evals-lab 0.15.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +53 -0
- package/README.md +35 -14
- package/bin/evals-lab.js +8 -1
- package/bin/presets.js +234 -0
- package/lab/VERSION +1 -1
- package/lab/demo/pipelines/demo-1.json +105 -106
- package/lab/demo/pipelines/demo-2.json +105 -106
- package/lab/demo/pipelines/documents.json +54 -51
- package/lab/demo/pipelines/sentiment.json +30 -30
- package/lab/demo/pipelines/weather-alerts.json +35 -35
- package/lab/demo/pipelines/wordings.json +33 -36
- package/lab/evals-core.mjs +815 -194
- package/lab/kinds/list.mjs +2 -2
- package/lab/metrics/builtin.mjs +24 -24
- package/lab/run-evals.js +34 -56
- package/lab/server.py +350 -180
- package/lab/web/dist/assets/gallery-ICJNZQIZ.js +5 -0
- package/lab/web/dist/assets/{gallery-eVJoaK-6.css → gallery-bG2vNzKT.css} +1 -1
- package/lab/web/dist/assets/main-BOkQWswU.css +1 -0
- package/lab/web/dist/assets/main-rJDsPFWT.js +18 -0
- package/lab/web/dist/assets/tokens-C5ist-xG.js +64 -0
- package/lab/web/dist/assets/tokens-CuK6hUMK.css +1 -0
- package/lab/web/dist/gallery.html +4 -4
- package/lab/web/dist/index.html +4 -4
- package/package.json +1 -1
- package/lab/presets.json +0 -62
- package/lab/web/dist/assets/gallery-Cy8oycht.js +0 -5
- package/lab/web/dist/assets/main-B-LGmhPu.css +0 -1
- package/lab/web/dist/assets/main-wqNvXpWp.js +0 -15
- package/lab/web/dist/assets/tokens-CWVzmtid.css +0 -1
- package/lab/web/dist/assets/tokens-mCS6IZGh.js +0 -64
package/lab/evals-core.mjs
CHANGED
|
@@ -255,7 +255,9 @@ import list, { ECHO, CASE_MODES, legacyTagsOut, legacyTagsModifiers, cleaningFil
|
|
|
255
255
|
|
|
256
256
|
/** A target (pipeline-model §16): one column through every job, compared
|
|
257
257
|
side by side in Results -- what version 9 called a scenario. Its step in
|
|
258
|
-
each job says what it sends there.
|
|
258
|
+
each job says what it sends there. From version 20 a Target is the
|
|
259
|
+
derived view of a Variant over the pipeline's Request steps (jobsOf,
|
|
260
|
+
targetsOf); the stored shape is the flat step list and its Variants (§20). */
|
|
259
261
|
|
|
260
262
|
|
|
261
263
|
|
|
@@ -270,6 +272,95 @@ import list, { ECHO, CASE_MODES, legacyTagsOut, legacyTagsModifiers, cleaningFil
|
|
|
270
272
|
|
|
271
273
|
|
|
272
274
|
|
|
275
|
+
// ---- the flat step list and Variants (version 20, pipeline-model §20) --------
|
|
276
|
+
//
|
|
277
|
+
// Version 20 reshapes a pipeline from jobs (each a stage holding every
|
|
278
|
+
// Target's step) into one ordered list of steps with Variants beside it. The
|
|
279
|
+
// steps are Get items, Request N, Read Response N and Filter; a Request holds
|
|
280
|
+
// the settings every Variant shares, and each Variant overrides any of them
|
|
281
|
+
// per Request. jobsOf / targetsOf derive the version-19 jobs and targets from
|
|
282
|
+
// this shape, so every reader (the page, the runner, the server through the
|
|
283
|
+
// core) keeps reading one shape; withJobs / withTargets serialise an edit
|
|
284
|
+
// back. The encoding is a verbatim re-packaging of jobs and targets, so a
|
|
285
|
+
// round-trip is byte-exact and stagesFor gives the same stages (#669).
|
|
286
|
+
|
|
287
|
+
/** Get items: the items a run goes over, before Request 1. There can be
|
|
288
|
+
several, each adding more items (§20); the upgrade from 19 makes one from
|
|
289
|
+
job 1's Attach Content. */
|
|
290
|
+
|
|
291
|
+
|
|
292
|
+
|
|
293
|
+
|
|
294
|
+
|
|
295
|
+
|
|
296
|
+
/** A Request: the settings every Variant shares in this job -- its base cell
|
|
297
|
+
(Variant A's: what it asks and of whom, its words, and an HTTP Request's
|
|
298
|
+
template), whether the item's image goes with it, the token mappings its
|
|
299
|
+
words resolve under, and the flow step its records are calls of, for a
|
|
300
|
+
workflow Request. Each Variant overrides its cell (Variant.overrides); the
|
|
301
|
+
image and the tokens are the Request's alone. */
|
|
302
|
+
|
|
303
|
+
|
|
304
|
+
|
|
305
|
+
|
|
306
|
+
|
|
307
|
+
|
|
308
|
+
|
|
309
|
+
|
|
310
|
+
|
|
311
|
+
|
|
312
|
+
|
|
313
|
+
|
|
314
|
+
|
|
315
|
+
|
|
316
|
+
|
|
317
|
+
|
|
318
|
+
/** Read Response N: how Request N's response is read -- the flow's reading of
|
|
319
|
+
it (`readAs`, a workflow Request's) and the output kind it is validated as
|
|
320
|
+
(`out`, absent for plain text). Comes after its Request. */
|
|
321
|
+
|
|
322
|
+
|
|
323
|
+
|
|
324
|
+
|
|
325
|
+
|
|
326
|
+
|
|
327
|
+
|
|
328
|
+
|
|
329
|
+
|
|
330
|
+
|
|
331
|
+
|
|
332
|
+
|
|
333
|
+
|
|
334
|
+
/** Filter: the filter sets a Request's response is cleaned and gated by, in
|
|
335
|
+
run order around its Response Format (the version-19 `filters` step, now
|
|
336
|
+
its own step in the list). Comes after its Request. */
|
|
337
|
+
|
|
338
|
+
|
|
339
|
+
|
|
340
|
+
|
|
341
|
+
|
|
342
|
+
|
|
343
|
+
|
|
344
|
+
|
|
345
|
+
|
|
346
|
+
/** One step of a version-20 pipeline. */
|
|
347
|
+
|
|
348
|
+
|
|
349
|
+
/** A Variant (version 20): a column through every Request, at most four. It
|
|
350
|
+
overrides a Request's shared settings, keyed by the Request's id -- AI
|
|
351
|
+
Connection, Model, Profile, Prompt (or Body) and a workflow Request's
|
|
352
|
+
Answered by (the cell's `type`); Send image and Tokens are the Request's.
|
|
353
|
+
An empty override inherits the Request's base; Variant A (the first)
|
|
354
|
+
inherits every one. The Variant's own `connection` / `profile` are the fallback a Request with no
|
|
355
|
+
override of its own uses, as a version-19 target's were. */
|
|
356
|
+
|
|
357
|
+
|
|
358
|
+
|
|
359
|
+
|
|
360
|
+
|
|
361
|
+
|
|
362
|
+
|
|
363
|
+
|
|
273
364
|
/** What every eval carries whatever its type: an id minted once and never
|
|
274
365
|
shown, an optional name ("Eval 2" when blank), and whether the evals after
|
|
275
366
|
it still read what it failed on (docs/pipeline-model.md §3). */
|
|
@@ -410,8 +501,13 @@ export const formatReadsItems = (kind ) =>
|
|
|
410
501
|
|
|
411
502
|
|
|
412
503
|
|
|
413
|
-
|
|
414
|
-
|
|
504
|
+
|
|
505
|
+
|
|
506
|
+
|
|
507
|
+
|
|
508
|
+
|
|
509
|
+
|
|
510
|
+
|
|
415
511
|
|
|
416
512
|
|
|
417
513
|
|
|
@@ -964,12 +1060,18 @@ export function kept(text , max = KEPT_CHARS) {
|
|
|
964
1060
|
|
|
965
1061
|
|
|
966
1062
|
|
|
1063
|
+
|
|
1064
|
+
|
|
1065
|
+
|
|
967
1066
|
|
|
968
1067
|
|
|
969
1068
|
/** One item of a queue row's results: a file or the pasted text. */
|
|
970
1069
|
|
|
971
1070
|
|
|
972
1071
|
|
|
1072
|
+
|
|
1073
|
+
|
|
1074
|
+
|
|
973
1075
|
|
|
974
1076
|
|
|
975
1077
|
|
|
@@ -2134,7 +2236,7 @@ function ollamaType(id , label , cloud ) {
|
|
|
2134
2236
|
description: cloud ? "Ollama's cloud at ollama.com, with a key." : "Ollama on a machine of your own; it takes no key.",
|
|
2135
2237
|
settings: [
|
|
2136
2238
|
{ key: "seed", label: "Seed", control: "text", default: "", hint: "" },
|
|
2137
|
-
{ key: "nPredict", label: "
|
|
2239
|
+
{ key: "nPredict", label: "Response tokens", control: "text", default: "", hint: "" },
|
|
2138
2240
|
{ key: "temperature", label: "Temperature", control: "text", default: "", hint: "" },
|
|
2139
2241
|
{ key: "repeatPenalty", label: "Repeat penalty", control: "text", default: "", hint: "" },
|
|
2140
2242
|
],
|
|
@@ -2187,7 +2289,7 @@ const sendsNothing = () => { throw new Error("Echo answers from the item
|
|
|
2187
2289
|
// stays so a stored run, or a profile made before, still reads.
|
|
2188
2290
|
CONNECTION_TYPES.echo = {
|
|
2189
2291
|
id: "echo", label: "Echo", settings: [], keyless: true, picker: false,
|
|
2190
|
-
description: "No model: answers with each text item itself, to grade
|
|
2292
|
+
description: "No model: answers with each text item itself, to grade responses recorded earlier.",
|
|
2191
2293
|
local: (item, sent) => item.text ?? sent,
|
|
2192
2294
|
request: sendsNothing, parseReply: sendsNothing, listModels: sendsNothing, parseModels: () => [],
|
|
2193
2295
|
};
|
|
@@ -2207,7 +2309,7 @@ function recordedRaw(record ) {
|
|
|
2207
2309
|
// the flow reads a reply; `local` is the no-Read-as fallback.
|
|
2208
2310
|
CONNECTION_TYPES.recorded = {
|
|
2209
2311
|
id: "recorded", label: "Recorded", settings: [], keyless: true, picker: false, replays: true,
|
|
2210
|
-
description: "Answers each
|
|
2312
|
+
description: "Answers each item with the response it got when it was recorded. Sends nothing.",
|
|
2211
2313
|
local: (_item, _sent, record) => recordedRaw(record),
|
|
2212
2314
|
request: sendsNothing, parseReply: sendsNothing, listModels: sendsNothing, parseModels: () => [],
|
|
2213
2315
|
};
|
|
@@ -2251,7 +2353,7 @@ CONNECTION_TYPES["openai-compatible"] = {
|
|
|
2251
2353
|
// A reasoning model refuses a temperature.
|
|
2252
2354
|
{ key: "temperature", label: "Temperature", control: "text", default: "", hint: "",
|
|
2253
2355
|
appliesTo: (conn) => !/^(gpt-5|o\d)/i.test(String(conn.model ?? "")) },
|
|
2254
|
-
{ key: "nPredict", label: "
|
|
2356
|
+
{ key: "nPredict", label: "Response tokens", control: "text", default: "", hint: "" },
|
|
2255
2357
|
],
|
|
2256
2358
|
// /v1/chat/completions, covering OpenAI, OpenRouter, vLLM, LM Studio.
|
|
2257
2359
|
// max_completion_tokens stays with this type, as the hosted providers
|
|
@@ -2295,7 +2397,7 @@ CONNECTION_TYPES.anthropic = {
|
|
|
2295
2397
|
settings: [
|
|
2296
2398
|
{ key: "temperature", label: "Temperature", control: "text", default: "", hint: "",
|
|
2297
2399
|
appliesTo: (conn) => !CLAUDE_FIXED_SAMPLING.test(String(conn.model ?? "")) },
|
|
2298
|
-
{ key: "maxTokens", label: "
|
|
2400
|
+
{ key: "maxTokens", label: "Response tokens", control: "text", default: "", hint: "" },
|
|
2299
2401
|
],
|
|
2300
2402
|
// Asked words (a Prompt step, a grader) or sent a flow's whole request (an
|
|
2301
2403
|
// HTTP Request step): one profile and one key do both.
|
|
@@ -2360,7 +2462,7 @@ CONNECTION_TYPES["llama.cpp"] = {
|
|
|
2360
2462
|
settings: [
|
|
2361
2463
|
{ key: "temperature", label: "Temperature", control: "text", default: "", hint: "" },
|
|
2362
2464
|
{ key: "seed", label: "Seed", control: "text", default: "", hint: "" },
|
|
2363
|
-
{ key: "nPredict", label: "
|
|
2465
|
+
{ key: "nPredict", label: "Response tokens", control: "text", default: "", hint: "" },
|
|
2364
2466
|
{ key: "repeatPenalty", label: "Repeat penalty", control: "text", default: "", hint: "" },
|
|
2365
2467
|
{ key: "frequencyPenalty", label: "Frequency penalty", control: "text", default: "", hint: "" },
|
|
2366
2468
|
{ key: "presencePenalty", label: "Presence penalty", control: "text", default: "", hint: "" },
|
|
@@ -2374,7 +2476,7 @@ CONNECTION_TYPES["llama.cpp"] = {
|
|
|
2374
2476
|
choices: [{ value: "on", label: "on" }, { value: "off", label: "off" }], hint: "" },
|
|
2375
2477
|
{ key: "thinking", label: "Thinking", control: "select", default: "off",
|
|
2376
2478
|
choices: [{ value: "on", label: "on" }, { value: "off", label: "off" }], hint: "" },
|
|
2377
|
-
{ key: "rejectLoops", label: "Reject looping
|
|
2479
|
+
{ key: "rejectLoops", label: "Reject looping responses", control: "select", default: "off",
|
|
2378
2480
|
choices: [{ value: "on", label: "on" }, { value: "off", label: "off" }], hint: "" },
|
|
2379
2481
|
],
|
|
2380
2482
|
request(conn, prompt, dataUrl, base, key) {
|
|
@@ -3104,7 +3206,7 @@ function singleRules(test ) {
|
|
|
3104
3206
|
}
|
|
3105
3207
|
|
|
3106
3208
|
/** "3 of 12 replies", or "the reply" where there was only one. */
|
|
3107
|
-
const ofReplies = (k , n ) => (n === 1 ? "the
|
|
3209
|
+
const ofReplies = (k , n ) => (n === 1 ? "the response" : `${k} of ${n} responses`);
|
|
3108
3210
|
const quoted = (s ) => `"${s.trim().slice(0, 60)}"`;
|
|
3109
3211
|
|
|
3110
3212
|
/**
|
|
@@ -3225,7 +3327,7 @@ function jobProblem(stages , call
|
|
|
3225
3327
|
const kind = OUTPUT_KINDS[kinds[from] ] ;
|
|
3226
3328
|
if (sub && !(kind.offers || []).includes(sub)) {
|
|
3227
3329
|
return `and stage ${from + 1} answers with ${kind.noun || kinds[from]}, which has no ${sub} — `
|
|
3228
|
-
+ `{${m ? `stage${from + 1}` : "reply"}} is its
|
|
3330
|
+
+ `{${m ? `stage${from + 1}` : "reply"}} is its response`;
|
|
3229
3331
|
}
|
|
3230
3332
|
return null;
|
|
3231
3333
|
};
|
|
@@ -3461,7 +3563,7 @@ function applyModifiers (list , kind ,
|
|
|
3461
3563
|
// pins one response (#596).
|
|
3462
3564
|
// 19: a job's filters are one list, run in order around its Response Format,
|
|
3463
3565
|
// and filters are seven verbs (#615).
|
|
3464
|
-
const PIPELINE_VERSION =
|
|
3566
|
+
const PIPELINE_VERSION = 20 ;
|
|
3465
3567
|
|
|
3466
3568
|
// Plain objects, so an entry is added by assignment and a reader never needs
|
|
3467
3569
|
// to know which registered it.
|
|
@@ -3581,9 +3683,9 @@ const clone = (v ) => JSON.parse(JSON.stringify(v));
|
|
|
3581
3683
|
// A reply read as itself: anything at all, handed on untouched.
|
|
3582
3684
|
OUTPUT_KINDS.text = {
|
|
3583
3685
|
label: "Plain text", noun: "text", offers: [],
|
|
3584
|
-
description: "Keeps the
|
|
3686
|
+
description: "Keeps the response as it came back, whole.",
|
|
3585
3687
|
read: reply => ({ value: String(reply ?? "") }),
|
|
3586
|
-
handsOn: (value ) => (value.trim() ? null : "empty
|
|
3688
|
+
handsOn: (value ) => (value.trim() ? null : "empty response"),
|
|
3587
3689
|
} ;
|
|
3588
3690
|
|
|
3589
3691
|
// ---- the connection a profile reference resolves to ------------------------
|
|
@@ -3694,6 +3796,28 @@ function connectionOf(p ) {
|
|
|
3694
3796
|
return out;
|
|
3695
3797
|
}
|
|
3696
3798
|
|
|
3799
|
+
/** The settings a Prompt Request may hold of its own (#672): any connection
|
|
3800
|
+
type's, but those that say how it is reached and the model, its own field. */
|
|
3801
|
+
const requestSettingKeys = () => [...new Set(Object.values(CONNECTION_TYPES)
|
|
3802
|
+
.flatMap(t => t.settings.filter(s => s.scope !== "connection" && s.key !== "model").map(s => s.key)))];
|
|
3803
|
+
|
|
3804
|
+
/** [conn] with a Request's own settings over it, as connectionOf carries
|
|
3805
|
+
them: a preset kept on a Variant (#672). One left blank is not sent, so a
|
|
3806
|
+
preset clears what it does not set, as Apply preset does a profile's. */
|
|
3807
|
+
function withOwnSettings (conn , own ) {
|
|
3808
|
+
if (!own || !Object.keys(own).length) return conn;
|
|
3809
|
+
const out = { ...conn };
|
|
3810
|
+
const options = { ...(conn.options || {}) };
|
|
3811
|
+
for (const [k, v] of Object.entries(own)) {
|
|
3812
|
+
const set = String(v ?? "").trim() !== "";
|
|
3813
|
+
if (k === "temperature") { if (set) out.temperature = v; else delete out.temperature; }
|
|
3814
|
+
else if (set) options[k] = v;
|
|
3815
|
+
else delete options[k];
|
|
3816
|
+
}
|
|
3817
|
+
if (Object.keys(options).length) out.options = options; else delete out.options;
|
|
3818
|
+
return out;
|
|
3819
|
+
}
|
|
3820
|
+
|
|
3697
3821
|
/** A run's connection as one flat set of settings, the shape a profile has. */
|
|
3698
3822
|
function connectionSettings(c ) {
|
|
3699
3823
|
const { options, ...rest } = c || {};
|
|
@@ -3776,7 +3900,7 @@ function answerOfKey(key )
|
|
|
3776
3900
|
|
|
3777
3901
|
/** Whom target [i] asks in job [k]: its step's own connection or profile,
|
|
3778
3902
|
where it names either, else the target's. */
|
|
3779
|
-
function targetAnswerOf(doc
|
|
3903
|
+
function targetAnswerOf(doc , i , k ) {
|
|
3780
3904
|
const t = targetsOf(doc)[i];
|
|
3781
3905
|
const st = targetStepOf(doc, i, k);
|
|
3782
3906
|
const own = st?.connection != null || st?.profile != null;
|
|
@@ -3787,7 +3911,7 @@ function targetAnswerOf(doc
|
|
|
3787
3911
|
}
|
|
3788
3912
|
|
|
3789
3913
|
/** The answer key of target [i] in job [k], or undefined where it asks none. */
|
|
3790
|
-
function targetAnswerKey(doc
|
|
3914
|
+
function targetAnswerKey(doc , i , k ) {
|
|
3791
3915
|
const a = targetAnswerOf(doc, i, k);
|
|
3792
3916
|
return a ? answerKey(a.connection.id, a.profile?.id) : undefined;
|
|
3793
3917
|
}
|
|
@@ -3868,9 +3992,9 @@ function encoderQuality(quality ) {
|
|
|
3868
3992
|
// ---- labels -----------------------------------------------------------------
|
|
3869
3993
|
// A name is optional everywhere; a blank one is the number the canvas has
|
|
3870
3994
|
// always shown, so a sentence names what the reader sees.
|
|
3871
|
-
const jobLabel = (doc
|
|
3872
|
-
(doc
|
|
3873
|
-
const scenarioLabel = (doc
|
|
3995
|
+
const jobLabel = (doc , k ) =>
|
|
3996
|
+
(jobsOf(doc)[k]?.name || "").trim() || `Job ${k + 1}`;
|
|
3997
|
+
const scenarioLabel = (doc , i ) => targetLabel(doc, i);
|
|
3874
3998
|
// The runner calls a job's link a stage; the model calls it a job.
|
|
3875
3999
|
const jobWords = (s ) => s && String(s).replace(/\bstage[ ](\d+)/g, "job $1");
|
|
3876
4000
|
|
|
@@ -3881,8 +4005,8 @@ function onlyFields(obj , at , allowed , bad
|
|
|
3881
4005
|
for (const k of Object.keys(obj)) {
|
|
3882
4006
|
if (allowed.includes(k)) continue;
|
|
3883
4007
|
bad.push(LOOKS_LIKE_A_KEY.test(k)
|
|
3884
|
-
? `${at} carries "${k}", and a key never goes in
|
|
3885
|
-
: `${at} has "${k}", which is not
|
|
4008
|
+
? `${at} carries "${k}", and a key never goes in an experiment — $${keyFrom || "EVALSLAB_API_KEY_<ID>"} supplies it`
|
|
4009
|
+
: `${at} has "${k}", which is not an experiment field`);
|
|
3886
4010
|
}
|
|
3887
4011
|
}
|
|
3888
4012
|
|
|
@@ -3915,9 +4039,9 @@ SOURCE_TYPES.workflow = {
|
|
|
3915
4039
|
icon: "workflow",
|
|
3916
4040
|
description: "A cloud flow, with records of its steps' calls from its run history.",
|
|
3917
4041
|
group: "Workflows",
|
|
3918
|
-
// Each item reads as
|
|
4042
|
+
// Each item reads as an item, as the Source's Items group names them, though
|
|
3919
4043
|
// the file holding one is a record.
|
|
3920
|
-
noun: "
|
|
4044
|
+
noun: "item",
|
|
3921
4045
|
uploads: [".json"],
|
|
3922
4046
|
sendsImage: false,
|
|
3923
4047
|
hasActions: true,
|
|
@@ -3959,9 +4083,9 @@ WORKFLOW_PLATFORMS["power-automate"] = { // platform: the Power Automate adapt
|
|
|
3959
4083
|
SOURCE_TYPES.dynamic = {
|
|
3960
4084
|
label: "Dynamic",
|
|
3961
4085
|
icon: "globe",
|
|
3962
|
-
description: "Data produced on demand -- an HTTP GET, an API, a file or a command -- captured as
|
|
4086
|
+
description: "Data produced on demand -- an HTTP GET, an API, a file or a command -- captured as Items.",
|
|
3963
4087
|
group: "Dynamic",
|
|
3964
|
-
noun: "
|
|
4088
|
+
noun: "item", // the generic default; the provider's own overrides it
|
|
3965
4089
|
uploads: [".txt", ".json", ".csv", ".md"],
|
|
3966
4090
|
sendsImage: false,
|
|
3967
4091
|
hasActions: false,
|
|
@@ -4090,7 +4214,7 @@ DYNAMIC_PROVIDERS["command"] = { // provider: the Command adapter's own id
|
|
|
4090
4214
|
id: "command", // provider: its id, the row's config.provider
|
|
4091
4215
|
label: "Command",
|
|
4092
4216
|
icon: "terminal",
|
|
4093
|
-
noun: "
|
|
4217
|
+
noun: "item",
|
|
4094
4218
|
produceIcon: "square-terminal",
|
|
4095
4219
|
tier: "local",
|
|
4096
4220
|
config: ["command", "args"],
|
|
@@ -4238,8 +4362,8 @@ CONTENT_TYPES.text = {
|
|
|
4238
4362
|
// model asked the prompt itself, or Echo answering with it, the prompt then
|
|
4239
4363
|
// being the recorded reply.
|
|
4240
4364
|
CONTENT_TYPES.prompt = {
|
|
4241
|
-
label: "
|
|
4242
|
-
description: "No item: each
|
|
4365
|
+
label: "(No content)",
|
|
4366
|
+
description: "No item: each variant's prompt is sent on its own.",
|
|
4243
4367
|
inline: true,
|
|
4244
4368
|
bare: true,
|
|
4245
4369
|
fields: ["type"],
|
|
@@ -4303,9 +4427,9 @@ LEGACY_TESTS.graded = {
|
|
|
4303
4427
|
validate(t, ctx, bad){
|
|
4304
4428
|
const d = t.dataset;
|
|
4305
4429
|
if (!isRef(d) || (d.version != null && !isStr(d.version))) {
|
|
4306
|
-
return void bad.push("a graded eval has to name its
|
|
4430
|
+
return void bad.push("a graded eval has to name its Library eval as { id, name }");
|
|
4307
4431
|
}
|
|
4308
|
-
if (ctx.groups && !ctx.groups.some(x => x.id === d.id)) bad.push(`Eval
|
|
4432
|
+
if (ctx.groups && !ctx.groups.some(x => x.id === d.id)) bad.push(`Eval ${d.name || d.id} not found`);
|
|
4309
4433
|
},
|
|
4310
4434
|
};
|
|
4311
4435
|
|
|
@@ -4507,9 +4631,9 @@ function readRun(list , run ) {
|
|
|
4507
4631
|
const off = each.filter(r => !r.pass);
|
|
4508
4632
|
const first = each[0];
|
|
4509
4633
|
if (!first) return [{ type: m.type, label: METRICS[m.type]?.label ?? m.type, weight: typeof m.weight === "number" ? m.weight : 1,
|
|
4510
|
-
pass: true, score: 1, reason: "no
|
|
4634
|
+
pass: true, score: 1, reason: "no responses" }];
|
|
4511
4635
|
return [{ ...first, pass: !off.length, score: each.length ? (each.length - off.length) / each.length : 1,
|
|
4512
|
-
reason: off.length ? `${off.length} of ${each.length}
|
|
4636
|
+
reason: off.length ? `${off.length} of ${each.length} responses: ${off[0] .reason}` : `all ${each.length} responses` }];
|
|
4513
4637
|
});
|
|
4514
4638
|
}
|
|
4515
4639
|
|
|
@@ -4518,7 +4642,7 @@ function readRun(list , run ) {
|
|
|
4518
4642
|
// pipeline-check.js hold the eval group to.
|
|
4519
4643
|
LEGACY_TESTS.metrics = {
|
|
4520
4644
|
label: "Metrics",
|
|
4521
|
-
description: "Checks of each
|
|
4645
|
+
description: "Checks of each response, or of the whole run, scored all-metrics-pass or in points.",
|
|
4522
4646
|
fields: ["type", "metrics", "mode", "threshold", "grader", "dataset", "over"],
|
|
4523
4647
|
// A metric that reads terms (a case) reads only a kind that yields them.
|
|
4524
4648
|
accepts: t => (Array.isArray(t.metrics) && t.metrics.some((m ) => METRICS[m?.type]?.needsTerms)
|
|
@@ -4533,8 +4657,8 @@ LEGACY_TESTS.metrics = {
|
|
|
4533
4657
|
}
|
|
4534
4658
|
// The dataset whose cases a case metric reads, and whose own metrics join.
|
|
4535
4659
|
if (t.dataset != null) {
|
|
4536
|
-
if (!isRef(t.dataset) || (t.dataset.version != null && !isStr(t.dataset.version))) bad.push("the Metrics name their
|
|
4537
|
-
else if (ctx.groups && !ctx.groups.some(x => x.id === t.dataset.id)) bad.push(`Eval
|
|
4660
|
+
if (!isRef(t.dataset) || (t.dataset.version != null && !isStr(t.dataset.version))) bad.push("the Metrics name their eval as { id, name }");
|
|
4661
|
+
else if (ctx.groups && !ctx.groups.some(x => x.id === t.dataset.id)) bad.push(`Eval ${t.dataset.name || t.dataset.id} not found`);
|
|
4538
4662
|
}
|
|
4539
4663
|
if (!SCORING_MODES.includes(t.mode)) bad.push(`the Metrics are scored ${SCORING_MODES.join(" or ")}`);
|
|
4540
4664
|
if (t.mode === "weighted" && typeof t.threshold !== "number") bad.push("weighted Metrics need a threshold: the points an item has to reach");
|
|
@@ -4626,8 +4750,8 @@ const wholeRunGroup = (t ) => isObj(t.own) && Array.isArray(t
|
|
|
4626
4750
|
// group does (readGroup), or -- a private group of Whole run metrics alone --
|
|
4627
4751
|
// the whole run (readGroupRun).
|
|
4628
4752
|
EVAL_TYPES.group = {
|
|
4629
|
-
label: "Eval
|
|
4630
|
-
description: "Checks of each
|
|
4753
|
+
label: "Eval",
|
|
4754
|
+
description: "Checks of each response, or of the whole run, from an eval: linked from the Library, or this experiment's own.",
|
|
4631
4755
|
fields: ["type", "group", "pin", "own", "grader"],
|
|
4632
4756
|
// A metric that reads terms (a case) reads only a kind that yields them.
|
|
4633
4757
|
accepts: t => ([...ownList(t)].some((m ) => METRICS[m?.type]?.needsTerms)
|
|
@@ -4640,17 +4764,17 @@ EVAL_TYPES.group = {
|
|
|
4640
4764
|
const named = (ref , what ) => {
|
|
4641
4765
|
const n = isObj(ref) ? ref.n : undefined;
|
|
4642
4766
|
if (!isRef(ref) || (ref.version != null && !isStr(ref.version)) || (n != null && !Number.isInteger(n))) {
|
|
4643
|
-
return void bad.push(`${what} names its eval
|
|
4767
|
+
return void bad.push(`${what} names its eval as { id, name }`);
|
|
4644
4768
|
}
|
|
4645
4769
|
onlyFields(ref, what, ["id", "name", "version", "n"], bad);
|
|
4646
4770
|
const held = ctx.groups?.find(x => x.id === ref.id);
|
|
4647
|
-
if (ctx.groups && !held) bad.push(`Eval
|
|
4771
|
+
if (ctx.groups && !held) bad.push(`Eval ${ref.name || ref.id} not found`);
|
|
4648
4772
|
return held;
|
|
4649
4773
|
};
|
|
4650
|
-
if (t.pin != null && !(Number.isInteger(t.pin) && t.pin > 0)) bad.push("a pin is
|
|
4774
|
+
if (t.pin != null && !(Number.isInteger(t.pin) && t.pin > 0)) bad.push("a pin is an eval's version: a whole number from 1");
|
|
4651
4775
|
if (t.grader != null && !isRef(t.grader)) bad.push("the link names its grader as { id, name }");
|
|
4652
4776
|
if (g != null) {
|
|
4653
|
-
if (own != null) return void bad.push("an eval links a Library
|
|
4777
|
+
if (own != null) return void bad.push("an eval links a Library eval or holds its own, not both");
|
|
4654
4778
|
const held = named(g, "the link");
|
|
4655
4779
|
if (held && t.pin != null && typeof held.versions === "number" && t.pin > held.versions) {
|
|
4656
4780
|
bad.push(`${g.name || g.id} has no version ${t.pin}`);
|
|
@@ -4662,17 +4786,17 @@ EVAL_TYPES.group = {
|
|
|
4662
4786
|
graderProblems(grader, ctx, bad);
|
|
4663
4787
|
return;
|
|
4664
4788
|
}
|
|
4665
|
-
if (!isObj(own)) return void bad.push("an eval links a Library
|
|
4666
|
-
if (t.pin != null) bad.push("
|
|
4667
|
-
if (own.version !== DATASET_BODY_VERSION) bad.push(`the
|
|
4668
|
-
for (const key of ["scoring", "every", "run", "cases"]) if (own[key] == null) bad.push(`the
|
|
4669
|
-
onlyFields(own, "the
|
|
4789
|
+
if (!isObj(own)) return void bad.push("an eval links a Library eval, or holds its own");
|
|
4790
|
+
if (t.pin != null) bad.push("an eval of this experiment's own versions with it, so it is never pinned");
|
|
4791
|
+
if (own.version !== DATASET_BODY_VERSION) bad.push(`the eval's own body is version ${DATASET_BODY_VERSION}`);
|
|
4792
|
+
for (const key of ["scoring", "every", "run", "cases"]) if (own[key] == null) bad.push(`the eval's own body has no ${key}`);
|
|
4793
|
+
onlyFields(own, "the eval's own body", ["version", "scoring", "grader", "every", "run", "cases", "casesFrom"], bad);
|
|
4670
4794
|
if (bad.length) return;
|
|
4671
4795
|
bad.push(...validateEvals(own));
|
|
4672
|
-
if (own.casesFrom != null) named(own.casesFrom, "
|
|
4796
|
+
if (own.casesFrom != null) named(own.casesFrom, "Eval cases from");
|
|
4673
4797
|
// Read item by item or over the run: one section at a time, until a
|
|
4674
4798
|
// group's Whole run is reported beside its items (#233).
|
|
4675
|
-
if (own.run.length && !wholeRunGroup(t)) bad.push("the
|
|
4799
|
+
if (own.run.length && !wholeRunGroup(t)) bad.push("the eval's own body reads each item or the whole run, not both");
|
|
4676
4800
|
// The group's own grader, or the lab's.
|
|
4677
4801
|
const grader = isRef(own.grader) ? own.grader : isRef(ctx.grader) ? ctx.grader : null;
|
|
4678
4802
|
if (gradedIn(own.every) && !grader) bad.push("a model-graded metric needs a grader: pick one on the eval, or make a connection the lab's grader");
|
|
@@ -4737,8 +4861,8 @@ function caseIn(body , item ) {
|
|
|
4737
4861
|
// ignoring case, as the Single Test did.
|
|
4738
4862
|
METRICS["has-items"] = {
|
|
4739
4863
|
label: "Has items",
|
|
4740
|
-
family: "The
|
|
4741
|
-
description: "Passes when the
|
|
4864
|
+
family: "The response's text",
|
|
4865
|
+
description: "Passes when the response holds all, or any, of the items listed.",
|
|
4742
4866
|
options: [{ key: "values", label: "Items", type: "textarea" },
|
|
4743
4867
|
{ key: "need", label: "Needs", type: "select", choices: [{ value: "all", label: "all" }, { value: "any", label: "any" }] }],
|
|
4744
4868
|
defaults: () => ({ values: "", need: "all" }),
|
|
@@ -4766,8 +4890,8 @@ const parseCountOf = (m ) =>
|
|
|
4766
4890
|
// A reply's length in characters, once trimmed.
|
|
4767
4891
|
METRICS.length = {
|
|
4768
4892
|
label: "Length",
|
|
4769
|
-
family: "The
|
|
4770
|
-
description: "Compares the
|
|
4893
|
+
family: "The response's text",
|
|
4894
|
+
description: "Compares the response's length in characters.",
|
|
4771
4895
|
options: [{ key: "op", label: "Is", type: "select", choices: LENGTH_OPS.map(([value, label]) => ({ value, label })) },
|
|
4772
4896
|
{ key: "n", label: "Characters", type: "number" }],
|
|
4773
4897
|
defaults: () => ({ op: "exactly", n: 0 }),
|
|
@@ -4787,7 +4911,7 @@ METRICS.length = {
|
|
|
4787
4911
|
METRICS["parse-count"] = {
|
|
4788
4912
|
label: "Parses",
|
|
4789
4913
|
family: "The result",
|
|
4790
|
-
description: "Passes when the
|
|
4914
|
+
description: "Passes when the response reads as the format chosen, holding the number of results set.",
|
|
4791
4915
|
// Unformatted is one result a reply, as the Single Test counted it.
|
|
4792
4916
|
options: [{ key: "parse", label: "As", type: "select", choices: [{ value: "", label: "Unformatted" }, { value: "csv", label: "csv" },
|
|
4793
4917
|
{ value: "json", label: "json" }, { value: "xml", label: "xml" }] },
|
|
@@ -4940,11 +5064,11 @@ function metricCheck(m ) {
|
|
|
4940
5064
|
// Attach Content: the items a run goes over, job 1's first step.
|
|
4941
5065
|
STEP_TYPES.attachContent = {
|
|
4942
5066
|
label: "Attach Content", slot: "content", rank: 0,
|
|
4943
|
-
description: "The items the run goes through: each is sent to every
|
|
5067
|
+
description: "The items the run goes through: each is sent to every variant.",
|
|
4944
5068
|
in: null, out: "items",
|
|
4945
5069
|
apply: "itemContent", // run-evals.js, through its file-type registry
|
|
4946
5070
|
fields: ["type", "content"],
|
|
4947
|
-
firstJobOnly: "only job 1 attaches content -- a later job is handed the
|
|
5071
|
+
firstJobOnly: "only job 1 attaches content -- a later job is handed the response before it",
|
|
4948
5072
|
validate(step, ctx, bad){
|
|
4949
5073
|
const c = step?.content;
|
|
4950
5074
|
if (!c) return void bad.push("set the content first");
|
|
@@ -5004,12 +5128,21 @@ STEP_TYPES.tokenMappings = {
|
|
|
5004
5128
|
// Prompt: a target's words, asked of its model.
|
|
5005
5129
|
STEP_TYPES.prompt = {
|
|
5006
5130
|
label: "Prompt", slot: "target", asks: "prompt", modelFrom: "profile",
|
|
5007
|
-
description: "Asks each
|
|
5131
|
+
description: "Asks each variant's model its prompt about the item.",
|
|
5008
5132
|
in: "item", out: "text",
|
|
5009
5133
|
apply: "runPipeline",
|
|
5010
|
-
|
|
5134
|
+
// A model and settings of its own go over its connection's (stagesFor):
|
|
5135
|
+
// a preset kept on a Variant (#672).
|
|
5136
|
+
fields: ["type", "prompt", "connection", "profile", "from", "model", "settings"],
|
|
5011
5137
|
validate(step, _ctx, bad, at){
|
|
5012
5138
|
if (!isStr(step.prompt)) bad.push(`${at}: prompt has to be text`);
|
|
5139
|
+
if (step.model != null && !isStr(step.model)) bad.push(`${at}: model has to be text`);
|
|
5140
|
+
if (step.settings != null) {
|
|
5141
|
+
if (!isObj(step.settings) || !Object.values(step.settings).every(isStr)) bad.push(`${at}: settings are names to text`);
|
|
5142
|
+
else for (const k of Object.keys(step.settings)) {
|
|
5143
|
+
if (!requestSettingKeys().includes(k)) bad.push(`${at}: no connection type has a setting ${k}`);
|
|
5144
|
+
}
|
|
5145
|
+
}
|
|
5013
5146
|
},
|
|
5014
5147
|
};
|
|
5015
5148
|
|
|
@@ -5031,7 +5164,7 @@ function keyField(v , at = "body") {
|
|
|
5031
5164
|
}
|
|
5032
5165
|
STEP_TYPES.httpRequest = {
|
|
5033
5166
|
label: "HTTP Request", slot: "target", asks: "request", verbatim: true, modelFrom: "step",
|
|
5034
|
-
description: "Sends a Power Automate step's own request, rebuilt from each record, with each
|
|
5167
|
+
description: "Sends a Power Automate step's own request, rebuilt from each record, with each variant's words in it.",
|
|
5035
5168
|
firstJobOnly: "an HTTP Request is sent from its item's record, so it is job 1's",
|
|
5036
5169
|
// A new target starts from the flow's own words and fields, read by the
|
|
5037
5170
|
// job's HTTP API ([step] carries it: requestStepOf).
|
|
@@ -5052,7 +5185,7 @@ STEP_TYPES.httpRequest = {
|
|
|
5052
5185
|
// A key is the Setup profile's, sent by its Key header; one in the
|
|
5053
5186
|
// template would be kept in the pipeline and every run of it.
|
|
5054
5187
|
const hit = keyField(step.body) ?? keyField(step.query, "query");
|
|
5055
|
-
if (hit) bad.push(`${at} carries a key at ${hit}, and a key never goes in
|
|
5188
|
+
if (hit) bad.push(`${at} carries a key at ${hit}, and a key never goes in an experiment -- the connection sends it`);
|
|
5056
5189
|
if (!api) return;
|
|
5057
5190
|
// Its fields, where its HTTP API has a place for them.
|
|
5058
5191
|
for (const f of ["system", "model", "prefill"] ) {
|
|
@@ -5083,7 +5216,7 @@ STEP_TYPES.httpRequest = {
|
|
|
5083
5216
|
// against; over Prompt only they are the reply.
|
|
5084
5217
|
STEP_TYPES.echo = {
|
|
5085
5218
|
label: "Echo", slot: "target", asks: "nothing", local: true, replaces: "echo",
|
|
5086
|
-
description: "Sends nothing: each text item is its own
|
|
5219
|
+
description: "Sends nothing: each text item is its own response, to grade responses recorded earlier.",
|
|
5087
5220
|
firstJobOnly: "Echo answers only job 1, with the item's own text",
|
|
5088
5221
|
in: "item", out: "text",
|
|
5089
5222
|
apply: "runPipeline",
|
|
@@ -5097,7 +5230,7 @@ STEP_TYPES.echo = {
|
|
|
5097
5230
|
// the flow reads one. No profile, no prompt -- the record is the reply.
|
|
5098
5231
|
STEP_TYPES.recorded = {
|
|
5099
5232
|
label: "Recorded", slot: "target", asks: "nothing", local: true, replaces: "recorded",
|
|
5100
|
-
firstJobOnly: "
|
|
5233
|
+
firstJobOnly: "an item's recorded response is job 1's",
|
|
5101
5234
|
in: "item", out: "text",
|
|
5102
5235
|
apply: "runPipeline",
|
|
5103
5236
|
// `prompt` only so the step satisfies the target-step shape; it is never
|
|
@@ -5122,7 +5255,7 @@ const localConnOf = (entry )
|
|
|
5122
5255
|
// Read as: what the flow does to the reply before it reads it.
|
|
5123
5256
|
STEP_TYPES.readAs = {
|
|
5124
5257
|
label: "Read as", slot: "responses", rank: 0,
|
|
5125
|
-
description: "What the flow makes of the
|
|
5258
|
+
description: "What the flow makes of the response before it reads it.",
|
|
5126
5259
|
in: "text", out: "text",
|
|
5127
5260
|
apply: "runPipeline",
|
|
5128
5261
|
fields: ["type", "readAs"],
|
|
@@ -5133,8 +5266,8 @@ STEP_TYPES.readAs = {
|
|
|
5133
5266
|
|
|
5134
5267
|
// Response Format Validation: the kind a reply is read as.
|
|
5135
5268
|
STEP_TYPES.readReply = {
|
|
5136
|
-
label: "Read
|
|
5137
|
-
description: "How the
|
|
5269
|
+
label: "Read Response", slot: "responses", rank: 1,
|
|
5270
|
+
description: "How the response is read before evals and later jobs see it.",
|
|
5138
5271
|
in: "text", out: step => step.out?.kind,
|
|
5139
5272
|
// Read inside the stage: runPipeline parses each reply as it comes back.
|
|
5140
5273
|
apply: "runPipeline",
|
|
@@ -5153,7 +5286,7 @@ STEP_TYPES.readReply = {
|
|
|
5153
5286
|
// A modifier: one change to the parsed reply, after the kind that takes it.
|
|
5154
5287
|
STEP_TYPES.modifier = {
|
|
5155
5288
|
label: "Modifier", slot: "responses", rank: 2, many: true,
|
|
5156
|
-
description: "Changes the
|
|
5289
|
+
description: "Changes the response after it is read.",
|
|
5157
5290
|
in: "value", out: "value",
|
|
5158
5291
|
apply: "runPipeline",
|
|
5159
5292
|
fields: ["type", "modifier"],
|
|
@@ -5175,7 +5308,7 @@ STEP_TYPES.modifier = {
|
|
|
5175
5308
|
// so only its shape is checked here.
|
|
5176
5309
|
STEP_TYPES.filters = {
|
|
5177
5310
|
label: "Filter sets", slot: "responses", rank: 2,
|
|
5178
|
-
description: "Cleans and gates the
|
|
5311
|
+
description: "Cleans and gates the response with linked or private filter sets.",
|
|
5179
5312
|
in: "value", out: "value",
|
|
5180
5313
|
apply: "runPipeline",
|
|
5181
5314
|
fields: ["type", "filterSets", "format"],
|
|
@@ -5223,17 +5356,263 @@ function withStep(job , type , step ) {
|
|
|
5223
5356
|
return { ...job, steps: ordered(step ? [...steps, step] : steps) };
|
|
5224
5357
|
}
|
|
5225
5358
|
|
|
5226
|
-
|
|
5227
|
-
|
|
5228
|
-
|
|
5359
|
+
// ---- the flat step list and Variants: derive jobs/targets, serialise back ----
|
|
5360
|
+
// Version 20 stores a pipeline as its step list and Variants; every reader
|
|
5361
|
+
// reads the version-19 jobs and targets jobsOf / targetsOf derive, and every
|
|
5362
|
+
// edit is serialised back by withJobs / withTargets. The encoding is a
|
|
5363
|
+
// verbatim re-packaging, so the round-trip is byte-exact and stagesFor gives
|
|
5364
|
+
// the same stages (pipeline-model §20, #669). A doc still carrying `jobs` or
|
|
5365
|
+
// `targets` -- the shape upgradePipeline's chain builds before it serialises,
|
|
5366
|
+
// and any legacy caller -- is read through as it is.
|
|
5367
|
+
|
|
5368
|
+
/** What a pipeline/run reader takes: the version-20 step list and Variants,
|
|
5369
|
+
or the jobs-and-targets intermediate the upgrade builds before it
|
|
5370
|
+
serialises -- the accessors read whichever is present. */
|
|
5371
|
+
|
|
5372
|
+
|
|
5373
|
+
|
|
5374
|
+
const isGetItems = (st ) => isObj(st) && st.type === "getItems";
|
|
5375
|
+
const isRequest = (st ) => isObj(st) && st.type === "request";
|
|
5376
|
+
const isReadResponse = (st ) => isObj(st) && st.type === "readResponse";
|
|
5377
|
+
const isFilterV20 = (st ) => isObj(st) && st.type === "filter";
|
|
5378
|
+
|
|
5379
|
+
/** Whether two JSON values say the same thing: object key order is ignored
|
|
5380
|
+
and a key holding `undefined` counts as absent, as JSON writes them;
|
|
5381
|
+
array order counts. */
|
|
5382
|
+
function sameJson(a , b ) {
|
|
5383
|
+
if (a === b) return true;
|
|
5384
|
+
if (Array.isArray(a)) return Array.isArray(b) && a.length === b.length && a.every((v, i) => sameJson(v, b[i]));
|
|
5385
|
+
if (!isObj(a) || !isObj(b)) return false;
|
|
5386
|
+
const keys = (o ) => Object.keys(o).filter(k => o[k] !== undefined);
|
|
5387
|
+
const ka = keys(a);
|
|
5388
|
+
return ka.length === keys(b).length && ka.every(k => b[k] !== undefined && sameJson(a[k], b[k]));
|
|
5389
|
+
}
|
|
5390
|
+
|
|
5391
|
+
/** The override to carry Variant [cur] over the Request's base [base], or
|
|
5392
|
+
none where they are the same. A different step type, or a different set of
|
|
5393
|
+
fields, is carried whole (it holds `type`, so mergeCell replaces); a cell
|
|
5394
|
+
of the same shape carries only the fields whose value differs. */
|
|
5395
|
+
function cellDiff(base , cur ) {
|
|
5396
|
+
if (!cur) return undefined;
|
|
5397
|
+
if (!base) return clone(cur);
|
|
5398
|
+
const bk = Object.keys(base).sort().join("\u0000"), ck = Object.keys(cur).sort().join("\u0000");
|
|
5399
|
+
if (cur.type !== base.type || bk !== ck) return clone(cur);
|
|
5400
|
+
const ov = {};
|
|
5401
|
+
for (const key of Object.keys(cur)) if (key !== "type" && !sameJson((cur )[key], (base )[key])) ov[key] = clone((cur )[key]);
|
|
5402
|
+
return Object.keys(ov).length ? ov : undefined;
|
|
5403
|
+
}
|
|
5404
|
+
|
|
5405
|
+
/** A Variant's cell over a Request's base [base]: the base where there is no
|
|
5406
|
+
override, the override whole where it carries its own `type` (a different
|
|
5407
|
+
Answered by, or a different shape), else the base with the override's
|
|
5408
|
+
fields. */
|
|
5409
|
+
function mergeCell(base , ov ) {
|
|
5410
|
+
if (!ov) return clone(base ?? { type: "prompt", prompt: "" }) ;
|
|
5411
|
+
if ("type" in ov) return clone(ov) ;
|
|
5412
|
+
return { ...clone(base ?? { type: "prompt", prompt: "" }), ...clone(ov) } ;
|
|
5413
|
+
}
|
|
5414
|
+
|
|
5415
|
+
/** [jobs] and [targets] as a version-20 step list and Variants. Variant A's
|
|
5416
|
+
step in each job is the Request's base; a later Variant carries only its
|
|
5417
|
+
overrides. A verbatim re-packaging: jobsOf and targetsOf read it back whole.
|
|
5418
|
+
Given the list it replaces ([old]), the steps keep their order there and
|
|
5419
|
+
the steps the jobs do not hold (keepOrder). */
|
|
5420
|
+
function serialize(jobs , targets , old ) {
|
|
5421
|
+
const { steps, variants } = serializeFresh(jobs, targets);
|
|
5422
|
+
return { steps: keepOrder(old, steps), variants };
|
|
5423
|
+
}
|
|
5424
|
+
|
|
5425
|
+
/** [next], a freshly serialised step list, in the order [old] held its steps
|
|
5426
|
+
(§20): the person moves steps in the Steps list, and an edit made through
|
|
5427
|
+
the jobs view must not move them back. Each step of [old] takes its new
|
|
5428
|
+
version where [next] has one (its id kept): a Request by its id, a Read
|
|
5429
|
+
Response or Filter by the Request it reads, and the first Get items by
|
|
5430
|
+
being the first. A later Get items, which the jobs view does not hold, is
|
|
5431
|
+
kept as it is; and a Request already here with no Read Response gets
|
|
5432
|
+
none, though the jobs view reads it as plain text. A step new
|
|
5433
|
+
in [next] goes after the step before it there. With no [old], [next]. */
|
|
5434
|
+
function keepOrder(old , next ) {
|
|
5435
|
+
if (!Array.isArray(old)) return next;
|
|
5436
|
+
const key = (st ) => st.type === "request" ? `request:${st.id}`
|
|
5437
|
+
: st.type === "getItems" ? null : `${st.type}:${st.request}`;
|
|
5438
|
+
const byKey = new Map(next.flatMap(st => { const k = key(st); return k ? [[k, st] ] : []; }));
|
|
5439
|
+
const items = next.find(isGetItems);
|
|
5440
|
+
const empty = (st ) => st.type === "readResponse" && st.readAs == null && !st.out && !st.modifiers?.length;
|
|
5441
|
+
// Each step of [next] placed, by the step it became in the result.
|
|
5442
|
+
const placed = new Map ();
|
|
5443
|
+
const out = [];
|
|
5444
|
+
const firstItems = old.find(isGetItems);
|
|
5445
|
+
for (const st of old.filter((o) => isObj(o))) {
|
|
5446
|
+
if (isGetItems(st)) {
|
|
5447
|
+
if (st !== firstItems) out.push(st);
|
|
5448
|
+
else if (items) { const put = { ...items, id: st.id }; out.push(put); placed.set(items, put); }
|
|
5449
|
+
continue;
|
|
5450
|
+
}
|
|
5451
|
+
const k = key(st);
|
|
5452
|
+
const n = k ? byKey.get(k) : undefined;
|
|
5453
|
+
if (n && !placed.has(n)) { const put = { ...n, id: st.id }; out.push(put); placed.set(n, put); }
|
|
5454
|
+
}
|
|
5455
|
+
// A Read Response the jobs view reads as plain text, for a Request already
|
|
5456
|
+
// here that had none, stays away: the person removed it, or never added one.
|
|
5457
|
+
const hadRead = new Set(old.flatMap(st => isReadResponse(st) ? [st.request] : []));
|
|
5458
|
+
next.forEach((n, x) => {
|
|
5459
|
+
if (placed.has(n)) return;
|
|
5460
|
+
if (empty(n) && !hadRead.has((n ).request)
|
|
5461
|
+
&& old.some(st => isRequest(st) && st.id === (n ).request)) return;
|
|
5462
|
+
let at = 0;
|
|
5463
|
+
for (let y = x - 1; y >= 0; y--) {
|
|
5464
|
+
const before = placed.get(next[y] );
|
|
5465
|
+
if (before) { at = out.indexOf(before) + 1; break; }
|
|
5466
|
+
}
|
|
5467
|
+
out.splice(at, 0, n);
|
|
5468
|
+
placed.set(n, n);
|
|
5469
|
+
});
|
|
5470
|
+
return out;
|
|
5229
5471
|
}
|
|
5230
5472
|
|
|
5231
|
-
|
|
5232
|
-
|
|
5233
|
-
|
|
5234
|
-
|
|
5473
|
+
function serializeFresh(jobs , targets ) {
|
|
5474
|
+
const steps = [];
|
|
5475
|
+
const content = stepOf (jobs[0], "attachContent")?.content ?? null;
|
|
5476
|
+
if (content) steps.push({ type: "getItems", id: `items-${jobs[0] .id}`, content });
|
|
5477
|
+
jobs.forEach((job, k) => {
|
|
5478
|
+
const base = targets[0]?.steps?.[k];
|
|
5479
|
+
const req = { type: "request", id: job.id, name: job.name,
|
|
5480
|
+
cell: base ? clone(base) : { type: "prompt", prompt: "" } };
|
|
5481
|
+
if (sendsImage(job)) req.image = true;
|
|
5482
|
+
const toks = tokensOf(job);
|
|
5483
|
+
if (toks.length) req.tokens = toks;
|
|
5484
|
+
const flow = flowStepOf(job);
|
|
5485
|
+
if (flow) req.flow = flow;
|
|
5486
|
+
steps.push(req);
|
|
5487
|
+
const read = { type: "readResponse", id: `read-${job.id}`, request: job.id };
|
|
5488
|
+
const readAs = readAsOf(job);
|
|
5489
|
+
if (readAs != null) read.readAs = readAs;
|
|
5490
|
+
const reply = replyOf(job);
|
|
5491
|
+
if (reply) read.out = reply.out;
|
|
5492
|
+
const mods = (job.steps || []).filter((st) => isObj(st) && st.type === "modifier").map(st => st.modifier);
|
|
5493
|
+
if (mods.length) read.modifiers = mods;
|
|
5494
|
+
steps.push(read);
|
|
5495
|
+
const filt = filterStepOf(job);
|
|
5496
|
+
if (filt) steps.push({ type: "filter", id: `filter-${job.id}`, request: job.id,
|
|
5497
|
+
filterSets: filt.filterSets, ...(filt.format != null ? { format: filt.format } : {}) });
|
|
5498
|
+
});
|
|
5499
|
+
const variants = targets.map((t, i) => {
|
|
5500
|
+
const overrides = {};
|
|
5501
|
+
if (i > 0) jobs.forEach((job, k) => {
|
|
5502
|
+
const ov = cellDiff(targets[0]?.steps?.[k], t.steps?.[k]);
|
|
5503
|
+
if (ov) overrides[job.id] = ov;
|
|
5504
|
+
});
|
|
5505
|
+
return { id: t.id, name: t.name, connection: t.connection ?? null, profile: t.profile ?? null, overrides };
|
|
5506
|
+
});
|
|
5507
|
+
return { steps, variants };
|
|
5508
|
+
}
|
|
5509
|
+
|
|
5510
|
+
/** The version-19 jobs a version-20 document holds: one per Request, its
|
|
5511
|
+
Content (job 1's Get items, the Request's image, tokens and flow) and its
|
|
5512
|
+
Responses (its Read Response and Filter) rebuilt from the step list. */
|
|
5513
|
+
function jobsOf(doc ) {
|
|
5514
|
+
if (Array.isArray(doc?.jobs)) return doc .jobs ;
|
|
5515
|
+
const all = Array.isArray(doc?.steps) ? doc .steps : [];
|
|
5516
|
+
const items = all.find(isGetItems);
|
|
5517
|
+
const requests = all.filter(isRequest);
|
|
5518
|
+
return requests.map((req, k) => {
|
|
5519
|
+
const read = all.find((st) => isReadResponse(st) && st.request === req.id);
|
|
5520
|
+
const filt = all.find((st) => isFilterV20(st) && st.request === req.id);
|
|
5521
|
+
const inner = [];
|
|
5522
|
+
if (k === 0 && items) inner.push({ type: "attachContent", content: items.content });
|
|
5523
|
+
if (req.flow) inner.push({ type: "flowStep", step: req.flow.step, loop: req.flow.loop ?? null, api: req.flow.api });
|
|
5524
|
+
if (req.image) inner.push({ type: "attachImage" });
|
|
5525
|
+
if (req.tokens && req.tokens.length) inner.push({ type: "tokenMappings", tokenMappings: req.tokens });
|
|
5526
|
+
if (read?.readAs != null) inner.push({ type: "readAs", readAs: read.readAs });
|
|
5527
|
+
if (read?.out) inner.push({ type: "readReply", out: read.out });
|
|
5528
|
+
for (const m of read?.modifiers ?? []) inner.push({ type: "modifier", modifier: m });
|
|
5529
|
+
if (filt) inner.push({ type: "filters", filterSets: filt.filterSets, ...(filt.format != null ? { format: filt.format } : {}) });
|
|
5530
|
+
return { type: "job", id: req.id, name: req.name, steps: ordered(inner) };
|
|
5531
|
+
});
|
|
5532
|
+
}
|
|
5533
|
+
|
|
5534
|
+
/** [doc] with its jobs set, serialised back into the step list beside its
|
|
5535
|
+
Variants (the targets left as they read now). */
|
|
5536
|
+
function withJobs (doc , jobs ) {
|
|
5537
|
+
return { ...doc, ...serialize(jobs, targetsOf(doc ), (doc ).steps) };
|
|
5538
|
+
}
|
|
5539
|
+
|
|
5540
|
+
/** [doc] with its targets set, serialised back into the Variants beside its
|
|
5541
|
+
step list (the jobs left as they read now). */
|
|
5542
|
+
function withTargets (doc , targets ) {
|
|
5543
|
+
return { ...doc, ...serialize(jobsOf(doc ), targets, (doc ).steps) };
|
|
5544
|
+
}
|
|
5545
|
+
|
|
5546
|
+
/** A pipeline's content: job 1's Get items step, or none. */
|
|
5547
|
+
function contentOf(doc ) {
|
|
5548
|
+
if (Array.isArray(doc?.jobs)) return stepOf (doc .jobs [0], "attachContent")?.content ?? null;
|
|
5549
|
+
return (Array.isArray(doc?.steps) ? doc .steps .find(isGetItems)?.content : null) ?? null;
|
|
5550
|
+
}
|
|
5551
|
+
|
|
5552
|
+
/** Every Get items step's content, in list order, with the step's id and the
|
|
5553
|
+
folder a run keeps its files under: the first at the top of the run's
|
|
5554
|
+
files, as a run over one Source always was, and each later one under
|
|
5555
|
+
`@N`, N its place among them -- a name no item can have, since an item's
|
|
5556
|
+
name starts with a letter, a digit or `_` (§20). A version-19 document's
|
|
5557
|
+
one content is the first. */
|
|
5558
|
+
function itemContents(doc ) {
|
|
5559
|
+
if (Array.isArray(doc?.jobs)) {
|
|
5560
|
+
const content = contentOf(doc);
|
|
5561
|
+
return content ? [{ step: "", content, dir: "" }] : [];
|
|
5562
|
+
}
|
|
5563
|
+
const steps = Array.isArray(doc?.steps) ? doc .steps : [];
|
|
5564
|
+
return steps.filter(isGetItems).map((st, k) => ({ step: st.id, content: st.content, dir: k ? `@${k + 1}` : "" }));
|
|
5565
|
+
}
|
|
5566
|
+
|
|
5567
|
+
/** One Get items step's content, as itemContents reads it. */
|
|
5568
|
+
|
|
5569
|
+
|
|
5570
|
+
|
|
5571
|
+
|
|
5572
|
+
|
|
5573
|
+
|
|
5574
|
+
/** One item a run goes over: a file of a Source (its name, under its step's
|
|
5575
|
+
folder), a text of the document's own, or the bare item a run with no
|
|
5576
|
+
content of its own goes over, its prompt alone. */
|
|
5577
|
+
|
|
5578
|
+
|
|
5579
|
+
|
|
5580
|
+
|
|
5581
|
+
|
|
5582
|
+
|
|
5583
|
+
|
|
5584
|
+
|
|
5585
|
+
|
|
5586
|
+
|
|
5587
|
+
/** The items [run] goes over, in order: each Get items step's in turn -- a
|
|
5588
|
+
Source's files (the run's `files`, stamped at submit) or a text -- so a
|
|
5589
|
+
second Get items adds more items, run through the same steps. A step
|
|
5590
|
+
with no item of its own (No content) adds none; a run where no step adds
|
|
5591
|
+
one goes over the bare item. */
|
|
5592
|
+
function runItemsOf(run ) {
|
|
5593
|
+
const out = [];
|
|
5594
|
+
const all = itemContents(run);
|
|
5595
|
+
for (const { step, content, dir } of all) {
|
|
5596
|
+
const type = isObj(content) ? CONTENT_TYPES[content.type] : undefined;
|
|
5597
|
+
if (content?.type === "source") {
|
|
5598
|
+
for (const name of content.files || []) out.push({ name, dir, step });
|
|
5599
|
+
} else if (type?.inline && !type.bare) {
|
|
5600
|
+
const text = type.text?.(content, {}) ?? "";
|
|
5601
|
+
if (text.trim()) out.push({ name: null, text, dir, step });
|
|
5602
|
+
}
|
|
5603
|
+
}
|
|
5604
|
+
const bare = all.find(x => isObj(x.content) && CONTENT_TYPES[x.content.type]?.bare);
|
|
5605
|
+
if (!out.length && bare) out.push({ name: null, bare: true, dir: "", step: bare.step });
|
|
5606
|
+
return out;
|
|
5607
|
+
}
|
|
5608
|
+
|
|
5609
|
+
/** [doc] with its content set -- job 1's Get items added, replaced, or taken
|
|
5610
|
+
away for null -- serialised back into the step list. */
|
|
5611
|
+
function withContent (doc , content ) {
|
|
5612
|
+
const jobs = jobsOf(doc );
|
|
5613
|
+
const [first, ...rest] = jobs;
|
|
5235
5614
|
if (!first) return doc;
|
|
5236
|
-
return
|
|
5615
|
+
return withJobs(doc, [withStep(first, "attachContent", content ? { type: "attachContent", content } : null), ...rest]);
|
|
5237
5616
|
}
|
|
5238
5617
|
|
|
5239
5618
|
/**
|
|
@@ -5244,7 +5623,7 @@ function withContent (doc , content )
|
|
|
5244
5623
|
* only where a token places it, and an image or a document is sent as a part
|
|
5245
5624
|
* of its own, so neither is joined. [ctx] reads a Source as validation does.
|
|
5246
5625
|
*/
|
|
5247
|
-
function contentJoined(doc
|
|
5626
|
+
function contentJoined(doc , i , k ,
|
|
5248
5627
|
ctx = {}) {
|
|
5249
5628
|
if (k !== 0 || targetEntryOf(doc, i, k)?.asks !== "prompt") return false;
|
|
5250
5629
|
if (String(targetStepOf(doc, i, k)?.prompt ?? "").includes("{text}")) return false;
|
|
@@ -5252,9 +5631,18 @@ function contentJoined(doc
|
|
|
5252
5631
|
return !!content && (CONTENT_TYPES[content.type]?.text?.(content, ctx) ?? null) != null;
|
|
5253
5632
|
}
|
|
5254
5633
|
|
|
5255
|
-
/** A document's targets, in column order
|
|
5256
|
-
|
|
5257
|
-
|
|
5634
|
+
/** A document's targets, in column order: each Variant as a column through
|
|
5635
|
+
the Request steps, its cell in each the Request's base with the Variant's
|
|
5636
|
+
override over it (version 20). A doc still carrying `targets` -- the shape
|
|
5637
|
+
the upgrade builds before it serialises -- is read as it is. */
|
|
5638
|
+
function targetsOf(doc ) {
|
|
5639
|
+
if (Array.isArray(doc?.targets)) return doc .targets ;
|
|
5640
|
+
const requests = (Array.isArray(doc?.steps) ? doc .steps : []).filter(isRequest);
|
|
5641
|
+
const variants = Array.isArray(doc?.variants) ? doc .variants : [];
|
|
5642
|
+
return variants.map(v => ({
|
|
5643
|
+
id: v.id, name: v.name, connection: v.connection ?? null, profile: v.profile ?? null,
|
|
5644
|
+
steps: requests.map(req => mergeCell(req.cell, v.overrides?.[req.id])),
|
|
5645
|
+
}));
|
|
5258
5646
|
}
|
|
5259
5647
|
|
|
5260
5648
|
/** Target [i]'s letter, A for the first: the same in every job, so a
|
|
@@ -5263,24 +5651,24 @@ function targetLetter(i ) {
|
|
|
5263
5651
|
return i < 26 ? String.fromCharCode(65 + i) : String(i + 1);
|
|
5264
5652
|
}
|
|
5265
5653
|
|
|
5266
|
-
/** Target [i]'s name, or its letter's: "
|
|
5267
|
-
function targetLabel(doc
|
|
5268
|
-
return (targetsOf(doc)[i]?.name || "").trim() || `
|
|
5654
|
+
/** Target [i]'s name, or its letter's: "Variant A". */
|
|
5655
|
+
function targetLabel(doc , i ) {
|
|
5656
|
+
return (targetsOf(doc)[i]?.name || "").trim() || `Variant ${targetLetter(i)}`;
|
|
5269
5657
|
}
|
|
5270
5658
|
|
|
5271
5659
|
/** What target [i] sends in job [k]: its step there. */
|
|
5272
|
-
function targetStepOf(doc
|
|
5660
|
+
function targetStepOf(doc , i , k ) {
|
|
5273
5661
|
const st = targetsOf(doc)[i]?.steps?.[k];
|
|
5274
5662
|
return isObj(st) ? st : undefined;
|
|
5275
5663
|
}
|
|
5276
5664
|
|
|
5277
5665
|
/** The registry entry of target [i]'s step in job [k]: what it asks, and of whom. */
|
|
5278
|
-
const targetEntryOf = (doc
|
|
5666
|
+
const targetEntryOf = (doc , i , k ) =>
|
|
5279
5667
|
STEP_TYPES[targetStepOf(doc, i, k)?.type ?? ""];
|
|
5280
5668
|
|
|
5281
5669
|
/** What job [k]'s targets send, as the job's editor shows it: the first
|
|
5282
5670
|
target's step type there, or Prompt's while there is none. */
|
|
5283
|
-
function jobTargetType(doc
|
|
5671
|
+
function jobTargetType(doc , k ) {
|
|
5284
5672
|
return targetStepOf(doc, 0, k)?.type ?? "prompt";
|
|
5285
5673
|
}
|
|
5286
5674
|
|
|
@@ -5289,18 +5677,19 @@ function jobTargetType(doc
|
|
|
5289
5677
|
connection that `replays`). A recorded-reply metric compares the recorded
|
|
5290
5678
|
reply with itself there, so it says nothing (n/a). Asked through the
|
|
5291
5679
|
registry, never a step id. */
|
|
5292
|
-
function recordedTargetOf(doc
|
|
5680
|
+
function recordedTargetOf(doc , i ) {
|
|
5293
5681
|
const entry = targetEntryOf(doc, i, 0);
|
|
5294
5682
|
return !!(entry?.local && CONNECTION_TYPES[localConnOf(entry).type]?.replays);
|
|
5295
5683
|
}
|
|
5296
5684
|
|
|
5297
5685
|
/** [doc] with target [i]'s step in job [k] changed: [change] merged into
|
|
5298
|
-
it, or the step [change] makes of it. */
|
|
5299
|
-
function withTarget
|
|
5686
|
+
it, or the step [change] makes of it. Serialised back into the Variants. */
|
|
5687
|
+
function withTarget (doc , i , k ,
|
|
5300
5688
|
change ) {
|
|
5301
|
-
|
|
5689
|
+
const targets = targetsOf(doc ).map((t, x) => (x !== i ? t : {
|
|
5302
5690
|
...t, steps: t.steps.map((st, j) => (j !== k ? st : typeof change === "function" ? change(st) : { ...st, ...change } )),
|
|
5303
|
-
}))
|
|
5691
|
+
}));
|
|
5692
|
+
return withTargets(doc, targets);
|
|
5304
5693
|
}
|
|
5305
5694
|
|
|
5306
5695
|
/** The token mappings [job]'s words are resolved under. */
|
|
@@ -5330,8 +5719,8 @@ const readAsOf = (job ) => stepOf (job, "readA
|
|
|
5330
5719
|
|
|
5331
5720
|
/** What a transport builds target [i]'s request in job [k] from: the
|
|
5332
5721
|
target's step, with the flow step and reading the job holds. */
|
|
5333
|
-
function requestStepOf(doc
|
|
5334
|
-
const job = doc
|
|
5722
|
+
function requestStepOf(doc , i , k ) {
|
|
5723
|
+
const job = jobsOf(doc)[k];
|
|
5335
5724
|
return { ...flowStepOf(job), ...targetStepOf(doc, i, k), readAs: readAsOf(job) } ;
|
|
5336
5725
|
}
|
|
5337
5726
|
|
|
@@ -5560,7 +5949,7 @@ STEP_TYPES.job = {
|
|
|
5560
5949
|
const unknown = steps.find(st => !slotOf(st));
|
|
5561
5950
|
if (unknown) return void bad.push(`${at}: "${isObj(unknown) ? unknown.type : unknown}" is not a step this lab has`);
|
|
5562
5951
|
if (steps.some(st => slotOf(st) === "target")) {
|
|
5563
|
-
return void bad.push(`${at}: what a
|
|
5952
|
+
return void bad.push(`${at}: what a variant sends is the variant's own step, not the job's`);
|
|
5564
5953
|
}
|
|
5565
5954
|
const twice = steps.find((st, x) => !STEP_TYPES[st.type] .many && steps.findIndex(o => o.type === st.type) !== x);
|
|
5566
5955
|
if (twice) return void bad.push(`${at} has two ${STEP_TYPES[twice.type] .label ?? twice.type} steps`);
|
|
@@ -5645,18 +6034,18 @@ STEP_TYPES.evals = {
|
|
|
5645
6034
|
|
|
5646
6035
|
// ---- the document -------------------------------------------------------------
|
|
5647
6036
|
|
|
5648
|
-
const PIPELINE_FIELDS = ["version", "id", "name", "
|
|
6037
|
+
const PIPELINE_FIELDS = ["version", "id", "name", "steps", "variants", "evals", "pass"];
|
|
5649
6038
|
// What resolving adds, and nothing else: the profiles it resolved to and the
|
|
5650
6039
|
// run's own comment, which belongs to the run and never to the pipeline.
|
|
5651
6040
|
const RUN_FIELDS = [...PIPELINE_FIELDS, "profiles", "comment", "plugins"];
|
|
5652
6041
|
|
|
5653
6042
|
/** The one rule for a document that is not this version: refuse it, by name. */
|
|
5654
6043
|
function versionProblem(doc ) {
|
|
5655
|
-
if (!isObj(doc)) return "
|
|
6044
|
+
if (!isObj(doc)) return "an experiment is a JSON object";
|
|
5656
6045
|
if (doc.version === PIPELINE_VERSION) return null;
|
|
5657
6046
|
return doc.version == null
|
|
5658
|
-
? `this
|
|
5659
|
-
: `this
|
|
6047
|
+
? `this experiment carries no version, and the lab reads version ${PIPELINE_VERSION}`
|
|
6048
|
+
: `this experiment is version ${JSON.stringify(doc.version)}, and the lab reads version ${PIPELINE_VERSION}`;
|
|
5660
6049
|
}
|
|
5661
6050
|
|
|
5662
6051
|
/** What an upgrade from version 2 needs from outside the document: the rules
|
|
@@ -5814,7 +6203,7 @@ function newFilterLink(set ) {
|
|
|
5814
6203
|
return { id: newId(), set: { id: set.id, name: set.name }, pin: null };
|
|
5815
6204
|
}
|
|
5816
6205
|
|
|
5817
|
-
function
|
|
6206
|
+
function chainsToJobs(next ) {
|
|
5818
6207
|
next.version = PIPELINE_VERSION;
|
|
5819
6208
|
if (Array.isArray(next.chains)) {
|
|
5820
6209
|
const entries = Object.entries(next).map(([k, v]) => (k === "chains" ? ["jobs", v] : [k, v]));
|
|
@@ -5970,23 +6359,57 @@ function ownBodiesNow(doc ) {
|
|
|
5970
6359
|
return out;
|
|
5971
6360
|
}
|
|
5972
6361
|
|
|
6362
|
+
/** A jobs-and-targets intermediate (the version-19 shape) as a version-20
|
|
6363
|
+
document: its step list and Variants, serialised, with a run's additions
|
|
6364
|
+
kept in place (version 20, §20). The fields are in RUN_FIELDS order, so a
|
|
6365
|
+
document re-read is byte-for-byte itself. */
|
|
6366
|
+
function serializeDoc(jt ) {
|
|
6367
|
+
const { steps, variants } = serialize(jobsOf(jt), targetsOf(jt), jt.steps);
|
|
6368
|
+
const out = { version: PIPELINE_VERSION, id: jt.id, name: jt.name, steps, variants,
|
|
6369
|
+
evals: jt.evals, pass: jt.pass };
|
|
6370
|
+
for (const f of ["profiles", "comment", "plugins"]) if (jt[f] !== undefined) out[f] = jt[f];
|
|
6371
|
+
return out;
|
|
6372
|
+
}
|
|
6373
|
+
|
|
6374
|
+
/** A version-20 document as the jobs-and-targets intermediate, so the
|
|
6375
|
+
normalisations that read jobs and targets (localSteps, ownBodiesNow,
|
|
6376
|
+
profileRefs) can run before it is serialised back. */
|
|
6377
|
+
function jtOf(doc ) {
|
|
6378
|
+
const out = { version: doc.version, id: doc.id, name: doc.name,
|
|
6379
|
+
jobs: jobsOf(doc), targets: targetsOf(doc), evals: doc.evals, pass: doc.pass };
|
|
6380
|
+
// The step list it came from, which serializeDoc keeps the order of.
|
|
6381
|
+
if (Array.isArray(doc.steps)) out.steps = doc.steps;
|
|
6382
|
+
for (const f of ["profiles", "comment", "plugins"]) if (doc[f] !== undefined) out[f] = doc[f];
|
|
6383
|
+
return out;
|
|
6384
|
+
}
|
|
6385
|
+
|
|
5973
6386
|
function upgradePipeline (doc , ctx = {}) {
|
|
5974
6387
|
// A current document is read as it is, but for a profile reference the Runs
|
|
5975
6388
|
// tab saved whole (see profileRefs), which is cut back, a Read Reply's
|
|
5976
6389
|
// leftover `preamble: false` (withoutLeftoverPreamble), and a step on a
|
|
5977
|
-
// profile a target step stands in for (localSteps).
|
|
6390
|
+
// profile a target step stands in for (localSteps). It is read as the jobs
|
|
6391
|
+
// intermediate, normalised, and serialised back (serializeDoc), so a clean
|
|
6392
|
+
// document comes back as the same object.
|
|
5978
6393
|
if (isObj(doc) && doc.version === PIPELINE_VERSION) {
|
|
5979
|
-
|
|
5980
|
-
if (fatProfileRef(
|
|
5981
|
-
|
|
5982
|
-
|
|
5983
|
-
|
|
5984
|
-
|
|
6394
|
+
const jt = jtOf(doc);
|
|
6395
|
+
if (fatProfileRef(jt)) profileRefs(jt);
|
|
6396
|
+
const out = serializeDoc(localSteps(ownBodiesNow(withoutLeftoverPreamble(jt)), ctx));
|
|
6397
|
+
// The same document back when nothing moved: the page saves a stored
|
|
6398
|
+
// pipeline whose upgrade is a new object, so a new one would rewrite every
|
|
6399
|
+
// experiment on every load. Compared by content, not as text: a document
|
|
6400
|
+
// stored in another key order (Import, the demo pack) has not changed.
|
|
6401
|
+
return (sameJson(out, doc) ? doc : out) ;
|
|
5985
6402
|
}
|
|
5986
|
-
if (!isObj(doc) || ![1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18].includes(doc.version )) return doc;
|
|
5987
|
-
// Every version reads
|
|
5988
|
-
//
|
|
5989
|
-
const
|
|
6403
|
+
if (!isObj(doc) || ![1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19].includes(doc.version )) return doc;
|
|
6404
|
+
// Every version reads up through the version-19 jobs-and-targets shape, then
|
|
6405
|
+
// reshapes to the version-20 step list and Variants (serializeDoc).
|
|
6406
|
+
const twenty = (jt ) => serializeDoc(localSteps(ownBodiesNow(jt), ctx)) ;
|
|
6407
|
+
// A version-19 document is already that shape; a Read Reply's leftover
|
|
6408
|
+
// preamble is cut and its profile references trimmed, then it reshapes.
|
|
6409
|
+
if (doc.version === 19) return twenty(withoutLeftoverPreamble(cutRefs(clone(doc) )));
|
|
6410
|
+
// Every version before 19 reads as 19 first: a job's filters are one list
|
|
6411
|
+
// around its Response Format (oneFilterList), then it reshapes.
|
|
6412
|
+
const nineteen = (d ) => twenty(oneFilterList(d));
|
|
5990
6413
|
if (doc.version === 18) return nineteen(withoutLeftoverPreamble(cutRefs(clone(doc) )));
|
|
5991
6414
|
// Every version reads as 17 first, then as 18: a saved-mode Source content
|
|
5992
6415
|
// pins one response (savedPinned).
|
|
@@ -6262,9 +6685,9 @@ function nineOf(next , ctx ) {
|
|
|
6262
6685
|
if (next.version === 9) { profileRefs(next); return next; }
|
|
6263
6686
|
if (next.version === 8) return metricsOf(next);
|
|
6264
6687
|
if (next.version === 7) return metricsOf(stepsOf(testsList(next)));
|
|
6265
|
-
if (next.version === 6) return metricsOf(stepsOf(
|
|
6266
|
-
if (next.version === 5) return metricsOf(stepsOf(
|
|
6267
|
-
if (next.version === 3 || next.version === 4) return metricsOf(stepsOf(
|
|
6688
|
+
if (next.version === 6) return metricsOf(stepsOf(chainsToJobs(next)));
|
|
6689
|
+
if (next.version === 5) return metricsOf(stepsOf(chainsToJobs(testsList(next))));
|
|
6690
|
+
if (next.version === 3 || next.version === 4) return metricsOf(stepsOf(chainsToJobs(testsList(upgradeIds(next)))));
|
|
6268
6691
|
if (next.version === 1) {
|
|
6269
6692
|
next.version = 2;
|
|
6270
6693
|
if (Array.isArray(next.chains)) {
|
|
@@ -6290,7 +6713,7 @@ function nineOf(next , ctx ) {
|
|
|
6290
6713
|
if (isObj(cell)) cell.prompt = renamed(cell.prompt);
|
|
6291
6714
|
}
|
|
6292
6715
|
}
|
|
6293
|
-
return metricsOf(stepsOf(
|
|
6716
|
+
return metricsOf(stepsOf(chainsToJobs(testsList(upgradeIds(next)))));
|
|
6294
6717
|
}
|
|
6295
6718
|
|
|
6296
6719
|
/** Version 1's two maps as a list of mappings. */
|
|
@@ -6303,13 +6726,14 @@ function tokenMappingsFromV1(set ) {
|
|
|
6303
6726
|
];
|
|
6304
6727
|
}
|
|
6305
6728
|
|
|
6306
|
-
/** A current-version document with one
|
|
6307
|
-
the item's image, as a job over a Source would; one over text content
|
|
6729
|
+
/** A current-version document with one Request and no Variant. Its Request
|
|
6730
|
+
sends the item's image, as a job over a Source would; one over text content
|
|
6308
6731
|
has none to send, and the page turns it off with the content. */
|
|
6309
6732
|
function blankPipeline(opts = {}) {
|
|
6310
6733
|
const job = withImage(jobDefaults(opts.kind, opts.tokens), true);
|
|
6734
|
+
const { steps, variants } = serialize([job], []);
|
|
6311
6735
|
return { version: PIPELINE_VERSION, id: newId(), name: opts.name || "",
|
|
6312
|
-
|
|
6736
|
+
steps, variants, evals: [], pass: { mode: "all" } };
|
|
6313
6737
|
}
|
|
6314
6738
|
|
|
6315
6739
|
/** What is wrong with how a target or a step [at] names whom it asks: each
|
|
@@ -6322,6 +6746,102 @@ function answerRefsProblems(o , at , bad ) {
|
|
|
6322
6746
|
}
|
|
6323
6747
|
}
|
|
6324
6748
|
|
|
6749
|
+
/** Each step of [doc]'s list as the Steps list names it, by its id: Get
|
|
6750
|
+
items, Request N (numbered by the Requests before it), Read Response N
|
|
6751
|
+
(the number of the Request it reads), Filter. */
|
|
6752
|
+
function stepNames(doc ) {
|
|
6753
|
+
const steps = Array.isArray(doc?.steps) ? doc .steps : [];
|
|
6754
|
+
const number = new Map(steps.filter(isRequest).map((r, k) => [r.id, k + 1]));
|
|
6755
|
+
const out = {};
|
|
6756
|
+
for (const st of steps) {
|
|
6757
|
+
if (!isObj(st)) continue;
|
|
6758
|
+
out[st.id] = st.type === "getItems" ? "Get items"
|
|
6759
|
+
: st.type === "request" ? `Request ${number.get(st.id)}`
|
|
6760
|
+
: st.type === "readResponse" ? `Read Response ${number.get(st.request) ?? ""}`.trim()
|
|
6761
|
+
: "Filter";
|
|
6762
|
+
}
|
|
6763
|
+
return out;
|
|
6764
|
+
}
|
|
6765
|
+
|
|
6766
|
+
/** Where each step of [doc]'s list that is out of order must go, by its id
|
|
6767
|
+
(the order rules stepsProblems holds the document to): a Get items after
|
|
6768
|
+
Request 1 comes before it; a Read Response or Filter above the Request it
|
|
6769
|
+
reads comes after it. A step in order is not listed. */
|
|
6770
|
+
function stepFaults(doc ) {
|
|
6771
|
+
const steps = Array.isArray(doc?.steps) ? doc .steps : [];
|
|
6772
|
+
const names = stepNames(doc);
|
|
6773
|
+
const first = steps.findIndex(isRequest);
|
|
6774
|
+
const out = {};
|
|
6775
|
+
steps.forEach((st, at) => {
|
|
6776
|
+
if (isGetItems(st) && first >= 0 && at > first) out[st.id] = `Comes before ${names[steps[first] .id]}`;
|
|
6777
|
+
if (isReadResponse(st) || isFilterV20(st)) {
|
|
6778
|
+
const req = steps.findIndex(o => isRequest(o) && o.id === st.request);
|
|
6779
|
+
if (req > at) out[st.id] = `Comes after ${names[steps[req] .id]}`;
|
|
6780
|
+
}
|
|
6781
|
+
});
|
|
6782
|
+
return out;
|
|
6783
|
+
}
|
|
6784
|
+
|
|
6785
|
+
/** The step types a version-20 pipeline's step list holds. */
|
|
6786
|
+
const PIPELINE_STEP_TYPES = ["getItems", "request", "readResponse", "filter"];
|
|
6787
|
+
|
|
6788
|
+
/**
|
|
6789
|
+
* Every reason [doc]'s step list and Variants are not a well-formed version-20
|
|
6790
|
+
* pipeline (§20): a step of an unknown type, a Get items after Request 1, a
|
|
6791
|
+
* Read Response or Filter before the Request it reads, and a Variant that is
|
|
6792
|
+
* not an object with its overrides keyed by Request ids. The order rules name
|
|
6793
|
+
* the step the out-of-order one must come before, so the page can point at it.
|
|
6794
|
+
*/
|
|
6795
|
+
function stepsProblems(doc , bad ) {
|
|
6796
|
+
if (!Array.isArray(doc.steps)) return void bad.push("steps has to be a list");
|
|
6797
|
+
const steps = doc.steps;
|
|
6798
|
+
const requestIds = [];
|
|
6799
|
+
const seen = new Set ();
|
|
6800
|
+
// The number of each Request, by its id, so a later step names it.
|
|
6801
|
+
for (const st of steps) if (isRequest(st)) requestIds.push(isStr(st.id) ? st.id : "");
|
|
6802
|
+
const numberOf = (id ) => requestIds.indexOf(id) + 1;
|
|
6803
|
+
let sawRequest = false;
|
|
6804
|
+
steps.forEach((st) => {
|
|
6805
|
+
if (!isObj(st) || !isStr(st.type) || !PIPELINE_STEP_TYPES.includes(st.type)) {
|
|
6806
|
+
return void bad.push(`"${isObj(st) ? st.type : st}" is not a step this lab has`);
|
|
6807
|
+
}
|
|
6808
|
+
if (!isStr(st.id) || !st.id.trim()) bad.push(`a ${st.type} step has no id`);
|
|
6809
|
+
else if (seen.has(st.id)) bad.push(`two steps have the id ${st.id}`);
|
|
6810
|
+
else seen.add(st.id);
|
|
6811
|
+
if (st.type === "getItems") {
|
|
6812
|
+
if (sawRequest) bad.push("Get items comes before Request 1");
|
|
6813
|
+
} else if (st.type === "request") {
|
|
6814
|
+
sawRequest = true;
|
|
6815
|
+
} else {
|
|
6816
|
+
// Read Response N and Filter come after Request N.
|
|
6817
|
+
const label = st.type === "readResponse" ? "Read Response" : "Filter";
|
|
6818
|
+
if (!isStr(st.request) || !requestIds.includes(st.request)) {
|
|
6819
|
+
bad.push(`a ${label} reads a Request the experiment does not have`);
|
|
6820
|
+
} else {
|
|
6821
|
+
const n = numberOf(st.request);
|
|
6822
|
+
const at = steps.indexOf(st), req = steps.findIndex(o => isRequest(o) && o.id === st.request);
|
|
6823
|
+
if (at < req) bad.push(`${label} ${n} comes after Request ${n}`);
|
|
6824
|
+
}
|
|
6825
|
+
}
|
|
6826
|
+
});
|
|
6827
|
+
if (!requestIds.length) bad.push("add a Request first");
|
|
6828
|
+
if (!Array.isArray(doc.variants)) return void bad.push("variants has to be a list");
|
|
6829
|
+
const vseen = new Set ();
|
|
6830
|
+
doc.variants.forEach((v , i ) => {
|
|
6831
|
+
const at = targetLabel(doc, i);
|
|
6832
|
+
if (!isObj(v)) return void bad.push(`${at} has to be an object`);
|
|
6833
|
+
onlyFields(v, at, ["id", "name", "connection", "profile", "overrides"], bad);
|
|
6834
|
+
if (!isStr(v.id) || !v.id.trim()) bad.push(`${at} has no id`);
|
|
6835
|
+
else if (vseen.has(v.id)) bad.push(`${at} has the id of another variant`);
|
|
6836
|
+
else vseen.add(v.id);
|
|
6837
|
+
if (v.name != null && !isStr(v.name)) bad.push(`${at}: name has to be text`);
|
|
6838
|
+
answerRefsProblems(v, at, bad);
|
|
6839
|
+
if (v.overrides != null && (!isObj(v.overrides) || Object.keys(v.overrides).some(k => !requestIds.includes(k)))) {
|
|
6840
|
+
bad.push(`${at}: overrides are keyed by Request ids`);
|
|
6841
|
+
}
|
|
6842
|
+
});
|
|
6843
|
+
}
|
|
6844
|
+
|
|
6325
6845
|
/**
|
|
6326
6846
|
* Every reason [doc] cannot run, as sentences -- none when it can. The
|
|
6327
6847
|
* reference lookups are [ctx]'s: `profiles(id)` and `sources(id)` answer with
|
|
@@ -6335,29 +6855,41 @@ function validatePipeline(input , ctx = {}) {
|
|
|
6335
6855
|
const run = doc.profiles !== undefined;
|
|
6336
6856
|
const c = { ...ctx, run };
|
|
6337
6857
|
if (run && !ctx.profiles) c.profiles = (key ) => (runAnswer(doc, key) ) ?? null;
|
|
6338
|
-
onlyFields(doc, "the
|
|
6339
|
-
if (!isStr(doc.id) || !doc.id.trim()) bad.push("the
|
|
6858
|
+
onlyFields(doc, "the experiment", run ? RUN_FIELDS : PIPELINE_FIELDS, bad);
|
|
6859
|
+
if (!isStr(doc.id) || !doc.id.trim()) bad.push("the experiment has no id");
|
|
6340
6860
|
if (doc.name != null && !isStr(doc.name)) bad.push("name has to be text");
|
|
6341
6861
|
if (run && doc.comment != null && !isStr(doc.comment)) bad.push("comment has to be text");
|
|
6342
|
-
|
|
6343
|
-
|
|
6344
|
-
|
|
6345
|
-
if (
|
|
6346
|
-
//
|
|
6862
|
+
// The version-20 step list and Variants are well-formed, so the jobs and
|
|
6863
|
+
// targets the rest reads (jobsOf, targetsOf) can be derived from them.
|
|
6864
|
+
stepsProblems(doc, bad);
|
|
6865
|
+
if (bad.length) return bad;
|
|
6866
|
+
// The derived jobs-and-targets view every semantic check below reads.
|
|
6867
|
+
const jobs = jobsOf(doc);
|
|
6868
|
+
const jdoc = { ...doc, jobs };
|
|
6869
|
+
if (!jobs.length) return [...bad, "add a Request first"];
|
|
6870
|
+
if (!targetsOf(doc).length) bad.push("add a variant first");
|
|
6871
|
+
// Content is job 1's Get items step; a job 1 without one has none yet.
|
|
6347
6872
|
if (!contentOf(doc)) bad.push("set the content first");
|
|
6348
6873
|
if (run) profilesProblems(doc.profiles, bad);
|
|
6349
|
-
|
|
6874
|
+
jobs.forEach((ch , k ) => STEP_TYPES.job .validate(ch, c, bad, k, jdoc));
|
|
6875
|
+
// A later Get items, which job 1 does not attach, is held to the same
|
|
6876
|
+
// rules, named by its place.
|
|
6877
|
+
itemContents(doc).slice(1).forEach(({ content }, k) => {
|
|
6878
|
+
const said = [];
|
|
6879
|
+
STEP_TYPES.attachContent .validate ({ type: "attachContent", content }, c, said);
|
|
6880
|
+
for (const s of said) bad.push(`Get items ${k + 2}: ${s}`);
|
|
6881
|
+
});
|
|
6350
6882
|
if (bad.length) return bad;
|
|
6351
6883
|
const content = contentOf(doc) ;
|
|
6352
6884
|
|
|
6353
6885
|
const targets = targetsOf(doc);
|
|
6354
|
-
const n =
|
|
6886
|
+
const n = jobs.length;
|
|
6355
6887
|
targets.forEach((t, i) => {
|
|
6356
6888
|
const at = targetLabel(doc, i);
|
|
6357
6889
|
if (!isObj(t)) return void bad.push(`${at} has to be an object`);
|
|
6358
6890
|
onlyFields(t, at, ["id", "name", "connection", "profile", "steps"], bad);
|
|
6359
6891
|
if (!isStr(t.id) || !t.id.trim()) bad.push(`${at} has no id`);
|
|
6360
|
-
else if (targets.findIndex((o) => isObj(o) && o.id === t.id) !== i) bad.push(`${at} has the id of another
|
|
6892
|
+
else if (targets.findIndex((o) => isObj(o) && o.id === t.id) !== i) bad.push(`${at} has the id of another variant`);
|
|
6361
6893
|
if (t.name != null && !isStr(t.name)) bad.push(`${at}: name has to be text`);
|
|
6362
6894
|
answerRefsProblems(t, at, bad);
|
|
6363
6895
|
if (!Array.isArray(t.steps) || t.steps.length !== n) {
|
|
@@ -6374,19 +6906,19 @@ function validatePipeline(input , ctx = {}) {
|
|
|
6374
6906
|
const where = `${at}, ${jobLabel(doc, k)}`;
|
|
6375
6907
|
if (!isObj(st)) return void bad.push(`${where} has to be an object`);
|
|
6376
6908
|
const entry = STEP_TYPES[st.type];
|
|
6377
|
-
if (entry?.slot !== "target") return void bad.push(`${where}: "${st.type}" is not something a
|
|
6909
|
+
if (entry?.slot !== "target") return void bad.push(`${where}: "${st.type}" is not something a variant can send`);
|
|
6378
6910
|
if (entry.local && (st.profile != null || st.connection != null)) return void bad.push(`${where}: ${entry.label} asks no connection`);
|
|
6379
6911
|
if (k > 0 && entry.firstJobOnly) return void bad.push(`${where}: ${entry.firstJobOnly}`);
|
|
6380
6912
|
onlyFields(st, where, entry.fields ?? [], bad);
|
|
6381
6913
|
// A request is read and placed by its job's HTTP API.
|
|
6382
|
-
const flow = entry.asks === "request" ? flowStepOf(
|
|
6914
|
+
const flow = entry.asks === "request" ? flowStepOf(jobs[k]) : null;
|
|
6383
6915
|
entry.validate(flow ? { ...st, api: flow.api } : st, c, bad, where);
|
|
6384
6916
|
answerRefsProblems(st, where, bad);
|
|
6385
6917
|
if (st.from != null && !isRef(st.from)) bad.push(`${where} has to name the prompt it was picked from as { id, name }`);
|
|
6386
6918
|
// A request is a flow step's, rebuilt from its records, and its
|
|
6387
6919
|
// template has no place for an image.
|
|
6388
|
-
if (entry.asks === "request" && !flowStepOf(
|
|
6389
|
-
if (entry.asks === "request" && sendsImage(
|
|
6920
|
+
if (entry.asks === "request" && !flowStepOf(jobs[k])) bad.push(`${where}: ${entry.label} needs the job's flow step`);
|
|
6921
|
+
if (entry.asks === "request" && sendsImage(jobs[k])) bad.push(`${where}: ${entry.label} has no place for an image`);
|
|
6390
6922
|
});
|
|
6391
6923
|
});
|
|
6392
6924
|
if (bad.length) return bad;
|
|
@@ -6441,7 +6973,7 @@ function validatePipeline(input , ctx = {}) {
|
|
|
6441
6973
|
const docType = (contentFiles || []).map(f => documentTypeOf(String((f )?.name ?? f)))
|
|
6442
6974
|
.find(Boolean) ?? null;
|
|
6443
6975
|
targets.forEach((_, i) => {
|
|
6444
|
-
const conns =
|
|
6976
|
+
const conns = jobs.map((_ , k ) => (targetEntryOf(doc, i, k)?.local ? localConnOf(targetEntryOf(doc, i, k)) : lookup(targetAnswerOf(doc, i, k) )));
|
|
6445
6977
|
conns.forEach((p , k ) => {
|
|
6446
6978
|
if (!local(p)) return;
|
|
6447
6979
|
const entry = CONNECTION_TYPES[typeOf(p )] ;
|
|
@@ -6463,27 +6995,28 @@ function validatePipeline(input , ctx = {}) {
|
|
|
6463
6995
|
}
|
|
6464
6996
|
// A request goes as the flow wrote it, so its connection has to speak
|
|
6465
6997
|
// the job's HTTP API; one the lab cannot tell (Any endpoint) passes.
|
|
6466
|
-
const api = step.asks === "request" ? flowStepOf(
|
|
6998
|
+
const api = step.asks === "request" ? flowStepOf(jobs[k])?.api : undefined;
|
|
6467
6999
|
const speaks = api && api !== "raw" && type?.speaks ? type.speaks(p ) : null;
|
|
6468
7000
|
if (api && speaks && speaks !== api) {
|
|
6469
7001
|
const name = targetAnswerOf(doc, i, k)?.connection.name || type .label;
|
|
6470
7002
|
bad.push(`${n > 1 ? `${jobLabel(doc, k)}, ` : ""}${targetLabel(doc, i)}: ${name} can't send an ${HTTP_APIS[api]?.label ?? api} request`);
|
|
6471
7003
|
}
|
|
6472
|
-
if (docType && type && !type.documents && sendsImage(
|
|
7004
|
+
if (docType && type && !type.documents && sendsImage(jobs[k])) {
|
|
6473
7005
|
const name = targetAnswerOf(doc, i, k)?.connection.name || type.label;
|
|
6474
7006
|
bad.push(`${n > 1 ? `${jobLabel(doc, k)}, ` : ""}${targetLabel(doc, i)}: ${name} can't read ${docType.label}s`);
|
|
6475
7007
|
}
|
|
6476
7008
|
});
|
|
6477
7009
|
// The model is the profile's unless the target's step carries it.
|
|
6478
7010
|
const bare = conns.findIndex((p , k ) => p && c.profiles && !local(p)
|
|
6479
|
-
&& targetEntryOf(doc, i, k)?.modelFrom !== "step" && !String(p.model || "").trim()
|
|
7011
|
+
&& targetEntryOf(doc, i, k)?.modelFrom !== "step" && !String(p.model || "").trim()
|
|
7012
|
+
&& !String(targetStepOf(doc, i, k)?.model || "").trim());
|
|
6480
7013
|
if (bare >= 0) {
|
|
6481
7014
|
bad.push(n > 1
|
|
6482
7015
|
? `no model on the connection ${jobLabel(doc, bare)} of ${targetLabel(doc, i)} uses — manage connections on the Setup tab`
|
|
6483
7016
|
: `no model on ${targetLabel(doc, i)}'s connection — manage connections on the Setup tab`);
|
|
6484
7017
|
}
|
|
6485
7018
|
});
|
|
6486
|
-
STEP_TYPES.evals .validate(doc.evals, c, bad, outOf(
|
|
7019
|
+
STEP_TYPES.evals .validate(doc.evals, c, bad, outOf(jobs.at(-1)).kind, doc);
|
|
6487
7020
|
passProblems(doc.pass, Array.isArray(doc.evals) ? doc.evals.length : 0, bad);
|
|
6488
7021
|
if (bad.length) return bad;
|
|
6489
7022
|
|
|
@@ -6491,9 +7024,9 @@ function validatePipeline(input , ctx = {}) {
|
|
|
6491
7024
|
// says which job, instead of failing every item the same way.
|
|
6492
7025
|
const text = CONTENT_TYPES[content?.type]?.text?.(content, c) ?? null;
|
|
6493
7026
|
targets.forEach((_, i) => {
|
|
6494
|
-
const stages =
|
|
7027
|
+
const stages = jobs.map((ch , k ) => ({ text: targetStepOf(doc, i, k) .prompt, kind: outOf(ch).kind,
|
|
6495
7028
|
verbatim: !!targetEntryOf(doc, i, k)?.verbatim }));
|
|
6496
|
-
const problem = jobProblem(stages, stages.map(() => () => {}),
|
|
7029
|
+
const problem = jobProblem(stages, stages.map(() => () => {}), jobs.map((ch ) => tokensOf(ch)), text);
|
|
6497
7030
|
if (problem) bad.push(`${targetLabel(doc, i)}: ${jobWords(problem)}`);
|
|
6498
7031
|
});
|
|
6499
7032
|
return bad;
|
|
@@ -6506,8 +7039,8 @@ function passProblems(pass , n , bad ) {
|
|
|
6506
7039
|
return void bad.push("pass has to be { mode: all } or { mode: atLeast, count }");
|
|
6507
7040
|
}
|
|
6508
7041
|
onlyFields(pass, "pass", pass.mode === "all" ? ["mode"] : ["mode", "count"], bad);
|
|
6509
|
-
if (pass.mode === "atLeast" && !(Number.isInteger(pass.count) && pass.count > 0)) bad.push("pass at least has to count a whole number of
|
|
6510
|
-
else if (pass.mode === "atLeast" && pass.count > n) bad.push(`pass asks for ${pass.count}
|
|
7042
|
+
if (pass.mode === "atLeast" && !(Number.isInteger(pass.count) && pass.count > 0)) bad.push("pass at least has to count a whole number of evals from 1");
|
|
7043
|
+
else if (pass.mode === "atLeast" && pass.count > n) bad.push(`pass asks for ${pass.count} evals, and the experiment has ${n}`);
|
|
6511
7044
|
}
|
|
6512
7045
|
|
|
6513
7046
|
/** How many of [items] are in [files] (names or `{ name }` entries): a
|
|
@@ -6540,7 +7073,7 @@ function pipelineWarnings(doc , ctx = {}) {
|
|
|
6540
7073
|
if (!items?.length) return;
|
|
6541
7074
|
const found = itemsFound(items, files);
|
|
6542
7075
|
if (found < items.length) {
|
|
6543
|
-
warn.push(`${evalLabel(doc , j)}: ${found} of ${ref .name || ref .id}'s ${items.length} cases have their item in ${where}`);
|
|
7076
|
+
warn.push(`${evalLabel(doc , j)}: ${found} of ${ref .name || ref .id}'s ${items.length} eval cases have their item in ${where}`);
|
|
6544
7077
|
}
|
|
6545
7078
|
});
|
|
6546
7079
|
return warn;
|
|
@@ -6591,7 +7124,7 @@ function connectionProblems(conn , at , from
|
|
|
6591
7124
|
// A key in an address is written into every log line and error message
|
|
6592
7125
|
// that prints it.
|
|
6593
7126
|
if (u && (u.username || u.password)) {
|
|
6594
|
-
bad.push(`${at} has a key in its address, and a key never goes in
|
|
7127
|
+
bad.push(`${at} has a key in its address, and a key never goes in an experiment — $${from} supplies it`);
|
|
6595
7128
|
}
|
|
6596
7129
|
}
|
|
6597
7130
|
if (!isStr(conn.model)) bad.push(`${at}: model has to be text`);
|
|
@@ -6658,10 +7191,12 @@ function answerKeys(doc )
|
|
|
6658
7191
|
* The run document: [doc] plus what its references resolved to, once, at
|
|
6659
7192
|
* submit. `ctx.profiles(key)` answers an answer key with what it resolves to
|
|
6660
7193
|
* (`answersOf`'s `resolve`), `ctx.files` is
|
|
6661
|
-
* the Source's file list as it reads now
|
|
7194
|
+
* the first Get items' Source's file list as it reads now and
|
|
7195
|
+
* `ctx.filesOf(id)` any Source's (a later Get items reads it), `ctx.comment`
|
|
7196
|
+
* is the run's own.
|
|
6662
7197
|
*/
|
|
6663
7198
|
function resolvePipeline(doc ,
|
|
6664
|
-
ctx
|
|
7199
|
+
ctx
|
|
6665
7200
|
= {}) {
|
|
6666
7201
|
const run = clone(doc) ;
|
|
6667
7202
|
// What the lab supplies an eval -- its grader -- before the profiles it
|
|
@@ -6678,12 +7213,14 @@ function resolvePipeline(doc ,
|
|
|
6678
7213
|
const cid = isStr(p.id) && p.id ? p.id : answerOfKey(key).connection;
|
|
6679
7214
|
run.profiles[key] = { ...conn, connection: cid };
|
|
6680
7215
|
}
|
|
6681
|
-
const content = contentOf(run) ;
|
|
6682
7216
|
// Saved response grades its one pinned response (#596), whatever else the
|
|
6683
7217
|
// Source holds; the server keeps that response's bytes with the run.
|
|
6684
|
-
|
|
6685
|
-
content
|
|
6686
|
-
|
|
7218
|
+
itemContents(run).forEach(({ content: c }, k) => {
|
|
7219
|
+
const content = c ;
|
|
7220
|
+
if (content?.type !== "source") return;
|
|
7221
|
+
const files = k === 0 && ctx.files !== undefined ? ctx.files : ctx.filesOf?.(content.ref?.id);
|
|
7222
|
+
content.files = CONTENT_TYPES.source .expand (content, isObj(content.pin) && content.live !== true ? [content.pin.name] : files);
|
|
7223
|
+
});
|
|
6687
7224
|
// Each Library group the evals read, as the version it grades with: the
|
|
6688
7225
|
// body's fingerprint and, where the lab numbers them, `n` -- a pinned
|
|
6689
7226
|
// link's pin.
|
|
@@ -6707,8 +7244,9 @@ function pipelineOfRun(run , ctx = {}) {
|
|
|
6707
7244
|
delete doc.profiles;
|
|
6708
7245
|
delete doc.comment;
|
|
6709
7246
|
delete doc.plugins;
|
|
6710
|
-
const content
|
|
6711
|
-
|
|
7247
|
+
for (const { content } of itemContents(doc)) {
|
|
7248
|
+
if (isObj(content)) { delete (content ).files; delete (content ).revs; }
|
|
7249
|
+
}
|
|
6712
7250
|
// What the run recorded of each group it read is the run's. The grader
|
|
6713
7251
|
// it carried stays, as a Metrics eval's did: a bundle writes it down.
|
|
6714
7252
|
for (const t of doc.evals || []) {
|
|
@@ -6743,16 +7281,22 @@ function pipelineToYaml(doc , ctx
|
|
|
6743
7281
|
return yaml.dump(out, { lineWidth: -1, noRefs: true });
|
|
6744
7282
|
}
|
|
6745
7283
|
|
|
6746
|
-
/** Every AI Connection reference in [doc] -- each
|
|
6747
|
-
steps', each eval's grader -- replaced, in place, by [f] of it.
|
|
6748
|
-
|
|
7284
|
+
/** Every AI Connection reference in [doc] -- each Variant's, each of its
|
|
7285
|
+
steps', each eval's grader -- replaced, in place, by [f] of it. The
|
|
7286
|
+
Variants and their Request steps are reached through the jobs-and-targets
|
|
7287
|
+
intermediate and serialised back; a Connection Profile is named by id and
|
|
7288
|
+
name, and left as it is. */
|
|
6749
7289
|
function eachProfileRef(doc , f ) {
|
|
6750
|
-
|
|
7290
|
+
const jt = jtOf(doc);
|
|
7291
|
+
for (const t of jt.targets) {
|
|
6751
7292
|
if (t.connection) t.connection = f(t.connection);
|
|
6752
7293
|
for (const st of Array.isArray(t.steps) ? t.steps : []) {
|
|
6753
7294
|
if (st?.connection) st.connection = f(st.connection);
|
|
6754
7295
|
}
|
|
6755
7296
|
}
|
|
7297
|
+
const re = serialize(jt.jobs, jt.targets, doc.steps);
|
|
7298
|
+
doc.steps = re.steps;
|
|
7299
|
+
doc.variants = re.variants;
|
|
6756
7300
|
for (const t of Array.isArray(doc.evals) ? doc.evals : []) {
|
|
6757
7301
|
if (isObj(t) && t.grader) t.grader = f(t.grader);
|
|
6758
7302
|
if (isObj(t?.own) && t.own.grader) t.own.grader = f(t.own.grader);
|
|
@@ -6779,14 +7323,17 @@ function importPipeline(input , ctx = {}) {
|
|
|
6779
7323
|
const v = versionProblem(doc);
|
|
6780
7324
|
if (v) return { error: v };
|
|
6781
7325
|
const profiles = ctx.profiles ?? [], sources = ctx.sources ?? [], groups = ctx.groups ?? [];
|
|
6782
|
-
|
|
7326
|
+
// The references are remapped on the jobs-and-targets intermediate (where
|
|
7327
|
+
// content, every target's connection and each step's live), then the result
|
|
7328
|
+
// is serialised back into the version-20 step list and Variants.
|
|
7329
|
+
const next = jtOf(doc);
|
|
6783
7330
|
const missing = [], seen = new Set ();
|
|
6784
7331
|
const lists = { profile: profiles, connectionProfile: ctx.connectionProfiles ?? [], source: sources, group: groups };
|
|
6785
7332
|
const words = {
|
|
6786
7333
|
profile: (ref ) => `Connection ${ref.slug || ref.name || ref.id} not found`,
|
|
6787
7334
|
connectionProfile: (ref ) => `Connection Profile ${ref.name || ref.id} not found`,
|
|
6788
7335
|
source: (ref ) => `Source ${ref.name || ref.id} not found`,
|
|
6789
|
-
group: (ref ) => `Eval
|
|
7336
|
+
group: (ref ) => `Eval ${ref.name || ref.id} not found`,
|
|
6790
7337
|
};
|
|
6791
7338
|
const remap = (ref , kind ) => {
|
|
6792
7339
|
const hit = lists[kind].find(x => x.id === ref.id) || lists[kind].find(x => x.name === ref.name);
|
|
@@ -6808,8 +7355,10 @@ function importPipeline(input , ctx = {}) {
|
|
|
6808
7355
|
if (!seen.has(sentence)) { seen.add(sentence); missing.push(sentence); }
|
|
6809
7356
|
return { id: ref.slug, name: isStr(ref.name) && ref.name ? ref.name : ref.slug };
|
|
6810
7357
|
};
|
|
6811
|
-
const content
|
|
6812
|
-
|
|
7358
|
+
for (const { content: c } of itemContents(next)) {
|
|
7359
|
+
const content = c ;
|
|
7360
|
+
if (content?.type === "source") content.ref = remap(content.ref, "source");
|
|
7361
|
+
}
|
|
6813
7362
|
const split = (connection ) => {
|
|
6814
7363
|
const p = preSplit && isRef(connection) && profiles.some(x => x.id === connection.id) ? ctx.connectionSplit?.(connection.id) : null;
|
|
6815
7364
|
return p ? { id: p.id, name: p.name ?? "" } : null;
|
|
@@ -6845,14 +7394,16 @@ function importPipeline(input , ctx = {}) {
|
|
|
6845
7394
|
// keep the ones it carries, so export → import → export is the same
|
|
6846
7395
|
// document and a re-import can recognise the same pipeline (#93).
|
|
6847
7396
|
mintIds(next);
|
|
7397
|
+
// Serialised back into the version-20 shape the lab reads.
|
|
7398
|
+
const out = serializeDoc(next);
|
|
6848
7399
|
// What a document may still lack on import is its references -- matched
|
|
6849
7400
|
// above and listed as missing. Anything else the model says is wrong is a
|
|
6850
7401
|
// document the lab cannot edit and run as it stands, so it is refused by
|
|
6851
7402
|
// the first such sentence. Validated with no lookups, so a reference a
|
|
6852
7403
|
// later step will satisfy is never the refusal.
|
|
6853
|
-
const bad = validatePipeline(
|
|
7404
|
+
const bad = validatePipeline(out).filter(s => !/ not found$/.test(s));
|
|
6854
7405
|
if (bad.length) return { error: bad[0] };
|
|
6855
|
-
return { doc:
|
|
7406
|
+
return { doc: out, missing };
|
|
6856
7407
|
}
|
|
6857
7408
|
|
|
6858
7409
|
/** Mint, in place, the ids a current-version document must have that it
|
|
@@ -6868,8 +7419,10 @@ function mintIds(doc ) {
|
|
|
6868
7419
|
for (const t of Array.isArray(doc.evals) ? doc.evals : []) {
|
|
6869
7420
|
if (isObj(t) && (!isStr(t.id) || !t.id)) t.id = newId();
|
|
6870
7421
|
}
|
|
6871
|
-
// Nor need it say how its linked groups pass together: all of them.
|
|
6872
|
-
|
|
7422
|
+
// Nor need it say how its linked groups pass together: all of them. (A
|
|
7423
|
+
// document serialised to version 20 carries a `pass` key even when it was
|
|
7424
|
+
// absent, so a null one is filled too.)
|
|
7425
|
+
if (doc.pass == null) doc.pass = { mode: "all" };
|
|
6873
7426
|
return doc;
|
|
6874
7427
|
}
|
|
6875
7428
|
|
|
@@ -7134,7 +7687,8 @@ function readBundle(files , ctx = {})
|
|
|
7134
7687
|
function stagesFor(run , i , resolveFilters )
|
|
7135
7688
|
|
|
7136
7689
|
{
|
|
7137
|
-
const
|
|
7690
|
+
const jobs = jobsOf(run);
|
|
7691
|
+
const stages = jobs.map((ch, k) => {
|
|
7138
7692
|
const { kind, modifiers: _mods, ...settings } = outOf(ch);
|
|
7139
7693
|
// The modifiers a run applies, with the filters below the format a
|
|
7140
7694
|
// synthetic modifier a link where the filters step sits (#339), and those
|
|
@@ -7149,7 +7703,7 @@ function stagesFor(run , i , resolveFilters
|
|
|
7149
7703
|
const { connection: _c, ...conn } = held || {} ;
|
|
7150
7704
|
return { id: answerConnectionId(key, held), ...conn };
|
|
7151
7705
|
};
|
|
7152
|
-
const connections =
|
|
7706
|
+
const connections = jobs.map((_, k) => {
|
|
7153
7707
|
// A step answered here asks nothing: the profile a run made before it
|
|
7154
7708
|
// kept is what it ran as, else Echo itself.
|
|
7155
7709
|
const key = targetAnswerKey(run, i, k);
|
|
@@ -7157,12 +7711,19 @@ function stagesFor(run , i , resolveFilters
|
|
|
7157
7711
|
const held = key != null ? runAnswer(run, key) : undefined;
|
|
7158
7712
|
return held && CONNECTION_TYPES[held.type]?.local ? asked(key , held) : { id: "", ...localConnOf(targetEntryOf(run, i, k)) };
|
|
7159
7713
|
}
|
|
7160
|
-
|
|
7714
|
+
if (key == null) return { id: "" };
|
|
7715
|
+
// A Request's own Model goes over its connection's, where the step's
|
|
7716
|
+
// model is the connection's to send (a Prompt; an HTTP Request places
|
|
7717
|
+
// its own in the body).
|
|
7718
|
+
const { model, settings: own } = targetStepOf(run, i, k) ?? {};
|
|
7719
|
+
const conn = asked(key, runAnswer(run, key));
|
|
7720
|
+
if (targetEntryOf(run, i, k)?.modelFrom !== "profile") return conn;
|
|
7721
|
+
return withOwnSettings(model?.trim() ? { ...conn, model: model.trim() } : conn, own);
|
|
7161
7722
|
});
|
|
7162
7723
|
// What a transport that builds its own request (an HTTP Request) builds
|
|
7163
7724
|
// it from in each job, and this target's step there.
|
|
7164
|
-
return { stages, tokens:
|
|
7165
|
-
calls:
|
|
7725
|
+
return { stages, tokens: jobs.map(tokensOf), connections,
|
|
7726
|
+
calls: jobs.map((_, k) => requestStepOf(run, i, k)), cells: jobs.map((_, k) => targetStepOf(run, i, k) ) };
|
|
7166
7727
|
}
|
|
7167
7728
|
|
|
7168
7729
|
/** The Setup profile scenario [i] of a run ran under, as History shows it: keyless. */
|
|
@@ -7180,11 +7741,11 @@ function scenarioProfile(run , i )
|
|
|
7180
7741
|
}
|
|
7181
7742
|
|
|
7182
7743
|
/** The kind a run's result is read as: its last job's. */
|
|
7183
|
-
const lastKind = (doc
|
|
7744
|
+
const lastKind = (doc ) => outOf(jobsOf(doc).at(-1))?.kind ?? null;
|
|
7184
7745
|
|
|
7185
7746
|
/** A value of a modifier on a run's last job -- the case mode History prints. */
|
|
7186
|
-
const lastModifier = (doc
|
|
7187
|
-
(outOf(doc.
|
|
7747
|
+
const lastModifier = (doc , type ) =>
|
|
7748
|
+
(outOf(jobsOf(doc).at(-1))?.modifiers || []).find(m => m.type === type) || null;
|
|
7188
7749
|
|
|
7189
7750
|
/** What a modifier is set to, in a few words, from its entry's own options:
|
|
7190
7751
|
a choice's label, a number, "3 rules" -- or "on" for one with none. What
|
|
@@ -7237,8 +7798,8 @@ function itemScores(run , kase , res
|
|
|
7237
7798
|
|
|
7238
7799
|
/** What production recorded as the reply to an item, for a run's metrics: its
|
|
7239
7800
|
last job's call says, from the item's record, or nobody does. */
|
|
7240
|
-
function recordedReplyOf(run
|
|
7241
|
-
const last = run.
|
|
7801
|
+
function recordedReplyOf(run , record ) {
|
|
7802
|
+
const last = jobsOf(run).at(-1), flow = flowStepOf(last);
|
|
7242
7803
|
return record && flow ? STEP_TYPES.flowStep .production ({ ...flow, readAs: readAsOf(last) }, record) : null;
|
|
7243
7804
|
}
|
|
7244
7805
|
|
|
@@ -7246,18 +7807,21 @@ function recordedReplyOf(run , record )
|
|
|
7246
7807
|
* itemScores for the runner, which can wait: an eval that reads every item
|
|
7247
7808
|
* (`read`: the Metrics, which may ask a grader) scores one with no case too.
|
|
7248
7809
|
*/
|
|
7249
|
-
async function itemScoresAsync(run , kase , res
|
|
7810
|
+
async function itemScoresAsync(run , kase , res ,
|
|
7250
7811
|
more = {}) {
|
|
7251
7812
|
const out = {};
|
|
7252
7813
|
let stopped = false;
|
|
7814
|
+
// The runner hands over a reply it just made, the re-score and the page a
|
|
7815
|
+
// stored one; a metric reads the same fields of either.
|
|
7816
|
+
const reply = res ;
|
|
7253
7817
|
for (const t of evalsOf(run)) {
|
|
7254
7818
|
const type = EVAL_TYPES[t.type];
|
|
7255
7819
|
if (isWholeRun(type, t) || (!type?.read && !(type?.score && kase))) continue;
|
|
7256
7820
|
if (stopped) { out[t.id] = SKIPPED; continue; }
|
|
7257
7821
|
// An eval with nothing to read on this item leaves no entry, as a graded
|
|
7258
7822
|
// eval does on an item with no case.
|
|
7259
|
-
const s = type.read ? await type.read(t, kase ?? null,
|
|
7260
|
-
: type.score (t, kase ,
|
|
7823
|
+
const s = type.read ? await type.read(t, kase ?? null, reply, { plain: !OUTPUT_KINDS[lastKind(run) ?? ""]?.terms, ...more })
|
|
7824
|
+
: type.score (t, kase , reply);
|
|
7261
7825
|
if (!s) continue;
|
|
7262
7826
|
out[t.id] = s;
|
|
7263
7827
|
if (!s.pass && !t.continueOnFailure) stopped = true;
|
|
@@ -7265,6 +7829,35 @@ async function itemScoresAsync(run , kase ,
|
|
|
7265
7829
|
return Object.keys(out).length ? out : null;
|
|
7266
7830
|
}
|
|
7267
7831
|
|
|
7832
|
+
/**
|
|
7833
|
+
* A finished run's stored results re-scored, the kept replies untouched and
|
|
7834
|
+
* each per-item eval's verdict recomputed against the eval bodies [more]
|
|
7835
|
+
* hands back. This is how the Results view scores a run against the Eval as
|
|
7836
|
+
* it reads now (#657): with [more.group] resolving each linked group to its
|
|
7837
|
+
* current version, and [more.item] each result's own name, an edited eval
|
|
7838
|
+
* case is read at its latest version, and the run itself is not touched --
|
|
7839
|
+
* nothing here is written back.
|
|
7840
|
+
*
|
|
7841
|
+
* No model is asked: pass no [more.grader] and a model-graded metric reads as
|
|
7842
|
+
* unmet, exactly as the eval case editor's live preview does, so the view
|
|
7843
|
+
* makes no call. The one scorer is itemScoresAsync -- the same the runner's
|
|
7844
|
+
* --rescore reaches -- so the page and the runner cannot disagree. A
|
|
7845
|
+
* whole-run eval settles in scenarioEvals from the replies, so it is not
|
|
7846
|
+
* re-read here; only per-item verdicts move.
|
|
7847
|
+
*/
|
|
7848
|
+
async function rescoreResults(run , results ,
|
|
7849
|
+
more = {}) {
|
|
7850
|
+
return Promise.all((results || []).map(async it => {
|
|
7851
|
+
if (!isObj(it) || !Array.isArray(it.scenarios)) return it ;
|
|
7852
|
+
const per = { ...more, item: isStr(it.name) ? it.name : null };
|
|
7853
|
+
const scenarios = await Promise.all((it.scenarios ).map(async side =>
|
|
7854
|
+
isObj(side) && side.res
|
|
7855
|
+
? { ...side, scores: await itemScoresAsync(run, null, side.res, per) }
|
|
7856
|
+
: side));
|
|
7857
|
+
return { ...(it ), scenarios } ;
|
|
7858
|
+
}));
|
|
7859
|
+
}
|
|
7860
|
+
|
|
7268
7861
|
/**
|
|
7269
7862
|
* Every eval's reading of scenario [i] of a run, in the run's order, from
|
|
7270
7863
|
* the items so far. [settled] says every item is in: only then has a
|
|
@@ -7310,13 +7903,41 @@ function scenarioEvals(run , i , items
|
|
|
7310
7903
|
});
|
|
7311
7904
|
}
|
|
7312
7905
|
|
|
7906
|
+
/**
|
|
7907
|
+
* A link's Whole run verdict, per Target, keyed by the eval's id: a Library
|
|
7908
|
+
* group's `run` metrics read over the replies so far (readGroupRun). A private
|
|
7909
|
+
* group's Whole run is scenarioEvals' own (it holds the body); a link's body
|
|
7910
|
+
* is held apart, so it is read here, beside the link's items (§17). [resolve]
|
|
7911
|
+
* gives a link's body by its reference: the runner's --groups, or the page's
|
|
7912
|
+
* kept or current versions. One that gives none reads nothing.
|
|
7913
|
+
*/
|
|
7914
|
+
function wholeRunsOf(run , items ,
|
|
7915
|
+
resolve ) {
|
|
7916
|
+
const kind = lastKind(run);
|
|
7917
|
+
return targetsOf(run).map((_, i) => {
|
|
7918
|
+
const out = {};
|
|
7919
|
+
const ress = (items || []).map(it => (it && !it.unrun ? it.scenarios?.[i]?.res : undefined));
|
|
7920
|
+
for (const t of evalsOf(run)) {
|
|
7921
|
+
if (t.type !== "group" || isWholeRun(EVAL_TYPES.group, t)) continue;
|
|
7922
|
+
const body = groupOf(t, { group: resolve });
|
|
7923
|
+
const v = Array.isArray(body.run) && body.run.length ? readGroupRun(body , ress, kind) : null;
|
|
7924
|
+
if (v) out[t.id] = v;
|
|
7925
|
+
}
|
|
7926
|
+
return out;
|
|
7927
|
+
});
|
|
7928
|
+
}
|
|
7929
|
+
|
|
7313
7930
|
/** One eval group's verdict over a scenario (§17): a whole-run group's own
|
|
7314
|
-
verdict, or a per-item group's "every item it read passed"
|
|
7315
|
-
|
|
7931
|
+
verdict, or a per-item group's "every item it read passed" -- and, for a
|
|
7932
|
+
link reading a Whole run beside its items, that too: either failing fails
|
|
7933
|
+
it, as the runner's verdict does. Null where it was skipped, or had
|
|
7934
|
+
nothing to read yet. */
|
|
7316
7935
|
function groupPass(o ) {
|
|
7317
7936
|
if (o.skipped) return null;
|
|
7318
7937
|
if (o.whole) return o.verdict?.ran ? o.verdict.pass : null;
|
|
7319
|
-
|
|
7938
|
+
const item = o.ran ? o.passed === o.ran : null;
|
|
7939
|
+
const whole = o.wholeRun?.ran ? o.wholeRun.pass : null;
|
|
7940
|
+
return item === false || whole === false ? false : item ?? whole;
|
|
7320
7941
|
}
|
|
7321
7942
|
|
|
7322
7943
|
/** A scenario's pass or fail over every eval that read it: null where none
|
|
@@ -7366,25 +7987,25 @@ function passVerdict(pass , passes
|
|
|
7366
7987
|
function validateEvals(ev , files ) {
|
|
7367
7988
|
const bad = [];
|
|
7368
7989
|
if (!ev || typeof ev !== "object" || Array.isArray(ev)) {
|
|
7369
|
-
return ["
|
|
7990
|
+
return ["an eval has to be a JSON object with a `cases` list"];
|
|
7370
7991
|
}
|
|
7371
7992
|
if (ev.cases != null && !Array.isArray(ev.cases)) bad.push("cases has to be a list");
|
|
7372
7993
|
// An eval group's own sections (version 7): what a Metrics eval held.
|
|
7373
7994
|
if (ev.scoring != null) {
|
|
7374
7995
|
const sc = ev.scoring;
|
|
7375
|
-
if (!isObj(sc) || !SCORING_MODES.includes(sc.mode)) bad.push(`
|
|
7376
|
-
else if (sc.mode === "weighted" && typeof sc.threshold !== "number") bad.push("
|
|
7377
|
-
else if (sc.threshold != null && typeof sc.threshold !== "number") bad.push("
|
|
7996
|
+
if (!isObj(sc) || !SCORING_MODES.includes(sc.mode)) bad.push(`an eval is scored ${SCORING_MODES.join(" or ")}`);
|
|
7997
|
+
else if (sc.mode === "weighted" && typeof sc.threshold !== "number") bad.push("an eval scored in points needs Pass at: the points an item has to reach");
|
|
7998
|
+
else if (sc.threshold != null && typeof sc.threshold !== "number") bad.push("an eval's Pass at has to be a number");
|
|
7378
7999
|
}
|
|
7379
|
-
if (ev.grader != null && !isRef(ev.grader)) bad.push("
|
|
7380
|
-
if (ev.every != null) metricsProblems(ev.every, "Every item", bad);
|
|
8000
|
+
if (ev.grader != null && !isRef(ev.grader)) bad.push("an eval names its grader as { id, name }, or null");
|
|
8001
|
+
if (ev.every != null) metricsProblems(ev.every, "Every item metrics", bad);
|
|
7381
8002
|
if (ev.run != null) {
|
|
7382
|
-
metricsProblems(ev.run, "Whole run", bad);
|
|
8003
|
+
metricsProblems(ev.run, "Whole run metrics", bad);
|
|
7383
8004
|
// Read once over every reply: nothing that reads one item, or asks a grader.
|
|
7384
8005
|
for (const m of Array.isArray(ev.run) ? ev.run : []) {
|
|
7385
8006
|
const entry = isObj(m) ? METRICS[m.type] : undefined;
|
|
7386
|
-
if (entry?.graded) bad.push(`Whole run: ${entry.label} is model-graded, so it cannot read a whole run`);
|
|
7387
|
-
else if (entry?.perItem) bad.push(`Whole run: ${entry.label} reads one item at a time, so it cannot read a whole run`);
|
|
8007
|
+
if (entry?.graded) bad.push(`Whole run metrics: ${entry.label} is model-graded, so it cannot read a whole run`);
|
|
8008
|
+
else if (entry?.perItem) bad.push(`Whole run metrics: ${entry.label} reads one item at a time, so it cannot read a whole run`);
|
|
7388
8009
|
}
|
|
7389
8010
|
}
|
|
7390
8011
|
if (bad.length) return bad;
|
|
@@ -7395,13 +8016,13 @@ function validateEvals(ev , files
|
|
|
7395
8016
|
const seen = new Set ();
|
|
7396
8017
|
for (const c of graded) {
|
|
7397
8018
|
if (!isObj(c)) {
|
|
7398
|
-
bad.push("cases holds something that is not
|
|
8019
|
+
bad.push("cases holds something that is not an eval case");
|
|
7399
8020
|
continue;
|
|
7400
8021
|
}
|
|
7401
|
-
if (!isStr(c.id) || !c.id.trim()) bad.push("
|
|
8022
|
+
if (!isStr(c.id) || !c.id.trim()) bad.push("an eval case has no id");
|
|
7402
8023
|
else if (seen.has(c.id)) bad.push(`${c.id} is graded twice`);
|
|
7403
8024
|
else seen.add(c.id);
|
|
7404
|
-
if (!caseItem(c).trim()) bad.push(`${c.id || "
|
|
8025
|
+
if (!caseItem(c).trim()) bad.push(`${c.id || "an eval case"} names no item`);
|
|
7405
8026
|
if (c.todo != null && typeof c.todo !== "boolean") bad.push(`${c.id}: todo is true or false`);
|
|
7406
8027
|
if (c.note != null && !isStr(c.note)) bad.push(`${c.id}: note has to be text`);
|
|
7407
8028
|
if (c.metrics != null) metricsProblems(c.metrics, String(c.id), bad);
|
|
@@ -7483,8 +8104,8 @@ function validateEvals(ev , files
|
|
|
7483
8104
|
// set and graded by nobody is one nobody discovers is ungraded.
|
|
7484
8105
|
for (const f of shown) {
|
|
7485
8106
|
if (!gradedAt(f)) {
|
|
7486
|
-
bad.push(`${f} is in the Source and this
|
|
7487
|
-
+ `so the Cases view does not list it`);
|
|
8107
|
+
bad.push(`${f} is in the Source and this eval does not grade it, `
|
|
8108
|
+
+ `so the Eval Cases view does not list it`);
|
|
7488
8109
|
}
|
|
7489
8110
|
}
|
|
7490
8111
|
return bad;
|
|
@@ -7548,18 +8169,18 @@ export {
|
|
|
7548
8169
|
tallyPercent, runPipeline, validateEvals, evalsWarnings, evalsJson, readList,
|
|
7549
8170
|
scoreSingle, singleRules, parseCount, lengthHolds, countHolds, COUNT_OPS, LENGTH_OPS,
|
|
7550
8171
|
COUNT_OP_LABEL, LENGTH_OP_LABEL,
|
|
7551
|
-
PIPELINE_VERSION, DATASET_BODY_VERSION, STEP_TYPES, CONTENT_TYPES, OUTPUT_KINDS, MODIFIERS, EVAL_TYPES, LEGACY_TESTS, METRICS, readMetrics, metricSummary, metricCheck, itemScoresAsync, recordedReplyOf,
|
|
8172
|
+
PIPELINE_VERSION, DATASET_BODY_VERSION, STEP_TYPES, CONTENT_TYPES, OUTPUT_KINDS, MODIFIERS, EVAL_TYPES, LEGACY_TESTS, METRICS, readMetrics, metricSummary, metricCheck, itemScoresAsync, rescoreResults, recordedReplyOf,
|
|
7552
8173
|
SOURCE_TYPES, DEFAULT_SOURCE_TYPE, sourceTypeOf, WORKFLOW_PLATFORMS, platformOf, DYNAMIC_PROVIDERS, providerOf, WIZARDS, wizardProgress,
|
|
7553
8174
|
registerKinds, defaultKind, jobDefaults, applyModifiers, keyVar, slugOf, uniqueSlug, isSlug, withSlugs, SLUG_MAX, connectionOf, addressOf, connectionSettings,
|
|
7554
8175
|
splitProfile, mergeAnswer, answerKey, answerOfKey, answersOf, runAnswer, answerConnectionId, connectionScoped,
|
|
7555
8176
|
connectionRequest, connectionBase, connectionSend, connectionReply, connectionReplyProblem, replyFinish, contentJoined, connectionImage, connectionProblems,
|
|
7556
8177
|
CONNECTION_FIELDS, SETTING_KEYS, OPTION_FIELDS, jobLabel, scenarioLabel, jobWords, versionProblem,
|
|
7557
|
-
contentOf, withContent, replyOf,
|
|
8178
|
+
contentOf, itemContents, runItemsOf, withContent, replyOf, jobsOf, withJobs, withTargets, serialize, stepNames, stepFaults,
|
|
7558
8179
|
targetsOf, targetLetter, targetLabel, targetStepOf, targetEntryOf, jobTargetType, targetAnswerOf, targetAnswerKey, recordedTargetOf, withTarget,
|
|
7559
8180
|
tokensOf, withTokens, sendsImage, withImage, flowStepOf, requestStepOf, readAsOf, outOf, withOut, slotEntry, SLOTS,
|
|
7560
8181
|
filterSetsOf, filterFormatOf, withFilterSets, formatPlace, stageModifiers, beforeFilters, gradedText,
|
|
7561
8182
|
blankPipeline, upgradePipeline, fatProfileRef, newId, newEval, newLink, validatePipeline, resolvePipeline, pipelineOfRun, stagesFor, profileIds, answerKeys,
|
|
7562
|
-
scenarioProfile, lastKind, lastModifier, modifierSummary, itemScores, scenarioEvals, scenarioPasses, groupPass, overallVerdict, passVerdict, isSkipped, failedScore,
|
|
8183
|
+
scenarioProfile, lastKind, lastModifier, modifierSummary, itemScores, scenarioEvals, wholeRunsOf, scenarioPasses, groupPass, overallVerdict, passVerdict, isSkipped, failedScore,
|
|
7563
8184
|
evalLabel, evalsOf, evalsDataset, casesRef, groupOf, groupOfMetrics, pipelineWarnings, itemsFound, previewOf, isWholeRun, EVAL_FIELDS,
|
|
7564
8185
|
pipelineToYaml, importPipeline, yamlToPipeline, exportBundle, readBundle, bundleProblems, junitXml,
|
|
7565
8186
|
pluginHost, legacyTagsOut, legacyTagsModifiers, legacyTagsJob, splitItems, tidy, tryExamples, itemRulesProblems, dropBy, rejectBy, matcher,
|