foldrun 2.6.0 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +95 -29
- package/LICENSING.md +101 -0
- package/README.md +9 -4
- package/bin/foldrun.mjs +18 -1
- package/docs/agents.md +11 -5
- package/docs/api-usage.md +8 -1
- package/docs/api.md +12 -4
- package/docs/cli.md +17 -3
- package/docs/evals.md +151 -1
- package/docs/flows.md +63 -2
- package/docs/hooks.md +2 -2
- package/docs/introduction.md +6 -4
- package/docs/network.md +6 -2
- package/docs/runs.md +28 -0
- package/docs/self-hosting.md +48 -10
- package/docs/tools.md +1 -1
- package/docs/versions.md +2 -2
- package/package.json +8 -4
- package/src/commands.mjs +488 -19
- package/src/guide.mjs +1 -1
package/docs/evals.md
CHANGED
|
@@ -35,7 +35,9 @@ satisfy. The heading is the case's name, so make it the thing being checked.
|
|
|
35
35
|
| `agent` **or** `flow` | what runs. A flow eval runs the whole flow per case |
|
|
36
36
|
| `model` | the **judge's** tier, not the subject's — cases run the agent on its own model. Unset, the workspace's `judge:` in `AGENTS.md` (see [The four jobs](four-jobs#judge--the-evaluation-default)), else the fast tier |
|
|
37
37
|
| `effort` | the judge's effort. PASS/FAIL against one sentence is not deep work |
|
|
38
|
-
| `trigger` | `deploy` (default) or `
|
|
38
|
+
| `trigger` | `deploy` (default), `manual`, or `runs` — grade real finished runs instead of running cases (see [Grading live runs](#grading-live-runs)) |
|
|
39
|
+
| `sample` | `trigger: runs` — the percent of finished runs graded, 0 to 100 (default 25) |
|
|
40
|
+
| `improve` | `trigger: runs` — `true` (5) or a count: once that many graded runs failed, a fix is proposed for a person to review (see [Proposed fixes](#proposed-fixes)) |
|
|
39
41
|
| `inputs` | `true` makes the file a flow's saved inputs, not a test: its cases are named tasks for [Run with…](flows#run-with-saved-inputs) and `foldrun invoke --inputs`, and nothing runs them as an eval |
|
|
40
42
|
| `live` | an eval's runs are [test runs](runs#test-runs) — nothing outward, `state/` untouched — unless this says `true`. A flow eval that must really send to prove itself opts in; nothing opts in by accident |
|
|
41
43
|
|
|
@@ -69,6 +71,154 @@ so it counts against the account's `concurrency:` and waits its turn.
|
|
|
69
71
|
eval runs the entire flow once per case, so a three-case eval on an hour-long
|
|
70
72
|
flow is three hours and the full cost, on every deploy.
|
|
71
73
|
|
|
74
|
+
## Grading live runs
|
|
75
|
+
|
|
76
|
+
An eval tests the cases someone wrote. A grader tests the runs that actually
|
|
77
|
+
happened: the 3am desk run nobody watched, held to the same checks. It is an
|
|
78
|
+
eval with `trigger: runs` and no tasks — each `##` heading is a check, with
|
|
79
|
+
only an `expect:` list:
|
|
80
|
+
|
|
81
|
+
```markdown
|
|
82
|
+
---
|
|
83
|
+
name: digest-quality
|
|
84
|
+
flow: weekly-digest # or agent: writer — that agent's steps, in any flow
|
|
85
|
+
trigger: runs
|
|
86
|
+
sample: 25 # percent of finished runs graded
|
|
87
|
+
model: fast # the judge
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## cites its sources
|
|
91
|
+
expect:
|
|
92
|
+
- judge: every claim names the source it came from
|
|
93
|
+
|
|
94
|
+
## no placeholders
|
|
95
|
+
expect:
|
|
96
|
+
- not-contains: lorem ipsum
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
- **What it grades.** For `flow:`, every step's reply joined; for `agent:`,
|
|
100
|
+
that agent's replies, in whichever flow it ran. The judge is also given
|
|
101
|
+
the run's task, so it grades the reply against what was asked.
|
|
102
|
+
- **Which runs.** Completed ones — a crash is already an
|
|
103
|
+
[issue](runs#issues). Not test runs, not eval runs, and only those that
|
|
104
|
+
finished after the grader was first deployed: a new grader starts from
|
|
105
|
+
now, it does not grade a week of runs at once. The sample is chosen by a
|
|
106
|
+
hash of the run id, so the same run is always in or out.
|
|
107
|
+
- **When.** The platform grades a few runs every five minutes, at most four
|
|
108
|
+
per account a pass, so a backlog drains steadily rather than spending at
|
|
109
|
+
once. A paused account or workspace is not graded.
|
|
110
|
+
- **The checks.** `contains:`, `not-contains:` and `judge:`, cheap ones
|
|
111
|
+
first; the judge only runs when they pass. `file:` and `run:` are refused
|
|
112
|
+
(`foldrun check` and the deploy say so): a finished run's sandbox is gone,
|
|
113
|
+
and they would act on the platform's host instead. `matches:` is refused
|
|
114
|
+
too — a regular expression over whatever a run said, on the platform's
|
|
115
|
+
scheduler, could stall it. A `task:` in a grader does nothing and is
|
|
116
|
+
refused as well.
|
|
117
|
+
- **A judge that does not answer** (a spent usage cap, a provider outage)
|
|
118
|
+
leaves the run *not decided*, which is not a failure. It is tried again
|
|
119
|
+
after an hour, three times at most.
|
|
120
|
+
- **What it costs.** One judge call per `judge:` line per graded run — at
|
|
121
|
+
most five lines in a grader — on the account's own model key like any
|
|
122
|
+
eval judge. A paused account or workspace is not graded, by the pass or
|
|
123
|
+
by **Grade a run**. The card shows what the
|
|
124
|
+
judges cost; the runs' own cost is on the runs.
|
|
125
|
+
|
|
126
|
+
The grades are on the Evals page: each grader's card has its pass rate over
|
|
127
|
+
7 and 30 days, a bar per day, the recent failures with the check that failed
|
|
128
|
+
and the judge's reason, and **Grade a run** to grade any finished run now,
|
|
129
|
+
sampled or not (a run graded again shows its newer grade). A run's own page
|
|
130
|
+
lists every grade it got. From the CLI, `foldrun eval grades <eval> --to
|
|
131
|
+
<workspace>` (`--status fail`, `--days 7`) and `foldrun eval grade <eval>
|
|
132
|
+
--run <id> --to <workspace>`; the API is `GET …/evals/<eval>/grades` and
|
|
133
|
+
`POST …/evals/<eval>/grade`. Grades live in `evals/.results/grades/`, kept
|
|
134
|
+
90 days; a deploy never touches them.
|
|
135
|
+
|
|
136
|
+
A grader never runs as cases: not on deploy, not from Run, not from
|
|
137
|
+
`foldrun eval`.
|
|
138
|
+
|
|
139
|
+
## Proposed fixes
|
|
140
|
+
|
|
141
|
+
A failure that keeps happening should turn into a change someone can look
|
|
142
|
+
at, not sit on a card. A grader with `improve: true` (or `improve: 3` — how
|
|
143
|
+
many failures) opens a **proposal** on its own once that many graded runs
|
|
144
|
+
have failed and no proposal has covered them, or once the oldest such failure
|
|
145
|
+
is a week old:
|
|
146
|
+
|
|
147
|
+
```yaml
|
|
148
|
+
trigger: runs
|
|
149
|
+
improve: true # 5 failed grades nobody has looked at → a proposal
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
- **The assistant drafts it.** It is given a short digest of the failures —
|
|
153
|
+
at most ten, each the run, the check and the judge's reason — and reads
|
|
154
|
+
the agent, the flow, their skills and the runs themselves. It is asked for
|
|
155
|
+
the smallest change that fixes the cause, never to loosen a check, and to
|
|
156
|
+
propose nothing when the fault is not in the files (a missing secret, an
|
|
157
|
+
outage).
|
|
158
|
+
- **Only agents, flows and skills.** A proposal may change
|
|
159
|
+
`agents/<name>/agent.md`, `flows/<name>.md` and `skills/<name>/…`; anything
|
|
160
|
+
else the assistant proposes is left out, and the proposal says so.
|
|
161
|
+
- **Nothing is written until you apply it.** Apply checks that every file
|
|
162
|
+
still reads what the proposal was drawn against and can be written, then
|
|
163
|
+
writes them all, each a revision by you (so History can take it back). A
|
|
164
|
+
file someone changed since is refused — dismiss it and propose again. A
|
|
165
|
+
check error the deploy would refuse blocks Apply.
|
|
166
|
+
- **At most one open proposal per eval**, and the automatic ones at most one
|
|
167
|
+
a day per grader. A failure is *looked at* once a proposal covers it —
|
|
168
|
+
drafting, ready, applied or dismissed; a draft that failed releases its
|
|
169
|
+
failures so they count again. A draft that has not finished in half an
|
|
170
|
+
hour has failed.
|
|
171
|
+
- **What it costs**: one assistant reply, paid the way Ask AI is.
|
|
172
|
+
|
|
173
|
+
They are on the eval's card under **Proposed fixes**: what failed, the
|
|
174
|
+
assistant's reasoning, a diff per file, and Apply or Dismiss. **Propose a
|
|
175
|
+
fix** asks for one now — for a grader, from the failures no proposal has
|
|
176
|
+
covered; for an eval that runs cases, from its last result's failed cases.
|
|
177
|
+
Ready proposals are in your [inbox](runs#your-inbox). From the CLI: `foldrun
|
|
178
|
+
eval improve <eval> --to <workspace>` asks for one and waits for the draft;
|
|
179
|
+
`foldrun proposals --to <workspace>` (`--status ready`), `foldrun proposals
|
|
180
|
+
show|apply|dismiss <id> --to <workspace>`. The API: `POST
|
|
181
|
+
…/evals/<eval>/improve`, `GET …/proposals`, `GET`/`POST …/proposals/<id>`.
|
|
182
|
+
|
|
183
|
+
## Comparing models
|
|
184
|
+
|
|
185
|
+
"Could this step run on the fast model?" is answered from the cases you
|
|
186
|
+
already have. **Compare models** on an eval's card runs every case under two
|
|
187
|
+
to four settings — a model (`fast`, `default`, `max`, or a model id), an
|
|
188
|
+
effort, and for a flow eval either every step or one agent's — and puts them
|
|
189
|
+
side by side: how many passed, what the runs and their judges cost, the cost
|
|
190
|
+
and time per run, and each case under each setting, so you see where they
|
|
191
|
+
differ.
|
|
192
|
+
|
|
193
|
+
- **The model is forced on the steps**, so it wins over the agent's own
|
|
194
|
+
`model:` and the flow's defaults. "every step" forces all of a flow's
|
|
195
|
+
steps; naming an agent forces only that agent's.
|
|
196
|
+
- **Every run is a test run**, whatever the eval or the flow says about
|
|
197
|
+
`live:` — a comparison is never the reason something was sent.
|
|
198
|
+
- **The judge is the eval's own** and the same for every setting: what
|
|
199
|
+
changes is the subject, never the grader.
|
|
200
|
+
- **`repeat`** (1–5) runs each case more than once; one run each tells you
|
|
201
|
+
little where the model is not deterministic. Repeats interleave — each
|
|
202
|
+
setting once, then again — so a provider that slows down mid-way slows
|
|
203
|
+
every setting alike.
|
|
204
|
+
- **Bounded twice**: at most 60 runs (cases × settings × repeat), and a spend
|
|
205
|
+
limit (default $5, at most $100) — once the runs and judges have spent it,
|
|
206
|
+
the rest are recorded as not run and the result says it stopped early.
|
|
207
|
+
- **Time per run** is from a worker picking the run up to its end; time in
|
|
208
|
+
the queue is left out, so a busy account does not read as a slow model.
|
|
209
|
+
- **What it writes**: the comparison's own result, kept with the last 20 —
|
|
210
|
+
never the eval's latest result or its history. One comparison of an eval
|
|
211
|
+
at a time.
|
|
212
|
+
|
|
213
|
+
It runs on the platform and can take as long as its runs do; leave the page
|
|
214
|
+
and come back. From the CLI: `foldrun eval compare <eval> --models
|
|
215
|
+
fast,default,max --to <workspace>` (`--models fast:low,max:high` for
|
|
216
|
+
efforts, `--agent <name>`, `--repeat 3`, `--max-usd 10`; `--list` and `--run
|
|
217
|
+
<id>` read the kept ones). The API is `POST …/evals/<eval>/compare`, then
|
|
218
|
+
`GET …/compare?run=<id>`. A grader (`trigger: runs`) has no cases, so it
|
|
219
|
+
cannot be compared, and neither can an eval with a `run:` check — a
|
|
220
|
+
comparison repeats every check many times, and `run:` executes a command.
|
|
221
|
+
|
|
72
222
|
## Promoting a run
|
|
73
223
|
|
|
74
224
|
A run is a case you already have — the task is on record. A run that worked
|
package/docs/flows.md
CHANGED
|
@@ -106,7 +106,7 @@ begins.
|
|
|
106
106
|
| `schema:` | with `output: json` — the JSON Schema the value must satisfy: one-line JSON, a YAML block under the key, or a file under the workspace |
|
|
107
107
|
| `model:` | override the tier for this step |
|
|
108
108
|
| `effort:` | override how hard it thinks for this step |
|
|
109
|
-
| `retry:` | attempts after the first, clamped to 5. Each retry waits — 15 s, 30 s, 1 m, 2 m, 4 m, jittered — holding no sandbox while it waits; a stop ends the wait. An attempt the cluster ended for want of memory or disk (OOMKilled, Evicted) is retried one `size:` class up. An `outward` step whose check failed after its tools ran is not retried — the retry would send again; `check` and `deploy` warn on `retry:` on an outward step, because a failure mid-send is still retried |
|
|
109
|
+
| `retry:` | attempts after the first, clamped to 5. Each retry waits — 15 s, 30 s, 1 m, 2 m, 4 m, jittered — holding no sandbox while it waits; a stop ends the wait. An attempt the cluster ended for want of memory or disk (OOMKilled, Evicted) is retried one `size:` class up. A step refused because the model usage cap is reached is not retried: the window resets in hours, not within the waits. An `outward` step whose check failed after its tools ran is not retried — the retry would send again; `check` and `deploy` warn on `retry:` on an outward step, because a failure mid-send is still retried |
|
|
110
110
|
| `timeout:` | seconds, or `90s` / `15m` / `4h` / `3d` — **the only clock there is** |
|
|
111
111
|
| `max_turns:` | the most model turns before the step is stopped (1 to 500) — the third bound beside `budget:` and `timeout:` |
|
|
112
112
|
| `limits:` | this step's call limits, on one line — `limits: {web.search: 10, calls: 50}` — over the agent's, key by key. A call past one is refused, never run ([Tools](tools#limits--how-many-calls-a-step-may-make)) |
|
|
@@ -115,7 +115,7 @@ begins.
|
|
|
115
115
|
| `ask:` | the same gate carrying a question; the typed answer reaches the prompt (a plain approval with no note reaches it as "approved with no note") |
|
|
116
116
|
| `preview:` | what the gate shows: paths under `storage/` (a `workspace/storage/` prefix is accepted), comma-separated, globs allowed — `draft/*.mdx, draft/images/*.webp`; at most 12 files |
|
|
117
117
|
| `wait:` | `3d` (`s`/`m`/`h`/`d`, capped at 30 days) or `event` — the run page shows the event URL and can release it with a payload, recorded as released by you (approve with `payload`) |
|
|
118
|
-
| `on-fail:` | another agent takes the step over, with the failure as context — and, for a gated step, the person's approval and answer, which it acts within. The rescuer owes what the step owed: its `verify:`, its `output:`, and a `loop:` step's `until:` marker (a loop that runs out of cycles goes to its rescuer too). A `verify:` that accepts `BLOCKED` never fails, so it never reaches the rescuer |
|
|
118
|
+
| `on-fail:` | another agent takes the step over, with the failure as context — and, for a gated step, the person's approval and answer, which it acts within. The rescuer owes what the step owed: its `verify:`, its `output:`, and a `loop:` step's `until:` marker (a loop that runs out of cycles goes to its rescuer too). A `verify:` that accepts `BLOCKED` never fails, so it never reaches the rescuer. A step refused by the model supply itself — a spent usage window, a credential refused — is not handed to a rescuer on the same supply (the same `provider:`, or both on the workspace's): it would be refused the same way, and the issue is filed as the cap or the credential |
|
|
119
119
|
| `delegate:` | the step's agent picks who runs next, from this set |
|
|
120
120
|
|
|
121
121
|
### What is retried without a `retry:`
|
|
@@ -329,6 +329,7 @@ normal outcome, recorded as one.
|
|
|
329
329
|
| `path` | the `storage/` prefix to watch, for `trigger: storage` |
|
|
330
330
|
| `url` / `every` | what to poll and how often (default 15m), for `trigger: watch` |
|
|
331
331
|
| `signature` / `signing_secret` | `github` \| `stripe` \| `slack` \| `hmac`, and the vault entry to check it against |
|
|
332
|
+
| `filter` | for a `webhook`, `email`, `flow` or `storage` trigger: the fields a delivery must have to start a run — see [Which deliveries start a run](#which-deliveries-start-a-run) |
|
|
332
333
|
| `model` / `effort` | defaults for every step |
|
|
333
334
|
| `budget` | the most one run may spend, in USD — a number, or `unlimited`; unset is no limit |
|
|
334
335
|
| `overlap` | `skip` or `queue` when a run is already live |
|
|
@@ -398,6 +399,66 @@ A flow may not chain on itself. First sightings for `watch` record without
|
|
|
398
399
|
firing, and a `once` instant more than six hours gone at first sighting is
|
|
399
400
|
recorded and never fired.
|
|
400
401
|
|
|
402
|
+
### Which deliveries start a run
|
|
403
|
+
|
|
404
|
+
`filter:` says which deliveries are this flow's. A GitHub hook for pull
|
|
405
|
+
requests posts for drafts, bot pushes and label changes alike; without a
|
|
406
|
+
filter each one is a run whose first step only says "not this one". With
|
|
407
|
+
one, a delivery that does not match starts nothing — no sandbox, no model
|
|
408
|
+
call — and the Triggers page lists it as *did not match the filter*, with the
|
|
409
|
+
condition that held it back.
|
|
410
|
+
|
|
411
|
+
```yaml
|
|
412
|
+
trigger: webhook
|
|
413
|
+
filter:
|
|
414
|
+
action: [opened, reopened] # any of these values
|
|
415
|
+
pull_request:
|
|
416
|
+
draft: false # exactly this value
|
|
417
|
+
base:
|
|
418
|
+
ref: { starts_with: release/ } # text tests: starts_with, ends_with, contains
|
|
419
|
+
sender:
|
|
420
|
+
type: { not_in: [Bot] } # none of these
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
What it is matched against depends on the trigger:
|
|
424
|
+
|
|
425
|
+
| trigger | the fields |
|
|
426
|
+
|---|---|
|
|
427
|
+
| `webhook` | the request's JSON body — a body that is not JSON matches no filter |
|
|
428
|
+
| `email` | `from`, `to`, `subject`, `text` |
|
|
429
|
+
| `flow` | `run`, `flow`, `status`, `ending` (`completed`, `failed` or `blocked`), `verdict`, `summary` |
|
|
430
|
+
| `storage` | `by` (who wrote), `paths` (the files that landed under `path:`) |
|
|
431
|
+
|
|
432
|
+
The rules, all of them:
|
|
433
|
+
|
|
434
|
+
- Every key must match. A nested map matches the nested field.
|
|
435
|
+
- A list is the values allowed — any of them. A plain value is the one value.
|
|
436
|
+
- An operator map says it explicitly: `in`, `not_in`, `exists` (`true` or
|
|
437
|
+
`false`), and the text tests `starts_with`, `ends_with` and `contains`.
|
|
438
|
+
There are no regular expressions: a sender controls the body, and a
|
|
439
|
+
pattern it could make backtrack has no place before a run starts.
|
|
440
|
+
- Strings compare exactly and case-sensitively; numbers numerically (`1`
|
|
441
|
+
equals `1.0`; the text `"1"` does not); `true`, `false` and `null` as
|
|
442
|
+
themselves.
|
|
443
|
+
- A missing field fails a value, `in` and the text tests, and passes
|
|
444
|
+
`not_in`. `exists: false` passes only when it is missing.
|
|
445
|
+
- A list in the delivery: a nested map matches when any element does;
|
|
446
|
+
`in` when any element is allowed, `not_in` when none is.
|
|
447
|
+
- It fails closed. A filter `foldrun check` cannot read is a check error,
|
|
448
|
+
and if one reaches the platform it matches nothing — a typo holds
|
|
449
|
+
deliveries back instead of letting all of them through. At most 10 levels
|
|
450
|
+
deep, 64 conditions, 32 values in a list.
|
|
451
|
+
|
|
452
|
+
A filtered delivery counts toward nothing: not `idempotency:`, not
|
|
453
|
+
`throttle:`, not `debounce:`, not `disable_after:`. A paused account or
|
|
454
|
+
workspace is still told it is paused first. `filter:` on a `schedule`,
|
|
455
|
+
`once`, `watch` or `manual` flow does nothing, and `foldrun check` says so.
|
|
456
|
+
|
|
457
|
+
Try a delivery without sending one: the Triggers page has a **Test filter**
|
|
458
|
+
box for each filtered flow, `foldrun flow filter <flow> --payload body.json`
|
|
459
|
+
reads the local file (add `--to <workspace>` for the deployed one), and the
|
|
460
|
+
API is `POST …/flows/<flow>/filter { payload }`.
|
|
461
|
+
|
|
401
462
|
### How many of those starts are real
|
|
402
463
|
|
|
403
464
|
A trigger says what happens. These four say how much of it becomes a run,
|
package/docs/hooks.md
CHANGED
|
@@ -105,8 +105,8 @@ hook can narrow what an agent does but never widen it.
|
|
|
105
105
|
## What a hook does not see
|
|
106
106
|
|
|
107
107
|
Tools the model provider runs on its own side never pass foldrun's dispatch.
|
|
108
|
-
With `web: {search: anthropic}` (or another provider's search
|
|
109
|
-
|
|
108
|
+
With `web: {search: anthropic}` (or another provider's search) the search
|
|
109
|
+
runs at the provider, and a `tool.call` hook
|
|
110
110
|
cannot refuse or rewrite it. foldrun's own `web` tool, the default, goes
|
|
111
111
|
through hooks like every other tool. `foldrun check` warns when an agent with
|
|
112
112
|
a `tool.call` hook uses a provider's own search.
|
package/docs/introduction.md
CHANGED
|
@@ -32,12 +32,14 @@ The design bets on three things:
|
|
|
32
32
|
| | you need | what runs where |
|
|
33
33
|
|---|---|---|
|
|
34
34
|
| **Locally** | `npm install -g foldrun` and a model key | everything on your machine; no account, no server |
|
|
35
|
-
| **
|
|
35
|
+
| **Your own server** | Docker and the public repositories | the whole platform on one machine of yours, for you or your organisation |
|
|
36
|
+
| **Hosted** | an account | ours: schedules, sandboxes, a vault, a team — nothing to run |
|
|
36
37
|
|
|
37
38
|
The same workspace moves between them with `foldrun deploy`.
|
|
38
|
-
Start locally; that is where every workspace should be born.
|
|
39
|
-
|
|
40
|
-
|
|
39
|
+
Start locally; that is where every workspace should be born. foldrun is
|
|
40
|
+
source-available under the Functional Source License: free for your own
|
|
41
|
+
use and your organisation's, not for offering it to others as a service —
|
|
42
|
+
see [Running it yourself](self-hosting).
|
|
41
43
|
|
|
42
44
|
## Where to go
|
|
43
45
|
|
package/docs/network.md
CHANGED
|
@@ -110,8 +110,12 @@ Every program in the sandbox reaches the internet through the **egress
|
|
|
110
110
|
proxy**: the pod's `HTTPS_PROXY` and `HTTP_PROXY` name it, with the step's
|
|
111
111
|
own lease as the password, and `NODE_USE_ENV_PROXY=1` turns that on for Node.
|
|
112
112
|
curl, Python's `requests` and `httpx`, pip, npm, git, apt and the Agent SDK
|
|
113
|
-
all read those variables.
|
|
114
|
-
|
|
113
|
+
all read those variables. A browser your own script launches does not:
|
|
114
|
+
Chromium ignores a password written in the proxy variable, so every page
|
|
115
|
+
fails `ERR_INVALID_AUTH_CREDENTIALS`. Hand Playwright the proxy yourself —
|
|
116
|
+
read `HTTPS_PROXY` with `new URL()` and pass `proxy: { server, username,
|
|
117
|
+
password }` to `launch()` (the `web` tool's browsers already do). The proxy
|
|
118
|
+
holds the step's list and checks every connection against it:
|
|
115
119
|
|
|
116
120
|
- a site **off the list** is refused — `403`, saying which site and what to
|
|
117
121
|
add — and the refusal is a line in the run trace;
|
package/docs/runs.md
CHANGED
|
@@ -266,6 +266,34 @@ Each agent also has an **Activity** page: every step it has ever run, across
|
|
|
266
266
|
flows and direct runs, with that step's own headline. It is where "how has
|
|
267
267
|
this agent been going" is answered without opening runs one at a time.
|
|
268
268
|
|
|
269
|
+
## Your inbox
|
|
270
|
+
|
|
271
|
+
**Inbox**, top right and under View in the account's sidebar, is what waits
|
|
272
|
+
on *you*, across every workspace you may see, in one list: gates to approve,
|
|
273
|
+
questions agents are asking, open [issues](#issues), [proposed
|
|
274
|
+
fixes](evals#proposed-fixes) to review, and your own runs that went wrong.
|
|
275
|
+
|
|
276
|
+
Three tabs that never mix:
|
|
277
|
+
|
|
278
|
+
| tab | what is in it |
|
|
279
|
+
|---|---|
|
|
280
|
+
| **For me** | assigned to you: you are named in the flow's `approvers:`, you own the flow or agent (`owner:`, down to the workspace's and the account's), or you asked for the proposal |
|
|
281
|
+
| **Started by me** | runs you started: one that failed or ended BLOCKED or BAD in the last week, a gate on one (someone else must decide it — your own approval is refused), a question one is asking |
|
|
282
|
+
| **Team** | everything else waiting in your workspaces, for whoever picks it up — a gate anyone may decide, an issue nobody owns |
|
|
283
|
+
|
|
284
|
+
Nothing is kept in it: each item is read from where it lives and leaves on
|
|
285
|
+
its own when it is done there — the gate decided, the issue resolved, the
|
|
286
|
+
fix applied. A gate you may not decide (`approvers:` names others) is not in
|
|
287
|
+
yours at all. What *is* kept is what you did to your own view: an item you
|
|
288
|
+
opened is read, and **Set aside** hides one from you — it is still waiting
|
|
289
|
+
for everyone else, and the gate is not decided. An item names its occurrence,
|
|
290
|
+
so an issue that happens again after you set it aside comes back. The count
|
|
291
|
+
on the top bar is what is unread in For me and Started by me.
|
|
292
|
+
|
|
293
|
+
`foldrun inbox` prints the same (`--team`, `--all`, `--kind approval`); a key
|
|
294
|
+
reads the inbox of the person who minted it, within the key's own workspaces.
|
|
295
|
+
The API is `GET /api/me/inbox`.
|
|
296
|
+
|
|
269
297
|
## Gates
|
|
270
298
|
|
|
271
299
|
A step marked `2!` or carrying `approve: true` / `ask: <question>` parks the
|
package/docs/self-hosting.md
CHANGED
|
@@ -1,20 +1,58 @@
|
|
|
1
1
|
# Running it yourself
|
|
2
2
|
|
|
3
|
-
There are
|
|
3
|
+
There are three places a workspace runs, and the same folder moves between
|
|
4
4
|
them unchanged.
|
|
5
5
|
|
|
6
6
|
| shape | for | what is yours |
|
|
7
7
|
|---|---|---|
|
|
8
8
|
| **your machine** | one person, no server — the foldrun CLI and runtime, free from npm | everything: the files, the runs, the model bill |
|
|
9
|
-
| **
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
9
|
+
| **your own server** | you or your organisation on one machine: the dashboard, schedules, sandboxes, gates, a vault and a team, with Docker Compose | everything, and the machine |
|
|
10
|
+
| **the hosted platform** | a team that would rather not run anything | your workspaces and their records; we run the machines |
|
|
11
|
+
|
|
12
|
+
foldrun is **source-available** under the Functional Source License 1.1
|
|
13
|
+
(FSL-1.1-ALv2): the format, the runner, the checks, the CLI, the engine and
|
|
14
|
+
the dashboard are free to use, change and run for yourself and inside your
|
|
15
|
+
own organisation, at any size. What needs a commercial licence is offering
|
|
16
|
+
foldrun to others as a product or service that competes with it. Each
|
|
17
|
+
release becomes Apache 2.0 two years after it ships. The licence and a
|
|
18
|
+
plain-English summary are in each repository (`LICENSE`, `LICENSING.md`);
|
|
19
|
+
for a commercial licence, write to hello@foldrun.io.
|
|
20
|
+
|
|
21
|
+
A self-hosted server has no billing: nothing is charged, and spend budgets
|
|
22
|
+
(`budget:`) still hold.
|
|
23
|
+
|
|
24
|
+
## Your own server
|
|
25
|
+
|
|
26
|
+
One machine with Docker. The platform runs in one container — the dashboard,
|
|
27
|
+
the API, the scheduler and the queue — and every agent step runs in a
|
|
28
|
+
throwaway container beside it, started through the Docker socket.
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
mkdir foldrun && cd foldrun
|
|
32
|
+
for r in foldrun-core foldrun-platform foldrun-web foldrun-docs; do
|
|
33
|
+
git clone https://github.com/foldrun-io/$r
|
|
34
|
+
done
|
|
35
|
+
cd foldrun-web/self-hosting
|
|
36
|
+
echo "FOLDRUN_SECRET_KEY=$(openssl rand -hex 32)" > .env
|
|
37
|
+
echo "FOLDRUN_PUBLIC_URL=http://<host>:3900" >> .env # the address people open
|
|
38
|
+
docker compose up -d --build
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Then open `http://<host>:3900/signup` (the first account needs no flag),
|
|
42
|
+
sign the CLI in to it, and deploy an account folder:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
npm install -g foldrun
|
|
46
|
+
foldrun login --url http://<host>:3900 # approve in the browser; or --token <a key from Settings>
|
|
47
|
+
foldrun deploy # from the account folder
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
`docker-compose.yml` beside the Dockerfile lists every setting with what it
|
|
51
|
+
does: the install's model key (or none, and each account brings its own),
|
|
52
|
+
open sign-up (off), retention. `FOLDRUN_PUBLIC_URL` is required: CLI sign-in,
|
|
53
|
+
approve links, password resets and shares are made with it, so it must be
|
|
54
|
+
the address people actually open. Back up the `foldrun-data` volume and `FOLDRUN_SECRET_KEY` and
|
|
55
|
+
you have backed the install up.
|
|
18
56
|
|
|
19
57
|
## Your machine
|
|
20
58
|
|
package/docs/tools.md
CHANGED
|
@@ -594,7 +594,7 @@ web:
|
|
|
594
594
|
| `parallel` | extract | full-page markdown; handles JavaScript pages and PDFs | up to 20 | `PARALLEL_API_KEY` |
|
|
595
595
|
| `zyte` | **unblocker** | the page rendered in a real browser behind a proxy pool; pays per success | 1 | `ZYTE_API_KEY_BASIC` |
|
|
596
596
|
| `scrapingbee` | **unblocker** | the page rendered in a real browser (`render_js`) behind a proxy pool; credits per call | 1 | `SCRAPINGBEE_API_KEY` |
|
|
597
|
-
| `anthropic` |
|
|
597
|
+
| `anthropic` | ours | accepted, and answered by foldrun's own fetch: Claude Code's `WebFetch` it used to hand over to fails its domain check behind the egress proxy | 1, or up to 20 with `urls=` | — |
|
|
598
598
|
|
|
599
599
|
A provider answers `markdown`, `text` and `html`. The modes only a raw read
|
|
600
600
|
can answer — `meta`, `links`, `raw` — are always read directly, and the tool
|
package/docs/versions.md
CHANGED
|
@@ -119,8 +119,8 @@ whatever `FOLDRUN_VERSION` it was started with.
|
|
|
119
119
|
## The API version
|
|
120
120
|
|
|
121
121
|
`api` in `/api/version`, and the `X-Foldrun-Api` header, is a date —
|
|
122
|
-
`2026-10-01` — that changes only when a route changes
|
|
123
|
-
caller would break on. A new route or a new field in an answer does not move
|
|
122
|
+
`2026-10-26` now, after `2026-10-01` — that changes only when a route changes
|
|
123
|
+
in a way an existing caller would break on ([the list](api-usage#versions)). A new route or a new field in an answer does not move
|
|
124
124
|
it. Breaking changes are announced in the release notes of the version that
|
|
125
125
|
makes them.
|
|
126
126
|
|
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "foldrun",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.8.0",
|
|
4
4
|
"description": "Agents are just folders. Write them, check them, run them — on your machine.",
|
|
5
|
-
"license": "
|
|
5
|
+
"license": "FSL-1.1-ALv2",
|
|
6
6
|
"type": "module",
|
|
7
7
|
"bin": {
|
|
8
8
|
"foldrun": "bin/foldrun.mjs"
|
|
@@ -11,17 +11,18 @@
|
|
|
11
11
|
"node": ">=22"
|
|
12
12
|
},
|
|
13
13
|
"dependencies": {
|
|
14
|
-
"@foldrun/core": "^2.
|
|
14
|
+
"@foldrun/core": "^2.12.0"
|
|
15
15
|
},
|
|
16
16
|
"files": [
|
|
17
17
|
"bin",
|
|
18
18
|
"src",
|
|
19
19
|
"README.md",
|
|
20
|
+
"LICENSING.md",
|
|
20
21
|
"docs"
|
|
21
22
|
],
|
|
22
23
|
"scripts": {
|
|
23
24
|
"e2e": "FOLDRUN_E2E=1 FOLDRUN_DATA=$(mktemp -d /tmp/foldrun-e2e-XXXXXX) node --test --test-timeout=300000 tests/e2e.test.ts",
|
|
24
|
-
"test": "FOLDRUN_DATA=$(mktemp -d /tmp/foldrun-test-XXXXXX) node --test tests/cli-account.test.ts tests/model-credential.test.ts tests/cli-approvals.test.ts tests/cli-issues.test.ts tests/cli-capture.test.ts tests/cli-stop.test.ts tests/cli-account-settings.test.ts tests/cli-schedule.test.ts tests/cli-billing.test.ts tests/cli-storage.test.ts tests/cli-scaffold.test.ts tests/cli-secrets.test.ts tests/cli-site-login.test.ts tests/cli-check-library.test.ts tests/cli-check-timezone.test.ts tests/cli-check-frontmatter.test.ts tests/cli-agent-import.test.ts tests/cli-check-subagents.test.ts tests/cli-connect.test.ts tests/cli-accounts.test.ts tests/cli-remote.test.ts tests/cli-stream.test.ts tests/cli-doctor.test.ts tests/cli-tool-test.test.ts tests/cli-agent-run.test.ts tests/cli-check-remote.test.ts tests/cli-shares.test.ts tests/cli-triggers.test.ts tests/cli-providers.test.ts tests/cli-rerun-put.test.ts tests/cli-guide.test.ts tests/cli-runtimes.test.ts tests/cli-deploy-safety.test.ts tests/cli-operator.test.ts tests/cli-flow-patterns.test.ts tests/cli-flow-rm-step.test.ts tests/cli-flow-dup-step.test.ts tests/cli-flow-write-stale.test.ts tests/cli-agent-link-stale.test.ts tests/cli-agent-link-isolated.test.ts tests/cli-parity.test.ts tests/cli-invoke-inputs.test.ts tests/cli-audit-keys.test.ts tests/cli-version.test.ts tests/cli-check-web-warnings.test.ts tests/cli-report-image.test.ts tests/cli-report-cache.test.ts tests/cli-check-workspace-paths.test.ts tests/cli-parity-dashboard.test.ts tests/cli-api-standards.test.ts tests/cli-backups-onboarding.test.ts tests/cli-check-limits.test.ts tests/cli-gallery.test.ts tests/cli-status-platform.test.ts tests/cli-webhooks-notifications.test.ts tests/cli-flags.test.ts tests/cli-api-coverage.test.ts tests/cli-hooks.test.ts tests/cli-release.test.ts tests/cli-terminal.test.ts tests/cli-network.test.ts tests/cli-assistant.test.ts tests/cli-check-scan.test.ts tests/cli-published.test.ts tests/cli-init-target.test.ts",
|
|
25
|
+
"test": "FOLDRUN_DATA=$(mktemp -d /tmp/foldrun-test-XXXXXX) node --test tests/cli-account.test.ts tests/model-credential.test.ts tests/cli-approvals.test.ts tests/cli-issues.test.ts tests/cli-capture.test.ts tests/cli-stop.test.ts tests/cli-account-settings.test.ts tests/cli-schedule.test.ts tests/cli-billing.test.ts tests/cli-storage.test.ts tests/cli-scaffold.test.ts tests/cli-secrets.test.ts tests/cli-site-login.test.ts tests/cli-check-library.test.ts tests/cli-check-timezone.test.ts tests/cli-check-frontmatter.test.ts tests/cli-agent-import.test.ts tests/cli-check-subagents.test.ts tests/cli-connect.test.ts tests/cli-accounts.test.ts tests/cli-remote.test.ts tests/cli-stream.test.ts tests/cli-doctor.test.ts tests/cli-tool-test.test.ts tests/cli-agent-run.test.ts tests/cli-check-remote.test.ts tests/cli-shares.test.ts tests/cli-triggers.test.ts tests/cli-providers.test.ts tests/cli-rerun-put.test.ts tests/cli-guide.test.ts tests/cli-runtimes.test.ts tests/cli-deploy-safety.test.ts tests/cli-operator.test.ts tests/cli-flow-patterns.test.ts tests/cli-flow-rm-step.test.ts tests/cli-flow-dup-step.test.ts tests/cli-flow-write-stale.test.ts tests/cli-agent-link-stale.test.ts tests/cli-agent-link-isolated.test.ts tests/cli-parity.test.ts tests/cli-invoke-inputs.test.ts tests/cli-audit-keys.test.ts tests/cli-version.test.ts tests/cli-check-web-warnings.test.ts tests/cli-report-image.test.ts tests/cli-report-cache.test.ts tests/cli-check-workspace-paths.test.ts tests/cli-parity-dashboard.test.ts tests/cli-api-standards.test.ts tests/cli-backups-onboarding.test.ts tests/cli-check-limits.test.ts tests/cli-gallery.test.ts tests/cli-status-platform.test.ts tests/cli-webhooks-notifications.test.ts tests/cli-flags.test.ts tests/cli-api-coverage.test.ts tests/cli-hooks.test.ts tests/cli-release.test.ts tests/cli-terminal.test.ts tests/cli-network.test.ts tests/cli-assistant.test.ts tests/cli-check-scan.test.ts tests/cli-published.test.ts tests/cli-init-target.test.ts tests/cli-flow-filter.test.ts tests/cli-eval-grades.test.ts tests/cli-eval-compare.test.ts tests/cli-proposals.test.ts tests/cli-inbox.test.ts",
|
|
25
26
|
"release": "node scripts/release.mjs",
|
|
26
27
|
"typecheck": "tsc -p tsconfig.json",
|
|
27
28
|
"docs:sync": "node scripts/sync-docs.mjs",
|
|
@@ -47,5 +48,8 @@
|
|
|
47
48
|
"@types/node": "^26.6.4",
|
|
48
49
|
"gray-matter": "^4.0.3",
|
|
49
50
|
"typescript": "^6.0.3"
|
|
51
|
+
},
|
|
52
|
+
"overrides": {
|
|
53
|
+
"zod": "4.6.7"
|
|
50
54
|
}
|
|
51
55
|
}
|