samagotchi 0.78.0 → 0.80.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +44 -1
- data/README.md +7 -5
- data/docs/cli.md +24 -7
- data/docs/configuration.md +11 -6
- data/docs/internals/anthropic.md +12 -5
- data/docs/internals/bundles.md +4 -0
- data/docs/internals/engine.md +15 -5
- data/docs/plugins.md +144 -3
- data/docs/releasing.md +2 -2
- data/docs/sessions.md +1 -1
- data/docs/testing.md +32 -2
- data/lib/samagotchi/bootstrap/config_writer.rb +13 -3
- data/lib/samagotchi/bootstrap/probe.rb +153 -31
- data/lib/samagotchi/bootstrap_command.rb +92 -15
- data/lib/samagotchi/bridge.rb +1 -1
- data/lib/samagotchi/bundles/system/manifest.yml +2 -2
- data/lib/samagotchi/bundles/system/self_map.md +1 -1
- data/lib/samagotchi/continue_offer.rb +5 -5
- data/lib/samagotchi/engine.rb +40 -37
- data/lib/samagotchi/guardrails/approval.rb +6 -4
- data/lib/samagotchi/guardrails/gate.rb +7 -3
- data/lib/samagotchi/hooks/bundle_loader.rb +31 -16
- data/lib/samagotchi/hooks/registry.rb +44 -3
- data/lib/samagotchi/host_registry.rb +12 -4
- data/lib/samagotchi/llm/adapter_common.rb +8 -4
- data/lib/samagotchi/llm/anthropic_binding_recovery.rb +198 -0
- data/lib/samagotchi/llm/anthropic_messages.rb +39 -178
- data/lib/samagotchi/llm/anthropic_wire.rb +7 -1
- data/lib/samagotchi/llm/chat_loop.rb +21 -12
- data/lib/samagotchi/llm/http.rb +12 -7
- data/lib/samagotchi/llm/model_listing.rb +29 -1
- data/lib/samagotchi/llm/refusal_memo.rb +3 -2
- data/lib/samagotchi/plugin/card.rb +44 -0
- data/lib/samagotchi/plugin/context.rb +53 -8
- data/lib/samagotchi/plugin/http.rb +154 -0
- data/lib/samagotchi/plugin/http_transport.rb +77 -0
- data/lib/samagotchi/plugin/loader.rb +29 -10
- data/lib/samagotchi/question_answer.rb +29 -0
- data/lib/samagotchi/question_desk.rb +30 -22
- data/lib/samagotchi/reasoning_history.rb +2 -1
- data/lib/samagotchi/relay_verifier.rb +1 -1
- data/lib/samagotchi/testing/boundary.rb +156 -0
- data/lib/samagotchi/testing/bundle.rb +349 -0
- data/lib/samagotchi/testing/fake_executable.rb +42 -0
- data/lib/samagotchi/testing/fake_http.rb +137 -0
- data/lib/samagotchi/testing/helpers.rb +56 -0
- data/lib/samagotchi/testing/host.rb +76 -0
- data/lib/samagotchi/testing/minitest.rb +31 -0
- data/lib/samagotchi/testing/recorder.rb +138 -0
- data/lib/samagotchi/testing/recording_api.rb +27 -0
- data/lib/samagotchi/testing/rspec.rb +31 -0
- data/lib/samagotchi/testing/sandbox.rb +163 -0
- data/lib/samagotchi/testing/sessions.rb +77 -0
- data/lib/samagotchi/testing/verdict.rb +28 -0
- data/lib/samagotchi/testing/waiting.rb +36 -0
- data/lib/samagotchi/testing.rb +26 -0
- data/lib/samagotchi/tools/delegate_relay.rb +7 -6
- data/lib/samagotchi/tools/tool_index.rb +3 -0
- data/lib/samagotchi/turn_state.rb +28 -1
- data/lib/samagotchi/version.rb +1 -1
- data/lib/samagotchi/web/public/app.js +79 -245
- data/lib/samagotchi/web/public/composer_grip.js +231 -0
- data/lib/samagotchi/web/public/images.js +33 -0
- data/lib/samagotchi/web/public/index.html +73 -40
- data/lib/samagotchi/web/public/level_chip.js +11 -11
- data/lib/samagotchi/web/public/lists_view.js +19 -14
- data/lib/samagotchi/web/public/llm_context_chip.js +15 -15
- data/lib/samagotchi/web/public/model_picker.js +5 -6
- data/lib/samagotchi/web/public/scope.js +3 -3
- data/lib/samagotchi/web/public/stage_view.js +3 -2
- data/lib/samagotchi/web/public/storage.js +23 -0
- data/lib/samagotchi/web/public/strip.js +35 -9
- data/lib/samagotchi/web/public/strip_grip.js +76 -0
- data/lib/samagotchi/web/public/thinking_chip.js +5 -5
- data/lib/samagotchi/web/public/toasts.js +5 -4
- metadata +24 -1
data/docs/sessions.md
CHANGED
|
@@ -76,7 +76,7 @@ chi sessions clean --dry-run --days 7 # test sessions older than 7 da
|
|
|
76
76
|
**Ordering:**
|
|
77
77
|
|
|
78
78
|
- `Session.list` / `SessionManager.list_sessions` / `GET /api/sessions?sort=&order=&limit=&offset=` default to `updated_at desc` (newest activity first). Also supports `created_at`, `asc`. `X-Total-Count` header when paginated.
|
|
79
|
-
- Web UI (`chi web`): the page's scope is in its URL. Started in a git repo, `chi web` opens `/?dir=<that folder>`: that project's sessions, and new chats start in that folder; the header chip says `<project> · all`, and `all` opens the same place without `?dir` (every session; a new chat there starts in the server's own folder, shown on the start page, and cards name their folder; `← <project>` goes back to the project view it came from, or to the server's own project). One server serves every project: a second `chi web` (from another repo) finds it through `GET /api/info` and prints (with `--open`, opens) its page for its own folder instead of starting another. The 3 latest sessions sit above the chat (`hide ▴` puts them away behind a `▾ N sessions` pill, remembered; in a window under 900 px wide, where the strip scrolls sideways, `▴ hide` sits in the top bar where the pill comes back; a window under 700 px tall hides them by itself, so a question or approval card in the stage keeps room, and the pill shows them until the window is tall again). A parent and its delegates are one card there: its `▸ 2 delegates · 1 live · 1 waiting` chip (the waiting part in the attention colour, the card's border too) opens their rows in a popover under it (a row per delegate at any depth, indented; a click opens that one, Esc or a click outside closes it, and the strip's order holds while it is open). A fork is a card of its own, and so is a delegate whose parent isn't listed (archived, another project), with its `↳ <parent>` chip; the pill counts cards, and every live or waiting session. "All sessions" (or `/`) opens every session at `#/sessions`, with a search over preview, id and status (Esc or Back returns). There a parent's card holds its delegates the same way, the chip opening their rows inside the card (each with an archive button on hover), open by itself while one of them waits on you; the count says `14 (+7 delegates)`, a search that matches a delegate shows its parent's card with the list open and the matching rows marked, and select mode picks cards (archiving a parent takes its delegates). The open session is in the URL (`#/s/<id>`), so a reload or a copied link opens it again; the chi logo top left goes back to the empty start for a new chat. The message box grows with its text; drag its top edge to keep it taller (double-click resets). A file in a tool row (a read, edit or write's title) and its diff lines are refs: hover one and click to open the current file in your editor at that line, on this machine only (`web.editor`; the browser's "always allow" for the page's port and VS Code's "allow opening local files" are asked once; see [Opening files from chi web](configuration.md#opening-files-from-chi-web)). So are an edit or write approval card's paths (at the first line the dry run changes) and its diff lines, before you approve, and the numbered lines of a read's output: `lines 40–60` under it opens the text as read, numbered, when the output is the file as-is (not the head/tail preview of a big file, an error or an image; no numbers after a cut). A `path:line` (or `path:line-line`) in an answer or a step's text is one too once the turn has ended, and after a reload, when it names a file a row or card on the page touched: the path under the session's folder, else the one touched file it ends with (a bare `app.js:12` when only one touched file is named so; two such files make it none); a path without a line counts only when it is a touched file exactly. Code blocks, links, bare URLs and inline code with more than the ref in it are left alone, and a file the session didn't touch is never linked; a link whose text is such a ref alone (github-pr's link to the PR diff) becomes a ref too, and its plain click still goes to the PR. A plugin's refs on an answer (an `after_turn` hook's, see [Refs on the answer](hooks.md#refs-on-the-answer-display-only)) are paired with its links and text whatever the rows touched: they can link a file the session didn't touch, or an issue or a commit, and add their own actions to the ref's menu. A plugin tool's row carries the refs its `refs:` gives (see [Refs on a tool's row](plugins.md#refs-on-a-tools-row)): its title opens the first, and its `▾` menu adds the others as `<ref> · <action>` (the URL on hover); their URL actions work on a phone too. A ref with more than one target shows a small `▾` on hover: it opens a menu of them, in order **PR diff** (a link's), **Open in editor**, **Show in turn** and **Copy path**, and a plain click runs the first (a link's own, else the editor, else show in turn). **Show in turn** scrolls to the tool row that touched the file (looked for in the ref's own step, then its own turn, then the rows before it, then the nearest after it; there, the last one showing the ref's line, else the last one), opens the steps, the stage's steps and the row's diff (or the read showing the line) and flashes the row and the line; an answer's or a step's ref and an approval card's path have it, not a row's own title or lines. It needs no editor, so a viewer without one (a phone on the LAN) gets an answer's refs with **Show in turn** as their click; a row's title and diff lines stay plain text there. On a touch screen a tap opens the menu. A ref with a target is a tab stop: Enter runs its first target, ↓ opens the menu (↑/↓ move, Enter picks, Esc closes back to the ref). Diff and read lines have their click only. The info bar copies `chi --attach <id>` for a terminal. A session whose prompt carried model notes has a `notes: deepseek` chip there (the names without `model_notes_`; the tooltip lists each with its scope and size; `session.prompt_notes` in `GET /api/sessions/:id`, the live worker's, else the session file's; see [Memory](memory.md#model-notes)). Its `llm ctx` chip names the LLM context strategy the next turn runs under (`llm ctx stale,forget · session`, in the accent colour when the session set its own; the tooltip gives each value's source), and its popover sets the session's own strategy, apply rule and budget, or follows the model again, by running `/llm-context` in the worker (the reply shows as a command bubble); `GET /api/sessions/:id`'s `session.llm_context` carries the values (the live worker's, else resolved from the session file), and a `command_ran` that changed them carries the new ones. Its `think` chip does the same for the thinking level (`/thinking`), and on a chat host its `reasoning` chip for the reasoning history (`reasoning all · session`, `/reasoning-history`; `session.reasoning_history`, null on a native host). A `/thinking` or `/reasoning-history` change that restarts the prompt cache asks first: the reply bubble has an **Apply (restarts the cache)** button that sends its `--yes` line (the chips' Set sends that line itself; see [Changes that restart the prompt cache](configuration.md#changes-that-restart-the-prompt-cache)). It shows the context used (`ctx 42%`) and the last generation's decode speed (`87 tok/s`, `~64 tok/s` when estimated; see `/stats` in [cli.md](cli.md#session-commands)); hovering `ctx` there or on a session card lists the session's tokens in (summed over every request, with the cached share), out (with reasoning) and the cost when the provider reports one, for this session only (one estimated from a `hosts.<name>.models` price shows with `~`: `cost: ~$0.54 ($0.42 reported, ~$0.12 estimated)`); after a `/model` switch the info bar's tooltip adds a `per model:` row per run of turns on one model above those totals (`splash:qwen3-35b · 6 turns · 96.0k in / 9.0k out · ~$0.02`, a switch's prefill under its row: `└ switch prefill 58.0k uncached prompt ≈$0.01`; the timing's `model_segments`, replaced by each `generation_completed`'s; the session cards stay as they were), and right after the switch its ctx % is the new model's window's; and the memory indexes its prompt holds (`memory index: ~3.4k tokens in this session's prompt (system 2.0k, project 1.4k)`; see [Memory](memory.md#the-indexs-size)). The start page's model picker (bottom left of the composer, the chosen name `▾`) chooses the model a new chat starts on, and a fresh chain link's (see [Session chains](#session-chains); there a pick shows as a `/model` command bubble in the link); it opens a search list over the conversation (↓ or a click; downward when the start page leaves no room above): an empty search shows the last 5 picks (Recent) and then each host's model ids A–Z under the host, the default host first (an id the host declares under `hosts.<name>.models` first, noted `config`, its tooltip `served by config (hosts.<name>.models)`; a host that failed to list its models keeps those ids, greyed and noted `host down`, still pickable; a remembered or default model stays chosen even then, greyed on the button); typed words must each match the host or the id (a substring, else for 3+ characters the letters in order, so `deepseek4.1 fla` finds `openrouter · deepseek/deepseek-v4.1-flash`), ⏎ picks, Esc/Tab close, a pick puts the focus in the message box: `GET /api/models` lists the hosts' models as chi spells them (`{default, models: [{name, host, id, configured?, unavailable?, sampling?, llm_context?}], warning?}`; `configured: true` marks an id declared under `hosts.<name>.models`, listed first under its host, `unavailable: true` one whose host failed to list its models; `sampling` is the model's configured request parameters as `/model` words them, the row's tooltip in the list; bare for the default host, `host:model` for the others, from the same cached lists as `/models`, a bounded wait, a host that is down noted in `warning`), and `POST /api/sessions` takes `model` (blank means the default). Beside it, an `llm ctx` chip shows the LLM context the picked model starts under (each `GET /api/models` row's `llm_context`, as the info bar's); its form sets one for the new chat only, sent as `POST /api/sessions`'s `llm_context` and never remembered (see [configuration.md](configuration.md#llm-context-a-sessions-own-strategy)). The start page's first message goes as later ones do: `POST /api/sessions` with `idle: true` and `preview` (the message, which names the session until its turn is saved), then `POST /api/sessions/:id/turn`, so a failed first turn puts its prompt back in the composer too; a first line that is one of the new session's commands (`/model …`) goes to `POST /api/sessions/:id/command` instead, as later ones do. The browser remembers the last model picked; a running session's model is in the info bar and changes only with `/model`. In the composer a line is a command when it is `!…` or starts with one of the session's command names (its `commands` list; `/stats`, `/exit` and the other terminal words get the page's own reply); an unknown `/word` (`/usr/bin/env is missing`) goes to the model as a prompt, as in a terminal; a one-word `/word` no command answers (`/modle`) is not sent: the page shows `Unknown command /modle. Did you mean /model? /help lists the commands.` (the "Did you mean" part only when a name is close). A `POST /turn` (or a session's first `prompt`) that is one of the worker's commands runs as that command too (`202 {status, command_id, session_id}`), so `chi send`, `chi -p` and a page whose command list lags all get the command. ↑/↓ in the composer walk the prompt history the TUI's ↑ walks ([Persistent Prompt History](cli.md#persistent-prompt-history)): ↑ with the caret on the first line, ↓ from the last line while browsing, ↓ past the newest brings the draft back; `GET /api/history` returns `{entries: [...]}`, oldest first (the page fetches it on focus and after each send). The server records a sent line in the routes once the worker took it (`POST /turn` accepted, a `!command` through `POST /command` or as a session's first line); `"history": false` in the body keeps it out (an image-only send).
|
|
79
|
+
- Web UI (`chi web`): the page's scope is in its URL. Started in a git repo, `chi web` opens `/?dir=<that folder>`: that project's sessions, and new sessions start in that folder; the header chip says `<project> · all`, and `all` opens the same place without `?dir` (every session; a new session there starts in the server's own folder, shown on the start page, and cards name their folder; `← <project>` goes back to the project view it came from, or to the server's own project). One server serves every project: a second `chi web` (from another repo) finds it through `GET /api/info` and prints (with `--open`, opens) its page for its own folder instead of starting another. The 3 latest sessions sit above the chat, and the grip on the chat's top edge — the same thin handle as the composer's, its twin — puts them away and brings them back: a drag up (or a click, or Enter/Space on it) hides them, a drag down (or a click) shows them again, and the choice is remembered. A hidden strip leaves its status in the top bar (`N sessions · ● 1 live · ● 1 waiting`, from the cards), and a click on that pill opens All sessions, without showing the strip first. In a window under 900 px wide, where the strip scrolls sideways, the grip stays under it, outside the scroller; a window under 700 px tall hides the strip by itself, so a question or approval card in the stage keeps room, and the grip shows it until the window is tall again. A parent and its delegates are one card there: its `▸ 2 delegates · 1 live · 1 waiting` chip (the waiting part in the attention colour, the card's border too) opens their rows in a popover under it (a row per delegate at any depth, indented; a click opens that one, Esc or a click outside closes it, and the strip's order holds while it is open). A fork is a card of its own, and so is a delegate whose parent isn't listed (archived, another project), with its `↳ <parent>` chip; the pill counts cards, and every live or waiting session. "All sessions" (or `/`) opens every session at `#/sessions`, with a search over preview, id and status (Esc or Back returns). There a parent's card holds its delegates the same way, the chip opening their rows inside the card (each with an archive button on hover), open by itself while one of them waits on you; the count says `14 (+7 delegates)`, a search that matches a delegate shows its parent's card with the list open and the matching rows marked, and select mode picks cards (archiving a parent takes its delegates). The open session is in the URL (`#/s/<id>`), so a reload or a copied link opens it again; the chi logo top left goes back to the empty start for a new session. The message box grows with its text; drag its top edge to keep it taller (double-click resets). A file in a tool row (a read, edit or write's title) and its diff lines are refs: hover one and click to open the current file in your editor at that line, on this machine only (`web.editor`; the browser's "always allow" for the page's port and VS Code's "allow opening local files" are asked once; see [Opening files from chi web](configuration.md#opening-files-from-chi-web)). So are an edit or write approval card's paths (at the first line the dry run changes) and its diff lines, before you approve, and the numbered lines of a read's output: `lines 40–60` under it opens the text as read, numbered, when the output is the file as-is (not the head/tail preview of a big file, an error or an image; no numbers after a cut). A `path:line` (or `path:line-line`) in an answer or a step's text is one too once the turn has ended, and after a reload, when it names a file a row or card on the page touched: the path under the session's folder, else the one touched file it ends with (a bare `app.js:12` when only one touched file is named so; two such files make it none); a path without a line counts only when it is a touched file exactly. Code blocks, links, bare URLs and inline code with more than the ref in it are left alone, and a file the session didn't touch is never linked; a link whose text is such a ref alone (github-pr's link to the PR diff) becomes a ref too, and its plain click still goes to the PR. A plugin's refs on an answer (an `after_turn` hook's, see [Refs on the answer](hooks.md#refs-on-the-answer-display-only)) are paired with its links and text whatever the rows touched: they can link a file the session didn't touch, or an issue or a commit, and add their own actions to the ref's menu. A plugin tool's row carries the refs its `refs:` gives (see [Refs on a tool's row](plugins.md#refs-on-a-tools-row)): its title opens the first, and its `▾` menu adds the others as `<ref> · <action>` (the URL on hover); their URL actions work on a phone too. A ref with more than one target shows a small `▾` on hover: it opens a menu of them, in order **PR diff** (a link's), **Open in editor**, **Show in turn** and **Copy path**, and a plain click runs the first (a link's own, else the editor, else show in turn). **Show in turn** scrolls to the tool row that touched the file (looked for in the ref's own step, then its own turn, then the rows before it, then the nearest after it; there, the last one showing the ref's line, else the last one), opens the steps, the stage's steps and the row's diff (or the read showing the line) and flashes the row and the line; an answer's or a step's ref and an approval card's path have it, not a row's own title or lines. It needs no editor, so a viewer without one (a phone on the LAN) gets an answer's refs with **Show in turn** as their click; a row's title and diff lines stay plain text there. On a touch screen a tap opens the menu. A ref with a target is a tab stop: Enter runs its first target, ↓ opens the menu (↑/↓ move, Enter picks, Esc closes back to the ref). Diff and read lines have their click only. The info bar copies `chi --attach <id>` for a terminal. A session whose prompt carried model notes has a `notes: deepseek` chip there (the names without `model_notes_`; the tooltip lists each with its scope and size; `session.prompt_notes` in `GET /api/sessions/:id`, the live worker's, else the session file's; see [Memory](memory.md#model-notes)). Its `llm ctx` chip names the LLM context strategy the next turn runs under (`llm ctx stale,forget · session`, in the accent colour when the session set its own; the tooltip gives each value's source), and its popover sets the session's own strategy, apply rule and budget, or follows the model again, by running `/llm-context` in the worker (the reply shows as a command bubble); `GET /api/sessions/:id`'s `session.llm_context` carries the values (the live worker's, else resolved from the session file), and a `command_ran` that changed them carries the new ones. Its `think` chip does the same for the thinking level (`/thinking`), and on a chat host its `reasoning` chip for the reasoning history (`reasoning all · session`, `/reasoning-history`; `session.reasoning_history`, null on a native host). A `/thinking` or `/reasoning-history` change that restarts the prompt cache asks first: the reply bubble has an **Apply (restarts the cache)** button that sends its `--yes` line (the chips' Set sends that line itself; see [Changes that restart the prompt cache](configuration.md#changes-that-restart-the-prompt-cache)). It shows the context used (`ctx 42%`) and the last generation's decode speed (`87 tok/s`, `~64 tok/s` when estimated; see `/stats` in [cli.md](cli.md#session-commands)); hovering `ctx` there or on a session card lists the session's tokens in (summed over every request, with the cached share), out (with reasoning) and the cost when the provider reports one, for this session only (one estimated from a `hosts.<name>.models` price shows with `~`: `cost: ~$0.54 ($0.42 reported, ~$0.12 estimated)`); after a `/model` switch the info bar's tooltip adds a `per model:` row per run of turns on one model above those totals (`splash:qwen3-35b · 6 turns · 96.0k in / 9.0k out · ~$0.02`, a switch's prefill under its row: `└ switch prefill 58.0k uncached prompt ≈$0.01`; the timing's `model_segments`, replaced by each `generation_completed`'s; the session cards stay as they were), and right after the switch its ctx % is the new model's window's; and the memory indexes its prompt holds (`memory index: ~3.4k tokens in this session's prompt (system 2.0k, project 1.4k)`; see [Memory](memory.md#the-indexs-size)). The start page's model picker (bottom left of the composer, the chosen name `▾`) chooses the model a new session starts on, and a fresh chain link's (see [Session chains](#session-chains); there a pick shows as a `/model` command bubble in the link); it opens a search list over the conversation (↓ or a click; downward when the start page leaves no room above): an empty search shows the last 5 picks (Recent) and then each host's model ids A–Z under the host, the default host first (an id the host declares under `hosts.<name>.models` first, noted `config`, its tooltip `served by config (hosts.<name>.models)`; a host that failed to list its models keeps those ids, greyed and noted `host down`, still pickable; a remembered or default model stays chosen even then, greyed on the button); typed words must each match the host or the id (a substring, else for 3+ characters the letters in order, so `deepseek4.1 fla` finds `openrouter · deepseek/deepseek-v4.1-flash`), ⏎ picks, Esc/Tab close, a pick puts the focus in the message box: `GET /api/models` lists the hosts' models as chi spells them (`{default, models: [{name, host, id, configured?, unavailable?, sampling?, llm_context?}], warning?}`; `configured: true` marks an id declared under `hosts.<name>.models`, listed first under its host, `unavailable: true` one whose host failed to list its models; `sampling` is the model's configured request parameters as `/model` words them, the row's tooltip in the list; bare for the default host, `host:model` for the others, from the same cached lists as `/models`, a bounded wait, a host that is down noted in `warning`), and `POST /api/sessions` takes `model` (blank means the default). Beside it, an `llm ctx` chip shows the LLM context the picked model starts under (each `GET /api/models` row's `llm_context`, as the info bar's); its form sets one for the new session only, sent as `POST /api/sessions`'s `llm_context` and never remembered (see [configuration.md](configuration.md#llm-context-a-sessions-own-strategy)). The start page's first message goes as later ones do: `POST /api/sessions` with `idle: true` and `preview` (the message, which names the session until its turn is saved), then `POST /api/sessions/:id/turn`, so a failed first turn puts its prompt back in the composer too; a first line that is one of the new session's commands (`/model …`) goes to `POST /api/sessions/:id/command` instead, as later ones do. The browser remembers the last model picked; a running session's model is in the info bar and changes only with `/model`. In the composer a line is a command when it is `!…` or starts with one of the session's command names (its `commands` list; `/stats`, `/exit` and the other terminal words get the page's own reply); an unknown `/word` (`/usr/bin/env is missing`) goes to the model as a prompt, as in a terminal; a one-word `/word` no command answers (`/modle`) is not sent: the page shows `Unknown command /modle. Did you mean /model? /help lists the commands.` (the "Did you mean" part only when a name is close). A `POST /turn` (or a session's first `prompt`) that is one of the worker's commands runs as that command too (`202 {status, command_id, session_id}`), so `chi send`, `chi -p` and a page whose command list lags all get the command. ↑/↓ in the composer walk the prompt history the TUI's ↑ walks ([Persistent Prompt History](cli.md#persistent-prompt-history)): ↑ with the caret on the first line, ↓ from the last line while browsing, ↓ past the newest brings the draft back; `GET /api/history` returns `{entries: [...]}`, oldest first (the page fetches it on focus and after each send). The server records a sent line in the routes once the worker took it (`POST /turn` accepted, a `!command` through `POST /command` or as a session's first line); `"history": false` in the body keeps it out (an image-only send).
|
|
80
80
|
- Web notifications: when a session needs you while the tab is hidden or behind another window (a question or a guardrail approval, a plugin card with buttons during a turn such as check-in's, a failed turn, a turn done after 10 s or more; a delegated session only for its questions, a canceled turn never), the page title counts it, `(N) Chi`, until you come back to the tab. The bell at the right of the top bar also turns on OS notifications ("needs an answer", "needs approval", "needs you", "turn failed", "done in 42 s", under the session's first message; a click opens the session): its first click asks the browser for the permission, and the choice is kept per browser. Any session of the page's scope counts, not only the open one. What was already so when the page loaded never counts, and several chi tabs show one notification per event. A chi tab in front tells the others (a `BroadcastChannel`), so while you look at one, the ones behind neither notify nor count what it shows; coming back to a tab clears those events in the other tabs' titles too. The hub's `session` frames carry what this needs: `pending_question` (`{id, kind}`, kind `question` or `approval`), `pending_card` (`{id, bundle}`: the running turn's card with actions, from `pending_card.json` that the worker keeps in the session's folder while one is open; a card without actions, such as loop-guard's, never counts) and `last_turn` (`{outcome, ended_at, seconds, origin}`, saved in the session's JSON as the engine ends each turn).
|
|
81
81
|
- The web frontend is a zero-build ES-module stack in `lib/samagotchi/web/public/`: `data.js` (retrieval, typed SSE `openStream` for a session and `openEvents` for the session list), `sessions_list.js` (the list as a pure reducer over `GET /api/events`: a snapshot replaces it, an upsert keeps a known card in place, a removal drops it; `families()` groups delegates under their parent), `families_view.js` (a parent's family chip, its rows and the strip's popover; the chip and rows as lit-html templates), `keep_focus.js` (lit reuses a list's nodes by position: a redraw keeps the focus on the same item by its `data-id` / `data-name`), `chains.js` (session chains: the links from `continues`, the cards folded into the latest link's, its `#N` chip and rows as lit-html templates, the info bar's ← → links, a refused continue, and the fresh-link rule), `notify.js` (which of those events needs the user, for the notifications), `app.js` (presentation/state; a session with no live stream gets one when its `session` event says `bridge_up`, after one re-read: `event_seq` starts over in each worker, so a dropped stream is never resumed with its old cursor), `bubbles.js` (the history's bubbles between the turns' rows: output chunks, end lines, every client's prompt bubbles by `enqueued_id` with their queued/steered badges, the idle recap, guardrail warnings, hooks' notices, plugins' steers, command and note bubbles; lit-html templates, put in the history with the sticky follow by `createBubbles`), `cards_place.js` (where plugin cards and a snapshot's notices go in the history: a card added or updated in place by its id, a running turn's into its step, a snapshot's each above the first turn after it, a finished turn's notice in its step's rows and its question resolved where it was asked; a card's action buttons), `format.js` (pure formatters such as `previewOf`), `plain_line.js` (markdown out of the plain one-line places: the stage headline, a closed step's title, an answered question card's kept-reply line), `prompt_history.js` (the composer's ↑/↓ over the shared prompt history), `commands.js` (session commands: which composer lines are commands, the page's own replies, a `command_ran`'s bubble, the page's command bubbles kept across a re-read), `composer.js` (where a composer line goes, `routeComposerLine`: /cut and /queue, the start page's own replies, a session command or `!cmd`, a typo hint, else a turn; and a session command sent from the composer or a chip), `delivery_line.js` (the composer's row above the text field in a session, always there so the composer keeps its height: the key hint while idle, and the delivery line while a turn runs: where a message typed now goes in, from the step's phase, clock and size, whether "cut in now" can land, and the queue link for a worker with the `delivery` feature, as a pure `deliveryLine`, its lit-html template and `createDeliveryControl`, which ticks it and keeps a choice armed from a link; see docs/cli.md, "Typing during a turn"), `restore.js` (a failed turn's prompt handed back into this tab's composer, before or after its `/turn` ack), `lists_view.js` (the session lists: the strip and its pill, all sessions with its search, count and select mode, their cards, the family and chain popovers; the cards, tile and pill as lit-html templates, drawn as new nodes each time, and the lists' orders as pure functions), `stream_handlers.js` (the open session's stream event handlers, by event type, over the page's views and hooks; a snapshot replays through them too), `stage_view.js` (the stage view: the running turn pinned above the composer, then handed off into the history; its pure rules and texts in `stage_model.js`, its live slots (headline, now row, trail, ticks) in `stage_slots.js`, its cap and held height in `stage_frame.js`, the in-use watch that times the hand-off in `stage_watch.js`), `info_bar.js` (the session bar's info line as a lit-html template over a plain model: id, status, model, chain links, chips, ctx, speed, clock, worker badge), `info_bar_model.js` (that model and the bar's buttons' states, built from the page's session state, its list and its in-flight continues and restarts as pure functions; `app.js`'s `updateInfoBar` applies them every second while a turn runs), `session_state.js` (what the page knows about the open session: its empty state, and `sessionStateFrom`, the fields a `GET /sessions/:id` payload sets), `refs.js` (refs: a kind and a target on the element, a plugin's ref data (label, http(s) URL actions) bound in a WeakMap by `bindRef`, the targets per kind, show in turn, one delegated set of click, key and hover listeners), `ref_menu.js` (a ref's `▾` and its menu), `ref_scan.js` (the `path:line` refs in an answer's or a step's text that name a file the session's rows touched), `answer_refs.js` (the refs a plugin attached to an answer, its `display_refs`: bound to the answer's bubble, paired with its links and text, bound with `bindRef`). Unit-tested via `npm test` (`spec/web/public/*.test.js`).
|
|
82
82
|
- Selecting a session in the web UI is read-only: `GET /api/sessions/:id` never spawns a worker (it reads a live worker's snapshot when one runs, else the session file). A prompt (`POST /turn`) or a command (`POST /command`) wakes the worker, and `/stream` briefly waits for a freshly-spawned bridge before answering. A caught-up SSE reconnect holds the stream open; `reset` markers are only sent for reconnects behind the ring window, or with a cursor from another worker (event ids are `<event_seq>-<epoch>`, one epoch per worker).
|
data/docs/testing.md
CHANGED
|
@@ -1,7 +1,9 @@
|
|
|
1
1
|
# Testing
|
|
2
2
|
|
|
3
3
|
```sh
|
|
4
|
-
bundle exec parallel_rspec -n 8 # Ruby specs, in parallel
|
|
4
|
+
bundle exec parallel_rspec -n 8 spec lib/samagotchi/bundles/*/spec # Ruby specs, both roots, in parallel
|
|
5
|
+
bundle exec rspec spec lib/samagotchi/bundles/*/spec # the same, plain
|
|
6
|
+
bundle exec rake bundles:specs # the shipped bundles' own specs, on the bundle test kit alone
|
|
5
7
|
npm test # web frontend specs
|
|
6
8
|
npm run e2e # web UI happy paths in Chromium, fake model (once: npx playwright install chromium)
|
|
7
9
|
npx playwright test spec/e2e/stage.spec.js # one e2e file (the root playwright.config.js is the suite's)
|
|
@@ -53,6 +55,34 @@ Every example also runs under a 120 s limit (`spec_helper`'s `around`; `integrat
|
|
|
53
55
|
`recap` examples are exempt), so a hang fails its example instead of the run. `SPEC_EXAMPLE_TIMEOUT=N` changes the
|
|
54
56
|
limit, `0` turns it off (a debugger session).
|
|
55
57
|
|
|
58
|
+
## Bundle specs: the test kit
|
|
59
|
+
|
|
60
|
+
A shipped bundle's plugin and hook files are tested through the bundle test kit (`samagotchi/testing`,
|
|
61
|
+
[plugins.md](plugins.md#testing-a-bundle-experimental)): `shipped_bundle("check-in", settings: …)` loads it through
|
|
62
|
+
chi's own loaders, registries, `ctx` and guardrail Gate, with the session, the hook helpers and `ctx.sessions`
|
|
63
|
+
recorded. No fake `chi` or `ctx`: what the kit can't express is a kit change (with a kit spec), not a local fake.
|
|
64
|
+
|
|
65
|
+
These specs live **next to the bundle**, in `lib/samagotchi/bundles/<name>/spec/` (their fixtures in its
|
|
66
|
+
`spec/fixtures/`), and require only `samagotchi/testing/rspec` (tagged `:bundle` by their path): no `spec_helper`, no
|
|
67
|
+
`spec/support`, as an external bundle's specs would run. `rake bundles:specs` runs them that way
|
|
68
|
+
(`rspec --options /dev/null -r samagotchi/testing/rspec`), in CI's bundles job; the full suite runs them too, under
|
|
69
|
+
`spec_helper`, given both roots (plain `rspec` runs `spec/` only: `.rspec` holds no `--pattern`, which would make
|
|
70
|
+
`rspec FILE` run everything). They are never shipped: the gem and an install leave `spec/` out, and the release
|
|
71
|
+
tools' version check ignores a bundle's `spec/` (a spec-only change needs no bump). A spec that runs real turns
|
|
72
|
+
(install → Engine → `run_turn`) stays in `spec/samagotchi/plugin/<name>_bundle_spec.rb`.
|
|
73
|
+
|
|
74
|
+
The kit's own specs, in `spec/samagotchi/testing/`, keep it and the bundles honest:
|
|
75
|
+
|
|
76
|
+
- `spec/samagotchi/testing/contract_spec.rb`: the kit sets every `Plugin::Host` and `Hooks::Runtime` member,
|
|
77
|
+
`Testing::Sessions` has `Plugin::Sessions`' methods and parameters, and each `ctx` helper lands in its recorder. A
|
|
78
|
+
member added to the Host fails it until the kit has it.
|
|
79
|
+
- `spec/samagotchi/testing/shipped_bundles_spec.rb`: each shipped bundle with a plugin or hooks registers the same
|
|
80
|
+
commands, tools, hooks, inits, toolkits and sources loaded by the kit as installed and loaded by chi.
|
|
81
|
+
- `spec/samagotchi/testing/boundary_spec.rb`, the boundary lint: each bundle's Ruby (plugin, hooks, scripts, specs)
|
|
82
|
+
reaches chi only through `Samagotchi::Testing::Boundary::ALLOWED` (the "Stable for bundles" list) and that
|
|
83
|
+
bundle's `Boundary::EXCEPTIONS` entry, each with why. A new internal reference fails it, and so does an exception
|
|
84
|
+
no longer used. Fix the reference (the kit or a stable API), or add the exception with its why.
|
|
85
|
+
|
|
56
86
|
## While you work: affected specs
|
|
57
87
|
|
|
58
88
|
```sh
|
|
@@ -73,7 +103,7 @@ The Ruby specs go to stdout; the node tests to run (`npm test` targets), "web ch
|
|
|
73
103
|
hint for a hub class (named in too many specs to list) go to stderr. Nothing printed means nothing matched.
|
|
74
104
|
|
|
75
105
|
It finds a change's own specs, not the regressions it causes elsewhere. So before committing, run the full suite
|
|
76
|
-
(`bundle exec parallel_rspec -n 8`), unless your task says the coordinator runs it; the coordinator runs it at merge
|
|
106
|
+
(`bundle exec parallel_rspec -n 8 spec lib/samagotchi/bundles/*/spec`), unless your task says the coordinator runs it; the coordinator runs it at merge
|
|
77
107
|
either way.
|
|
78
108
|
|
|
79
109
|
## Worktrees
|
|
@@ -6,6 +6,7 @@ require "fileutils"
|
|
|
6
6
|
require "ipaddr"
|
|
7
7
|
require_relative "../atomic_file"
|
|
8
8
|
require_relative "../config"
|
|
9
|
+
require_relative "../host_config"
|
|
9
10
|
require_relative "../yaml_lines"
|
|
10
11
|
require_relative "../redacted_inspect"
|
|
11
12
|
|
|
@@ -91,6 +92,8 @@ module Samagotchi
|
|
|
91
92
|
end
|
|
92
93
|
|
|
93
94
|
# The name of an existing entry that points at the same server, or nil.
|
|
95
|
+
# An api: anthropic entry and any other on one server are different
|
|
96
|
+
# hosts (Fireworks serves both protocols).
|
|
94
97
|
def duplicate_of(fields)
|
|
95
98
|
current_hosts.find { |_, existing| same_server?(existing, fields) }&.first
|
|
96
99
|
end
|
|
@@ -267,15 +270,22 @@ module Samagotchi
|
|
|
267
270
|
def same_server?(existing, fields)
|
|
268
271
|
a = location(existing)
|
|
269
272
|
b = location(fields)
|
|
270
|
-
a && b && a == b
|
|
273
|
+
a && b && a == b && anthropic?(existing) == anthropic?(fields)
|
|
271
274
|
end
|
|
272
275
|
|
|
273
|
-
|
|
276
|
+
def anthropic?(fields) = fields["api"].to_s.strip.downcase == "anthropic"
|
|
277
|
+
|
|
278
|
+
# [scheme-less host, port, path] of an entry. An api: anthropic url's
|
|
279
|
+
# path loses a trailing /v1 (the adapter adds it), and one with
|
|
280
|
+
# neither url: nor host: is Anthropic's own API (HostConfig's rule).
|
|
274
281
|
def location(fields)
|
|
275
282
|
url = fields["url"].to_s.strip
|
|
283
|
+
url = ANTHROPIC_URL if url.empty? && anthropic?(fields) && fields["host"].to_s.strip.empty? && fields["port"].to_s.strip.empty?
|
|
276
284
|
unless url.empty?
|
|
277
285
|
uri = URI.parse(url)
|
|
278
|
-
|
|
286
|
+
path = uri.path.to_s.chomp("/")
|
|
287
|
+
path = path.delete_suffix("/v1") if anthropic?(fields)
|
|
288
|
+
return [uri.host.to_s.downcase, uri.port, path]
|
|
279
289
|
end
|
|
280
290
|
host = fields["host"].to_s.strip.downcase
|
|
281
291
|
return nil if host.empty?
|
|
@@ -7,6 +7,8 @@ require "uri"
|
|
|
7
7
|
require "ipaddr"
|
|
8
8
|
require_relative "../version"
|
|
9
9
|
require_relative "../llm/openai_chat"
|
|
10
|
+
require_relative "../llm/anthropic_messages"
|
|
11
|
+
require_relative "../llm/dialect"
|
|
10
12
|
require_relative "../llm/http"
|
|
11
13
|
require_relative "../llm/api_key"
|
|
12
14
|
require_relative "../host_keys"
|
|
@@ -15,9 +17,12 @@ require_relative "../redacted_inspect"
|
|
|
15
17
|
module Samagotchi
|
|
16
18
|
module Bootstrap
|
|
17
19
|
# What `chi bootstrap TARGET` finds at a model server: llama.cpp's native
|
|
18
|
-
# API (/props), an OpenAI-compatible one (/v1/models),
|
|
19
|
-
# API
|
|
20
|
-
#
|
|
20
|
+
# API (/props), an OpenAI-compatible one (/v1/models), Anthropic's
|
|
21
|
+
# Messages API (its /v1/models, asked only after the OpenAI list was
|
|
22
|
+
# refused with 400/401/403: the Anthropic list reads an OpenAI-shaped
|
|
23
|
+
# one too), one that wants an API key, or nothing. One attempt per
|
|
24
|
+
# request, no retries: a refused port answers at once instead of after
|
|
25
|
+
# chi's usual backoff.
|
|
21
26
|
class Probe
|
|
22
27
|
include RedactedInspect
|
|
23
28
|
|
|
@@ -31,6 +36,14 @@ module Samagotchi
|
|
|
31
36
|
SCAN_TIMEOUT = 1
|
|
32
37
|
TEST_TIMEOUT = 60
|
|
33
38
|
TEST_PROMPT = "Reply with the word ok."
|
|
39
|
+
# The OpenAI list's statuses after which the Anthropic list is asked:
|
|
40
|
+
# Anthropic answers a Bearer request 400 (no anthropic-version) or
|
|
41
|
+
# 401/403. Not 404/405: a typed URL path sends the two lists to
|
|
42
|
+
# different paths, so a 404 there says nothing about Anthropic.
|
|
43
|
+
ANTHROPIC_AFTER = [400, 401, 403].freeze
|
|
44
|
+
AUTH_STATUSES = [401, 403].freeze
|
|
45
|
+
FIREWORKS_HOST = "fireworks.ai"
|
|
46
|
+
FIREWORKS_MODEL_PREFIX = "accounts/fireworks/"
|
|
34
47
|
|
|
35
48
|
# One place a server may be: root is scheme://host:port (llama.cpp's
|
|
36
49
|
# /props), base the OpenAI API base (<root>/v1, or a URL's own path).
|
|
@@ -44,17 +57,34 @@ module Samagotchi
|
|
|
44
57
|
def label = "#{host}:#{port}"
|
|
45
58
|
end
|
|
46
59
|
|
|
47
|
-
# kind: :native, :openai, :needs_key, :unreachable or
|
|
48
|
-
# props: the /props JSON (native); models: [LLM::ModelInfo];
|
|
60
|
+
# kind: :native, :openai, :anthropic, :needs_key, :unreachable or
|
|
61
|
+
# :unknown. props: the /props JSON (native); models: [LLM::ModelInfo];
|
|
49
62
|
# status: the HTTP status that decided (:unknown, :needs_key);
|
|
50
|
-
# reason: why it's unreachable
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
63
|
+
# reason: why it's unreachable; dialect: an :anthropic result's
|
|
64
|
+
# LLM::Dialect name when not the standard one ("fireworks"), else nil;
|
|
65
|
+
# auth: :bearer when an :anthropic list took the key only as Bearer
|
|
66
|
+
# (a Symbol, LLM::ApiKey::SCHEMES), else nil.
|
|
67
|
+
Result = Data.define(:kind, :candidate, :props, :models, :status, :reason, :dialect, :auth) do
|
|
68
|
+
def self.of(kind, candidate, props: nil, models: [], status: nil, reason: nil, dialect: nil, auth: nil)
|
|
69
|
+
new(kind: kind, candidate: candidate, props: props, models: models, status: status, reason: reason,
|
|
70
|
+
dialect: dialect, auth: auth)
|
|
54
71
|
end
|
|
55
72
|
|
|
56
73
|
def native? = kind == :native
|
|
57
|
-
def
|
|
74
|
+
def anthropic? = kind == :anthropic
|
|
75
|
+
def reached? = %i[native openai anthropic].include?(kind)
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# A failed test turn: the message, and the HTTP status and request path
|
|
79
|
+
# when the server answered (BootstrapCommand's --api anthropic hint).
|
|
80
|
+
class TestFailed < RuntimeError
|
|
81
|
+
attr_reader :status, :path
|
|
82
|
+
|
|
83
|
+
def initialize(message = nil, status: nil, path: nil)
|
|
84
|
+
@status = status
|
|
85
|
+
@path = path
|
|
86
|
+
super(message)
|
|
87
|
+
end
|
|
58
88
|
end
|
|
59
89
|
|
|
60
90
|
# The target the user typed, as the candidates to try in order.
|
|
@@ -110,6 +140,21 @@ module Samagotchi
|
|
|
110
140
|
false
|
|
111
141
|
end
|
|
112
142
|
|
|
143
|
+
# The Messages API root: the OpenAI base without its trailing /v1
|
|
144
|
+
# (HostEntry#api_base_url's rule), e.g. https://api.anthropic.com.
|
|
145
|
+
def self.api_root(candidate) = candidate.base.chomp("/").delete_suffix("/v1")
|
|
146
|
+
|
|
147
|
+
# Fireworks by its host (fireworks.ai or a subdomain), or by a model id
|
|
148
|
+
# in its accounts/fireworks/ namespace.
|
|
149
|
+
def self.fireworks?(candidate, models = [])
|
|
150
|
+
fireworks_host?(candidate.host) || models.any? { |m| m.id.to_s.start_with?(FIREWORKS_MODEL_PREFIX) }
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
def self.fireworks_host?(host)
|
|
154
|
+
name = host.to_s.downcase
|
|
155
|
+
name == FIREWORKS_HOST || name.end_with?(".#{FIREWORKS_HOST}")
|
|
156
|
+
end
|
|
157
|
+
|
|
113
158
|
def self.parse_uri(text)
|
|
114
159
|
URI.parse(text)
|
|
115
160
|
rescue URI::InvalidURIError
|
|
@@ -128,18 +173,27 @@ module Samagotchi
|
|
|
128
173
|
# when this one refused the connection or failed its TLS handshake
|
|
129
174
|
# (https → http), not on a timeout.
|
|
130
175
|
# @param key_env [String, nil] the API key's environment variable
|
|
176
|
+
# @param api [Symbol, nil] :openai or :anthropic to skip the detection
|
|
177
|
+
# (--api); nil detects
|
|
178
|
+
# @param dialect [String, nil] the Messages dialect (--dialect); nil
|
|
179
|
+
# detects Fireworks
|
|
131
180
|
# @return [Result]
|
|
132
|
-
def classify_target(candidates, key_env: nil)
|
|
181
|
+
def classify_target(candidates, key_env: nil, api: nil, dialect: nil)
|
|
133
182
|
result = nil
|
|
134
183
|
candidates.each do |candidate|
|
|
135
|
-
result = classify(candidate, key_env: key_env)
|
|
184
|
+
result = classify(candidate, key_env: key_env, api: api, dialect: dialect)
|
|
136
185
|
break unless result.kind == :unreachable && result.reason.to_s.match?(/refused|SSL|TLS|certificate/i)
|
|
137
186
|
end
|
|
138
187
|
result
|
|
139
188
|
end
|
|
140
189
|
|
|
190
|
+
# /props (llama.cpp), then the OpenAI list, then (only after the OpenAI
|
|
191
|
+
# list's 400/401/403) the Anthropic one. api: :anthropic asks only the
|
|
192
|
+
# Anthropic list; api: :openai never asks it.
|
|
141
193
|
# @return [Result]
|
|
142
|
-
def classify(candidate, key_env: nil)
|
|
194
|
+
def classify(candidate, key_env: nil, api: nil, dialect: nil)
|
|
195
|
+
return classify_anthropic(candidate, key_env, dialect, nil) if api&.to_sym == :anthropic
|
|
196
|
+
|
|
143
197
|
props = begin
|
|
144
198
|
status, body = get(URI("#{candidate.root}/props"), key_env)
|
|
145
199
|
status == 200 ? parse_object(body) : nil
|
|
@@ -151,8 +205,9 @@ module Samagotchi
|
|
|
151
205
|
begin
|
|
152
206
|
Result.of(:openai, candidate, models: list_models(candidate.base, key_env))
|
|
153
207
|
rescue LLM::ProviderError => e
|
|
154
|
-
|
|
155
|
-
|
|
208
|
+
return failure(candidate, e) if api || !ANTHROPIC_AFTER.include?(e.status)
|
|
209
|
+
|
|
210
|
+
classify_anthropic(candidate, key_env, dialect, e)
|
|
156
211
|
rescue *network_errors => e
|
|
157
212
|
Result.of(:unreachable, candidate, reason: reason(e))
|
|
158
213
|
end
|
|
@@ -178,25 +233,27 @@ module Samagotchi
|
|
|
178
233
|
|
|
179
234
|
# One short chat request: does the server generate for +model+? Any 200
|
|
180
235
|
# with a choice counts (a thinking model may spend its 16 tokens
|
|
181
|
-
# reasoning).
|
|
182
|
-
#
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
236
|
+
# reasoning). api: :anthropic posts {root}/v1/messages instead, and any
|
|
237
|
+
# 200 with a content array counts (stop_reason max_tokens too).
|
|
238
|
+
# @param auth [Symbol, nil] the key's header on :anthropic (nil: the
|
|
239
|
+
# dialect's)
|
|
240
|
+
# @return [Float] seconds taken
|
|
241
|
+
# @raise [TestFailed] with what went wrong
|
|
242
|
+
def test_turn(candidate, model, key_env: nil, api: :openai, dialect: nil, auth: nil, read_timeout: TEST_TIMEOUT)
|
|
243
|
+
anthropic = api&.to_sym == :anthropic
|
|
244
|
+
uri, request, scheme = anthropic ? messages_request(candidate, model, dialect, auth) : chat_request(candidate, model)
|
|
189
245
|
started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
190
|
-
response = send_request(uri, request, key_env, read_timeout: read_timeout)
|
|
246
|
+
response = send_request(uri, request, key_env, read_timeout: read_timeout, scheme: scheme)
|
|
191
247
|
elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
|
|
192
248
|
body = parse_object(response.body)
|
|
193
|
-
unless response.code.to_i == 200 && body
|
|
194
|
-
raise "HTTP #{response.code}: #{error_text(body, response.body)}"
|
|
249
|
+
unless response.code.to_i == 200 && answered?(body, anthropic ? "content" : "choices")
|
|
250
|
+
raise TestFailed.new("HTTP #{response.code}: #{error_text(body, response.body)}", status: response.code.to_i,
|
|
251
|
+
path: uri.path)
|
|
195
252
|
end
|
|
196
253
|
|
|
197
254
|
elapsed
|
|
198
255
|
rescue *network_errors => e
|
|
199
|
-
raise "#{reason(e)} (#{candidate.label})"
|
|
256
|
+
raise TestFailed, "#{reason(e)} (#{candidate.label})"
|
|
200
257
|
end
|
|
201
258
|
|
|
202
259
|
private
|
|
@@ -224,16 +281,81 @@ module Samagotchi
|
|
|
224
281
|
timeout: @timeout, env: @env).list_models
|
|
225
282
|
end
|
|
226
283
|
|
|
284
|
+
# The Anthropic list at the candidate's API root, the key in the
|
|
285
|
+
# dialect's header (Bearer on Fireworks, detected by its host before
|
|
286
|
+
# listing); a 401/403 with the standard dialect's x-api-key is asked
|
|
287
|
+
# once more with Bearer (a gateway that wants it). Ids in Fireworks'
|
|
288
|
+
# namespace make the result's dialect fireworks. +openai_error+ is the
|
|
289
|
+
# OpenAI list's refusal, the result again when the Anthropic list fails
|
|
290
|
+
# other than 401/403 (nil when --api anthropic skipped it).
|
|
291
|
+
def classify_anthropic(candidate, key_env, dialect, openai_error)
|
|
292
|
+
forced = dialect
|
|
293
|
+
dialect ||= LLM::Dialect::FIREWORKS.name if self.class.fireworks_host?(candidate.host)
|
|
294
|
+
auth = nil
|
|
295
|
+
models = begin
|
|
296
|
+
anthropic_models(candidate, key_env, dialect: dialect, auth: nil)
|
|
297
|
+
rescue LLM::ProviderError => e
|
|
298
|
+
raise unless AUTH_STATUSES.include?(e.status) && key_env && LLM::Dialect.for(dialect).auth != :bearer
|
|
299
|
+
|
|
300
|
+
auth = :bearer
|
|
301
|
+
anthropic_models(candidate, key_env, dialect: dialect, auth: auth)
|
|
302
|
+
end
|
|
303
|
+
dialect ||= LLM::Dialect::FIREWORKS.name if forced.nil? && self.class.fireworks?(candidate, models)
|
|
304
|
+
dialect = nil if dialect == LLM::Dialect::STANDARD.name
|
|
305
|
+
Result.of(:anthropic, candidate, models: models, dialect: dialect, auth: auth)
|
|
306
|
+
rescue LLM::ProviderError => e
|
|
307
|
+
return Result.of(:needs_key, candidate, status: e.status) if AUTH_STATUSES.include?(e.status)
|
|
308
|
+
|
|
309
|
+
failure(candidate, openai_error || e)
|
|
310
|
+
rescue *network_errors => e
|
|
311
|
+
Result.of(:unreachable, candidate, reason: reason(e))
|
|
312
|
+
end
|
|
313
|
+
|
|
314
|
+
def anthropic_models(candidate, key_env, dialect:, auth:)
|
|
315
|
+
LLM::AnthropicMessages.new(base_url: self.class.api_root(candidate), host_name: candidate.host, api_key_env: key_env,
|
|
316
|
+
auth: auth, dialect: dialect, stream: false, retries: false, timeout: @timeout,
|
|
317
|
+
env: @env).list_models
|
|
318
|
+
end
|
|
319
|
+
|
|
320
|
+
# A list's refusal as a Result: :needs_key on 401/403, else :unknown.
|
|
321
|
+
def failure(candidate, error)
|
|
322
|
+
kind = AUTH_STATUSES.include?(error.status) ? :needs_key : :unknown
|
|
323
|
+
Result.of(kind, candidate, status: error.status, reason: error.status ? nil : error.message)
|
|
324
|
+
end
|
|
325
|
+
|
|
326
|
+
def chat_request(candidate, model)
|
|
327
|
+
uri = URI("#{candidate.base}/chat/completions")
|
|
328
|
+
request = json_post(uri, model: model, messages: [{ role: "user", content: TEST_PROMPT }], max_tokens: 16, stream: false)
|
|
329
|
+
[uri, request, :bearer]
|
|
330
|
+
end
|
|
331
|
+
|
|
332
|
+
# No stream and no thinking field: the model's own defaults.
|
|
333
|
+
def messages_request(candidate, model, dialect, auth)
|
|
334
|
+
uri = URI("#{self.class.api_root(candidate)}/v1/messages")
|
|
335
|
+
request = json_post(uri, model: model, max_tokens: 16, messages: [{ role: "user", content: TEST_PROMPT }])
|
|
336
|
+
request["anthropic-version"] = LLM::AnthropicMessages::API_VERSION
|
|
337
|
+
[uri, request, auth || LLM::Dialect.for(dialect).auth]
|
|
338
|
+
end
|
|
339
|
+
|
|
340
|
+
def json_post(uri, body)
|
|
341
|
+
request = Net::HTTP::Post.new(uri)
|
|
342
|
+
request["Content-Type"] = "application/json"
|
|
343
|
+
request.body = JSON.generate(body)
|
|
344
|
+
request
|
|
345
|
+
end
|
|
346
|
+
|
|
347
|
+
def answered?(body, key) = body.is_a?(Hash) && body[key].is_a?(Array) && !body[key].empty?
|
|
348
|
+
|
|
227
349
|
def get(uri, key_env)
|
|
228
350
|
response = send_request(uri, Net::HTTP::Get.new(uri), key_env, read_timeout: @timeout)
|
|
229
351
|
[response.code.to_i, response.body]
|
|
230
352
|
end
|
|
231
353
|
|
|
232
|
-
# One attempt through LLM::HTTP (User-Agent, the key's
|
|
233
|
-
# any status returned as is. A key neither its variable nor the
|
|
354
|
+
# One attempt through LLM::HTTP (User-Agent, the key in +scheme+'s
|
|
355
|
+
# header), any status returned as is. A key neither its variable nor the
|
|
234
356
|
# Keychain holds sends no header: the server's 401 then says one is needed.
|
|
235
|
-
def send_request(uri, request, key_env, read_timeout:)
|
|
236
|
-
key = key_env && HostKeys.fetch(key_env, env: @env) ? LLM::ApiKey.for(key_env, host: uri.host, env: @env) : nil
|
|
357
|
+
def send_request(uri, request, key_env, read_timeout:, scheme: :bearer)
|
|
358
|
+
key = key_env && HostKeys.fetch(key_env, env: @env) ? LLM::ApiKey.for(key_env, host: uri.host, env: @env, scheme: scheme) : nil
|
|
237
359
|
LLM::HTTP.new(label: uri.host, open_timeout: @timeout, read_timeout: read_timeout, api_key: key,
|
|
238
360
|
retry_policy: LLM::HTTP::RetryPolicy.none)
|
|
239
361
|
.fetch(uri, request, retries: false, check_status: false, log_fields: { purpose: "probe" })
|