isorite 0.3.0 → 0.5.0

package/docs/setup-for-agents.md

@@ -14,12 +14,13 @@ turn off certificate checking (`NODE_TLS_REJECT_UNAUTHORIZED=0`).
 
 | Command | What it does |
 |---|---|
+| `isorite new <folder>` | Makes the folder (and its parents), empty but for `.isorite/`. Refuses one with files in it already: that is opened as it is. |
 | `isorite doctor [<notes>] [--report] [--offline]` | Checks the setup a step at a time (below). Exit 1 if anything failed. `--offline`: asks the provider nothing. |
 | `isorite serve <notes> [--port <n>]` | The browser app at `http://localhost:4767/`, on 127.0.0.1 only. |
 | `isorite term [<notes>] [--port <n>]` | OpenCode 2 in this terminal, in the current folder (the code), with the notes beside it. Starts the isorite server if none answers on the port, and stops it on exit. |
 | `isorite check <notes>` | Lints the notes: frontmatter, links, cycles. |
-| `isorite --tutorial` | A tour on an example base with a scripted model: no key or network needed. Good to check the install alone. |
-| `isorite model [--from <folder>]` | Optional: fetches the model for search by meaning (53 MB). |
+| `isorite guide [<folder>]` | The guide, a knowledge base about isorite, written into `~/isorite-guide` (or the folder named) and served. With no provider set up, Iso is the guide's finder: no key or network needed. Good to check the install alone. `isorite --tutorial` is another name for it. |
+| `isorite model [--from <folder>]` | Optional: the model for search by meaning. Says which is installed and where, else fetches it (53 MB) from npm, a GitHub release, then Hugging Face, saying why each failed. `--from`: granite's files, or any compatible model (below). |
 | `isorite help setup` | Prints this. |
 
 ## Files
@@ -29,7 +30,10 @@ turn off certificate checking (`NODE_TLS_REJECT_UNAUTHORIZED=0`).
 | `~/.config/isorite/config.toml` (or `$XDG_CONFIG_HOME/isorite/`) | isorite's settings for this machine: `device`, `[provider]`, `[tiers]`, `[prices]`. Mode 600. Holds the key. |
 | `~/.config/isorite/opencode/` | OpenCode's settings as isorite writes them at each `isorite term` start (`opencode.json`, theme, plugin link). Rewritten; don't edit. |
 | `~/.config/isorite/opencode/local.json` | The person's own OpenCode settings, laid over isorite's. Never rewritten. |
+| `<notes>/AGENTS.md` | The base's own instructions (notation, level, how notes are named), sent to Iso with every turn and named in OpenCode's `instructions`. Not a note: not on the Atlas, not linted. |
 | `<notes>/.isorite/` | isorite's own folder in the notes: journals of every change, `settings.toml` (`repository = "<path>"`, where code links point). Made on first open; commit it with the notes. |
+| `~/.local/share/isorite/models/` (or `$XDG_DATA_HOME/isorite/models/`) | The models for search by meaning, a folder each: `granite-embedding-small-english-r2/`, and any other installed with `--from`. `use` names the one used, if not granite. |
+| `~/.cache/isorite/vectors-<hash>.json` | Each base's vectors, made by the model; made again when the model changes. |
 | `$TMPDIR/isorite-term-<port>.log` | The isorite server's output, when `isorite term` started it. Look here when "the isorite server did not start". |
 | `$TMPDIR/isorite-opencode-<port>.log` | OpenCode's server's output, under `isorite term`. |
 
@@ -44,6 +48,7 @@ url = "https://…/v1"                  # OpenAI-compatible; /chat/completions a
 key = "…"                             # the key; also settable in the browser (behind the logo)
 key_header = "api-key"                # send the key in this header, as it is, instead of Authorization: Bearer
 stream_usage = false                  # don't send stream_options (some gateways answer 400 to it)
+max_tokens = 16000                    # the least output budget any call gets, for models that think at length
 cache_marks = true                    # send Anthropic cache marks (OpenRouter-style gateways only)
 pfx = "~/certs/me.pfx"                # client certificate (mTLS), PKCS#12; or cert + cert_key (PEM)
 pfx_passphrase = "$PFX_PASS"          # "$NAME": read from that environment variable
@@ -82,6 +87,8 @@ Rules:
 | `ISORITE_NODE_DEFAULTS=0` | Leave Node as it is: no system authorities, no proxy unless Node is told. |
 | `NODE_EXTRA_CA_CERTS` | A .pem of extra authorities, read by Node at start. |
 | `ISORITE_GATEWAY_URL`, `ISORITE_GATEWAY_MODELS`, `ISORITE_GATEWAY_KEY` | The terminal only: OpenCode asks this OpenAI-compatible address directly, with these models (comma-separated) and key, instead of `config.toml`'s provider. No certificate or headers apply. |
+| `ISORITE_EMBED_DIR` | The model for search by meaning is in this folder (and nowhere else is looked). |
+| `ISORITE_EMBED=off` | No search by meaning, whatever is installed. |
 | `ISORITE_OPENCODE` | The OpenCode 2 binary to run. Otherwise `opencode2`, then `opencode`, must be version 2. |
 | `XDG_CONFIG_HOME` | Where `isorite/config.toml` is looked for. |
 | `SSH_CONNECTION` | Set over SSH: `term`, `serve` and `doctor` print the `ssh -L` command for the browser. |
@@ -96,6 +103,12 @@ Rules:
   server's relay, and with the server's token as its key. The relay adds the real key, certificate, headers and
   query, and passes `/chat/completions` and `/models` on. So whatever works for Iso works for the terminal. The
   relay answers only on 127.0.0.1, and only that token.
+- **Retries.** A 502, 503 or 504, a dropped connection or a timeout is asked again up to 3 times (after about 1, 2
+  and 4 s), and a 429 once (after its `Retry-After`, if 10 s or less), but only before any of the reply has come.
+  The relay does the same before it passes anything to OpenCode (which retries on its own as well). 4xx answers
+  and names not found are the settings', and are not asked again.
+- **An empty reply.** A reply cut off at its length limit with nothing written (a reasoning model that spent the
+  budget thinking) is asked once more with three times the room.
 - With no key and no certificate in `config.toml`, and no `ISORITE_GATEWAY_*`, OpenCode uses its own models.
   `isorite term` prints which models it uses, and from where.
 
@@ -131,6 +144,10 @@ Rules:
 | `ETIMEDOUT`, `UND_ERR_CONNECT_TIMEOUT` | No answer: often a proxy is needed. | Set `HTTPS_PROXY` (and `NO_PROXY` for internal hosts reached directly). |
 | `said 401` / `said 403` | The key, or where it's sent, isn't accepted. | Check the key, `key_header`, and any header the gateway wants. The gateway's own message usually names it. |
 | `said 404 … no model called X, which is OpenRouter's name for it` | No `[tiers]`. | Name the gateway's models under `[tiers]`. `doctor` lists the models `/models` offers. |
+| `… is unavailable (503 …), tried 4 times over N s: it is likely …, not your settings` | The gateway, or the model behind it, is down. | Wait, and send it again (↻ under the turn). If it lasts, the gateway's status page or owner. |
+| `… used its whole output budget (N tokens) thinking, and wrote nothing` | A reasoning model ran out of room before it wrote. | A lower tier, or `max_tokens` under `[provider]`. |
+| `! search by meaning: no model …` | Not installed (optional). | `isorite model`. Its failures name each source; offline, `isorite model --from <folder>` with a copy. |
+| `The model could not be fetched:` and a line for npm, GitHub, Hugging Face | None reachable from here. | Copy the model's folder from a machine that has it (`isorite model` there says where), then `isorite model --from <folder>`; or any compatible model. |
 | `said 400: … stream_options` | The gateway refuses `stream_options`. | `stream_usage = false`. |
 | `✗ tools: … answered without calling the tool` | The model or gateway drops tools. | Choose a model that supports tool calls. Iso and the terminal need them. |
 | `isorite term needs OpenCode 2` | OpenCode 1, or none. | `npm i -g @opencode/cli`, or `ISORITE_OPENCODE=…`. |
@@ -138,5 +155,13 @@ Rules:
 | The link doesn't open, over SSH | `localhost` is the remote machine. | `ssh -L 4767:localhost:4767 <host>` from the person's own machine. |
 | The terminal works but uses an unexpected model | Check the line `isorite term` prints: `config.toml`, `ISORITE_GATEWAY_URL`, or OpenCode's own (no key or certificate). | Fix what it names. |
 
+## Another model for search by meaning
+
+Any ONNX model that gives 384 numbers for a text, with its `tokenizer.json`, can stand in for granite:
+`isorite model --from <folder>`. It's checked by use (loaded, and asked for one vector), copied into the models
+folder, and used from then on; each base's vectors are made again with it. An optional `isorite-model.json` beside
+its files gives `{"name": "…", "pooling": "cls" | "mean"}`: `cls` (the default) takes the first token's state, as
+granite is used; `mean` averages them all, as sentence-transformers models such as all-MiniLM-L6-v2 are used.
+
 If the gateway isn't OpenAI-compatible (Bedrock's or Vertex's own APIs, for example), isorite can't use it directly.
 An OpenAI-compatible proxy in front of it, such as LiteLLM, can.

package/docs/setup.md

@@ -9,7 +9,7 @@ run `isorite doctor`: it checks each step and says what to change.
 - **isorite**: `npm i -g isorite`.
 - For the terminal, **OpenCode 2**: `npm i -g @opencode/cli`.
 
-`isorite --tutorial` now shows you around, on an example base, with a scripted model: no key, nothing sent anywhere.
+`isorite guide` now opens the guide, a knowledge base about isorite with things to do in it. Until a provider is set up, Iso there is the guide's finder: no key, nothing sent anywhere.
 
 ## 2. A model provider
 
@@ -45,6 +45,7 @@ url = "https://llm-gateway.corp.example/v1"
 key = "…"
 key_header = "api-key"                     # the header the key goes in, as it is, instead of Authorization: Bearer
 stream_usage = false                       # for a gateway that refuses stream_options; the gauge then counts nothing
+max_tokens = 16000                         # the least room a reply is given, for models that think at length
 
 # A client certificate (mTLS), for a gateway that asks for one: a .pfx (or .p12) ...
 pfx = "~/certs/me.pfx"
@@ -77,12 +78,20 @@ If the company's authority isn't installed on the machine, name its certificate
 
 ## 3. Your notes
 
-A folder of Markdown files; empty is fine.
+A folder of Markdown files. To start from nothing:
 
 ```sh
+isorite new ~/notes/my-project
 isorite serve ~/notes/my-project
 ```
 
+`isorite new` makes the folder, empty but for isorite's own `.isorite/`. A folder of notes you have already is
+opened as it is, with `isorite serve`.
+
+How you want the base kept (your notation, the level, the language, how notes are named) goes in `AGENTS.md` at
+the top of the notes, which Iso and the terminal's agent are both given. Edit it in Iso's settings, the icon at the
+right of its pane's header, or in any editor. Iso may offer a line for it when you say how you want things done.
+
 It prints a link to click: `http://localhost:4767/`. isorite keeps what it needs in the folder's `.isorite/`.
 
 ## 4. The terminal
@@ -95,8 +104,25 @@ isorite term ~/notes/my-project
 It prints the same link, and which models the terminal uses. OpenCode uses the provider above, through isorite. More
 is in [terminal.md](terminal.md).
 
+## Search by meaning (optional)
+
+```sh
+isorite model
+```
+
+It says which model is installed and where, or fetches it (53 MB, once): from npm, else from a release on GitHub,
+else from Hugging Face. Without it, Iso searches by words and follows links. Where none of those can be reached,
+copy the model's folder from a machine that has it (`isorite model` there says where it is) and run
+`isorite model --from <folder>`. Any compatible model works the same way: an ONNX model that gives 384 numbers,
+with its `tokenizer.json` (for example all-MiniLM-L6-v2, with `{"pooling": "mean"}` in an `isorite-model.json`
+beside it).
+
 ## When something fails
 
+If a gateway is down for a moment (it answers 502, 503 or 504, or drops the connection), Iso and the terminal ask
+it again, up to three times. When it stays down, Iso says it's likely the gateway, not your settings, and the ↻
+under the turn sends it again.
+
 ```sh
 isorite doctor ~/notes/my-project
 ```

package/docs/terminal.md

@@ -17,7 +17,8 @@ Setting isorite up the first time, or at work behind a company's gateway, is in
 
 ## Your notes for the project
 
-Keep the project's notes in a folder of their own, for example `~/notes/<project>`. It can start empty. isorite
+Keep the project's notes in a folder of their own, for example `~/notes/<project>`. It can start empty:
+`isorite new ~/notes/<project>` makes it. isorite
 keeps what it needs in the folder's `.isorite/` (the journals of every change, and its settings), made the first
 time the folder is opened. Commit `.isorite/` with the notes, if they're in git.
 
@@ -71,7 +72,12 @@ terminal's session:
 - **Notes the agent changes** are written at once, under its name. Turn on provenance in the Note pane's header to
   see the agent's words marked, and the timeline to put a note back.
 
-With both off, the pane is Iso's own again.
+With both off, the pane is Iso's own again. The ≡ in its header, or a click on its gauge, opens what Iso is sent:
+a row for each part, with its size. These are the notes you gave and the notes Iso read, the conversation's turns,
+and what is always sent. Iso keeps a note it read in hand, and sets it aside after three turns unused, as a line it
+can read again. A click switches a row on or off. At the head of the conversation, one icon sums the earlier turns
+up into a briefing sent instead of them, and another starts a new conversation from that briefing. Iso sums up on
+its own only when a turn would be more than the model takes.
 
 ## The models
 
@@ -88,6 +94,9 @@ OpenCode doesn't know a gateway's prices, so Iso's pane shows the session's toke
 
 ## OpenCode's settings
 
+OpenCode is given the base's `AGENTS.md` (in its `instructions`), when `isorite term` is named the notes, and the
+knowledge base's ways through `brief_base`: how a note is written and linked, and how the base is kept.
+
 isorite keeps OpenCode's settings in `~/.config/isorite/opencode/`, rewritten at each start, so your own OpenCode
 settings are never read or changed. Anything of your own goes in `~/.config/isorite/opencode/local.json`, which is
 never rewritten, and is laid over what isorite writes. For example, to have OpenCode ask before every command (by
@@ -104,8 +113,11 @@ gateway, or a model served on your own machine). What it fetches, it fetches onc
 
 - isorite and OpenCode, from npm: through your company's npm mirror, if it has one.
 - The model for search by meaning (optional): `isorite model` gets it from npm too (`isorite-model-granite`), so the
-  same mirror serves it; else from Hugging Face. On a machine with neither, copy its files from one that has them:
-  `isorite model --from <folder>`. Every file is checked against its SHA-256.
+  same mirror serves it; else from its release on GitHub, else from Hugging Face. On a machine with none of them,
+  copy its folder from one that has it (`isorite model` there says where), or download the package from
+  `https://github.com/lachlanjs/isorite/releases/tag/model-granite-1.0.0` and unpack it, then
+  `isorite model --from <folder>`. Granite's files are checked against their SHA-256; any other compatible model is
+  checked by use ([setup-for-agents.md](setup-for-agents.md) says which).
 
 ## Who can reach it
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "isorite",
-  "version": "0.3.0",
+  "version": "0.5.0",
   "description": "An agentic harness for understanding: your notes as a knowledge base, an Atlas of them, an agent that reads them, and study.",
   "license": "MIT",
   "type": "module",

package/plugin/tui.tsx

@@ -5,7 +5,7 @@
 import { spawn } from "node:child_process";
 import { BoxRenderable, fg, ImageRenderable, RGBA, StyledText, TextAttributes, TextRenderable, type Renderable, type TextChunk } from "@opentui/core";
 import { Plugin } from "@opencode/plugin/tui";
-import { createEffect, createResource, createSignal, For, onCleanup, Show, untrack } from "solid-js";
+import { createEffect, createResource, createSignal, For, onCleanup, Show } from "solid-js";
 import type { ScrollBoxRenderable } from "@opentui/core";
 import { ago, anchorOf, blocks, describe, momentKey, moments, reach, type Entry, type MapModel } from "../dist/core.js";
 import { drawAtlas, runsOf, type Role } from "./oneatlas.ts";
@@ -15,7 +15,6 @@ import * as Isorite from "./isorite.ts";
 import * as Logo from "./logo.ts";
 import { lapis, quartz, routeColours } from "./palette.ts";
 import { Play, type Step } from "./play.ts";
-import { TOUR, type TourState } from "./tour.ts";
 import { parseSources } from "./sources.ts";
 import * as Wordmark from "./wordmark.ts";
 
@@ -265,9 +264,6 @@ export default Plugin.define({
         </text>
       ),
     });
-    // What the tour (isorite --tutorial) waits on that nothing else counts: the agent's turns ended, Iso's answers,
-    // and notes sent from here.
-    const [turns, setTurns] = createSignal(0), [isoAnswers, setIsoAnswers] = createSignal(0), [sent, setSent] = createSignal(0);
 
     /** [s]: a note, or a passage of it, sent to the agent from the terminal, as from the browser: under "from you". */
     const sendToAgent = async (note: string, passage?: string) => {
@@ -275,7 +271,7 @@ export default Plugin.define({
       const said = await ctx.ui.dialog.prompt({ title: "Send to the agent", description: passage ? "the passage the cursor is on" : `the note “${(titles().get(note) ?? note).toLowerCase()}”`, placeholder: "what you want of it (optional)" });
       if (said === undefined) return;
       await Isorite.post("/api/agent/send", { note, ...(passage ? { passage } : {}), said: said.trim() })
-        .then(() => setSent((n) => n + 1), (e: Error) => ctx.ui.toast?.show?.({ message: e.message, variant: "error" }));
+        .catch((e: Error) => ctx.ui.toast?.show?.({ message: e.message, variant: "error" }));
     };
 
     const stop = new AbortController();
@@ -293,8 +289,7 @@ export default Plugin.define({
         if (sessionID) void deliver(sessionID);
         return;
       }
-      if (type === "agent-done" && typeof data.session === "string") { reading.delete(data.session); hear((n) => n + 1); setTurns((n) => n + 1); }
-      if (type === "iso-done") setIsoAnswers((n) => n + 1);
+      if (type === "agent-done" && typeof data.session === "string") { reading.delete(data.session); hear((n) => n + 1); }
       if (type === "note-changed") bumpMap((n) => n + 1);
       if (type !== "agent-step" || data.agent !== Isorite.AGENT || typeof data.session !== "string") return;
       const play = plays.get(data.session) ?? new Play();
@@ -324,75 +319,6 @@ export default Plugin.define({
       return new TextRenderable(ctx.renderer, { content: new StyledText(chunks) });
     });
 
-    // ------------------------------------------------------------ the tour (isorite --tutorial)
-    const TUTORIAL = process.env.ISORITE_TUTORIAL === "1";
-    const [stepAt, setStepAt] = createSignal(TUTORIAL ? 0 : TOUR.length);
-    const [cardShown, setCardShown] = createSignal(true);
-    const tourState = (): TourState => {
-      heard();
-      const route = ctx.ui.router.current(), shown = note();
-      return {
-        page: route.type === "plugin" ? (route.name === SPLASH ? "splash" : route.name === NOTE ? "note" : route.name) : route.type,
-        turns: turns(),
-        inPlay: route.type === "session" ? plays.get(route.sessionID)?.size ?? 0 : 0,
-        atlasOpen: ctx.ui.panel.current()?.name === ATLAS,
-        note: shownNote(),
-        // Not known until the note is read: no step moves on for a note still opening.
-        waiting: shown && shown.id === shownNote() ? shown.items.filter((i) => i.type === "change").length : -1,
-        past: moment() !== null,
-        isoAnswers: isoAnswers(),
-        sent: sent(),
-      };
-    };
-    /** Puts the step's prompt in the prompt being typed in, once it has focus and while it is empty. */
-    const fillPrompt = (text: string) => {
-      let tries = 0;
-      const timer = setInterval(() => {
-        const editor = ctx.renderer.currentFocusedEditor;
-        if (editor && !editor.plainText.trim()) editor.insertText(text);
-        if (editor || ++tries > 50) clearInterval(timer);
-      }, 100);
-    };
-    ctx.ui.slot({
-      append: "app",
-      render() {
-        if (!TUTORIAL) return null;
-        // A step moves on when what it waits for holds, judged against what held when it began.
-        let at = -1, began = untrack(tourState);
-        createEffect(() => {
-          const i = stepAt(), now = tourState(), step = TOUR[i];
-          if (i !== at) { at = i; began = now; if (step?.fill) fillPrompt(step.fill); return; }
-          if (step?.until?.(now, began)) setStepAt(i + 1);
-        });
-        ctx.keymap.layer(() => ({
-          mode: "global",
-          commands: [
-            { id: "isorite.tour.next", title: "Tour: the next step", palette: true, slash: { name: "next" }, run: () => { setStepAt((i) => Math.min(i + 1, TOUR.length)); } },
-            { id: "isorite.tour.card", title: "Tour: hide or show the card", palette: true, slash: { name: "tour" }, run: () => { setCardShown((v) => !v); } },
-          ],
-        }));
-        return (
-          <Show when={cardShown() && TOUR[stepAt()]}>
-            {(step) => (
-              <box position="absolute" right={2} bottom={4} width={48} zIndex={3000} flexDirection="column" paddingLeft={1} paddingRight={1}
-                border borderStyle="rounded" borderColor={ground().growth} backgroundColor={ground().g0} title={` tour · ${stepAt() + 1} of ${TOUR.length} `}>
-                <text fg={ground().text} wrapMode="word"><b>{step().title}</b></text>
-                <text fg={ground().soft} wrapMode="word">{step().text}</text>
-                <text> </text>
-                <box flexDirection="row">
-                  <text wrapMode="none">
-                    <For each={TOUR}>{(_, k) => <span style={{ fg: k() <= stepAt() ? ground().growth : ground().line }}>{k() <= stepAt() ? "■" : "□"}</span>}</For>
-                  </text>
-                  <box flexGrow={1} />
-                  <text fg={ground().faint} wrapMode="none">{step().until ? "/tour hides" : "/next   /tour hides"}</text>
-                </box>
-              </box>
-            )}
-          </Show>
-        );
-      },
-    });
-
     ctx.ui.slot({
       append: "sidebar.content",
       render: (input) => {