@javierkaiser/opencode-indexing 0.1.1 → 0.2.1

package/CHANGELOG.md

@@ -5,12 +5,166 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.2.1] - 2026-10-15
+
+### Added
+
+- **`opencode plugin add @javierkaiser/opencode-indexing` is now the documented
+  install.** It was already possible — the package was published and exported
+  `./tui` — but the README told people to run `npm install` and edit
+  `opencode.json` by hand, which puts the package in whatever directory they were
+  standing in and never tells OpenCode about it. The CLI command installs into
+  OpenCode's own cache, records the entry, and brings the update and removal path
+  (`plugin check`, `plugin update`, `plugin remove`) with it.
+- **The installer has a package mode**, and picks between the two from where it
+  runs: a checkout installs the working tree, an installed copy installs the
+  published package. `npx @javierkaiser/opencode-indexing` therefore does the
+  right thing with no flags, including the permission rules and the restart that
+  `plugin add` alone does not do.
+- `bin` in `package.json`, so `npx` has something to run.
+- The plugin options and `ignoreKiloConfig` are documented in the README. The
+  option existed in the code and had tests, but appeared in no documentation.
+
+### Fixed
+
+- **"No Kilo index found" for a workspace that has one.** The server plugin is
+  instantiated for the directory the service runs in, so every RPC that omitted a
+  `directory` described *that* project. `kilo.discover` took no directory at all,
+  which is why opening the palette in one project and importing reported nothing
+  while the footer, using the same client and passing the directory, looked
+  correct. `kilo.discover` and `settings.test` now accept a directory, and the
+  palette, the settings page and the index actions all send the caller's. The
+  directory is injected once by `bindWorkspace` rather than passed at each call
+  site, so a new workspace-scoped RPC is correct by default instead of correct by
+  remembering.
+- **The tools read and indexed the service's directory.** All seven were pinned to
+  the plugin instance's root, so `indexing_search` in a session answered from
+  another codebase — results from the wrong project read as plausible — and
+  `indexing_import` refused an index that was sitting right there. They now use
+  `context.directory`, with the instance root as the fallback for hosts that do
+  not report one.
+- **`@lancedb/lancedb` could never load, so no Kilo index was ever importable
+  from LanceDB.** `apache-arrow` is that package's *peerDependency*, and the
+  installer runs `npm install --omit=peer`, which drops peers from the tree: the
+  native binary loaded and then threw `Cannot find module 'apache-arrow'` on first
+  use. It is now declared explicitly. This also broke `vectorStore: lancedb`
+  outright.
+- **An unreadable index was reported as a missing one.** Discovery swallowed
+  backend failures, so "Kilo is not running here" and "Kilo is not installed here"
+  both surfaced as *No Kilo index found*. It now names the path it found and the
+  reason it could not be read, in `indexing_status`, `indexing_build`,
+  `indexing_config`, `settings test kilo`, the import tool and the empty-state
+  toast. `indexing_build` is the one that matters most: with nothing reported, a
+  build in a workspace whose Kilo index could not be read re-embedded everything
+  without saying why.
+- **`apache-arrow` is declared in `package.json`, not only in the installer.**
+  It is a *peer* of `@lancedb/lancedb`, and the two install paths disagreed about
+  peers: `install.mjs` runs `npm install --omit=peer`, which drops them, while
+  the npm-based install OpenCode performs keeps them. Declaring it makes the
+  dependency true for both instead of true for whichever path happened to be
+  tested.
+- The installer no longer keeps its own copy of the dependency list; it derives
+  the installed `package.json` from the published one. That duplication is how
+  `apache-arrow` came to be declared for one install path and missing from the
+  other.
+- `--package` deletes a previously copied plugin. Both sources define
+  `id: "opencode.indexing"` and the config directory is scanned automatically, so
+  leaving both would load the plugin twice.
+
+## [0.2.0] - 2026-10-14
+
+### Added
+
+- **`gemini`, `voyage` and `openrouter` are implemented.** They were selectable in
+  the provider list, passed every validation gate, and then threw
+  `Unsupported embedder provider` once indexing had already started and been billed
+  for a probe. A test asserted that they were unimplemented, which pinned the gap in
+  place.
+- **The model catalog is visible.** The TUI model field is now a chooser built from
+  the catalog, with a *Custom* entry for anything not listed and for providers with
+  no catalog at all. `indexing_config` lists every model with its dimension and
+  whether Kilo also knows it — which is the whole answer to whether a free import
+  will work.
+- **Per-model input ceilings and per-provider batch limits.** The ceilings differ by
+  more than an order of magnitude (`mxbai-embed-large` takes 512 tokens,
+  `voyage-code-4` takes 32,000), and one global constant truncated every chunk past
+  the smallest model's limit. Batch limits are now per provider: OpenAI caps a
+  request at 300,000 tokens whatever the item count, Gemini accepts a fixed number of
+  items, Voyage has both.
+- `voyage-code-4`, `voyage-4`, `voyage-4-large`, `voyage-4-lite`,
+  `gemini-embedding-2` and Ollama's `bge-m3` in the catalog, each marked as not
+  present in Kilo's registry so it is clear they cannot be imported.
+- `indexing_status` warns before a model change re-embeds the workspace, with the
+  point count, rather than after the embeddings have been paid for.
+
+### Changed
+
+- **A search that cannot read part of the corpus now fails instead of shrinking.**
+  An embedding failure per profile was pushed onto a list rendered as "Warnings", and
+  the results that came back looked exactly like a complete set whose missing half
+  ranked lower. An index holding points whose profile cannot be embedded is now an
+  error naming the profile. An index with no points is not an error, so the common
+  case is unaffected.
+- Changing provider clears the model, because model ids are namespaced per provider.
+  The previous code claimed to do this and did not, so switching to Voyage produced
+  `voyage` + `codestral-embed-2505`, which outranks the provider default.
+- The stored dimension of a previous index is no longer inherited across a provider
+  change. A 1536-wide Mistral index made any new model look 1536-dimensional, and the
+  inherited number was then persisted as though it had been measured, making the
+  mismatch self-consistent and invisible on every later run.
+- The `apiKeys` merge is derived from the incoming keys instead of a hand-written list
+  of provider names, which silently dropped any credential added to the shape but not
+  to the list.
+- The settings page reports which providers have credentials instead of deriving it
+  from two booleans, which reported "none configured" for a Gemini, Voyage,
+  OpenRouter, Ollama or OpenAI-compatible setup.
+- `indexing_config` and `settings test kilo` pass a target profile to the Kilo source
+  scan. Without it every source was reported as compatible, including ones that would
+  be refused a moment later.
+- Ollama and Voyage are sent `truncate` / `truncation: false`, so an over-long chunk
+  fails instead of being silently shortened into a valid-looking wrong vector.
+- Embedding results are placed by `data[].index` where the provider supplies it.
+  OpenAI documents the field rather than promising response order, and its own samples
+  fetch by it; a repeated or out-of-range index is now an error.
+- The catalog's `openrouter` Gemini entry names `google/gemini-embedding-2-preview`,
+  which is what Kilo writes. It said `google/gemini-embedding-001`, and since the
+  profile comparison is exact string equality the drift was invisible and made every
+  such index un-importable.
+
+### Fixed
+
+- The model field is cleared on a provider change in the TUI as well as the chat
+  command; both paths had the same stale-model bug.
+- `searchCode` and `getStatus` had no unit tests — only the e2e and cli scripts
+  exercised them, and those assert nothing. Both have tests now.
+
+### Documentation
+
+- The README states how far each provider has actually been tested. `mistral` has
+  touched its real API; `openrouter` reached the API and authenticated but was
+  refused by the key's own spend limit; `openai`, `gemini`, `voyage`,
+  `openai-compatible` and `ollama` have never contacted a real endpoint and are
+  covered only by fixtures written alongside their adapters. The same fact is
+  surfaced in the model chooser and in `indexing_config`, and a test pins the list so
+  the claim cannot quietly widen.
 
 ## [0.1.1] - 2026-10-14
 
+### Added
+
+- `src/lancedb-support.ts` reports up front that LanceDB ships one native binary per
+  platform and that there is no `lancedb-darwin-x64` build (there is no WASM
+  fallback), so the embedded store is unavailable on macOS on Intel — instead of
+  failing later with a module-not-found error.
+
 ### Changed
 
+- **The own index now defaults to embedded LanceDB** instead of inheriting Kilo's
+  `vectorStore`. Kilo's setting is read as `kiloVectorStore`: it still decides where
+  an *import* reads from, and it is still reported, but it no longer decides where
+  the own index writes to. Dropping that inheritance is reported as a warning rather
+  than applied silently, so an existing Qdrant own index is not discovered to be
+  empty.
 - npm releases are now automated: pushing a `v*` tag triggers a GitHub Actions
   workflow that publishes through npm trusted publishing (OIDC), with no
   long-lived token.
@@ -37,6 +191,7 @@ First public release.
 - Cross-platform installer (`install.mjs`, plus `install.sh` / `install.ps1`
   wrappers).
 
-[Unreleased]: https://github.com/Javierkaiser/opencode-indexing/compare/v0.1.1...HEAD
+[Unreleased]: https://github.com/Javierkaiser/opencode-indexing/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/Javierkaiser/opencode-indexing/releases/tag/v0.2.0
 [0.1.1]: https://github.com/Javierkaiser/opencode-indexing/releases/tag/v0.1.1
 [0.1.0]: https://github.com/Javierkaiser/opencode-indexing/releases/tag/v0.1.0

package/README.md

@@ -26,44 +26,83 @@ index is still installed.
 
 ## Install
 
-### From npm (recommended)
+### With the OpenCode CLI (recommended)
 
 ```bash
-npm install @javierkaiser/opencode-indexing
+opencode plugin add @javierkaiser/opencode-indexing
 ```
 
-Then reference it from `opencode.json`:
+That is the whole install. OpenCode downloads the package into its own cache and
+records it in the global `plugins` array, and it loads both halves of the plugin:
+the server side (`index.ts`, which registers the tools) and the terminal UI
+(`tui.ts`, which adds the prompt-footer indicator and the `/indexing` page).
+
+Update and remove it with the same command family:
+
+```bash
+opencode plugin check     # is there a newer version?
+opencode plugin update    # install it
+opencode plugin remove @javierkaiser/opencode-indexing
+```
+
+`plugin add` installs the plugin but knows nothing about its tools, so it does
+not write permission rules. Add this one to `opencode.json` to keep the host
+asking before the only tool that writes your settings file:
 
 ```jsonc
 {
-  "plugins": [
-    { "package": "@javierkaiser/opencode-indexing" }
+  "permissions": [
+    { "action": "indexing_config_set", "resource": "*", "effect": "ask" }
   ]
 }
 ```
 
-### Global installer (alternative)
+The other six tools only read. OpenCode's base policy already allows everything
+(`{ "action": "*", "resource": "*", "effect": "allow" }`), so they need no rule;
+write them out only if you want to tighten the set.
+
+### One command, permissions included
 
-The installer is cross-platform; it only needs Node >= 22.6.
+To skip the manual rule:
 
 ```bash
-# Linux / macOS
-./install.sh
+npx @javierkaiser/opencode-indexing
+```
 
-# Windows
-powershell -ExecutionPolicy Bypass -File .\install.ps1
+The installer runs `opencode plugin add`, writes the permission rules, and
+restarts the service. It picks its strategy from where it runs: from an installed
+copy — an npx cache, an unpacked tarball — it installs the published package;
+from a checkout it installs your working tree instead, so edits take effect on
+restart with no publish in between. It prints which one it chose.
 
-# Any platform (the wrappers just call this)
-node install.mjs
-```
+| Flag | Effect |
+| --- | --- |
+| `--package [spec]` | Install the published package (default spec `@latest`) |
+| `--local` | Copy this directory into the config dir (development) |
+| `--skip-deps` | `--local` only: copy the files, install no dependencies |
+| `--no-restart` | Do not restart the service |
 
-It copies the plugin to the global config directory (`$XDG_CONFIG_HOME/opencode` or `~/.config/opencode`, same path used on Windows), installs runtime deps (`ignore`, plus optional `@lancedb/lancedb@0.26.2` and the tree-sitter WASM packages), adds one `permissions` rule per tool to `opencode.json` (read tools `allow`, `indexing_config_set` `ask`), and restarts the OpenCode service.
+`install.sh` and `install.ps1` are thin wrappers over it; PowerShell also accepts
+`-Local`, `-Package`, `-Spec`, `-SkipDeps` and `-NoRestart`.
 
-Those are V2 rules — `{ "action": "indexing_search", "resource": "*", "effect": "allow" }` inside the `permissions` array. The older `permission` action map is a V1 shape that V2 ignores, so the installer removes any of its own entries from that map rather than leaving something that looks configured but grants nothing.
+`--package` deletes any copied plugin it finds: both sources define the same
+plugin id and the config directory is scanned automatically, so leaving both
+would load the plugin twice.
 
-Options: `--skip-deps` (copy files only), `--no-restart`. PowerShell accepts `-SkipDeps` / `-NoRestart` too.
+If no package manager is available, a `--local` install still copies the files
+and the plugin runs with reduced features (line chunker instead of tree-sitter,
+no LanceDB).
 
-If no package manager is available the copy still happens and the plugin runs with reduced features (line chunker instead of tree-sitter, no LanceDB).
+### Manual
+
+To manage the directory yourself, point `opencode.json` at it. Relative paths
+resolve from the config file containing the entry:
+
+```jsonc
+{
+  "plugins": ["./plugins/opencode-indexing"]
+}
+```
 
 ## How it works
 
@@ -176,11 +215,123 @@ Configurable from all three:
 - **Vector store** for the own index: Qdrant or LanceDB.
 - **Qdrant URL + API key** (local or remote) with connection test.
 - **LanceDB directory** with availability test.
-- **Embedding provider/model** (mistral, openai, ollama, openai-compatible, gemini, voyage, openrouter) and **API key**.
+- **Embedding provider/model** (mistral, openai, ollama, openai-compatible, gemini, voyage, openrouter) and **API key**. The model is chosen from a catalog rather than typed blind. The model is chosen from a catalog rather than typed blind.
 - **Import from Kilo** toggle with live source detection ("Qdrant ws-… (31,261 pts) · LanceDB …").
 - **Auto-refresh on search** toggle.
 - Build / refresh / import actions and a status summary.
 
+### Embedding providers
+
+All seven have an adapter. Every model below is in the catalog, which is what
+supplies the dimension, the score threshold and the input ceiling; a model that
+is not listed still works, and its dimension is probed from the provider on
+first use. See the next section for how far each one has actually been tested.
+
+| Provider | Default model | Also in the catalog | Input ceiling |
+| --- | --- | --- | --- |
+| `mistral` | `codestral-embed-2505` | `codestral-embed`, `mistral-embed` | 8,192 |
+| `openai` | `text-embedding-3-small` | `text-embedding-3-large`, `text-embedding-ada-002` | 8,192 |
+| `voyage` | `voyage-code-3` | **`voyage-code-4`**, `voyage-4`, `voyage-4-large`, `voyage-4-lite`, `voyage-3-lite` | **32,000** |
+| `gemini` | `gemini-embedding-001` | **`gemini-embedding-2`**, `text-embedding-004`, `embedding-001` | 2,048 (001) / 8,192 (2) |
+| `openrouter` | `openai/text-embedding-3-small` | `openai/text-embedding-3-large`, `google/gemini-embedding-2-preview` | 8,192 |
+| `ollama` | `nomic-embed-text` | `mxbai-embed-large`, `all-minilm`, `bge-m3` | 512 – 8,192 |
+| `openai-compatible` | none | whatever your endpoint serves | probed |
+
+#### Getting an API key (Mistral)
+
+Mistral is the default provider and the only one tested end to end here, so it is
+the shortest path to a working index. API access is enabled by default on a free
+account — **no credit card** — with a usage limit that is enough to index a
+repository.
+
+1. Create an account at <https://console.mistral.ai> if you do not have one.
+2. Open the **Studio** console and click **API Keys** in the left sidebar.
+3. Click **Create new key**: give it a name, set an expiration date, and choose the
+   connector access scope (*Shared connectors only* is enough for an API key).
+4. Click **Create new key** and **copy the key right away** — it is shown once and
+   cannot be retrieved later.
+
+Then hand it to the plugin. The simplest way keeps the key out of any file:
+
+```bash
+export KILO_INDEX_MISTRAL_KEY=...          # Linux / macOS
+# PowerShell: $env:KILO_INDEX_MISTRAL_KEY = "..."
+```
+
+Equivalently, set it from **`/indexing-config`** in any client or the `/indexing`
+page in the terminal (the field is masked), or through the plugin's options in
+`opencode.json` — see
+[Configuration](#configuration-optional-opencodejson). Settings resolve in the
+order *plugin options → `indexing.json` → environment → defaults*, so an
+environment variable is overridden by anything set in the file or in the options.
+
+To check the key before indexing:
+
+```bash
+curl -s https://api.mistral.ai/v1/models \
+  -H "Authorization: Bearer $KILO_INDEX_MISTRAL_KEY" | head -c 200
+```
+
+A list of models means it works; a `401` means the key is wrong or the header is
+missing.
+
+#### How far each provider has actually been tested
+
+Worth stating plainly, because "implemented" and "seen working" are not the
+same claim and only one of them is safe to rely on.
+
+| Provider | Against the real API | What that means |
+| --- | --- | --- |
+| `mistral` | **Yes** | Two inputs, 1536 dimensions, distinct vectors, ~680 ms |
+| `openrouter` | **Partly** | The request reached the API and authenticated; it was refused by the key's own $0 spend limit, so the response parser is unverified |
+| `ollama` | **No** | The local server was not running during the check |
+| `openai`, `gemini`, `voyage`, `openai-compatible` | **No** | No credentials available; only mock-based tests |
+
+The adapters for the last four were written from each provider's documentation
+and tested against fixtures written alongside them, which means the tests encode
+the same reading of the docs as the code. A wrong assumption about a field name
+or a response shape would pass every test and fail on first use.
+
+Three specific things this leaves open, in descending order of how quietly they
+would fail:
+
+- **Gemini's aggregation.** `gemini-embedding-2` returns a single aggregated
+  vector for a plain list of inputs, where `gemini-embedding-001` returns one per
+  input. The adapter wraps each input in its own `Content` to make both behave
+  the same way, and a test asserts the count — but the test is against a fixture,
+  not the model. If the wrapping is wrong, N inputs silently become one vector.
+- **Voyage's response shape.** It is treated as OpenAI-shaped
+  (`data[].embedding` with an `index`). This matches its published API reference
+  and was not confirmed against the service.
+- **Voyage and Ollama truncation.** Both are sent `truncation`/`truncate: false`
+  so that an over-long chunk fails instead of being silently shortened. If a
+  provider rejects that field rather than honouring it, indexing stops with an
+  error rather than degrading — which is the intended direction to fail, but it
+  is a guess about the field being accepted.
+
+If you use one of these providers and it fails, the error message names the
+provider and the endpoint; please report it, since this is the part of the
+plugin most likely to be wrong.
+
+Two things worth knowing before switching:
+
+- **Model ids are namespaced per provider.** Changing provider clears the model
+  so the new provider's default applies. A leftover id like
+  `codestral-embed-2505` under `provider: "voyage"` would be sent to Voyage and
+  rejected.
+- **Changing the model invalidates the own index.** A different model means a
+  different vector space, so the next build deletes the existing points and
+  re-embeds the whole workspace. `indexing_status` warns with the point count
+  *before* that happens. Re-embedding a repository is cheap — well under a
+  couple of dollars at any paid provider — but a large one takes a while.
+
+**Which models can import from Kilo.** Kilo's registry is mirrored entry for
+entry, because an import is accepted only when provider, model id and dimension
+match exactly. Models marked *ours* in the table above are not in Kilo's
+registry and can never be imported; `indexing_config` says which is which. If
+your Kilo index uses a different model, set the same provider and model here and
+the import costs nothing.
+
 Settings are stored in `~/.config/opencode/indexing.json` (plugin options → `indexing.json` → environment → defaults). Plugin options in `opencode.json` still work and take precedence. API keys are never written to `indexing.json` in plain text; set them through the environment or plugin options.
 
 `~/.config/kilo/kilo.jsonc` is still read, and still supplies the embedding provider, model, dimensions and API keys — and it still says where *Kilo* keeps its index. It no longer decides where this plugin's own index is written; that stopped being coupled so that the default could actually take effect for people running Kilo. See [Where each index lives](#where-each-index-lives).
@@ -190,16 +341,36 @@ Settings are stored in `~/.config/opencode/indexing.json` (plugin options → `i
 ```jsonc
 {
   "plugins": [
-    { "package": "./plugins/opencode-indexing", "options": {
+    { "package": "@javierkaiser/opencode-indexing", "options": {
       "vectorStore": "qdrant",
       "qdrantUrl": "http://localhost:6333",
       "importFromKilo": true,
-      "autoRefresh": false
+      "autoRefresh": false,
+      "ignoreKiloConfig": false
     }}
   ]
 }
 ```
 
+| Option | Meaning |
+| --- | --- |
+| `vectorStore` | `qdrant` or `lancedb` for the **own** index. Does not affect where an import reads from. |
+| `qdrantUrl` / `qdrantApiKey` | Qdrant connection |
+| `provider` / `model` / `dimension` | Embedding profile, overriding the resolved one |
+| `importFromKilo` | Import a compatible Kilo index on build instead of re-embedding |
+| `autoRefresh` | Refresh inline on search when few files changed |
+| `enabled` | Master switch: when false the own index is neither written nor refreshed |
+| `ignoreKiloConfig` | Stop reading `~/.config/kilo/kilo.jsonc` entirely |
+
+`~/.config/kilo/kilo.jsonc` is read as a fallback for the embedding provider,
+model, dimensions **and API keys**, which is what makes a free import possible:
+the same model has to be used to reuse Kilo's vectors. Set `ignoreKiloConfig` to
+`true` to cut that off, or override any single value with the options above, an
+environment variable, or `~/.config/opencode/indexing.json` — in that order of
+precedence, with plugin options winning.
+
+### Environment variables
+
 | Variable | Meaning |
 | --- | --- |
 | `KILO_INDEX_VECTOR_STORE` | `lancedb` (default) or `qdrant` for the own index |