@3sln/trove 0.0.19 → 0.0.20

package/README.md

@@ -356,6 +356,44 @@ create databases on demand and a scope key contains the user's id, so per-scope
 databases are not expressible here. Their tables sit side by side, which is weaker
 isolation than the file-per-scope a self-hosted run gets.
 
+#### Plugin storage
+
+A plugin with the `storage` capability gets an isolated SQLite database per scope — its
+own, and optionally one shared with the rest of its vendor's plugins. The scope key
+embeds the user *and* the plugin (`pstore:<principal>:plg:<pluginId>`), so it can never be
+pre-bound: **D1 cannot create a database on demand.**
+
+So a Worker deployment has two options, and they are not equivalent.
+
+**A Durable Object per scope** — bind `PLUGIN_STORE` to the `TrovePluginStore` class. A
+Durable Object is addressable by name, so each `(user, plugin)` becomes its own object with
+its own SQLite database, created on first use. The isolation is structural.
+
+```toml
+[[durable_objects.bindings]]
+name = "PLUGIN_STORE"
+class_name = "TrovePluginStore"
+
+[[migrations]]
+tag = "v1"
+new_sqlite_classes = ["TroveTasks", "TrovePluginStore"]
+```
+
+and export it beside the fetch handler:
+
+```js
+export { default, TroveTasks, TrovePluginStore } from '@3sln/trove/server/adapters/worker.js';
+```
+
+**One shared D1** — bind `PLUGIN_DB` and every plugin's tables for every user live in it
+side by side. The keys stay distinct; the boundary is a naming convention rather than a
+wall. It works, and it is what you get without the Durable Object, and it is worth knowing
+which of the two you have.
+
+Core stores — metadata, KV, install records, the keyword index — stay on D1 either way.
+They are one per deployment, so they can simply be bound, and routing the whole drive
+through one single-threaded object would be a bottleneck rather than an isolation win.
+
 #### Work that outlives a request
 
 A Worker isolate is not a server: it may be discarded as soon as the response resolves,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@3sln/trove",
-  "version": "0.0.19",
+  "version": "0.0.20",
   "type": "module",
   "description": "Trove — a self-hostable, plugin-extensible Google Drive. Semantic search, pluggable storage (S3 / filesystem / NAS), and a VS Code-style contribution system with sandboxed plugins.",
   "repository": {

package/packages/core/src/index.js

@@ -82,6 +82,9 @@ export {
 export { SqliteDatabase, SqliteProvider, LocalSqliteProvider, assertSafePluginSql, stripSqlLiterals } from './sqlite.js';
 // SQLite on Cloudflare D1, so a Worker deployment has a metadata store that exists.
 export { D1SqliteProvider } from './sqlite-d1.js';
+// A Durable Object per plugin scope — what D1 cannot do, because a scope key embeds the
+// runtime principal and D1 cannot create a database on demand. See sqlite-do.js.
+export { DurableObjectSqliteProvider, createPluginStore, isPluginScope } from './sqlite-do.js';
 
 // Identity (BYO IdP — Cloudflare Access / Zero Trust / a proxy).
 export {

package/packages/core/src/sqlite-d1.js

@@ -144,6 +144,12 @@ export class D1SqliteProvider extends SqliteProvider {
     // the local provider's file-per-scope — the tables live side by side — but the keys
     // are still distinct per (user, plugin) and it is the strongest thing D1's model
     // allows. Stated here rather than discovered.
+    // ONE BINDING FOR EVERY PLUGIN SCOPE, and it is a compromise rather than a design.
+    // The tables live side by side: distinct names per (user, plugin), no boundary
+    // between them. `DurableObjectSqliteProvider` is the one that gets this right — a DO
+    // is addressable by name, so each scope is its own object with its own database, and
+    // creating one on demand is the thing D1 cannot do. Prefer it where a Durable Object
+    // is available; this stays for deployments that have only D1.
     if (this.pluginStore) return this.pluginStore;
     // Plugin scopes are an isolation boundary. Handing back the main database would
     // put a plugin's tables next to the drive's metadata, which is precisely what the

package/packages/core/src/sqlite-do.js

@@ -0,0 +1,182 @@
+// A Durable Object per scope — plugin storage on Workers, done properly.
+//
+// `D1SqliteProvider` cannot do this, and its own comment says why: a plugin scope key
+// embeds the runtime principal (`pstore:alice@x.com:plg:acme/notes`), so it can never be
+// pre-bound, and D1 cannot create a database on demand. What shipped instead was one
+// nominated binding holding every plugin's tables for every user side by side — weaker
+// isolation than the local provider's file-per-scope, kept only because it was the
+// strongest thing D1's model allowed. The keys stayed distinct; the boundary was a naming
+// convention.
+//
+// A Durable Object is addressable BY NAME, which is exactly the missing primitive:
+// `obtain({ key })` becomes a lookup, the object is created on first use, and each one
+// carries its own SQLite storage. The isolation is structural rather than a prefix.
+//
+// The seam does not change. This is a third sibling next to the local and D1 providers,
+// returning the same `SqliteDatabase` surface, and a deployment picks it by declaring the
+// binding.
+//
+// WHAT A DO COSTS, so nobody is surprised: it is single-threaded and lives in one place,
+// so every query for a scope routes there. For per-user plugin state — a listening
+// position, a reader's notes — that is exactly right. For something read from everywhere
+// at once it would be a bottleneck, and that is a different problem than this one.
+
+import { TroveError } from './errors.js';
+import { SqliteDatabase, SqliteProvider } from './sqlite.js';
+
+/**
+ * One scope's database, reached through its Durable Object.
+ *
+ * Every call is a `fetch` at the stub. Batched where the interface allows it, because a
+ * round trip per statement to a single-threaded object is the one thing that would make
+ * this slower than the shared table it replaces.
+ */
+class DurableObjectDatabase extends SqliteDatabase {
+  constructor(stub, key) {
+    super();
+    this.stub = stub;
+    this.key = key;
+  }
+
+  async #call(op, sql, params) {
+    const res = await this.stub.fetch('https://trove.store/sql', {
+      method: 'POST',
+      headers: { 'content-type': 'application/json' },
+      body: JSON.stringify({ op, sql, params }),
+    });
+    const body = await res.json().catch(() => ({ error: `store returned ${res.status}` }));
+    // The DO reports a SQL error as data rather than as a 500, so the message survives the
+    // trip — a plugin that wrote bad SQL should read its own error, not "500".
+    if (!res.ok || body?.error) throw TroveError.invalid(body?.error || `store returned ${res.status}`);
+    return body.result;
+  }
+
+  async exec(sql) { await this.#call('exec', sql); }
+  async run(sql, ...params) { return this.#call('run', sql, params); }
+  async get(sql, ...params) { return this.#call('get', sql, params); }
+  async all(sql, ...params) { return this.#call('all', sql, params); }
+  async batch(statements) { return this.#call('batch', null, statements); }
+}
+
+export class DurableObjectSqliteProvider extends SqliteProvider {
+  /**
+   * @param {object} opts
+   * @param {{idFromName: Function, get: Function}} opts.namespace a DurableObjectNamespace
+   * @param {SqliteProvider} [opts.core] where the CORE keys go — metadata, kv, installs,
+   *   search. Those are one-per-deployment and already have a home; this provider exists
+   *   for the keys that are one-per-(user, plugin).
+   */
+  constructor({ namespace, core = null } = {}) {
+    super();
+    if (!namespace?.idFromName) throw TroveError.invalid('DurableObjectSqliteProvider requires a Durable Object namespace');
+    this.namespace = namespace;
+    this.core = core;
+    this._dbs = new Map();
+  }
+
+  // A Durable Object's storage survives the isolate, the deploy and the restart. That is
+  // the whole question this flag answers.
+  get durable() { return true; }
+
+  async obtain({ key }) {
+    if (!key) throw TroveError.invalid('a scope key is required');
+    // Core keys keep whatever the deployment already gave them. Routing the metadata
+    // store through a DO would put the whole drive behind one single-threaded object,
+    // which is the bottleneck this file's header warns about.
+    if (this.core && !isPluginScope(key)) return this.core.obtain({ key });
+
+    let db = this._dbs.get(key);
+    if (!db) {
+      // NAMED BY THE SCOPE KEY, which is what makes the isolation structural: two scopes
+      // are two objects with two databases, not two prefixes in one.
+      const stub = this.namespace.get(this.namespace.idFromName(key));
+      db = new DurableObjectDatabase(stub, key);
+      this._dbs.set(key, db);
+    }
+    return db;
+  }
+
+  async drop({ key }) {
+    if (this.core && !isPluginScope(key)) return this.core.drop({ key });
+    const db = this._dbs.get(key);
+    this._dbs.delete(key);
+    // Ask the object to empty itself. Its storage outlives this process, so forgetting the
+    // handle here would leave the data behind — which for an uninstalled plugin is the
+    // difference between "removed" and "invisible".
+    if (db) await db.stub.fetch('https://trove.store/drop', { method: 'POST' }).catch(() => {});
+  }
+
+  async close() { this._dbs.clear(); }
+}
+
+/** Plugin scopes are the ones that cannot be pre-bound — see the header. */
+export function isPluginScope(key) {
+  return typeof key === 'string' && key.startsWith('pstore:');
+}
+
+/**
+ * The Durable Object class itself.
+ *
+ * Deliberately thin: it owns `state.storage.sql` and answers the verbs above. Everything
+ * that decides WHAT may run — `assertSafePluginSql`, the scope gate — already happened on
+ * the way in, and repeating it here would put two authorities on one question.
+ *
+ * Export it from the Worker entry and declare it in wrangler.toml with a
+ * `new_sqlite_classes` migration, the same way TroveTasks is declared.
+ */
+export function createPluginStore() {
+  return class TrovePluginStore {
+    constructor(state) {
+      this.state = state;
+      this.sql = state.storage.sql;
+    }
+
+    async fetch(request) {
+      const url = new URL(request.url);
+      if (url.pathname === '/drop') {
+        // `deleteAll` takes the tables with it, which is the point: an uninstalled
+        // plugin's data should stop existing rather than stop being addressed.
+        await this.state.storage.deleteAll();
+        return json({ result: { ok: true } });
+      }
+      const { op, sql, params } = await request.json().catch(() => ({}));
+      try {
+        return json({ result: this.#run(op, sql, params) });
+      } catch (err) {
+        // As DATA, not a 500: the message is the plugin author's own SQL error and is the
+        // only useful thing anyone will get back.
+        return json({ error: err?.message || String(err) });
+      }
+    }
+
+    #run(op, sql, params) {
+      const bind = (s, p) => [...this.sql.exec(s, ...(p || []))];
+      switch (op) {
+        case 'exec':
+          // Multi-statement schema setup. `exec` takes one statement at a time here, so a
+          // schema arrives split — which is also what makes it safe to run per statement.
+          for (const s of String(sql).split(';').map((x) => x.trim()).filter(Boolean)) this.sql.exec(s);
+          return { ok: true };
+        case 'run': {
+          bind(sql, params);
+          return { changes: this.sql.rowsWritten, lastInsertRowid: null };
+        }
+        case 'get': return bind(sql, params)[0] ?? null;
+        case 'all': return bind(sql, params);
+        case 'batch': {
+          // ATOMIC. A DO's storage transaction is what makes a batch mean what it says;
+          // running the statements loose would leave a half-applied schema behind on the
+          // first bad one.
+          const out = [];
+          this.state.storage.transactionSync(() => {
+            for (const st of params || []) out.push(bind(st.sql, st.params));
+          });
+          return out;
+        }
+        default: throw new Error(`Unknown store op "${op}"`);
+      }
+    }
+  };
+}
+
+const json = (body) => new Response(JSON.stringify(body), { headers: { 'content-type': 'application/json' } });

package/packages/server/src/adapters/worker.js

@@ -15,7 +15,7 @@
 //   [[durable_objects.bindings]] name = "TASKS" class_name = "TroveTasks"
 //                                            -> owns scans and reindexes (see below)
 
-import { D1SqliteProvider } from '@3sln/trove/core';
+import { D1SqliteProvider, DurableObjectSqliteProvider, createPluginStore } from '@3sln/trove/core';
 import { createServer, configFromEnv } from '../index.js';
 import { createTaskHost, remoteBackground } from './worker-tasks.js';
 
@@ -42,14 +42,24 @@ async function getServer(env, buildVfs, { delegate = true } = {}) {
   // Worker falls back to in-memory everything, which looks like it works right up until
   // the isolate is recycled and the drive is empty. Bind `DB` and it persists.
   if (env.DB && !config.sqlite) {
-    config.sqlite = new D1SqliteProvider({
+    const d1 = new D1SqliteProvider({
       db: env.DB,
-      // Plugin storage. `scopes: { plugins: … }` named a key that is already a CORE key
-      // (the install-record store), so the binding was silently ignored and every
-      // /api/plugins/:id/sql call was a 501 — the real keys look like
-      // `pstore:<user>:plg:<pluginId>` and cannot be pre-bound at all.
+      // Plugin storage of last resort. `scopes: { plugins: … }` named a key that is
+      // already a CORE key (the install-record store), so the binding was silently
+      // ignored and every /api/plugins/:id/sql call was a 501 — the real keys look like
+      // `pstore:<user>:plg:<pluginId>` and cannot be pre-bound at all. One binding then
+      // holds every plugin's tables side by side, which is a compromise, not a design.
       pluginStore: env.PLUGIN_DB || null,
     });
+    // A DURABLE OBJECT PER SCOPE where one is available, which is what plugin storage
+    // actually wants: a DO is addressable by name, so each (user, plugin) is its own
+    // object with its own SQLite database, created on demand. That is the primitive D1
+    // does not have. Core stores — metadata, kv, installs, search — stay on D1: routing
+    // the whole drive through one single-threaded object would be a bottleneck, and they
+    // are one-per-deployment so they can simply be bound.
+    config.sqlite = env.PLUGIN_STORE
+      ? new DurableObjectSqliteProvider({ namespace: env.PLUGIN_STORE, core: d1 })
+      : d1;
     config.metadata = { driver: 'sqlite' };
   }
   // Cloudflare Vectorize binding → first-class vector store (no REST creds needed).
@@ -162,4 +172,13 @@ export default {
  */
 export const TroveTasks = createTaskHost((env) => getServer(env, undefined, { delegate: false }));
 
+/**
+ * One SQLite database per plugin scope, addressed by name.
+ *
+ * Export it alongside TroveTasks and declare it in wrangler.toml with a
+ * `new_sqlite_classes` migration; `PLUGIN_STORE` is then picked up above. Without the
+ * binding a drive falls back to the shared D1 store, which still works and isolates less.
+ */
+export const TrovePluginStore = createPluginStore();
+
 export { getServer };