bonecode 1.0.0

package/src/engine/sync/README.md

@@ -0,0 +1,179 @@
+tl;dr All of these APIs work, are properly type-checked, and are sync events are backwards compatible with `Bus`:
+
+```ts
+// The schema from `Updated` typechecks the object correctly
+SyncEvent.run(Updated, { sessionID: id, info: { title: "foo" } })
+
+// `subscribeAll` passes a generic sync event
+SyncEvent.subscribeAll((event) => {
+  // These will be type-checked correctly
+  event.id
+  event.seq
+  // This will be unknown because we are listening for all events,
+  // and this API is only used to record them
+  event.data
+})
+
+// This works, but you shouldn't publish sync event like this (should fail in the future)
+Bus.publish(Updated, { sessionID: id, info: { title: "foo" } })
+
+// Update event is fully type-checked
+Bus.subscribe(Updated, (event) => event.properties.info.title)
+
+// Update event is fully type-checked
+client.subscribe("session.updated", (evt) => evt.properties.info.title)
+```
+
+# Goal
+
+## Syncing with only one writer
+
+This system defines a basic event sourcing system for session replayability. The goal is to allow for one device to control and modify the session, and allow multiple other devices to "sync" session data. The sync works by getting a log of events to replay and replaying them locally.
+
+Because only one device is allowed to write, we don't need any kind of sophisticated distributed system clocks or causal ordering. We implement total ordering with a simple sequence id (a number) and increment it by one every time we generate an event.
+
+## Bus event integration and backwards compatibility
+
+This initial implementation aims to be fully backwards compatible. We should be able to land this without any visible changes to the user.
+
+An existing `Bus` abstraction to send events already exists. We already send events like `session.created` through the system. We should not duplicate this.
+
+The difference in event sourcing is events are sent _before_ the mutation happens, and "projectors" handle the effects and perform the mutations. This difference is subtle, and a necessary change for syncing to work.
+
+So the goal is:
+
+- Introduce a new syncing abstraction to handle event sourcing and projectors
+- Seamlessly integrate these new events into the same existing `Bus` abstraction
+- Maintain full backwards compatibility to reduce risk
+
+## My approach
+
+This directory introduces a new abstraction: `SyncEvent`. This handles all of the event sourcing.
+
+There are now "sync events" which are different than "bus events". Bus events are defined like this:
+
+```ts
+const Diff = BusEvent.define(
+  "session.diff",
+  Schema.Struct({
+    sessionID: SessionID,
+    diff: Schema.Array(Snapshot.FileDiff),
+  }),
+)
+```
+
+You can do `Bus.publish(Diff, { ... })` to push these events, and `Bus.subscribe(Diff, handler)` to listen to them.
+
+Sync events are a lower-level abstraction which are similar, but also handle the requirements for recording and replaying. Defining them looks like this:
+
+```ts
+const Created = SyncEvent.define({
+  type: "session.created",
+  version: 1,
+  aggregate: "sessionID",
+  schema: Schema.Struct({
+    sessionID: SessionID,
+    info: Info,
+  }),
+})
+```
+
+Not too different, except they track a version and an "aggregate" field (will explain that later).
+
+You do this to run an event, which is kind of like `Bus.publish` except that it runs through the event sourcing system:
+
+```
+SyncEvent.run(Created, { ... })
+```
+
+The data passed as the second argument is properly type-checked based on the schema defined in `Created`.
+
+Importantly, **sync events automatically re-publish as bus events**. This makes them backwards compatible, and allows the `Bus` to still be the single abstraction that the system uses to listen for individual events.
+
+**We have upgraded many of the session events to be sync events** (all of the ones that mutate the db). Sync and bus events are largely compatible. Here are the differences:
+
+### Event shape
+
+- The shape of the events are slightly different. A sync event has the `type`, `id`, `seq`, `aggregateID`, and `data` fields. A bus event has the `type` and `properties` fields. `data` and `properties` are largely the same thing. This conversion is automatically handled when the sync system re-published the event through the bus.
+
+The reason for this is because sync events need to track more information. I chose not to copy the `properties` naming to more clearly disambiguate the event types.
+
+### Event flow
+
+There is no way to subscribe to individual sync events in `SyncEvent`. You can use `subscribeAll` to receive _all_ of the events, which is needed for clients that want to record them.
+
+To listen for individual events, use `Bus.subscribe`. You can pass in a sync event definition to it: `Bus.subscribe(Created, handler)`. This is fully supported.
+
+You should never "publish" a sync event however: `Bus.publish(Created, ...)`. I would like to force this to be a type error in the future. You should never be touching the db directly, and should not be manually handling these events.
+
+### Backwards compatibility
+
+The system install projectors in `server/projectors.js`. It calls `SyncEvent.init` to do this. It also installs a hook for dynamically converting an event at runtime (`convertEvent`).
+
+This allows you to "reshape" an event from the sync system before it's published to the bus. This should be avoided, but might be necessary for temporary backwards compat.
+
+The only time we use this is the `session.updated` event. Previously this event contained the entire session object. The sync event only contains the fields updated. We convert the event to contain the full object for backwards compatibility (but ideally we'd remove this).
+
+It's very important that types are correct when working with events. Event definitions have a `schema` which carries the definition of the event shape. Examples:
+
+```ts
+// The schema from `Updated` typechecks the object correctly
+SyncEvent.run(Updated, { sessionID: id, info: { title: "foo" } })
+
+// `subscribeAll` passes a generic sync event
+SyncEvent.subscribeAll((event) => {
+  // These will be type-checked correctly
+  event.id
+  event.seq
+  // This will be unknown because we are listening for all events,
+  // and this API is only used to record them
+  event.data
+})
+
+// This works, but you shouldn't publish sync event like this (should fail in the future)
+Bus.publish(Updated, { sessionID: id, info: { title: "foo" } })
+
+// Update event is fully type-checked
+Bus.subscribe(Updated, (event) => event.properties.info.title)
+
+// Update event is fully type-checked
+client.subscribe("session.updated", (evt) => evt.properties.info.title)
+```
+
+The last two examples look similar to `SyncEvent.run`, but they were the cause of a lot of grief. Those are existing APIs that we can't break, but we are passing in the new sync event definitions to these APIs, which sometimes have a different event shape.
+
+I previously mentioned the runtime conversion of events, but we still need to the types to work! To do that, the `define` API supports an optional `busSchema` prop to give it the schema for backwards compatibility. For example this is the full definition of `Session.Update`:
+
+```ts
+const Update = SyncEvent.define({
+  type: "session.updated",
+  version: 1,
+  aggregate: "sessionID",
+  schema: Schema.Struct({
+    sessionID: SessionID,
+    info: partialSchema(Info),
+  }),
+  busSchema: Schema.Struct({
+    sessionID: SessionID,
+    info: Info,
+  }),
+})
+```
+
+_Important_: the conversion done in `convertEvent` is not automatically type-checked with `busSchema`. It's very important they match, but because we need this at type-checking time this needs to live here.
+
+Internally, the way this works is `busSchema` is stored on a `properties` field which is what the bus system expects. Doing this made everything with `Bus` "just work". This is why you can pass a sync event to the bus APIs.
+
+_Alternatives_
+
+These are some other paths I explored:
+
+- Providing a way to subscribe to individual sync events, and change all the instances of `Bus.subscribe` in our code to it. Then you are directly only working with sync events always.
+  - Two big problems. First, `Bus` is instance-scoped, and we'd need to make the sync event system instance-scoped too for backwards compat. If we didn't, those listeners would get calls for events they weren't expecting.
+  - Second, we can't change consumers of our SDK. So they still have to use the old events, and we might as well stick with them for consistency
+- Directly add sync event support to bus system
+  - I explored adding sync events to the bus, but due to backwards compat, it only made it more complicated (still need to support both shapes)
+- I explored a `convertSchema` function to convert the event schema at runtime so we didn't need `busSchema`
+  - Fatal flaw: we need type-checking done earlier. We can't do this at run-time. This worked for consumers of our SDK (because it gets generated TS types from the converted schema) but breaks for our internal usage of `Bus.subscribe` calls
+
+I explored many other permutations of the above solutions. What we have today I think is the best balance of backwards compatibility while opening a path forward for the new events.

package/src/engine/sync/event.sql.ts

@@ -0,0 +1,17 @@
+import { sqliteTable, text, integer } from "drizzle-orm/sqlite-core"
+
+export const EventSequenceTable = sqliteTable("event_sequence", {
+  aggregate_id: text().notNull().primaryKey(),
+  seq: integer().notNull(),
+  owner_id: text(),
+})
+
+export const EventTable = sqliteTable("event", {
+  id: text().primaryKey(),
+  aggregate_id: text()
+    .notNull()
+    .references(() => EventSequenceTable.aggregate_id, { onDelete: "cascade" }),
+  seq: integer().notNull(),
+  type: text().notNull(),
+  data: text({ mode: "json" }).$type<Record<string, unknown>>().notNull(),
+})

package/src/engine/sync/index.ts

@@ -0,0 +1,410 @@
+// Legacy sync event system. It should stay unaware of core EventV2 execution;
+// the only temporary V2 coupling here is exposing versioned core event schemas
+// in effectPayloads() so existing HTTP/SDK schema generation remains stable.
+// Remove that registry read when event schemas are generated from core directly.
+import { Database } from "@/storage/db"
+import { eq } from "drizzle-orm"
+import { GlobalBus } from "@/bus/global"
+import { Bus as ProjectBus } from "@/bus"
+import { BusEvent } from "@/bus/bus-event"
+import type { InstanceContext } from "@/project/instance-context"
+import { EventSequenceTable, EventTable } from "./event.sql"
+import type { WorkspaceID } from "@/control-plane/schema"
+import { EventID } from "./schema"
+import { Context, Effect, Layer, Schema as EffectSchema } from "effect"
+import type { DeepMutable } from "@opencode-ai/core/schema"
+import { EventV2 } from "@opencode-ai/core/event"
+import { serviceUse } from "@/effect/service-use"
+import { InstanceState } from "@/effect/instance-state"
+import { RuntimeFlags } from "@/effect/runtime-flags"
+
+// Keep `Event["data"]` mutable because projectors mutate the persisted shape
+// when writing to the database. Bus payloads (`Properties`) stay readonly —
+// subscribers only read.
+
+export type Definition<
+  Type extends string = string,
+  Schema extends EffectSchema.Top = EffectSchema.Top,
+  BusSchema extends EffectSchema.Top = Schema,
+> = {
+  type: Type
+  version: number
+  aggregate: string
+  schema: Schema
+  // Bus event payload schema. Defaults to `schema` unless `busSchema` was
+  // passed at definition time (see `session.updated`, whose projector
+  // expands the persisted data to a `{ sessionID, info }` bus payload).
+  properties: BusSchema
+}
+
+export type Event<Def extends Definition = Definition> = {
+  id: string
+  seq: number
+  aggregateID: string
+  data: DeepMutable<EffectSchema.Schema.Type<Def["schema"]>>
+}
+
+export type Properties<Def extends Definition = Definition> = EffectSchema.Schema.Type<Def["properties"]>
+
+export type SerializedEvent<Def extends Definition = Definition> = Event<Def> & { type: string }
+
+type ProjectorFunc = (db: Database.TxOrDb, data: unknown, event: Event) => void
+type ConvertEvent = (type: string, data: Event["data"]) => unknown | Promise<unknown>
+type PublishContext = {
+  instance?: InstanceContext
+  workspace?: WorkspaceID
+}
+
+export interface Interface {
+  readonly run: <Def extends Definition>(
+    def: Def,
+    data: Event<Def>["data"],
+    options?: { publish?: boolean },
+  ) => Effect.Effect<void>
+  readonly replay: (event: SerializedEvent, options?: { publish: boolean; ownerID?: string }) => Effect.Effect<void>
+  readonly replayAll: (
+    events: SerializedEvent[],
+    options?: { publish: boolean; ownerID?: string },
+  ) => Effect.Effect<string | undefined>
+  readonly remove: (aggregateID: string) => Effect.Effect<void>
+  readonly claim: (aggregateID: string, ownerID: string) => Effect.Effect<void>
+}
+
+export class Service extends Context.Service<Service, Interface>()("@opencode/SyncEvent") {}
+
+export const layer = Layer.effect(Service)(
+  Effect.gen(function* () {
+    const flags = yield* RuntimeFlags.Service
+
+    const replay: Interface["replay"] = Effect.fn("SyncEvent.replay")(function* (event, options) {
+      const def = registry.get(event.type)
+      if (!def) {
+        throw new Error(`Unknown event type: ${event.type}`)
+      }
+
+      const row = Database.use((db) =>
+        db
+          .select({ seq: EventSequenceTable.seq, ownerID: EventSequenceTable.owner_id })
+          .from(EventSequenceTable)
+          .where(eq(EventSequenceTable.aggregate_id, event.aggregateID))
+          .get(),
+      )
+
+      const latest = row?.seq ?? -1
+      if (event.seq <= latest) return
+
+      if (row?.ownerID && row.ownerID !== options?.ownerID) {
+        return
+      }
+
+      const expected = latest + 1
+      if (event.seq !== expected) {
+        throw new Error(
+          `Sequence mismatch for aggregate "${event.aggregateID}": expected ${expected}, got ${event.seq}`,
+        )
+      }
+
+      const publish = !!options?.publish
+      const context = publish
+        ? {
+            instance: yield* InstanceState.context,
+            workspace: yield* InstanceState.workspaceID,
+          }
+        : undefined
+      process(def, event, {
+        publish,
+        context,
+        ownerID: options?.ownerID,
+        experimentalWorkspaces: flags.experimentalWorkspaces,
+      })
+    })
+
+    const replayAll: Interface["replayAll"] = Effect.fn("SyncEvent.replayAll")(function* (events, options) {
+      const source = events[0]?.aggregateID
+      if (!source) return undefined
+      if (events.some((item) => item.aggregateID !== source)) {
+        throw new Error("Replay events must belong to the same session")
+      }
+      const start = events[0].seq
+      for (const [i, item] of events.entries()) {
+        const seq = start + i
+        if (item.seq !== seq) {
+          throw new Error(`Replay sequence mismatch at index ${i}: expected ${seq}, got ${item.seq}`)
+        }
+      }
+      for (const item of events) {
+        yield* replay(item, options)
+      }
+      return source
+    })
+
+    const run: Interface["run"] = Effect.fn("SyncEvent.run")(function* (def, data, options) {
+      const agg = (data as Record<string, string>)[def.aggregate]
+      // This should never happen: we've enforced it via typescript in
+      // the definition
+      if (agg == null) {
+        throw new Error(`SyncEvent.run: "${def.aggregate}" required but not found: ${JSON.stringify(data)}`)
+      }
+
+      if (def.version !== versions.get(def.type)) {
+        throw new Error(`SyncEvent.run: running old versions of events is not allowed: ${def.type}`)
+      }
+
+      const { publish = true } = options || {}
+      const context = publish
+        ? {
+            instance: yield* InstanceState.context,
+            workspace: yield* InstanceState.workspaceID,
+          }
+        : undefined
+
+      // Note that this is an "immediate" transaction which is critical.
+      // We need to make sure we can safely read and write with nothing
+      // else changing the data from under us
+      Database.transaction(
+        (tx) => {
+          const id = EventID.ascending()
+          const row = tx
+            .select({ seq: EventSequenceTable.seq })
+            .from(EventSequenceTable)
+            .where(eq(EventSequenceTable.aggregate_id, agg))
+            .get()
+          const seq = row?.seq != null ? row.seq + 1 : 0
+
+          const event = { id, seq, aggregateID: agg, data }
+          process(def, event, { publish, context, experimentalWorkspaces: flags.experimentalWorkspaces })
+        },
+        {
+          behavior: "immediate",
+        },
+      )
+    })
+
+    const remove: Interface["remove"] = Effect.fn("SyncEvent.remove")(function* (aggregateID) {
+      Database.transaction((tx) => {
+        tx.delete(EventSequenceTable).where(eq(EventSequenceTable.aggregate_id, aggregateID)).run()
+        tx.delete(EventTable).where(eq(EventTable.aggregate_id, aggregateID)).run()
+      })
+    })
+
+    const claim: Interface["claim"] = Effect.fn("SyncEvent.claim")((aggregateID, ownerID) =>
+      Effect.sync(() =>
+        Database.use((db) =>
+          db
+            .update(EventSequenceTable)
+            .set({ owner_id: ownerID })
+            .where(eq(EventSequenceTable.aggregate_id, aggregateID))
+            .run(),
+        ),
+      ),
+    )
+
+    return Service.of({
+      run,
+      replay,
+      replayAll,
+      remove,
+      claim,
+    })
+  }),
+)
+
+export const defaultLayer = layer.pipe(Layer.provide(RuntimeFlags.defaultLayer))
+
+export const use = serviceUse(Service)
+
+export const registry = new Map<string, Definition>()
+let projectors: Map<string, ProjectorFunc> | undefined
+const versions = new Map<string, number>()
+let frozen = false
+let convertEvent: ConvertEvent
+
+export function reset() {
+  frozen = false
+  projectors = undefined
+  convertEvent = (_, data) => data
+}
+
+export function init(input: { projectors: Array<[Definition, ProjectorFunc]>; convertEvent?: ConvertEvent }) {
+  projectors = new Map(input.projectors.map(([def, func]) => [versionedType(def.type, def.version), func]))
+  for (let entry of EventV2.registry.values()) {
+    if (!entry.version || !entry.aggregate) continue
+    register({
+      type: entry.type,
+      version: entry.version,
+      aggregate: entry.aggregate,
+      properties: entry.data,
+      schema: entry.data,
+    })
+  }
+
+  // Install all the latest event defs to the bus. We only ever emit
+  // latest versions from code, and keep around old versions for
+  // replaying. Replaying does not go through the bus, and it
+  // simplifies the bus to only use unversioned latest events
+  for (let [type, version] of versions.entries()) {
+    let def = registry.get(versionedType(type, version))!
+    BusEvent.define(def.type, def.properties)
+  }
+
+  // Freeze the system so it clearly errors if events are defined
+  // after `init` which would cause bugs
+  frozen = true
+  convertEvent = input.convertEvent ?? ((_, data) => data)
+}
+
+export function versionedType<A extends string>(type: A): A
+export function versionedType<A extends string, B extends number>(type: A, version: B): `${A}/${B}`
+export function versionedType(type: string, version?: number) {
+  return version ? `${type}.${version}` : type
+}
+
+export function define<
+  Type extends string,
+  Agg extends string,
+  Schema extends EffectSchema.Top,
+  BusSchema extends EffectSchema.Top = Schema,
+>(input: {
+  type: Type
+  version: number
+  aggregate: Agg
+  schema: Schema
+  busSchema?: BusSchema
+}): Definition<Type, Schema, BusSchema> {
+  if (frozen) {
+    throw new Error("Error defining sync event: sync system has been frozen")
+  }
+
+  const def = {
+    type: input.type,
+    version: input.version,
+    aggregate: input.aggregate,
+    schema: input.schema,
+    properties: (input.busSchema ?? input.schema) as BusSchema,
+  }
+
+  register(def)
+
+  return def
+}
+
+export function project<Def extends Definition>(
+  def: Def,
+  func: (db: Database.TxOrDb, data: Event<Def>["data"], event: Event<Def>) => void,
+): [Definition, ProjectorFunc] {
+  return [def, func as ProjectorFunc]
+}
+
+function register(def: Definition) {
+  versions.set(def.type, Math.max(def.version, versions.get(def.type) || 0))
+  registry.set(versionedType(def.type, def.version), def)
+}
+
+function process<Def extends Definition>(
+  def: Def,
+  event: Event<Def>,
+  options: { publish: boolean; context?: PublishContext; ownerID?: string; experimentalWorkspaces: boolean },
+) {
+  if (projectors == null) {
+    throw new Error("No projectors available. Call `SyncEvent.init` to install projectors")
+  }
+
+  const projector = projectors.get(versionedType(def.type, def.version))
+  if (!projector) {
+    if (!def.type.includes("next")) throw new Error(`Projector not found for event: ${def.type}`)
+    return
+  }
+
+  Database.transaction((tx) => {
+    projector(tx, event.data, event)
+
+    if (options.experimentalWorkspaces) {
+      tx.insert(EventSequenceTable)
+        .values({
+          aggregate_id: event.aggregateID,
+          seq: event.seq,
+          owner_id: options?.ownerID,
+        })
+        .onConflictDoUpdate({
+          target: EventSequenceTable.aggregate_id,
+          set: { seq: event.seq },
+        })
+        .run()
+      tx.insert(EventTable)
+        .values({
+          id: event.id,
+          seq: event.seq,
+          aggregate_id: event.aggregateID,
+          type: versionedType(def.type, def.version),
+          data: event.data as Record<string, unknown>,
+        })
+        .run()
+    }
+
+    Database.effect(() => {
+      if (options?.publish) {
+        if (!options.context?.instance) {
+          throw new Error("SyncEvent.process: publish requires instance context")
+        }
+
+        const result = convertEvent(def.type, event.data)
+        const publish = (data: unknown) => ProjectBus.publish(def, data as Properties<Def>, { id: event.id })
+        if (result instanceof Promise) {
+          void result.then(publish)
+        } else {
+          void publish(result)
+        }
+
+        GlobalBus.emit("event", {
+          directory: options.context.instance.directory,
+          project: options.context.instance.project.id,
+          workspace: options.context.workspace,
+          payload: {
+            type: "sync",
+            syncEvent: {
+              type: versionedType(def.type, def.version),
+              ...event,
+            },
+          },
+        })
+      }
+    })
+  })
+}
+
+export function effectPayloads() {
+  return [
+    ...registry
+      .entries()
+      .map(([type, def]) =>
+        EffectSchema.Struct({
+          type: EffectSchema.Literal("sync"),
+          name: EffectSchema.Literal(type),
+          id: EffectSchema.String,
+          seq: EffectSchema.Finite,
+          aggregateID: EffectSchema.Literal(def.aggregate),
+          data: def.schema,
+        }).annotate({ identifier: `SyncEvent.${type}` }),
+      )
+      .toArray(),
+    ...EventV2.registry
+      .values()
+      .filter(
+        (definition) =>
+          definition.version !== undefined && !registry.has(versionedType(definition.type, definition.version)),
+      )
+      .map((definition) =>
+        EffectSchema.Struct({
+          type: EffectSchema.Literal("sync"),
+          name: EffectSchema.Literal(versionedType(definition.type, definition.version!)),
+          id: EffectSchema.String,
+          seq: EffectSchema.Finite,
+          aggregateID: EffectSchema.Literal(definition.aggregate!),
+          data: definition.data,
+        }).annotate({ identifier: `SyncEvent.${definition.type}` }),
+      )
+      .toArray(),
+  ]
+}
+
+export * as SyncEvent from "."

package/src/engine/sync/schema.ts

@@ -0,0 +1,11 @@
+import { Schema } from "effect"
+
+import { Identifier } from "@/id/id"
+import { withStatics } from "@opencode-ai/core/schema"
+
+export const EventID = Schema.String.check(Schema.isStartsWith("evt")).pipe(
+  Schema.brand("EventID"),
+  withStatics((s) => ({
+    ascending: (id?: string) => s.make(Identifier.ascending("event", id)),
+  })),
+)

package/src/engine/temporary.ts

@@ -0,0 +1,33 @@
+import yargs from "yargs"
+import { TuiThreadCommand } from "./cli/cmd/tui/thread"
+import { InstallationVersion } from "@opencode-ai/core/installation/version"
+import { hideBin } from "yargs/helpers"
+import { Log } from "./node"
+
+Log.init({
+  print: false,
+})
+
+const cli = yargs(hideBin(process.argv))
+  .parserConfiguration({ "populate--": true })
+  .scriptName("opencode")
+  .wrap(100)
+  .help("help", "show help")
+  .alias("help", "h")
+  .version("version", "show version number", InstallationVersion)
+  .alias("version", "v")
+  .option("print-logs", {
+    describe: "print logs to stderr",
+    type: "boolean",
+  })
+  .option("log-level", {
+    describe: "log level",
+    type: "string",
+    choices: ["DEBUG", "INFO", "WARN", "ERROR"],
+  })
+  .option("pure", {
+    describe: "run without external plugins",
+    type: "boolean",
+  })
+  .command(TuiThreadCommand)
+  .parse()