edinburgh 0.3.0 → 0.4.1

package/README.md

@@ -1,63 +1,124 @@
 # Edinburgh
-**Very fast object persistence for TypeScript, supporting optimistic transactions, lazy loading and indexes.**
+**TypeScript objects that live in the database.**
 
-Edinburgh is a high-performance ORM built on [OLMDB](https://github.com/vanviegen/olmdb), providing type-safe model definitions with automatic field validation, ACID transactions, and efficient LMDB-based storage.
+Edinburgh blurs the line between in-memory objects and database records. Define a model class, and its instances *are* the database rows. They are read directly from a memory-mapped [LMDB](http://www.lmdb.tech/doc/) store on first access, and mutations are written back in an ACID transaction on commit. There is no SQL layer, no query builder, no network round-trip, and no result-set marshalling. A primary-key lookup completes in about 1 µs.
 
-**Features:**
+This makes problems like n+1 queries irrelevant: traversing `post.author.department.manager` is just a chain of microsecond memory-mapped reads, not a cascade of network calls.
 
-- 🚀 **Type-Safe Models**: Define models with automatic TypeScript type inference and runtime validation
-- 🔒 **ACID Transactions**: Optimistic locking with automatic retry on conflicts  
-- 🔗 **Relationships**: Model instances can reference each other, and will be lazy-loaded on access
-- 📦 **Embedded Database**: Negligible query latency, due to a blazing fast embedded database (LMDB)
-- 📊 **Custom Indexing**: Efficient querying with primary, unique, and multi-value indexes
+Built on [OLMDB](https://github.com/vanviegen/olmdb) (an optimistic-locking wrapper around LMDB).
+
+- **Objects are records**: model fields are backed by memory-mapped storage; no serialization boundary between your code and the database
+- **Sub-microsecond reads**: embedded B+ tree in the same process, no network hop, no query parsing
+- **Type-safe at every layer**: TypeScript inference at compile time, runtime validation at write time
+- **First-class relationships**: `E.link(OtherModel)` fields load lazily and transparently on access
+- **Indexes**: primary, unique, and secondary indexes with efficient range queries
+- **ACID transactions**: optimistic locking with automatic retry on conflict (up to 6 attempts)
+- **Zero-downtime schema evolution**: old rows are lazily migrated on read; no batch DDL required
 
 ## Quick Demo
 ```typescript
 import * as E from "edinburgh";
 
-// Initialize the database (optional, defaults to "./.olmdb")
+// Initialize the database (optional, defaults to ".edinburgh")
 E.init("./my-database");
 
 // Define a model
 @E.registerModel
 class User extends E.Model<User> {
-  // Define a primary key (optional, defaults to using the "id" field)
-  static pk = E.index(User, ["id"], "primary");
-  // Define a unique index on the email field
-  static byEmail = E.index(User, "email", "unique");
-
-  // Define fields with simple types -- they will be type-checked at compile time and validated at runtime.
-  id = E.field(E.identifier);
-  name = E.field(E.string);
-  email = E.field(E.string);
-  age = E.field(E.opt(E.number));
-
-  // A field with a more elaborate type. In Typescript: `User | User[] | "self" | "spouse"`
-  supervisor = E.field(E.choice(E.link(User), E.array(E.link(User)), E.literal("self"), E.literal("spouse")));
+    // Define a primary key (optional, defaults to using the "id" field)
+    static pk = E.primary(User, "id");
+    // Define a unique index on the email field
+    static byEmail = E.unique(User, "email");
+
+    // Define fields with simple types -- they will be type-checked at compile time and validated at runtime.
+    id = E.field(E.identifier);
+    name = E.field(E.string);
+    age = E.field(E.number);
+    email = E.field(E.opt(E.string)); // TypeScript: undefined | string
+
+    // Link to another instance of this model
+    supervisor = E.field(E.opt(E.link(User)));
+
+    // A field with a more elaborate type. In TypeScript: `User | User[] | "unknown" | "whatever"`
+    something = E.field(E.or(E.link(User), E.array(E.link(User)), E.literal("unknown"), E.literal("whatever")), { default: "unknown" });
 }
 
 // Use in transactions
 await E.transact(() => {
-  const user = new User({
-    name: "John Doe", 
-    email: "john@example.com"
-  });
+    const boss = new User({
+        name: "Big Boss",
+        age: 50,
+    });
+    const john = new User({ // Unique 'id' is automatically generated if not provided
+        name: "John Doe", 
+        age: 41,
+        email: "john@example.com",
+        supervisor: boss, // Link to another model instance
+    });
 });
 
 await E.transact(() => {
-  // Query by unique index
-  const user = User.byEmail.get("john@example.com")!;
-  // The transaction will retry if there's a conflict, such as another transaction
-  // modifying the same user (from another async function or another process)
-  user.age++;
-});
+    // Query by unique index
+    const john = User.byEmail.get("john@example.com")!;
+
+    // The transaction will retry if there's a conflict, such as another transaction
+    // modifying the same user (from another async function or another process)
+    john.age++;
+
+    // The supervisor object is lazy loaded on first access
+    console.log(`${john.supervisor!.name} is ${john.name}'s supervisor`);  
+});    
+```
+
+## TypeScript Configuration
+
+When using TypeScript to transpile to JavaScript, make sure to enable the following options in your `tsconfig.json`:
+
+```json
+{
+    "compilerOptions": {
+        "target": "es2022",
+        "experimentalDecorators": true
+    }
+}
 ```
 
+## Logging
+
+You can enable debug logging to stdout by setting the `EDINBURGH_LOG_LEVEL` environment variable to a number from 0 to 3. Higher numbers produce more verbose logs, including model-level operations, updates, and reads.
+
+- 0: no logging (default)
+- 1: model-level logs
+- 2: + update logs
+- 3: + read logs
+
 ## API Reference
 
 The following is auto-generated from `src/edinburgh.ts`:
 
-### transact · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L89)
+### scheduleInit · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L6)
+
+**Signature:** `() => void`
+
+### init · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L65)
+
+Initialize the database with the specified directory path.
+This function may be called multiple times with the same parameters. If it is not called before the first transact(),
+the database will be automatically initialized with the default directory.
+
+**Signature:** `(dbDir: string) => void`
+
+**Parameters:**
+
+- `dbDir: string`
+
+**Examples:**
+
+```typescript
+init("./my-database");
+```
+
+### transact · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L116)
 
 Executes a function within a database transaction context.
 
@@ -66,7 +127,7 @@ within a transaction.
 
 Transactions have a consistent view of the database, and changes made within a transaction are
 isolated from other transactions until they are committed. In case a commit clashes with changes
-made by another transaction, the transaction function will automatically be re-executed up to 10
+made by another transaction, the transaction function will automatically be re-executed up to 6
 times.
 
 **Signature:** `<T>(fn: () => T) => Promise<T>`
@@ -77,15 +138,13 @@ times.
 
 **Parameters:**
 
-- `fn: () => T` - - The function to execute within the transaction context.
+- `fn: () => T` - - The function to execute within the transaction context. Receives a Transaction instance.
 
 **Returns:** A promise that resolves with the function's return value.
 
 **Throws:**
 
-- If nested transactions are attempted.
 - With code "RACING_TRANSACTION" if the transaction fails after retries due to conflicts.
-- With code "TRANSACTION_FAILED" if the transaction fails for other reasons.
 - With code "TXN_LIMIT" if maximum number of transactions is reached.
 - With code "LMDB-{code}" for LMDB-specific errors.
 
@@ -94,7 +153,6 @@ times.
 ```typescript
 const paid = await E.transact(() => {
   const user = User.pk.get("john_doe");
-  // This is concurrency-safe - the function will rerun if it is raced by another transaction
   if (user.credits > 0) {
     user.credits--;
     return true;
@@ -105,32 +163,42 @@ const paid = await E.transact(() => {
 ```typescript
 // Transaction with automatic retry on conflicts
 await E.transact(() => {
-  const counter = Counter.load("global") || new Counter({id: "global", value: 0});
+  const counter = Counter.pk.get("global") || new Counter({id: "global", value: 0});
   counter.value++;
 });
 ```
 
-### setOnSaveCallback · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L146)
+### setMaxRetryCount · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L206)
+
+Set the maximum number of retries for a transaction in case of conflicts.
+The default value is 6. Setting it to 0 will disable retries and cause transactions to fail immediately on conflict.
+
+**Signature:** `(count: number) => void`
+
+**Parameters:**
+
+- `count: number` - The maximum number of retries for a transaction.
+
+### setOnSaveCallback · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L220)
 
 Set a callback function to be called after a model is saved and committed.
 
-**Signature:** `(callback: (commitId: number, items: ChangedModel[]) => void) => void`
+**Signature:** `(callback: (commitId: number, items: Map<Model<any>, Change>) => void) => void`
 
 **Parameters:**
 
-- `callback: ((commitId: number, items: ChangedModel[]) => void) | undefined` - [object Object],[object Object],[object Object]
+- `callback: ((commitId: number, items: Map<Model<any>, Change>) => void) | undefined` - The callback function to set. It gets called after each successful
+`transact()` commit that has changes, with the following arguments:
+- A sequential number. Higher numbers have been committed after lower numbers.
+- A map of model instances to their changes. The change can be "created", "deleted", or an object containing the old values.
 
-### deleteEverything · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L151)
+### deleteEverything · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L225)
 
 **Signature:** `() => Promise<void>`
 
-### Model · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
-
-Base class for all database models in the Edinburgh ORM.
+### Model · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L218)
 
-Models represent database entities with typed fields, automatic serialization,
-change tracking, and relationship management. All model classes should extend
-this base class and be decorated with `@registerModel`.
+[object Object],[object Object],[object Object],[object Object],[object Object]
 
 **Type Parameters:**
 
@@ -141,29 +209,70 @@ this base class and be decorated with `@registerModel`.
 ```typescript
 ⁣@E.registerModel
 class User extends E.Model<User> {
-  static pk = E.index(User, ["id"], "primary");
+  static pk = E.primary(User, "id");
   
   id = E.field(E.identifier);
   name = E.field(E.string);
   email = E.field(E.string);
   
-  static byEmail = E.index(User, "email", "unique");
+  static byEmail = E.unique(User, "email");
 }
 ```
 
-#### Model.tableName · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### Model.tableName · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L226)
 
 The database table name (defaults to class name).
 
 **Type:** `string`
 
-#### Model.fields · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### Model.override · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L230)
+
+When true, registerModel replaces an existing model with the same tableName.
+
+**Type:** `boolean`
+
+#### Model.fields · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L231)
 
 Field configuration metadata.
 
 **Type:** `Record<string | number | symbol, FieldConfig<unknown>>`
 
-#### Model.findAll · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### Model.migrate · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+Optional migration function called when deserializing rows written with an older schema version.
+Receives a plain record with all fields (primary key fields + value fields) and should mutate it
+in-place to match the current schema.
+
+This is called both during lazy loading (when a row is read from disk) and during batch
+migration (via `runMigration()` / `npx migrate-edinburgh`). The function's source code is hashed
+to detect changes. Modifying `migrate()` triggers a new schema version.
+
+If `migrate()` changes values of fields used in secondary or unique indexes, those indexes
+will only be updated when `runMigration()` is run (not during lazy loading).
+
+**Signature:** `(record: Record<string, any>) => void`
+
+**Parameters:**
+
+- `record: Record<string, any>` - - A plain object with all field values from the old schema version.
+
+**Examples:**
+
+```typescript
+⁣@E.registerModel
+class User extends E.Model<User> {
+  static pk = E.primary(User, "id");
+  id = E.field(E.identifier);
+  name = E.field(E.string);
+  role = E.field(E.string);  // new field
+
+  static migrate(record: Record<string, any>) {
+    record.role ??= "user";  // default for rows that predate the 'role' field
+  }
+}
+```
+
+#### Model.findAll · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Find all instances of this model in the database, ordered by primary key.
 
@@ -176,27 +285,74 @@ Find all instances of this model in the database, ordered by primary key.
 
 **Returns:** An iterator.
 
-#### model.changed · [property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### Model.replaceInto · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+Load an existing instance by primary key and update it, or create a new one.
+
+The provided object must contain all primary key fields. If a matching row exists,
+the remaining properties from `obj` are set on the loaded instance. Otherwise a
+new instance is created with `obj` as its initial properties.
+
+**Signature:** `<T extends typeof Model<any>>(this: T, obj: Partial<Omit<InstanceType<T>, "constructor">>) => InstanceType<T>`
+
+**Parameters:**
+
+- `this: T`
+- `obj: Partial<Omit<InstanceType<T>, "constructor">>` - - Partial model data that **must** include every primary key field.
+
+**Returns:** The loaded-and-updated or newly created instance.
+
+#### model.preCommit · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-This property can be used in `setOnSave` callbacks to determine how a model instance has changed.
-If the value is undefined, the instance has been created. If it's "deleted" the instance has
-been deleted. If its an object, the instance has been modified and the object contains the old values.
+Optional hook called on each modified instance right before the transaction commits.
+Runs before data is written to disk, so changes made here are included in the commit.
 
-Note: this property should **not** be accessed *during* a `transact()` -- it's state is an implementation
-detail that may change semantics at any minor release.
+Common use cases:
+- Computing derived or denormalized fields
+- Enforcing cross-field validation rules
+- Creating or updating related model instances (newly created instances will also
+  have their `preCommit()` called)
 
-**Type:** `Record<any, any> | "deleted" | "created"`
+**Signature:** `() => void`
+
+**Parameters:**
 
-#### model.getPrimaryKey · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+
+**Examples:**
+
+```typescript
+⁣@E.registerModel
+class Post extends E.Model<Post> {
+  static pk = E.primary(Post, "id");
+  id = E.field(E.identifier);
+  title = E.field(E.string);
+  slug = E.field(E.string);
+
+  preCommit() {
+    this.slug = this.title.toLowerCase().replace(/\s+/g, "-");
+  }
+}
+```
+
+#### model.getPrimaryKey · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 **Signature:** `() => Uint8Array<ArrayBufferLike>`
 
 **Parameters:**
 
 
-**Returns:** The primary key for this instance, or undefined if not yet saved.
+**Returns:** The primary key for this instance.
+
+#### model.getPrimaryKeyHash · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => number`
+
+**Parameters:**
+
+
+**Returns:** A 53-bit positive integer non-cryptographic hash of the primary key, or undefined if not yet saved.
 
-#### model.isLazyField · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### model.isLazyField · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 **Signature:** `(field: keyof this) => boolean`
 
@@ -204,7 +360,7 @@ detail that may change semantics at any minor release.
 
 - `field: keyof this`
 
-#### model.preventPersist · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### model.preventPersist · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Prevent this instance from being persisted to the database.
 
@@ -223,7 +379,7 @@ user.name = "New Name";
 user.preventPersist(); // Changes won't be saved
 ```
 
-#### model.delete · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### model.delete · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Delete this model instance from the database.
 
@@ -241,7 +397,7 @@ const user = User.load("user123");
 user.delete(); // Removes from database
 ```
 
-#### model.validate · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### model.validate · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Validate all fields in this model instance.
 
@@ -263,7 +419,7 @@ if (errors.length > 0) {
 }
 ```
 
-#### model.isValid · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### model.isValid · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Check if this model instance is valid.
 
@@ -281,7 +437,28 @@ const user = new User({name: "John"});
 if (!user.isValid()) shoutAtTheUser();
 ```
 
-### registerModel · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L91)
+#### model.getState · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => "created" | "deleted" | "loaded" | "lazy"`
+
+**Parameters:**
+
+
+#### model.toString · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => string`
+
+**Parameters:**
+
+
+#### model.[Symbol.for('nodejs.util.inspect.custom')] · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => string`
+
+**Parameters:**
+
+
+### registerModel · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L111)
 
 Register a model class with the Edinburgh ORM system.
 
@@ -308,7 +485,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### field · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L60)
+### field · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L87)
 
 Create a field definition for a model property.
 
@@ -338,13 +515,13 @@ class User extends E.Model<User> {
 }
 ```
 
-### string · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### string · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Type wrapper instance for the string type.
 
 **Value:** `TypeWrapper<string>`
 
-### orderedString · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### orderedString · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Type wrapper instance for the ordered string type, which is just like a string
 except that it sorts lexicographically in the database (instead of by incrementing
@@ -354,37 +531,37 @@ may not contain null characters.
 
 **Value:** `TypeWrapper<string>`
 
-### number · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### number · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Type wrapper instance for the number type.
 
 **Value:** `TypeWrapper<number>`
 
-### dateTime · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### dateTime · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Type wrapper instance for the date/time type.
 
 **Value:** `TypeWrapper<Date>`
 
-### boolean · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### boolean · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Type wrapper instance for the boolean type.
 
 **Value:** `TypeWrapper<boolean>`
 
-### identifier · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### identifier · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Type wrapper instance for the identifier type.
 
 **Value:** `TypeWrapper<string>`
 
-### undef · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### undef · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Type wrapper instance for the 'undefined' type.
 
 **Value:** `TypeWrapper<undefined>`
 
-### opt · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### opt · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create an optional type wrapper (allows undefined).
 
@@ -407,7 +584,7 @@ const optionalString = E.opt(E.string);
 const optionalNumber = E.opt(E.number);
 ```
 
-### or · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### or · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a union type wrapper from multiple type choices.
 
@@ -430,7 +607,7 @@ const stringOrNumber = E.or(E.string, E.number);
 const status = E.or("active", "inactive", "pending");
 ```
 
-### array · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### array · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create an array type wrapper with optional length constraints.
 
@@ -454,7 +631,7 @@ const stringArray = E.array(E.string);
 const boundedArray = E.array(E.number, {min: 1, max: 10});
 ```
 
-### literal · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### literal · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a literal type wrapper for a constant value.
 
@@ -477,7 +654,7 @@ const statusType = E.literal("active");
 const countType = E.literal(42);
 ```
 
-### link · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### link · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a link type wrapper for model relationships.
 
@@ -505,7 +682,7 @@ class Post extends E.Model<Post> {
 }
 ```
 
-### index · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### index · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a secondary index on model fields.
 
@@ -532,7 +709,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### primary · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### primary · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a primary index on model fields.
 
@@ -559,7 +736,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### unique · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### unique · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a unique index on model fields.
 
@@ -586,7 +763,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### dump · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### dump · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Dump database contents for debugging.
 
@@ -595,18 +772,7 @@ This is primarily useful for development and debugging purposes.
 
 **Signature:** `() => void`
 
-### setLogLevel · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L55)
-
-Global log level for debugging output.
-0 = no logging, 1 = model-level logs, 2 = update logs, 3 = read logs.
-
-**Signature:** `(level: number) => void`
-
-**Parameters:**
-
-- `level: number`
-
-### BaseIndex · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L129)
+### BaseIndex · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L128)
 
 Base class for database indexes for efficient lookups on model fields.
 
@@ -622,73 +788,33 @@ Indexes enable fast queries on specific field combinations and enforce uniquenes
 - `MyModel`: - The model class this index belongs to.
 - `_fieldNames`: - Array of field names that make up this index.
 
-#### baseIndex.find · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
-
-Find model instances using flexible range query options.
-
-Supports exact matches, inclusive/exclusive range queries, and reverse iteration.
-For single-field indexes, you can pass values directly or in arrays.
-For multi-field indexes, pass arrays or partial arrays for prefix matching.
+#### baseIndex.find · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 **Signature:** `(opts?: FindOptions<IndexArgTypes<M, F>>) => IndexRangeIterator<M>`
 
 **Parameters:**
 
-- `opts: FindOptions<IndexArgTypes<M, F>>` (optional) - - Query options object
-
-**Returns:** An iterable of model instances matching the query
+- `opts: FindOptions<IndexArgTypes<M, F>>` (optional)
 
-**Examples:**
+#### baseIndex.batchProcess · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-```typescript
-// Exact match
-for (const user of User.byEmail.find({is: "john@example.com"})) {
-  console.log(user.name);
-}
+[object Object],[object Object],[object Object]
 
-// Range query (inclusive)
-for (const user of User.byEmail.find({from: "a@", to: "m@"})) {
-  console.log(user.email);
-}
+**Signature:** `(opts: FindOptions<IndexArgTypes<M, F>> & { limitSeconds?: number; limitRows?: number; }, callback: (row: InstanceType<M>) => void | Promise<...>) => Promise<...>`
 
-// Range query (exclusive)
-for (const user of User.byEmail.find({after: "a@", before: "m@"})) {
-  console.log(user.email);
-}
-
-// Open-ended ranges
-for (const user of User.byEmail.find({from: "m@"})) { // m@ and later
-  console.log(user.email);
-}
-
-for (const user of User.byEmail.find({to: "m@"})) { // up to and including m@
-  console.log(user.email);
-}
-
-// Reverse iteration
-for (const user of User.byEmail.find({reverse: true})) {
-  console.log(user.email); // Z to A order
-}
-
-// Multi-field index prefix matching
-for (const item of CompositeModel.pk.find({from: ["electronics", "phones"]})) {
-  console.log(item.name); // All electronics/phones items
-}
+**Parameters:**
 
-// For single-field indexes, you can use the value directly
-for (const user of User.byEmail.find({is: "john@example.com"})) {
-  console.log(user.name);
-}
-```
+- `opts: FindOptions<IndexArgTypes<M, F>> & { limitSeconds?: number; limitRows?: number }` (optional) - - Query options (same as `find()`), plus:
+- `callback: (row: InstanceType<M>) => void | Promise<void>` - - Called for each matching row within a transaction
 
-#### baseIndex.toString · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### baseIndex.toString · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 **Signature:** `() => string`
 
 **Parameters:**
 
 
-### UniqueIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### UniqueIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Unique index that stores references to the primary key.
 
@@ -697,7 +823,7 @@ Unique index that stores references to the primary key.
 - `M extends typeof Model` - The model class this index belongs to.
 - `F extends readonly (keyof InstanceType<M> & string)[]` - The field names that make up this index.
 
-#### uniqueIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### uniqueIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Get a model instance by unique index key values.
 
@@ -715,7 +841,7 @@ Get a model instance by unique index key values.
 const userByEmail = User.byEmail.get("john@example.com");
 ```
 
-### PrimaryIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+### PrimaryIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Primary index that stores the actual model data.
 
@@ -724,15 +850,15 @@ Primary index that stores the actual model data.
 - `M extends typeof Model` - The model class this index belongs to.
 - `F extends readonly (keyof InstanceType<M> & string)[]` - The field names that make up this index.
 
-#### primaryIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### primaryIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Get a model instance by primary key values.
 
-**Signature:** `(...args: IndexArgTypes<M, F> | [Uint8Array<ArrayBufferLike>]) => InstanceType<M>`
+**Signature:** `(...args: IndexArgTypes<M, F>) => InstanceType<M>`
 
 **Parameters:**
 
-- `args: IndexArgTypes<M, F> | [Uint8Array]` - - The primary key values.
+- `args: IndexArgTypes<M, F>` - - The primary key values.
 
 **Returns:** The model instance if found, undefined otherwise.
 
@@ -742,125 +868,169 @@ Get a model instance by primary key values.
 const user = User.pk.get("john_doe");
 ```
 
-#### primaryIndex.getLazy · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### primaryIndex.getLazy · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Does the same as as `get()`, but will delay loading the instance from disk until the first
 property access. In case it turns out the instance doesn't exist, an error will be thrown
 at that time.
 
-**Signature:** `(...args: IndexArgTypes<M, F> | [Uint8Array<ArrayBufferLike>]) => InstanceType<M>`
+**Signature:** `(...args: IndexArgTypes<M, F>) => InstanceType<M>`
 
 **Parameters:**
 
-- `args: IndexArgTypes<M, F> | [Uint8Array]` - Primary key field values. (Or a single Uint8Array containing the key.)
+- `args: IndexArgTypes<M, F>` - Primary key field values. (Or a single Uint8Array containing the key.)
 
 **Returns:** The (lazily loaded) model instance.
 
-### init · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L141)
+### SecondaryIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-Initialize the database with the specified directory path.
-This function may only be called once. If it is not called before the first transact(),
-the database will be automatically initialized with the default directory.
+Secondary index for non-unique lookups.
 
-**Signature:** `(dbDir?: string) => void`
+**Type Parameters:**
 
-**Parameters:**
+- `M extends typeof Model` - The model class this index belongs to.
+- `F extends readonly (keyof InstanceType<M> & string)[]` - The field names that make up this index.
 
-- `dbDir?: string` - - Optional directory path for the database (defaults to environment variable $OLMDB_DIR or "./.olmdb").
+### Change · [type](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L94)
 
-**Throws:**
+**Type:** `Record<any, any> | "created" | "deleted"`
 
-- With code "DUP_INIT" if database is already initialized.
-- With code "CREATE_DIR_FAILED" if directory creation fails.
-- With code "LMDB-{code}" for LMDB-specific errors.
+### Transaction · [interface](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L38)
 
-**Examples:**
+#### transaction.id · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L41)
 
-```typescript
-init("./my-database");
-```
+**Type:** `number`
 
-### onCommit · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L159)
+#### transaction.instances · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L42)
 
-Registers a callback to be executed when the current transaction commits successfully.
-The callback will be executed outside of transaction context.
+**Type:** `Set<Model<unknown>>`
 
-**Signature:** `(callback: (commitSeq: number) => void) => void`
+#### transaction.instancesByPk · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L44)
 
-**Parameters:**
+**Type:** `Map<number, Model<unknown>>`
 
-- `callback: (commitSeq: number) => void` - - Function to execute when transaction commits. It receives the commit sequence, which is an always-increasing number that provides a global ordering of commits, as an argument.
+### DatabaseError · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L156)
 
-**Throws:**
+The DatabaseError class is used to represent errors that occur during database operations.
+It extends the built-in Error class and has a machine readable error code string property.
 
-- If called outside of a transaction context
+The lowlevel API will throw DatabaseError instances for all database-related errors.
+Invalid function arguments will throw TypeError.
 
-### onRevert · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L151)
+**Value:** `DatabaseErrorConstructor`
+
+### runMigration · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L116)
 
-Registers a callback to be executed when the current transaction is reverted (aborted due to error).
-The callback will be executed outside of transaction context.
+Run database migration: upgrade all rows to the latest schema version,
+convert old primary indices, and clean up orphaned secondary indices.
 
-**Signature:** `(callback: (commitSeq: number) => void) => void`
+**Signature:** `(options?: MigrationOptions) => Promise<MigrationResult>`
 
 **Parameters:**
 
-- `callback: (commitSeq: number) => void` - - Function to execute when transaction is reverted. It receives the dummy (always 0) commit sequence indicating failure as an argument.
+- `options: MigrationOptions` (optional)
 
-**Throws:**
+### MigrationOptions · [interface](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L16)
 
-- If called outside of a transaction context
+#### migrationOptions.tables · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L18)
 
-### getTransactionData · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L55)
+Limit migration to specific table names.
 
-Retrieves data from the current transaction context.
+**Type:** `string[]`
 
-**Signature:** `(key: symbol) => any`
+#### migrationOptions.convertOldPrimaries · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L22)
 
-**Parameters:**
+Whether to convert old primary indices for known tables (default: true).
 
-- `key: symbol` - - A symbol key to retrieve data from the current transaction context.
+**Type:** `boolean`
 
-**Returns:** - The value associated with the key, or undefined if not set.
+#### migrationOptions.deleteOrphanedIndexes · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L30)
 
-**Throws:**
+Whether to delete orphaned secondary/unique indices (default: true).
 
-- If called outside of a transaction context.
+**Type:** `boolean`
 
-### setTransactionData · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L46)
+#### migrationOptions.upgradeVersions · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L38)
 
-Attach some arbitrary user data to the current transaction context, which is
-attached to the currently running (async) task.
+Whether to upgrade rows to the latest version (default: true).
 
-**Signature:** `(key: symbol, value: any) => void`
+**Type:** `boolean`
 
-**Parameters:**
+#### migrationOptions.onProgress · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L42)
 
-- `key: symbol` - - A symbol key to store data in the current transaction context.
-- `value: any` - - The value to store.
+Progress callback.
 
-**Throws:**
+**Type:** `(info: ProgressInfo) => void`
 
-- If called outside of a transaction context.
+### MigrationResult · [interface](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L47)
 
-**Examples:**
+#### migrationResult.secondaries · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L49)
+
+Per-table stats for row upgrades.
+
+**Type:** `Record<string, number>`
+
+#### migrationResult.primaries · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L50)
+
+Per-table stats for old primary conversions.
+
+**Type:** `Record<string, number>`
+
+#### migrationResult.conversionFailures · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L56)
+
+Per-table conversion failure counts by reason.
+
+**Type:** `Record<string, Record<string, number>>`
+
+#### migrationResult.orphaned · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L57)
+
+Number of orphaned index entries deleted.
+
+**Type:** `number`
+
+## Schema Migrations
+
+Edinburgh automatically tracks the schema version of each model. When you change fields, field types, indexes, or the `migrate()` function, Edinburgh detects a new schema version.
+
+### What happens automatically (lazy migration)
+
+Changes to regular (non-index) field values are migrated lazily. When a row with an old schema version is loaded from disk, it is deserialized using the old field types and transformed by the optional static `migrate()` function. This is transparent and requires no downtime.
 
 ```typescript
-const MY_SYMBOL = Symbol("myKey");
-await transact(async () => {
-  setTransactionData(MY_SYMBOL, "myValue");
-  await somethingAsync(); // Can be interleaved with other transactions
-  const value = getTransactionData(MY_SYMBOL);
-  console.log(value); // "myValue"
-});
+@E.registerModel
+class User extends E.Model<User> {
+  static pk = E.primary(User, "id");
+  id = E.field(E.identifier);
+  name = E.field(E.string);
+  role = E.field(E.string);  // newly added field
+
+  static migrate(record: Record<string, any>) {
+    record.role ??= "user";  // provide a default for old rows
+  }
+}
 ```
 
-### DatabaseError · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L142)
+### What requires `migrate-edinburgh`
 
-The DatabaseError class is used to represent errors that occur during database operations.
-It extends the built-in Error class and has a machine readable error code string property.
+The `migrate-edinburgh` CLI tool (or the `runMigration()` API) must be run when:
 
-The lowlevel API will throw DatabaseError instances for all database-related errors.
-Invalid function arguments will throw TypeError.
+- **Adding or removing** secondary or unique indexes
+- **Changing the fields or types** of an existing index
+- A **`migrate()` function changes values** that are used in index fields
 
-**Value:** `DatabaseErrorConstructor`
+The tool populates new indexes, removes orphaned ones, and updates index entries whose values were changed by `migrate()`. It does *not* rewrite primary data rows - lazy migration handles that on read.
+
+```bash
+npx migrate-edinburgh --import ./src/models.ts
+```
+
+Run `npx migrate-edinburgh` to see all of its options.
+
+You can also call `runMigration()` programmatically:
 
+```typescript
+import { runMigration } from "edinburgh";
+
+const result = await runMigration({ tables: ["User"] });
+console.log(result.upgraded);  // { User: 1500 }
+```