edinburgh 0.1.3 → 0.4.1

package/README.md

@@ -1,63 +1,124 @@
 # Edinburgh
-**Very fast object persistence for TypeScript, supporting optimistic transactions, on-demand reference loading, automatic back-references and indexes.**
+**TypeScript objects that live in the database.**
 
-Edinburgh is a high-performance ORM built on [OLMDB](https://github.com/vanviegen/olmdb), providing type-safe model definitions with automatic field validation, ACID transactions, and efficient LMDB-based storage.
+Edinburgh blurs the line between in-memory objects and database records. Define a model class, and its instances *are* the database rows. They are read directly from a memory-mapped [LMDB](http://www.lmdb.tech/doc/) store on first access, and mutations are written back in an ACID transaction on commit. There is no SQL layer, no query builder, no network round-trip, and no result-set marshalling. A primary-key lookup completes in about 1 µs.
 
-**Features:**
+This makes problems like n+1 queries irrelevant: traversing `post.author.department.manager` is just a chain of microsecond memory-mapped reads, not a cascade of network calls.
 
-- 🚀 **Type-Safe Models**: Define models with automatic TypeScript type inference and runtime validation
-- 🔒 **ACID Transactions**: Optimistic locking with automatic retry on conflicts  
-- 🔗 **Relationship Management**: Automatic back-references and link handling between models
-- 📊 **Custom Indexing**: Efficient querying with primary, unique, and multi-value indexes
-- 📦 **Zero Dependencies**: Minimal footprint, built on LMDB for maximum performance
+Built on [OLMDB](https://github.com/vanviegen/olmdb) (an optimistic-locking wrapper around LMDB).
+
+- **Objects are records**: model fields are backed by memory-mapped storage; no serialization boundary between your code and the database
+- **Sub-microsecond reads**: embedded B+ tree in the same process, no network hop, no query parsing
+- **Type-safe at every layer**: TypeScript inference at compile time, runtime validation at write time
+- **First-class relationships**: `E.link(OtherModel)` fields load lazily and transparently on access
+- **Indexes**: primary, unique, and secondary indexes with efficient range queries
+- **ACID transactions**: optimistic locking with automatic retry on conflict (up to 6 attempts)
+- **Zero-downtime schema evolution**: old rows are lazily migrated on read; no batch DDL required
 
 ## Quick Demo
 ```typescript
 import * as E from "edinburgh";
 
-// Initialize the database (optional, defaults to "./.olmdb")
+// Initialize the database (optional, defaults to ".edinburgh")
 E.init("./my-database");
 
 // Define a model
 @E.registerModel
 class User extends E.Model<User> {
-  // Define a primary key (optional, defaults to using the "id" field)
-  static pk = E.index(User, ["id"], "primary");
-  // Define a unique index on the email field
-  static byEmail = E.index(User, "email", "unique");
-
-  // Define fields with simple types -- they will be type-checked at compile time and validated at runtime.
-  id = E.field(E.identifier);
-  name = E.field(E.string);
-  email = E.field(E.string);
-  age = E.field(E.opt(E.number));
-
-  // A field with a more elaborate type. In Typescript: `User | User[] | "self" | "spouse"`
-  supervisor = E.field(E.choice(E.link(User), E.array(E.link(User)), E.literal("self"), E.literal("spouse")));
+    // Define a primary key (optional, defaults to using the "id" field)
+    static pk = E.primary(User, "id");
+    // Define a unique index on the email field
+    static byEmail = E.unique(User, "email");
+
+    // Define fields with simple types -- they will be type-checked at compile time and validated at runtime.
+    id = E.field(E.identifier);
+    name = E.field(E.string);
+    age = E.field(E.number);
+    email = E.field(E.opt(E.string)); // TypeScript: undefined | string
+
+    // Link to another instance of this model
+    supervisor = E.field(E.opt(E.link(User)));
+
+    // A field with a more elaborate type. In TypeScript: `User | User[] | "unknown" | "whatever"`
+    something = E.field(E.or(E.link(User), E.array(E.link(User)), E.literal("unknown"), E.literal("whatever")), { default: "unknown" });
 }
 
 // Use in transactions
 await E.transact(() => {
-  const user = new User({
-    name: "John Doe", 
-    email: "john@example.com"
-  });
+    const boss = new User({
+        name: "Big Boss",
+        age: 50,
+    });
+    const john = new User({ // Unique 'id' is automatically generated if not provided
+        name: "John Doe", 
+        age: 41,
+        email: "john@example.com",
+        supervisor: boss, // Link to another model instance
+    });
 });
 
 await E.transact(() => {
-  // Query by unique index
-  const user = User.byEmail.get("john@example.com")!;
-  // The transaction will retry if there's a conflict, such as another transaction
-  // modifying the same user (from another async function or another process)
-  user.age++;
-});
+    // Query by unique index
+    const john = User.byEmail.get("john@example.com")!;
+
+    // The transaction will retry if there's a conflict, such as another transaction
+    // modifying the same user (from another async function or another process)
+    john.age++;
+
+    // The supervisor object is lazy loaded on first access
+    console.log(`${john.supervisor!.name} is ${john.name}'s supervisor`);  
+});    
+```
+
+## TypeScript Configuration
+
+When using TypeScript to transpile to JavaScript, make sure to enable the following options in your `tsconfig.json`:
+
+```json
+{
+    "compilerOptions": {
+        "target": "es2022",
+        "experimentalDecorators": true
+    }
+}
 ```
 
+## Logging
+
+You can enable debug logging to stdout by setting the `EDINBURGH_LOG_LEVEL` environment variable to a number from 0 to 3. Higher numbers produce more verbose logs, including model-level operations, updates, and reads.
+
+- 0: no logging (default)
+- 1: model-level logs
+- 2: + update logs
+- 3: + read logs
+
 ## API Reference
 
 The following is auto-generated from `src/edinburgh.ts`:
 
-### transact · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L83)
+### scheduleInit · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L6)
+
+**Signature:** `() => void`
+
+### init · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L65)
+
+Initialize the database with the specified directory path.
+This function may be called multiple times with the same parameters. If it is not called before the first transact(),
+the database will be automatically initialized with the default directory.
+
+**Signature:** `(dbDir: string) => void`
+
+**Parameters:**
+
+- `dbDir: string`
+
+**Examples:**
+
+```typescript
+init("./my-database");
+```
+
+### transact · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L116)
 
 Executes a function within a database transaction context.
 
@@ -66,7 +127,7 @@ within a transaction.
 
 Transactions have a consistent view of the database, and changes made within a transaction are
 isolated from other transactions until they are committed. In case a commit clashes with changes
-made by another transaction, the transaction function will automatically be re-executed up to 10
+made by another transaction, the transaction function will automatically be re-executed up to 6
 times.
 
 **Signature:** `<T>(fn: () => T) => Promise<T>`
@@ -77,15 +138,13 @@ times.
 
 **Parameters:**
 
-- `fn: () => T` - - The function to execute within the transaction context.
+- `fn: () => T` - - The function to execute within the transaction context. Receives a Transaction instance.
 
 **Returns:** A promise that resolves with the function's return value.
 
 **Throws:**
 
-- If nested transactions are attempted.
 - With code "RACING_TRANSACTION" if the transaction fails after retries due to conflicts.
-- With code "TRANSACTION_FAILED" if the transaction fails for other reasons.
 - With code "TXN_LIMIT" if maximum number of transactions is reached.
 - With code "LMDB-{code}" for LMDB-specific errors.
 
@@ -93,8 +152,7 @@ times.
 
 ```typescript
 const paid = await E.transact(() => {
-  const user = User.load("john_doe");
-  // This is concurrency-safe - the function will rerun if it is raced by another transaction
+  const user = User.pk.get("john_doe");
   if (user.credits > 0) {
     user.credits--;
     return true;
@@ -105,22 +163,42 @@ const paid = await E.transact(() => {
 ```typescript
 // Transaction with automatic retry on conflicts
 await E.transact(() => {
-  const counter = Counter.load("global") || new Counter({id: "global", value: 0});
+  const counter = Counter.pk.get("global") || new Counter({id: "global", value: 0});
   counter.value++;
 });
 ```
 
-### deleteEverything · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L112)
+### setMaxRetryCount · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L206)
 
-**Signature:** `() => Promise<void>`
+Set the maximum number of retries for a transaction in case of conflicts.
+The default value is 6. Setting it to 0 will disable retries and cause transactions to fail immediately on conflict.
+
+**Signature:** `(count: number) => void`
 
-### Model · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Parameters:**
 
-Base class for all database models in the Edinburgh ORM.
+- `count: number` - The maximum number of retries for a transaction.
 
-Models represent database entities with typed fields, automatic serialization,
-change tracking, and relationship management. All model classes should extend
-this base class and be decorated with `@registerModel`.
+### setOnSaveCallback · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L220)
+
+Set a callback function to be called after a model is saved and committed.
+
+**Signature:** `(callback: (commitId: number, items: Map<Model<any>, Change>) => void) => void`
+
+**Parameters:**
+
+- `callback: ((commitId: number, items: Map<Model<any>, Change>) => void) | undefined` - The callback function to set. It gets called after each successful
+`transact()` commit that has changes, with the following arguments:
+- A sequential number. Higher numbers have been committed after lower numbers.
+- A map of model instances to their changes. The change can be "created", "deleted", or an object containing the old values.
+
+### deleteEverything · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L225)
+
+**Signature:** `() => Promise<void>`
+
+### Model · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L218)
+
+[object Object],[object Object],[object Object],[object Object],[object Object]
 
 **Type Parameters:**
 
@@ -131,54 +209,160 @@ this base class and be decorated with `@registerModel`.
 ```typescript
 ⁣@E.registerModel
 class User extends E.Model<User> {
-  static pk = E.index(User, ["id"], "primary");
+  static pk = E.primary(User, "id");
   
   id = E.field(E.identifier);
   name = E.field(E.string);
   email = E.field(E.string);
   
-  static byEmail = E.index(User, "email", "unique");
+  static byEmail = E.unique(User, "email");
 }
 ```
 
-#### Model.tableName · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### Model.tableName · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L226)
 
 The database table name (defaults to class name).
 
 **Type:** `string`
 
-#### Model.fields · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### Model.override · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L230)
+
+When true, registerModel replaces an existing model with the same tableName.
+
+**Type:** `boolean`
+
+#### Model.fields · [static property](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L231)
 
 Field configuration metadata.
 
-**Type:** `Record<string, FieldConfig<unknown>>`
+**Type:** `Record<string | number | symbol, FieldConfig<unknown>>`
+
+#### Model.migrate · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-#### Model.load · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+Optional migration function called when deserializing rows written with an older schema version.
+Receives a plain record with all fields (primary key fields + value fields) and should mutate it
+in-place to match the current schema.
 
-Load a model instance by primary key.
+This is called both during lazy loading (when a row is read from disk) and during batch
+migration (via `runMigration()` / `npx migrate-edinburgh`). The function's source code is hashed
+to detect changes. Modifying `migrate()` triggers a new schema version.
 
-**Signature:** `<SUB>(this: typeof Model<SUB>, ...args: any[]) => SUB`
+If `migrate()` changes values of fields used in secondary or unique indexes, those indexes
+will only be updated when `runMigration()` is run (not during lazy loading).
+
+**Signature:** `(record: Record<string, any>) => void`
 
 **Parameters:**
 
-- `this: typeof Model<SUB>`
-- `args: any[]` - - Primary key field values.
+- `record: Record<string, any>` - - A plain object with all field values from the old schema version.
+
+**Examples:**
+
+```typescript
+⁣@E.registerModel
+class User extends E.Model<User> {
+  static pk = E.primary(User, "id");
+  id = E.field(E.identifier);
+  name = E.field(E.string);
+  role = E.field(E.string);  // new field
+
+  static migrate(record: Record<string, any>) {
+    record.role ??= "user";  // default for rows that predate the 'role' field
+  }
+}
+```
+
+#### Model.findAll · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+Find all instances of this model in the database, ordered by primary key.
+
+**Signature:** `<T extends typeof Model<unknown>>(this: T, opts?: { reverse?: boolean; }) => IndexRangeIterator<T>`
+
+**Parameters:**
+
+- `this: T`
+- `opts?: {reverse?: boolean}` - - Optional parameters.
+
+**Returns:** An iterator.
+
+#### Model.replaceInto · [static method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+Load an existing instance by primary key and update it, or create a new one.
+
+The provided object must contain all primary key fields. If a matching row exists,
+the remaining properties from `obj` are set on the loaded instance. Otherwise a
+new instance is created with `obj` as its initial properties.
+
+**Signature:** `<T extends typeof Model<any>>(this: T, obj: Partial<Omit<InstanceType<T>, "constructor">>) => InstanceType<T>`
+
+**Parameters:**
+
+- `this: T`
+- `obj: Partial<Omit<InstanceType<T>, "constructor">>` - - Partial model data that **must** include every primary key field.
+
+**Returns:** The loaded-and-updated or newly created instance.
+
+#### model.preCommit · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+Optional hook called on each modified instance right before the transaction commits.
+Runs before data is written to disk, so changes made here are included in the commit.
+
+Common use cases:
+- Computing derived or denormalized fields
+- Enforcing cross-field validation rules
+- Creating or updating related model instances (newly created instances will also
+  have their `preCommit()` called)
+
+**Signature:** `() => void`
+
+**Parameters:**
 
-**Returns:** The model instance if found, undefined otherwise.
 
 **Examples:**
 
 ```typescript
-const user = User.load("user123");
-const post = Post.load("post456", "en");
+⁣@E.registerModel
+class Post extends E.Model<Post> {
+  static pk = E.primary(Post, "id");
+  id = E.field(E.identifier);
+  title = E.field(E.string);
+  slug = E.field(E.string);
+
+  preCommit() {
+    this.slug = this.title.toLowerCase().replace(/\s+/g, "-");
+  }
+}
 ```
 
-#### model.preventPersist · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### model.getPrimaryKey · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-Prevent this instance from being persisted to the database.
+**Signature:** `() => Uint8Array<ArrayBufferLike>`
+
+**Parameters:**
+
+
+**Returns:** The primary key for this instance.
+
+#### model.getPrimaryKeyHash · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => number`
+
+**Parameters:**
+
+
+**Returns:** A 53-bit positive integer non-cryptographic hash of the primary key, or undefined if not yet saved.
 
-Removes the instance from the modified instances set and disables
-automatic persistence at transaction commit.
+#### model.isLazyField · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `(field: keyof this) => boolean`
+
+**Parameters:**
+
+- `field: keyof this`
+
+#### model.preventPersist · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+Prevent this instance from being persisted to the database.
 
 **Signature:** `() => this`
 
@@ -195,7 +379,7 @@ user.name = "New Name";
 user.preventPersist(); // Changes won't be saved
 ```
 
-#### model.delete · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### model.delete · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Delete this model instance from the database.
 
@@ -213,11 +397,11 @@ const user = User.load("user123");
 user.delete(); // Removes from database
 ```
 
-#### model.validate · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### model.validate · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Validate all fields in this model instance.
 
-**Signature:** `(raise?: boolean) => DatabaseError[]`
+**Signature:** `(raise?: boolean) => Error[]`
 
 **Parameters:**
 
@@ -235,7 +419,7 @@ if (errors.length > 0) {
 }
 ```
 
-#### model.isValid · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### model.isValid · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Check if this model instance is valid.
 
@@ -253,13 +437,30 @@ const user = new User({name: "John"});
 if (!user.isValid()) shoutAtTheUser();
 ```
 
-### registerModel · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### model.getState · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => "created" | "deleted" | "loaded" | "lazy"`
+
+**Parameters:**
+
+
+#### model.toString · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => string`
+
+**Parameters:**
+
+
+#### model.[Symbol.for('nodejs.util.inspect.custom')] · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
+
+**Signature:** `() => string`
+
+**Parameters:**
 
-Register a model class with the Edinburgh ORM system.
 
-This decorator function transforms the model class to use a proxy-based constructor
-that enables change tracking and automatic field initialization. It also extracts
-field metadata and sets up default values on the prototype.
+### registerModel · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L111)
+
+Register a model class with the Edinburgh ORM system.
 
 **Signature:** `<T extends typeof Model<unknown>>(MyModel: T) => T`
 
@@ -284,7 +485,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### field · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L55)
+### field · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L87)
 
 Create a field definition for a model property.
 
@@ -314,51 +515,57 @@ class User extends E.Model<User> {
 }
 ```
 
-### setOnSaveCallback · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L85)
+### string · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-Set a callback function to be called after a model is saved and committed.
+Type wrapper instance for the string type.
 
-**Signature:** `(callback: OnSaveType) => void`
+**Value:** `TypeWrapper<string>`
 
-**Parameters:**
+### orderedString · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-- `callback: OnSaveType | undefined` - The callback function to set. As arguments, it receives the model instance, the new key (undefined in case of a delete), and the old key (undefined in case of a create).
+Type wrapper instance for the ordered string type, which is just like a string
+except that it sorts lexicographically in the database (instead of by incrementing
+length first), making it suitable for index fields that want lexicographic range
+scans. Ordered strings are implemented as null-terminated UTF-8 strings, so they
+may not contain null characters.
 
-### string · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Value:** `TypeWrapper<string>`
 
-Constant representing the string type.
+### number · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Value:** `StringType`
+Type wrapper instance for the number type.
 
-### number · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Value:** `TypeWrapper<number>`
 
-Constant representing the number type.
+### dateTime · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Value:** `NumberType`
+Type wrapper instance for the date/time type.
 
-### boolean · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Value:** `TypeWrapper<Date>`
 
-Constant representing the boolean type.
+### boolean · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Value:** `BooleanType`
+Type wrapper instance for the boolean type.
 
-### identifier · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Value:** `TypeWrapper<boolean>`
 
-Constant representing the identifier type.
+### identifier · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Value:** `IdentifierType`
+Type wrapper instance for the identifier type.
 
-### undef · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Value:** `TypeWrapper<string>`
 
-Constant representing the 'undefined' type.
+### undef · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Value:** `LiteralType<any>`
+Type wrapper instance for the 'undefined' type.
 
-### opt · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Value:** `TypeWrapper<undefined>`
+
+### opt · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create an optional type wrapper (allows undefined).
 
-**Signature:** `<const T extends TypeWrapper<unknown> | BasicType>(inner: T) => OrType<any>`
+**Signature:** `<const T extends TypeWrapper<unknown> | BasicType>(inner: T) => TypeWrapper<T extends TypeWrapper<infer U> ? U : T>`
 
 **Type Parameters:**
 
@@ -377,11 +584,11 @@ const optionalString = E.opt(E.string);
 const optionalNumber = E.opt(E.number);
 ```
 
-### or · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### or · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a union type wrapper from multiple type choices.
 
-**Signature:** `<const T extends (TypeWrapper<unknown> | BasicType)[]>(...choices: T) => OrType<UnwrapTypes<T>>`
+**Signature:** `<const T extends (TypeWrapper<unknown> | BasicType)[]>(...choices: T) => TypeWrapper<UnwrapTypes<T>>`
 
 **Type Parameters:**
 
@@ -400,11 +607,11 @@ const stringOrNumber = E.or(E.string, E.number);
 const status = E.or("active", "inactive", "pending");
 ```
 
-### array · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### array · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create an array type wrapper with optional length constraints.
 
-**Signature:** `<const T>(inner: TypeWrapper<T>, opts?: { min?: number; max?: number; }) => ArrayType<T>`
+**Signature:** `<const T>(inner: TypeWrapper<T>, opts?: { min?: number; max?: number; }) => TypeWrapper<T[]>`
 
 **Type Parameters:**
 
@@ -424,11 +631,11 @@ const stringArray = E.array(E.string);
 const boundedArray = E.array(E.number, {min: 1, max: 10});
 ```
 
-### literal · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### literal · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a literal type wrapper for a constant value.
 
-**Signature:** `<const T>(value: T) => LiteralType<T>`
+**Signature:** `<const T>(value: T) => TypeWrapper<T>`
 
 **Type Parameters:**
 
@@ -447,11 +654,11 @@ const statusType = E.literal("active");
 const countType = E.literal(42);
 ```
 
-### link · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### link · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a link type wrapper for model relationships.
 
-**Signature:** `<const T extends typeof Model<any>>(TargetModel: T) => LinkType<T>`
+**Signature:** `<const T extends typeof Model<any>>(TargetModel: T) => TypeWrapper<InstanceType<T>>`
 
 **Type Parameters:**
 
@@ -475,7 +682,7 @@ class Post extends E.Model<Post> {
 }
 ```
 
-### index · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### index · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a secondary index on model fields.
 
@@ -502,7 +709,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### primary · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### primary · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a primary index on model fields.
 
@@ -529,7 +736,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### unique · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### unique · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Create a unique index on model fields.
 
@@ -556,7 +763,7 @@ class User extends E.Model<User> {
 }
 ```
 
-### dump · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### dump · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Dump database contents for debugging.
 
@@ -565,7 +772,7 @@ This is primarily useful for development and debugging purposes.
 
 **Signature:** `() => void`
 
-### BaseIndex · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L104)
+### BaseIndex · [abstract class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L128)
 
 Base class for database indexes for efficient lookups on model fields.
 
@@ -581,66 +788,33 @@ Indexes enable fast queries on specific field combinations and enforce uniquenes
 - `MyModel`: - The model class this index belongs to.
 - `_fieldNames`: - Array of field names that make up this index.
 
-#### baseIndex.find · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
-
-Find model instances using flexible range query options.
-
-Supports exact matches, inclusive/exclusive range queries, and reverse iteration.
-For single-field indexes, you can pass values directly or in arrays.
-For multi-field indexes, pass arrays or partial arrays for prefix matching.
+#### baseIndex.find · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Signature:** `(opts?: FindOptions<IndexArgTypes<M, F>>) => IndexRangeIterator<M, F>`
+**Signature:** `(opts?: FindOptions<IndexArgTypes<M, F>>) => IndexRangeIterator<M>`
 
 **Parameters:**
 
-- `opts: FindOptions<IndexArgTypes<M, F>>` (optional) - - Query options object
+- `opts: FindOptions<IndexArgTypes<M, F>>` (optional)
 
-**Returns:** An iterable of model instances matching the query
+#### baseIndex.batchProcess · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Examples:**
+[object Object],[object Object],[object Object]
 
-```typescript
-// Exact match
-for (const user of User.byEmail.find({is: "john@example.com"})) {
-  console.log(user.name);
-}
+**Signature:** `(opts: FindOptions<IndexArgTypes<M, F>> & { limitSeconds?: number; limitRows?: number; }, callback: (row: InstanceType<M>) => void | Promise<...>) => Promise<...>`
 
-// Range query (inclusive)
-for (const user of User.byEmail.find({from: "a@", to: "m@"})) {
-  console.log(user.email);
-}
-
-// Range query (exclusive)
-for (const user of User.byEmail.find({after: "a@", before: "m@"})) {
-  console.log(user.email);
-}
+**Parameters:**
 
-// Open-ended ranges
-for (const user of User.byEmail.find({from: "m@"})) { // m@ and later
-  console.log(user.email);
-}
+- `opts: FindOptions<IndexArgTypes<M, F>> & { limitSeconds?: number; limitRows?: number }` (optional) - - Query options (same as `find()`), plus:
+- `callback: (row: InstanceType<M>) => void | Promise<void>` - - Called for each matching row within a transaction
 
-for (const user of User.byEmail.find({to: "m@"})) { // up to and including m@
-  console.log(user.email);
-}
+#### baseIndex.toString · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-// Reverse iteration
-for (const user of User.byEmail.find({reverse: true})) {
-  console.log(user.email); // Z to A order
-}
+**Signature:** `() => string`
 
-// Multi-field index prefix matching
-for (const item of CompositeModel.pk.find({from: ["electronics", "phones"]})) {
-  console.log(item.name); // All electronics/phones items
-}
+**Parameters:**
 
-// For single-field indexes, you can use the value directly
-for (const user of User.byEmail.find({is: "john@example.com"})) {
-  console.log(user.name);
-}
-```
 
-### UniqueIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### UniqueIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Unique index that stores references to the primary key.
 
@@ -649,7 +823,7 @@ Unique index that stores references to the primary key.
 - `M extends typeof Model` - The model class this index belongs to.
 - `F extends readonly (keyof InstanceType<M> & string)[]` - The field names that make up this index.
 
-#### uniqueIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### uniqueIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Get a model instance by unique index key values.
 
@@ -667,7 +841,7 @@ Get a model instance by unique index key values.
 const userByEmail = User.byEmail.get("john@example.com");
 ```
 
-### PrimaryIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### PrimaryIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Primary index that stores the actual model data.
 
@@ -676,7 +850,7 @@ Primary index that stores the actual model data.
 - `M extends typeof Model` - The model class this index belongs to.
 - `F extends readonly (keyof InstanceType<M> & string)[]` - The field names that make up this index.
 
-#### primaryIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### primaryIndex.get · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
 Get a model instance by primary key values.
 
@@ -694,111 +868,169 @@ Get a model instance by primary key values.
 const user = User.pk.get("john_doe");
 ```
 
-### init · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+#### primaryIndex.getLazy · [method](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-Initialize the database with the specified directory path.
-This function may only be called once. If it is not called before the first transact(),
-the database will be automatically initialized with the default directory.
+Does the same as as `get()`, but will delay loading the instance from disk until the first
+property access. In case it turns out the instance doesn't exist, an error will be thrown
+at that time.
 
-**Signature:** `(dbDir?: string) => void`
+**Signature:** `(...args: IndexArgTypes<M, F>) => InstanceType<M>`
 
 **Parameters:**
 
-- `dbDir?: string` - - Optional directory path for the database (defaults to environment variable $OLMDB_DIR or "./.olmdb").
+- `args: IndexArgTypes<M, F>` - Primary key field values. (Or a single Uint8Array containing the key.)
 
-**Throws:**
+**Returns:** The (lazily loaded) model instance.
 
-- With code "DUP_INIT" if database is already initialized.
-- With code "CREATE_DIR_FAILED" if directory creation fails.
-- With code "LMDB-{code}" for LMDB-specific errors.
+### SecondaryIndex · [class](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L251)
 
-**Examples:**
+Secondary index for non-unique lookups.
 
-```typescript
-init("./my-database");
-```
+**Type Parameters:**
 
-### onCommit · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+- `M extends typeof Model` - The model class this index belongs to.
+- `F extends readonly (keyof InstanceType<M> & string)[]` - The field names that make up this index.
 
-Registers a callback to be executed when the current transaction commits successfully.
-The callback will be executed outside of transaction context.
+### Change · [type](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L94)
 
-**Signature:** `(callback: (commitSeq: number) => void) => void`
+**Type:** `Record<any, any> | "created" | "deleted"`
 
-**Parameters:**
+### Transaction · [interface](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L38)
 
-- `callback: (commitSeq: number) => void` - - Function to execute when transaction commits. It receives the commit sequence, which is an always-increasing number that provides a global ordering of commits, as an argument.
+#### transaction.id · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L41)
 
-**Throws:**
+**Type:** `number`
 
-- If called outside of a transaction context
+#### transaction.instances · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L42)
 
-### onRevert · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+**Type:** `Set<Model<unknown>>`
 
-Registers a callback to be executed when the current transaction is reverted (aborted due to error).
-The callback will be executed outside of transaction context.
+#### transaction.instancesByPk · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L44)
 
-**Signature:** `(callback: (commitSeq: number) => void) => void`
+**Type:** `Map<number, Model<unknown>>`
 
-**Parameters:**
+### DatabaseError · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L156)
 
-- `callback: (commitSeq: number) => void` - - Function to execute when transaction is reverted. It receives the dummy (always 0) commit sequence indicating failure as an argument.
+The DatabaseError class is used to represent errors that occur during database operations.
+It extends the built-in Error class and has a machine readable error code string property.
 
-**Throws:**
+The lowlevel API will throw DatabaseError instances for all database-related errors.
+Invalid function arguments will throw TypeError.
 
-- If called outside of a transaction context
+**Value:** `DatabaseErrorConstructor`
 
-### getTransactionData · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L51)
+### runMigration · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L116)
 
-Retrieves data from the current transaction context.
+Run database migration: upgrade all rows to the latest schema version,
+convert old primary indices, and clean up orphaned secondary indices.
 
-**Signature:** `(key: symbol) => any`
+**Signature:** `(options?: MigrationOptions) => Promise<MigrationResult>`
 
 **Parameters:**
 
-- `key: symbol` - - A symbol key to retrieve data from the current transaction context.
+- `options: MigrationOptions` (optional)
 
-**Returns:** - The value associated with the key, or undefined if not set.
+### MigrationOptions · [interface](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L16)
 
-**Throws:**
+#### migrationOptions.tables · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L18)
 
-- If called outside of a transaction context.
+Limit migration to specific table names.
 
-### setTransactionData · [function](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L44)
+**Type:** `string[]`
 
-Attach some arbitrary user data to the current transaction context, which is
-attached to the currently running (async) task.
+#### migrationOptions.convertOldPrimaries · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L22)
 
-**Signature:** `(key: symbol, value: any) => void`
+Whether to convert old primary indices for known tables (default: true).
 
-**Parameters:**
+**Type:** `boolean`
 
-- `key: symbol` - - A symbol key to store data in the current transaction context.
-- `value: any` - - The value to store.
+#### migrationOptions.deleteOrphanedIndexes · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L30)
 
-**Throws:**
+Whether to delete orphaned secondary/unique indices (default: true).
 
-- If called outside of a transaction context.
+**Type:** `boolean`
 
-**Examples:**
+#### migrationOptions.upgradeVersions · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L38)
+
+Whether to upgrade rows to the latest version (default: true).
+
+**Type:** `boolean`
+
+#### migrationOptions.onProgress · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L42)
+
+Progress callback.
+
+**Type:** `(info: ProgressInfo) => void`
+
+### MigrationResult · [interface](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L47)
+
+#### migrationResult.secondaries · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L49)
+
+Per-table stats for row upgrades.
+
+**Type:** `Record<string, number>`
+
+#### migrationResult.primaries · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L50)
+
+Per-table stats for old primary conversions.
+
+**Type:** `Record<string, number>`
+
+#### migrationResult.conversionFailures · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L56)
+
+Per-table conversion failure counts by reason.
+
+**Type:** `Record<string, Record<string, number>>`
+
+#### migrationResult.orphaned · [member](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L57)
+
+Number of orphaned index entries deleted.
+
+**Type:** `number`
+
+## Schema Migrations
+
+Edinburgh automatically tracks the schema version of each model. When you change fields, field types, indexes, or the `migrate()` function, Edinburgh detects a new schema version.
+
+### What happens automatically (lazy migration)
+
+Changes to regular (non-index) field values are migrated lazily. When a row with an old schema version is loaded from disk, it is deserialized using the old field types and transformed by the optional static `migrate()` function. This is transparent and requires no downtime.
 
 ```typescript
-const MY_SYMBOL = Symbol("myKey");
-await transact(async () => {
-  setTransactionData(MY_SYMBOL, "myValue");
-  await somethingAsync(); // Can be interleaved with other transactions
-  const value = getTransactionData(MY_SYMBOL);
-  console.log(value); // "myValue"
-});
+@E.registerModel
+class User extends E.Model<User> {
+  static pk = E.primary(User, "id");
+  id = E.field(E.identifier);
+  name = E.field(E.string);
+  role = E.field(E.string);  // newly added field
+
+  static migrate(record: Record<string, any>) {
+    record.role ??= "user";  // provide a default for old rows
+  }
+}
 ```
 
-### DatabaseError · [constant](https://github.com/vanviegen/edinburgh/blob/main/src/edinburgh.ts#L120)
+### What requires `migrate-edinburgh`
 
-The DatabaseError class is used to represent errors that occur during database operations.
-It extends the built-in Error class and has a machine readable error code string property.
+The `migrate-edinburgh` CLI tool (or the `runMigration()` API) must be run when:
 
-The lowlevel API will throw DatabaseError instances for all database-related errors.
-Invalid function arguments will throw TypeError.
+- **Adding or removing** secondary or unique indexes
+- **Changing the fields or types** of an existing index
+- A **`migrate()` function changes values** that are used in index fields
 
-**Value:** `DatabaseErrorConstructor`
+The tool populates new indexes, removes orphaned ones, and updates index entries whose values were changed by `migrate()`. It does *not* rewrite primary data rows - lazy migration handles that on read.
 
+```bash
+npx migrate-edinburgh --import ./src/models.ts
+```
+
+Run `npx migrate-edinburgh` to see all of its options.
+
+You can also call `runMigration()` programmatically:
+
+```typescript
+import { runMigration } from "edinburgh";
+
+const result = await runMigration({ tables: ["User"] });
+console.log(result.upgraded);  // { User: 1500 }
+```