solve-engine 2.23.0 → 2.25.0

package/README.md

@@ -52,8 +52,14 @@ const value = engine.evaluateExpression("2 + 2 * 10");
 console.log(value.toNumber()); // 22
 ```
 
-`evaluateExpression` throws an `EngineError` (see `solve-engine/errors`) on a parse or
-evaluation failure, wrap calls with untrusted input in a `try`/`catch`.
+Two kinds of failure, deliberately different. A line the parser cannot read
+(`10 +`), or one that names a variable no line defined, throws an `EngineError`
+(see `solve-engine/errors`), so wrap calls on untrusted input in a `try`/`catch`.
+A line the engine can run but cannot answer (an impossible conversion, a rate it
+has no data for, a live value still loading) comes back as a `Value` whose
+`isError()` or `isPending()` says so, never as a throw, which is the right shape
+for input being typed one character at a time. Check `isFault()` before
+`toNumber()`: a faulted value reads as `0` through it.
 
 For line-oriented input (e.g. a document made of multiple expressions, some referencing
 variables defined on earlier lines), use `evaluateLine`/`parseDocument` instead, see the
@@ -104,13 +110,15 @@ they are:
 
 | Subpath | Purpose |
 |---|---|
-| `solve-engine` | Start here, `ExpressionEngine`, `createEngine`, `IEnginePackage`. |
+| `solve-engine` | Start here: `createEngine`, `ExpressionEngine`, `defineFunction`, `IEnginePackage`, and the result surface `Value`, `ValueType`, `formatValue`. |
 | `solve-engine/engine` | `ExpressionEngine` and its supporting types (`Explanation`, `EngineSnapshot`, etc.) directly, without the package-registration wrapper. |
-| `solve-engine/vm` | The bytecode VM: `Value`/`ValueType`, opcode dispatch, `allocatePluginFunctionIndex`. |
-| `solve-engine/format` | Turning a `Value` into a display string (numbers, dates, units, vectors, ...). |
+| `solve-engine/vm` | The bytecode VM: `Value`/`ValueType`, the value constructors, opcode dispatch. |
+| `solve-engine/format` | Turning a `Value` into a display string (numbers, dates, units, vectors, ...) and the formatting settings. |
 | `solve-engine/language` | Editor-agnostic language service: token categories, completions, highlighting. |
 | `solve-engine/packages` | The built-in packages (arithmetic, datetime, time, dice, uom, currency, vector, conditionals, converters, mathphrases, ...). |
-| `solve-engine/constants` | Engine configuration types and defaults (`EngineConfig`, `VMConfig`, ...). |
+| `solve-engine/constants` | Engine configuration types and defaults (`EngineConfig`, `VMConfig`, `NetworkConfig`, ...). |
+| `solve-engine/worker` | Off-main-thread evaluation: `createWorkerEngine`, `startWorkerRuntime`, the transports and the result DTOs. |
+| `solve-engine/testing` | A test kit for package authors: `createTestEngine`, `expectExpression`, `expectPackage`. |
 
 The following subpaths are **advanced-public**, everything a third-party package author
 needs to extend the engine, but with a looser stability contract than the tier above (these
@@ -132,32 +140,70 @@ Anything not listed above (`telemetry`, `cache`, `diagnostics`, `types`, `worker
 internal and not part of the package's public contract, it may change or disappear between
 minor versions without notice.
 
+## Adding a function
+
+The shortest way to teach the engine a new name is `defineFunction`: declare the
+name, the argument types and the return type, and get back a package that plugs
+in beside the built-ins. No parser, no bytecode.
+
+```typescript
+import { createEngine, defineFunction } from "solve-engine";
+
+const vat = defineFunction({
+  name: "vat",
+  args: [{ name: "amount", type: "number" }],
+  returns: "number",
+  call: (amount) => amount * 1.2,
+});
+
+const engine = createEngine({ extraPackages: [vat] });
+engine.evaluateExpression("vat(100)").toNumber(); // 120
+```
+
+Arguments are a fixed list of `number`, `string` or `boolean`, and `call` is
+synchronous; a mismatched call produces a structured error naming the argument.
+Anything beyond that shape (a unit-bearing argument, a phrase rather than a
+call, live data) is a package proper.
+
 ## Authoring a package
 
-A **package** (`IEnginePackage`) is a plain data descriptor bundling everything needed to
-extend the engine with a new domain: custom tokens, parselets, VM opcode handlers, and
-optional async resolvers. See
-[`solve-engine/api`'s `IEnginePackage`](https://github.com/LiamRiddell/solve-engine/blob/main/packages/engine/src/api/PackageRegistry.ts) for the full field list
-with inline documentation and examples for each field.
+A **package** (`IEnginePackage`) is a plain data descriptor bundling everything
+needed to extend the engine with a new domain: token vocabulary, normaliser
+rules, parselets, plugin functions, `as` converters, completions, and optional
+async resolvers. The [package author guides](https://liamriddell.github.io/solve-engine/packages/authoring-a-package/)
+walk through each extension point; [`IEnginePackage`](https://github.com/LiamRiddell/solve-engine/blob/main/packages/engine/src/api/PackageRegistry.ts)
+carries the full field list with inline documentation.
 
-Minimal shape:
+Minimal shape, a keyword that calls a function:
 
 ```typescript
-import { allocatePluginFunctionIndex } from "solve-engine/vm";
 import type { IEnginePackage } from "solve-engine";
-
-const MY_FN_IDX = allocatePluginFunctionIndex();
+import type { PrefixParselet } from "solve-engine/parser";
+import { stringValue } from "solve-engine/vm";
+
+const reverseParselet: PrefixParselet = {
+  category: "Text",
+  parse(parser, _token, builder) {
+    parser.consume("LPAREN");
+    parser.parseExpression(0, builder);
+    parser.consume("RPAREN");
+    // By name; the engine assigns the index when the package registers.
+    builder.emitPluginCall("reverse", 1);
+  },
+};
 
 export const MY_PACKAGE: IEnginePackage = {
-  name: "MyPackage",
-  // engineVersion: "^0.1.0", // optional, see below
-  prefixParselets: [{ tokenType: "MY_FUNC", parselet: new MyParselet() }],
-  pluginFunctions: [{ index: MY_FN_IDX, handler: (args) => /* ... */ }],
+  name: "my-package",
+  // engineVersion: "^2.0.0", // optional, see below
+  lexerVocabulary: { keywords: { reverse: "REVERSE_FN" } },
+  prefixParselets: { REVERSE_FN: reverseParselet },
+  pluginFunctions: { reverse: ([text]) => stringValue([...String(text.value)].reverse().join("")) },
 };
 ```
 
-Register it either as one of the packages passed to the `ExpressionEngine` constructor, or
-at runtime via `ExpressionEngine.registerPackage()` / `unregisterPackage()`.
+Register it as one of the packages passed to the `ExpressionEngine` constructor
+(or `createEngine`'s `extraPackages`), or at runtime via
+`ExpressionEngine.registerPackage()` / `unregisterPackage()`.
 
 ### Declaring engine-version compatibility
 
@@ -210,17 +256,21 @@ published package, see `files` in `package.json`, only `dist/` ships):
 
 ## Known limitations
 
-**Cross-instance isolation is partial.** Plugin functions and the opcode registry are
-now owned per `ExpressionEngine`, so two engines with different package sets no longer
-interfere across those. The lexer and the currency exchange rates
-are still module-level singletons, so full isolation between two engines in one process
-cannot yet be assumed. Tracked as "L1, EngineContext"; three of its five migrations have
-landed and the remaining two are a prerequisite for 1.0.0 proper.
-
-**Async results need a host hook.** `AsyncResolutionBatcher.onLineResult` is the only
-mechanism that patches a resolved async value back into the document model, and it is not
-wired inside the package. A host that does not supply it gets async values that never
-resolve, with no error to explain why.
+**Exchange rates are shared across engines, by design.** Plugin functions, the
+opcode registry and the lexer are owned per `ExpressionEngine`, so two engines
+with different package sets do not interfere. The currency rate cache is the one
+deliberate exception: rates are market data with a fifteen-minute freshness
+window, and two engines in one process fetching the same pair separately could
+disagree about it. A host that primes rates (`currencyExchangeService.primeRates`)
+primes them for every engine in the process.
+
+**A live value reaches the document only through the event stream.** When a
+fetch lands, the engine does not push the new value at you; it emits a
+`lines-updated` event on `engine.getEventStream()` naming the lines to
+re-evaluate. A host that reads neither that stream nor sets
+`AsyncResolutionBatcher.onLineResult` sees the line stay pending, and a warning
+says so once. The [async guide](https://liamriddell.github.io/solve-engine/guide/async-and-live-data/)
+shows the loop.
 
 ## Development
 

package/dist/{BytecodeBuilder-aqVa7Plx.d.cts → BytecodeBuilder-DSWKZi4f.d.cts}

@@ -199,6 +199,8 @@ declare class BytecodeBuilder {
     private numbers;
     private strings;
     private stringIndex;
+    /** Value (or {@link NEGATIVE_ZERO_KEY}) to its slot in `numbers`, so a repeated literal is emitted once. */
+    private numberIndex;
     private _hasAsync;
     private userFunctionBodies;
     private anonymousBodies;
@@ -222,17 +224,17 @@ declare class BytecodeBuilder {
     /** Emit an {@link OpCode} instruction. */
     emitOpcode(op: OpCode): void;
     /**
-     * Emit a numeric literal: appends `n` to the program's constant pool and
-     * writes its index into the opcode stream (read back by the VM as e.g.
-     * `PUSH_NUMBER <idx>`).
+     * Emit a numeric literal: interns `n` into the program's constant pool
+     * (deduplicated, like {@link emitString}) and writes its index into the
+     * opcode stream (read back by the VM as e.g. `PUSH_NUMBER <idx>`).
      *
-     * Numeric constants are NOT deduplicated (unlike {@link emitString})
-     * every call appends a new entry, so an expression with more than
-     * {@link MAX_CONSTANT_POOL_INDEX}+1 distinct numeric-literal occurrences
-     * throws rather than silently wrapping the index (see
-     * `MAX_CONSTANT_POOL_INDEX`'s doc for what that would otherwise do).
+     * Deduplicated so that the 256-entry pool counts distinct values rather
+     * than occurrences: a long line of repeated literals used to exhaust it
+     * long before it held 256 different numbers. Negative zero keeps its own
+     * slot, because `1 / -0` is not `1 / 0`; NaN shares one, since every NaN
+     * reads the same.
      *
-     * @throws If the constant pool would exceed 256 entries.
+     * @throws If the pool would exceed 256 distinct entries.
      */
     emitNumber(n: number): void;
     /**

package/dist/{BytecodeBuilder-aqVa7Plx.d.ts → BytecodeBuilder-DSWKZi4f.d.ts}

@@ -199,6 +199,8 @@ declare class BytecodeBuilder {
     private numbers;
     private strings;
     private stringIndex;
+    /** Value (or {@link NEGATIVE_ZERO_KEY}) to its slot in `numbers`, so a repeated literal is emitted once. */
+    private numberIndex;
     private _hasAsync;
     private userFunctionBodies;
     private anonymousBodies;
@@ -222,17 +224,17 @@ declare class BytecodeBuilder {
     /** Emit an {@link OpCode} instruction. */
     emitOpcode(op: OpCode): void;
     /**
-     * Emit a numeric literal: appends `n` to the program's constant pool and
-     * writes its index into the opcode stream (read back by the VM as e.g.
-     * `PUSH_NUMBER <idx>`).
+     * Emit a numeric literal: interns `n` into the program's constant pool
+     * (deduplicated, like {@link emitString}) and writes its index into the
+     * opcode stream (read back by the VM as e.g. `PUSH_NUMBER <idx>`).
      *
-     * Numeric constants are NOT deduplicated (unlike {@link emitString})
-     * every call appends a new entry, so an expression with more than
-     * {@link MAX_CONSTANT_POOL_INDEX}+1 distinct numeric-literal occurrences
-     * throws rather than silently wrapping the index (see
-     * `MAX_CONSTANT_POOL_INDEX`'s doc for what that would otherwise do).
+     * Deduplicated so that the 256-entry pool counts distinct values rather
+     * than occurrences: a long line of repeated literals used to exhaust it
+     * long before it held 256 different numbers. Negative zero keeps its own
+     * slot, because `1 / -0` is not `1 / 0`; NaN shares one, since every NaN
+     * reads the same.
      *
-     * @throws If the constant pool would exceed 256 entries.
+     * @throws If the pool would exceed 256 distinct entries.
      */
     emitNumber(n: number): void;
     /**

package/dist/{Configuration-BGQn-cJ8.d.cts → Configuration-DnzYPmoK.d.cts}

@@ -207,6 +207,37 @@ interface BackgroundRefreshConfig {
     /** Master switch. When false, no timers run and every value refreshes only on re-evaluation. */
     readonly enabled: boolean;
 }
+/**
+ * Whether the engine may reach the network at all.
+ *
+ * On by default, which is the historic behaviour: a city name or a currency
+ * pair in a line fetches live data from a public endpoint as soon as the
+ * line evaluates. A host embedding the engine somewhere that must not make
+ * outbound requests (an offline document, a sandboxed evaluation, a tenant
+ * with egress rules) switches it off here and gets the same engine with no
+ * egress: every live-data form answers with a `NETWORK_DISABLED` error
+ * naming the setting, instead of a request.
+ *
+ * The gate closes before any request is made. It stops every async
+ * resolver a package registers (weather, currency, stocks, crypto,
+ * knowledge, and any host-supplied one) unless the resolver declares
+ * itself `local` (it reads engine state, never a network; the engine's own
+ * global-variable resolver is the one built-in case). It also refuses the
+ * result of a plugin function that returns a promise. That last case is a
+ * boundary rather than a guarantee: such a function has already run by the
+ * time the engine sees the promise, so a package that fetches inside a
+ * plugin function rather than through an async resolver may still have
+ * started its request. The documented shape for live data is the resolver
+ * (`createQueryResolver`), which this gate stops cold.
+ *
+ * Rates a host primes by hand (`currencyExchangeService.primeRates`) keep
+ * working with the network off, which is what an offline host with its own
+ * rate table wants.
+ */
+interface NetworkConfig {
+    /** Master switch. When false, no async resolver runs and no live value is fetched. */
+    readonly enabled: boolean;
+}
 /**
  * Virtual Machine configuration.
  * Controls the internal bytecode VM that executes compiled expressions.
@@ -321,6 +352,8 @@ interface EngineConfig {
     readonly diagnostic: DiagnosticConfig;
     /** Proactive background refresh of live async values */
     readonly backgroundRefresh: BackgroundRefreshConfig;
+    /** Whether live data may be fetched at all. See {@link NetworkConfig}. */
+    readonly network: NetworkConfig;
 }
 /**
  * A partial {@link EngineConfig} override, one section at a time.

package/dist/{Configuration-BGQn-cJ8.d.ts → Configuration-DnzYPmoK.d.ts}

@@ -207,6 +207,37 @@ interface BackgroundRefreshConfig {
     /** Master switch. When false, no timers run and every value refreshes only on re-evaluation. */
     readonly enabled: boolean;
 }
+/**
+ * Whether the engine may reach the network at all.
+ *
+ * On by default, which is the historic behaviour: a city name or a currency
+ * pair in a line fetches live data from a public endpoint as soon as the
+ * line evaluates. A host embedding the engine somewhere that must not make
+ * outbound requests (an offline document, a sandboxed evaluation, a tenant
+ * with egress rules) switches it off here and gets the same engine with no
+ * egress: every live-data form answers with a `NETWORK_DISABLED` error
+ * naming the setting, instead of a request.
+ *
+ * The gate closes before any request is made. It stops every async
+ * resolver a package registers (weather, currency, stocks, crypto,
+ * knowledge, and any host-supplied one) unless the resolver declares
+ * itself `local` (it reads engine state, never a network; the engine's own
+ * global-variable resolver is the one built-in case). It also refuses the
+ * result of a plugin function that returns a promise. That last case is a
+ * boundary rather than a guarantee: such a function has already run by the
+ * time the engine sees the promise, so a package that fetches inside a
+ * plugin function rather than through an async resolver may still have
+ * started its request. The documented shape for live data is the resolver
+ * (`createQueryResolver`), which this gate stops cold.
+ *
+ * Rates a host primes by hand (`currencyExchangeService.primeRates`) keep
+ * working with the network off, which is what an offline host with its own
+ * rate table wants.
+ */
+interface NetworkConfig {
+    /** Master switch. When false, no async resolver runs and no live value is fetched. */
+    readonly enabled: boolean;
+}
 /**
  * Virtual Machine configuration.
  * Controls the internal bytecode VM that executes compiled expressions.
@@ -321,6 +352,8 @@ interface EngineConfig {
     readonly diagnostic: DiagnosticConfig;
     /** Proactive background refresh of live async values */
     readonly backgroundRefresh: BackgroundRefreshConfig;
+    /** Whether live data may be fetched at all. See {@link NetworkConfig}. */
+    readonly network: NetworkConfig;
 }
 /**
  * A partial {@link EngineConfig} override, one section at a time.

package/dist/{EngineError-DTk7I7hZ.d.cts → EngineError-fR1K1rvx.d.cts}

@@ -38,6 +38,8 @@ declare const CoreErrorCodes: {
     readonly UNEXPECTED_TRAILING_TOKEN: "UNEXPECTED_TRAILING_TOKEN";
     readonly PARSE_ERROR: "PARSE_ERROR";
     readonly TOO_MANY_NUMERIC_CONSTANTS: "TOO_MANY_NUMERIC_CONSTANTS";
+    /** A raw bytecode operand outside 0 to 255, or a jump patch outside the emitted stream (`parser/BytecodeBuilder.ts`'s `emitIndex`, `emitByte`, `patchJump`). `build()` keeps one byte per operand, so 300 used to become 44 with no error and the program read the wrong constant or plugin function. A package-authoring fault, reported at compile time. */
+    readonly BYTECODE_OPERAND_OUT_OF_RANGE: "BYTECODE_OPERAND_OUT_OF_RANGE";
     readonly TOO_MANY_STRING_CONSTANTS: "TOO_MANY_STRING_CONSTANTS";
     readonly NO_MATCHING_PHRASE_ALTERNATIVE: "NO_MATCHING_PHRASE_ALTERNATIVE";
     readonly INVALID_PHRASE_PATTERN: "INVALID_PHRASE_PATTERN";
@@ -103,6 +105,8 @@ declare const CoreErrorCodes: {
     readonly INTERNAL_MISSING_FUNCTION_BODY: "INTERNAL_MISSING_FUNCTION_BODY";
     /** `MAP_INVOKE`/`REDUCE_INVOKE`'s anonymous-body-index lookup failing. Same class as `INTERNAL_MISSING_FUNCTION_BODY` above, for the `anonymousBodies` side-table instead of `userFunctionBodies`. */
     readonly INTERNAL_MISSING_ANONYMOUS_BODY: "INTERNAL_MISSING_ANONYMOUS_BODY";
+    /** `CALL_BUILTIN` naming an index no builtin is registered at. It used to pop its arguments and push nothing, so the opcode after it read a neighbour's operand as its own; now an Error value on the stack, the shape `map`/`reduce` already used for the same fault. Bytecode-level, so unreachable from source without a compiler or snapshot fault. */
+    readonly UNKNOWN_BUILTIN_FUNCTION: "UNKNOWN_BUILTIN_FUNCTION";
     /** An operand byte read past the end of the stream: the program ends in the middle of an instruction. */
     readonly MALFORMED_BYTECODE_TRUNCATED: "MALFORMED_BYTECODE_TRUNCATED";
     /** A constant-pool operand indexing a `numbers`/`strings` entry that does not exist, or that is not of the pool's type. */
@@ -164,6 +168,8 @@ declare const CoreErrorCodes: {
     readonly UNKNOWN_SAVINGS_PERIOD: "UNKNOWN_SAVINGS_PERIOD";
     /** Colon-separated numbers that are not a time any clock can show ("24:00", "9:60", "100:5"). Raised by the labeled-line fallback, which used to answer them with whatever stood after the colon. */
     readonly INVALID_TIME_LITERAL: "INVALID_TIME_LITERAL";
+    /** A live-data form evaluated on an engine whose host switched the network off (`network.enabled: false`, see `constants/Configuration.ts`'s `NetworkConfig`). A recoverable Error value, raised by the VM for a currency conversion with no primed rate and for a plugin function that returned a promise, and by `createQueryResolver`'s plugin function when its preflight was skipped. Names the setting, so the reader knows it is policy rather than an outage. */
+    readonly NETWORK_DISABLED: "NETWORK_DISABLED";
     /** `fromJSON()` handed an object that is not a snapshot at all, or whose serialised-shape version does not match this engine's reader. The versioning gate that refuses an incompatible snapshot clearly rather than restoring it wrongly. See `engine/EngineSnapshot.ts`'s `assertRestorable()`. */
     readonly SNAPSHOT_VERSION_MISMATCH: "SNAPSHOT_VERSION_MISMATCH";
     /** A snapshot with the right envelope but internally inconsistent contents (an unrecognised number sentinel, an unknown value tag). Distinct from a version mismatch: the format is right, the payload is not. */

package/dist/{EngineError-DTk7I7hZ.d.ts → EngineError-fR1K1rvx.d.ts}

@@ -38,6 +38,8 @@ declare const CoreErrorCodes: {
     readonly UNEXPECTED_TRAILING_TOKEN: "UNEXPECTED_TRAILING_TOKEN";
     readonly PARSE_ERROR: "PARSE_ERROR";
     readonly TOO_MANY_NUMERIC_CONSTANTS: "TOO_MANY_NUMERIC_CONSTANTS";
+    /** A raw bytecode operand outside 0 to 255, or a jump patch outside the emitted stream (`parser/BytecodeBuilder.ts`'s `emitIndex`, `emitByte`, `patchJump`). `build()` keeps one byte per operand, so 300 used to become 44 with no error and the program read the wrong constant or plugin function. A package-authoring fault, reported at compile time. */
+    readonly BYTECODE_OPERAND_OUT_OF_RANGE: "BYTECODE_OPERAND_OUT_OF_RANGE";
     readonly TOO_MANY_STRING_CONSTANTS: "TOO_MANY_STRING_CONSTANTS";
     readonly NO_MATCHING_PHRASE_ALTERNATIVE: "NO_MATCHING_PHRASE_ALTERNATIVE";
     readonly INVALID_PHRASE_PATTERN: "INVALID_PHRASE_PATTERN";
@@ -103,6 +105,8 @@ declare const CoreErrorCodes: {
     readonly INTERNAL_MISSING_FUNCTION_BODY: "INTERNAL_MISSING_FUNCTION_BODY";
     /** `MAP_INVOKE`/`REDUCE_INVOKE`'s anonymous-body-index lookup failing. Same class as `INTERNAL_MISSING_FUNCTION_BODY` above, for the `anonymousBodies` side-table instead of `userFunctionBodies`. */
     readonly INTERNAL_MISSING_ANONYMOUS_BODY: "INTERNAL_MISSING_ANONYMOUS_BODY";
+    /** `CALL_BUILTIN` naming an index no builtin is registered at. It used to pop its arguments and push nothing, so the opcode after it read a neighbour's operand as its own; now an Error value on the stack, the shape `map`/`reduce` already used for the same fault. Bytecode-level, so unreachable from source without a compiler or snapshot fault. */
+    readonly UNKNOWN_BUILTIN_FUNCTION: "UNKNOWN_BUILTIN_FUNCTION";
     /** An operand byte read past the end of the stream: the program ends in the middle of an instruction. */
     readonly MALFORMED_BYTECODE_TRUNCATED: "MALFORMED_BYTECODE_TRUNCATED";
     /** A constant-pool operand indexing a `numbers`/`strings` entry that does not exist, or that is not of the pool's type. */
@@ -164,6 +168,8 @@ declare const CoreErrorCodes: {
     readonly UNKNOWN_SAVINGS_PERIOD: "UNKNOWN_SAVINGS_PERIOD";
     /** Colon-separated numbers that are not a time any clock can show ("24:00", "9:60", "100:5"). Raised by the labeled-line fallback, which used to answer them with whatever stood after the colon. */
     readonly INVALID_TIME_LITERAL: "INVALID_TIME_LITERAL";
+    /** A live-data form evaluated on an engine whose host switched the network off (`network.enabled: false`, see `constants/Configuration.ts`'s `NetworkConfig`). A recoverable Error value, raised by the VM for a currency conversion with no primed rate and for a plugin function that returned a promise, and by `createQueryResolver`'s plugin function when its preflight was skipped. Names the setting, so the reader knows it is policy rather than an outage. */
+    readonly NETWORK_DISABLED: "NETWORK_DISABLED";
     /** `fromJSON()` handed an object that is not a snapshot at all, or whose serialised-shape version does not match this engine's reader. The versioning gate that refuses an incompatible snapshot clearly rather than restoring it wrongly. See `engine/EngineSnapshot.ts`'s `assertRestorable()`. */
     readonly SNAPSHOT_VERSION_MISMATCH: "SNAPSHOT_VERSION_MISMATCH";
     /** A snapshot with the right envelope but internally inconsistent contents (an unrecognised number sentinel, an unknown value tag). Distinct from a version mismatch: the format is right, the payload is not. */

package/dist/{Lexer-CqagTewQ.d.ts → Lexer-Bmctxl0n.d.ts}

@@ -1,4 +1,5 @@
 import { T as Token } from './Token-CbP_OutD.js';
+import { E as EngineError } from './EngineError-fR1K1rvx.js';
 
 /** Trie node for multi-word phrase matching. */
 interface PhraseNode {
@@ -101,6 +102,14 @@ interface ScanLineResult {
     tokens: Token[];
     /** Inline solve spans found in this line (empty if none). */
     inlineSolves: InlineSolveSpan[];
+    /**
+     * The structured error the tokeniser raised on this line, if it raised one
+     * (an unterminated string literal is the one it can raise today). The line
+     * carries no tokens then, and the rest of the document is unaffected: a
+     * half-typed quote used to escape {@link ExpressionLexer.scanDocument} as
+     * a throw and take every other line's result down with it.
+     */
+    error?: EngineError;
 }
 /**
  * A token, as the lexer produces it.
@@ -208,16 +217,6 @@ interface LexerVocabulary {
  */
 declare class ExpressionLexer {
     private static readonly CHAR_CLASS;
-    /**
-     * Configured TokenLookup from TokenClassRegistry. When set, replaces
-     * the internal keyword map and unit set with registry-built equivalents.
-     * Enables data-driven keyword/unit registration across locale keywords,
-     * provider keywords, and plugins.
-     *
-     * Set at construction time via the constructor parameter. Plugin-registered
-     * keywords/units (via registerVocabulary()) are checked alongside
-     * the configuredLookup, neither source is bypassed.
-     */ private configuredLookup;
     private input;
     private pos;
     private len;
@@ -227,7 +226,16 @@ declare class ExpressionLexer {
     private keywordMap;
     private mergedKeywords;
     private mergedUnits;
-    private pluginKeywordMap;
+    /**
+     * Who registered each plugin keyword, unit and two-character operator, in
+     * registration order. Two packages may claim the same word (the
+     * compatibility index warns, and the last registered wins, as before); the
+     * owner lists make unregistration exact, so removing one package hands the
+     * word back to the other rather than deleting it for both.
+     */
+    private pluginKeywordOwners;
+    private pluginUnitOwners;
+    private pluginOperatorOwners;
     private pluginOperators;
     private pluginUnits;
     private hasPluginOps;
@@ -235,7 +243,7 @@ declare class ExpressionLexer {
     private locale;
     /**
      * Inline solve spans collected during the most recent tokenization pass.
-     * Populated by [Symbol.iterator]() and consumed by scanDocument().
+     * Populated by tokenizeInto() and consumed by scanDocument().
      */
     _inlineSolveSpans: InlineSolveSpan[];
     private pluginRawLinePatterns;
@@ -260,7 +268,14 @@ declare class ExpressionLexer {
     private matchRawLine;
     /** Rebuild merged keyword and unit collections after plugin registration. */
     private rebuildMergedUnits;
-    constructor(localeCode?: string, lookup?: TokenLookup);
+    /**
+     * @param localeCode - Locale whose keyword table seeds the lexer.
+     * @param _lookup - Ignored. The lexer built its keyword, unit and phrase
+     *   tables from the locale and the registered packages and never read the
+     *   lookup it was handed; the parameter stays so existing callers compile.
+     *   @deprecated Removed in 3.0.
+     */
+    constructor(localeCode?: string, _lookup?: TokenLookup);
     /**
      * Register a plugin to extend the lexer with custom tokens.
      *
@@ -276,6 +291,10 @@ declare class ExpressionLexer {
      * that conflicts with a built-in one.
      */
     registerVocabulary(plugin: LexerVocabulary): void;
+    /** Record `owner`'s claim on `key`. The newest claim is the one in force. */
+    private claim;
+    /** Drop `owner`'s claim on `key`, and return the claim now in force, if any is left. */
+    private release;
     /**
      * Unregister a plugin, removing its custom tokens from the lexer.
      *
@@ -307,9 +326,9 @@ declare class ExpressionLexer {
      *   lines that classifyLine() marks as having inline solves.
      *
      * Tokenization is scoped to each line by temporarily restricting
-     * `this.len` to the line end position, so the [Symbol.iterator]
-     * generator naturally stops at the line boundary. After tokenization,
-     * `this.len` is restored and `this.pos` advances past the newline.
+     * `this.len` to the line end position, so tokenizeInto() naturally
+     * stops at the line boundary. After tokenization, `this.len` is
+     * restored and `this.pos` advances past the newline.
      *
      * @param text The full document text (with newlines).
      * @returns Array of ScanLineResult, one per line, in document order.
@@ -318,9 +337,7 @@ declare class ExpressionLexer {
     /**
      * Tokenize an expression string into an array of Tokens.
      *
-     * Delegates to the lazy [Symbol.iterator]() generator and collects all
-     * yielded tokens via Array.from(). For memory-sensitive use cases, prefer
-     * iterating the lexer directly with for...of to avoid array allocation.
+     * Runs {@link tokenizeInto} into a fresh array.
      *
      * Optimizations:
      *  - CHAR_CLASS jump table (Uint8Array) → switch on small integers
@@ -332,20 +349,31 @@ declare class ExpressionLexer {
      */
     tokenizeAll(): Token[];
     /**
-     * Lazy token-by-token generator. Yields each token without allocating an
-     * intermediate Token[] array. Supports for...of and spread usage.
+     * Scan the current input and append every token to `out`.
+     *
+     * This is the scanner itself: {@link tokenizeAll} and the iterator are
+     * thin wrappers over it. It pushes into an array the caller owns rather
+     * than yielding, because a generator paid a resume per token and
+     * `Array.from` a second pass on top, and because a caller that catches a
+     * tokeniser fault (highlighting a line with an unterminated quote) still
+     * holds the tokens read before it.
      *
-     * Usage:
-     *   for (const t of lexer) { ... }  // lazy, no array allocation
-     *   const tokens = [...lexer];       // materializes via spread
-     *   const tokens = lexer.tokenizeAll(); // materializes via Array.from()
+     * IMPORTANT: `this.len` is read ONCE on entry (const len = this.len).
+     * `scanDocument()` relies on this to scope a pass to a single line by
+     * restricting `this.len` to the line end before calling. Do not re-read
+     * `this.len` mid-loop without also updating `scanDocument()`.
      *
-     * IMPORTANT: This generator captures `this.len` ONCE at creation time
-     * (const len = this.len). `scanDocument()` relies on this behavior to
-     * scope tokenization to a single line by temporarily restricting
-     * `this.len` to the line end position before creating the iterator.
-     * Do NOT refactor to re-read `this.len` mid-loop without also updating
-     * `scanDocument()`.
+     * @param out - The array to append to. Left as it was if the input is empty.
+     */
+    tokenizeInto(out: Token[]): void;
+    /**
+     * Iterate the tokens of the current input.
+     *
+     * Runs {@link tokenizeInto} first, so the tokens are the same as before,
+     * but a fault part way through the line is thrown from the first
+     * `next()` rather than at the token it occurred on. A caller that wants
+     * the tokens read before a fault calls {@link tokenizeInto} with its own
+     * array.
      */
     [Symbol.iterator](): Generator<Token, void, undefined>;
     /**
@@ -456,6 +484,27 @@ declare class ExpressionLexer {
      * `#12345` (length 5), `#ff0000zz` (trailing word) do not.
      */
     private matchHexColourEnd;
+    /**
+     * Whether an inline-solve opener (`s` + backtick) occurs in `[from, end)`.
+     *
+     * Bounded by hand rather than written as `input.indexOf("s\`", from)`. This
+     * used to be the `indexOf` form with an `idx < end` check afterwards, which
+     * is correct but not bounded: when {@link scanDocument} is classifying,
+     * `this.input` is the whole document, so a line with no marker scanned to
+     * the END OF THE DOCUMENT before the check could reject the hit. Every line
+     * paid for every line after it, and a whole-document parse was quadratic in
+     * line count (measured at 28.9 us per line at 10,000 lines against 10.1 us
+     * at 1,000, and two thirds of the parse's self time in a profile). A prose
+     * document with many candidate `s` characters was worse still. The
+     * wikilink close in {@link indexOfWithin} had the same shape.
+     */
+    private hasInlineSolveWithin;
+    /**
+     * `input.indexOf(needle, from)` restricted to `[from, end)`, returning -1
+     * when the needle does not occur wholly before `end`. See
+     * {@link hasInlineSolveWithin} for why the plain `indexOf` was not enough.
+     */
+    private indexOfWithin;
     private classifyFromPositions;
     /**
      * Classify a single line of markdown text.
@@ -530,11 +579,11 @@ declare class Lexer {
     private tokenIdx;
     /**
      * @param localeCode - Locale code (e.g., "en", "de"). Defaults to "en".
-     * @param tokenLookup - Optional TokenLookup from TokenClassRegistry.
-     *   When provided, configures ExpressionLexer to use registry-built
-     *   keyword/unit/phrase lookups instead of internal instance maps.
+     * @param _tokenLookup - Ignored. The lexer never read the lookup it was
+     *   handed; the parameter stays so existing callers compile.
+     *   @deprecated Removed in 3.0.
      */
-    constructor(localeCode?: string, tokenLookup?: TokenLookup);
+    constructor(localeCode?: string, _tokenLookup?: TokenLookup);
     reset(input: string, state?: LexerState): void;
     /**
      * Classify a single line of markdown text (Phase B).