@malloydata/malloy 0.0.335 → 0.0.336

package/CONTEXT.md

@@ -10,9 +10,9 @@ packages/malloy/
 │   ├── lang/              # Translator: Parse tree → AST → IR (see src/lang/CONTEXT.md)
 │   ├── model/             # Compiler: IR → SQL (see src/model/CONTEXT.md)
 │   ├── dialect/           # Database-specific SQL generation
-│   ├── api/               # Public API interfaces
-│   ├── connection/        # Database connection abstractions
-│   └── malloy.ts          # Main entry point
+│   ├── api/               # API layers (see src/api/CONTEXT.md)
+│   │   └── foundation/    # Public API classes (see MALLOY_API.md)
+│   └── connection/        # Database connection abstractions
 ```
 
 ## Two-Phase Architecture
@@ -53,5 +53,6 @@ For detailed information about the compiler, see [src/model/CONTEXT.md](src/mode
 ## Subsystem Context
 
 For deeper details on specific subsystems:
+- [MALLOY_API.md](MALLOY_API.md) - Public API classes (Model, PreparedQuery, Runtime, Materializers)
 - [src/lang/CONTEXT.md](src/lang/CONTEXT.md) - Translator architecture (grammar, AST, IR generation)
 - [src/model/CONTEXT.md](src/model/CONTEXT.md) - Compiler architecture (SQL generation, expression compilation)

package/MALLOY_API.md

@@ -0,0 +1,129 @@
+# Foundation API - Public API Classes
+
+The directory `src/api/foundation/` is the main public API surface for the Malloy package. It exports classes that wrap internal IR types and provide user-facing functionality. See `src/api/CONTEXT.md` for a complete overview of all API layers.
+
+## Core Data Classes
+
+### `Model`
+Wraps `ModelDef` (internal IR). Represents a compiled Malloy model.
+
+- `explores: Explore[]` - All sources in the model (wrapped)
+- `getPreparedQueryByName(name): PreparedQuery`
+- `getPreparedQueryByIndex(index): PreparedQuery`
+- `getExploreByName(name): Explore`
+- `_modelDef` - Escape hatch to raw IR
+
+### `PreparedQuery`
+Wraps `Query` (internal IR) + `ModelDef`. A query that can be compiled to SQL.
+
+- `getPreparedResult(options?): PreparedResult` - Compile to SQL
+- `dialect: string`
+- `name?: string`
+- `_query`, `_modelDef` - Escape hatches
+
+**Issue:** Each `getPreparedResult()` call creates a new `QueryModel`, reprocessing the ModelDef.
+
+### `PreparedResult`
+Wraps `CompiledQuery` (internal). The compiled SQL and metadata.
+
+- `sql: string` - The generated SQL
+- `connectionName: string` - Which connection to run against
+- `resultExplore: Explore` - Schema of the result
+
+### `Result`
+Extends `PreparedResult`. Adds actual query result data.
+
+- `data: DataArray` - The result rows
+- `totalRows: number`
+- `runStats: QueryRunStats`
+
+### `Explore`
+Wraps `StructDef` (internal). Represents a source (historical name was "explore").
+
+- `name: string`
+- `allFields: Field[]`
+- `getFieldByName(name): Field`
+- `getQueryByName(name): PreparedQuery` - Get a view as a query
+
+### `ExploreField`
+Extends `Explore`. A joined source (appears as a field in parent).
+
+### `Query`
+Wraps `TurtleDef`. Represents a view definition. **Different from `PreparedQuery`.**
+
+### `QueryField`
+Extends `Query`. A view that appears as a field in an explore.
+
+### `AtomicField` and subclasses
+Field wrappers: `StringField`, `NumberField`, `DateField`, `TimestampField`, `BooleanField`, `JSONField`, `UnsupportedField`
+
+## Runtime & Materializer Classes
+
+### `Runtime`
+Entry point for loading and running Malloy. Holds URLReader, connections, event stream.
+
+- `loadModel(source): ModelMaterializer`
+- `loadQuery(query): QueryMaterializer`
+- `getModel(source): Promise<Model>`
+- `getQuery(query): Promise<PreparedQuery>`
+
+### `SingleConnectionRuntime`
+Extends `Runtime`. For single-connection use cases.
+
+### `ConnectionRuntime`
+Extends `Runtime`. Holds array of connections.
+
+### `ModelMaterializer`
+Fluent builder for loading models. Returned by `runtime.loadModel()`.
+
+- `getModel(): Promise<Model>`
+- `loadQueryByName(name): QueryMaterializer`
+- `loadExploreByName(name): ExploreMaterializer`
+- `extendModel(source): ModelMaterializer`
+
+### `QueryMaterializer`
+Fluent builder for queries. Returned by `modelMaterializer.loadQueryByName()`.
+
+- `getPreparedQuery(): Promise<PreparedQuery>`
+- `getPreparedResult(): Promise<PreparedResult>`
+- `getSQL(): Promise<string>`
+- `run(options?): Promise<Result>`
+
+### `PreparedResultMaterializer`
+Fluent builder for prepared results.
+
+### `ExploreMaterializer`
+Fluent builder for explores/sources.
+
+## Result Data Classes
+
+### `DataArray`
+Iterable array of `DataRecord`. Query result rows.
+
+### `DataRecord`
+Single result row. Field values accessed by name.
+
+## Utility Classes
+
+- `Parse` - Parsed (not compiled) Malloy document
+- `Malloy` - Static methods: `parse()`, `compile()`, `run()`
+- `MalloyError` - Error with structured problems
+- `EmptyURLReader`, `InMemoryURLReader` - URLReader implementations
+- `FixedConnectionMap` - LookupConnection implementation
+- `CacheManager`, `InMemoryModelCache` - Caching
+- `JSONWriter`, `CSVWriter` - Result writers
+- `DocumentSymbol`, `DocumentPosition`, `DocumentRange`, etc. - IDE support
+
+## Naming Issues
+
+| Current Name | What It Actually Is |
+|--------------|---------------------|
+| `Explore` | A source (historical name) |
+| `PreparedQuery` | Uncompiled query holding IR |
+| `PreparedResult` | Compiled query with SQL (not a "result") |
+| `Query` | A view/turtle definition |
+
+## Known Architectural Issues
+
+1. **Transient QueryModel**: `PreparedQuery.getPreparedResult()` creates new `QueryModel` each call - expensive.
+3. **Name collision**: `Query` class (view wrapper) vs internal `Query` type vs `PreparedQuery`

package/dist/annotation.d.ts

@@ -4,9 +4,7 @@ import type { LogMessage } from './lang';
 export interface TagParseSpec {
     prefix?: RegExp;
     extending?: Tag;
-    scopes?: Tag[];
 }
-export declare function addModelScope(spec: TagParseSpec | undefined, modelScope: Tag): TagParseSpec;
 export declare function annotationToTaglines(annote: Annotation | undefined, prefix?: RegExp): string[];
 export interface MalloyTagParse {
     tag: Tag;

package/dist/annotation.js

@@ -1,19 +1,8 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.addModelScope = addModelScope;
 exports.annotationToTaglines = annotationToTaglines;
 exports.annotationToTag = annotationToTag;
 const malloy_tag_1 = require("@malloydata/malloy-tag");
-function addModelScope(spec, modelScope) {
-    const useSpec = spec ? { ...spec } : {};
-    if (useSpec.scopes) {
-        useSpec.scopes = useSpec.scopes.concat(modelScope);
-    }
-    else {
-        useSpec.scopes = [modelScope];
-    }
-    return useSpec;
-}
 function annotationToTaglines(annote, prefix) {
     annote || (annote = {});
     const tagLines = annote.inherits
@@ -31,7 +20,6 @@ function annotationToTaglines(annote, prefix) {
     return tagLines.concat(prefixed(annote.blockNotes), prefixed(annote.notes));
 }
 function annotationToTag(annote, spec = {}) {
-    var _a;
     let extending = spec.extending || new malloy_tag_1.Tag();
     const prefix = spec.prefix || /^##? /;
     annote || (annote = {});
@@ -54,21 +42,39 @@ function annotationToTag(annote, spec = {}) {
             matchingNotes.push(note);
         }
     }
-    for (let i = 0; i < matchingNotes.length; i++) {
-        const note = matchingNotes[i];
-        if (note.text.match(prefix)) {
-            const noteParse = malloy_tag_1.Tag.fromTagLine(note.text, i, extending, ...((_a = spec.scopes) !== null && _a !== void 0 ? _a : []));
-            extending = noteParse.tag;
-            allErrs.push(...noteParse.log.map((e) => mapMalloyError(e, note)));
-        }
+    for (const note of matchingNotes) {
+        const noteParse = (0, malloy_tag_1.parseTag)(note.text, extending);
+        extending = noteParse.tag;
+        allErrs.push(...noteParse.log.map((e) => mapMalloyError(e, note)));
+    }
+    // Validate references and add any warnings
+    const refErrors = extending.validateReferences();
+    for (const refError of refErrors) {
+        allErrs.push({
+            code: 'tag-reference-error',
+            severity: 'warn',
+            message: refError,
+        });
     }
     return { tag: extending, log: allErrs };
 }
 function mapMalloyError(e, note) {
-    const loc = {
-        line: note.at.range.start.line,
-        character: note.at.range.start.character + e.offset,
-    };
+    // Calculate prefix length (same logic as parseTagLine)
+    let prefixLen = 0;
+    if (note.text[0] === '#') {
+        const skipTo = note.text.indexOf(' ');
+        if (skipTo > 0) {
+            prefixLen = skipTo;
+        }
+    }
+    // Map error position to source location
+    // e.line is 0-based line within the (stripped) input
+    // e.offset is 0-based column within that line
+    const line = note.at.range.start.line + e.line;
+    const character = e.line === 0
+        ? note.at.range.start.character + prefixLen + e.offset
+        : e.offset;
+    const loc = { line, character };
     return {
         code: 'tag-parse-error',
         severity: 'error',

package/dist/api/asynchronous.d.ts

@@ -1,7 +1,7 @@
 import type * as Malloy from '@malloydata/malloy-interfaces';
 import type { InfoConnection, Connection, LookupConnection } from './connection';
 import type { URLReader } from '../runtime_types';
-import type { CacheManager } from '../malloy';
+import type { CacheManager } from './foundation';
 export interface CompilerNeedFetch<T extends InfoConnection> {
     connections: LookupConnection<T>;
     urls: URLReader;

package/dist/api/foundation/cache.d.ts

@@ -0,0 +1,32 @@
+import type { ModelDef } from '../../model';
+import type { InvalidationKey, URLReader } from '../../runtime_types';
+interface CacheGetModelDefResponse {
+    modelDef: ModelDef;
+    invalidationKeys: {
+        [url: string]: InvalidationKey;
+    };
+}
+export interface ModelCache {
+    getModel(url: URL): Promise<CachedModel | undefined>;
+    setModel(url: URL, cachedModel: CachedModel): Promise<boolean>;
+}
+export interface CachedModel {
+    modelDef: ModelDef;
+    invalidationKeys: {
+        [url: string]: InvalidationKey;
+    };
+}
+export declare class CacheManager {
+    private modelCache;
+    private modelDependencies;
+    private modelInvalidationKeys;
+    constructor(modelCache: ModelCache);
+    getCachedModelDef(urlReader: URLReader, url: string): Promise<CacheGetModelDefResponse | undefined>;
+    setCachedModelDef(url: string, cachedModel: CachedModel): Promise<boolean>;
+}
+export declare class InMemoryModelCache implements ModelCache {
+    private readonly models;
+    getModel(url: URL): Promise<CachedModel | undefined>;
+    setModel(url: URL, cachedModel: CachedModel): Promise<boolean>;
+}
+export {};

package/dist/api/foundation/cache.js

@@ -0,0 +1,92 @@
+"use strict";
+/*
+ * Copyright Contributors to the Malloy project
+ * SPDX-License-Identifier: MIT
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.InMemoryModelCache = exports.CacheManager = void 0;
+const readers_1 = require("./readers");
+class CacheManager {
+    constructor(modelCache) {
+        this.modelCache = modelCache;
+        this.modelDependencies = new Map();
+        this.modelInvalidationKeys = new Map();
+    }
+    async getCachedModelDef(urlReader, url) {
+        const _dependencies = this.modelDependencies.get(url);
+        if (_dependencies === undefined) {
+            return undefined;
+        }
+        const dependencies = [url, ...flatDeps(_dependencies)];
+        const invalidationKeys = {};
+        for (const dependency of dependencies) {
+            const invalidationKey = this.modelInvalidationKeys.get(dependency);
+            if (invalidationKey === undefined || invalidationKey === null) {
+                return undefined;
+            }
+            invalidationKeys[dependency] = invalidationKey;
+        }
+        for (const dependency of dependencies) {
+            const invalidationKey = await (0, readers_1.getInvalidationKey)(urlReader, new URL(dependency));
+            if (invalidationKey !== invalidationKeys[dependency]) {
+                return undefined;
+            }
+        }
+        const cached = await this.modelCache.getModel(new URL(url));
+        if (cached === undefined) {
+            return undefined;
+        }
+        for (const dependency of dependencies) {
+            if (cached.invalidationKeys[dependency] !== invalidationKeys[dependency]) {
+                return undefined;
+            }
+        }
+        // Return the cached model def and the invalidation keys for this
+        // model def's dependencies
+        return { modelDef: cached.modelDef, invalidationKeys };
+    }
+    async setCachedModelDef(url, cachedModel) {
+        this.modelDependencies.set(url, cachedModel.modelDef.dependencies);
+        const invalidationKeys = {};
+        for (const dependency of [
+            url,
+            ...flatDeps(cachedModel.modelDef.dependencies),
+        ]) {
+            if (cachedModel.invalidationKeys[dependency] === null) {
+                return false;
+            }
+            if (cachedModel.invalidationKeys[dependency] === undefined) {
+                throw new Error(`Missing invalidation key for dependency ${dependency}`);
+            }
+            this.modelInvalidationKeys.set(dependency, cachedModel.invalidationKeys[dependency]);
+            invalidationKeys[dependency] = cachedModel.invalidationKeys[dependency];
+        }
+        const result = await this.modelCache.setModel(new URL(url), {
+            modelDef: cachedModel.modelDef,
+            invalidationKeys,
+        });
+        if (result) {
+            return true; // TODO just return `result` when it's a boolean
+        }
+        return false;
+    }
+}
+exports.CacheManager = CacheManager;
+function flatDeps(tree) {
+    return [...Object.keys(tree), ...Object.values(tree).map(flatDeps).flat()];
+}
+// TODO maybe make this memory bounded....
+class InMemoryModelCache {
+    constructor() {
+        this.models = new Map();
+    }
+    async getModel(url) {
+        return Promise.resolve(this.models.get(url.toString()));
+    }
+    async setModel(url, cachedModel) {
+        this.models.set(url.toString(), cachedModel);
+        return Promise.resolve(true);
+    }
+}
+exports.InMemoryModelCache = InMemoryModelCache;
+//# sourceMappingURL=cache.js.map

package/dist/api/foundation/compile.d.ts

@@ -0,0 +1,175 @@
+import type { LogMessage } from '../../lang';
+import type { Connection, InfoConnection, LookupConnection, FetchSchemaOptions } from '../../connection/types';
+import type { URLReader, EventStream } from '../../runtime_types';
+import type { SQLSourceDef, QueryRunStats } from '../../model';
+import type { RunSQLOptions } from '../../run_sql_options';
+import type { CacheManager } from './cache';
+import { Parse } from './document';
+import type { ParseOptions, CompileOptions, CompileQueryOptions } from './types';
+import type { PreparedResult } from './core';
+import { Model } from './core';
+import { Result, DataRecord } from './result';
+type Compilable = {
+    url: URL;
+    source?: undefined;
+    parse?: undefined;
+} | {
+    source: string;
+    url?: URL;
+    parse?: undefined;
+} | {
+    parse: Parse;
+    url?: URL;
+    source?: undefined;
+};
+export interface MalloyCompileOptions {
+    url?: URL;
+    source?: string;
+    parse?: Parse;
+    urlReader: URLReader;
+    connections: LookupConnection<InfoConnection>;
+    model?: Model;
+    cacheManager?: CacheManager;
+    refreshSchemaCache?: boolean | number;
+    noThrowOnError?: boolean;
+    eventStream?: EventStream;
+    importBaseURL?: URL;
+    testEnvironment?: boolean;
+}
+export interface MalloyRunOptions {
+    connection?: Connection;
+    preparedResult?: PreparedResult;
+    sqlStruct?: SQLSourceDef;
+    connections?: LookupConnection<Connection>;
+    options?: RunSQLOptions;
+}
+/**
+ * A Malloy error, which may contain log messages produced during compilation.
+ */
+export declare class MalloyError extends Error {
+    readonly problems: LogMessage[];
+    /**
+     * An array of log messages produced during compilation.
+     */
+    constructor(message: string, problems?: LogMessage[]);
+}
+export declare class Malloy {
+    static get version(): string;
+    private static _parse;
+    /**
+     * Parse a Malloy document by URL.
+     *
+     * @param url The URL of the Malloy document to parse.
+     * @param urlReader Object capable of fetching URL contents.
+     * @return A (promise of a) `Parse` result.
+     */
+    static parse({ url, urlReader, eventStream, options, }: {
+        url: URL;
+        urlReader: URLReader;
+        eventStream?: EventStream;
+        options?: ParseOptions;
+    }): Promise<Parse>;
+    /**
+     * Parse a Malloy document by contents.
+     *
+     * @param url The URL of the Malloy document to parse (optional).
+     * @param source The contents of the Malloy document to parse.
+     * @return A `Parse` result.
+     */
+    static parse({ source, url, eventStream, options, }: {
+        url?: URL;
+        source: string;
+        eventStream?: EventStream;
+        options?: ParseOptions;
+    }): Parse;
+    /**
+     * Compile a parsed Malloy document.
+     *
+     * @param urlReader Object capable of reading contents of a URL.
+     * @param connections Mapping of connection names to objects capable of reading Malloy schemas.
+     * @param parse The parsed Malloy document.
+     * @param model A compiled model to build upon (optional).
+     * @return A (promise of a) compiled `Model`.
+     */
+    static compile({ url, source, parse, urlReader, connections, model, refreshSchemaCache, noThrowOnError, eventStream, importBaseURL, cacheManager, }: {
+        urlReader: URLReader;
+        connections: LookupConnection<InfoConnection>;
+        model?: Model;
+        cacheManager?: CacheManager;
+    } & Compilable & CompileOptions & CompileQueryOptions & ParseOptions): Promise<Model>;
+    /**
+     * A dialect must provide a response for every table, or the translator loop
+     * will never exit. Because there was a time when this happened, we throw
+     * instead of looping forever, but the fix is to correct the dialect.
+     */
+    static safelyFetchTableSchema(connection: InfoConnection, toFetch: Record<string, string>, opts: FetchSchemaOptions): Promise<{
+        schemas: Record<string, import("../../model").TableSourceDef>;
+        errors: Record<string, string>;
+    }>;
+    /**
+     * Run a fully-prepared query.
+     *
+     * @param get A mapping from connection names to objects capable of running SQL.
+     * @param preparedResult A fully-prepared query which is ready to run (a `PreparedResult`).
+     * @return Query result data and associated metadata.
+     */
+    static run(params: {
+        connections: LookupConnection<Connection>;
+        preparedResult: PreparedResult;
+        options?: RunSQLOptions;
+    }): Promise<Result>;
+    static run(params: {
+        connection: Connection;
+        preparedResult: PreparedResult;
+        options?: RunSQLOptions;
+    }): Promise<Result>;
+    static run(params: {
+        connection: Connection;
+        sqlStruct: SQLSourceDef;
+        options?: RunSQLOptions;
+    }): Promise<Result>;
+    static run(params: {
+        connections: LookupConnection<Connection>;
+        sqlStruct: SQLSourceDef;
+        options?: RunSQLOptions;
+    }): Promise<Result>;
+    static run(params: {
+        connection: Connection;
+        sqlStruct: SQLSourceDef;
+        options?: RunSQLOptions;
+    }): Promise<Result>;
+    static run(params: {
+        connections: LookupConnection<Connection>;
+        sqlStruct: SQLSourceDef;
+        options?: RunSQLOptions;
+    }): Promise<Result>;
+    static runStream(params: {
+        connections: LookupConnection<Connection>;
+        preparedResult: PreparedResult;
+        options?: RunSQLOptions;
+    }): AsyncIterableIterator<DataRecord>;
+    static runStream(params: {
+        connection: Connection;
+        preparedResult: PreparedResult;
+        options?: RunSQLOptions;
+    }): AsyncIterableIterator<DataRecord>;
+    static runStream(params: {
+        connection: Connection;
+        sqlStruct: SQLSourceDef;
+        options?: RunSQLOptions;
+    }): AsyncIterableIterator<DataRecord>;
+    static runStream(params: {
+        connections: LookupConnection<Connection>;
+        sqlStruct: SQLSourceDef;
+        options?: RunSQLOptions;
+    }): AsyncIterableIterator<DataRecord>;
+    static estimateQueryCost(params: {
+        connections: LookupConnection<Connection>;
+        preparedResult: PreparedResult;
+    }): Promise<QueryRunStats>;
+    static estimateQueryCost(params: {
+        connections: LookupConnection<Connection>;
+        sqlStruct: SQLSourceDef;
+    }): Promise<QueryRunStats>;
+}
+export {};