@malloydata/render 0.0.423 → 0.0.425

package/docs/plans/field-creation-analysis.md

@@ -0,0 +1,178 @@
+# Field Creation Analysis: Excessive Field Creation and Tag Parsing in malloy-render
+
+## Problem Statement
+
+In the malloy-render package, we're experiencing significant performance overhead during data visualization preprocessing. For a simple two-column Malloy query like:
+
+```malloy
+view: two_column is {
+  select: brand, department
+  limit: 1
+}
+```
+
+We expect to see:
+- 2 Field objects created (one for each column)
+- 2 tagFor calls (one for each field's metadata parsing)
+
+However, we're actually seeing:
+- **5 Field objects created** (2.5x expected)
+- **7-8 tagFor calls** (3.5-4x expected)
+
+Since tag parsing is an expensive operation (parsing annotations and metadata), this overhead becomes significant with large datasets. For example, with 5000 rows containing nested data, this can lead to hundreds of thousands of unnecessary field creations and tag parsing operations.
+
+## Root Cause Analysis
+
+### The Problem Flow
+
+1. **RootField Creation** (extends RepeatedRecordField)
+   - Creates RootField instance → 1 tagFor call for RootField itself
+
+2. **ArrayField Constructor** (line 27 in nest.ts)
+   - Creates an `elementField` for the array's element type
+   - This creates a RecordField named "element" → 1 Field.from call + 1 tagFor call
+
+3. **RecordField Constructor** (line 159 in nest.ts)
+   - Creates fields for brand and department as children of "element"
+   - 2 Field.from calls + 2 tagFor calls
+
+4. **RepeatedRecordField Constructor** (line 54 in nest.ts)
+   - **DUPLICATES** the field creation for brand and department
+   - Creates them again as direct children of RepeatedRecordField
+   - 2 more Field.from calls + 2 more tagFor calls
+
+5. **Additional RecordField Creation** (lines 58-71 in nest.ts)
+   - Creates a `nestedRecordField` instance
+   - 1 more tagFor call
+
+### The Duplication Issue
+
+The main issue is that **RepeatedRecordField creates fields TWICE**:
+
+1. First in ArrayField's constructor via `elementField`
+2. Then again in RepeatedRecordField's own constructor at line 54
+
+This happens because:
+- ArrayField creates an elementField (RecordField) which creates brand/department fields
+- RepeatedRecordField then directly parses the record type and creates brand/department fields again
+- The `nestedRecordField` creation adds another layer of overhead
+
+## Current Architecture Analysis
+
+### Why the Extra Fields Exist
+
+1. **ArrayField's `elementField`**
+   - **Purpose**: Provides schema for array elements (could be any type: string, number, record, etc.)
+   - **Used by**: ArrayCell to create appropriate cell types for each array element
+   - **Necessity**: Required for generic array handling
+
+2. **RepeatedRecordField's Triple Structure**
+   - **`elementField`** (inherited from ArrayField): Generic element schema
+   - **`fields` array**: Direct access to record fields for performance
+   - **`nestedRecordField`**: RecordField instance for cell creation compatibility
+
+3. **The Architecture's Intent**
+   
+   The design appears to be intentional for several reasons:
+   - **Type Safety**: Each field type knows exactly what kind of data it contains
+   - **Performance**: Direct field access via `fields` array avoids indirection
+   - **Compatibility**: The `nestedRecordField` allows RepeatedRecordField to work seamlessly with RecordCell
+
+### Why This Architecture Evolved
+
+1. **Type System Alignment**: Mirrors Malloy's type system where arrays and records are distinct
+2. **Performance Optimization**: Direct field access without indirection
+3. **API Compatibility**: RecordCell expects a RecordField, not a RepeatedRecordField
+4. **Incremental Development**: Features added over time without refactoring base structures
+
+## Rearchitecture Proposals
+
+### Option 1: Lazy Field Creation (Minimal Change)
+**Concept**: Create fields only when accessed, cache aggressively
+
+**Pros**: 
+- Minimal API changes
+- Reduces upfront cost
+- Fields created only once
+
+**Cons**: 
+- Doesn't eliminate architectural duplication
+- Just defers the problem
+
+### Option 2: Unified Field Registry (Medium Change)
+**Concept**: Single source of truth for all fields with automatic deduplication
+
+**Pros**: 
+- Single point of field creation
+- Automatic deduplication
+- Easy to add metrics/debugging
+
+**Cons**: 
+- Requires passing registry throughout
+- May complicate field lifecycle
+
+### Option 3: Rethink Inheritance (Major Change)
+**Concept**: RepeatedRecordField shouldn't extend ArrayField since it's conceptually a table, not an array
+
+**Pros**: 
+- Eliminates conceptual mismatch
+- No duplicate fields
+- Cleaner mental model
+
+**Cons**: 
+- Breaking API change
+- Significant refactoring needed
+
+### Option 4: Cell-Centric Architecture (Revolutionary)
+**Concept**: Fields become lightweight metadata, cells handle complexity and create child fields on demand
+
+**Pros**: 
+- Fields created only when data is processed
+- Minimal memory for schema
+- Natural lazy evaluation
+
+**Cons**: 
+- Major paradigm shift
+- May impact performance differently
+
+## Recommendation
+
+**Short term**: Implement the current approach with `skipFieldCreation` flag. This solves the immediate performance issue by:
+- Creating fields once in RepeatedRecordField constructor
+- Sharing these fields with the nestedRecordField
+- Preventing RecordField from creating its own duplicate fields
+
+**Long term**: Consider **Option 3 (Rethink Inheritance)**. The fundamental issue is that RepeatedRecordField is conceptually not an array - it's a table. The inheritance from ArrayField creates unnecessary complexity.
+
+The ideal architecture would:
+1. Create each field exactly once
+2. Share field instances where semantically equivalent
+3. Align with the mental model of tables vs arrays
+4. Minimize memory usage for large datasets
+
+## Impact
+
+For the example two-column query, fixing this would reduce:
+- Field creation from 5 to 2 (60% reduction)
+- tagFor calls from 7-8 to 2 (71-75% reduction)
+
+For larger datasets with nested structures, this optimization becomes even more critical, potentially eliminating hundreds of thousands of unnecessary operations.
+
+## Implementation Status
+
+### Completed (Short-term Fix)
+✅ **Lazy elementField creation in ArrayField**: The elementField is now created only when accessed, preventing duplicate field creation for RepeatedRecordField instances.
+
+✅ **Field sharing in RepeatedRecordField**: The RepeatedRecordField now shares its fields with both the nestedRecordField and the elementField (when accessed), eliminating duplicate Field.from calls.
+
+✅ **Skip redundant tag parsing**: Added skipTagParsing parameter to prevent unnecessary tagFor calls for the synthetic RecordField instance.
+
+**Results achieved**:
+- Field.from calls reduced from 5 to 2 ✓
+- tagFor calls reduced from 7-8 to 4 (further optimization possible)
+
+### Future Work (Long-term Architecture)
+The long-term architectural improvements (Option 3: Rethink Inheritance) remain as future work. This would involve:
+- Redesigning RepeatedRecordField to not extend ArrayField
+- Creating a more appropriate inheritance hierarchy that reflects the conceptual difference between arrays and tables
+- Potentially achieving the ideal 2 tagFor calls by eliminating all redundant metadata parsing

package/docs/plugin-api-reference.md

@@ -0,0 +1,238 @@
+# Plugin API Reference
+
+## Core Types
+
+### RenderPluginFactory<TInstance>
+
+```typescript
+interface RenderPluginFactory<TInstance extends RenderPluginInstance> {
+  readonly name: string;
+  getValidationSpec?(): RendererValidationSpec;
+  matches(field: Field, fieldTag: Tag, fieldType: FieldType): boolean;
+  create(field: Field, pluginOptions?: unknown, modelTag?: Tag): TInstance;
+}
+```
+
+#### Properties
+- `name`: Unique identifier for the plugin
+
+#### Methods
+- `getValidationSpec()`: Declares which tag paths this plugin semantically owns, so the unread-tag detector does not false-warn on valid uses. Required for plugins that read tags at render time. See [validation.md](validation.md) for the ownership model.
+
+- `matches()`: Determines if plugin should handle a field
+  - `field`: Field metadata and structure
+  - `fieldTag`: Field annotations/tags
+  - `fieldType`: Enumerated field type
+  - Returns: `boolean`
+  - Throw when the user's tag asks for this plugin but the field cannot support it — the user sees a red error tile. See [validation.md](validation.md) for the throw-vs-log rule.
+
+- `create()`: Instantiates plugin for a matched field
+  - `field`: Field to render
+  - `pluginOptions`: Configuration from renderer options
+  - `modelTag`: Model-level annotations
+  - Returns: Plugin instance
+  - Do all tag reads here (setup time), not in `renderComponent()` or `beforeRender()` — setup-time reads are validated and marked as consumed automatically.
+
+### RendererValidationSpec
+
+```typescript
+interface RendererValidationSpec {
+  readonly renderer: string;
+  readonly ownedPaths?: string[][];
+  readonly childOwnedPaths?: string[][];
+}
+```
+
+Declares the tag paths a renderer owns — tags whose meaning depends on this renderer being active. See [validation.md](validation.md) for when to declare ownership.
+
+### RenderPluginInstance
+
+Base type: Union of `SolidJSRenderPluginInstance` and `DOMRenderPluginInstance`
+
+#### Common Properties
+```typescript
+interface BaseRenderPluginInstance<TMetadata = unknown> {
+  readonly name: string;
+  readonly field: Field;
+  readonly sizingStrategy: 'fixed' | 'fill';
+  
+  getMetadata(): TMetadata;
+  processData?(field: NestField, cell: NestCell): void;
+  beforeRender?(metadata: RenderMetadata, options: GetResultMetadataOptions): void;
+}
+```
+
+### SolidJSRenderPluginInstance
+
+```typescript
+interface SolidJSRenderPluginInstance<TMetadata = unknown>
+  extends BaseRenderPluginInstance<TMetadata> {
+  readonly renderMode: 'solidjs';
+  renderComponent(props: RenderProps): JSXElement;
+}
+```
+
+### DOMRenderPluginInstance
+
+```typescript
+interface DOMRenderPluginInstance<TMetadata = unknown>
+  extends BaseRenderPluginInstance<TMetadata> {
+  readonly renderMode: 'dom';
+  renderToDOM(container: HTMLElement, props: RenderProps): void;
+  cleanup?(container: HTMLElement): void;
+}
+```
+
+### CoreVizPluginInstance
+
+```typescript
+interface CoreVizPluginMethods {
+  getSchema(): JSONSchemaObject;
+  getSettings(): Record<string, unknown>;
+  getDefaultSettings(): Record<string, unknown>;
+  settingsToTag(settings: Record<string, unknown>): Tag;
+}
+
+type CoreVizPluginInstance<TMetadata = unknown> = 
+  SolidJSRenderPluginInstance<TMetadata> & CoreVizPluginMethods;
+```
+
+## Supporting Types
+
+### RenderProps
+
+```typescript
+interface RenderProps {
+  dataColumn: Cell;
+  field: Field;
+  customProps?: Record<string, unknown>;
+}
+```
+
+### FieldType Enum
+
+```typescript
+enum FieldType {
+  String = 'string',
+  Number = 'number',
+  Boolean = 'boolean',
+  Date = 'date',
+  Timestamp = 'timestamp',
+  JSON = 'json',
+  Record = 'record',
+  RepeatedRecord = 'repeated_record',
+  SQLNative = 'sql_native'
+}
+```
+
+### Field Interface (Key Methods)
+
+```typescript
+interface Field {
+  name: string;
+  key: string;
+  tag: Tag;
+  
+  isString(): boolean;
+  isNumber(): boolean;
+  isBoolean(): boolean;
+  isDate(): boolean;
+  isTime(): boolean;
+  isJSON(): boolean;
+  isNest(): boolean;
+  isBasic(): boolean;
+  
+  // For nested fields
+  fields?: Field[];
+  fieldAt(path: string): Field | undefined;
+}
+```
+
+### Cell Interface (Key Methods)
+
+```typescript
+interface Cell {
+  value: any;
+  
+  isNull(): boolean;
+  isString(): boolean;
+  isNumber(): boolean;
+  isBoolean(): boolean;
+  isDate(): boolean;
+  isTime(): boolean;
+  isJSON(): boolean;
+  isRepeatedRecord(): boolean;
+  
+  // For nested cells
+  column(name: string): Cell;
+}
+```
+
+### Tag Interface (Key Methods)
+
+```typescript
+interface Tag {
+  has(key: string): boolean;
+  text(key: string): string | undefined;
+  boolean(key: string): boolean | undefined;
+  number(key: string): number | undefined;
+  json(key: string): any | undefined;
+}
+```
+
+## Renderer Integration
+
+### MalloyRendererOptions
+
+```typescript
+interface MalloyRendererOptions {
+  plugins?: RenderPluginFactory[];
+  pluginOptions?: Record<string, unknown>;
+  // ... other options
+}
+```
+
+### Plugin Registration
+
+```typescript
+const renderer = new MalloyRenderer({
+  plugins: [Plugin1Factory, Plugin2Factory],
+  pluginOptions: {
+    'plugin1': { /* options */ },
+    'plugin2': { /* options */ }
+  }
+});
+```
+
+## Utility Functions
+
+### isCoreVizPluginInstance
+
+```typescript
+function isCoreVizPluginInstance(
+  plugin: RenderPluginInstance
+): plugin is CoreVizPluginInstance
+```
+
+Checks if a plugin implements the core visualization interface.
+
+## Lifecycle Methods
+
+### processData(field, cell)
+- Called during data processing phase
+- Use for expensive calculations
+- Results can be stored in plugin instance
+
+### beforeRender(metadata, options)
+- Called before rendering
+- Access to full result metadata
+- Can access Vega config overrides
+
+### renderComponent(props) / renderToDOM(container, props)
+- Called to render the visualization
+- Access to current data cell and field metadata
+
+### cleanup(container)
+- DOM plugins only
+- Called when visualization is removed
+- Clean up event listeners, timers, etc.

package/docs/plugin-quick-start.md

@@ -0,0 +1,103 @@
+# Malloy Render Plugin Quick Start
+
+## Minimal Plugin Example
+
+```typescript
+import type { RenderPluginFactory, SolidJSRenderPluginInstance } from '@/api/plugin-types';
+
+export const MinimalPluginFactory: RenderPluginFactory<SolidJSRenderPluginInstance> = {
+  name: 'minimal',
+
+  matches: (field, fieldTag) => fieldTag.has('minimal'),
+
+  create: (field) => ({
+    name: 'minimal',
+    field,
+    renderMode: 'solidjs',
+    sizingStrategy: 'fixed',
+
+    renderComponent: (props) => (
+      <div>{props.dataColumn.value}</div>
+    ),
+
+    getMetadata: () => ({ type: 'minimal' })
+  })
+};
+```
+
+## Key Concepts
+
+### 1. Factory Pattern
+
+- **Factory** creates plugin instances
+- **Instance** handles rendering
+
+### 2. Matching Fields
+
+Plugins match fields based on:
+
+- Tags (e.g., `# my_plugin`)
+- Field types (String, Number, RepeatedRecord, etc.)
+- Custom logic
+
+### 3. Render Modes
+
+- **solidjs**: Reactive components (recommended)
+- **dom**: Direct DOM manipulation
+
+### 4. Sizing Strategies
+
+- **fixed**: Plugin controls its size
+- **fill**: Adapts to container
+
+## Common Patterns
+
+### Chart Plugin
+
+```typescript
+matches: (field, fieldTag, fieldType) => {
+  return fieldType === FieldType.RepeatedRecord && fieldTag.has('my_chart');
+};
+```
+
+### Formatter Plugin
+
+```typescript
+matches: (field, fieldTag, fieldType) => {
+  return fieldType === FieldType.Number && fieldTag.has('currency');
+};
+```
+
+### Conditional Styling
+
+```typescript
+renderComponent: (props) => {
+  const value = props.dataColumn.value;
+  const className = value > 100 ? 'high' : 'low';
+  return <div class={className}>{value}</div>;
+}
+```
+
+## Usage in Malloy
+
+```malloy
+source: data is table('data.csv') {
+  group_by:
+    # currency
+    price
+    # my_chart
+    sales_trend
+}
+```
+
+## Registration
+
+```typescript
+const renderer = new MalloyRenderer({
+  plugins: [MinimalPluginFactory],
+});
+```
+
+## Next steps
+
+Once your plugin reads tags beyond just the one that triggers `matches()`, read [validation.md](validation.md). It covers when to throw (red error tile), when to log (source-located validation error), and how to declare tag ownership so valid tag uses don't trigger unread-tag warnings.