@gblikas/querykit 0.2.0 → 0.4.0

package/.cursor/BUGBOT.md

@@ -1,21 +1,84 @@
 # Project review guidelines
 
-Bellow is a list of generally accepted best-practices to prevent bugs in projects. Not all guidelines may apply to the current project; please make sure to read any README.md files in order to correlate goals for bug detection. 
+Below is a list of generally accepted best-practices to prevent bugs in QueryKit. Not all guidelines may apply to every component; please make sure to read the README.md for context on the project's goals.
 
 ## Security focus areas
 
 - Validate user input in API endpoints
 - Check for SQL injection vulnerabilities in database queries
 - Ensure proper authentication on protected routes
+- Validate query inputs using `parseWithContext` with security options
+- Use `allowedFields` and `denyFields` to restrict queryable fields
+- Set `maxQueryDepth` and `maxClauseCount` to prevent DoS attacks
+
+### Query parsing security
+
+When using the input parser or `parseWithContext`:
+
+1. **Never trust user-provided queries** - Always validate with security options:
+   ```typescript
+   const result = parser.parseWithContext(userQuery, {
+     securityOptions: {
+       allowedFields: ['name', 'status', 'priority'],
+       denyFields: ['password', 'secret'],
+       maxQueryDepth: 5,
+       maxClauseCount: 20
+     }
+   });
+   
+   if (!result.security?.passed) {
+     // Reject query - contains violations
+   }
+   ```
+
+2. **Schema validation** - Use schema to detect typos and invalid fields early:
+   ```typescript
+   const result = parser.parseWithContext(userQuery, { schema });
+   if (!result.fieldValidation?.valid) {
+     // Show user-friendly error with suggestions
+   }
+   ```
+
+3. **Input parser limitations** - The input parser (`parseQueryInput`, `parseQueryTokens`) is regex-based for performance. It may accept inputs that the main parser rejects. Always validate with `parseWithContext` or `parser.parse()` before executing queries.
 
 ## Architecture patterns
 
 - Use dependency injection for services
 - Follow the repository pattern for data access
 - Implement proper error handling with custom error classes
+- Parser components follow Single Responsibility Principle:
+  - `input-parser.ts` - Fast, regex-based tokenization for UI feedback
+  - `parser.ts` - Full Liqe-based parsing with AST generation
+  - `parseWithContext` - Orchestrates both for rich context
+
+### Parser architecture
+
+The parsing system has two tiers:
+
+1. **Input Parser** (`parseQueryInput`, `parseQueryTokens`)
+   - Purpose: Real-time UI feedback (highlighting, cursor context)
+   - Performance: O(n) regex-based, no AST generation
+   - Error handling: Best-effort, never throws
+   - Use for: Search bar highlighting, autocomplete triggering
+
+2. **Query Parser** (`parser.parse`, `parseWithContext`)
+   - Purpose: Query validation and execution
+   - Performance: Full Liqe grammar parsing
+   - Error handling: Strict validation, detailed error messages
+   - Use for: Query execution, security validation
 
 ## Common issues
 
 - Memory leaks in React components (check useEffect cleanup)
 - Missing error boundaries in UI components
-- Inconsistent naming conventions (use camelCase for functions)
+- Inconsistent naming conventions (use camelCase for functions)
+- Not checking `result.success` before accessing `result.ast`
+- Using input parser for security validation (use `parseWithContext` instead)
+- Forgetting to provide `cursorPosition` when autocomplete is needed
+
+## Testing guidelines
+
+- All parser features require co-located tests
+- Use divergence tests to document differences between input parser and main parser
+- Token consistency tests verify `parseWithContext` tokens match `parseQueryTokens`
+- Security tests should cover field restrictions, depth limits, and value sanitization

package/.husky/pre-commit

@@ -14,16 +14,16 @@ echo "Running pre-commit checks..."
 
 # Run lint-staged to process only changed files
 echo "🔍 Running lint-staged..."
-pnpm lint-staged || { echo "❌ Linting failed"; exit 1; }
+npx lint-staged || { echo "❌ Linting failed"; exit 1; }
 
 # Check TypeScript compilation
 echo "🔍 Checking TypeScript compilation..."
-pnpm exec tsc --noEmit || { echo "❌ TypeScript check failed"; exit 1; }
+npx tsc --noEmit || { echo "❌ TypeScript check failed"; exit 1; }
 
 # Run tests
 if [ "$RUN_TESTS" = "true" ]; then
   echo "🧪 Running tests..."
-  pnpm test || { echo "❌ Tests failed"; exit 1; }
+  npm test || { echo "❌ Tests failed"; exit 1; }
 fi
 
 # If everything passes, allow the commit

package/README.md

@@ -275,6 +275,511 @@ const publicSearchKit = createQueryKit({
 });
 ```
 
+## Input Parsing for Search UIs
+
+QueryKit provides utilities for building rich search bar experiences with real-time feedback, including key:value highlighting, autocomplete suggestions, and error recovery hints.
+
+### Real-Time Token Parsing
+
+Use `parseQueryInput` and `parseQueryTokens` for lightweight, real-time parsing as users type:
+
+```typescript
+import { parseQueryInput, parseQueryTokens } from '@gblikas/querykit';
+
+// Parse input to get terms and cursor context
+const input = 'status:done AND priority:';
+const result = parseQueryInput(input, { cursorPosition: 25 });
+
+// result.terms contains parsed terms:
+// [{ key: 'status', value: 'done', ... }, { key: 'priority', value: null, ... }]
+
+// result.cursorContext tells you where the cursor is: 'key', 'value', or 'operator'
+console.log(result.cursorContext); // 'value' (cursor is after 'priority:')
+
+// Get interleaved tokens (terms + operators) for highlighting
+const tokens = parseQueryTokens(input);
+// [
+//   { type: 'term', key: 'status', value: 'done', startPosition: 0, endPosition: 11 },
+//   { type: 'operator', operator: 'AND', startPosition: 12, endPosition: 15 },
+//   { type: 'term', key: 'priority', value: null, startPosition: 16, endPosition: 25 }
+// ]
+```
+
+### Rich Context with parseWithContext
+
+For comprehensive parsing with schema validation, autocomplete, and error recovery:
+
+```typescript
+import { QueryParser } from '@gblikas/querykit';
+
+const parser = new QueryParser();
+
+// Define your schema for validation and autocomplete
+const schema = {
+  status: {
+    type: 'string',
+    allowedValues: ['todo', 'doing', 'done'],
+    description: 'Task status'
+  },
+  priority: { type: 'number', description: 'Priority level (1-5)' },
+  assignee: { type: 'string', description: 'Assigned user' }
+};
+
+const result = parser.parseWithContext('status:do', {
+  cursorPosition: 9,
+  schema,
+  securityOptions: { maxClauseCount: 10 }
+});
+
+// Always returns a result object (never throws)
+console.log(result.success);    // true/false - whether parsing succeeded
+console.log(result.tokens);     // Tokenized input (always available)
+console.log(result.structure);  // Query structure analysis
+console.log(result.ast);        // AST (if successful)
+console.log(result.error);      // Error details (if failed)
+
+// Autocomplete suggestions based on cursor position
+console.log(result.suggestions);
+// {
+//   context: 'value',
+//   currentField: 'status',
+//   values: [
+//     { value: 'doing', score: 80 },
+//     { value: 'done', score: 80 }
+//   ]
+// }
+
+// Schema validation results
+console.log(result.fieldValidation);
+// { valid: true, fields: [...], unknownFields: [] }
+
+// Security pre-check
+console.log(result.security);
+// { passed: true, violations: [], warnings: [] }
+```
+
+### Error Recovery
+
+When parsing fails, `parseWithContext` provides helpful recovery hints:
+
+```typescript
+const result = parser.parseWithContext('status:"incomplete');
+
+console.log(result.recovery);
+// {
+//   issue: 'unclosed_quote',
+//   message: 'Unclosed double quote detected',
+//   suggestion: 'Add a closing " to complete the quoted value',
+//   autofix: 'status:"incomplete"',
+//   position: 7
+// }
+```
+
+Error types detected:
+- `unclosed_quote` - Missing closing quote (with autofix)
+- `unclosed_parenthesis` - Unbalanced parentheses (with autofix)
+- `trailing_operator` - Query ends with AND/OR/NOT (with autofix)
+- `missing_value` - Field has colon but no value
+- `syntax_error` - Generic syntax issue
+
+### Building a Search Bar with Highlighting
+
+Here's a React example using the input parser for highlighting:
+
+```tsx
+import { parseQueryTokens } from '@gblikas/querykit';
+
+function SearchBar({ value, onChange }) {
+  const tokens = parseQueryTokens(value);
+
+  const renderHighlightedQuery = () => {
+    if (!value) return null;
+
+    return tokens.map((token, idx) => {
+      const text = value.slice(token.startPosition, token.endPosition);
+
+      if (token.type === 'operator') {
+        return <span key={idx} className="text-purple-500">{text}</span>;
+      }
+
+      // Term token - highlight key and value differently
+      if (token.key && token.operator) {
+        const keyEnd = token.startPosition + token.key.length;
+        const opEnd = keyEnd + token.operator.length;
+        return (
+          <span key={idx}>
+            <span className="text-orange-400">{token.key}</span>
+            <span className="text-gray-500">{token.operator}</span>
+            <span className="text-blue-400">{value.slice(opEnd, token.endPosition)}</span>
+          </span>
+        );
+      }
+
+      return <span key={idx}>{text}</span>;
+    });
+  };
+
+  return (
+    <div className="relative">
+      <div className="absolute inset-0 pointer-events-none">
+        {renderHighlightedQuery()}
+      </div>
+      <input
+        value={value}
+        onChange={(e) => onChange(e.target.value)}
+        className="bg-transparent text-transparent caret-black"
+      />
+    </div>
+  );
+}
+```
+
+## Virtual Fields
+
+Virtual fields enable powerful shortcuts in your queries that expand to real schema fields at query execution time based on runtime context. This allows you to support queries like `my:assigned` which expands to `assignee_id == <current_user_id>` using the currently logged-in user's ID.
+
+### Why Virtual Fields?
+
+Virtual fields are useful when:
+- You want to provide user-friendly shortcuts (e.g., `my:assigned` instead of `assignee_id:123`)
+- The query depends on runtime context (current user, permissions, tenant, etc.)
+- You want to abstract complex field mappings from end users
+- You need consistent query shortcuts across your application
+
+### Basic Usage
+
+Define virtual fields when creating your QueryKit instance:
+
+```typescript
+import { createQueryKit } from '@gblikas/querykit';
+import { drizzleAdapter } from '@gblikas/querykit/adapters/drizzle';
+
+const qk = createQueryKit({
+  adapter: drizzleAdapter,
+  schema: { tasks, users },
+
+  // Define virtual fields
+  virtualFields: {
+    my: {
+      allowedValues: ['assigned', 'created', 'watching'] as const,
+      description: 'Filter by your relationship to items',
+      
+      resolve: (input, ctx, { fields }) => {
+        // Map virtual values to real schema fields
+        const fieldMap = fields({
+          assigned: 'assignee_id',
+          created: 'creator_id',
+          watching: 'watcher_ids'
+        });
+
+        return {
+          type: 'comparison',
+          field: fieldMap[input.value],
+          operator: '==',
+          value: ctx.currentUserId
+        };
+      }
+    }
+  },
+
+  // Provide runtime context
+  createContext: async () => ({
+    currentUserId: await getCurrentUserId(),
+    currentUserTeamIds: await getUserTeamIds()
+  })
+});
+
+// Use virtual fields in queries
+const myTasks = await qk
+  .query('tasks')
+  .where('my:assigned AND status:active')
+  .execute();
+```
+
+### Configuration Options
+
+Each virtual field definition supports:
+
+```typescript
+{
+  // Required: allowed values for this virtual field
+  allowedValues: ['value1', 'value2'] as const,
+
+  // Optional: allow comparison operators (>, <, >=, <=)
+  // Default: false (only equality ":" is allowed)
+  allowOperators?: boolean,
+
+  // Required: resolver function
+  resolve: (input, context, helpers) => {
+    // Return a query expression that replaces the virtual field
+    return {
+      type: 'comparison',
+      field: 'real_field',
+      operator: '==',
+      value: context.someValue
+    };
+  },
+
+  // Optional: human-readable description
+  description?: string,
+
+  // Optional: descriptions for each value
+  valueDescriptions?: {
+    value1: 'Description of value1',
+    value2: 'Description of value2'
+  }
+}
+```
+
+### Type-Safe Field Mapping
+
+The `fields()` helper provides compile-time validation that all mapped fields exist in your schema:
+
+```typescript
+virtualFields: {
+  my: {
+    allowedValues: ['assigned', 'created'] as const,
+    resolve: (input, ctx, { fields }) => {
+      // TypeScript validates:
+      // 1. All allowedValues keys are mapped
+      // 2. All field values exist in the schema
+      const fieldMap = fields({
+        assigned: 'assignee_id',  // ✓ Valid schema field
+        created: 'creator_id'      // ✓ Valid schema field
+        // Missing 'watching' → TypeScript error!
+        // assigned: 'invalid_field' → TypeScript error!
+      });
+
+      return {
+        type: 'comparison',
+        field: fieldMap[input.value],
+        operator: '==',
+        value: ctx.currentUserId
+      };
+    }
+  }
+}
+```
+
+### Context Factory
+
+The `createContext` function is called once per query execution to provide runtime values:
+
+```typescript
+createContext: async () => {
+  const user = await getCurrentUser();
+  const permissions = await getUserPermissions(user.id);
+  
+  return {
+    currentUserId: user.id,
+    currentUserTeamIds: user.teamIds,
+    canSeeArchived: permissions.includes('view:archived')
+  };
+}
+```
+
+Context is type-safe and can include any data your resolvers need:
+
+```typescript
+interface MyQueryContext extends IQueryContext {
+  currentUserId: number;
+  currentUserTeamIds: number[];
+  canSeeArchived: boolean;
+}
+
+const qk = createQueryKit<typeof schema, MyQueryContext>({
+  // ... configuration
+});
+```
+
+### Complex Resolvers
+
+Virtual fields can return logical expressions for more complex scenarios:
+
+```typescript
+virtualFields: {
+  myItems: {
+    allowedValues: ['all'] as const,
+    resolve: (input, ctx) => ({
+      // Return a logical OR expression
+      type: 'logical',
+      operator: 'OR',
+      left: {
+        type: 'comparison',
+        field: 'assignee_id',
+        operator: '==',
+        value: ctx.currentUserId
+      },
+      right: {
+        type: 'comparison',
+        field: 'creator_id',
+        operator: '==',
+        value: ctx.currentUserId
+      }
+    })
+  }
+}
+
+// Expands to: (assignee_id == currentUserId OR creator_id == currentUserId)
+await qk.query('tasks').where('myItems:all').execute();
+```
+
+### Allowing Comparison Operators
+
+By default, only equality (`:`) is allowed. Enable other operators with `allowOperators: true`:
+
+```typescript
+virtualFields: {
+  priority: {
+    allowedValues: ['high', 'low'] as const,
+    allowOperators: true,  // Enable >, <, etc.
+    
+    resolve: (input, ctx) => {
+      const threshold = input.value === 'high' ? 7 : 3;
+      
+      return {
+        type: 'comparison',
+        field: 'priority',
+        operator: input.operator,  // Use the operator from the query
+        value: threshold
+      };
+    }
+  }
+}
+
+// Both work:
+qk.query('tasks').where('priority:high')    // priority == 7
+qk.query('tasks').where('priority:>high')  // priority > 7
+```
+
+### Error Handling
+
+QueryKit throws `QueryParseError` for invalid virtual field usage:
+
+```typescript
+// Invalid value
+qk.query('tasks').where('my:invalid')
+// Error: Invalid value "invalid" for virtual field "my". 
+//        Allowed values: "assigned", "created", "watching"
+
+// Operator not allowed (when allowOperators: false)
+qk.query('tasks').where('my:>assigned')
+// Error: Virtual field "my" does not allow comparison operators. 
+//        Only equality (":") is permitted.
+```
+
+### Complete Example
+
+Here's a full example with multiple virtual fields:
+
+```typescript
+import { createQueryKit, IQueryContext, ComparisonOperator } from '@gblikas/querykit';
+import { drizzleAdapter } from '@gblikas/querykit/adapters/drizzle';
+
+// Define your context type
+interface TaskQueryContext extends IQueryContext {
+  currentUserId: number;
+  currentUserTeamIds: number[];
+  currentTenantId: string;
+}
+
+// Create QueryKit with virtual fields
+const qk = createQueryKit<typeof schema, TaskQueryContext>({
+  adapter: drizzleAdapter,
+  schema: { tasks, users },
+
+  virtualFields: {
+    // User relationship shortcuts
+    my: {
+      allowedValues: ['assigned', 'created', 'watching'] as const,
+      description: 'Filter by your relationship to tasks',
+      valueDescriptions: {
+        assigned: 'Tasks assigned to you',
+        created: 'Tasks you created',
+        watching: 'Tasks you are watching'
+      },
+      resolve: (input, ctx, { fields }) => {
+        const fieldMap = fields({
+          assigned: 'assignee_id',
+          created: 'creator_id',
+          watching: 'watcher_ids'
+        });
+        return {
+          type: 'comparison',
+          field: fieldMap[input.value],
+          operator: '==',
+          value: ctx.currentUserId
+        };
+      }
+    },
+
+    // Team shortcuts
+    team: {
+      allowedValues: ['assigned', 'owned'] as const,
+      description: 'Filter by team relationship',
+      resolve: (input, ctx, { fields }) => {
+        const fieldMap = fields({
+          assigned: 'assignee_id',
+          owned: 'owner_id'
+        });
+        return {
+          type: 'comparison',
+          field: fieldMap[input.value],
+          operator: 'IN',
+          value: ctx.currentUserTeamIds
+        };
+      }
+    },
+
+    // Priority shortcuts with operators
+    priority: {
+      allowedValues: ['critical', 'high', 'normal', 'low'] as const,
+      allowOperators: true,
+      description: 'Filter by priority level',
+      resolve: (input) => {
+        const priorityMap = {
+          critical: 10,
+          high: 7,
+          normal: 5,
+          low: 3
+        };
+        return {
+          type: 'comparison',
+          field: 'priority',
+          operator: input.operator as ComparisonOperator,
+          value: priorityMap[input.value as keyof typeof priorityMap]
+        };
+      }
+    }
+  },
+
+  // Context factory
+  createContext: async () => {
+    const user = await getCurrentUser();
+    const teams = await getUserTeams(user.id);
+    
+    return {
+      currentUserId: user.id,
+      currentUserTeamIds: teams.map(t => t.id),
+      currentTenantId: user.tenantId
+    };
+  }
+});
+
+// Example queries using virtual fields
+// "my:assigned AND status:active"
+// "team:assigned OR my:created"
+// "priority:>high AND my:watching"
+// "(my:assigned OR team:assigned) AND status:active"
+
+const results = await qk
+  .query('tasks')
+  .where('my:assigned AND priority:>high')
+  .orderBy('created_at', 'desc')
+  .limit(10)
+  .execute();
+```
+
 ## Roadmap
 
 ### Core Parsing Engine and DSL
@@ -283,6 +788,9 @@ const publicSearchKit = createQueryKit({
 - [x] Develop internal AST representation
 - [x] Implement consistent syntax for logical operators (AND, OR, NOT)
 - [x] Support standard comparison operators (==, !=, >, >=, <, <=)
+- [x] Real-time input parsing for search UIs
+- [x] Autocomplete suggestions with schema awareness
+- [x] Error recovery hints with autofix
 
 ### First Adapters
 - [x] Drizzle ORM integration
@@ -297,9 +805,10 @@ const publicSearchKit = createQueryKit({
 - [x] Support for complex nested expressions
 - [ ] Custom function support
 - [ ] Pagination helpers
+- [x] Virtual fields for context-aware query expansion
 
 ### Ecosystem Expansion
-- [ ] Frontend query builder components
+- [x] Frontend query builder components (input parser)
 - [ ] Additional ORM adapters
 - [ ] Server middleware for Express/Fastify
 - [ ] TypeScript SDK generation