@yesiree/jsonq 1.0.0

package/README.md

@@ -0,0 +1,891 @@
+# jsonq
+
+Developer documentation for contributors.
+
+> **User documentation:** See [DOCS.md](DOCS.md)
+
+## Overview
+
+jsonq is a lightweight JSON query engine that implements a custom expression language for querying and transforming JSON data. The library provides a functional programming interface with method chaining, similar to JSONPath but with richer expression support.
+
+## Architecture
+
+### Core Components
+
+**Tokenizer** (`tokenizePipeline`)
+
+- Parses query strings into token streams
+- Handles root reference (`$`), property access (`.prop`), array indexing (`[0]`), and method calls (`.method()`)
+- Tracks token positions for extracting method parameters
+
+**Evaluator** (`evaluatePipeline`)
+
+- Executes token pipeline against data
+- Maintains current value through transformations
+- Handles nested data navigation
+
+**Expression Parser** (`parseExpr`)
+
+- Recursive descent parser for expressions
+- Generates AST with nodes: literal, variable, property, arrayIndex, unary, binary, methodCall
+- Supports operators: arithmetic, comparison, logical
+
+**Expression Evaluator** (`evalExpr`, `evalNode`)
+
+- Detects arrow function syntax with regex
+- Creates variable bindings for named parameters
+- Evaluates AST nodes with context (data, item, index, bindings)
+
+**Method Executor** (`applyMethod`)
+
+- Switch statement dispatching to method implementations
+- Propagates bindings through nested method calls
+- Handles recursive evaluation for nested transformations
+
+## Public API
+
+```javascript
+export { query };
+```
+
+**query(expression, data)**
+
+- Main entry point
+- Returns transformed data based on expression
+
+## Development
+
+### Setup
+
+```bash
+npm install
+npm test
+```
+
+### Running Tests
+
+```bash
+npm test
+```
+
+Test suite includes 98 tests covering:
+
+- Tokenization and parsing
+- All methods (map, filter, sort, etc.)
+- Expression evaluation (arithmetic, logical, comparison)
+- Nested method calls
+- Arrow function syntax and underscore shorthand
+- Root ($) and array ($array) variable access
+- Edge cases and error handling
+
+### Building
+
+```bash
+npm run build
+```
+
+Outputs to `dist/`:
+
+- `index.cjs` - CommonJS
+- `index.mjs` - ES Module
+- `index.umd.js` - UMD (browser)
+- `index.d.ts` - TypeScript definitions
+
+## Implementation Details
+
+### Supported Methods
+
+```javascript
+const METHODS = [
+  "map",
+  "filter",
+  "join",
+  "sort",
+  "unique",
+  "first",
+  "last",
+  "count",
+  "sum",
+  "avg",
+  "min",
+  "max",
+];
+```
+
+### Token Types
+
+- `root` - `$` reference
+- `property` - `.prop` access
+- `arrayIndex` - `[0]` access
+- `method` - `.method()` call with params
+
+### Expression Syntax Features
+
+**Variable syntaxes:**
+
+1. Root reference: `$`
+2. Built-in variables: `$item`, `$index`, `$array`
+3. Underscore shorthand: `_` (alias for `$item`)
+4. Arrow functions: `x => x.prop` (creates binding for `x`)
+
+**Arrow function detection:**
+
+```javascript
+const arrowMatch = expr.match(/^([a-zA-Z_][a-zA-Z0-9_]*)\s*=>\s*(.+)$/);
+```
+
+**Binding propagation:**
+Arrow function bindings are passed through nested method calls, enabling parent scope access in nested iterations.
+
+### AST Node Types
+
+- `literal` - Numbers, strings, booleans, null, undefined
+- `variable` - `$`, `$item`, `$index`, `$array`, `_`, or named params
+- `property` - Property access (`obj.prop`)
+- `arrayIndex` - Array indexing (`arr[0]`)
+- `unary` - Unary operators (`!`, `-`)
+- `binary` - Binary operators (arithmetic, comparison, logical)
+- `methodCall` - Nested method invocations
+
+### Sort Implementation
+
+Descending sort detected by `-` prefix:
+
+```javascript
+const descending = expr.startsWith("-");
+const sortExpr = descending ? expr.substring(1) : expr;
+```
+
+### Date Handling
+
+Dates prefixed with `@` are parsed to timestamps:
+
+```javascript
+if (token.startsWith("@")) {
+  return new Date(token.substring(1)).getTime();
+}
+```
+
+## Contributing
+
+### Code Style
+
+- No comments in implementation code
+- Variable names provide documentation
+- Concise, functional style
+- Manual parsing (no external parser libraries)
+
+### Adding Methods
+
+1. Add method name to `METHODS` array
+2. Add case to `applyMethod` switch statement
+3. Propagate `bindings` parameter for nested support
+4. Add tests to `index.test.js`
+5. Update DOCS.md with user-facing documentation
+
+### Adding Operators
+
+1. Update `tokenizeExpr` regex to include new operator
+2. Add operator precedence in `parseExpr`
+3. Handle in `evalNode` binary operator cases
+4. Add tests
+
+### Testing Guidelines
+
+- Test each method with basic usage
+- Test chaining combinations
+- Test edge cases (empty arrays, null values)
+- Test error conditions
+- Test nested method scenarios
+- Test all three syntax styles (`$item`, `_`, arrow functions)
+
+## Project Structure
+
+```
+├── src/
+│   ├── index.js         # Main implementation
+│   └── cli.js           # CLI tool (if applicable)
+├── test/
+│   └── index.test.js    # Test suite
+├── examples/
+│   └── example.js       # Usage examples
+├── dist/                # Build outputs (generated)
+├── DOCS.md              # User documentation
+├── README.md            # Developer documentation (this file)
+├── LICENSE              # MIT License
+├── .gitignore
+├── package.json
+├── rollup.config.mjs
+└── jest.config.js
+```
+
+## Design Decisions
+
+### Manual Parsing
+
+We use a manual tokenizer/parser instead of a library to:
+
+- Keep bundle size minimal
+- Avoid external dependencies
+- Maintain full control over syntax
+- Optimize for our specific use case
+
+### Three Syntax Styles
+
+Supporting `$item`, `_`, and arrow functions provides:
+
+- Backward compatibility (`$item`)
+- Conciseness (`_`)
+- Clarity and parent scope access (arrow functions)
+
+### No Pipeline Operator
+
+The `>` operator was removed to avoid:
+
+- Confusion with greater-than comparisons
+- Additional parsing complexity
+- Preference for dot-chaining which is more familiar
+
+### Method Chaining Only
+
+All operations chain with `.method()` syntax for consistency with JavaScript's native array methods.
+
+## Performance Considerations
+
+- No AST caching (expressions evaluated fresh each time)
+- Recursive descent parser (adequate for expression complexity)
+- Methods execute eagerly (no lazy evaluation)
+- Suitable for small to medium datasets
+
+## Known Limitations
+
+- No async support
+- No custom function definitions
+- Single-parameter arrow functions only
+- No array/object literal construction in expressions
+- Limited to JavaScript-compatible JSON data
+
+## Release Process
+
+1. Update version in `package.json`
+2. Run tests: `npm test`
+3. Build: `npm run build`
+4. Commit changes
+5. Tag release: `git tag v1.0.0`
+6. Push: `git push && git push --tags`
+7. Publish: `npm publish`
+
+## License
+
+MIT
+query("$.users.map($item.name)", data);
+
+// Arithmetic
+query("$.items.map(price * 1.1)", data);
+query("$.items.map(\_.price \* 1.1)", data);
+
+// Complex expressions
+query("$.users.map(u => u.age \* 2 + 10)", data);
+
+````
+
+### filter(condition)
+
+Select elements matching a condition.
+
+```javascript
+// Comparison operators
+query("$.users.filter(age > 25)", data);
+query('$.users.filter(name == "Alice")', data);
+
+// Using underscore
+query("$.users.filter(_.age > 25)", data);
+
+// Using arrow syntax
+query("$.users.filter(u => u.age > 25 && u.active)", data);
+
+// Logical operators
+query("$.users.filter(age > 25 && active == true)", data);
+query("$.users.filter(age < 20 || age > 60)", data);
+
+// Negation
+query("$.users.filter(!active)", data);
+query("$.users.filter(!_.deleted)", data);
+````
+
+### sort(expression)
+
+Sort array elements. Use negative expression for descending order.
+
+```javascript
+// Sort ascending (default)
+query("$.users.sort(age)", data);
+query("$.users.sort(name)", data);
+
+// Sort descending (use negative)
+query("$.users.sort(-age)", data);
+query("$.users.sort(-name)", data);
+
+// Sort with expressions
+query("$.items.sort(price * quantity)", data);
+query("$.items.sort(-(price * quantity))", data); // Descending
+```
+
+### join(separator)
+
+Join array elements into a string.
+
+```javascript
+query('$.tags.join(", ")', data);
+query('$.users.map(name).join(" | ")', data);
+```
+
+### unique(expression)
+
+Get unique values from an array.
+
+```javascript
+// Unique primitives
+query("$.tags.unique($item)", data);
+
+// Unique by property
+query("$.users.unique(department)", data);
+```
+
+### first()
+
+Get the first element.
+
+```javascript
+query("$.users.first()", data);
+query("$.users.filter(active).first()", data);
+```
+
+### last()
+
+Get the last element.
+
+```javascript
+query("$.users.last()", data);
+query("$.users.sort(age).last()", data);
+```
+
+### count()
+
+Count array elements.
+
+```javascript
+query("$.users.count()", data);
+query("$.users.filter(active).count()", data);
+```
+
+### sum(expression)
+
+Sum values in an array.
+
+```javascript
+query("$.items.sum(price)", data);
+query("$.numbers.sum($item)", data);
+```
+
+### avg(expression)
+
+Calculate average of values.
+
+```javascript
+query("$.users.avg(age)", data);
+query("$.items.avg(price * quantity)", data);
+```
+
+### min(expression)
+
+Find minimum value.
+
+```javascript
+query("$.users.min(age)", data);
+query("$.items.min(price)", data);
+```
+
+### max(expression)
+
+Find maximum value.
+
+```javascript
+query("$.users.max(age)", data);
+query("$.items.max(price)", data);
+```
+
+## Expressions
+
+### Variables
+
+**Built-in variables:**
+
+- `$item` - Current element in iteration
+- `$index` - Current index in iteration
+- `$data` - Root data object
+- `_` - Shorthand for `$item` (concise syntax)
+
+**Arrow function syntax:**
+Use arrow functions to name your parameters:
+
+```javascript
+// Named parameter (any name you want)
+query("$.users.map(user => user.name)", data);
+query("$.users.filter(person => person.age > 25)", data);
+
+// Access parent scope in nested iterations
+query(
+  '$.departments.map(dept => dept.employees.map(emp => dept.name + ": " + emp.name))',
+  data,
+);
+```
+
+**Examples:**
+
+```javascript
+// Using built-in variables
+query("$.users.map($item.name)", data);
+query("$.users.map($index)", data);
+query("$.users.filter($item.age > $data.minAge)", data);
+
+// Using underscore shorthand (more concise)
+query("$.users.map(_.name)", data);
+query("$.items.filter(_.price < 100)", data);
+
+// Using arrow syntax for clarity
+query("$.users.map(u => u.name)", data);
+query("$.items.filter(item => item.price < 100)", data);
+```
+
+**Nested references:**
+
+```javascript
+// Underscore shadows in nested contexts
+query("$.departments.map(_.employees.map(_.name))", data);
+// Outer _ = department, inner _ = employee (shadowed)
+
+// Use arrow syntax when you need parent access
+query(
+  '$.departments.map(dept => dept.employees.map(emp => dept.name + ": " + emp.name))',
+  data,
+);
+
+// Mix both styles
+query("$.groups.map(g => g.items.filter(_.value > 10).map(_.name))", data);
+```
+
+### Operators
+
+**Arithmetic**: `+`, `-`, `*`, `/`, `%`
+
+```javascript
+query("$.items.map(price * 1.2)", data);
+query("$.users.filter(age % 2 == 0)", data);
+```
+
+**Comparison**: `==`, `!=`, `<`, `>`, `<=`, `>=`
+
+```javascript
+query("$.users.filter(age >= 18)", data);
+query("$.items.filter(stock != 0)", data);
+```
+
+**Logical**: `&&`, `||`, `!`
+
+```javascript
+query("$.users.filter(age > 18 && active == true)", data);
+query("$.items.filter(inStock || onOrder)", data);
+query("$.users.filter(!deleted)", data);
+```
+
+### Literals
+
+- **Numbers**: `42`, `3.14`, `-10`, `1.5e10`
+- **Strings**: `"hello"`, `'world'`
+- **Booleans**: `true`, `false`
+- **Null**: `null`
+- **Undefined**: `undefined`
+
+```javascript
+query("$.users.filter(age > 25)", data);
+query('$.users.filter(name == "Alice")', data);
+query("$.users.filter(deleted == null)", data);
+```
+
+### Dates
+
+Use `@` prefix for ISO date strings:
+
+```javascript
+const data = [
+  { event: "Launch", date: "2024-01-15" },
+  { event: "Update", date: "2024-06-20" },
+];
+
+query("filter(date > @2024-03-01).map(event)", data);
+// => ['Update']
+
+// With timestamps
+query("filter(createdAt > @2024-01-01T12:00:00Z)", data);
+```
+
+### Property Access
+
+Access nested properties in expressions:
+
+```javascript
+query("$.users.map($item.address.city)", data);
+query("$.users.map(_.address.city)", data);
+query("$.users.map(u => u.address.city)", data);
+query("$.items.filter($item.specs.weight > 100)", data);
+```
+
+### Array Access
+
+Use brackets in expressions:
+
+```javascript
+query("$.users.map($item.tags[0])", data);
+query("$.users.map(_.tags[0])", data);
+query("$.data.map($item.values[$index])", data);
+```
+
+### Nested Method Calls
+
+Call methods within expressions for complex transformations:
+
+```javascript
+const data = {
+  departments: [
+    {
+      name: "Engineering",
+      employees: [
+        { name: "Alice", age: 30 },
+        { name: "Bob", age: 25 },
+        { name: "Carol", age: 35 },
+      ],
+    },
+    {
+      name: "Sales",
+      employees: [
+        { name: "Dave", age: 40 },
+        { name: "Eve", age: 28 },
+      ],
+    },
+  ],
+};
+
+// Filter and map employees within each department (concise)
+query("$.departments.map(_.employees.filter(age > 28).map(name))", data);
+// => [['Alice', 'Carol'], ['Dave']]
+
+// With arrow syntax for parent access
+query(
+  '$.departments.map(dept => dept.employees.map(emp => dept.name + ": " + emp.name))',
+  data,
+);
+// => [['Engineering: Alice', 'Engineering: Bob', 'Engineering: Carol'], ['Sales: Dave', 'Sales: Eve']]
+
+// Count employees per department
+query("$.departments.map(_.employees.count())", data);
+// => [3, 2]
+
+// Average age per department
+query("$.departments.map(_.employees.avg(age))", data);
+// => [30, 34]
+
+// Multiple levels of nesting with underscore
+query("$.departments.map(_.employees.sort(age).first().name)", data);
+// => ['Bob', 'Eve']
+```
+
+## Examples
+
+### E-commerce
+
+```javascript
+const orders = {
+  items: [
+    { name: "Laptop", price: 999, quantity: 2, inStock: true },
+    { name: "Mouse", price: 25, quantity: 5, inStock: true },
+    { name: "Keyboard", price: 75, quantity: 0, inStock: false },
+  ],
+};
+
+// Total value of in-stock items
+query("$.items.filter(inStock).map(price * quantity).sum($item)", orders);
+// => 2048
+
+// Most expensive item name
+query("$.items.sort(price).last().name", orders);
+// => 'Laptop'
+
+// Average price
+query("$.items.avg(price)", orders);
+// => 366.33...
+```
+
+### User Management
+
+```javascript
+const company = {
+  departments: [
+    {
+      name: "Engineering",
+      employees: [
+        { name: "Alice", salary: 120000, startDate: "2020-01-15" },
+        { name: "Bob", salary: 95000, startDate: "2021-06-01" },
+      ],
+    },
+    {
+      name: "Sales",
+      employees: [{ name: "Carol", salary: 85000, startDate: "2019-03-10" }],
+    },
+  ],
+};
+
+// All department names
+query('$.departments.map(name).join(", ")', company);
+// => 'Engineering, Sales'
+
+// High earners in each department
+query(
+  "$.departments.map($item.employees.filter(salary > 100000).map(name))",
+  company,
+);
+// => [['Alice'], []]
+
+// Recent hires
+query(
+  "$.departments[0].employees.filter(startDate > @2021-01-01).map(name)",
+  company,
+);
+// => ['Bob']
+
+// Department sizes
+query("$.departments.map($item.employees.count())", company);
+// => [2, 1]
+```
+
+### Data Analysis
+
+```javascript
+const metrics = {
+  daily: [
+    { date: "2024-01-01", views: 1200, clicks: 45 },
+    { date: "2024-01-02", views: 1500, clicks: 67 },
+    { date: "2024-01-03", views: 980, clicks: 32 },
+  ],
+};
+
+// Click-through rates
+query("$.daily.map(clicks / views * 100)", metrics);
+// => [3.75, 4.47, 3.27]
+
+// Best performing day
+query("$.daily.sort(clicks).last().date", metrics);
+// => '2024-01-02'
+
+// Total engagement
+query("$.daily.sum(views + clicks)", metrics);
+// => 3824
+```
+
+## API Reference
+
+### query(expression, data)
+
+Execute a query against data.
+
+**Parameters:**
+
+- `expression` (string): Query expression
+- `data` (any): Data to query
+
+**Returns:** Query result
+
+```javascript
+const result = query("$.users.filter(age > 25)", data);
+```
+
+### tokenize(expression)
+
+Parse expression into tokens (exported for testing/debugging).
+
+**Parameters:**
+
+- `expression` (string): Query expression
+
+**Returns:** Array of tokens
+
+```javascript
+import { tokenize } from "jsonq";
+const tokens = tokenize("$.users.map(name)");
+```
+
+### evaluate(tokens, data)
+
+Evaluate parsed tokens against data (exported for testing/debugging).
+
+**Parameters:**
+
+- `tokens` (Array): Parsed tokens
+- `data` (any): Data to evaluate
+
+**Returns:** Evaluation result
+
+```javascript
+import { evaluate, tokenize } from "jsonq";
+const tokens = tokenize("$.users.count()");
+const result = evaluate(tokens, data);
+```
+
+## Advanced Usage
+
+### Complex Filters
+
+Combine multiple conditions:
+
+```javascript
+query(
+  `
+  $.users
+    .filter(age >= 21 && age <= 65)
+    .filter(active == true)
+    .filter(verified == true)
+    .sort(name)
+`,
+  data,
+);
+```
+
+### Nested Access Patterns
+
+Access deeply nested data:
+
+```javascript
+query('$.company.departments[0].teams[2].members.filter(role == "lead")', data);
+```
+
+Combine nested methods with property navigation:
+
+```javascript
+// Get top performer from each department
+query("$.departments.map($item.employees.sort(performance).last())", data);
+
+// Filter departments by employee criteria
+query(
+  "$.departments.filter($item.employees.filter(certified == true).count() > 5)",
+  data,
+);
+
+// Aggregate across multiple levels
+query("$.regions.map($item.stores.map($item.sales.sum(amount)))", data);
+```
+
+### Dynamic Calculations
+
+Use expressions in any method:
+
+```javascript
+query("$.products.filter((price - cost) / price > 0.3)", data);
+```
+
+### Index-Based Operations
+
+Use `$index` for position-dependent logic:
+
+```javascript
+query("$.items.filter($index % 2 == 0)", data); // Even indices
+query("$.items.map($index * 10 + $item.value)", data);
+```
+
+## Type Handling
+
+### Null and Undefined
+
+Safe navigation with optional chaining behavior:
+
+```javascript
+const data = [{ user: { name: "Alice" } }, { user: null }, {}];
+
+query("map($item.user.name)", data);
+// => ['Alice', undefined, undefined]
+```
+
+### Missing Properties
+
+Accessing non-existent properties returns `undefined`:
+
+```javascript
+query("$.users.map(nonexistent)", data);
+// => [undefined, undefined, ...]
+```
+
+### Type Coercion
+
+Dates are automatically coerced for comparison:
+
+```javascript
+query("filter(dateString > @2024-01-01)", data);
+```
+
+## Performance Tips
+
+1. **Filter Early**: Apply filters before expensive operations
+
+   ```javascript
+   // Good
+   query("$.users.filter(active).map(expensiveOperation)", data);
+
+   // Less efficient
+   query("$.users.map(expensiveOperation).filter(active)", data);
+   ```
+
+2. **Avoid Redundant Sorts**: Sort only when necessary
+
+   ```javascript
+   // If you only need first/last, consider avoiding sort
+   query("$.users.sort(age).first()", data);
+   ```
+
+3. **Use Specific Expressions**: More specific filters run faster
+
+   ```javascript
+   // More specific
+   query("$.users.filter(id == 123)", data);
+
+   // Less specific
+   query("$.users.filter(id > 0)", data);
+   ```
+
+## Error Handling
+
+The library throws errors for:
+
+- Unknown methods
+- Invalid expressions
+- Unknown variables
+- Unexpected tokens
+- Invalid literals
+
+```javascript
+try {
+  query("$.users.unknownMethod()", data);
+} catch (error) {
+  console.error(error.message); // "Unknown method: unknownMethod"
+}
+```
+
+## Testing
+
+Run the test suite:
+
+```bash
+npm test
+```
+
+## License
+
+MIT