json-rest-api 1.0.7 → 1.0.8

package/docs/ONBOARDING/Computed_Properties_And_Sparse_Fields.md

@@ -0,0 +1,398 @@
+# Computed Properties and Sparse Fields: Deep Dive
+
+This guide explains how computed fields, sparse fieldsets, and the `enrichAttributes` method work together in the json-rest-api library.
+
+## Table of Contents
+1. [Overview](#overview)
+2. [Key Concepts](#key-concepts)
+3. [The Data Flow](#the-data-flow)
+4. [enrichAttributes Method](#enrichattributes-method)
+5. [Computed Dependencies](#computed-dependencies)
+6. [Sparse Fieldsets](#sparse-fieldsets)
+7. [Complete Example Walkthrough](#complete-example-walkthrough)
+
+## Overview
+
+The json-rest-api library supports computed fields - virtual fields that are calculated on-the-fly based on other fields. When combined with sparse fieldsets, the system intelligently fetches only the necessary dependencies from the database, computes the requested fields, and then removes any dependencies that weren't explicitly requested.
+
+## Key Concepts
+
+### Computed Fields
+Fields that don't exist in the database but are calculated from other fields:
+
+```javascript
+await api.addResource('products', {
+  schema: {
+    name: { type: 'string', required: true },
+    price: { type: 'number' },
+    cost: { type: 'number', normallyHidden: true },
+    profit_margin: {
+      type: 'number',
+      computed: true,  // Mark as computed field
+      dependencies: ['price', 'cost'],
+      compute: ({ attributes }) => {
+        if (!attributes.price || !attributes.cost) return null;
+        return ((attributes.price - attributes.cost) / attributes.price * 100).toFixed(2);
+      }
+    }
+  }
+});
+```
+
+### Sparse Fieldsets
+JSON:API feature allowing clients to request only specific fields:
+```
+GET /products/1?fields[products]=name,profit_margin
+```
+
+### Computed Dependencies
+Fields that are needed to compute other fields but weren't explicitly requested. In the above example, if only `profit_margin` is requested, both `price` and `cost` are computed dependencies.
+
+## The Data Flow
+
+Here's the complete flow when a request with sparse fieldsets and computed fields is made:
+
+```
+1. Request arrives: GET /products/1?fields[products]=name,profit_margin
+                                                       ↓
+2. buildFieldSelection() determines what to SELECT from database
+                                                       ↓
+3. Database query: SELECT id, name, price, cost FROM products WHERE id = 1
+                                                       ↓
+4. Raw data: { id: 1, name: 'Widget', price: 100, cost: 60 }
+                                                       ↓
+5. enrichAttributes() is called to compute fields and filter data
+                                                       ↓
+6. Final response: { id: '1', name: 'Widget', profit_margin: '40.00' }
+```
+
+## enrichAttributes Method
+
+### When It's Called
+
+`enrichAttributes` is called in two main scenarios:
+
+1. **For main resources** - in `dataGetQuery` and `dataGet` methods:
+```javascript
+// rest-api-plugin.js, line 497
+entry.attributes = await scope.enrichAttributes({ 
+  attributes: entry.attributes, 
+  parentContext: context,
+  requestedComputedFields: requestedComputedFields,
+  isMainResource: true,
+  computedDependencies: context.computedDependencies
+})
+```
+
+2. **For included resources** - when processing relationships:
+```javascript
+// rest-api-plugin.js, line 515
+entry.attributes = await entryScope.enrichAttributes({ 
+  attributes: entry.attributes, 
+  parentContext: context,
+  requestedComputedFields: entryRequestedComputed,
+  isMainResource: false,
+  computedDependencies: entry.__$jsonrestapi_computed_deps$__
+})
+```
+
+### Parameters
+
+```javascript
+{
+  attributes,              // Raw attributes from database
+  parentContext,          // Original request context (has queryParams, transaction, etc.)
+  requestedComputedFields, // Array of computed fields to calculate
+  isMainResource,         // Boolean - true for main, false for included
+  computedDependencies    // Array of fields to remove if not explicitly requested
+}
+```
+
+### What It Does
+
+Here's the implementation with annotations:
+
+```javascript
+// rest-api-plugin.js, line 2368
+addScopeMethod('enrichAttributes', async ({ context, params, runHooks, scopeName, scopes, api, helpers }) => {
+  const { attributes, parentContext, requestedComputedFields, isMainResource, computedDependencies } = params || {};
+  
+  // Step 1: Get schema information
+  const schemaStructure = scopes[scopeName]?.vars?.schemaInfo?.schemaStructure || {};
+  const computedFields = scopes[scopeName]?.vars?.schemaInfo?.computed || {};
+  
+  // Step 2: Filter hidden fields (removes hidden:true and normallyHidden:true unless requested)
+  const requestedFields = parentContext?.queryParams?.fields?.[scopeName];
+  const filteredAttributes = filterHiddenFields(attributes, { structure: schemaStructure }, requestedFields);
+  
+  // Step 3: Handle virtual fields based on sparse fieldsets
+  // Virtual fields (marked with virtual: true in schema) are already in filteredAttributes
+  // We need to remove them if they weren't requested in sparse fieldsets
+  if (requestedFields && requestedFields.length > 0) {
+    Object.entries(schemaStructure).forEach(([fieldName, fieldDef]) => {
+      if (fieldDef.virtual === true && fieldName in filteredAttributes) {
+        if (!requestedFields.includes(fieldName)) {
+          delete filteredAttributes[fieldName];
+        }
+      }
+    });
+  }
+  
+  // Step 4: Determine which computed fields to calculate
+  let fieldsToCompute = [];
+  if (requestedComputedFields) {
+    fieldsToCompute = requestedComputedFields;
+  } else if (isMainResource || !parentContext?.queryParams?.fields) {
+    fieldsToCompute = Object.keys(computedFields);
+  }
+  
+  // Step 5: Calculate computed fields
+  const computeContext = {
+    attributes: attributes,  // ALL attributes including dependencies
+    record: { ...attributes },
+    context: parentContext,
+    helpers,
+    api,
+  };
+  
+  for (const fieldName of fieldsToCompute) {
+    const fieldDef = computedFields[fieldName];
+    if (fieldDef && fieldDef.compute) {
+      try {
+        filteredAttributes[fieldName] = await fieldDef.compute(computeContext);
+      } catch (error) {
+        console.error(`Error computing field '${fieldName}' for ${scopeName}:`, error);
+        filteredAttributes[fieldName] = null;
+      }
+    }
+  }
+  
+  // Step 6: Remove computed dependencies that weren't explicitly requested
+  const finalAttributes = { ...filteredAttributes };
+  if (requestedFields && computedDependencies && computedDependencies.length > 0) {
+    const requested = typeof requestedFields === 'string'
+      ? requestedFields.split(',').map(f => f.trim()).filter(f => f)
+      : requestedFields;
+    
+    for (const dep of computedDependencies) {
+      if (!requested.includes(dep)) {
+        delete finalAttributes[dep];
+      }
+    }
+  }
+  
+  // Step 7: Run enrichAttributes hooks
+  Object.assign(context, {
+    parentContext,
+    attributes: finalAttributes,
+    computedFields,
+    requestedComputedFields: fieldsToCompute,
+    scopeName,
+    helpers,
+    api
+  });
+  
+  await runHooks('enrichAttributes');
+  
+  return context.attributes;
+});
+```
+
+## Computed Dependencies
+
+### How They're Determined
+
+In `buildFieldSelection` (knex-field-helpers.js):
+
+```javascript
+// When a computed field is requested, add its dependencies
+if (computedDef && computedDef.dependencies) {
+  for (const dep of computedDef.dependencies) {
+    if (!fieldsToSelect.has(dep)) {
+      fieldsToSelect.add(dep);
+      // Track that this is a dependency (not explicitly requested)
+      if (!requested.includes(dep)) {
+        computedDependencies.add(dep);
+      }
+    }
+  }
+}
+```
+
+### Flow Through the System
+
+1. **Database Query Level**: Dependencies are added to SELECT clause
+2. **Knex Plugin**: Passes `computedDependencies` in context
+3. **REST API Plugin**: Receives them in `context.computedDependencies`
+4. **enrichAttributes**: Removes them if not explicitly requested
+
+### For Included Resources
+
+The fix we implemented ensures computed dependencies flow through for included resources:
+
+```javascript
+// In knex-relationship-includes.js, when creating JSON:API records:
+if (fieldSelectionInfo?.computedDependencies) {
+  jsonApiRecord[COMPUTED_DEPENDENCIES_KEY] = fieldSelectionInfo.computedDependencies;
+}
+
+// Then in rest-api-plugin.js when processing includes:
+entry.attributes = await entryScope.enrichAttributes({ 
+  // ...
+  computedDependencies: entry.__$jsonrestapi_computed_deps$__
+})
+```
+
+## Sparse Fieldsets
+
+### Implementation Details
+
+1. **Field Validation**: The system validates requested fields exist:
+```javascript
+// knex-field-helpers.js
+if (!fieldDef) throw new Error(`Unknown sparse field '${field}' requested for '${scopeName}'`);
+```
+
+2. **Hidden Field Handling**: 
+   - `hidden: true` fields are NEVER returned
+   - `normallyHidden: true` fields are returned ONLY if explicitly requested
+
+3. **Computed Field Handling**: Computed fields can be requested like regular fields
+
+## Complete Example Walkthrough
+
+Let's trace a request through the system:
+
+### Request
+```
+GET /users/1?include=posts&fields[users]=name,profit_margin&fields[posts]=title,popularity
+```
+
+### Step 1: Build Field Selection for Main Resource
+
+```javascript
+// For users (main resource)
+buildFieldSelection(usersScope, {
+  context: {
+    queryParams: { fields: { users: 'name,profit_margin' } }
+  }
+})
+// Returns:
+{
+  fieldsToSelect: ['id', 'name', 'revenue', 'cost'],  // Added dependencies
+  requestedFields: ['name', 'profit_margin'],
+  computedDependencies: ['revenue', 'cost'],  // To be removed later
+  idProperty: 'id'
+}
+```
+
+### Step 2: Database Queries
+
+```sql
+-- Main resource
+SELECT id, name, revenue, cost FROM users WHERE id = 1
+
+-- Included posts (hasMany)
+SELECT id, title, views, internal_score, author_id 
+FROM posts 
+WHERE author_id IN (1)
+```
+
+### Step 3: enrichAttributes for Main Resource
+
+```javascript
+// Input attributes from DB
+{ id: 1, name: 'John Doe', revenue: 100000, cost: 60000 }
+
+// After enrichAttributes
+{ id: '1', name: 'John Doe', profit_margin: '40.00' }
+// Note: revenue and cost are removed as computed dependencies
+```
+
+### Step 4: enrichAttributes for Included Resources
+
+```javascript
+// Each post goes through enrichAttributes
+// Input: { id: 1, title: 'First Post', views: 150, internal_score: 20, author_id: 1 }
+// Output: { id: '1', title: 'First Post', popularity: 'popular' }
+// Note: views and internal_score removed as dependencies
+```
+
+### Step 5: Final JSON:API Response
+
+```json
+{
+  "data": {
+    "type": "users",
+    "id": "1",
+    "attributes": {
+      "name": "John Doe",
+      "profit_margin": "40.00"
+    },
+    "relationships": {
+      "posts": {
+        "data": [
+          { "type": "posts", "id": "1" },
+          { "type": "posts", "id": "2" }
+        ]
+      }
+    }
+  },
+  "included": [
+    {
+      "type": "posts",
+      "id": "1",
+      "attributes": {
+        "title": "First Post",
+        "popularity": "popular"
+      }
+    },
+    {
+      "type": "posts",
+      "id": "2",
+      "attributes": {
+        "title": "Second Post",
+        "popularity": "normal"
+      }
+    }
+  ]
+}
+```
+
+### Step 6: Transformation to Simplified Format
+
+```javascript
+{
+  "id": "1",
+  "name": "John Doe",
+  "profit_margin": "40.00",
+  "posts_ids": ["1", "2"],  // Added from relationships
+  "posts": [
+    {
+      "id": "1",
+      "title": "First Post",
+      "popularity": "popular"
+    },
+    {
+      "id": "2", 
+      "title": "Second Post",
+      "popularity": "normal"
+    }
+  ]
+}
+```
+
+## Key Takeaways
+
+1. **enrichAttributes is the central processing point** for computed fields and field filtering
+2. **Computed dependencies are tracked throughout the pipeline** to ensure they're available for computation but removed from final output
+3. **The fix for included resources** ensures the same dependency removal logic applies to all resources
+4. **Sparse fieldsets control attributes**, not relationships (hence `_ids` fields always appear)
+5. **The system is efficient** - it only fetches and computes what's necessary
+
+## Debugging Tips
+
+1. Check `context.computedDependencies` to see what will be removed
+2. Use `console.log` in `enrichAttributes` to trace field transformations
+3. Verify `fieldSelectionInfo` contains expected dependencies
+4. Remember that `_ids` fields are relationships, not attributes