forge-sql-orm 2.1.3 → 2.1.5

package/README.md

@@ -19,10 +19,12 @@
 - ✅ **Custom Drizzle Driver** for direct integration with @forge/sql
 - ✅ **Local Cache System (Level 1)** for in-memory query optimization within single resolver invocation scope
 - ✅ **Global Cache System (Level 2)** with cross-invocation caching, automatic cache invalidation and context-aware operations (using [@forge/kvs](https://developer.atlassian.com/platform/forge/storage-reference/storage-api-custom-entities/) )
+- ✅ **Memory Usage Monitoring**: Automated detection of memory-intensive queries with configurable thresholds (essential for Atlassian's 16 MiB per query limit)
 - ✅ **Type-Safe Query Building**: Write SQL queries with full TypeScript support
 - ✅ **Supports complex SQL queries** with joins and filtering using Drizzle ORM
 - ✅ **Advanced Query Methods**: `selectFrom()`, `selectDistinctFrom()`, `selectCacheableFrom()`, `selectDistinctCacheableFrom()` for all-column queries with field aliasing
 - ✅ **Raw SQL Execution**: `execute()` and `executeCacheable()` methods for direct SQL queries with local and global caching
+- ✅ **Query Execution with Metadata**: `executeWithMetadata()` method for capturing detailed execution metrics including database execution time, response size, and Forge SQL metadata
 - ✅ **Common Table Expressions (CTEs)**: `with()` method for complex queries with subqueries
 - ✅ **Schema migration support**, allowing automatic schema evolution
 - ✅ **Automatic entity generation** from MySQL/tidb databases
@@ -63,6 +65,7 @@
 ### 🔒 Advanced Features
 - [Optimistic Locking](#optimistic-locking)
 - [Query Analysis and Performance Optimization](#query-analysis-and-performance-optimization)
+- [Memory Usage Monitoring](#memory-usage-monitoring)
 - [Date and Time Types](#date-and-time-types)
 
 ### 🛠️ Development Tools
@@ -79,6 +82,7 @@
 - [Query Analysis Example](examples/forge-sql-orm-example-query-analyses)
 - [Organization Tracker Example](examples/forge-sql-orm-example-org-tracker)
 - [Checklist Example](examples/forge-sql-orm-example-checklist)
+- [Cache Example](examples/forge-sql-orm-example-cache) - Advanced caching capabilities with `topSlowestStatementLastHourTrigger` memory monitoring
 
 ### 📚 Reference
 - [ForgeSqlOrmOptions](#forgesqlormoptions)
@@ -95,6 +99,7 @@
 - [Global Cache System (Level 2)](#global-cache-system-level-2) - Cross-invocation persistent caching
 - [Local Cache System (Level 1)](#local-cache-operations-level-1) - In-memory invocation caching
 - [Optimistic Locking](#optimistic-locking) - Data consistency
+- [Memory Usage Monitoring](#memory-usage-monitoring) - Memory-intensive query detection
 - [Migration Tools](#web-triggers-for-migrations) - Database migrations
 - [Query Analysis](#query-analysis-and-performance-optimization) - Performance optimization
 
@@ -103,6 +108,7 @@
 - [Optimistic Locking Example](examples/forge-sql-orm-example-optimistic-locking) - Real-world conflict handling
 - [Organization Tracker Example](examples/forge-sql-orm-example-org-tracker) - Complex relationships
 - [Checklist Example](examples/forge-sql-orm-example-checklist) - Jira integration
+- [Cache Example](examples/forge-sql-orm-example-cache) - Advanced caching with memory monitoring
 
 ## Usage Approaches
 
@@ -328,6 +334,16 @@ const cachedRawUsers = await forgeSQL.executeCacheable(
   300
 );
 
+// Raw SQL with execution metadata
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+  async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+  (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+    console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+    console.log(`Response size: ${totalResponseSize} bytes`);
+    console.log('Forge metadata:', forgeMetadata);
+  }
+);
+
 // Common Table Expressions (CTEs)
 const userStats = await forgeSQL
   .with(
@@ -401,6 +417,16 @@ const cachedRawUsers = await forgeSQL.executeCacheable(
   [true], 
   300
 );
+
+// Raw SQL with execution metadata
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+  async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+  (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+    console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+    console.log(`Response size: ${totalResponseSize} bytes`);
+    console.log('Forge metadata:', forgeMetadata);
+  }
+);
 ```
 
 ## Setting Up Caching with @forge/kvs (Optional)
@@ -772,6 +798,7 @@ const optimizedData = await forgeSQL.executeWithLocalCacheContextAndReturnValue(
 | `selectDistinctCacheableFrom()` | Distinct all-column queries with field aliasing and caching | ❌ No | Local + Global Cache |
 | `execute()` | Raw SQL queries with local caching | ❌ No | Local Cache |
 | `executeCacheable()` | Raw SQL queries with local and global caching | ❌ No | Local + Global Cache |
+| `executeWithMetadata()` | Raw SQL queries with execution metrics capture | ❌ No | Local Cache |
 | `with()` | Common Table Expressions (CTEs) | ❌ No | Local Cache |
 where Cache context - allows you to batch cache invalidation events and bypass cache reads for affected tables.
 
@@ -1158,6 +1185,17 @@ const users = await forgeSQL
 const users = await forgeSQL
   .executeCacheable("SELECT * FROM users WHERE active = ?", [true], 300);
 
+// Using executeWithMetadata() for capturing execution metrics
+const usersWithMetadata = await forgeSQL
+  .executeWithMetadata(
+    async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+    (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+      console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+      console.log(`Response size: ${totalResponseSize} bytes`);
+      console.log('Forge metadata:', forgeMetadata);
+    }
+  );
+
 // Using execute() with complex queries
 const userStats = await forgeSQL
   .execute(`
@@ -1473,6 +1511,16 @@ await forgeSQL.executeWithLocalContext(async () => {
     [true]
   );
   
+  // Raw SQL with execution metadata and local caching
+  const usersWithMetadata = await forgeSQL.executeWithMetadata(
+    async () => await forgeSQL.execute("SELECT id, name FROM users WHERE active = ?", [true]),
+    (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+      console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+      console.log(`Response size: ${totalResponseSize} bytes`);
+      console.log('Forge metadata:', forgeMetadata);
+    }
+  );
+  
   // Insert operation - evicts local cache for users table
   await forgeSQL.insert(users).values({ name: 'New User', active: true });
   
@@ -1641,6 +1689,16 @@ const userStats = await forgeSQL
   })
   .from(sql`activeUsers au`)
   .leftJoin(sql`completedOrders co`, eq(sql`au.id`, sql`co.userId`));
+
+// Using executeWithMetadata() for capturing execution metrics with caching
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+  async () => await forgeSQL.executeCacheable("SELECT * FROM users WHERE active = ?", [true], 300),
+  (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+    console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+    console.log(`Response size: ${totalResponseSize} bytes`);
+    console.log('Forge metadata:', forgeMetadata);
+  }
+);
 ```
 
 ### Manual Cache Management
@@ -1716,6 +1774,7 @@ The `ForgeSqlOrmOptions` object allows customization of ORM behavior:
 | Option                     | Type      | Description                                                                                                                                                                                                                     |
 | -------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `logRawSqlQuery`           | `boolean` | Enables logging of raw SQL queries in the Atlassian Forge Developer Console. Useful for debugging and monitoring. Defaults to `false`.                                                                                         |
+| `logCache`                 | `boolean` | Enables logging of cache operations (hits, misses, evictions) in the Atlassian Forge Developer Console. Useful for debugging caching issues. Defaults to `false`.                                                               |
 | `disableOptimisticLocking` | `boolean` | Disables optimistic locking. When set to `true`, no additional condition (e.g., a version check) is added during record updates, which can improve performance. However, this may lead to conflicts when multiple transactions attempt to update the same record concurrently. |
 | `additionalMetadata`       | `object`  | Allows adding custom metadata to all entities. This is useful for tracking common fields across all tables (e.g., `createdAt`, `updatedAt`, `createdBy`, etc.). The metadata will be automatically added to all generated entities. |
 | `cacheEntityName`          | `string`  | KVS Custom entity name for cache storage. Must match the `name` in your `manifest.yml` storage entities configuration. Required for caching functionality. Defaults to `"cache"`.                                                                                                                         |
@@ -2015,6 +2074,228 @@ This analysis provides insights into:
 - Resource usage at each step
 - Performance optimization opportunities
 
+## Memory Usage Monitoring
+
+[↑ Back to Top](#table-of-contents)
+
+Forge-SQL-ORM provides automated memory usage monitoring capabilities to help you identify and track memory-intensive queries in your Forge SQL instance. This feature is **essential for Atlassian Forge applications** as it helps you stay within the **16 MiB per query memory limit** and provides detailed insights for optimization.
+
+### Why Memory Monitoring is Critical
+
+Atlassian Forge SQL has a strict **16 MiB memory limit per query**. Unlike slow query detection (which is available in the Forge Developer Console), there's **no built-in way to monitor memory usage** of your queries. This monitoring system fills that gap by:
+
+- **Detecting memory-intensive queries** before they hit the 16 MiB limit
+- **Providing detailed memory metrics** including average and maximum memory usage
+- **Showing execution plans** to help optimize memory consumption
+- **Configurable thresholds** to match your application's memory requirements
+- **Scheduled monitoring** via Forge scheduler triggers
+
+### Overview
+
+The memory monitoring system:
+- **Automatically detects memory-intensive queries** based on configurable memory thresholds
+- **Provides detailed memory metrics** including execution time, memory usage, and execution plans
+- **Logs memory issues** to the Forge Developer Console for easy debugging
+- **Supports scheduled monitoring** via Forge scheduler triggers
+- **Filters out system queries** to focus on your application's performance
+
+### Key Features
+
+- **Memory-Focused Monitoring**: Primary focus on memory usage with configurable thresholds
+- **Atlassian 16 MiB Limit Awareness**: Designed specifically for Forge SQL's memory constraints
+- **Execution Plan Analysis**: Shows detailed query plans to help optimize memory consumption
+- **Configurable Thresholds**: Set custom memory usage thresholds (default: 4MB warning)
+- **Automatic Filtering**: Excludes system queries (`Use`, `Set`, `Show`) and empty queries
+- **Scheduled Monitoring**: Run automatically on configurable intervals
+
+### Basic Usage
+
+#### 1. Import the Trigger
+
+```typescript
+import ForgeSQL, { topSlowestStatementLastHourTrigger } from "forge-sql-orm";
+```
+
+#### 2. Create a Scheduler Function
+
+```typescript
+import ForgeSQL, { topSlowestStatementLastHourTrigger } from 'forge-sql-orm';
+
+// Initialize ForgeSQL ORM instance
+const forgeSQL = new ForgeSQL();
+
+// Basic usage with default thresholds (300ms latency, 8MB memory warning)
+export const memoryUsageTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL);
+
+// Conservative memory monitoring: 4MB warning (well below 16MB limit)
+export const conservativeMemoryTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { memoryThresholdBytes: 4 * 1024 * 1024 });
+
+// Aggressive memory monitoring: 12MB warning (75% of 16MB limit)
+export const aggressiveMemoryTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { memoryThresholdBytes: 12 * 1024 * 1024 });
+
+// Memory-only monitoring: Only trigger on memory usage (latency effectively disabled)
+export const memoryOnlyTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { warnThresholdMs: 10000, memoryThresholdBytes: 4 * 1024 * 1024 });
+
+// Latency-only monitoring: Only trigger on slow queries (memory effectively disabled)
+export const latencyOnlyTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { warnThresholdMs: 500, memoryThresholdBytes: 16 * 1024 * 1024 });
+
+// With execution plan in logs
+export const withPlanTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { showPlan: true });
+
+// With cache logging enabled
+export const withCacheLoggingTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { logCache: true });
+
+// With both execution plan and cache logging
+export const withFullLoggingTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { showPlan: true, logCache: true });
+
+// With custom ORM options for debugging
+const forgeSQL = new ForgeSQL({
+  logRawSqlQuery: true,
+  logCache: true,
+  cacheEntityName: "cache"
+});
+
+
+#### 3. Configure in manifest.yml
+
+**As Scheduler Trigger (Recommended for Production):**
+```yaml
+scheduledTrigger:
+  - key: memory-usage-trigger
+    function: memoryUsageTrigger
+    interval: hour  # Required: only hour interval is supported
+
+function:
+  - key: memoryUsageTrigger
+    handler: index.memoryUsageTrigger
+```
+
+**As Web Trigger (Development Only):**
+```yaml
+webtrigger:
+  - key: print-slowest-queries
+    function: memoryUsageTrigger
+
+function:
+  - key: memoryUsageTrigger
+    handler: index.memoryUsageTrigger
+```
+
+> **⚠️ Important**: Web triggers are not recommended for production as they violate the "run-on-atlassian" principle. Use scheduler triggers for production monitoring.
+
+### How It Works
+
+The memory monitoring trigger works differently depending on how it's configured:
+
+#### Scheduler Trigger Mode (Production)
+
+When used as a **scheduler trigger**, the system:
+- **Runs automatically** on the configured interval (hour only)
+- **Logs to Forge Developer Console** only when thresholds are exceeded
+- **No HTTP response** - operates silently in the background
+- **Perfect for production** monitoring without violating "run-on-atlassian"
+
+**Example Console Log Output:**
+```
+1. Select  avg=3006.03ms  max=3006.03ms  mem≈0.08MB(max 0.08MB)  exec=1 
+   digest=28344800f90f6c929484e83337404df7e55a660c5f4ce922c4b298ab5e90c425
+   sql=select `demo_users` . `id` as `a_userid_id` , `demo_users` . `name` as `a_username_name` , `demo_orders` . `product` as `a_product_product` , `demo_orders` . `id` as `a_productid_id` , `sleep` ( ? ) from `demo_users` inner join `demo_orders` on `demo_orders` . `user_id` = `demo_users` . `id`
+
+full plan:
+id task estRows operator info actRows execution info memory disk
+Projection_7 root 2.50 forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_users.id, forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_users.name, forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_orders.product, forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_orders.id, sleep(?)->Column#7 3 time:3s, loops:2, Concurrency:OFF 1.98 KB N/A
+└─IndexHashJoin_14 root 2.50 inner join, inner:IndexLookUp_11, outer key:forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_users.id, inner key:forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_orders.user_id, equal cond:eq(forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_users.id, forge_38dd1c6156b94bb59c2c9a45582bbfc7.demo_orders.user_id) 3 time:2.11ms, loops:2, inner:{total:1.31ms, concurrency:5, task:1, construct:6.62µs, fetch:1.3ms, build:4.68µs, join:6.14µs} 57.9 KB N/A
+```
+
+#### Web Trigger Mode (Development)
+
+When used as a **web trigger**, the system:
+- **Runs on-demand** when the web endpoint is called
+- **Returns JSON response** with detailed metrics
+- **Logs to console** AND provides structured data
+- **Useful for development** and debugging
+
+**Example JSON Response:**
+```json
+{
+  "success": true,
+  "window": "last_1h",
+  "top": 1,
+  "warnThresholdMs": 300,
+  "memoryThresholdBytes": 8388608,
+  "rows": [
+    {
+      "rank": 1,
+      "digest": "abc123...",
+      "stmtType": "Select",
+      "schemaName": "myapp",
+      "execCount": 150,
+      "avgLatencyMs": 450.25,
+      "maxLatencyMs": 1200.50,
+      "minLatencyMs": 200.10,
+      "avgProcessTimeMs": 400.20,
+      "avgWaitTimeMs": 50.05,
+      "avgBackoffTimeMs": 0.00,
+      "avgMemMB": 2.5,
+      "maxMemMB": 8.2,
+      "avgMemBytes": 2621440,
+      "maxMemBytes": 8598323,
+      "avgTotalKeys": 1000,
+      "firstSeen": "2024-01-15 10:30:00",
+      "lastSeen": "2024-01-15 11:30:00",
+      "planInCache": true,
+      "planCacheHits": 120,
+      "digestText": "SELECT * FROM users WHERE active = ?",
+      "plan": "IndexScan(users, idx_active)..."
+    }
+  ],
+  "generatedAt": "2024-01-15T11:30:00.000Z"
+}
+```
+
+### Configuration Options
+
+#### Threshold Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `warnThresholdMs` | `number` | `300` | Latency threshold in milliseconds (secondary) |
+| `memoryThresholdBytes` | `number` | `8 * 1024 * 1024` | **Memory usage threshold in bytes (primary focus)** |
+| `showPlan` | `boolean` | `false` | Whether to include execution plan in logs |
+| `logCache` | `boolean` | `false` | Whether to log cache operations |
+
+**⚠️ Important: OR Logic**
+The monitoring uses **OR logic** - if **either** threshold is exceeded, the query will be logged/returned:
+- Query exceeds `warnThresholdMs` **OR** `memoryThresholdBytes` → Included in results
+- This means you can set different thresholds for different monitoring priorities
+- No need to exceed both thresholds simultaneously
+
+**💡 Pro Tips:**
+- **Memory-only monitoring**: Set `warnThresholdMs` to a very high value (e.g., 10000ms) to trigger only on memory usage
+- **Latency-only monitoring**: Set `memoryThresholdBytes` to 16MB (16 * 1024 * 1024) to trigger only on latency
+- **Combined monitoring**: Use both thresholds for comprehensive monitoring
+- **Execution plan analysis**: Set `showPlan: true` to include detailed execution plans in logs (useful for debugging)
+- **Cache debugging**: Set `logCache: true` to log cache operations and debug caching issues
+
+**Memory Threshold Guidelines:**
+- **Conservative**: 4MB (25% of 16MB limit)
+- **Default**: 8MB (50% of 16MB limit) 
+- **Aggressive**: 12MB (75% of 16MB limit)
+- **Critical**: 14MB (87.5% of 16MB limit)
+
+#### Available Intervals
+
+- `hour` - **Every hour (Required)** - Statistics are available for approximately 12 hours
+
+> **⚠️ Important**: Due to Forge SQL's statistics retention policy (approximately 12 hours), **only `hour` interval is supported**. Using `day` or `week` intervals will result in incomplete or missing data.
 
 ## Migration Guide
 
@@ -2107,6 +2388,16 @@ const cachedRawUsers = await forgeSQL.executeCacheable(
   300
 );
 
+// ✅ Raw SQL execution with metadata capture
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+  async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+  (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+    console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+    console.log(`Response size: ${totalResponseSize} bytes`);
+    console.log('Forge metadata:', forgeMetadata);
+  }
+);
+
 // ✅ Common Table Expressions (CTEs)
 const userStats = await forgeSQL
   .with(