forge-sql-orm 2.1.4 → 2.1.6

package/README.md

@@ -19,11 +19,12 @@
 - ✅ **Custom Drizzle Driver** for direct integration with @forge/sql
 - ✅ **Local Cache System (Level 1)** for in-memory query optimization within single resolver invocation scope
 - ✅ **Global Cache System (Level 2)** with cross-invocation caching, automatic cache invalidation and context-aware operations (using [@forge/kvs](https://developer.atlassian.com/platform/forge/storage-reference/storage-api-custom-entities/) )
-- ✅ **Memory Usage Monitoring**: Automated detection of memory-intensive queries with configurable thresholds (essential for Atlassian's 16 MiB per query limit)
+- ✅ **Performance Monitoring**: Automated detection of memory-intensive queries with configurable thresholds (essential for Atlassian's 16 MiB per query limit)
 - ✅ **Type-Safe Query Building**: Write SQL queries with full TypeScript support
 - ✅ **Supports complex SQL queries** with joins and filtering using Drizzle ORM
 - ✅ **Advanced Query Methods**: `selectFrom()`, `selectDistinctFrom()`, `selectCacheableFrom()`, `selectDistinctCacheableFrom()` for all-column queries with field aliasing
-- ✅ **Raw SQL Execution**: `execute()` and `executeCacheable()` methods for direct SQL queries with local and global caching
+- ✅ **Query Execution with Metadata**: `executeWithMetadata()` method for capturing detailed execution metrics including database execution time, response size, and Forge SQL metadata
+- ✅ **Raw SQL Execution**: `execute()`, `executeCacheable()`, `executeDDL()`, and `executeDDLActions()` methods for direct SQL queries with local and global caching
 - ✅ **Common Table Expressions (CTEs)**: `with()` method for complex queries with subqueries
 - ✅ **Schema migration support**, allowing automatic schema evolution
 - ✅ **Automatic entity generation** from MySQL/tidb databases
@@ -64,7 +65,7 @@
 ### 🔒 Advanced Features
 - [Optimistic Locking](#optimistic-locking)
 - [Query Analysis and Performance Optimization](#query-analysis-and-performance-optimization)
-- [Memory Usage Monitoring](#memory-usage-monitoring)
+- [Performance Monitoring](#performance-monitoring)
 - [Date and Time Types](#date-and-time-types)
 
 ### 🛠️ Development Tools
@@ -81,7 +82,7 @@
 - [Query Analysis Example](examples/forge-sql-orm-example-query-analyses)
 - [Organization Tracker Example](examples/forge-sql-orm-example-org-tracker)
 - [Checklist Example](examples/forge-sql-orm-example-checklist)
-- [Cache Example](examples/forge-sql-orm-example-cache) - Advanced caching capabilities with `topSlowestStatementLastHourTrigger` memory monitoring
+- [Cache Example](examples/forge-sql-orm-example-cache) - Advanced caching capabilities with `topSlowestStatementLastHourTrigger` performance monitoring
 
 ### 📚 Reference
 - [ForgeSqlOrmOptions](#forgesqlormoptions)
@@ -333,6 +334,42 @@ const cachedRawUsers = await forgeSQL.executeCacheable(
   300
 );
 
+// Raw SQL with execution metadata
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+  async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+  (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+    console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+    console.log(`Response size: ${totalResponseSize} bytes`);
+    console.log('Forge metadata:', forgeMetadata);
+  }
+);
+
+// DDL operations for schema modifications
+await forgeSQL.executeDDL(`
+  CREATE TABLE users (
+    id INT PRIMARY KEY AUTO_INCREMENT,
+    name VARCHAR(255) NOT NULL,
+    email VARCHAR(255) UNIQUE
+  )
+`);
+
+// Execute regular SQL queries in DDL context for performance monitoring
+await forgeSQL.executeDDLActions(async () => {
+  // Execute regular SQL queries in DDL context for monitoring
+  const slowQueries = await forgeSQL.execute(`
+    SELECT * FROM INFORMATION_SCHEMA.STATEMENTS_SUMMARY 
+    WHERE AVG_LATENCY > 1000000
+  `);
+  
+  // Execute complex analysis queries in DDL context
+  const performanceData = await forgeSQL.execute(`
+    SELECT * FROM INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_HISTORY
+    WHERE SUMMARY_END_TIME > DATE_SUB(NOW(), INTERVAL 1 HOUR)
+  `);
+  
+  return { slowQueries, performanceData };
+});
+
 // Common Table Expressions (CTEs)
 const userStats = await forgeSQL
   .with(
@@ -406,6 +443,16 @@ const cachedRawUsers = await forgeSQL.executeCacheable(
   [true], 
   300
 );
+
+// Raw SQL with execution metadata
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+  async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+  (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+    console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+    console.log(`Response size: ${totalResponseSize} bytes`);
+    console.log('Forge metadata:', forgeMetadata);
+  }
+);
 ```
 
 ## Setting Up Caching with @forge/kvs (Optional)
@@ -757,6 +804,8 @@ const optimizedData = await forgeSQL.executeWithLocalCacheContextAndReturnValue(
 | `selectDistinctCacheableFrom()` | Distinct all-column queries with field aliasing and caching | ❌ No | Local + Global Cache |
 | `execute()` | Raw SQL queries with local caching | ❌ No | Local Cache |
 | `executeCacheable()` | Raw SQL queries with local and global caching | ❌ No | Local + Global Cache |
+| `executeDDL()` | DDL operations (CREATE, ALTER, DROP, etc.) | ❌ No | No Caching |
+| `executeDDLActions()` | Execute regular SQL queries in DDL operation context | ❌ No | No Caching |
 | `with()` | Common Table Expressions (CTEs) | ❌ No | Local Cache |
 
 
@@ -777,6 +826,9 @@ const optimizedData = await forgeSQL.executeWithLocalCacheContextAndReturnValue(
 | `selectDistinctCacheableFrom()` | Distinct all-column queries with field aliasing and caching | ❌ No | Local + Global Cache |
 | `execute()` | Raw SQL queries with local caching | ❌ No | Local Cache |
 | `executeCacheable()` | Raw SQL queries with local and global caching | ❌ No | Local + Global Cache |
+| `executeWithMetadata()` | Raw SQL queries with execution metrics capture | ❌ No | Local Cache |
+| `executeDDL()` | DDL operations (CREATE, ALTER, DROP, etc.) | ❌ No | No Caching |
+| `executeDDLActions()` | Execute regular SQL queries in DDL operation context | ❌ No | No Caching |
 | `with()` | Common Table Expressions (CTEs) | ❌ No | Local Cache |
 where Cache context - allows you to batch cache invalidation events and bypass cache reads for affected tables.
 
@@ -1163,6 +1215,51 @@ const users = await forgeSQL
 const users = await forgeSQL
   .executeCacheable("SELECT * FROM users WHERE active = ?", [true], 300);
 
+// Using executeWithMetadata() for capturing execution metrics
+const usersWithMetadata = await forgeSQL
+    .executeWithMetadata(
+        async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+        (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+            console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+            console.log(`Response size: ${totalResponseSize} bytes`);
+            console.log('Forge metadata:', forgeMetadata);
+        }
+    );
+
+// Using executeDDL() for DDL operations (CREATE, ALTER, DROP, etc.)
+await forgeSQL.executeDDL(`
+  CREATE TABLE users (
+    id INT PRIMARY KEY AUTO_INCREMENT,
+    name VARCHAR(255) NOT NULL,
+    email VARCHAR(255) UNIQUE
+  )
+`);
+
+await forgeSQL.executeDDL(sql`
+  ALTER TABLE users 
+  ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+`);
+
+await forgeSQL.executeDDL("DROP TABLE IF EXISTS old_users");
+
+// Using executeDDLActions() for executing regular SQL queries in DDL context
+// This method executes a series of actions within a DDL operation context for monitoring
+await forgeSQL.executeDDLActions(async () => {
+  // Execute regular SQL queries in DDL context for performance monitoring
+  const slowQueries = await forgeSQL.execute(`
+    SELECT * FROM INFORMATION_SCHEMA.STATEMENTS_SUMMARY 
+    WHERE AVG_LATENCY > 1000000
+  `);
+  
+  // Execute complex analysis queries in DDL context
+  const performanceData = await forgeSQL.execute(`
+    SELECT * FROM INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_HISTORY
+    WHERE SUMMARY_END_TIME > DATE_SUB(NOW(), INTERVAL 1 HOUR)
+  `);
+  
+  return { slowQueries, performanceData };
+});
+
 // Using execute() with complex queries
 const userStats = await forgeSQL
   .execute(`
@@ -1478,6 +1575,16 @@ await forgeSQL.executeWithLocalContext(async () => {
     [true]
   );
   
+  // Raw SQL with execution metadata and local caching
+  const usersWithMetadata = await forgeSQL.executeWithMetadata(
+    async () => await forgeSQL.execute("SELECT id, name FROM users WHERE active = ?", [true]),
+    (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+      console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+      console.log(`Response size: ${totalResponseSize} bytes`);
+      console.log('Forge metadata:', forgeMetadata);
+    }
+  );
+  
   // Insert operation - evicts local cache for users table
   await forgeSQL.insert(users).values({ name: 'New User', active: true });
   
@@ -1646,6 +1753,16 @@ const userStats = await forgeSQL
   })
   .from(sql`activeUsers au`)
   .leftJoin(sql`completedOrders co`, eq(sql`au.id`, sql`co.userId`));
+
+// Using executeWithMetadata() for capturing execution metrics with caching
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+  async () => await forgeSQL.executeCacheable("SELECT * FROM users WHERE active = ?", [true], 300),
+  (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+    console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+    console.log(`Response size: ${totalResponseSize} bytes`);
+    console.log('Forge metadata:', forgeMetadata);
+  }
+);
 ```
 
 ### Manual Cache Management
@@ -1721,6 +1838,7 @@ The `ForgeSqlOrmOptions` object allows customization of ORM behavior:
 | Option                     | Type      | Description                                                                                                                                                                                                                     |
 | -------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `logRawSqlQuery`           | `boolean` | Enables logging of raw SQL queries in the Atlassian Forge Developer Console. Useful for debugging and monitoring. Defaults to `false`.                                                                                         |
+| `logCache`                 | `boolean` | Enables logging of cache operations (hits, misses, evictions) in the Atlassian Forge Developer Console. Useful for debugging caching issues. Defaults to `false`.                                                               |
 | `disableOptimisticLocking` | `boolean` | Disables optimistic locking. When set to `true`, no additional condition (e.g., a version check) is added during record updates, which can improve performance. However, this may lead to conflicts when multiple transactions attempt to update the same record concurrently. |
 | `additionalMetadata`       | `object`  | Allows adding custom metadata to all entities. This is useful for tracking common fields across all tables (e.g., `createdAt`, `updatedAt`, `createdBy`, etc.). The metadata will be automatically added to all generated entities. |
 | `cacheEntityName`          | `string`  | KVS Custom entity name for cache storage. Must match the `name` in your `manifest.yml` storage entities configuration. Required for caching functionality. Defaults to `"cache"`.                                                                                                                         |
@@ -2020,13 +2138,13 @@ This analysis provides insights into:
 - Resource usage at each step
 - Performance optimization opportunities
 
-## Memory Usage Monitoring
+## Performance Monitoring
 
 [↑ Back to Top](#table-of-contents)
 
-Forge-SQL-ORM provides automated memory usage monitoring capabilities to help you identify and track memory-intensive queries in your Forge SQL instance. This feature is **essential for Atlassian Forge applications** as it helps you stay within the **16 MiB per query memory limit** and provides detailed insights for optimization.
+Forge-SQL-ORM provides automated performance monitoring capabilities to help you identify and track memory-intensive queries in your Forge SQL instance. This feature is **essential for Atlassian Forge applications** as it helps you stay within the **16 MiB per query memory limit** and provides detailed insights for optimization.
 
-### Why Memory Monitoring is Critical
+### Why Performance Monitoring is Critical
 
 Atlassian Forge SQL has a strict **16 MiB memory limit per query**. Unlike slow query detection (which is available in the Forge Developer Console), there's **no built-in way to monitor memory usage** of your queries. This monitoring system fills that gap by:
 
@@ -2038,7 +2156,7 @@ Atlassian Forge SQL has a strict **16 MiB memory limit per query**. Unlike slow
 
 ### Overview
 
-The memory monitoring system:
+The performance monitoring system:
 - **Automatically detects memory-intensive queries** based on configurable memory thresholds
 - **Provides detailed memory metrics** including execution time, memory usage, and execution plans
 - **Logs memory issues** to the Forge Developer Console for easy debugging
@@ -2047,7 +2165,7 @@ The memory monitoring system:
 
 ### Key Features
 
-- **Memory-Focused Monitoring**: Primary focus on memory usage with configurable thresholds
+- **Memory-Focused Performance Monitoring**: Primary focus on memory usage with configurable thresholds
 - **Atlassian 16 MiB Limit Awareness**: Designed specifically for Forge SQL's memory constraints
 - **Execution Plan Analysis**: Shows detailed query plans to help optimize memory consumption
 - **Configurable Thresholds**: Set custom memory usage thresholds (default: 4MB warning)
@@ -2071,56 +2189,60 @@ import ForgeSQL, { topSlowestStatementLastHourTrigger } from 'forge-sql-orm';
 const forgeSQL = new ForgeSQL();
 
 // Basic usage with default thresholds (300ms latency, 8MB memory warning)
-export const memoryUsageTrigger = () =>
+export const performanceTrigger = () =>
   topSlowestStatementLastHourTrigger(forgeSQL);
 
-// Conservative memory monitoring: 4MB warning (well below 16MB limit)
+// Conservative performance monitoring: 4MB warning (well below 16MB limit)
 export const conservativeMemoryTrigger = () =>
-  topSlowestStatementLastHourTrigger(forgeSQL, 300, 4 * 1024 * 1024);
+  topSlowestStatementLastHourTrigger(forgeSQL, { memoryThresholdBytes: 4 * 1024 * 1024 });
 
-// Aggressive memory monitoring: 12MB warning (75% of 16MB limit)
+// Aggressive performance monitoring: 12MB warning (75% of 16MB limit)
 export const aggressiveMemoryTrigger = () =>
-  topSlowestStatementLastHourTrigger(forgeSQL, 300, 12 * 1024 * 1024);
+  topSlowestStatementLastHourTrigger(forgeSQL, { memoryThresholdBytes: 12 * 1024 * 1024 });
 
-// Memory-only monitoring: Only trigger on memory usage (latency effectively disabled)
+// Memory-only performance monitoring: Only trigger on memory usage (latency effectively disabled)
 export const memoryOnlyTrigger = () =>
-  topSlowestStatementLastHourTrigger(forgeSQL, 10000, 4 * 1024 * 1024);
+  topSlowestStatementLastHourTrigger(forgeSQL, { warnThresholdMs: 10000, memoryThresholdBytes: 4 * 1024 * 1024 });
 
 // Latency-only monitoring: Only trigger on slow queries (memory effectively disabled)
 export const latencyOnlyTrigger = () =>
-  topSlowestStatementLastHourTrigger(forgeSQL, 500, 16 * 1024 * 1024);
+  topSlowestStatementLastHourTrigger(forgeSQL, { warnThresholdMs: 500, memoryThresholdBytes: 16 * 1024 * 1024 });
 
+// With execution plan in logs
+export const performanceWithPlanTrigger = () =>
+  topSlowestStatementLastHourTrigger(forgeSQL, { showPlan: true });
+```
 
 #### 3. Configure in manifest.yml
 
 **As Scheduler Trigger (Recommended for Production):**
 ```yaml
 scheduledTrigger:
-  - key: memory-usage-trigger
-    function: memoryUsageTrigger
+  - key: performance-trigger
+    function: perfTrigger
     interval: hour  # Required: only hour interval is supported
 
 function:
-  - key: memoryUsageTrigger
-    handler: index.memoryUsageTrigger
+  - key: perfTrigger
+    handler: index.performanceTrigger
 ```
 
 **As Web Trigger (Development Only):**
 ```yaml
 webtrigger:
   - key: print-slowest-queries
-    function: memoryUsageTrigger
+    function: perfTrigger
 
 function:
-  - key: memoryUsageTrigger
-    handler: index.memoryUsageTrigger
+  - key: perfTrigger
+    handler: index.performanceTrigger
 ```
 
 > **⚠️ Important**: Web triggers are not recommended for production as they violate the "run-on-atlassian" principle. Use scheduler triggers for production monitoring.
 
 ### How It Works
 
-The memory monitoring trigger works differently depending on how it's configured:
+The performance monitoring trigger works differently depending on how it's configured:
 
 #### Scheduler Trigger Mode (Production)
 
@@ -2196,7 +2318,8 @@ When used as a **web trigger**, the system:
 |-----------|------|---------|-------------|
 | `warnThresholdMs` | `number` | `300` | Latency threshold in milliseconds (secondary) |
 | `memoryThresholdBytes` | `number` | `8 * 1024 * 1024` | **Memory usage threshold in bytes (primary focus)** |
-| `options` | `ForgeSqlOrmOptions` | `undefined` | Optional ORM configuration |
+| `showPlan` | `boolean` | `false` | Whether to include execution plan in logs |
+| `logCache` | `boolean` | `false` | Whether to log cache operations |
 
 **⚠️ Important: OR Logic**
 The monitoring uses **OR logic** - if **either** threshold is exceeded, the query will be logged/returned:
@@ -2205,9 +2328,11 @@ The monitoring uses **OR logic** - if **either** threshold is exceeded, the quer
 - No need to exceed both thresholds simultaneously
 
 **💡 Pro Tips:**
-- **Memory-only monitoring**: Set `warnThresholdMs` to a very high value (e.g., 10000ms) to trigger only on memory usage
+- **Memory-only performance monitoring**: Set `warnThresholdMs` to a very high value (e.g., 10000ms) to trigger only on memory usage
 - **Latency-only monitoring**: Set `memoryThresholdBytes` to 16MB (16 * 1024 * 1024) to trigger only on latency
 - **Combined monitoring**: Use both thresholds for comprehensive monitoring
+- **Execution plan analysis**: Set `showPlan: true` to include detailed execution plans in logs (useful for debugging)
+- **Cache debugging**: Set `logCache: true` to log cache operations and debug caching issues
 
 **Memory Threshold Guidelines:**
 - **Conservative**: 4MB (25% of 16MB limit)
@@ -2276,6 +2401,8 @@ This section covers the breaking changes introduced in version 2.1.x and how to
 - `forgeSQL.selectDistinctCacheableFrom()` - Distinct all-column queries with field aliasing and caching
 - `forgeSQL.execute()` - Raw SQL queries with local caching
 - `forgeSQL.executeCacheable()` - Raw SQL queries with local and global caching
+- `forgeSQL.executeDDL()` - DDL operations (CREATE, ALTER, DROP, etc.)
+- `forgeSQL.executeDDLActions()` - Execute actions within DDL operation context
 - `forgeSQL.with()` - Common Table Expressions (CTEs)
 
 **Optional Migration:**
@@ -2312,6 +2439,47 @@ const cachedRawUsers = await forgeSQL.executeCacheable(
   300
 );
 
+// ✅ Raw SQL execution with metadata capture
+const usersWithMetadata = await forgeSQL.executeWithMetadata(
+    async () => await forgeSQL.execute("SELECT * FROM users WHERE active = ?", [true]),
+    (totalDbExecutionTime, totalResponseSize, forgeMetadata) => {
+        console.log(`DB execution time: ${totalDbExecutionTime}ms`);
+        console.log(`Response size: ${totalResponseSize} bytes`);
+        console.log('Forge metadata:', forgeMetadata);
+    }
+);
+
+// ✅ DDL operations for schema modifications
+await forgeSQL.executeDDL(`
+  CREATE TABLE users (
+    id INT PRIMARY KEY AUTO_INCREMENT,
+    name VARCHAR(255) NOT NULL,
+    email VARCHAR(255) UNIQUE
+  )
+`);
+
+await forgeSQL.executeDDL(sql`
+  ALTER TABLE users 
+  ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+`);
+
+// ✅ Execute regular SQL queries in DDL context for performance monitoring
+await forgeSQL.executeDDLActions(async () => {
+  // Execute regular SQL queries in DDL context for monitoring
+  const slowQueries = await forgeSQL.execute(`
+    SELECT * FROM INFORMATION_SCHEMA.STATEMENTS_SUMMARY 
+    WHERE AVG_LATENCY > 1000000
+  `);
+  
+  // Execute complex analysis queries in DDL context
+  const performanceData = await forgeSQL.execute(`
+    SELECT * FROM INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_HISTORY
+    WHERE SUMMARY_END_TIME > DATE_SUB(NOW(), INTERVAL 1 HOUR)
+  `);
+  
+  return { slowQueries, performanceData };
+});
+
 // ✅ Common Table Expressions (CTEs)
 const userStats = await forgeSQL
   .with(