@flisk/analyze-tracking 0.7.1 → 0.7.2

package/README.md

@@ -1,8 +1,8 @@
 # @flisk/analyze-tracking
 
-Automatically document your analytics setup by analyzing tracking code and generating data schemas from tools like Segment, Amplitude, Mixpanel, and more 🚀.
+Automatically document your analytics setup by analyzing tracking code and generating data schemas from tools like Segment, Amplitude, Mixpanel, and more 🚀
 
-[![NPM version](https://img.shields.io/npm/v/@flisk/analyze-tracking.svg)](https://www.npmjs.com/package/@flisk/analyze-tracking)
+[![NPM version](https://img.shields.io/npm/v/@flisk/analyze-tracking.svg)](https://www.npmjs.com/package/@flisk/analyze-tracking) [![Tests](https://github.com/fliskdata/analyze-tracking/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/fliskdata/analyze-tracking/actions/workflows/tests.yml)
 
 
 ## Why Use @flisk/analyze-tracking?
@@ -30,7 +30,7 @@ npx @flisk/analyze-tracking /path/to/project [options]
 - `-o, --output <output_file>`: Name of the output file (default: `tracking-schema.yaml`)
 - `-c, --customFunction <function_name>`: Specify a custom tracking function
 
-🔑&nbsp; **Important:** If you are using `generateDescription`, you must set the appropriate credentials for the provider you are using as an environment variable. OpenAI uses `OPENAI_API_KEY` and Google Vertex AI uses `GOOGLE_APPLICATION_CREDENTIALS`.
+🔑&nbsp; **Important:** If you are using `generateDescription`, you must set the appropriate credentials for the LLM provider you are using as an environment variable. OpenAI uses `OPENAI_API_KEY` and Google Vertex AI uses `GOOGLE_APPLICATION_CREDENTIALS`.
 
 <details>
   <summary>Note on Custom Functions 💡</summary>
@@ -96,10 +96,10 @@ See [schema.json](schema.json) for a JSON Schema of the output.
 | Mixpanel          | ✅ | ✅ | ✅ | ✅ |
 | Amplitude         | ✅ | ✅ | ❌ | ✅ |
 | Rudderstack       | ✅ | ✅ | ✳️ | ✳️ |
-| mParticle         | ✅ | ✅ | ❌ | ❌ |
+| mParticle         | ✅ | ❌ | ❌ | ❌ |
 | PostHog           | ✅ | ✅ | ✅ | ✅ |
-| Pendo             | ✅ | ✅ | ❌ | ❌ |
-| Heap              | ✅ | ✅ | ❌ | ❌ |
+| Pendo             | ✅ | ❌ | ❌ | ❌ |
+| Heap              | ✅ | ❌ | ❌ | ❌ |
 | Snowplow          | ✅ | ✅ | ✅ | ✅ |
 | Custom Function   | ✅ | ✅ | ✅ | ✅ |
 
@@ -198,16 +198,22 @@ See [schema.json](schema.json) for a JSON Schema of the output.
 
   **JavaScript/TypeScript**
   ```js
-  amplitude.logEvent('<event_name>', {
+  amplitude.track('<event_name>', {
     <event_parameters>
   });
   ```
 
   **Python**
   ```python
-  amplitude.track('<event_name>', {
-    '<property_name>': '<property_value>'
-  })
+  client.track(
+    BaseEvent(
+      event_type="<event_name>",
+      user_id="<user_id>",
+      event_properties={
+        "<property_name>": "<property_value>",
+      },
+    )
+  )
   ```
 
   **Go**
@@ -266,19 +272,10 @@ See [schema.json](schema.json) for a JSON Schema of the output.
 
   **JavaScript/TypeScript**
   ```js
-  mParticle.logEvent('<event_name>', {
+  mParticle.logEvent('<event_name>', mParticle.EventType.<event_type>, {
     '<property_name>': '<property_value>'
   });
   ```
-
-  **Python**
-  ```python
-  mParticle.logEvent('<event_name>', {
-    '<property_name>': '<property_value>'
-  })
-  ```
-
-
 </details>
 
 <details>
@@ -293,13 +290,9 @@ See [schema.json](schema.json) for a JSON Schema of the output.
 
   **Python**
   ```python
-  posthog.capture(
-    'distinct_id',
-    '<event_name>',
-    {
-      '<property_name>': '<property_value>'
-    }
-  )
+  posthog.capture('distinct_id', '<event_name>', {
+    '<property_name>': '<property_value>'
+  })
   # Or
   posthog.capture(
     'distinct_id',
@@ -377,61 +370,42 @@ See [schema.json](schema.json) for a JSON Schema of the output.
 
   **JavaScript/TypeScript**
   ```js
-  snowplow('trackStructEvent', {
+  tracker.track(buildStructEvent({
+    action: '<event_name>',
     category: '<category>',
-    action: '<action>',
     label: '<label>',
     property: '<property>',
-    value: '<value>'
-  });
+    value: <value>
+  }));
   ```
 
   **Python**
   ```python
-  # Direct tracking
-  trackStructEvent({
-    'category': '<category>',
-    'action': '<action>',
-    'label': '<label>',
-    'property': '<property>',
-    'value': '<value>'
-  })
-  
-  # Builder pattern
-  buildStructEvent({
-    'category': '<category>',
-    'action': '<action>',
-    'label': '<label>',
-    'property': '<property>',
-    'value': '<value>'
-  })
-  
-  # Function call pattern
-  snowplow('trackStructEvent', {
-    'category': '<category>',
-    'action': '<action>',
-    'label': '<label>',
-    'property': '<property>',
-    'value': '<value>'
-  })
+  tracker.track(StructuredEvent(
+    action="<event_name>",
+    category="<category>",
+    label="<label>",
+    property_="<property>",
+    value=<value>,
+  ))
   ```
 
   **Ruby**
   ```ruby
   tracker.track_struct_event(
+    action: '<event_name>',
     category: '<category>',
-    action: '<action>',
     label: '<label>',
     property: '<property>',
-    value: '<value>'
+    value: <value>
   )
   ```
 
   **Go**
   ```go
   tracker.TrackStructEvent(sp.StructuredEvent{
+		Action:   sp.NewString("<event_name>"),
 		Category: sp.NewString("<category>"),
-		Action:   sp.NewString("<action>"),
 		Label:    sp.NewString("<label>"),
 		Property: sp.NewString("<property>"),
 		Value:    sp.NewFloat64(<value>),
@@ -441,4 +415,4 @@ See [schema.json](schema.json) for a JSON Schema of the output.
 
 
 ## Contribute
-We're actively improving this package. Found a bug? Want to request a feature? Open an issue or contribute directly!
+We're actively improving this package. Found a bug? Have a feature request? Open an issue or submit a pull request!

package/package.json

@@ -1,14 +1,26 @@
 {
   "name": "@flisk/analyze-tracking",
-  "version": "0.7.1",
+  "version": "0.7.2",
   "description": "Analyzes tracking code in a project and generates data schemas",
   "main": "src/index.js",
   "bin": {
     "analyze-tracking": "bin/cli.js"
   },
   "scripts": {
-    "test": "jest"
+    "test": "node tests",
+    "test:js": "node --test tests/analyzeJsFile.test.js",
+    "test:ts": "node --test tests/analyzeTsFile.test.js",
+    "test:python": "node --experimental-vm-modules --test tests/analyzePythonFile.test.js",
+    "test:ruby": "node --experimental-vm-modules --test tests/analyzeRubyFile.test.js",
+    "test:go": "node --test tests/analyzeGoFile.test.js",
+    "test:cli": "node --test tests/cli.test.js",
+    "test:schema": "node --test tests/schema.test.js"
   },
+  "files": [
+    "bin",
+    "src",
+    "schema.json"
+  ],
   "repository": {
     "type": "git",
     "url": "git+https://github.com/fliskdata/analyze-tracking.git"
@@ -38,6 +50,7 @@
     "zod": "^3.24.4"
   },
   "devDependencies": {
-    "jest": "^29.7.0"
+    "ajv": "^8.17.1",
+    "lodash": "^4.17.21"
   }
 }

package/src/analyze/analyzeJsFile.js

@@ -33,10 +33,20 @@ function analyzeJsFile(filePath, customFunction) {
           if (source === 'googleanalytics' && node.arguments.length >= 3) {
             eventName = node.arguments[1]?.value || null;
             propertiesNode = node.arguments[2];
-          } else if (source === 'snowplow' && node.arguments.length >= 2) {
-            const actionProperty = node.arguments[1].properties.find(prop => prop.key.name === 'action');
-            eventName = actionProperty ? actionProperty.value.value : null;
-            propertiesNode = node.arguments[1];
+          } else if (source === 'snowplow' && node.arguments.length > 0) {
+            // Snowplow pattern: tracker.track(buildStructEvent({...}))
+            const firstArg = node.arguments[0];
+            if (firstArg.type === 'CallExpression' && firstArg.arguments.length > 0) {
+              const structEventArg = firstArg.arguments[0];
+              if (structEventArg.type === 'ObjectExpression') {
+                const actionProperty = structEventArg.properties.find(prop => prop.key.name === 'action');
+                eventName = actionProperty ? actionProperty.value.value : null;
+                propertiesNode = structEventArg;
+              }
+            }
+          } else if (source === 'mparticle' && node.arguments.length >= 3) {
+            eventName = node.arguments[0]?.value || null;
+            propertiesNode = node.arguments[2];
           } else if (node.arguments.length >= 2) {
             eventName = node.arguments[0]?.value || null;
             propertiesNode = node.arguments[1];
@@ -46,7 +56,12 @@ function analyzeJsFile(filePath, customFunction) {
           const functionName = findWrappingFunctionJs(node, ancestors);
 
           if (eventName && propertiesNode && propertiesNode.type === 'ObjectExpression') {
-            const properties = extractJsProperties(propertiesNode);
+            let properties = extractJsProperties(propertiesNode);
+            
+            // For Snowplow, remove 'action' from properties since it's used as the event name
+            if (source === 'snowplow' && properties.action) {
+              delete properties.action;
+            }
 
             events.push({
               eventName,

package/src/analyze/analyzePythonFile.js

@@ -25,6 +25,7 @@ async function analyzePythonFile(filePath, customFunction) {
     py.globals.set('code', code);
     py.globals.set('filepath', filePath);
     py.globals.set('custom_function', customFunction || null);
+    py.globals.set('__name__', null);
     
     // Run the Python analyzer
     py.runPython(analyzerCode);

package/src/analyze/analyzeRubyFile.js

@@ -46,12 +46,16 @@ class TrackingVisitor {
       const objectName = node.receiver.name;
       const methodName = node.name;
 
-      // Segment
-      if (objectName === 'Analytics' && methodName === 'track') return 'segment';
+      // Segment and Rudderstack (both use similar format)
+      // Analytics.track (Segment) or analytics.track (Rudderstack)
+      if ((objectName === 'Analytics' || objectName === 'analytics') && methodName === 'track') {
+        // Try to determine if it's Rudderstack based on context
+        // For now, we'll treat lowercase 'analytics' as Rudderstack
+        return objectName === 'analytics' ? 'rudderstack' : 'segment';
+      }
       
       // Mixpanel (Ruby SDK uses Mixpanel::Tracker instance)
-      if (methodName === 'track' && node.receiver.type === 'CallNode' && 
-          node.receiver.name === 'tracker') return 'mixpanel';
+      if (methodName === 'track' && objectName === 'tracker') return 'mixpanel';
       
       // PostHog
       if (objectName === 'posthog' && methodName === 'capture') return 'posthog';
@@ -67,7 +71,8 @@ class TrackingVisitor {
   }
 
   extractEventName(node, source) {
-    if (source === 'segment') {
+    if (source === 'segment' || source === 'rudderstack') {
+      // Both Segment and Rudderstack use the same format
       const params = node.arguments_.arguments_[0].elements;
       const eventProperty = params.find(param => param?.key?.unescaped?.value === 'event');
       return eventProperty?.value?.unescaped?.value || null;
@@ -111,7 +116,8 @@ class TrackingVisitor {
   async extractProperties(node, source) {
     const { HashNode, ArrayNode } = await import('@ruby/prism');
 
-    if (source === 'segment') {
+    if (source === 'segment' || source === 'rudderstack') {
+      // Both Segment and Rudderstack use the same format
       const params = node.arguments_.arguments_[0].elements;
       const properties = {};
 
@@ -158,10 +164,10 @@ class TrackingVisitor {
       const args = node.arguments_.arguments_;
       const properties = {};
       
-      // Add distinct_id as property
-      if (args && args.length > 0 && args[0]?.unescaped?.value) {
+      // Add distinct_id as property (even if it's a variable)
+      if (args && args.length > 0) {
         properties.distinct_id = {
-          type: 'string'
+          type: await this.getValueType(args[0])
         };
       }
       
@@ -300,7 +306,7 @@ class TrackingVisitor {
   }
 
   async visit(node) {
-    const { CallNode, ProgramNode, StatementsNode, DefNode, IfNode, BlockNode, ArgumentsNode, HashNode, AssocNode, ClassNode } = await import('@ruby/prism');
+    const { CallNode, ProgramNode, StatementsNode, DefNode, IfNode, BlockNode, ArgumentsNode, HashNode, AssocNode, ClassNode, ModuleNode } = await import('@ruby/prism');
     if (!node) return;
 
     this.ancestors.push(node);
@@ -344,6 +350,10 @@ class TrackingVisitor {
       if (node.body) {
         await this.visit(node.body);
       }
+    } else if (node instanceof ModuleNode) {
+      if (node.body) {
+        await this.visit(node.body);
+      }
     } else if (node instanceof DefNode) {
       if (node.body) {
         await this.visit(node.body);

package/src/analyze/analyzeTsFile.js

@@ -1,5 +1,65 @@
 const ts = require('typescript');
-const { detectSourceTs, findWrappingFunctionTs, extractTsProperties } = require('./helpers');
+const { detectSourceTs, findWrappingFunctionTs, extractTsProperties, resolveIdentifierToInitializer, extractInterfaceProperties } = require('./helpers');
+
+function resolveUnresolvedTypes(properties, checker, sourceFile) {
+  const resolved = {};
+  
+  for (const [key, value] of Object.entries(properties)) {
+    if (value && typeof value === 'object') {
+      if (value.__unresolved) {
+        // Try to find and resolve the type
+        const typeName = value.__unresolved;
+        delete value.__unresolved;
+        
+        // This is a simplified approach - in practice, you'd need to find the actual type declaration
+        // For now, we'll keep the object type but remove the unresolved marker
+        resolved[key] = value;
+      } else if (value.type === 'array' && value.items && value.items.__unresolved) {
+        // Handle unresolved array element types
+        const itemTypeName = value.items.__unresolved;
+        delete value.items.__unresolved;
+        resolved[key] = value;
+      } else if (value.type === 'array' && value.items && typeof value.items.type === 'string' && value.items.type.includes(' ')) {
+        // Handle types like "readonly Product" - extract the actual type name
+        const typeString = value.items.type;
+        const actualType = typeString.replace(/^readonly\s+/, '').trim();
+        
+        // If it looks like a custom type, mark it as object
+        if (actualType[0] === actualType[0].toUpperCase() && !actualType.includes('<')) {
+          resolved[key] = {
+            ...value,
+            items: {
+              type: 'object'
+            }
+          };
+        } else {
+          resolved[key] = value;
+        }
+      } else if (value.type === 'object' && value.properties) {
+        // Recursively resolve nested properties
+        resolved[key] = {
+          ...value,
+          properties: resolveUnresolvedTypes(value.properties, checker, sourceFile)
+        };
+      } else if (value.type === 'array' && value.items && value.items.properties) {
+        // Recursively resolve array item properties
+        resolved[key] = {
+          ...value,
+          items: {
+            ...value.items,
+            properties: value.items.properties ? resolveUnresolvedTypes(value.items.properties, checker, sourceFile) : undefined
+          }
+        };
+      } else {
+        resolved[key] = value;
+      }
+    } else {
+      resolved[key] = value;
+    }
+  }
+  
+  return resolved;
+}
 
 function analyzeTsFile(filePath, program, customFunction) {
   let events = [];
@@ -24,10 +84,50 @@ function analyzeTsFile(filePath, program, customFunction) {
           if (source === 'googleanalytics' && node.arguments.length >= 3) {
             eventName = node.arguments[1]?.text || null;
             propertiesNode = node.arguments[2];
-          } else if (source === 'snowplow' && node.arguments.length >= 2) {
-            const actionProperty = node.arguments[1].properties.find(prop => prop.name.escapedText === 'action');
-            eventName = actionProperty ? actionProperty.initializer.text : null;
-            propertiesNode = node.arguments[1];
+          } else if (source === 'snowplow' && node.arguments.length > 0) {
+            // Snowplow pattern: tracker.track(buildStructEvent({...})) or tracker.track(payload)
+            const firstArg = node.arguments[0];
+            
+            // Check if it's a direct buildStructEvent call
+            if (ts.isCallExpression(firstArg) && 
+                ts.isIdentifier(firstArg.expression) && 
+                firstArg.expression.escapedText === 'buildStructEvent' &&
+                firstArg.arguments.length > 0) {
+              const structEventArg = firstArg.arguments[0];
+              if (ts.isObjectLiteralExpression(structEventArg)) {
+                // Find the action property for event name
+                const actionProp = structEventArg.properties.find(
+                  prop => prop.name && prop.name.escapedText === 'action'
+                );
+                if (actionProp && actionProp.initializer && ts.isStringLiteral(actionProp.initializer)) {
+                  eventName = actionProp.initializer.text;
+                  propertiesNode = structEventArg;
+                }
+              }
+            } 
+            // Check if it's a variable reference (e.g., const payload = buildStructEvent({...}))
+            else if (ts.isIdentifier(firstArg)) {
+              const resolvedNode = resolveIdentifierToInitializer(checker, firstArg, sourceFile);
+              if (resolvedNode && ts.isCallExpression(resolvedNode) &&
+                  ts.isIdentifier(resolvedNode.expression) &&
+                  resolvedNode.expression.escapedText === 'buildStructEvent' &&
+                  resolvedNode.arguments.length > 0) {
+                const structEventArg = resolvedNode.arguments[0];
+                if (ts.isObjectLiteralExpression(structEventArg)) {
+                  const actionProp = structEventArg.properties.find(
+                    prop => prop.name && prop.name.escapedText === 'action'
+                  );
+                  if (actionProp && actionProp.initializer && ts.isStringLiteral(actionProp.initializer)) {
+                    eventName = actionProp.initializer.text;
+                    propertiesNode = structEventArg;
+                  }
+                }
+              }
+            }
+          } else if (source === 'mparticle' && node.arguments.length >= 3) {
+            // mParticle: first param is event name, second is event type (ignored), third is properties
+            eventName = node.arguments[0]?.text || null;
+            propertiesNode = node.arguments[2];
           } else if (node.arguments.length >= 2) {
             eventName = node.arguments[0]?.text || null;
             propertiesNode = node.arguments[1];
@@ -36,17 +136,40 @@ function analyzeTsFile(filePath, program, customFunction) {
           const line = sourceFile.getLineAndCharacterOfPosition(node.getStart()).line + 1;
           const functionName = findWrappingFunctionTs(node);
 
-          if (eventName && propertiesNode && ts.isObjectLiteralExpression(propertiesNode)) {
+          if (eventName && propertiesNode) {
             try {
-              const properties = extractTsProperties(checker, propertiesNode);
-              events.push({
-                eventName,
-                source,
-                properties,
-                filePath,
-                line,
-                functionName
-              });
+              let properties = null;
+              
+              // Check if properties is an object literal
+              if (ts.isObjectLiteralExpression(propertiesNode)) {
+                properties = extractTsProperties(checker, propertiesNode);
+              } 
+              // Check if properties is an identifier (variable reference)
+              else if (ts.isIdentifier(propertiesNode)) {
+                const resolvedNode = resolveIdentifierToInitializer(checker, propertiesNode, sourceFile);
+                if (resolvedNode && ts.isObjectLiteralExpression(resolvedNode)) {
+                  properties = extractTsProperties(checker, resolvedNode);
+                }
+              }
+              
+              if (properties) {
+                // For Snowplow, remove 'action' from properties since it's used as the event name
+                if (source === 'snowplow' && properties.action) {
+                  delete properties.action;
+                }
+                
+                // Clean up any unresolved type markers
+                const cleanedProperties = resolveUnresolvedTypes(properties, checker, sourceFile);
+                
+                events.push({
+                  eventName,
+                  source,
+                  properties: cleanedProperties,
+                  filePath,
+                  line,
+                  functionName
+                });
+              }
             } catch (propertyError) {
               console.error(`Error extracting properties in ${filePath} at line ${line}`);
             }