@stackql/provider-utils 0.4.0 → 0.4.2

package/README.md

@@ -1,42 +1,33 @@
 # StackQL Provider Utils
 
-A comprehensive toolkit for transforming OpenAPI specs into StackQL providers. Includes parsing, mapping, validation, testing, and documentation generation utilities.
+![NPM Version](https://img.shields.io/npm/v/%40stackql%2Fprovider-utils)
+![GitHub Downloads (all assets, all releases)](https://img.shields.io/github/downloads/stackql/stackql/total?style=plastic&label=stackql%20downloads)
+
+A comprehensive toolkit for transforming OpenAPI specifications into StackQL providers. This library streamlines the process of parsing, mapping, validating, testing, and generating documentation for StackQL providers.
 
 ## Table of Contents
 
 - [Prerequisites](#prerequisites)
 - [Installation](#installation)
-- [Local Development Setup](#local-development-setup)
-- [Testing with Node.js](#testing-with-nodejs)
-- [Using the Documentation Generator](#using-the-documentation-generator)
-- [API Reference](#api-reference)
+- [Directory Structure](#directory-structure)
+- [Provider Development Workflow](#provider-development-workflow)
+  - [`providerdev.split`](docs/split.md) - Divide a large OpenAPI specification into smaller, service-specific files
+  - [`providerdev.analyze`](docs/analyze.md) - Examine split API specifications to generate mapping recommendations
+  - [`providerdev.generate`](docs/generate.md) - Create StackQL provider extensions from specifications and mappings
+  - [`docgen.generateDocs`](docs/docgen.md) - Generate comprehensive documentation for StackQL providers
 - [Contributing](#contributing)
+- [License](#license)
+- [Support](#support)
 
 ## Prerequisites
 
-### For Node.js
 - Node.js >= 20
-- npm or yarn
-- StackQL server (for documentation generation)
-
-### Installing StackQL
-
-Download and install StackQL from [stackql.io/downloads](https://stackql.io/downloads)
-
-```bash
-# macOS
-brew install stackql
-
-# Linux
-curl -L https://bit.ly/stackql-zip -O && unzip stackql-zip
-
-# Windows
-# Download from https://stackql.io/downloads
-```
+- `npm` or `yarn`
+- [`stackql`](https://stackql.io/docs/installing-stackql) for testing 
 
 ## Installation
 
-### For Node.js Projects
+Add `@stackql/provider-utils` to your `package.json`:
 
 ```bash
 npm install @stackql/provider-utils
@@ -44,229 +35,55 @@ npm install @stackql/provider-utils
 yarn add @stackql/provider-utils
 ```
 
-## Local Development Setup
-
-1. Clone the repository:
-```bash
-git clone https://github.com/stackql/stackql-provider-utils.git
-cd stackql-provider-utils
-```
-
-2. Install dependencies (Node.js):
-```bash
-npm install
-```
-
-## Testing with Node.js
-
-### 1. Create a Test Script
-
-Create a file `test-docgen.js`:
-
-```javascript
-import { docgen } from './src/index.js';
-
-// Test the documentation generator
-async function testDocGen() {
-    try {
-        const result = await docgen.generateDocs({
-            providerName: 'myservice',
-            providerDir: './test-data/output/src/myservice/v00.00.00000',
-            outputDir: './test-output',
-            providerDataDir: './test-data/provider-data',
-        });
-        
-        console.log('Documentation generated successfully:', result);
-    } catch (error) {
-        console.error('Error generating documentation:', error);
-    }
-}
-
-testDocGen();
-```
-
-### 2. Set Up Test Data
-
-Create the required directory structure:
-
-```bash
-mkdir -p test-data/output/src/myservice/v00.00.00000/services
-mkdir -p test-data/provider-data
-```
-
-Add test files:
-
-`test-data/provider-data/headerContent1.txt`:
-```
----
-title: myservice
-hide_title: false
-hide_table_of_contents: false
-keywords:
-  - myservice
-  - stackql
-  - infrastructure-as-code
-  - configuration-as-data
-description: Query and manage myservice resources using SQL
----
-
-# myservice Provider
-
-The myservice provider for StackQL allows you to query, deploy, and manage myservice resources using SQL.
-```
-
-`test-data/provider-data/headerContent2.txt`:
-```
-See the [myservice provider documentation](https://myservice.com/docs) for more information.
-```
-
-`test-data/output/src/myservice/v00.00.00000/services/example.yaml`:
-```yaml
-openapi: 3.0.0
-info:
-  title: Example Service
-  version: 1.0.0
-paths:
-  /examples:
-    get:
-      operationId: listExamples
-      responses:
-        '200':
-          description: Success
-components:
-  x-stackQL-resources:
-    examples:
-      id: myservice.example.examples
-      name: examples
-      title: Examples
-      methods:
-        list:
-          operation:
-            $ref: '#/paths/~1examples/get'
-          response:
-            mediaType: application/json
-            openAPIDocKey: '200'
-      sqlVerbs:
-        select:
-          - $ref: '#/components/x-stackQL-resources/examples/methods/list'
-```
+## Directory Structure
 
-### 3. Run the Test
+A typical project structure for the development of a `stackql` provider would be...
 
 ```bash
-node tests/docgen/test-docgen.js snowflake
-node tests/docgen/test-docgen.js google
-node tests/docgen/test-docgen.js homebrew
-```
-
-
-node tests/providerdev/test-split.js okta tests/providerdev/split-source/okta/management-minimal.yaml path
-node tests/providerdev/test-analyze.js okta
-node tests/providerdev/test-generate.js okta
-
-## Using the Documentation Generator
-
-### Basic Example
-
-```javascript
-import { docgen } from '@stackql/provider-utils';
-
-const options = {
-    providerName: 'github',
-    providerDir: './output/src/github/v00.00.00000',
-    outputDir: './docs',
-    providerDataDir: './config/provider-data',
-};
-
-const result = await docgen.generateDocs(options);
-console.log(`Generated docs for ${result.totalServices} services and ${result.totalResources} resources`);
-console.log(`Output location: ${result.outputPath}`);
-```
-
-### Options
-
-| Option | Type | Description | Default |
-|--------|------|-------------|---------|
-| `providerName` | string | Name of the provider (e.g., 'github', 'aws') | Required |
-| `providerDir` | string | Path to provider spec directory | Required |
-| `outputDir` | string | Directory for generated documentation | Required |
-| `providerDataDir` | string | Directory containing provider header files | Required |
-
-## Directory Structure Requirements
-
-### Provider Data Directory
-```
-provider-data/
-├── headerContent1.txt    # Provider introduction
-├── headerContent2.txt    # Additional provider info
-```
-
-### Provider Spec Directory
-```
-output/src/{provider}/v00.00.00000/
-├── provider.yaml
-└── services/
-    ├── service1.yaml
-    ├── service2.yaml
-    └── ...
-```
-
-### Generated Output
-```
-docs/{provider}-docs/
-├── index.md
-└── {service}/
-    ├── index.md
-    └── {resource}/
-        └── index.md
-```
-
-## Troubleshooting
-
-### Missing Provider Data
-- Ensure `headerContent1.txt` and `headerContent2.txt` exist in provider data directory
-- Check file permissions
-
-### Empty Documentation
-- Verify provider specs have `x-stackQL-resources` components
-- Check that resources have proper method definitions
-
-## API Reference
-
-### `docgen.generateDocs(options)`
-
-Generates documentation for a StackQL provider.
-
-**Parameters:**
-- `options` (Object): Configuration options
-
-**Returns:**
-- Promise<Object>: Result object containing:
-  - `totalServices`: Number of services processed
-  - `totalResources`: Number of resources documented
-  - `outputPath`: Path to generated documentation
-
-**Example:**
-```javascript
-const result = await docgen.generateDocs({
-    providerName: 'aws',
-    providerDir: './providers/src/aws/v00.00.00000',
-    outputDir: './documentation',
-    providerDataDir: './config/aws',
-});
-```
-
-### `docgen.createResourceIndexContent(...)`
-
-Creates markdown content for a single resource. This is a lower-level function used internally by `generateDocs`.
+.
+├── bin # convinience scripts
+│   ├── ... 
+├── provider-dev                            
+│   ├── config
+│   │   └── all_services.csv  # mappings generated or updated by the `providerdev.analyze` function, used by `providerdev.generate`
+│   ├── docgen
+│   │   └── provider-data  # provider metadata used by `docgen.generateDocs`
+│   │       ├── headerContent1.txt
+│   │       └── headerContent2.txt
+│   ├── downloaded # used to store the original spec for the provider
+│   │   └── management-minimal.yaml
+│   ├── openapi # output from `providerdev.generate`, this is the stackql provider
+│   │   └── src
+│   │       └── okta
+│   │           └── v00.00.00000
+│   │               ├── provider.yaml
+│   │               └── services
+│   │                   ├── agentpools.yaml
+│   │                   ├── ...
+│   ├── scripts # optional scripts for pre or post processing if required
+│   │   └── post_processing.sh
+│   └── source  # output from `providerdev.split` if used, this is the source used with the mappings to generate the provider
+│       ├── agentpools.yaml
+│       ├── ...
+└── website # docusaurus site
+    ├── docs # output from `docgen.generateDocs`
+    │   ├── ...
+```
+
+> see [__stackql-provider-okta__](https://github.com/stackql/stackql-provider-okta) for a working example.
+
+## Provider Development Workflow
+
+The library provides a streamlined workflow for creating StackQL providers from OpenAPI specifications:
+
+1. [`providerdev.split`](docs/split.md) - Divide a large OpenAPI specification into smaller, service-specific files
+2. [`providerdev.analyze`](docs/analyze.md) - Examine split API specifications to generate mapping recommendations
+3. [`providerdev.generate`](docs/generate.md) - Create StackQL provider extensions from specifications and mappings
+4. [`docgen.generateDocs`](docs/docgen.md) - Generate comprehensive documentation for StackQL providers
 
 ## Contributing
 
-1. Fork the repository
-2. Create your feature branch (`git checkout -b feature/amazing-feature`)
-3. Commit your changes (`git commit -m 'Add some amazing feature'`)
-4. Push to the branch (`git push origin feature/amazing-feature`)
-5. Open a Pull Request
+Contributions are welcome!
 
 ## License
 
@@ -276,4 +93,5 @@ MIT
 
 - [StackQL Documentation](https://stackql.io/docs)
 - [GitHub Issues](https://github.com/stackql/stackql-provider-utils/issues)
-- [StackQL Discord](https://discord.gg/stackql)
+- [StackQL Community Slack](https://stackqlcommunity.slack.com/join/shared_invite/zt-1cbdq9s5v-CkY65IMAesCgFqjN6FU6hg)
+- [StackQL Discord](https://discord.com/invite/xVXZ9d5NxN)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stackql/provider-utils",
-  "version": "0.4.0",
+  "version": "0.4.2",
   "description": "Utilities for building StackQL providers from OpenAPI specifications.",
   "type": "module",
   "main": "./src/index.js",

package/src/docgen/resource/fields.js

@@ -7,6 +7,13 @@ import { docView } from './view.js';
 
 const mdCodeAnchor = "`";
 
+// List of meaningless descriptions to filter out (all lowercase)
+const meaninglessDescriptions = [
+    'ok',
+    'successful response',
+    'success'
+];
+
 export function createFieldsSection(resourceType, resourceData, dereferencedAPI) {
     let content = '## Fields\n\n';
 
@@ -41,14 +48,14 @@ export function createFieldsSection(resourceType, resourceData, dereferencedAPI)
                 // Start the TabItem
                 content += `<TabItem value="${methodName}">\n\n`;
                 
-                // Add the method description if available
-                if (methodData.respDescription 
-                    && methodData.respDescription.trim().toUpperCase() !== 'OK' 
-                    && methodData.respDescription.trim() !== 'Successful response'
-                ) {
+                // Add the method description if available and not in the meaningless list
+                if (methodData.respDescription && 
+                    !meaninglessDescriptions.includes(methodData.respDescription.trim().toLowerCase()) &&
+                    methodData.respDescription.trim().length > 0) {
                     content += `${sanitizeHtml(methodData.respDescription)}\n\n`;
                 }
                 
+
                 // Add the table header
                 content += `<table>
 <thead>

package/src/docgen/resource/methods.js

@@ -4,6 +4,24 @@ import {
     sanitizeHtml
 } from '../helpers.js';
 
+const getRequiredBodyParams = (methodDetails, accessType) => {
+    // Only process request body for insert, update, replace, and exec
+    if (!['insert', 'update', 'replace', 'exec'].includes(accessType)) {
+        return [];
+    }
+    
+    // Get required body params if they exist
+    const requiredBodyProps = methodDetails.requestBody?.required ? methodDetails.requestBody.required : [];
+    
+    // For insert, update, and replace, prefix with data__
+    if (['insert', 'update', 'replace'].includes(accessType)) {
+        return requiredBodyProps.map(prop => `data__${prop}`);
+    } else {
+        // For exec, don't prefix
+        return requiredBodyProps;
+    }
+};
+
 export function createMethodsSection(resourceData, dereferencedAPI) {
     
     let content = `\n## Methods\n\n`;
@@ -37,10 +55,16 @@ export function createMethodsSection(resourceData, dereferencedAPI) {
         for (const [methodName, methodDetails] of Object.entries(methods)) {
             console.info(`Adding ${accessType} method to table: ${methodName}`);
             
+            // Get required params from both the standard params and the request body
+            const reqParamsArr = Object.keys(methodDetails.requiredParams || {});
+            const reqBodyParamsArr = getRequiredBodyParams(methodDetails, accessType);
+            
+            // Combine both types of required parameters
+            const allReqParamsArr = [...reqParamsArr, ...reqBodyParamsArr];
+            
             // Format required params as comma-delimited list with hyperlinks
-            const requiredParamsArr = Object.keys(methodDetails.requiredParams || {});
-            const requiredParamsStr = requiredParamsArr.length > 0 
-                ? requiredParamsArr.map(param => `<a href="#parameter-${param}"><code>${param}</code></a>`).join(', ') 
+            const requiredParamsStr = allReqParamsArr.length > 0 
+                ? allReqParamsArr.map(param => `<a href="#parameter-${param}"><code>${param}</code></a>`).join(', ') 
                 : '';
             
             // Format optional params as comma-delimited list with hyperlinks
@@ -48,7 +72,7 @@ export function createMethodsSection(resourceData, dereferencedAPI) {
             const optionalParamsStr = optionalParamsArr.length > 0 
                 ? optionalParamsArr.map(param => `<a href="#parameter-${param}"><code>${param}</code></a>`).join(', ') 
                 : '';
-            
+
             // Add the method row to the table
             content += `
 <tr>

package/src/providerdev/analyze.js

@@ -95,6 +95,23 @@ function findExistingMapping(spec, pathRef) {
   };
 }
 
+/**
+ * Escape and sanitize a CSV field value
+ * @param {string} value - Field value to escape
+ * @returns {string} - Escaped value
+ */
+function escapeCsvField(value) {
+  if (!value) return '';
+  
+  // If the value contains commas, double quotes, or newlines, wrap it in quotes
+  // and escape any existing double quotes by doubling them
+  if (value.includes(',') || value.includes('"') || value.includes('\n')) {
+    return `"${value.replace(/"/g, '""')}"`;
+  }
+  
+  return value;
+}
+
 /**
  * Analyze OpenAPI specs and generate mapping CSV
  * @param {Object} options - Options for analysis
@@ -159,7 +176,7 @@ export async function analyze(options) {
 
     // Only write header if creating a new file
     if (!fileExists) {
-      writer.write('filename,path,operationId,formatted_op_id,verb,response_object,tags,formatted_tags,stackql_resource_name,stackql_method_name,stackql_verb\n');
+      writer.write('filename,path,operationId,formatted_op_id,verb,response_object,tags,formatted_tags,stackql_resource_name,stackql_method_name,stackql_verb,op_description\n');
     }
     
     const files = fs.readdirSync(inputDir);
@@ -216,15 +233,27 @@ export async function analyze(options) {
           // Find existing mapping if available
           const { resourceName, methodName, sqlVerb } = findExistingMapping(spec, pathRef);
           
-          // Escape commas in fields
-          const escapedPath = pathKey.includes(',') ? `"${pathKey}"` : pathKey;
-          const escapedOperationId = operationId.includes(',') ? `"${operationId}"` : operationId;
-          const escapedFormattedOpId = formattedOpId.includes(',') ? `"${formattedOpId}"` : formattedOpId;
-          const escapedTags = tagsStr.includes(',') ? `"${tagsStr}"` : tagsStr;
-          const escapedFormattedTags = formattedTags.includes(',') ? `"${formattedTags}"` : formattedTags;
+          // Get operation description
+          const opDescription = operation.summary || operation.description || '';
+          
+          // Escape fields that might contain commas, quotes, or other special characters
+          const escapedFields = {
+            filename: escapeCsvField(filename),
+            path: escapeCsvField(pathKey),
+            operationId: escapeCsvField(operationId),
+            formattedOpId: escapeCsvField(formattedOpId),
+            verb: escapeCsvField(verb),
+            responseRef: escapeCsvField(responseRef),
+            tagsStr: escapeCsvField(tagsStr),
+            formattedTags: escapeCsvField(formattedTags),
+            resourceName: escapeCsvField(resourceName),
+            methodName: escapeCsvField(methodName),
+            sqlVerb: escapeCsvField(sqlVerb),
+            opDescription: escapeCsvField(opDescription)
+          };
           
           // Write row
-          writer.write(`${filename},${escapedPath},${escapedOperationId},${escapedFormattedOpId},${verb},${responseRef},${escapedTags},${escapedFormattedTags},${resourceName},${methodName},${sqlVerb}\n`);
+          writer.write(`${escapedFields.filename},${escapedFields.path},${escapedFields.operationId},${escapedFields.formattedOpId},${escapedFields.verb},${escapedFields.responseRef},${escapedFields.tagsStr},${escapedFields.formattedTags},${escapedFields.resourceName},${escapedFields.methodName},${escapedFields.sqlVerb},${escapedFields.opDescription}\n`);
         }
       }
     }

package/src/providerdev/split.js

@@ -11,7 +11,6 @@ import {
 
 // Constants
 const OPERATIONS = ["get", "post", "put", "delete", "patch", "options", "head", "trace"];
-const NON_OPERATIONS = ["parameters", "servers", "summary", "description"];
 const COMPONENTS_CHILDREN = ["schemas", "responses", "parameters", "examples", "requestBodies", "headers", "securitySchemes", "links", "callbacks"];
 
 /**
@@ -42,9 +41,10 @@ function isOperationExcluded(exclude, opItem) {
  * @param {string} svcDiscriminator - Service discriminator
  * @param {Object[]} allTags - All tags from API doc
  * @param {boolean} debug - Debug flag
+ * @param {Object} svcNameOverrides - Service name overrides
  * @returns {[string, string]} - [service name, service description]
  */
-function retServiceNameAndDesc(providerName, opItem, pathKey, svcDiscriminator, allTags, debug) {
+function retServiceNameAndDesc(providerName, opItem, pathKey, svcDiscriminator, allTags, debug, svcNameOverrides) {
   let service = "default";
   let serviceDesc = `${providerName} API`;
   
@@ -84,9 +84,25 @@ function retServiceNameAndDesc(providerName, opItem, pathKey, svcDiscriminator,
     return ["skip", ""];
   }
   
+  // Apply service name overrides if present
+  if (svcNameOverrides && svcNameOverrides[service]) {
+    const newName = svcNameOverrides[service];
+    if (debug) {
+      logger.debug(`Overriding service name: ${service} -> ${newName}`);
+    }
+    
+    // Update service description for path-based services
+    if (svcDiscriminator === "path") {
+      serviceDesc = `${providerName} ${newName} API`;
+    }
+    
+    service = newName;
+  }
+  
   return [service, serviceDesc];
 }
 
+
 /**
  * Initialize service map
  * @param {Object} services - Services map
@@ -192,43 +208,6 @@ function getPathLevelRefs(pathItem) {
   return refs;
 }
 
-/**
- * Add referenced components to service
- * @param {Set<string>} refs - Set of refs
- * @param {Object} service - Service object
- * @param {Object} components - Components from API doc
- * @param {boolean} debug - Debug flag
- */
-function addRefsToComponents(refs, service, components, debug) {
-  for (const ref of refs) {
-    const parts = ref.split('/');
-    
-    // Only process refs that point to components
-    if (parts.length >= 4 && parts[1] === "components") {
-      const componentType = parts[2];
-      const componentName = parts[3];
-      
-      // Check if component type exists in service
-      if (!service.components[componentType]) {
-        service.components[componentType] = {};
-      }
-      
-      // Skip if component already added
-      if (service.components[componentType][componentName]) {
-        continue;
-      }
-      
-      // Add component if it exists in source document
-      if (components[componentType] && components[componentType][componentName]) {
-        service.components[componentType][componentName] = components[componentType][componentName];
-        if (debug) {
-          logger.debug(`Added component ${componentType}/${componentName}`);
-        }
-      }
-    }
-  }
-}
-
 /**
  * Add missing type: object to schema objects
  * @param {any} obj - Object to process
@@ -338,7 +317,8 @@ export async function split(options) {
     svcDiscriminator = "tag",
     exclude = null,
     overwrite = true,
-    verbose = false
+    verbose = false,
+    svcNameOverrides = {}  // Add this new parameter with default empty object
   } = options;
   
   // Setup logging based on verbosity
@@ -351,6 +331,13 @@ export async function split(options) {
   logger.info(`Output: ${outputDir}`);
   logger.info(`Service Discriminator: ${svcDiscriminator}`);
   
+  if (Object.keys(svcNameOverrides).length > 0) {
+    logger.info(`Using ${Object.keys(svcNameOverrides).length} service name overrides`);
+    if (verbose) {
+      logger.debug(`Service name overrides: ${JSON.stringify(svcNameOverrides, null, 2)}`);
+    }
+  }
+  
   // Process exclude list
   const excludeList = exclude ? exclude.split(",") : [];
   
@@ -409,12 +396,11 @@ export async function split(options) {
         continue;
       }
       
-      // Determine service name
       const [service, serviceDesc] = retServiceNameAndDesc(
         providerName, opItem, pathKey, svcDiscriminator, 
-        apiDocObj.tags || [], verbose
-      );
-      
+        apiDocObj.tags || [], verbose, svcNameOverrides
+      );      
+
       // Skip if service is marked to skip
       if (service === 'skip') {
         logger.warn(`⭐️ Skipping service: ${service}`);