@jackchuka/gql-ingest 1.2.0 → 1.3.0

package/README.md

@@ -12,6 +12,7 @@ A TypeScript CLI tool that reads CSV files and ingests data into GraphQL APIs th
 - ✅ Configurable GraphQL endpoint and headers
 - ✅ **Parallel processing** with dependency management
 - ✅ Entity-level and row-level concurrency control
+- ✅ **Retry capabilities** with exponential backoff and configurable error handling
 - ✅ Comprehensive metrics and progress tracking
 
 ## Installation
@@ -78,7 +79,7 @@ GQL Ingest supports advanced parallel processing with dependency management for
 ### Key Capabilities
 
 - **Entity-level parallelism**: Process multiple entities (users, products, orders) concurrently
-- **Row-level parallelism**: Process multiple CSV rows within an entity concurrently  
+- **Row-level parallelism**: Process multiple CSV rows within an entity concurrently
 - **Dependency management**: Ensure entities process in the correct order (e.g., users before orders)
 - **Smart batching**: Control exactly how many entities/rows process simultaneously
 - **Real-time metrics**: Track progress, success rates, and performance
@@ -88,20 +89,60 @@ GQL Ingest supports advanced parallel processing with dependency management for
 ```yaml
 # config.yaml - Add to your configuration directory
 parallelProcessing:
-  concurrency: 10          # Process up to 10 CSV rows per entity concurrently
-  entityConcurrency: 3     # Process up to 3 entities simultaneously
-  preserveRowOrder: false  # Allow rows to complete out of order for speed
+  concurrency: 10 # Process up to 10 CSV rows per entity concurrently
+  entityConcurrency: 3 # Process up to 3 entities simultaneously
+  preserveRowOrder: false # Allow rows to complete out of order for speed
 
-# Define dependencies between entities  
+# Define dependencies between entities
 entityDependencies:
-  products: ["users"]       # Products must wait for users to complete
-  orders: ["products"]      # Orders must wait for products to complete
+  products: ["users"] # Products must wait for users to complete
+  orders: ["products"] # Orders must wait for products to complete
 ```
 
 **Performance Impact**: This configuration can process data **10-50x faster** than sequential processing, depending on your GraphQL API's capabilities.
 
 👉 **[Full Parallel Processing Guide](PARALLEL_PROCESSING.md)** - Detailed configuration options, performance tuning, and examples.
 
+## Retry Capabilities 🔄
+
+GQL Ingest includes robust retry functionality to handle transient failures and improve reliability:
+
+### Key Features
+
+- **Automatic retries**: Failed GraphQL mutations are retried automatically
+- **Exponential backoff**: Intelligent delay increases between retry attempts
+- **Jitter**: Randomization prevents thundering herd problems
+- **Configurable error codes**: Control which HTTP status codes trigger retries
+- **Per-entity overrides**: Different retry settings for different entities
+- **Metrics tracking**: Monitor retry success rates and attempt counts
+
+### Quick Example
+
+```yaml
+# config.yaml - Add to your configuration directory
+retry:
+  maxAttempts: 5 # Retry up to 5 times (default: 3)
+  baseDelay: 2000 # Start with 2s delay (default: 1000ms)
+  maxDelay: 60000 # Cap delays at 60s (default: 30000ms)
+  exponentialBackoff: true # Double delay each retry (default: true)
+  retryableStatusCodes: # Which HTTP errors to retry (defaults shown)
+    - 408 # Request Timeout
+    - 429 # Too Many Requests
+    - 500 # Internal Server Error
+    - 502 # Bad Gateway
+    - 503 # Service Unavailable
+    - 504 # Gateway Timeout
+
+# Per-entity retry overrides
+entityConfig:
+  critical-orders:
+    retry:
+      maxAttempts: 10 # More retries for critical data
+      baseDelay: 500 # Faster initial retry
+```
+
+**Reliability Impact**: Retry capabilities can improve success rates from 95% to 99.9%+ for APIs with transient failures.
+
 ## Configuration
 
 The `--config` flag points to a configuration directory containing three subdirectories:
@@ -109,7 +150,7 @@ The `--config` flag points to a configuration directory containing three subdire
 - `data/` - CSV files with actual data
 - `graphql/` - GraphQL mutation definitions
 - `mappings/` - JSON files that map CSV columns to GraphQL variables
-- `config.yaml` - *(Optional)* Parallel processing and dependency configuration
+- `config.yaml` - _(Optional)_ Parallel processing and dependency configuration
 
 Each entity has three corresponding files across these directories with matching names.
 
@@ -148,16 +189,32 @@ mutation CreateItem($name: String!, $sku: String!) {
 }
 ```
 
-**examples/demo/config.yaml** *(Optional - for parallel processing)*:
+**examples/demo/config.yaml** _(Optional - for parallel processing and retry configuration)_:
 
 ```yaml
+# Parallel processing configuration
 parallelProcessing:
-  concurrency: 5           # Process 5 rows per entity concurrently
-  entityConcurrency: 2     # Process 2 entities simultaneously
-  preserveRowOrder: false  # Allow faster out-of-order completion
+  concurrency: 5 # Process 5 rows per entity concurrently
+  entityConcurrency: 2 # Process 2 entities simultaneously
+  preserveRowOrder: false # Allow faster out-of-order completion
 
+# Global retry configuration
+retry:
+  maxAttempts: 3 # Retry failed requests up to 3 times
+  baseDelay: 1000 # Start with 1s delay between retries
+  exponentialBackoff: true # Double delay each retry
+
+# Entity dependencies
 entityDependencies:
-  items: ["users"]         # Items depend on users being processed first
+  items: ["users"] # Items depend on users being processed first
+
+# Per-entity overrides (optional)
+entityConfig:
+  users:
+    retry:
+      maxAttempts: 5 # More retries for user creation
+  items:
+    concurrency: 10 # Higher concurrency for items
 ```
 
 ## Development
@@ -165,21 +222,11 @@ entityDependencies:
 ### Scripts
 
 ```bash
-npm run build        # Build CLI bundle with esbuild
-npm run build:types  # Generate TypeScript declarations
-npm run build:all    # Build bundle + types
-npm run dev          # Run in development mode
-npm run test         # Run test suite
-npm run test:watch   # Run tests in watch mode
-npm run test:coverage # Run tests with coverage report
-```
-
-### Testing
-
-The project includes comprehensive unit tests for all modules:
-
-```bash
-npm test              # Run all tests
+npm run build       # Build CLI bundle with esbuild
+npm run build:types # Generate TypeScript declarations
+npm run build:all   # Build bundle + types
+npm run dev         # Run in development mode
+npm run test        # Run test suite
 ```
 
 ## How It Works
@@ -194,9 +241,13 @@ npm test              # Run all tests
    - Loads the GraphQL mutation definition
    - Maps CSV columns to GraphQL variables using the mapping configuration
    - Executes the mutation against the GraphQL endpoint
-5. **Error Handling & Metrics**: 
-   - Failed mutations are logged but don't stop processing
-   - Real-time progress tracking and success/failure metrics
+5. **Error Handling & Retries**:
+   - Failed mutations are automatically retried with exponential backoff
+   - Non-retryable errors (e.g., validation failures) are logged and skipped
+   - Configurable retry policies per entity type
+6. **Metrics & Monitoring**:
+   - Real-time progress tracking and success/failure rates
+   - Retry attempt counts and success rates
    - Detailed per-entity performance breakdown
 
 ## License