@jackchuka/gql-ingest 1.4.0 → 2.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 jackchuka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -3,12 +3,14 @@
 [![npm version](https://badge.fury.io/js/%40jackchuka%2Fgql-ingest.svg)](https://badge.fury.io/js/%40jackchuka%2Fgql-ingest)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-A TypeScript CLI tool that reads CSV files and ingests data into GraphQL APIs through configurable mutations.
+A TypeScript CLI tool that reads data from multiple formats (CSV, JSON, YAML, JSONL) and ingests it into GraphQL APIs through configurable mutations.
 
 ## Features
 
+- ✅ **Supported data formats**: CSV, JSON, YAML, JSONL
+- ✅ **Complex nested data support** for sophisticated GraphQL mutations
 - ✅ External GraphQL mutation definitions (separate .graphql files)
-- ✅ CSV-to-GraphQL variable mapping via JSON configuration
+- ✅ Flexible data-to-GraphQL variable mapping via JSON configuration
 - ✅ Configurable GraphQL endpoint and headers
 - ✅ **Parallel processing** with dependency management
 - ✅ Entity-level and row-level concurrency control
@@ -47,7 +49,10 @@ Options:
   -V, --version            output the version number
   -e, --endpoint <url>     GraphQL endpoint URL (required)
   -c, --config <path>      Path to configuration directory (required)
+  -n, --entities <list>    Comma-separated list of specific entities to process
   -h, --headers <headers>  JSON string of headers to include in requests
+  -f, --format <format>    Override data format detection (csv, json, yaml, jsonl)
+  -v, --verbose            Show detailed request results and responses
   --help                   display help for command
 ```
 
@@ -70,6 +75,18 @@ npx @jackchuka/gql-ingest \
   --endpoint https://api.example.com/graphql \
   --config ./my-config \
   --headers '{"X-API-Key": "your-api-key", "Content-Type": "application/json"}'
+
+# Process specific entities only
+npx @jackchuka/gql-ingest \
+  --endpoint https://your-graphql-api.com/graphql \
+  --config ./examples/demo \
+  --entities users,products
+
+# Process a single entity
+npx @jackchuka/gql-ingest \
+  --endpoint https://your-graphql-api.com/graphql \
+  --config ./examples/demo \
+  --entities items
 ```
 
 ## Parallel Processing 🚀
@@ -143,12 +160,21 @@ entityConfig:
 
 **Reliability Impact**: Retry capabilities can improve success rates from 95% to 99.9%+ for APIs with transient failures.
 
+## Selective Entity Processing
+
+The `--entities` flag allows you to process specific entities instead of all discovered mappings:
+
+- Process multiple entities: `--entities users,products,orders`
+- Process a single entity: `--entities items`
+- Entities are processed in dependency order automatically
+- Missing dependencies will trigger a warning but not prevent execution
+
+**Note**: When using `--entities` with entity dependencies defined in `config.yaml`, the tool will warn you about any missing dependencies but will still attempt to process the selected entities. Ensure dependent data exists in your GraphQL API before processing entities with unmet dependencies.
+
 ## Configuration
 
-The `--config` flag points to a configuration directory containing three subdirectories:
+The `--config` flag points to a configuration directory containing these necessary files:
 
-- `data/` - CSV files with actual data
-- `graphql/` - GraphQL mutation definitions
 - `mappings/` - JSON files that map CSV columns to GraphQL variables
 - `config.yaml` - _(Optional)_ Parallel processing and dependency configuration
 
@@ -160,7 +186,8 @@ Each entity has three corresponding files across these directories with matching
 
 ```json
 {
-  "csvFile": "data/items.csv",
+  "dataFile": "data/items.csv",
+  "dataFormat": "csv",
   "graphqlFile": "graphql/items.graphql",
   "mapping": {
     "name": "item_name",
@@ -217,6 +244,135 @@ entityConfig:
     concurrency: 10 # Higher concurrency for items
 ```
 
+## Supported Data Formats 📄
+
+GQL Ingest now supports multiple data formats beyond CSV for more flexible data ingestion, especially for complex nested GraphQL mutations:
+
+### Supported Formats
+
+- **CSV** - Traditional flat file format
+- **JSON** - Perfect for nested/complex data structures
+- **YAML** - Human-friendly alternative to JSON
+- **JSONL** - JSON Lines format for streaming large datasets
+
+### Format Selection
+
+The tool automatically detects the format based on file extension, or you can specify it explicitly:
+
+```bash
+# Auto-detect from mapping configuration
+gql-ingest --endpoint <url> --config ./config
+
+# Force specific format
+gql-ingest --endpoint <url> --config ./config --format json
+```
+
+### JSON/YAML Format Examples
+
+#### Direct Mapping (Entire Object)
+
+For complex GraphQL mutations with nested input types, you can map the entire data object:
+
+**data/products.json**:
+
+```json
+[
+  {
+    "name": "Premium T-Shirt",
+    "type": "PHYSICAL",
+    "options": [
+      {
+        "name": "Color",
+        "values": ["Red", "Blue", "Green"]
+      },
+      {
+        "name": "Size",
+        "values": ["S", "M", "L", "XL"]
+      }
+    ],
+    "variants": [
+      {
+        "name": "Red Small",
+        "sku": "TS-RED-S",
+        "optionMappings": [
+          { "name": "Color", "value": "Red" },
+          { "name": "Size", "value": "S" }
+        ]
+      }
+    ]
+  }
+]
+```
+
+**mappings/products.json**:
+
+```json
+{
+  "dataFile": "data/products.json",
+  "dataFormat": "json",
+  "graphqlFile": "graphql/newProduct.graphql",
+  "mapping": {
+    "input": "$" // Map entire object to input variable
+  }
+}
+```
+
+#### Path-Based Mapping
+
+For transforming flat JSON into nested structures:
+
+**data/products-flat.json**:
+
+```json
+[
+  {
+    "product_name": "Notebook",
+    "product_type": "PHYSICAL",
+    "brand": "ACME"
+  }
+]
+```
+
+**mappings/products-flat.json**:
+
+```json
+{
+  "dataFile": "data/products-flat.json",
+  "graphqlFile": "graphql/newProduct.graphql",
+  "mapping": {
+    "input": {
+      "name": "$.product_name",
+      "type": "$.product_type",
+      "brandCode": "$.brand"
+    }
+  }
+}
+```
+
+### YAML Format
+
+YAML provides a more readable alternative:
+
+**data/products.yaml**:
+
+```yaml
+- name: Premium T-Shirt
+  type: PHYSICAL
+  options:
+    - name: Color
+      values: [Red, Blue, Green]
+    - name: Size
+      values: [S, M, L, XL]
+  variants:
+    - name: Red Small
+      sku: TS-RED-S
+      optionMappings:
+        - name: Color
+          value: Red
+        - name: Size
+          value: S
+```
+
 ## Development
 
 ### Scripts