@oronts/vendure-data-hub-plugin 0.1.0

package/docs/reference/step-types.md

@@ -0,0 +1,222 @@
+# Step Types Reference
+
+Complete reference for all pipeline step types.
+
+## Overview
+
+Every pipeline is composed of steps connected by edges. Each step has a **type**
+that determines its role in the data flow. The `StepType` enum defines the
+following 11 step types:
+
+| Step Type | Purpose |
+|-----------|---------|
+| `TRIGGER` | Starts pipeline execution |
+| `EXTRACT` | Pulls data from external sources |
+| `TRANSFORM` | Modifies records with operators |
+| `VALIDATE` | Validates records against rules |
+| `ENRICH` | Adds data from external lookups |
+| `ROUTE` | Splits data flow by conditions |
+| `LOAD` | Creates or updates Vendure entities |
+| `EXPORT` | Sends data to external destinations |
+| `FEED` | Generates product feeds |
+| `SINK` | Indexes data to search engines and queues |
+| `GATE` | Human-in-the-loop approval gate |
+
+---
+
+## TRIGGER
+
+Defines how a pipeline starts. Every pipeline must have exactly one trigger step
+as its root node.
+
+**Trigger types:** `MANUAL`, `SCHEDULE`, `WEBHOOK`, `EVENT`, `FILE`, `MESSAGE`
+
+See [Pipeline Builder - trigger](../developer-guide/dsl/pipeline-builder.md#trigger) for full configuration.
+
+---
+
+## EXTRACT
+
+Pulls data from external sources into the pipeline. Supports HTTP APIs, GraphQL
+endpoints, files, databases, S3, FTP, CDC, and Vendure entity queries.
+
+See [Extractors Reference](./extractors.md) for all extractor adapters and their configuration.
+
+---
+
+## TRANSFORM
+
+Modifies records using one or more operators. Supports 61 built-in operators
+across 11 categories (data, string, numeric, date, logic, JSON, enrichment,
+aggregation, file, validation, and scripting).
+
+Optionally supports per-record retry via `retryPerRecord` configuration.
+
+See [Operators Reference](./operators.md) for all operators and their arguments.
+
+---
+
+## VALIDATE
+
+Validates records against business rules and schemas. Supports `FAIL_FAST`
+(stop on first error) and `ACCUMULATE` (collect all errors) modes.
+
+See [Pipeline Builder - validate](../developer-guide/dsl/pipeline-builder.md#validate) for configuration.
+
+---
+
+## ENRICH
+
+Adds data from external lookups to records. Uses adapter-based enrichment
+with configurable lookup sources.
+
+See [Pipeline Builder - enrich](../developer-guide/dsl/pipeline-builder.md#enrich) for configuration.
+
+---
+
+## ROUTE
+
+Splits data flow based on field conditions. Each branch defines a set of
+conditions using comparison operators (`eq`, `ne`, `gt`, `lt`, `in`,
+`contains`, `regex`, etc.). An optional `defaultTo` branch handles
+unmatched records.
+
+See [Pipeline Builder - route](../developer-guide/dsl/pipeline-builder.md#route) for configuration.
+
+---
+
+## LOAD
+
+Creates, updates, or deletes Vendure entities. Supports 22 entity types
+including products, variants, customers, collections, facets, orders,
+promotions, assets, and more.
+
+**Strategies:** `CREATE`, `UPDATE`, `UPSERT`, `MERGE`, `SOFT_DELETE`, `HARD_DELETE`
+
+See [Loaders Reference](./loaders.md) for all loader adapters and their configuration.
+
+---
+
+## EXPORT
+
+Sends data to external destinations including files, S3, SFTP, HTTP, and email.
+
+**Formats:** `CSV`, `JSON`, `XML`, `XLSX`, `NDJSON`, `PARQUET`
+
+**Targets:** `FILE`, `API`, `WEBHOOK`, `S3`, `SFTP`, `EMAIL`
+
+See [Pipeline Builder - export](../developer-guide/dsl/pipeline-builder.md#export) for configuration.
+
+---
+
+## FEED
+
+Generates product feeds for marketing platforms.
+
+**Feed types:** `GOOGLE_SHOPPING`, `META_CATALOG`, `AMAZON`, `CUSTOM`
+
+See [Feed Generators Reference](./feeds.md) for all feed adapters and their configuration.
+
+---
+
+## SINK
+
+Indexes data to search engines and publishes to message queues.
+
+**Sink types:** `ELASTICSEARCH`, `OPENSEARCH`, `MEILISEARCH`, `ALGOLIA`, `TYPESENSE`, `WEBHOOK`, `CUSTOM`
+
+See [Sinks Reference](./sinks.md) for all sink adapters and their configuration.
+
+---
+
+## GATE
+
+Adds a human-in-the-loop approval gate that pauses pipeline execution until
+approval is granted. Gates are useful for reviewing data before it is loaded
+into Vendure, especially for large or high-risk imports.
+
+### Configuration
+
+```typescript
+.gate('step-key', {
+    approvalType: 'MANUAL' | 'THRESHOLD' | 'TIMEOUT',
+    timeoutSeconds?: number,
+    errorThresholdPercent?: number,
+    notifyWebhook?: string,
+    notifyEmail?: string,
+    previewCount?: number,
+})
+```
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `approvalType` | string | Yes | Approval mode (see below) |
+| `timeoutSeconds` | number | No | Auto-approve after N seconds (`TIMEOUT` mode) |
+| `errorThresholdPercent` | number | No | Auto-approve if error rate below threshold, 0-100 (`THRESHOLD` mode) |
+| `notifyWebhook` | string | No | Webhook URL to call when gate is reached |
+| `notifyEmail` | string | No | Email address to notify when gate is reached |
+| `previewCount` | number | No | Number of records to include in the gate preview (default: 10) |
+
+### Approval Types
+
+| Type | Behavior |
+|------|----------|
+| `MANUAL` | Pipeline pauses until a user explicitly approves or rejects via the dashboard or API |
+| `THRESHOLD` | Auto-approves if the upstream error rate is below `errorThresholdPercent`; otherwise pauses for manual review |
+| `TIMEOUT` | Auto-approves after `timeoutSeconds` if no manual action is taken |
+
+### Examples
+
+**Manual approval with email notification:**
+
+```typescript
+.gate('review-before-load', {
+    approvalType: 'MANUAL',
+    notifyEmail: 'admin@example.com',
+    previewCount: 25,
+})
+```
+
+**Threshold-based auto-approval:**
+
+```typescript
+.gate('error-check', {
+    approvalType: 'THRESHOLD',
+    errorThresholdPercent: 5,
+    notifyWebhook: 'https://hooks.example.com/gate-reached',
+})
+```
+
+**Timeout-based auto-approval:**
+
+```typescript
+.gate('timed-review', {
+    approvalType: 'TIMEOUT',
+    timeoutSeconds: 3600,
+    notifyEmail: 'team@example.com',
+    previewCount: 50,
+})
+```
+
+### Pipeline Example with Gate
+
+```typescript
+createPipeline()
+    .name('Reviewed Product Import')
+    .trigger('start', { type: 'MANUAL' })
+    .extract('fetch-erp', { adapterCode: 'httpApi', /* ... */ })
+    .transform('map-fields', { operators: [ /* ... */ ] })
+    .validate('check-data', { errorHandlingMode: 'ACCUMULATE', rules: [ /* ... */ ] })
+    .gate('review-before-load', {
+        approvalType: 'MANUAL',
+        notifyEmail: 'data-team@example.com',
+        previewCount: 20,
+    })
+    .load('upsert-products', { adapterCode: 'productUpsert', strategy: 'UPSERT', matchField: 'slug' })
+    .edge('start', 'fetch-erp')
+    .edge('fetch-erp', 'map-fields')
+    .edge('map-fields', 'check-data')
+    .edge('check-data', 'review-before-load')
+    .edge('review-before-load', 'upsert-products')
+    .build();
+```

package/docs/user-guide/README.md

@@ -0,0 +1,65 @@
+# User Guide
+
+This guide covers using the Data Hub Admin UI to create and manage data pipelines.
+
+<p align="center">
+  <img src="../images/08-pipeline-editor-workflow.png" alt="Visual Pipeline Editor" width="700">
+  <br>
+  <em>Visual Pipeline Editor - Drag-and-drop interface for building data pipelines</em>
+</p>
+
+## Contents
+
+1. [Creating Pipelines](./pipelines.md) - Build and edit pipelines with the visual editor
+2. [Import/Export Wizards](./wizards.md) - Step-by-step guide to data import and export wizards
+3. [Advanced Recipes](./recipes.md) - Real-world pipeline examples and patterns
+4. [Managing Connections](./connections.md) - Set up database and API connections
+5. [Secrets Management](./secrets.md) - Store API keys and credentials securely
+6. [Scheduling Pipelines](./scheduling.md) - Automate pipeline execution
+7. [Monitoring & Logs](./monitoring.md) - Track runs and debug issues
+8. [Product Feeds](./feeds.md) - Generate feeds for Google, Meta, and more
+9. [Queue & Messaging](./queue-messaging.md) - Consume from and produce to message queues
+10. [External Integrations](./external-integrations.md) - Search engines, webhooks, and external APIs
+
+## Accessing Data Hub
+
+1. Log in to the Vendure Admin UI
+2. Click **Data Hub** in the left navigation menu
+3. You'll see the main dashboard with pipeline overview
+
+## Dashboard Overview
+
+The Data Hub dashboard shows:
+
+- **Active Pipelines** - Pipelines currently enabled
+- **Recent Runs** - Latest pipeline executions
+- **Failed Records** - Records requiring attention
+- **Upcoming Schedules** - Next scheduled runs
+
+## Navigation
+
+| Section | Purpose |
+|---------|---------|
+| Pipelines | Create, edit, run pipelines |
+| Runs | View execution history and logs |
+| Connections | Manage external system connections |
+| Secrets | Store sensitive credentials |
+| Errors | Review and retry failed records |
+| Analytics | View performance metrics |
+| Settings | Configure plugin options |
+
+## Required Permissions
+
+To use Data Hub, your admin role needs these permissions:
+
+| Action | Permission Required |
+|--------|---------------------|
+| View pipelines | `ReadDataHubPipeline` |
+| Create/edit pipelines | `CreateDataHubPipeline`, `UpdateDataHubPipeline` |
+| Run pipelines | `RunDataHubPipeline` |
+| View run history | `ViewDataHubRuns` |
+| Manage connections | `ManageDataHubConnections` |
+| Manage secrets | `CreateDataHubSecret`, `ReadDataHubSecret` |
+| Configure settings | `UpdateDataHubSettings` |
+
+Ask your administrator to assign these permissions if you don't have access.

package/docs/user-guide/connections.md

@@ -0,0 +1,234 @@
+# Managing Connections
+
+Connections store reusable configuration for external systems like databases, APIs, and cloud storage.
+
+<p align="center">
+  <img src="../images/02-connections-list.png" alt="Connections List" width="700">
+  <br>
+  <em>Connections List - Manage all external system connections</em>
+</p>
+
+## Why Use Connections
+
+- **Reusability** - Use the same connection in multiple pipelines
+- **Security** - Credentials are stored securely, not in pipeline config
+- **Maintainability** - Update connection details in one place
+- **Environment Flexibility** - Different connections for dev/staging/prod
+
+## Creating a Connection
+
+1. Go to **Data Hub > Connections**
+2. Click **Create Connection**
+3. Select the connection type
+4. Configure the connection settings
+5. Click **Save**
+
+<p align="center">
+  <img src="../images/03-connection-detail.png" alt="Connection Configuration" width="700">
+  <br>
+  <em>Connection Configuration - Multiple connection types supported</em>
+</p>
+
+## Connection Types
+
+### HTTP / REST API
+
+Connect to REST APIs:
+
+| Field | Description |
+|-------|-------------|
+| Base URL | API base URL (e.g., `https://api.example.com`) |
+| Timeout | Request timeout in milliseconds |
+| Headers | Default headers for all requests |
+| Auth Type | None, Bearer Token, Basic Auth, API Key |
+| Secret Code | Reference to secret for credentials |
+
+Example:
+```
+Code: supplier-api
+Type: http
+Base URL: https://api.supplier.com/v1
+Timeout: 30000
+Auth Type: Bearer Token
+Secret Code: supplier-api-token
+```
+
+### PostgreSQL
+
+Connect to PostgreSQL databases:
+
+| Field | Description |
+|-------|-------------|
+| Host | Database server hostname |
+| Port | Server port (default: 5432) |
+| Database | Database name |
+| Username | Database user |
+| Password Secret | Reference to password secret |
+| SSL | Enable SSL connection |
+| Pool Size | Connection pool size |
+
+Example:
+```
+Code: erp-db
+Type: postgres
+Host: db.example.com
+Port: 5432
+Database: erp
+Username: vendure_reader
+Password Secret: erp-db-password
+SSL: true
+Pool Size: 5
+```
+
+### MySQL
+
+Connect to MySQL databases:
+
+| Field | Description |
+|-------|-------------|
+| Host | Database server hostname |
+| Port | Server port (default: 3306) |
+| Database | Database name |
+| Username | Database user |
+| Password Secret | Reference to password secret |
+
+### Microsoft SQL Server
+
+Connect to MSSQL databases:
+
+| Field | Description |
+|-------|-------------|
+| Host | Server hostname |
+| Port | Server port (default: 1433) |
+| Database | Database name |
+| Username | Database user |
+| Password Secret | Reference to password secret |
+| Encrypt | Enable encryption |
+| Trust Server Certificate | Trust self-signed certs |
+
+### Amazon S3
+
+Connect to S3 or compatible storage (MinIO, DigitalOcean Spaces):
+
+| Field | Description |
+|-------|-------------|
+| Region | AWS region (e.g., `us-east-1`) |
+| Bucket | Default bucket name |
+| Access Key ID Secret | Reference to access key secret |
+| Secret Access Key Secret | Reference to secret key |
+| Endpoint | Custom endpoint for S3-compatible services |
+
+Example for MinIO:
+```
+Code: local-storage
+Type: s3
+Region: us-east-1
+Bucket: imports
+Endpoint: http://minio:9000
+Access Key ID Secret: minio-access-key
+Secret Access Key Secret: minio-secret-key
+```
+
+### FTP / SFTP
+
+Connect to file servers:
+
+| Field | Description |
+|-------|-------------|
+| Host | Server hostname |
+| Port | Server port (FTP: 21, SFTP: 22) |
+| Protocol | FTP or SFTP |
+| Username | Login username |
+| Password Secret | Reference to password secret |
+| Private Key Secret | For SFTP key-based auth |
+| Base Path | Default directory |
+
+## Testing Connections
+
+1. After creating a connection, click **Test**
+2. The system will attempt to connect
+3. View success or error message
+4. Fix any issues before using in pipelines
+
+## Using Connections in Pipelines
+
+Reference a connection by its code:
+
+### In Visual Editor
+
+1. Add an Extract or Export step
+2. Select the adapter (e.g., Database)
+3. Choose the connection from the dropdown
+4. Configure step-specific settings
+
+### In DSL
+
+```typescript
+.extract('fetch-products', {
+    adapterCode: 'httpApi',
+    connectionCode: 'erp-api',  // Connection code
+    url: '/products',
+})
+```
+
+## Environment Variables
+
+Connection settings can reference environment variables:
+
+```
+Host: ${DB_HOST}
+Port: ${DB_PORT}
+Database: ${DB_NAME}
+```
+
+Variables are resolved at runtime. This allows different values for different environments without changing the connection configuration.
+
+## Connection Health
+
+View connection status:
+
+1. Go to **Data Hub > Connections**
+2. Each connection shows a health indicator:
+   - 🟢 Healthy - Last test successful
+   - 🟡 Unknown - Not tested recently
+   - 🔴 Failed - Last test failed
+
+Run **Test** to update the status.
+
+## Editing Connections
+
+1. Go to **Data Hub > Connections**
+2. Click on a connection
+3. Modify settings
+4. Click **Save**
+
+Note: Changing a connection affects all pipelines using it.
+
+## Deleting Connections
+
+1. Go to **Data Hub > Connections**
+2. Click the menu (⋮) on a connection
+3. Select **Delete**
+4. Confirm deletion
+
+Warning: You cannot delete a connection used by active pipelines.
+
+## Best Practices
+
+### Naming
+
+- Use descriptive codes: `production-mysql`, `staging-api`
+- Include environment: `dev-erp-db`, `prod-erp-db`
+- Include purpose: `supplier-catalog-api`
+
+### Security
+
+- Always use secrets for credentials, never inline passwords
+- Use read-only database users when possible
+- Limit pool sizes to prevent overwhelming external systems
+
+### Testing
+
+- Test connections before using in pipelines
+- Set up monitoring for production connections
+- Have backup connections for critical systems

package/docs/user-guide/external-integrations.md

@@ -0,0 +1,200 @@
+# External Integrations
+
+Data Hub includes integrations for external services with built-in fault tolerance.
+
+## Search Engine Sinks
+
+Index your data to search engines with automatic circuit breaker protection.
+
+### Supported Search Engines
+
+| Engine | Adapter Code | Features |
+|--------|--------------|----------|
+| MeiliSearch | `meilisearch` | Full-text search, faceting |
+| Elasticsearch | `elasticsearch` | Advanced querying, analytics |
+| OpenSearch | `elasticsearch` | AWS-compatible (uses Elasticsearch adapter) |
+| Algolia | `algolia` | Instant search, relevance tuning |
+| Typesense | `typesense` | Typo-tolerant, fast |
+
+### Configuration
+
+```typescript
+// Pipeline with search sink
+.sink('index-products', {
+    adapterCode: 'meilisearch',
+    host: 'http://localhost:7700',
+    apiKeySecretCode: 'meili-api-key',
+    indexName: 'products',
+    primaryKey: 'id',
+    batchSize: 100,
+})
+```
+
+## Webhook Sink
+
+Send data to external HTTP endpoints with retry and circuit breaker.
+
+```typescript
+.sink('notify-external', {
+    adapterCode: 'webhook',
+    url: 'https://api.example.com/webhook',
+    method: 'POST',
+    bearerTokenSecretCode: 'webhook-token',
+    batchSize: 50,
+    timeoutMs: 30000,
+    retries: 3,
+})
+```
+
+## Queue Producer Sink
+
+Publish messages to message queues. See [Queue & Messaging](./queue-messaging.md) for details.
+
+```typescript
+.sink('to-queue', {
+    adapterCode: 'queueProducer',
+    queueType: 'rabbitmq-amqp',
+    connectionCode: 'my-rabbitmq',
+    queueName: 'product-updates',
+})
+```
+
+## Circuit Breaker Protection
+
+All external sinks include automatic circuit breaker protection to prevent cascading failures.
+
+### How It Works
+
+1. **Closed State** (Normal): Requests flow normally
+2. **Open State** (Protected): After 5 failures in 60 seconds, circuit opens and requests fail fast
+3. **Half-Open State** (Testing): After 30 seconds, allows test requests
+4. **Recovery**: After 3 consecutive successes, circuit closes
+
+### Configuration
+
+Circuit breaker uses defaults from plugin configuration:
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `failureThreshold` | 5 | Failures before opening |
+| `successThreshold` | 3 | Successes to close |
+| `resetTimeoutMs` | 30000 | Time before half-open |
+| `failureWindowMs` | 60000 | Window for counting failures |
+
+### Monitoring
+
+When circuit breaker activates, warnings are logged:
+
+```
+WARN [SinkExecutor] Circuit breaker OPEN for sink:elasticsearch:https://search.example.com
+     - Blocking requests until reset
+     - Time until half-open: 25000ms
+```
+
+## HTTP Lookup Operator
+
+Enrich records by fetching data from external APIs during transformation.
+
+### Features
+
+- **Caching**: Configurable TTL to reduce API calls
+- **Rate Limiting**: Limit requests per second per domain
+- **Authentication**: Bearer token, API key, Basic auth
+- **Retry Logic**: Automatic retry on transient errors
+- **Batch Processing**: Parallel requests for efficiency
+
+### Example
+
+```typescript
+.transform('enrich-inventory', {
+    operators: [
+        {
+            op: 'httpLookup',
+            args: {
+                url: 'https://api.inventory.com/stock/{{sku}}',
+                target: 'stockLevel',
+                responsePath: 'data.available',
+                bearerTokenSecretCode: 'inventory-api-key',
+                cacheTtlSec: 300,
+                rateLimitPerSecond: 50,
+                maxRetries: 3,
+                default: 0,
+            },
+        },
+    ],
+})
+```
+
+### Options
+
+| Option | Description |
+|--------|-------------|
+| `url` | Endpoint URL with `{{field}}` placeholders |
+| `method` | GET or POST |
+| `target` | Field to store response |
+| `responsePath` | JSON path to extract |
+| `cacheTtlSec` | Cache duration (0 = disabled) |
+| `rateLimitPerSecond` | Max requests per second |
+| `maxRetries` | Retry attempts on failure |
+| `timeoutMs` | Request timeout |
+| `bearerTokenSecretCode` | Bearer token secret |
+| `apiKeySecretCode` | API key secret |
+| `default` | Value if lookup fails |
+
+## Best Practices
+
+### 1. Use Caching
+
+Always enable caching for HTTP lookups to reduce load:
+
+```typescript
+{ op: 'httpLookup', args: { cacheTtlSec: 300, ... } }
+```
+
+### 2. Set Rate Limits
+
+Respect external API limits:
+
+```typescript
+{ op: 'httpLookup', args: { rateLimitPerSecond: 10, ... } }
+```
+
+### 3. Provide Defaults
+
+Handle failures gracefully:
+
+```typescript
+{ op: 'httpLookup', args: { default: null, skipOn404: true, ... } }
+```
+
+### 4. Monitor Circuit Breakers
+
+Check logs for circuit breaker activations and investigate root causes.
+
+### 5. Use Batch Sizes
+
+Tune batch sizes based on external service capacity:
+
+```typescript
+.sink('index', { batchSize: 100, ... })
+```
+
+## Troubleshooting
+
+### Circuit Breaker Keeps Opening
+
+- Check external service health
+- Increase `failureThreshold` for flaky services
+- Add retry logic with `maxRetries`
+
+### High Latency
+
+- Enable caching
+- Reduce batch sizes
+- Add rate limiting
+
+### Authentication Errors
+
+- Verify secret values
+- Check token expiration
+- Confirm header names