@aifabrix/builder 2.42.1 → 2.44.0

package/integration/roundtrip-test-local/README.md

@@ -0,0 +1,144 @@
+# Roundtrip Test
+
+Roundtrip validation test
+
+## System Information
+
+- **System Key**: `roundtrip-test-local`
+- **System Type**: `openapi`
+- **Datasources**: 1
+
+## Files
+
+- `application.yaml` – Application configuration with `app` and `externalIntegration` blocks
+- `roundtrip-test-local-system.yaml` – External system definition (authentication, OpenAPI/MCP, RBAC)
+- `roundtrip-test-local-datasource-roundtrip-test-company.yaml` – Datasource: Company
+- `env.template` – Environment variables template (secrets, API keys)
+- `roundtrip-test-local-deploy.json` – Deployment manifest (generated by `aifabrix json roundtrip-test-local`)
+- `deploy.js` – Deploy script for the integration
+- `wizard.yaml` – Wizard configuration (if created via wizard)
+
+Optional: `rbac.yaml` – Roles and permissions merged into the system when present.
+
+## Quick Start
+
+Login to your controller
+```bash
+aifabrix auth config --set-controller <url> --set-environment dev
+aifabrix login
+```
+
+### 1. Extend External System
+
+Use the interactive wizard to extend your existing system:
+
+```bash
+aifabrix wizard --app roundtrip-test-local
+```
+
+### 2. Configure Authentication and Datasources
+
+Edit files in `integration/roundtrip-test-local/`:
+
+- **Authentication**: `roundtrip-test-local-system.yaml` (auth type, credentials placeholders)
+- **Field mappings**: `roundtrip-test-local-datasource-*-datasource.yaml` (dimensions, attributes, operations)
+- **Credential and configuration**: `env.template` (security settings and configuration variables)
+
+
+### Secrets
+
+Secrets are resolved from `.aifabrix` or key vault. Set them with:
+
+```bash
+aifabrix secret set roundtrip-test-local/apiKey <your value>   # API Key
+```
+
+
+### 3. Validate Configuration
+
+```bash
+aifabrix validate roundtrip-test-local
+```
+
+### 4. Repair Deployment Manifest
+
+**Run repair regularly.** It keeps naming conventions, filenames, and the deployment manifest aligned with AI Fabrix platform best practices. Use it after editing datasources, env.template, or system config—and run it often to catch drift early.
+
+```bash
+aifabrix repair roundtrip-test-local
+```
+
+Options:
+  --auth <method>  Set authentication method (oauth2, aad, apikey, basic, queryParam, oidc, hmac, none); updates system file and env.template
+  --dry-run        Report changes only; do not write
+  --rbac           Ensure RBAC permissions per datasource and add default Admin/Reader roles if none exist
+  --expose         Set exposed.attributes on each datasource to all fieldMappings.attributes keys
+  --sync           Add default sync section to datasources that lack it
+  --test           Generate testPayload.payloadTemplate and testPayload.expectedResult from attributes
+
+### 5. Upload to dataplane
+
+```bash
+aifabrix upload roundtrip-test-local
+```
+
+## Testing
+
+### Unit Tests (Local Validation, No API)
+
+```bash
+aifabrix test roundtrip-test-local
+```
+
+### Integration Tests (Via Dataplane)
+
+```bash
+aifabrix test-integration roundtrip-test-local
+```
+
+### End-to-end Tests (Via Dataplane)
+
+```bash
+aifabrix test-e2e roundtrip-test-local
+```
+
+Options:
+  -e, --env <env>  Environment: dev, tst, or pro (builder: dev/tst for container)
+  -v, --verbose    Show detailed step output and poll progress
+  --debug          Include debug output and write log to integration/roundtrip-test-local/logs/
+  --no-async       Use sync mode (no polling); single POST per datasource
+
+### E2E tests per datasource
+
+To run a full E2E test for a single datasource (config, credential, sync, data, CIP), use `aifabrix datasource test-e2e` with the datasource key and app:
+
+```bash
+# Company
+aifabrix datasource test-e2e roundtrip-test-company --app roundtrip-test-local
+
+```
+
+Options:
+  -a, --app roundtrip-test-local              App key (default: resolve from cwd if inside integration/roundtrip-test-local/)
+  -e, --env <env>                    Environment: dev, tst, or pro
+  -v, --verbose                      Show detailed step output and poll progress
+  --debug                            Include debug output and write log to integration/roundtrip-test-local/logs/
+  --test-crud                        Enable CRUD lifecycle test (body testCrud: true)
+  --record-id <id>                   Record ID for test (body recordId)
+  --no-cleanup                       Disable cleanup after test (body cleanup: false)
+  --primary-key-value <value|@path>  Primary key value or path to JSON file (e.g. @pk.json) for body primaryKeyValue
+  --no-async                         Use sync mode (no polling); single POST, no asyncRun
+
+## Deployment
+
+Deploy via miso-controller pipeline (same as regular apps). Auth and controller come from `aifabrix login` and `aifabrix auth config`:
+
+```bash
+aifabrix deploy roundtrip-test-local
+```
+
+## Troubleshooting
+
+- **Validation errors**: Run `aifabrix validate roundtrip-test-local` to see schema and manifest errors.
+- **Deployment / auth**: Run `aifabrix auth config --set-controller <url> --set-environment <env>` and `aifabrix login` before `aifabrix deploy`.
+- **File not found**: Run commands from the project root (where `package.json` and `integration/` live).

package/integration/roundtrip-test-local/application.yaml

@@ -0,0 +1,13 @@
+app:
+  key: roundtrip-test-local
+  displayName: Roundtrip Test
+  description: Roundtrip validation test
+  type: external
+externalIntegration:
+  schemaBasePath: ./
+  systems:
+    - roundtrip-test-local-system.yaml
+  dataSources:
+    - roundtrip-test-local-datasource-roundtrip-test-company.yaml
+  autopublish: true
+  version: 1.0.0

package/integration/roundtrip-test-local/env.template

@@ -0,0 +1,15 @@
+# Environment variables for external system integration
+# Use kv:// (or aifabrix secret set) for sensitive values; plain values for non-sensitive configuration.
+#
+
+# Authentication
+# Type: oauth2
+KV_ROUNDTRIP_TEST_LOCAL_CLIENTID=kv://roundtrip-test-local/clientId
+KV_ROUNDTRIP_TEST_LOCAL_CLIENTSECRET=kv://roundtrip-test-local/clientSecret
+# Non-secure (e.g. URLs): baseUrl, tokenUrl, authorizationUrl
+
+# Configuration
+# KV_ROUNDTRIP_TEST_CLIENTID
+KV_ROUNDTRIP_TEST_CLIENTID=kv://roundtrip-test/clientId
+# KV_ROUNDTRIP_TEST_CLIENTSECRET
+KV_ROUNDTRIP_TEST_CLIENTSECRET=kv://roundtrip-test/clientSecret

package/integration/roundtrip-test-local/roundtrip-test-local-datasource-roundtrip-test-company.yaml

@@ -0,0 +1,14 @@
+key: roundtrip-test-company
+displayName: Company
+systemKey: roundtrip-test-local
+entityType: record-storage
+resourceType: customer
+primaryKey:
+  - country
+fieldMappings:
+  dimensions:
+    country: metadata.country
+  attributes:
+    country:
+      expression: '{{metadata.country}}'
+      type: string

package/integration/roundtrip-test-local/roundtrip-test-local-deploy.json

@@ -0,0 +1,61 @@
+{
+  "system": {
+    "key": "roundtrip-test-local",
+    "displayName": "Roundtrip Test",
+    "description": "Roundtrip validation test",
+    "type": "openapi",
+    "enabled": true,
+    "authentication": {
+      "method": "oauth2",
+      "variables": {
+        "baseUrl": "https://api.example.com",
+        "tokenUrl": "https://api.example.com/oauth/token",
+        "authorizationUrl": "https://api.example.com/oauth/authorize"
+      },
+      "security": {
+        "clientId": "kv://roundtrip-test/clientId",
+        "clientSecret": "kv://roundtrip-test/clientSecret"
+      }
+    },
+    "configuration": [
+      {
+        "name": "KV_ROUNDTRIP_TEST_CLIENTID",
+        "value": "roundtrip-test/clientId",
+        "location": "keyvault",
+        "required": true
+      },
+      {
+        "name": "KV_ROUNDTRIP_TEST_CLIENTSECRET",
+        "value": "roundtrip-test/clientSecret",
+        "location": "keyvault",
+        "required": true
+      }
+    ],
+    "dataSources": [
+      "roundtrip-test-company"
+    ]
+  },
+  "dataSources": [
+    {
+      "key": "roundtrip-test-company",
+      "displayName": "Company",
+      "systemKey": "roundtrip-test-local",
+      "entityType": "record-storage",
+      "resourceType": "customer",
+      "primaryKey": [
+        "country"
+      ],
+      "fieldMappings": {
+        "dimensions": {
+          "country": "metadata.country"
+        },
+        "attributes": {
+          "country": {
+            "expression": "{{metadata.country}}",
+            "type": "string"
+          }
+        }
+      }
+    }
+  ]
+}

package/integration/roundtrip-test-local/roundtrip-test-local-system.yaml

@@ -0,0 +1,25 @@
+key: roundtrip-test-local
+displayName: Roundtrip Test
+description: Roundtrip validation test
+type: openapi
+enabled: true
+authentication:
+  method: oauth2
+  variables:
+    baseUrl: https://api.example.com
+    tokenUrl: https://api.example.com/oauth/token
+    authorizationUrl: https://api.example.com/oauth/authorize
+  security:
+    clientId: kv://roundtrip-test/clientId
+    clientSecret: kv://roundtrip-test/clientSecret
+configuration:
+  - name: KV_ROUNDTRIP_TEST_CLIENTID
+    value: roundtrip-test/clientId
+    location: keyvault
+    required: true
+  - name: KV_ROUNDTRIP_TEST_CLIENTSECRET
+    value: roundtrip-test/clientSecret
+    location: keyvault
+    required: true
+dataSources:
+  - roundtrip-test-company

package/integration/roundtrip-test-local2/README.md

@@ -0,0 +1,144 @@
+# Roundtrip Test
+
+Roundtrip validation test
+
+## System Information
+
+- **System Key**: `roundtrip-test-local2`
+- **System Type**: `openapi`
+- **Datasources**: 1
+
+## Files
+
+- `application.yaml` – Application configuration with `app` and `externalIntegration` blocks
+- `roundtrip-test-local2-system.yaml` – External system definition (authentication, OpenAPI/MCP, RBAC)
+- `roundtrip-test-local2-datasource-company.yaml` – Datasource: Company
+- `env.template` – Environment variables template (secrets, API keys)
+- `roundtrip-test-local2-deploy.json` – Deployment manifest (generated by `aifabrix json roundtrip-test-local2`)
+- `deploy.js` – Deploy script for the integration
+- `wizard.yaml` – Wizard configuration (if created via wizard)
+
+Optional: `rbac.yaml` – Roles and permissions merged into the system when present.
+
+## Quick Start
+
+Login to your controller
+```bash
+aifabrix auth config --set-controller <url> --set-environment dev
+aifabrix login
+```
+
+### 1. Extend External System
+
+Use the interactive wizard to extend your existing system:
+
+```bash
+aifabrix wizard --app roundtrip-test-local2
+```
+
+### 2. Configure Authentication and Datasources
+
+Edit files in `integration/roundtrip-test-local2/`:
+
+- **Authentication**: `roundtrip-test-local2-system.yaml` (auth type, credentials placeholders)
+- **Field mappings**: `roundtrip-test-local2-datasource-*-datasource.yaml` (dimensions, attributes, operations)
+- **Credential and configuration**: `env.template` (security settings and configuration variables)
+
+
+### Secrets
+
+Secrets are resolved from `.aifabrix` or key vault. Set them with:
+
+```bash
+aifabrix secret set roundtrip-test-local2/apiKey <your value>   # API Key
+```
+
+
+### 3. Validate Configuration
+
+```bash
+aifabrix validate roundtrip-test-local2
+```
+
+### 4. Repair Deployment Manifest
+
+**Run repair regularly.** It keeps naming conventions, filenames, and the deployment manifest aligned with AI Fabrix platform best practices. Use it after editing datasources, env.template, or system config—and run it often to catch drift early.
+
+```bash
+aifabrix repair roundtrip-test-local2
+```
+
+Options:
+  --auth <method>  Set authentication method (oauth2, aad, apikey, basic, queryParam, oidc, hmac, none); updates system file and env.template
+  --dry-run        Report changes only; do not write
+  --rbac           Ensure RBAC permissions per datasource and add default Admin/Reader roles if none exist
+  --expose         Set exposed.attributes on each datasource to all fieldMappings.attributes keys
+  --sync           Add default sync section to datasources that lack it
+  --test           Generate testPayload.payloadTemplate and testPayload.expectedResult from attributes
+
+### 5. Upload to dataplane
+
+```bash
+aifabrix upload roundtrip-test-local2
+```
+
+## Testing
+
+### Unit Tests (Local Validation, No API)
+
+```bash
+aifabrix test roundtrip-test-local2
+```
+
+### Integration Tests (Via Dataplane)
+
+```bash
+aifabrix test-integration roundtrip-test-local2
+```
+
+### End-to-end Tests (Via Dataplane)
+
+```bash
+aifabrix test-e2e roundtrip-test-local2
+```
+
+Options:
+  -e, --env <env>  Environment: dev, tst, or pro (builder: dev/tst for container)
+  -v, --verbose    Show detailed step output and poll progress
+  --debug          Include debug output and write log to integration/roundtrip-test-local2/logs/
+  --no-async       Use sync mode (no polling); single POST per datasource
+
+### E2E tests per datasource
+
+To run a full E2E test for a single datasource (config, credential, sync, data, CIP), use `aifabrix datasource test-e2e` with the datasource key and app:
+
+```bash
+# Company
+aifabrix datasource test-e2e roundtrip-test-local2-company --app roundtrip-test-local2
+
+```
+
+Options:
+  -a, --app roundtrip-test-local2              App key (default: resolve from cwd if inside integration/roundtrip-test-local2/)
+  -e, --env <env>                    Environment: dev, tst, or pro
+  -v, --verbose                      Show detailed step output and poll progress
+  --debug                            Include debug output and write log to integration/roundtrip-test-local2/logs/
+  --test-crud                        Enable CRUD lifecycle test (body testCrud: true)
+  --record-id <id>                   Record ID for test (body recordId)
+  --no-cleanup                       Disable cleanup after test (body cleanup: false)
+  --primary-key-value <value|@path>  Primary key value or path to JSON file (e.g. @pk.json) for body primaryKeyValue
+  --no-async                         Use sync mode (no polling); single POST, no asyncRun
+
+## Deployment
+
+Deploy via miso-controller pipeline (same as regular apps). Auth and controller come from `aifabrix login` and `aifabrix auth config`:
+
+```bash
+aifabrix deploy roundtrip-test-local2
+```
+
+## Troubleshooting
+
+- **Validation errors**: Run `aifabrix validate roundtrip-test-local2` to see schema and manifest errors.
+- **Deployment / auth**: Run `aifabrix auth config --set-controller <url> --set-environment <env>` and `aifabrix login` before `aifabrix deploy`.
+- **File not found**: Run commands from the project root (where `package.json` and `integration/` live).

package/integration/roundtrip-test-local2/application.yaml

@@ -0,0 +1,13 @@
+app:
+  key: roundtrip-test-local2
+  displayName: Roundtrip Test
+  description: Roundtrip validation test
+  type: external
+externalIntegration:
+  schemaBasePath: ./
+  systems:
+    - roundtrip-test-local2-system.yaml
+  dataSources:
+    - roundtrip-test-local2-datasource-company.yaml
+  autopublish: true
+  version: 1.0.0

package/integration/roundtrip-test-local2/env.template

@@ -0,0 +1,15 @@
+# Environment variables for external system integration
+# Use kv:// (or aifabrix secret set) for sensitive values; plain values for non-sensitive configuration.
+#
+
+# Authentication
+# Type: oauth2
+KV_ROUNDTRIP_TEST_LOCAL2_CLIENTID=kv://roundtrip-test-local2/clientId
+KV_ROUNDTRIP_TEST_LOCAL2_CLIENTSECRET=kv://roundtrip-test-local2/clientSecret
+# Non-secure (e.g. URLs): baseUrl, tokenUrl, authorizationUrl
+
+# Configuration
+# KV_ROUNDTRIP_TEST_CLIENTID
+KV_ROUNDTRIP_TEST_CLIENTID=kv://roundtrip-test-local2/clientId
+# KV_ROUNDTRIP_TEST_CLIENTSECRET
+KV_ROUNDTRIP_TEST_CLIENTSECRET=kv://roundtrip-test-local2/clientSecret

package/integration/roundtrip-test-local2/roundtrip-test-local2-datasource-company.yaml

@@ -0,0 +1,31 @@
+key: roundtrip-test-local2-company
+displayName: Company
+systemKey: roundtrip-test-local2
+entityType: recordStorage
+resourceType: record
+primaryKey:
+  - id
+labelKey:
+  - id
+metadataSchema:
+  type: object
+  required:
+    - id
+    - externalId
+  properties:
+    id:
+      type: string
+      index: true
+    externalId:
+      type: string
+      index: true
+fieldMappings:
+  attributes:
+    id:
+      expression: '{{raw.id}}'
+    externalId:
+      expression: '{{raw.externalId}}'
+exposed:
+  schema:
+    id: metadata.id
+    externalId: metadata.externalId

package/integration/roundtrip-test-local2/roundtrip-test-local2-deploy.json

@@ -0,0 +1,86 @@
+{
+  "system": {
+    "key": "roundtrip-test-local2",
+    "displayName": "Roundtrip Test",
+    "description": "Roundtrip validation test",
+    "type": "openapi",
+    "enabled": true,
+    "authentication": {
+      "method": "oauth2",
+      "variables": {
+        "baseUrl": "https://api.example.com",
+        "tokenUrl": "https://api.example.com/oauth/token",
+        "authorizationUrl": "https://api.example.com/oauth/authorize"
+      },
+      "security": {
+        "clientId": "kv://roundtrip-test-local2/clientId",
+        "clientSecret": "kv://roundtrip-test-local2/clientSecret"
+      }
+    },
+    "configuration": [
+      {
+        "name": "KV_ROUNDTRIP_TEST_CLIENTID",
+        "value": "roundtrip-test-local2/clientId",
+        "location": "keyvault",
+        "required": true
+      },
+      {
+        "name": "KV_ROUNDTRIP_TEST_CLIENTSECRET",
+        "value": "roundtrip-test-local2/clientSecret",
+        "location": "keyvault",
+        "required": true
+      }
+    ],
+    "dataSources": [
+      "roundtrip-test-local2-company"
+    ]
+  },
+  "dataSources": [
+    {
+      "key": "roundtrip-test-local2-company",
+      "displayName": "Company",
+      "systemKey": "roundtrip-test-local2",
+      "entityType": "recordStorage",
+      "resourceType": "record",
+      "primaryKey": [
+        "id"
+      ],
+      "labelKey": [
+        "id"
+      ],
+      "metadataSchema": {
+        "type": "object",
+        "required": [
+          "id",
+          "externalId"
+        ],
+        "properties": {
+          "id": {
+            "type": "string",
+            "index": true
+          },
+          "externalId": {
+            "type": "string",
+            "index": true
+          }
+        }
+      },
+      "fieldMappings": {
+        "attributes": {
+          "id": {
+            "expression": "{{raw.id}}"
+          },
+          "externalId": {
+            "expression": "{{raw.externalId}}"
+          }
+        }
+      },
+      "exposed": {
+        "schema": {
+          "id": "metadata.id",
+          "externalId": "metadata.externalId"
+        }
+      }
+    }
+  ]
+}

package/integration/roundtrip-test-local2/roundtrip-test-local2-system.yaml

@@ -0,0 +1,25 @@
+key: roundtrip-test-local2
+displayName: Roundtrip Test
+description: Roundtrip validation test
+type: openapi
+enabled: true
+authentication:
+  method: oauth2
+  variables:
+    baseUrl: https://api.example.com
+    tokenUrl: https://api.example.com/oauth/token
+    authorizationUrl: https://api.example.com/oauth/authorize
+  security:
+    clientId: kv://roundtrip-test-local2/clientId
+    clientSecret: kv://roundtrip-test-local2/clientSecret
+configuration:
+  - name: KV_ROUNDTRIP_TEST_CLIENTID
+    value: roundtrip-test-local2/clientId
+    location: keyvault
+    required: true
+  - name: KV_ROUNDTRIP_TEST_CLIENTSECRET
+    value: roundtrip-test-local2/clientSecret
+    location: keyvault
+    required: true
+dataSources:
+  - roundtrip-test-local2-company

package/integration/test/wizard.yaml

@@ -0,0 +1,8 @@
+appName: test
+mode: create-system
+source:
+  type: openapi-file
+  filePath: /workspace/aifabrix-builder/integration/hubspot-test/companies.json
+credential:
+  action: create
+preferences: {}

package/jest.config.default.js

@@ -0,0 +1,10 @@
+/**
+ * Default test project only (no isolated suites). Used by test-wrapper pass 1.
+ * @fileoverview
+ */
+
+const { defaultProject } = require('./jest.projects');
+
+module.exports = {
+  projects: [defaultProject]
+};

package/jest.config.integration.fixtures.js

@@ -0,0 +1,22 @@
+/**
+ * Integration tests that do not build images or talk to Docker.
+ * Use: npm run test:integration:fixtures
+ *
+ * Full Docker lifecycle tests: npm run test:integration (includes steps/ + workflow).
+ *
+ * @fileoverview Fixture-only integration Jest config
+ */
+
+const { sharedTransform } = require('./jest.projects');
+
+module.exports = {
+  displayName: 'integration-fixtures',
+  testEnvironment: 'node',
+  transform: sharedTransform,
+  testMatch: ['**/tests/integration/hubspot/**/*.test.js'],
+  testPathIgnorePatterns: ['/node_modules/', '\\\\node_modules\\\\'],
+  maxWorkers: 1,
+  setupFiles: ['<rootDir>/tests/capture-real-fs.js'],
+  setupFilesAfterEnv: ['<rootDir>/tests/setup.js'],
+  testTimeout: 60000
+};