@aifabrix/builder 2.36.2 → 2.37.5

package/.cursor/rules/project-rules.mdc

@@ -70,6 +70,23 @@ lib/
 └── schema/             # JSON schemas
 ```
 
+### Generated Output (integration/ and builder/)
+
+Files under **integration/** and **builder/** are **auto-generated** by the CLI. When fixing bugs or changing behavior, validate **where** each file is produced so fixes go into the generator, not only into the generated artifact.
+
+- **integration/** – External system / wizard output:
+  - Path: `integration/<appName>/` (see `lib/utils/paths.js` → `getIntegrationPath`).
+  - Generated by: `lib/generator/wizard.js` (`generateWizardFiles`, `generateConfigFilesForWizard`), `lib/external-system/download.js`, wizard commands in `lib/commands/wizard-core.js` and `lib/commands/wizard.js`.
+  - Typical outputs: `variables.yaml`, `env.template`, `README.md`, `*-system.json`, `*-datasource*.json`, `*-deploy.json`, deploy script (`deploy.js`), and optionally `wizard.yaml`, `error.log`.
+- **builder/** – Application (non-external) output:
+  - Path: `builder/<appName>/` or custom root via `AIFABRIX_BUILDER_DIR` (see `lib/utils/paths.js` → `getBuilderPath`).
+  - Generated by: app create/register flow, `lib/generator/index.js`, `lib/commands/up-common.js`, `lib/core/secrets.js`, and related app/deploy logic.
+  - Typical outputs: `variables.yaml`, `env.template`, deploy JSON, `.env` (from secrets), and app-specific config.
+
+**Editable vs generated:**
+- Some generated files are **intended to be edited** (e.g. `variables.yaml`, `env.template`, `README.md`, `wizard.yaml`). Improvements to defaults or structure still belong in the generator/templates.
+- **When debugging:** First identify the **source of generation** (which module and function write the file). Fix bugs in that generator or template; avoid treating a one-off edit in integration/ or builder/ as the permanent fix unless it’s a deliberate local override.
+
 ### CLI Command Pattern
 Commands are defined in `lib/cli.js` using Commander.js:
 ```javascript
@@ -850,6 +867,7 @@ Define request/response types using JSDoc `@typedef`:
 
 ### Must Do (✅)
 - ✅ Validate all inputs (app names, file paths, URLs)
+- ✅ When fixing bugs in integration/ or builder/ output: identify the generator that produces the file and fix the source (lib/generator, lib/commands, templates), not only the generated artifact
 - ✅ Use try-catch for all async operations
 - ✅ Provide meaningful error messages with context
 - ✅ Use JSDoc for all public functions
@@ -874,6 +892,7 @@ Define request/response types using JSDoc `@typedef`:
 - ❌ Never use `eval()` or `Function()` constructor
 - ❌ Never use raw paths (always use path.join)
 - ❌ Never make direct API calls using `makeApiCall` in new code (use `lib/api/` modules)
+- ❌ Never treat one-off edits in integration/ or builder/ as the permanent fix for a bug—update the generator or template that produces the file
 - ❌ Never skip type definitions for API request/response types
 - ❌ Never log authentication tokens or secrets in API calls
 

package/README.md

@@ -1,9 +1,13 @@
-# AI Fabrix - Builder SDK
+# AI Fabrix - Builder
 
 [![npm version](https://img.shields.io/npm/v/@aifabrix/builder.svg)](https://www.npmjs.com/package/@aifabrix/builder)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Local development infrastructure + Azure deployment tool.
+Install the AI Fabrix platform and test it locally. Then add external integrations or build your own applications.
+
+← **Full documentation:** [docs/README.md](docs/README.md) (table of contents for all guides)
+
+---
 
 ## Install
 
@@ -11,126 +15,86 @@ Local development infrastructure + Azure deployment tool.
 npm install -g @aifabrix/builder
 ```
 
-## Quick Start
+**Alias:** You can use `aifx` instead of `aifabrix` in any command.
 
-```bash
-aifabrix up              # Start Postgres + Redis
-aifabrix create myapp    # Create your app
-aifabrix build myapp     # Build Docker image
-aifabrix run myapp       # Run locally
-# Stop the app (optionally remove its data volume)
-aifabrix down myapp
-# aifabrix down myapp --volumes
-```
+---
 
-→ [Full Guide](docs/quick-start.md) | [CLI Commands](docs/cli-reference.md)
+## Goal 1: Start and test the AI Fabrix platform
 
-## What You Get
+Get the platform running locally so you can try it.
 
-- **Local Postgres + Redis infrastructure** - Runs in Docker
-- **Auto-generated Dockerfiles** - TypeScript and Python templates
-- **Environment variable management** - Secret resolution with kv:// references
-- **Azure deployment pipeline** - Push to ACR and deploy via controller
+1. **Start local infrastructure** (Postgres, Redis, optional Traefik):
 
-## Optional Platform Apps
+   ```bash
+   aifabrix up-infra
+   ```
 
-Want authentication or deployment controller?
+2. **Start the platform** (Keycloak, Miso Controller, Dataplane) from community images:
 
-**Quick install from images (no build):**
-```bash
-aifabrix up              # Start Postgres + Redis first
-aifabrix up-miso         # Install Keycloak + Miso Controller from images (auto-generated secrets for testing)
-```
+   ```bash
+   aifabrix up-platform
+   ```
 
-**Or create and build from templates:**
-```bash
-# Keycloak for authentication
-aifabrix create keycloak --port 8082 --database --template keycloak
-aifabrix build keycloak
-aifabrix run keycloak
-
-# Miso Controller for Azure deployments
-aifabrix create miso-controller --port 3000 --database --redis --template miso-controller
-aifabrix build miso-controller
-aifabrix run miso-controller
-```
+   Or run platform apps separately: `aifabrix up-miso` then `aifabrix up-dataplane`. Infra must be up first.
 
-**Dataplane in dev (after login):**
-```bash
-aifabrix login --environment dev
-aifabrix up-dataplane    # Register or rotate, run, and deploy dataplane in dev
-```
+3. **Configure secrets** – You need either **OpenAI** or **Azure OpenAI**:
+
+   - **OpenAI:** set your API key:
+     ```bash
+     aifabrix secrets set secrets-openaiApiKeyVault <your-openai-secret-key>
+     ```
+   - **Azure OpenAI:** set endpoint and API key:
+     ```bash
+     aifabrix secrets set azure-openaiapi-urlKeyVault <your-azure-openai-endpoint-url>
+     aifabrix secrets set secrets-azureOpenaiApiKeyVault <your-azure-openai-secret-key>
+     ```
+
+Secrets are stored in `~/.aifabrix/secrets.local.yaml` or the file from `aifabrix-secrets` in your config (e.g. `builder/secrets.local.yaml`).
+
+→ [Infrastructure guide](docs/infrastructure.md)
 
-→ [Infrastructure Guide](docs/infrastructure.md)
+---
+
+## Goal 2: External system integration
+
+Create and deploy an external system (e.g. HubSpot): wizard or manual setup, then validate and deploy.
+
+**Example: HubSpot**
+
+- Create: `aifabrix create hubspot-test --type external` (or `aifabrix wizard` for guided setup).
+- Configure auth and datasources under `integration/hubspot-test/`.
+- Validate: `aifabrix validate hubspot-test`
+- Deploy: `aifabrix deploy hubspot-test`
+
+→ [External systems guide](docs/external-systems.md) · [Wizard](docs/wizard.md)
+
+---
+
+## Goal 3: Build your own application
+
+Create, configure, and run your own AI Fabrix application locally or deploy it (create app → configure → build → run / deploy).
+
+→ [Your own applications](docs/your-own-applications.md)
+
+---
 
 ## Documentation
 
-- [Quick Start](docs/quick-start.md) - Get running in 5 minutes
-- [Infrastructure](docs/infrastructure.md) - What runs and why
-- [Configuration](docs/configuration.md) - Config file reference
-- [Building](docs/building.md) - Build process explained
-- [Running](docs/running.md) - Run apps locally
-- [Deploying](docs/deploying.md) - Deploy to Azure
-- [CLI Reference](docs/cli-reference.md) - All commands
-
-## How It Works
-
-1. **Infrastructure** - Minimal baseline (Postgres + Redis)
-2. **Create** - Generate config files for your app
-3. **Build** - Auto-detect runtime and build Docker image
-4. **Run** - Start locally, connected to infrastructure
-5. **Deploy** - Push to ACR and deploy via controller
-
-```mermaid
-%%{init: {
-  "theme": "base",
-  "themeVariables": {
-    "fontFamily": "Poppins, Arial Rounded MT Bold, Arial, sans-serif",
-    "fontSize": "16px",
-    "background": "#FFFFFF",
-    "primaryColor": "#F8FAFC",
-    "primaryTextColor": "#0B0E15",
-    "primaryBorderColor": "#E2E8F0",
-    "lineColor": "#E2E8F0",
-    "textColor": "#0B0E15",
-    "borderRadius": 16
-  },
-  "flowchart": {
-    "curve": "linear",
-    "nodeSpacing": 34,
-    "rankSpacing": 34,
-    "padding": 10
-  }
-}}%%
-
-flowchart TD
-
-%% =======================
-%% Styles
-%% =======================
-classDef base fill:#FFFFFF,color:#0B0E15,stroke:#E2E8F0,stroke-width:1.5px;
-classDef primary fill:#0062FF,color:#ffffff,stroke-width:0px;
-
-%% =======================
-%% Flow
-%% =======================
-Install[Install CLI]:::primary --> Up[Start Infrastructure]:::base
-Up --> Create[Create App]:::base
-Create --> Build[Build Image]:::base
-Build --> Run[Run Locally]:::base
-Run --> Deploy[Deploy to Azure]:::primary
-```
+All guides and references are listed in **[docs/README.md](docs/README.md)** (table of contents).
 
-## Development
+- [CLI Reference](docs/cli-reference.md) – All commands
+- [Infrastructure](docs/infrastructure.md) – What runs and why
+- [Configuration](docs/configuration.md) – Config files
 
-- **Tests**: `npm test` (runs via wrapper; handles known Jest/Node exit issues).
-- **Coverage**: `npm run test:coverage` — runs tests with coverage through the same wrapper. May take 3–5 minutes for the full suite. If the process exits with a signal after "Ran all test suites", the wrapper treats it as success and coverage is written to `coverage/`. Use `test:coverage:nyc` only if you need nyc-specific reporters.
+---
 
 ## Requirements
 
-- **Docker Desktop** - For running containers
-- **Node.js 18+** - For running the CLI
-- **Azure CLI** - For deploying to Azure (optional)
+- **Docker Desktop** – For running containers
+- **Node.js 18+** – For running the CLI
+- **Azure CLI** – For deploying to Azure (optional)
+
+---
 
 ## License
 

package/integration/hubspot/test.js

@@ -511,7 +511,7 @@ async function checkAppDirectory(appPath) {
  * @throws {Error} If required files are missing
  */
 async function validateRequiredFiles(appPath, entries) {
-  const requiredFiles = ['variables.yaml', 'env.template', 'README.md', 'deploy.sh', 'deploy.ps1'];
+  const requiredFiles = ['variables.yaml', 'env.template', 'README.md', 'deploy.js'];
   const missingFiles = [];
   for (const fileName of requiredFiles) {
     const filePath = path.join(appPath, fileName);

package/lib/api/wizard.api.js

@@ -311,7 +311,7 @@ async function testMcpConnection(dataplaneUrl, authConfig, serverUrl, token) {
 }
 
 /**
- * Get deployment documentation for a system
+ * Get deployment documentation for a system (from dataplane DB only)
  * GET /api/v1/wizard/deployment-docs/{systemKey}
  * @async
  * @function getDeploymentDocs
@@ -326,6 +326,28 @@ async function getDeploymentDocs(dataplaneUrl, authConfig, systemKey) {
   return await client.get(`/api/v1/wizard/deployment-docs/${systemKey}`);
 }
 
+/**
+ * Generate deployment documentation with variables.yaml and deploy JSON for better quality
+ * POST /api/v1/wizard/deployment-docs/{systemKey}
+ * Sends deployJson and variablesYaml in the request body so the dataplane can align README with the integration folder.
+ * @async
+ * @function postDeploymentDocs
+ * @param {string} dataplaneUrl - Dataplane base URL
+ * @param {Object} authConfig - Authentication configuration
+ * @param {string} systemKey - System key identifier
+ * @param {Object} [body] - Optional request body (WizardDeploymentDocsRequest)
+ * @param {Object} [body.deployJson] - Deploy JSON object (e.g. *-deploy.json content)
+ * @param {string} [body.variablesYaml] - variables.yaml file content as string
+ * @returns {Promise<Object>} Deployment documentation response (content, contentType, systemKey)
+ * @throws {Error} If request fails
+ */
+async function postDeploymentDocs(dataplaneUrl, authConfig, systemKey, body = null) {
+  const client = new ApiClient(dataplaneUrl, authConfig);
+  return await client.post(`/api/v1/wizard/deployment-docs/${systemKey}`, {
+    body: body || {}
+  });
+}
+
 /**
  * Get known wizard platforms from dataplane.
  * GET /api/v1/wizard/platforms
@@ -365,5 +387,6 @@ module.exports = {
   getPreview,
   testMcpConnection,
   getDeploymentDocs,
+  postDeploymentDocs,
   getWizardPlatforms
 };

package/lib/app/deploy.js

@@ -15,7 +15,7 @@ const yaml = require('js-yaml');
 const chalk = require('chalk');
 const pushUtils = require('../deployment/push');
 const logger = require('../utils/logger');
-const { detectAppType } = require('../utils/paths');
+const { detectAppType, getBuilderPath, getIntegrationPath } = require('../utils/paths');
 const { checkApplicationExists } = require('../utils/app-existence');
 const { loadDeploymentConfig } = require('./deploy-config');
 
@@ -219,15 +219,16 @@ function displayDeploymentResults(result) {
 }
 
 /**
- * Check if app is external and handle external deployment
+ * Check if app is external and handle external deployment.
+ * When options.type === 'external', forces deployment from integration/<app> (no app register needed).
  * @async
  * @function handleExternalDeployment
  * @param {string} appName - Application name
- * @param {Object} options - Deployment options
+ * @param {Object} options - Deployment options (type: 'external' to force integration/<app>)
  * @returns {Promise<Object|null>} Deployment result if external, null otherwise
  */
 async function handleExternalDeployment(appName, options) {
-  const { isExternal } = await detectAppType(appName);
+  const { isExternal } = await detectAppType(appName, options);
   if (isExternal) {
     const externalDeploy = require('../external-system/deploy');
     await externalDeploy.deployExternalSystem(appName, options);
@@ -317,6 +318,37 @@ async function executeStandardDeployment(appName, options) {
   }
 }
 
+/**
+ * Tries external deploy when builder/<app> does not exist but integration/<app> does.
+ * @async
+ * @param {string} appName - Application name
+ * @param {Object} options - Deployment options
+ * @returns {Promise<{usedExternalDeploy: boolean, result: Object|null}>}
+ */
+async function tryExternalDeployFallback(appName, options) {
+  const builderPath = getBuilderPath(appName);
+  const integrationPath = getIntegrationPath(appName);
+  let builderExists = false;
+  let integrationExists = false;
+  try {
+    await fs.access(builderPath);
+    builderExists = true;
+  } catch (e) {
+    if (e.code !== 'ENOENT') throw e;
+  }
+  try {
+    await fs.access(integrationPath);
+    integrationExists = true;
+  } catch (e) {
+    if (e.code !== 'ENOENT') throw e;
+  }
+  if (!builderExists && integrationExists) {
+    const fallbackResult = await handleExternalDeployment(appName, { ...options, type: 'external' });
+    if (fallbackResult) return { usedExternalDeploy: true, result: fallbackResult };
+  }
+  return { usedExternalDeploy: false, result: null };
+}
+
 /**
  * Deploys application to Miso Controller
  * Orchestrates manifest generation, key creation, and deployment
@@ -338,7 +370,7 @@ async function executeStandardDeployment(appName, options) {
  */
 async function deployApp(appName, options = {}) {
   let controllerUrl = null;
-  let usedExternalDeploy = false;
+  let usedExternalDeploy = options.type === 'external';
 
   try {
     if (!appName || typeof appName !== 'string' || appName.trim().length === 0) {
@@ -347,8 +379,12 @@ async function deployApp(appName, options = {}) {
     validateAppName(appName);
 
     const externalResult = await handleExternalDeployment(appName, options);
-    if (externalResult) {
-      return externalResult;
+    if (externalResult) return externalResult;
+
+    const fallback = await tryExternalDeployFallback(appName, options);
+    if (fallback.result) {
+      usedExternalDeploy = fallback.usedExternalDeploy;
+      return fallback.result;
     }
     usedExternalDeploy = false;
 

package/lib/app/display.js

@@ -50,7 +50,7 @@ function displayWebappSuccess(appName, config, envConversionMessage) {
 
   logger.log(chalk.green('\nNext steps:'));
   logger.log(chalk.white('1. Copy env.template to .env and fill in your values'));
-  logger.log(chalk.white('2. Run: aifabrix up'));
+  logger.log(chalk.white('2. Run: aifabrix up-infra'));
   logger.log(chalk.white('3. Run: aifabrix build ' + appName));
   logger.log(chalk.white('4. Run: aifabrix run ' + appName));
 }

package/lib/app/list.js

@@ -164,9 +164,11 @@ function displayApplications(applications, environment, controllerUrl) {
 
   logger.log(chalk.bold(`\n📱 ${header}:\n`));
   applications.forEach((app) => {
+    const isExternal = app.configuration?.type === 'external';
+    const externalIcon = isExternal ? '🔗 ' : '';
     const hasPipeline = app.configuration?.pipeline?.isActive ? '✓' : '✗';
     const urlAndPort = formatUrlAndPort(app);
-    logger.log(`${hasPipeline} ${chalk.cyan(app.key)} - ${app.displayName} (${app.status || 'unknown'})${urlAndPort}`);
+    logger.log(`${externalIcon}${hasPipeline} ${chalk.cyan(app.key)} - ${app.displayName} (${app.status || 'unknown'})${urlAndPort}`);
   });
   logger.log(chalk.gray('  To show details for an app: aifabrix app show <appKey>\n'));
 }

package/lib/app/run-helpers.js

@@ -177,7 +177,7 @@ async function checkPrerequisites(appName, appConfig, debug = false, skipInfraCh
     .map(([service, _]) => service);
 
   if (unhealthyServices.length > 0) {
-    throw new Error(`Infrastructure services not healthy: ${unhealthyServices.join(', ')}\nRun 'aifabrix up' first`);
+    throw new Error(`Infrastructure services not healthy: ${unhealthyServices.join(', ')}\nRun 'aifabrix up-infra' first`);
   }
   logger.log(chalk.green('✓ Infrastructure is running'));
 }

package/lib/build/index.js

@@ -200,7 +200,7 @@ async function postBuildTasks(appName, buildConfig) {
 }
 
 /**
- * Check if app is external type and handle accordingly
+ * External apps have no Docker image; deploy JSON is generated by aifabrix json.
  * @async
  * @param {string} appName - Application name
  * @returns {Promise<boolean>} True if external (handled), false if should continue
@@ -208,9 +208,8 @@ async function postBuildTasks(appName, buildConfig) {
 async function checkExternalAppType(appName) {
   const variables = await loadVariablesYaml(appName);
   if (variables.app && variables.app.type === 'external') {
-    const generator = require('../generator');
-    const jsonPath = await generator.generateDeployJson(appName);
-    logger.log(chalk.green(`✓ Generated deployment JSON: ${jsonPath}`));
+    logger.log(chalk.blue(`External system: ${appName}`));
+    logger.log(chalk.gray('To regenerate deployment JSON, run: aifabrix json ' + appName));
     return true;
   }
   return false;

package/lib/cli/index.js

@@ -0,0 +1,45 @@
+/**
+ * AI Fabrix Builder CLI Command Definitions
+ *
+ * This module wires all CLI command setup and re-exports setupCommands,
+ * validateCommand, and handleCommandError for backward compatibility.
+ *
+ * @fileoverview CLI entry for AI Fabrix Builder; command setup orchestration
+ * @author AI Fabrix Team
+ * @version 2.0.0
+ */
+
+const { validateCommand, handleCommandError } = require('../utils/cli-utils');
+const { setupAuthCommands } = require('./setup-auth');
+const { setupInfraCommands } = require('./setup-infra');
+const { setupAppCommands } = require('./setup-app');
+const { setupEnvironmentCommands } = require('./setup-environment');
+const { setupUtilityCommands } = require('./setup-utility');
+const { setupDevCommands } = require('./setup-dev');
+const { setupSecretsCommands } = require('./setup-secrets');
+const { setupExternalSystemCommands } = require('./setup-external-system');
+const { setupAppCommands: setupAppManagementCommands } = require('../commands/app');
+const { setupDatasourceCommands } = require('../commands/datasource');
+
+/**
+ * Sets up all CLI commands on the Commander program instance
+ * @param {Command} program - Commander program instance
+ */
+function setupCommands(program) {
+  setupInfraCommands(program);
+  setupAuthCommands(program);
+  setupAppCommands(program);
+  setupEnvironmentCommands(program);
+  setupAppManagementCommands(program);
+  setupDatasourceCommands(program);
+  setupUtilityCommands(program);
+  setupExternalSystemCommands(program);
+  setupDevCommands(program);
+  setupSecretsCommands(program);
+}
+
+module.exports = {
+  setupCommands,
+  validateCommand,
+  handleCommandError
+};