@ibm-cloud/cd-tools 1.13.1 → 1.14.0

package/README.md

@@ -35,19 +35,19 @@ brew install hashicorp/tap/terraform
 The tools are provided as an [npx](https://docs.npmjs.com/cli/commands/npx) command. [npx](https://docs.npmjs.com/cli/commands/npx) (Node Package Execute) is a utility provided with [Node.js](https://nodejs.org/) which automatically downloads a module and its dependencies, and runs it. To see the available commands, run `npx @ibm-cloud/cd-tools` on your command line.
 
 ```shell-session
-$ npx @ibm-cloud/cd-tools
+$ npx @ibm-cloud/cd-tools -h
 Usage: @ibm-cloud/cd-tools [options] [command]
 
-Tools for migrating Toolchains, Delivery Pipelines, and Git Repos and Issue Tracking projects.
+Tools and utilities for the IBM Cloud Continuous Delivery service and resources.
 
 Options:
   -V, --version                 output the version number
   -h, --help                    display help for command
 
 Commands:
-  copy-project-group [options]  Bulk migrate GitLab group projects
-  copy-toolchain [options]      Copies a toolchain, including tool integrations and Tekton pipelines, to another region or resource group
-  export-secrets [options]      Checks if you have any stored secrets in your toolchain or pipelines, and exports them to Secrets Manager
+  copy-project-group [options]  Copies all Git Repos and Issue Tracking projects in a group to another region.
+  copy-toolchain [options]      Copies a toolchain, including tool integrations and Tekton pipelines, to another region or resource group.
+  export-secrets [options]      Exports Toolchain stored secrets to a Secrets Manager instance
   help [command]                display help for command
 ```
 
@@ -88,6 +88,7 @@ Options:
   -g, --group-id <id>           The id of the group to copy from the source region (e.g. "1796019"), or the group name (e.g. "mygroup") for top-level groups. For sub-groups, a path
                                 is also allowed, e.g. "mygroup/subgroup"
   -n, --new-group-slug <slug>   (Optional) Destination group URL slug (single path segment, e.g. "mygroup-copy"). Must be unique. Group display name remains the same as source.
+  -v, --verbose                 Enable verbose output (debug logs + wait details)
   -h, --help                    display help for command
 ```
 
@@ -190,11 +191,45 @@ The command will output a collection of `.tf` files in the `terraform` directory
 
 ### Copying toolchains to a different account
 
-The `copy-toolchain` command copies a toolchain within an IBM Cloud account. However it is possible to copy a toolchain to a different account with a few extra steps. Note that any tool integrations that access services in the source account, such as [Secrets Manager](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-getting-started), [Event Notifications](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-getting-started), etc. are not supported for cross-account copying.
+The `copy-toolchain` command copies a toolchain within an IBM Cloud account. However it is possible to copy a toolchain to a different account with a few extra steps. Note that any tool integrations that access services in the source account, such as [Secrets Manager](https://cloud.ibm.com/docs/secrets-manager), [Event Notifications](https://cloud.ibm.com/docs/event-notifications), etc. are not supported for cross-account copying.
 1. Run the `copy-toolchain` command with the `-D, --dry-run` option to first generate the Terraform (.tf) files to a directory (See [Getting the Terraform code for a toolchain](#getting-the-terraform-code-for-a-toolchain)).
 2. Edit the `cd_toolchain.tf` file, replacing the `resource_group_id` with a valid resource group id in the target account. You can find the resource group id in the IBM Cloud console under [Manage > Account > Resource groups](https://cloud.ibm.com/account/resource-groups).
 3. Switch to the directory containing the Terraform files, and run `terraform init`, then `terraform apply`.
 4. When prompted for the API key, provide an API key for the target account you wish to copy the toolchain to.
 
+## export-secrets
+
+### Overview
+The `export-secrets` command copies secrets stored directly in your toolchain or Tekton pipeline into [Secrets Manager](https://cloud.ibm.com/docs/secrets-manager), and then updates the toolchain and pipeline to reference the secrets in Secrets Manager. The `copy-toolchain` command does not copy secrets stored directly in the toolchain or its Tekton pipeline environment properties or trigger properties, however [secret references](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-cd_data_security#cd_secrets_references) to secrets in a secret store such as [Secrets Manager](https://cloud.ibm.com/docs/secrets-manager) or [Key Protect](https://cloud.ibm.com/docs/key-protect) can be copied. The `export-secrets` command is useful for moving your secrets out before copying a toolchain. You can also use it to check whether a toolchain or its Tekton pipeline(s) contain any stored secrets. Storing secrets in a proper secret store like Secrets Manager is a recommended practice for added security.
+
+### Limitations
+1. The Secrets Manager instance must be in the account that owns the API key you'll be using.
+2. Only [arbitrary type](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-arbitrary-secrets) secrets are supported.
+3. If you opt to create a Secrets Manager tool integration while running the command, it will not automatically create an IAM authorization policy to allow the toolchain to read secrets from the Secrets Manager instance. If the tool integration in the toolchain shows an error status due to the missing authorization policy, you can click the **Create Authorization** button to create a default one.
+
+### Prerequisites
+- You must first provision a [Secrets Manager](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-create-instance) instance before running the command.
+- The API key you use must be from the same account as the Secrets Manager instance.
+- The API key must have IAM permission to read the toolchain and create secrets in the selected Secrets Manager instance.
+
+### Recommendations
+- After running the command, open the toolchain and verify that the tool integration shows a healthy status. If it shows an error status due to a missing authorization policy, you can reconfigure the tool integration and click the **Create Authorization** button to create a default one.
+- You can run the command as many times as you like until all secrets are exported.
+
+### Usage
+```shell-session
+$ npx @ibm-cloud/cd-tools export-secrets -h
+Usage: @ibm-cloud/cd-tools export-secrets [options]
+
+Exports Toolchain stored secrets to a Secrets Manager instance
+
+Options:
+  -c, --toolchain-crn <crn>  The CRN of the toolchain to check
+  -a, --apikey <api_key>     API key used to authenticate. Must have IAM permission to read toolchains and create secrets in Secrets Manager
+  --check                    (Optional) Checks and lists any stored secrets in your toolchain
+  -v, --verbose              (Optional) Increase log output
+  -h, --help                 display help for command
+```
+
 ## Test
 All test setup and usage instructions are documented in [test/README.md](./test/README.md).

package/cmd/copy-toolchain.js

@@ -85,7 +85,7 @@ async function main(options) {
 	const verbosity = options.quiet ? 0 : options.verbose ? 2 : 1;
 
 	logger.setVerbosity(verbosity);
-	if (LOG_DUMP) logger.createLogStream(`${LOGS_DIR}/copy-toolchain-${new Date().getTime()}.log`);
+	if (LOG_DUMP) logger.createLogStream(`${LOGS_DIR}/copy-toolchain-${TIME_SUFFIX}.log`);
 	logger.dump(`Options: ${JSON.stringify(options)}\n`);
 
 	let bearer;

package/cmd/direct-transfer.js

@@ -10,8 +10,10 @@
 import { Command } from 'commander';
 import axios from 'axios';
 import { writeFile } from 'fs/promises';
-import { COPY_PROJECT_GROUP_DESC, SOURCE_REGIONS } from '../config.js';
-import { getWithRetry } from './utils/requests.js';
+import { COPY_PROJECT_GROUP_DESC, SOURCE_REGIONS, BROKER_REGIONS } from '../config.js';
+import { getWithRetry, shouldFailover } from './utils/requests.js';
+import Papa from 'papaparse';
+import fs from 'fs';
 import { logger, LOG_STAGES } from './utils/logger.js';
 import { promptUserYesNo } from './utils/utils.js';
 
@@ -158,11 +160,59 @@ class GitLabClient {
     return out;
   }
 
+  async getGroupPlaceholderCsv(groupId) {
+    const response = await this.client.get(`/groups/${groupId}/placeholder_reassignments`);
+    return response.data;
+  }
+
+  async reassignGroupPlaceholder(groupId, form) {
+    const response = await this.client.postForm(`/groups/${groupId}/placeholder_reassignments`, form);
+    return response.data;
+  }
+
   async getGroup(groupId) {
     const response = await this.client.get(`/groups/${groupId}`);
     return response.data;
   }
 
+  async syncUser(syncData) {
+    const preferred = syncData?.destRegion;
+    const candidates = [
+      ...(preferred ? [preferred] : []),
+      ...BROKER_REGIONS,
+    ];
+
+    const seen = new Set();
+    const brokerRegions = candidates.filter(r => (seen.has(r) ? false : (seen.add(r), true)));
+
+    let lastErr;
+
+    for (const brokerRegion of brokerRegions) {
+      const url = `https://otc-github-consolidated-broker.${brokerRegion}.devops.cloud.ibm.com/git-user-sync`;
+      try {
+        const response = await this.client.post(url, syncData);
+        return response.data;
+      } catch (err) {
+        lastErr = err;
+
+        if (!shouldFailover(err)) throw err;
+
+        continue;
+      }
+    }
+
+    const status = lastErr?.response?.status;
+    const msg =
+      lastErr?.response?.data?.message ||
+      lastErr?.message ||
+      'Unknown error';
+
+    throw new Error(
+      `git-user-sync failed via all brokers (destRegion=${syncData?.destRegion}). ` +
+      `Last error: ${status || 'NO_RESPONSE'} - ${msg}`
+    );
+  }
+
   async createBulkImport(importData) {
     const response = await this.client.post('/bulk_imports', importData);
     return response.data;
@@ -487,6 +537,160 @@ async function handleBulkImportConflict({ destination, destUrl, sourceGroupFullP
   }
 }
 
+async function handlePlaceholderReassignments({ destination, options, destinationGroupPath }) {
+  const SRC_COL = 'Source user identifier';
+  const DEST_COL = 'GitLab username';
+
+  logger.print();
+  logger.info('Checking for placeholder users to reassign...', LOG_STAGES.info);
+
+  let destGroup;
+  try {
+    destGroup = await destination.getGroupByFullPath(destinationGroupPath);
+  } catch (e) {
+    logger.warn(
+      `Unable to look up destination group "${destinationGroupPath}" to process placeholder reassignment. Skipping.`,
+      LOG_STAGES.import
+    );
+    logger.debug(`Group lookup error: ${e?.message || e}`, LOG_STAGES.import);
+    return;
+  }
+
+  const destGroupId = destGroup?.id;
+  if (!destGroupId) {
+    logger.warn('Destination group ID not available. Skipping placeholder reassignment.', LOG_STAGES.import);
+    return;
+  }
+
+  const csvText = await logger.withSpinner(
+    () => destination.getGroupPlaceholderCsv(destGroupId),
+    'Fetching placeholder reassignment data...',
+    'Fetched placeholder reassignment data.',
+    LOG_STAGES.request
+  );
+
+  if (!csvText) {
+    logger.info('No placeholder reassignment support detected (or no placeholders). Skipping.', LOG_STAGES.import);
+    return;
+  }
+
+  const parsed = Papa.parse(csvText, { header: true, skipEmptyLines: true });
+  const rows = Array.isArray(parsed?.data) ? parsed.data.filter(Boolean) : [];
+
+  if (parsed?.errors?.length) {
+    logger.warn(`Placeholder CSV parse warnings: ${parsed.errors.length}`, LOG_STAGES.import);
+    if (options.verbose) {
+      parsed.errors.slice(0, 5).forEach(e => logger.log(`CSV parse warning: ${JSON.stringify(e)}`, LOG_STAGES.import, true));
+    }
+  }
+
+  if (!rows.length) {
+    logger.info('No placeholder users found. Skipping reassignment.', LOG_STAGES.import);
+    return;
+  }
+
+  if (!(SRC_COL in rows[0])) {
+    logger.warn(`Placeholder CSV missing expected column "${SRC_COL}". Skipping reassignment.`, LOG_STAGES.import);
+    if (options.verbose) logger.debug(`CSV headers: ${Object.keys(rows[0] || {}).join(', ')}`, LOG_STAGES.import);
+    return;
+  }
+
+  logger.info(`Found ${rows.length} placeholder record(s). Resolving users...`, LOG_STAGES.info);
+
+  const ids = [...new Set(rows.map(r => (r?.[SRC_COL] || '').trim()).filter(Boolean))];
+
+  if (!ids.length) {
+    logger.info('No valid placeholder user identifiers found. Skipping reassignment.', LOG_STAGES.import);
+    return;
+  }
+
+  const idToUsername = new Map();
+  const failedIds = [];
+
+  const resolveUsers = async () => {
+    for (let i = 0; i < ids.length; i++) {
+      const userId = ids[i];
+      logger.updateSpinnerMsg(`Resolving placeholder users (${i + 1}/${ids.length})...`);
+
+      try {
+        const resp = await destination.syncUser({
+          sourceRegion: options.sourceRegion,
+          destRegion: options.destRegion,
+          groupId: destGroupId,
+          userId,
+        });
+
+        const username = resp?.username;
+        if (username) {
+          idToUsername.set(userId, username);
+        } else {
+          failedIds.push(userId);
+          if (options.verbose) logger.warn(`User sync returned no username for userId=${userId}`, LOG_STAGES.import);
+        }
+      } catch (e) {
+        failedIds.push(userId);
+        logger.warn(`Failed to sync userId=${userId}: ${e?.message || e}`, LOG_STAGES.request);
+      }
+    }
+    return { resolved: idToUsername.size, total: ids.length };
+  };
+
+  const resolvedSummary = await logger.withSpinner(
+    resolveUsers,
+    `Resolving placeholder users (0/${ids.length})...`,
+    `Resolved placeholder users.`,
+    LOG_STAGES.request
+  );
+
+  logger.info(`Resolved ${resolvedSummary.resolved}/${resolvedSummary.total} user(s).`, LOG_STAGES.info);
+
+  if (idToUsername.size === 0) {
+    logger.warn('No users could be resolved. Skipping placeholder reassignment upload.', LOG_STAGES.info);
+    return;
+  }
+
+  let updatedCount = 0;
+  for (const r of rows) {
+    const id = (r?.[SRC_COL] || '').trim();
+    const username = idToUsername.get(id);
+    if (username) {
+      r[DEST_COL] = username;
+      updatedCount++;
+    }
+  }
+
+  logger.info(`Prepared reassignment CSV for ${updatedCount}/${rows.length} record(s).`, LOG_STAGES.setup);
+
+  const outPath = `groupPlaceholders.csv`;
+  const csvOut = Papa.unparse(rows);
+
+  await fs.promises.writeFile(outPath, csvOut, 'utf8');
+  logger.info(`Wrote placeholder reassignment CSV: ${outPath}`, LOG_STAGES.setup);
+
+  const csvConfig = { file: fs.createReadStream(outPath) };
+
+  const reassignRes = await logger.withSpinner(
+    () => destination.reassignGroupPlaceholder(destGroupId, csvConfig),
+    'Submitting placeholder reassignment...',
+    'Placeholder reassignment submitted.',
+    LOG_STAGES.request
+  );
+
+  if (!reassignRes) {
+    logger.warn('Placeholder reassignment endpoint not supported (or returned 404). Skipping.', LOG_STAGES.import);
+    return;
+  }
+
+  if (options.verbose) logger.debug(`Placeholder reassignment response: ${JSON.stringify(reassignRes)}`, LOG_STAGES.import);
+
+  if (failedIds.length) {
+    logger.warn(`Some users could not be resolved: ${failedIds.length}/${ids.length}`, LOG_STAGES.import);
+    if (options.verbose) failedIds.slice(0, 20).forEach(id => logger.log(`Unresolved userId: ${id}`, LOG_STAGES.import, true));
+  } else {
+    logger.success('✔ Placeholder reassignment completed.', LOG_STAGES.info);
+  }
+}
+
 async function directTransfer(options) {
   const sourceUrl = validateAndConvertRegion(options.sourceRegion);
   const destUrl = validateAndConvertRegion(options.destRegion);
@@ -695,6 +899,8 @@ async function directTransfer(options) {
       logger.info(`${summary.entityFailed} entities failed to copy`, LOG_STAGES.import);
       if (newGroupUrl) logger.info(`New group URL: ${newGroupUrl}`, LOG_STAGES.import);
 
+      await handlePlaceholderReassignments({destination, options, destinationGroupPath});
+
       // show failed list only in verbose (or if failures exist)
       if (summary.entityFailed > 0) {
         logger.print();

package/cmd/utils/import-terraform.js

@@ -15,9 +15,12 @@ import { jsonToTf } from 'json-to-tf';
 import { getPipelineData, getToolchainTools } from './requests.js';
 import { runTerraformPlanGenerate, setTerraformEnv } from './terraform.js';
 import { getRandChars, isSecretReference, normalizeName } from './utils.js';
+import { logger } from './logger.js';
 
 import { SECRET_KEYS_MAP, SUPPORTED_TOOLS_MAP } from '../../config.js';
 
+const DEBUG_MODE = process.env['DEBUG_MODE'] === 'true'; // when true, log extra errors for debugging
+
 export async function importTerraform(token, apiKey, region, toolchainId, toolchainName, dir, isCompact, verbosity) {
     // STEP 1/2: set up terraform file with import blocks
     const importBlocks = []; // an array of objects representing import blocks, used in importBlocksToTf
@@ -157,7 +160,7 @@ export async function importTerraform(token, apiKey, region, toolchainId, toolch
 
     // STEP 2/2: run terraform import and post-processing
     setTerraformEnv(apiKey, verbosity);
-    await runTerraformPlanGenerate(dir, 'generated/draft.tf').catch(() => { }); // temp fix for errors due to bugs in the provider
+    await runTerraformPlanGenerate(dir, 'generated/draft.tf').catch((err) => { DEBUG_MODE && logger.dump(`\n[DEBUG_MODE=true] Draft errors: ${err}`) }); // temp fix for errors before post-processing
 
     const generatedFile = fs.readFileSync(`${dir}/generated/draft.tf`);
     const generatedFileJson = await tfToJson('draft.tf', generatedFile.toString());

package/cmd/utils/requests.js

@@ -502,6 +502,16 @@ async function getWithRetry(client, path, params = {}, { retries = 3, retryDelay
     throw lastError;
 }
 
+function shouldFailover(err) {
+  if (!err?.response) return true;
+  const status = err.response.status;
+  if (status >= 500) return true;
+  if (status === 404) return true;
+  if (status === 401) return true;
+
+  return false;
+}
+
 export {
     getBearerToken,
     getAccountId,
@@ -521,5 +531,6 @@ export {
     createTool,
     getSmInstances,
     migrateToolchainSecrets,
-    getWithRetry
+    getWithRetry,
+    shouldFailover
 }

package/config.js

@@ -55,6 +55,20 @@ const TARGET_REGIONS = [
 	'us-south'
 ];
 
+const BROKER_REGIONS = [
+  	'au-syd',
+	'br-sao',
+	'ca-mon',
+	'ca-tor',
+	'eu-de',
+	'eu-es',
+	'eu-gb',
+	'jp-osa',
+	'jp-tok',
+	'us-east',
+	'us-south'
+];
+
 const TERRAFORM_REQUIRED_VERSION = '1.13.3';
 
 // see https://docs.gitlab.com/user/reserved_names/
@@ -232,6 +246,7 @@ export {
 	DOCS_URL,
 	SOURCE_REGIONS,
 	TARGET_REGIONS,
+	BROKER_REGIONS,
 	TERRAFORM_REQUIRED_VERSION,
 	RESERVED_GRIT_PROJECT_NAMES,
 	RESERVED_GRIT_GROUP_NAMES,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ibm-cloud/cd-tools",
-  "version": "1.13.1",
+  "version": "1.14.0",
   "description": "Tools and utilities for the IBM Cloud Continuous Delivery service and resources",
   "repository": {
     "type": "git",
@@ -25,6 +25,7 @@
     "commander": "^14.0.1",
     "json-to-tf": "^0.3.1",
     "ora": "^9.0.0",
+    "papaparse": "^5.5.3",
     "strip-ansi": "^7.1.2"
   },
   "bin": {