ec2-instance-running-scheduler 0.1.9 → 0.2.0

package/README.md

@@ -4,22 +4,23 @@
 [![npm](https://img.shields.io/npm/v/ec2-instance-running-scheduler?style=flat-square)](https://www.npmjs.com/package/ec2-instance-running-scheduler)
 [![GitHub Workflow Status (branch)](https://img.shields.io/github/actions/workflow/status/gammarers-aws-cdk-constructs/ec2-instance-running-scheduler/release.yml?branch=main&label=release&style=flat-square)](https://github.com/gammarers-aws-cdk-constructs/ec2-instance-running-scheduler/actions/workflows/release.yml)
 [![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/gammarers-aws-cdk-constructs/ec2-instance-running-scheduler?sort=semver&style=flat-square)](https://github.com/gammarers-aws-cdk-constructs/ec2-instance-running-scheduler/releases)
-
 [![View on Construct Hub](https://constructs.dev/badge?package=ec2-instance-running-scheduler)](https://constructs.dev/packages/ec2-instance-running-scheduler)
 
-AWS CDK construct that starts and stops EC2 instances on a cron schedule using **EventBridge Scheduler** and a **Durable Execution Lambda**. The handler discovers instances with the **Resource Groups Tagging API**, runs start/stop and **polls until the instance reaches the target state** (durable `step` / `wait`), processes **multiple instances in parallel** (bounded concurrency), and posts **Slack** summary and per-instance thread messages using a secret from **Secrets Manager**. The Lambda emits **structured application logs** (invocation, EC2 transitions, Slack steps, completion) alongside JSON platform logs.
+AWS CDK construct library that starts and stops EC2 instances on a cron schedule using **EventBridge Scheduler** and a **Durable Execution Lambda**. The handler discovers instances with the **Resource Groups Tagging API**, issues start/stop, **polls until each instance reaches a stable target state** (durable `step` / `wait`), processes **multiple instances in parallel** (bounded concurrency), and posts **Slack** summary and per-instance thread messages using a secret from **Secrets Manager**. The Lambda emits **structured application logs** alongside JSON platform logs.
 
 ## Features
 
 - **Tag-based targeting** – Select EC2 instances by tag key and values (e.g. `Schedule` / `YES`) via `tag:GetResources`.
 - **EventBridge Scheduler** – Separate cron rules for start and stop, with per-rule timezone (`aws-cdk-lib` `TimeZone`).
 - **Durable Lambda** – One Lambda with AWS Lambda Durable Execution (`step`, `wait`, `map`, child contexts per instance) for long-running workflows without Step Functions.
-- **Stable-state polling** – After start/stop, the function waits and re-describes instances until `running` (start mode) or `stopped` (stop mode), or until a terminal error.
-- **Slack notifications** – Required for a successful run: parent message plus threaded updates per instance; credentials come from Secrets Manager (`token` and `channel` JSON). The secret name is passed as **`SLACK_SECRET_NAME`** on the function (from `secrets.slackSecretName`) and validated at runtime via **safe-env-getter**.
-- **Structured logging** – The handler uses the durable execution **`ctx.logger`** for traceable JSON application logs (e.g. invocation, describe/start/stop/wait loops, target count, Slack posts, errors before instance state failures).
+- **Stable-state polling** – After start/stop, the function waits (20 seconds between attempts) and re-describes instances until `running` (start mode) or `stopped` (stop mode).
+- **Configurable polling limits** – Per-instance **max loop count** and **max elapsed time** via `resourcePolling` (default: 90 loops / 1800 seconds). Failures use explicit `ResourcePollingFailed:*` messages instead of running until the Durable execution timeout (construct default: 2 hours).
+- **Validated environment variables** – The bundled handler parses required and polling env vars at invocation using strict decimal integer rules; polling limits must be **positive integers** (`> 0`).
+- **Slack notifications** – Parent message plus threaded updates per instance; credentials from Secrets Manager JSON (`token`, `channel`). The construct sets **`SLACK_SECRET_NAME`** on the function.
+- **Structured logging** – Durable execution **`ctx.logger`** for traceable JSON application logs (invocation, describe/start/stop/wait loops, polling limit errors, Slack steps, completion).
 - **Scheduling toggle** – Enable or disable both schedules without removing the stack (`enableScheduling`).
 - **Configurable schedules** – Optional cron overrides for start and stop (`minute`, `hour`, `week`, `timezone`); sensible defaults if omitted.
-- **IAM and observability** – EC2 and tagging API permissions, Slack secret read grant, **Parameters and Secrets Lambda Extension** (configured on the function for secret access patterns), JSON logging with configurable system/application levels, and a dedicated log group (construct defaults).
+- **IAM and observability** – EC2 and tagging API permissions, Slack secret read grant, **Parameters and Secrets Lambda Extension**, JSON logging, and a dedicated log group (construct defaults).
 
 ## Installation
 
@@ -74,10 +75,14 @@ new EC2InstanceRunningScheduler(stack, 'EC2InstanceRunningScheduler', {
     week: 'MON-FRI',
   },
   enableScheduling: true,
+  resourcePolling: {
+    maxLoopCount: 120,
+    maxElapsedSeconds: 3600,
+  },
 });
 ```
 
-Use the **stack** `EC2InstanceRunningScheduleStack` when deploying the scheduler as its own stack. It accepts the **same scheduler options** as the construct (plus standard `StackProps` such as `env`).
+Use the **stack** `EC2InstanceRunningScheduleStack` when deploying the scheduler as its own stack. It accepts the same **targeting, schedules, secrets, and enable flag** as the construct (plus standard `StackProps` such as `env`). For **`resourcePolling`**, use the construct directly or extend the stack in your app.
 
 ```typescript
 import * as cdk from 'aws-cdk-lib';
@@ -110,19 +115,32 @@ new EC2InstanceRunningScheduleStack(app, 'EC2InstanceRunningScheduleStack', {
 });
 ```
 
-EventBridge Scheduler invokes the Lambda with `Params.TagKey`, `Params.TagValues`, and `Params.Mode` (`Start` or `Stop`); the construct wires this for you. The function environment includes **`SLACK_SECRET_NAME`** set to `secrets.slackSecretName`.
+EventBridge Scheduler invokes the Lambda with `Params.TagKey`, `Params.TagValues`, and `Params.Mode` (`Start` or `Stop`); the construct wires this for you. The function environment includes:
+
+| Variable | Source | Purpose |
+|----------|--------|---------|
+| `SLACK_SECRET_NAME` | `secrets.slackSecretName` | Secrets Manager secret for Slack (required) |
+| `PROCESS_RESOURCE_MAX_LOOP_COUNT` | `resourcePolling.maxLoopCount` (default `90`) | Max describe/poll iterations per instance |
+| `PROCESS_RESOURCE_MAX_ELAPSED_SECONDS` | `resourcePolling.maxElapsedSeconds` (default `1800`) | Max wall-clock seconds polling one instance |
+
+When you set polling limits via `resourcePolling`, the construct writes them as decimal integer strings. If you override these variables manually, each value must be a **strict decimal integer** (e.g. `"120"` is valid; `"0x10"`, `"3.14"`, or `"10abc"` are rejected) and **greater than zero**. Invalid or missing `SLACK_SECRET_NAME` causes the handler to fail at the start of an invocation.
 
 ## Options
 
-These options apply to **`EC2InstanceRunningScheduler`** and to **`EC2InstanceRunningScheduleStack`** (stack props include them alongside `StackProps`).
+### EC2InstanceRunningScheduler
 
 | Option | Type | Required | Description |
 |--------|------|----------|-------------|
 | `targetResource` | `TargetResource` | Yes | Tag key and values used to select EC2 instances. |
-| `secrets` | `Secrets` | Yes | Identifies the Secrets Manager secret used for Slack (`slackSecretName`). |
+| `secrets` | `Secrets` | Yes | Secrets Manager secret for Slack (`slackSecretName`). |
 | `startSchedule` | `Schedule` | No | Cron for starting instances (default: `50 7 ? * MON-FRI *` in `Etc/UTC`). |
 | `stopSchedule` | `Schedule` | No | Cron for stopping instances (default: `5 19 ? * MON-FRI *` in `Etc/UTC`). |
 | `enableScheduling` | `boolean` | No | Whether both scheduler rules are enabled (default: `true`). |
+| `resourcePolling` | `ResourcePollingLimits` | No | Per-instance polling caps (see below). |
+
+### EC2InstanceRunningScheduleStack
+
+Includes `targetResource`, `secrets`, `startSchedule`, `stopSchedule`, `enableScheduling`, and standard `StackProps`. Does **not** expose `resourcePolling`; use `EC2InstanceRunningScheduler` when you need custom polling limits.
 
 ### TargetResource
 
@@ -138,13 +156,22 @@ These options apply to **`EC2InstanceRunningScheduler`** and to **`EC2InstanceRu
 
 ### Secrets
 
-- `slackSecretName` – Name of the AWS Secrets Manager secret. The Lambda expects JSON with **`token`** (Slack bot token) and **`channel`** (channel ID or name for `chat.postMessage`). The construct sets **`SLACK_SECRET_NAME`** on the function to this value; the handler reads and validates it at runtime (via **safe-env-getter**).
+- `slackSecretName` – Name of the AWS Secrets Manager secret. The Lambda expects JSON with **`token`** (Slack bot token) and **`channel`** (channel ID or name for `chat.postMessage`).
+
+### ResourcePollingLimits
+
+Written to `PROCESS_RESOURCE_MAX_LOOP_COUNT` and `PROCESS_RESOURCE_MAX_ELAPSED_SECONDS` on the running scheduler Lambda.
+
+- `maxLoopCount` – Maximum describe/poll loop iterations per instance (default: **90**). Must be a positive integer when set.
+- `maxElapsedSeconds` – Maximum wall-clock seconds spent polling one instance (default: **1800**, 30 minutes). Must be a positive integer when set.
+
+When a limit is exceeded during polling, the handler throws an error with prefix `ResourcePollingFailed:` (`MaxLoopCountExceeded`, `MaxElapsedTimeExceeded`, or `UnexpectedInstanceState` for unknown EC2 states).
 
 ## Requirements
 
 - **Node.js** ≥ 20.0.0 (for developing or synthesizing CDK apps that depend on this package).
 - **aws-cdk-lib** ^2.232.0 and **constructs** ^10.5.1 (peer dependencies).
-- **AWS** – EventBridge Scheduler; Lambda with **Durable Execution** (requires **Node.js 22+** in the deployment region—runtime is the latest Node.js available in the region per the construct), a **live alias**, **Parameters and Secrets Lambda Extension**; EC2 (describe/start/stop); Resource Groups Tagging API; Secrets Manager. The deployed function uses **arm64**, Durable Execution–compatible IAM and settings, and a bundled handler that loads secrets via **aws-lambda-secret-fetcher**.
+- **AWS** – EventBridge Scheduler; Lambda with **Durable Execution** (Node.js **24.x** runtime in the construct; Durable Execution requires a supported Node.js runtime in your region), a **live alias**, **Parameters and Secrets Lambda Extension**; EC2 (`DescribeInstances`, `StartInstances`, `StopInstances`); Resource Groups Tagging API (`tag:GetResources`); Secrets Manager. The deployed function uses **arm64**, Durable Execution IAM policies, a 2-hour Durable execution timeout (construct default), and a bundled handler that loads secrets via **aws-lambda-secret-fetcher**.
 
 ## License
 

package/assets/funcs/running-scheduler.lambda/index.js

@@ -20017,7 +20017,22 @@ var require_lib3 = __commonJS({
     "use strict";
     Object.defineProperty(exports2, "__esModule", { value: true });
     exports2.SafeEnvGetter = exports2.SafeEnvType = exports2.SafeEnvGetterValidationError = exports2.SafeEnvGetterError = void 0;
+    var STRICT_INTEGER_PATTERN = /^-?\d+$/;
+    var parseStrictInteger = (raw) => {
+      const trimmed = raw.trim();
+      if (trimmed === "" || !STRICT_INTEGER_PATTERN.test(trimmed)) {
+        return void 0;
+      }
+      const n = Number(trimmed);
+      if (!Number.isFinite(n) || !Number.isInteger(n)) {
+        return void 0;
+      }
+      return n;
+    };
     var SafeEnvGetterError = class extends Error {
+      /**
+       * @param message - Error message.
+       */
       constructor(message) {
         super(message);
         this.name = "SafeEnvGetterError";
@@ -20027,6 +20042,10 @@ var require_lib3 = __commonJS({
     var SafeEnvGetterValidationError = class _SafeEnvGetterValidationError extends SafeEnvGetterError {
       /**
        * Formats validation errors into a human-readable error message.
+       *
+       * @template K - Environment variable key type.
+       * @param errors - Validation error entries to format.
+       * @returns Multi-line summary listing each key and message.
        */
       static format(errors) {
         const lines = errors.map((e) => `- ${e.key}: ${e.message}${e.raw == null ? "" : ` (raw="${e.raw}")`}`);
@@ -20035,6 +20054,8 @@ ${lines.join("\n")}`;
       }
       /**
        * Creates a new validation error from one or more `SafeEnvError` entries.
+       *
+       * @param errors - One or more structured validation errors.
        */
       constructor(errors) {
         const msg = _SafeEnvGetterValidationError.format(errors);
@@ -20046,9 +20067,12 @@ ${lines.join("\n")}`;
     };
     exports2.SafeEnvGetterValidationError = SafeEnvGetterValidationError;
     exports2.SafeEnvType = {
-      /** Spec for a string value. */
+      /** Spec for a string value (returned as-is). */
       String: { type: "string" },
-      /** Spec for a numeric value. */
+      /**
+       * Spec for a strict decimal integer.
+       * Rejects whitespace-only, hex, `Infinity`, `NaN`, and non-integer forms.
+       */
       Number: { type: "number" },
       /** Spec for a boolean value (1/true/yes/on → true). */
       Boolean: { type: "boolean" },
@@ -20073,8 +20097,8 @@ ${lines.join("\n")}`;
       }
       switch (spec.type) {
         case "number": {
-          const n = Number(raw);
-          if (Number.isNaN(n)) {
+          const n = parseStrictInteger(raw);
+          if (n === void 0) {
             throw new SafeEnvGetterValidationError([
               { key, message: `Env ${key}: expected number, got "${raw}"`, raw, kind: "invalid_number" }
             ]);
@@ -20113,8 +20137,8 @@ ${lines.join("\n")}`;
           continue;
         }
         if (spec.type === "number") {
-          const n = Number(raw);
-          if (Number.isNaN(n)) {
+          const n = parseStrictInteger(raw);
+          if (n === void 0) {
             errors.push({ key, message: `Env ${key}: expected number, got "${raw}"`, raw, kind: "invalid_number" });
             continue;
           }
@@ -20145,7 +20169,9 @@ ${lines.join("\n")}`;
       return envs;
     };
     exports2.SafeEnvGetter = {
+      /** Reads and parses a single environment variable. See {@link getEnv}. */
       getEnv,
+      /** Reads and parses multiple environment variables. See {@link getEnvs}. */
       getEnvs
     };
   }
@@ -23677,10 +23703,73 @@ var import_client_ec2 = require("@aws-sdk/client-ec2");
 var import_client_resource_groups_tagging_api = require("@aws-sdk/client-resource-groups-tagging-api");
 var import_web_api = __toESM(require_dist5());
 var import_aws_lambda_secret_fetcher = __toESM(require_lib2());
+var import_safe_env_getter2 = __toESM(require_lib3());
+
+// src/funcs/running-scheduler-polling-env.ts
 var import_safe_env_getter = __toESM(require_lib3());
 
+// src/funcs/running-scheduler-polling-config.ts
+var PROCESS_RESOURCE_MAX_LOOP_COUNT_ENV = "PROCESS_RESOURCE_MAX_LOOP_COUNT";
+var PROCESS_RESOURCE_MAX_ELAPSED_SECONDS_ENV = "PROCESS_RESOURCE_MAX_ELAPSED_SECONDS";
+
 // src/funcs/running-scheduler-predicates.ts
+var DEFAULT_RESOURCE_POLLING_LIMITS = {
+  maxLoopCount: 90,
+  maxElapsedSeconds: 1800
+};
 var isDesiredStableState = (mode, currentState) => mode === "Start" && currentState === "running" || mode === "Stop" && currentState === "stopped";
+var isTransitioningState = (mode, currentState) => mode === "Start" && currentState === "pending" || mode === "Stop" && (currentState === "stopping" || currentState === "shutting-down");
+var getPollingAbortReason = (loopCount, startedAtMs, nowMs, limits) => {
+  if (loopCount >= limits.maxLoopCount) {
+    return "MaxLoopCountExceeded";
+  }
+  const elapsedSeconds = (nowMs - startedAtMs) / 1e3;
+  if (elapsedSeconds >= limits.maxElapsedSeconds) {
+    return "MaxElapsedTimeExceeded";
+  }
+  return void 0;
+};
+var formatResourcePollingFailure = (reason, context) => {
+  const common = `identifier=${context.identifier} mode=${context.mode} currentState=${context.currentState} loopCount=${context.loopCount}`;
+  if (reason === "MaxLoopCountExceeded") {
+    return `ResourcePollingFailed:MaxLoopCountExceeded: ${common} maxLoopCount=${context.limits.maxLoopCount}`;
+  }
+  if (reason === "MaxElapsedTimeExceeded") {
+    return `ResourcePollingFailed:MaxElapsedTimeExceeded: ${common} maxElapsedSeconds=${context.limits.maxElapsedSeconds}`;
+  }
+  return `ResourcePollingFailed:UnexpectedInstanceState: ${common}`;
+};
+
+// src/funcs/running-scheduler-polling-env.ts
+var assertPositiveEnvInt = (key, value) => {
+  if (value <= 0) {
+    const raw = process.env[key];
+    throw new Error(`Invalid ${key}: must be a positive integer (got "${raw ?? ""}")`);
+  }
+  return value;
+};
+var parseResourcePollingLimitsFromEnv = () => {
+  const parsed = import_safe_env_getter.SafeEnvGetter.getEnvs({
+    [PROCESS_RESOURCE_MAX_LOOP_COUNT_ENV]: [
+      import_safe_env_getter.SafeEnvType.Number,
+      { default: DEFAULT_RESOURCE_POLLING_LIMITS.maxLoopCount }
+    ],
+    [PROCESS_RESOURCE_MAX_ELAPSED_SECONDS_ENV]: [
+      import_safe_env_getter.SafeEnvType.Number,
+      { default: DEFAULT_RESOURCE_POLLING_LIMITS.maxElapsedSeconds }
+    ]
+  });
+  return {
+    maxLoopCount: assertPositiveEnvInt(
+      PROCESS_RESOURCE_MAX_LOOP_COUNT_ENV,
+      parsed[PROCESS_RESOURCE_MAX_LOOP_COUNT_ENV]
+    ),
+    maxElapsedSeconds: assertPositiveEnvInt(
+      PROCESS_RESOURCE_MAX_ELAPSED_SECONDS_ENV,
+      parsed[PROCESS_RESOURCE_MAX_ELAPSED_SECONDS_ENV]
+    )
+  };
+};
 
 // src/funcs/running-scheduler.lambda.ts
 var STATE_LIST = [
@@ -23692,7 +23781,7 @@ var getStateDisplay = (current) => {
   const found = STATE_LIST.find((s) => s.state === current);
   return found ? { emoji: found.emoji, name: found.name } : void 0;
 };
-var processOneResource = async (ctx, targetResource, params, resourceIndex) => {
+var processOneResource = async (ctx, targetResource, params, resourceIndex, pollingLimits) => {
   const parts = targetResource.split("/");
   const identifier = parts[parts.length - 1] ?? "unknown";
   const arnParts = targetResource.split(":");
@@ -23704,11 +23793,36 @@ var processOneResource = async (ctx, targetResource, params, resourceIndex) => {
     identifier,
     region,
     account,
-    mode: params.Mode
+    mode: params.Mode,
+    maxLoopCount: pollingLimits.maxLoopCount,
+    maxElapsedSeconds: pollingLimits.maxElapsedSeconds
   });
+  const startedAtMs = await ctx.step(`${stepPrefix}-polling-started-at`, async () => Date.now());
   let loopCount = 0;
   let currentState = "";
   do {
+    const abortReason = await ctx.step(
+      `${stepPrefix}-poll-limit-check-${loopCount}`,
+      async () => getPollingAbortReason(loopCount, startedAtMs, Date.now(), pollingLimits)
+    );
+    if (abortReason) {
+      const message = formatResourcePollingFailure(abortReason, {
+        identifier,
+        mode: params.Mode,
+        currentState: currentState || "unknown",
+        loopCount,
+        limits: pollingLimits
+      });
+      ctx.logger.error("processOneResource: polling limit exceeded", {
+        identifier,
+        abortReason,
+        loopCount,
+        currentState,
+        maxLoopCount: pollingLimits.maxLoopCount,
+        maxElapsedSeconds: pollingLimits.maxElapsedSeconds
+      });
+      throw new Error(message);
+    }
     currentState = await ctx.step(`${stepPrefix}-describe-${loopCount}`, async () => {
       const ec2 = new import_client_ec2.EC2Client({});
       const out = await ec2.send(new import_client_ec2.DescribeInstancesCommand({ InstanceIds: [identifier] }));
@@ -23750,8 +23864,7 @@ var processOneResource = async (ctx, targetResource, params, resourceIndex) => {
       continue;
     }
     if (!isDesiredStableState(mode, currentState)) {
-      const transitioning = mode === "Start" && currentState === "pending" || mode === "Stop" && (currentState === "stopping" || currentState === "shutting-down");
-      if (transitioning) {
+      if (isTransitioningState(mode, currentState)) {
         ctx.logger.info("processOneResource: wait while transitioning", {
           identifier,
           loopCount,
@@ -23763,13 +23876,20 @@ var processOneResource = async (ctx, targetResource, params, resourceIndex) => {
         loopCount += 1;
         continue;
       }
+      const message = formatResourcePollingFailure("UnexpectedInstanceState", {
+        identifier,
+        mode,
+        currentState,
+        loopCount,
+        limits: pollingLimits
+      });
       ctx.logger.error("processOneResource: unexpected state", {
         identifier,
         mode,
         currentState,
         loopCount
       });
-      throw new Error(`instance status fail: mode=${mode} currentState=${currentState}`);
+      throw new Error(message);
     }
   } while (!isDesiredStableState(params.Mode, currentState));
   ctx.logger.info("processOneResource: reached desired stable state", {
@@ -23796,7 +23916,8 @@ var handler = withDurableExecution(async (event, ctx) => {
   if (!params?.TagKey || !params?.TagValues || !params?.Mode) {
     throw new Error("Invalid event: Params.TagKey, Params.TagValues, Params.Mode are required.");
   }
-  const slackSecretName = import_safe_env_getter.SafeEnvGetter.getEnv("SLACK_SECRET_NAME");
+  const pollingLimits = parseResourcePollingLimitsFromEnv();
+  const slackSecretName = import_safe_env_getter2.SafeEnvGetter.getEnv("SLACK_SECRET_NAME");
   const slackSecretValue = await ctx.step("fetch-slack-secret", async () => {
     ctx.logger.info("running-scheduler: fetching Slack secret", { secretName: slackSecretName });
     return import_aws_lambda_secret_fetcher.secretFetcher.getSecretValue(slackSecretName);
@@ -23847,7 +23968,7 @@ var handler = withDurableExecution(async (event, ctx) => {
     //   ),
     async (mapCtx, targetResource, index) => {
       return mapCtx.runInChildContext(`resource-${index}`, async (childCtx) => {
-        const result = await processOneResource(childCtx, targetResource, params, index);
+        const result = await processOneResource(childCtx, targetResource, params, index, pollingLimits);
         childCtx.logger.info("running-scheduler: posting thread Slack message", {
           index,
           identifier: result.identifier,