screwdriver-queue-service 5.0.2 → 6.0.0

package/config/default.yaml

@@ -172,8 +172,8 @@ queue:
 
 plugins:
   blockedBy:
-    # re-enqueue in 1 mins if blocked
-    reenqueueWaitTime: 1
+    # re-enqueue in 10 seconds if blocked (0.167 minutes)
+    reenqueueWaitTime: 0.167
     # job is blocking for maximum 120 mins = build timeout
     blockTimeout: 120
     # job blocked by itself

package/docs/ARCHITECTURE_REDESIGN.md

@@ -0,0 +1,214 @@
+# BlockedBy Architecture Redesign
+
+**Purpose**: Propose a cleaner, more maintainable architecture for the build start/stop Queue system (incl. BlockedBy, Collapse, Timeout)
+**Design Proposal:**
+
+---
+
+## Current Architecture - Problems
+
+### 1. **Complexity Analysis**
+
+#### **Current Issues**
+
+**A. State Scattered Across Redis Keys**
+```
+running_job_{jobId}         # Am I running?
+last_running_job_{jobId}    # Who ran last?
+waiting_job_{jobId}         # Who's waiting?
+deleted_{jobId}_{buildId}   # Am I aborted?
+buildConfigs[buildId]       # What's my config?
+timeoutConfigs[buildId]     # When do I timeout?
+```
+
+**Problem**: No single source of truth. State reconstruction requires 6+ Redis reads.
+
+**B. Implicit State Machine**
+```javascript
+// State is implicit in Redis key presence/absence
+if (runningKey exists) → RUNNING
+if (in waitingKey) → BLOCKED
+if (deleteKey exists) → ABORTED
+if (buildId < lastRunning && collapse) → COLLAPSED
+```
+
+**Problem**: State transitions scattered across code. Hard to reason about.
+
+**C. Race Conditions Everywhere**
+```javascript
+// Non-atomic check-then-act
+const value = await redis.get(key);  // Read
+if (value) {
+    await redis.set(otherKey, ...);  // Write
+}
+// Another worker can interleave here!
+```
+
+**Problem**: Locking added as band-aid. Real issue is non-atomic operations.
+
+**D. Mixed Concerns**
+```javascript
+async beforePerform() {
+    // Concern 1: Filtering (job ownership)
+    // Concern 2: Abort checking
+    // Concern 3: Collapse logic
+    // Concern 4: Blocking logic
+    // Concern 5: Queue management
+    // Concern 6: Lock management
+    // Concern 7: Status updates
+    // All in one 300-line function!
+}
+```
+
+**Problem**: Single Responsibility Principle violated. Hard to test, modify, understand.
+
+**E. Implicit Dependencies**
+```
+beforePerform → checkBlockingJob
+              → blockedBySelf
+              → collapseBuilds
+              → reEnqueue
+              → helper.updateBuildStatus
+```
+
+**Problem**: Deep call stack. Side effects hidden. Hard to trace execution flow.
+
+---
+
+## Proposed Architecture - Principles
+
+### **Core Principles**
+
+1. **Single Source of Truth** - One place holds authoritative state
+2. **Explicit State Machine** - States and transitions clearly defined
+3. **Atomic Operations** - Use Lua scripts or transactions
+4. **Separation of Concerns** - Each class has one responsibility
+5. **Immutable Events** - Events log what happened, states derive from events
+6. **Testability** - Pure functions, dependency injection, clear interfaces
+
+---
+
+## Redesign Proposal
+
+### **State Machine**
+
+#### **Architecture**
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                      Build Lifecycle                        │
+├─────────────────────────────────────────────────────────────┤
+│                                                             │
+│  Events (Immutable Log)        States (Derived)             │
+│  ────────────────────────      ──────────────────           │
+│  BuildEnqueued                 QUEUED                       │
+│  BuildBlocked                  BLOCKED                      │
+│  BuildUnblocked                READY                        │
+│  BuildStarted                  RUNNING                      │
+│  BuildCompleted                SUCCESS/FAILURE              │
+│  BuildAborted                  ABORTED                      │
+│  BuildCollapsed                COLLAPSED                    │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
+
+#### **Components**
+
+**1. LuaScriptLoader** - Loads and executes Lua scripts on Redis server
+
+**2. startBuild.lua** - Main script handling blocking/collapse/abort logic atomically
+
+**3. checkTimeout.lua** - Timeout detection and cleanup script
+
+**4. stopBuild.lua** - Main script for handling cleanup for stopped build
+
+**5. Helper Modules** (Pure Logic - No Redis calls):
+- **CollapseDecider.lua** - Build collapse logic
+- **DependencyResolver.lua** - Dependency blocking logic
+- **StateValidator.lua** - State transition validation
+- **TimeoutDecider.lua** - Timeout calculation logic
+
+### **Future Phase: Introduce Event Log**
+
+**Goal**: Add event sourcing for observability
+
+**Changes**:
+```
+1. Add Redis Streams for events
+   build:events:{buildId} → stream
+
+2. Record all state changes as events
+   - BuildEnqueued
+   - BuildBlocked
+   - BuildStarted
+   - BuildCompleted
+
+3. Keep existing state keys (dual-write)
+   - Events for debugging
+   - Keys for fast reads
+
+4. Build debugging tools (may be)
+   - Event replay UI
+   - State reconstruction
+```
+
+## Trade-offs & Decisions
+
+### **Lua Scripts**
+
+**Pros**:
+- Atomic operations
+- No locks needed
+- Better performance
+- Simpler code
+
+**Cons**:
+- Lua learning curve
+- Harder to debug
+- Can't use debugger
+
+**Decision**: YES
+- Lua simple (we write once)
+- Atomicity > debuggability
+- Can test Lua separately
+- Worth the complexity reduction
+
+### **Event Sourcing**
+
+**Pros**:
+- Full audit trail
+- Easy debugging
+- Can replay state
+- Append-only (fast)
+
+**Cons**:
+- More storage
+- Event schema versioning
+- Eventual consistency?
+
+**Decision**: NO (we can add in a later phase)
+- Redis Streams are cheap
+- Debugging builds is easier
+- Can keep short retention (7 days)
+
+---
+
+### **Architecture Diagram**
+
+![Architecture Diagram](./QS-REDIS-ATOMIC-REDESIGN.png)
+
+KEY BENEFITS:
+1 Redis roundtrip (was 6+)
+Zero race conditions (atomic execution)
+No distributed locks (eliminated Redlock)
+Modular design (reusable helper modules)
+
+---
+
+## Success Metrics
+
+- Redis roundtrips: **6+ reduced to 1** (single Lua script execution)
+- Lock contention: **Eliminated entirely** (no Redlock)
+- Race conditions: **Zero** (Lua atomicity guarantees)
+
+---

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "screwdriver-queue-service",
-  "version": "5.0.2",
+  "version": "6.0.0",
   "description": "Screwdriver Queue Service API",
   "main": "app.js",
   "directories": {
@@ -51,6 +51,7 @@
     "mocha-sonarqube-reporter": "^1.0.2",
     "mockery": "^2.1.0",
     "nyc": "^15.1.0",
+    "redis-memory-server": "^0.13.0",
     "sinon": "^15.0.0",
     "snyk": "^1.814.0",
     "util": "^0.12.5"

package/plugins/queue/scheduler.js

@@ -422,51 +422,14 @@ async function start(executor, config) {
             throw value.error;
         }
 
-        let buildUpdatePayload;
-
         if (isVirtualJob(annotations)) {
             // Bypass execution of the build if the job is virtual
-            buildUpdatePayload = {
+            const buildUpdatePayload = {
                 status: 'SUCCESS',
                 statusMessage: 'Skipped execution of the virtual job',
                 statusMessageType: 'INFO'
             };
-        } else {
-            const token = executor.tokenGen(
-                Object.assign(tokenConfig, { scope: ['temporal'] }),
-                TEMPORAL_TOKEN_TIMEOUT
-            );
 
-            // set the start time in the queue
-            Object.assign(config, { token });
-            // Store the config in redis
-            await executor.redisBreaker.runCommand('hset', executor.buildConfigTable, buildId, JSON.stringify(config));
-
-            const blockedBySameJob = reach(config, 'annotations>screwdriver.cd/blockedBySameJob', {
-                separator: '>',
-                default: true
-            });
-            const blockedBySameJobWaitTime = reach(config, 'annotations>screwdriver.cd/blockedBySameJobWaitTime', {
-                separator: '>',
-                default: BLOCKED_BY_SAME_JOB_WAIT_TIME
-            });
-
-            // Note: arguments to enqueue are [queue name, job name, array of args]
-            enq = await executor.queueBreaker.runCommand('enqueue', executor.buildQueue, 'start', [
-                {
-                    buildId,
-                    jobId,
-                    blockedBy: blockedBy.toString(),
-                    blockedBySameJob,
-                    blockedBySameJobWaitTime
-                }
-            ]);
-            if (buildStats) {
-                buildUpdatePayload = { stats: build.stats, status: 'QUEUED' };
-            }
-        }
-
-        if (buildUpdatePayload) {
             await helper
                 .updateBuild(
                     {
@@ -478,10 +441,68 @@ async function start(executor, config) {
                     helper.requestRetryStrategy
                 )
                 .catch(err => {
-                    logger.error(`Failed to update build status for build ${buildId}: ${err}`);
+                    logger.error(`Failed to update virtual build status for build ${buildId}: ${err}`);
 
                     throw err;
                 });
+        } else {
+            if (buildStats) {
+                await helper
+                    .updateBuild(
+                        {
+                            buildId,
+                            token: buildToken,
+                            apiUri,
+                            payload: { stats: build.stats, status: 'QUEUED' }
+                        },
+                        helper.requestRetryStrategy
+                    )
+                    .catch(err => {
+                        logger.error(`Failed to update build status to QUEUED for build ${buildId}: ${err}`);
+                        throw err;
+                    });
+            }
+
+            try {
+                const token = executor.tokenGen(
+                    Object.assign(tokenConfig, { scope: ['temporal'] }),
+                    TEMPORAL_TOKEN_TIMEOUT
+                );
+
+                // set the start time in the queue
+                Object.assign(config, { token });
+                // Store the config in redis
+                await executor.redisBreaker.runCommand(
+                    'hset',
+                    executor.buildConfigTable,
+                    buildId,
+                    JSON.stringify(config)
+                );
+
+                const blockedBySameJob = reach(config, 'annotations>screwdriver.cd/blockedBySameJob', {
+                    separator: '>',
+                    default: true
+                });
+                const blockedBySameJobWaitTime = reach(config, 'annotations>screwdriver.cd/blockedBySameJobWaitTime', {
+                    separator: '>',
+                    default: BLOCKED_BY_SAME_JOB_WAIT_TIME
+                });
+
+                // Note: arguments to enqueue are [queue name, job name, array of args]
+                enq = await executor.queueBreaker.runCommand('enqueue', executor.buildQueue, 'start', [
+                    {
+                        buildId,
+                        jobId,
+                        blockedBy: blockedBy.toString(),
+                        blockedBySameJob,
+                        blockedBySameJobWaitTime
+                    }
+                ]);
+            } catch (err) {
+                logger.error(`Redis enqueue failed for build ${buildId}: ${err}`);
+
+                throw err;
+            }
         }
     }