ai-inference-stepper 1.0.0

package/src/queue/How queue works in package stepper.md

@@ -0,0 +1,149 @@
+# Chat Conversation
+
+Note: _This is purely the output of the chat conversation and does not contain any raw data, codebase snippets, etc. used to generate the output._
+
+### User Input
+
+lets break down the explanation of  @[packages/stepper/src/queue/producer.ts] 
+ . i want to know what each functions means, how they work, possible results of it, why they work like that and how they help achieve what the inference stepper package is for.
+break down evey detail and explanton for me, to the lowest and clear enough for even a non programmer or technial person
+
+*Viewed [producer.ts](file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts) *
+
+*Viewed [index.ts](file:///home/blaze/mine/commitdiary/packages/stepper/src/index.ts) *
+
+### Planner Response
+
+This file acts as the **"Dispatcher"** or **"Order Taker"** for the system.
+
+Imagine a busy coffee shop. Making a complex latte takes time. If the cashier made the coffee for every customer right then and there, the line would never move.
+Instead, the cashier (Producer) takes the order, writes it on a cup, and places it in a line (Queue) for the baristas (Workers) to make. The customer gets a receipt number to check when it's ready.
+
+This [producer.ts](cci:7://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:0:0-0:0) file is exactly that **Cashier**. It takes requests to generate reports and puts them in a line so the main application stays fast and doesn't freeze while waiting for the work to be done.
+
+Here is the breakdown of every part:
+
+### 1. [getQueue()](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:11:0-25:1)
+**The "Connection Starter"**
+
+*   **What it does:** This function checks if we are already connected to the "Job Line" (the Queue system). If we are, it just returns that connection. If not, it creates a new connection to the system (Redis/BullMQ).
+*   **Why it works like this:** Opening a new connection every single time you want to add a job is slow and wasteful (like buying a new phone every time you want to make a call). Ideally, you want to keep one phone line open and re-use it.
+*   **Result:** A working link to the queue system, ready to accept jobs.
+
+### 2. [enqueueReportJob(input, cacheKey, options)](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:27:0-60:1)
+**The "Order Taker"**
+
+*   **What it does:** This is the most important function. It takes the "Order" (the commit details, user ID, etc.), creates a unique "Ticket Number" (Job ID), and places the order into the queue.
+*   **How it works:**
+    1.  It generates a unique ID (like `uuidv4`) so we can track this specific job later.
+    2.  It packages all the data (userId, commit info) into a neat little bundle (`jobData`).
+    3.  It pushes this bundle onto the queue with instructions (like "if this fails, keep the record for a bit so we can see why").
+*   **Why it works like this:** By "enqueueing" (putting in line) the job instead of doing it immediately, the function returns the `Job ID` **instantly**. This means the user's browser isn't stuck loading for 20 seconds while the report generates. It feels instant.
+*   **Result:** A returned `Job ID` string (e.g., `"1234-5678..."`) that serves as a receipt.
+
+### 3. [getJobStatus(jobId)](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:62:0-92:1)
+**The "Status Board"**
+
+*   **What it does:** This allows us to check on a specific order using the "Ticket Number" (Job ID) we got earlier.
+*   **How it works:** It asks the queue system: "Hey, what is happening with Job #123?".
+*   **Possible Results:**
+    *   **"waiting"**: Still in line.
+    *   **"active"**: Currently being worked on.
+    *   **"completed"**: Done! Here is the result (the report).
+    *   **"failed"**: Something went wrong (and here is the error reason).
+*   **Why it works like this:** Since the background worker is doing the job separately, we need a way to "peek" at its progress without interrupting it.
+
+### 4. [closeQueue()](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:94:0-103:1)
+**The "Shutdown Switch"**
+
+*   **What it does:** Safely disconnects from the queue system.
+*   **Why it works like this:** When the application needs to restart or shut down, you don't want to just "pull the plug." This function politely closes the connection so no data is corrupted or lost.
+*   **Result:** The connection variable is reset to `null`.
+
+---
+
+### **Summary: How this helps the Inference Stepper**
+
+The Inference Stepper is designed to use AI to analyze code, which is **slow** (it can take 5-30 seconds).
+
+Without this file:
+> User clicks "Generate" -> **Wait 30 seconds...** -> Screen freezes -> Finally sees result.
+
+With this file:
+> User clicks "Generate" -> **Instant response** ("We're working on it! Order #123") -> The user can continue doing other things -> The system checks [getJobStatus](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:62:0-92:1) periodically -> Pop! The result appears when ready.
+
+This file ensures the system feels **premium, fast, and responsive**, even though the work being done behind the scenes is heavy and slow.
+
+### User Input
+
+lets break down the explanation of  
+ . i want to know what each functions means, how they work, possible results of it, why they work like that and how they help achieve what the inference stepper package is for.
+break down evey detail and explanton for me, to the lowest and clear enough for even a non programmer or technial person
+
+### User Input
+
+lets break down the explanation of  @[packages/stepper/src/queue/worker.ts] 
+ . i want to know what each functions means, how they work, possible results of it, why they work like that and how they help achieve what the inference stepper package is for.
+break down evey detail and explanton for me, to the lowest and clear enough for even a non programmer or technial person
+
+*Viewed [worker.ts](file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts) *
+
+### Planner Response
+
+This file acts as the **"Kitchen Staff"** or the **"Barista"** of the system.
+
+If the [producer.ts](cci:7://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:0:0-0:0) we discussed earlier is the **Cashier** taking orders and putting them on a line, then this [worker.ts](cci:7://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:0:0-0:0) is the **Worker** who stands at the back, picks up those orders one by one, and actually does the hard work of "brewing the coffee."
+
+Here is the breakdown of every part:
+
+---
+
+### 1. [processReportJob(job)](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:12:0-56:1)
+**The "Cooking Instructions"**
+
+*   **What it does:** This is the core logic. It tells the worker exactly *how* to handle an order once it picks it up from the line.
+*   **How it works:**
+    1.  **Check the "Fridge" (Cache):** First, it checks if someone else already made this specific report while it was waiting. If it's already done, it just stops to save time.
+    2.  **Do the Hard Work:** It calls `generateReportNow`, which is the part that actually talks to the AI and asks it to analyze the code.
+    3.  **Save the Result:** Once the AI finishes, it saves that result into the "Fridge" (Cache) so that when the user checks their "Receipt" (Job ID), the info is ready for them.
+    4.  **Mark as Done:** It updates the status to "100% complete."
+*   **Why it works like this:** It runs "asynchronously." This means it doesn't matter if it takes 5 seconds or 30 seconds; it's happening in the background without slowing down the rest of the website.
+*   **Possible Results:**
+    *   **Success:** The report is generated and saved.
+    *   **Failure:** If the AI is down or something breaks, it records the error so the user knows *why* it failed.
+
+### 2. [startWorker()](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:58:0-89:1)
+**The "Opening the Kitchen"**
+
+*   **What it does:** This is the button that turns the worker "ON."
+*   **How it works:**
+    1.  It checks if a worker is already running (no need to hire two people for the same spot).
+    2.  It connects to the same "Order Line" (the Queue) as the Cashier.
+    3.  It sets the **Concurrency** (how many jobs it can do at once). If concurrency is `5`, it’s like having 5 baristas working simultaneously.
+    4.  It sets up "Listeners" (Ears) to hear when a job finishes or fails, so it can write it down in the logs for the developers to see.
+*   **Why it works like this:** By separating the [startWorker](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:58:0-89:1) function, we can choose *where* to run the worker. We could have 10 different computers all running this worker to handle thousands of users at once.
+*   **Result:** The system starts listening for new orders and begins processing them immediately.
+
+### 3. [stopWorker()](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:91:0-100:1)
+**The "Closing Time"**
+
+*   **What it does:** Politely tells the worker to stop taking new orders and finish what they are doing before going home.
+*   **How it works:** It closes the connection to the queue system.
+*   **Why it works like this:** If you just "killed" the computer while it was halfway through a report, the report would be lost forever. [stopWorker](cci:1://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:91:0-100:1) ensures a "Graceful Shutdown"—finishing the current task before turning off the lights.
+*   **Result:** The worker shuts down safely.
+
+---
+
+### **How this helps the Inference Stepper**
+
+The **Inference Stepper** package is meant to be a high-performance tool for generating AI-powered commit messages or reports. 
+
+Without this [worker.ts](cci:7://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:0:0-0:0):
+*   The system would have a list of orders (from the [producer.ts](cci:7://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:0:0-0:0)) but **no one to actually do the work.**
+*   The AI analysis would never happen, and every order would just sit in line forever.
+
+**In simple terms:** 
+*   [producer.ts](cci:7://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/producer.ts:0:0-0:0) says: "Someone wants a report!"
+*   [worker.ts](cci:7://file:///home/blaze/mine/commitdiary/packages/stepper/src/queue/worker.ts:0:0-0:0) says: "I'm on it. I'll ask the AI, save the answer, and let everyone know when I'm done."
+
+By having this background worker, the **Inference Stepper** can handle hundreds of requests at the same time without ever making the user’s screen feel "laggy" or "frozen." It turns a slow AI process into a smooth, professional-feeling experience.

package/src/queue/README.md

@@ -0,0 +1,41 @@
+# 📥 Async Job Queue
+
+AI generation is slow (it can take 30-60 seconds). To keep the application responsive, the **Inference Stepper** uses a background job queue powered by **BullMQ** and **Redis**.
+
+## 🎯 Purpose
+
+- **Responsiveness**: Users get an immediate "Job Enqueued" response instead of waiting for the AI.
+- **Reliability**: If a job fails due to a temporary glitch, it can be retried automatically.
+- **Concurrency Control**: We can limit how many AI requests run at the same time to avoid hitting global rate limits.
+
+## 👥 Major Organs
+
+### 1. Producer (`producer.ts`)
+
+The "Cashier" of the system.
+
+- Takes the incoming request.
+- Assigns a unique `jobId`.
+- Puts the task into the Redis queue.
+- Returns the `jobId` to the caller.
+
+### 2. Worker (`worker.ts`)
+
+The "Kitchen Staff" who does the actual work.
+
+- Sits in the background and waits for jobs.
+- Picks up a job when one is available.
+- Calls the **Orchestrator** to do the AI inference.
+- Updates the cache with the final result.
+
+## 📋 Functions
+
+| Function             | Role                                                              |
+| -------------------- | ----------------------------------------------------------------- |
+| `enqueueReportJob()` | Adds a new report request to the queue.                           |
+| `getJobStatus()`     | Checks if a job is `waiting`, `active`, `completed`, or `failed`. |
+| `processReportJob()` | The internal logic the worker runs for every ticket.              |
+
+## 🚀 Scaling
+
+This architecture allows for **horizontal scaling**. You can run one instance of the API server (Producer) and ten instances of the Worker on different machines to handle high traffic, all sharing the same Redis queue.

package/src/queue/producer.ts

@@ -0,0 +1,108 @@
+// packages/stepper/src/queue/producer.ts`
+
+import { Queue } from 'bullmq';
+import { v4 as uuidv4 } from 'uuid';
+import { getRedisClient } from '../cache/redisCache.js';
+import { PromptInput, ReportJobData } from '../types.js';
+import { config } from '../config.js';
+import { logger } from '../logging.js';
+
+let queue: Queue<ReportJobData> | null = null;
+
+/**
+ * Get or create BullMQ queue
+ */
+export function getQueue(): Queue<ReportJobData> {
+    if (!queue) {
+        const connection = getRedisClient();
+        queue = new Queue<ReportJobData>(config.queue.name, { connection });
+
+        logger.info({ queueName: config.queue.name }, 'BullMQ queue initialized');
+    }
+
+    return queue;
+}
+
+/**
+ * Enqueue a report generation job
+ * The "Order Taker"
+ */
+export async function enqueueReportJob(
+    input: PromptInput,
+    cacheKey: string,
+    options: { priority?: number; callbackUrl?: string } = {}
+): Promise<string> {
+    const jobId = uuidv4();
+    const queue = getQueue();
+
+    const jobData: ReportJobData = {
+        jobId,
+        input,
+        cacheKey,
+        priority: options.priority,
+        callbackUrl: options.callbackUrl,
+    };
+
+    try {
+        await queue.add('generate-report', jobData, {
+            jobId,
+            priority: options.priority,
+            removeOnComplete: 100, // Keep last 100 completed jobs
+            removeOnFail: 500, // Keep last 500 failed jobs
+            attempts: 5, // Retry up to 5 times if all providers fail
+            backoff: {
+                type: 'exponential',
+                delay: 10000, // Start with 10 seconds, doubles each retry
+            },
+        });
+
+        logger.info({ jobId, userId: input.userId, commitSha: input.commitSha }, 'Job enqueued');
+        return jobId;
+    } catch (error) {
+        logger.error({ error, jobId }, 'Failed to enqueue job');
+        throw error;
+    }
+}
+
+/**
+ * Get job status
+ */
+export async function getJobStatus(jobId: string): Promise<{
+    id: string;
+    state: string;
+    progress?: number;
+    result?: unknown;
+    failedReason?: string;
+    data?: unknown;
+} | null> {
+    const queue = getQueue();
+
+    try {
+        const job = await queue.getJob(jobId);
+        if (!job) return null;
+
+        const state = await job.getState();
+        return {
+            id: job.id!,
+            state,
+            progress: job.progress as number | undefined,
+            result: job.returnvalue,
+            failedReason: job.failedReason,
+            data: job.data,
+        };
+    } catch (error) {
+        logger.error({ error, jobId }, 'Failed to get job status');
+        return null;
+    }
+}
+
+/**
+ * Close queue connection (for graceful shutdown)
+ */
+export async function closeQueue(): Promise<void> {
+    if (queue) {
+        await queue.close();
+        queue = null;
+        logger.info('Queue closed');
+    }
+}

package/src/queue/worker.ts

@@ -0,0 +1,170 @@
+// packages/stepper/src/queue/worker.ts`
+
+import { Worker, Job } from 'bullmq';
+import { getRedisClient, setHydrated, markFailed, getReportCache } from '../cache/redisCache.js';
+import { generateReportNow } from '../stepper/orchestrator.js';
+import { ReportJobData } from '../types.js';
+import { config } from '../config.js';
+import { logger, createChildLogger } from '../logging.js';
+import { recordJobProcessed, recordJobFailed } from '../metrics/metrics.js';
+import { sendDiscordAlert } from '../alerts/discord.js';
+import { notifyWebhookSuccess, notifyWebhookFailure } from '../webhooks/delivery.js';
+
+let worker: Worker<ReportJobData> | null = null;
+
+/**
+ * Job processor function
+ */
+async function processReportJob(job: Job<ReportJobData>): Promise<void> {
+    const { jobId, input, cacheKey } = job.data;
+    const log = createChildLogger({ jobId, userId: input.userId, commitSha: input.commitSha });
+
+    log.info('Processing report job');
+
+    try {
+        // Check cache again (avoid race condition)
+        const cached = await getReportCache(cacheKey);
+        if (cached && cached.status === 'hydrated') {
+            log.info('Report already hydrated in cache, skipping generation');
+            return;
+        }
+
+        // Generate report
+        const result = await generateReportNow(input, jobId);
+
+        // Store in cache
+        await setHydrated(cacheKey, result.result, result.providersAttempted, result.fallback);
+
+        // Update job progress
+        await job.updateProgress(100);
+
+        recordJobProcessed();
+        log.info({ usedProvider: result.usedProvider, fallback: result.fallback }, 'Job completed successfully');
+
+        // Note: input.callbacks are already executed in orchestrator immediately after generation
+        // The callbackUrl below is the legacy webhook for backwards compatibility
+        if (job.data.callbackUrl && config.webhook.enabled) {
+            log.info({ callbackUrl: job.data.callbackUrl }, 'Sending success webhook');
+            await notifyWebhookSuccess(
+                job.data.callbackUrl,
+                config.webhook.secret,
+                jobId,
+                result.result
+            );
+        }
+    } catch (error) {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        log.error({ error: errorMessage }, 'Job failed');
+
+        // Mark cache as failed
+        await markFailed(cacheKey, errorMessage, []);
+
+        recordJobFailed();
+
+        // Execute failure callbacks if configured
+        // Note: We handle this directly here since orchestrator threw before callbacks could execute
+        // The callbacks are fire-and-forget to not delay the job failure handling
+        if (input.callbacks && input.callbacks.length > 0) {
+            const failurePayload = {
+                success: false,
+                error: errorMessage,
+                metadata: {
+                    jobId,
+                    userId: input.userId,
+                    commitSha: input.commitSha,
+                    repo: input.repo,
+                    timestamp: new Date().toISOString(),
+                },
+            };
+
+            for (const callback of input.callbacks) {
+                fetch(callback.url, {
+                    method: 'POST',
+                    headers: {
+                        'Content-Type': 'application/json',
+                        'User-Agent': 'Stepper/1.0',
+                        'X-Stepper-Timestamp': Date.now().toString(),
+                        ...callback.headers,
+                    },
+                    body: JSON.stringify(failurePayload),
+                    signal: AbortSignal.timeout(10000),
+                }).catch((err: Error) => {
+                    log.warn({ url: callback.url, error: err.message }, 'Failed to send failure callback');
+                });
+            }
+        }
+
+        // Send failure webhook notification if configured (legacy)
+        if (job.data.callbackUrl && config.webhook.enabled) {
+            log.info({ callbackUrl: job.data.callbackUrl }, 'Sending failure webhook');
+            await notifyWebhookFailure(
+                job.data.callbackUrl,
+                config.webhook.secret,
+                jobId,
+                errorMessage
+            ).catch(err => {
+                log.warn({ error: err.message }, 'Failed to send failure webhook');
+            });
+        }
+
+        throw error; // Let BullMQ handle retry logic
+    }
+}
+
+/**
+ * Start worker
+ */
+export function startWorker(): void {
+    if (worker) {
+        logger.warn('Worker already started');
+        return;
+    }
+
+    const connection = getRedisClient();
+
+    worker = new Worker<ReportJobData>(config.queue.name, processReportJob, {
+        connection,
+        concurrency: config.queue.concurrency, //(how many jobs it can do at once).
+        removeOnComplete: { count: 100 },
+        removeOnFail: { count: 500 },
+    });
+
+    worker.on('completed', (job) => {
+        logger.info({ jobId: job.id }, 'Job completed');
+    });
+
+    worker.on('failed', (job, err) => {
+        logger.error({ jobId: job?.id, error: err.message }, 'Job failed');
+        if (job) {
+            void sendDiscordAlert({
+                title: 'Job Failed Permanently',
+                message: `Job **${job.id}** failed after all retries.\n\n**Error:**\n\`${err.message}\``,
+                severity: 'warning',
+                metadata: { jobId: job.id, error: err.message, userId: job.data.input.userId }
+            });
+        }
+    });
+
+    worker.on('error', (err) => {
+        logger.error({ error: err }, 'Worker error');
+        void sendDiscordAlert({
+            title: 'Worker System Error',
+            message: `The job queue worker encountered a system error: ${err.message}`,
+            severity: 'critical',
+            metadata: { error: err.message }
+        });
+    });
+
+    logger.info({ concurrency: config.queue.concurrency }, 'Worker started');
+}
+
+/**
+ * Stop worker gracefully
+ */
+export async function stopWorker(): Promise<void> {
+    if (worker) {
+        await worker.close();
+        worker = null;
+        logger.info('Worker stopped');
+    }
+}