@govtechsg/oobee 0.10.69 → 0.10.72

package/DETAILS.md

@@ -168,7 +168,6 @@ Note: Level AAA are disabled by default.  Please specify `enable-wcag-aaa` in ru
 | Issue ID                            | Issue Description                                                                                                                              | Severity    |
 | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
 | accesskeys                          | Ensures every accesskey attribute value is unique                                                                                              | Good to Fix |
-| aria-allowed-role                   | Ensures role attribute has an appropriate value for the element                                                                                | Good to Fix |
 | aria-dialog-name                    | Ensures every ARIA dialog and alertdialog node has an accessible name                                                                          | Good to Fix |
 | aria-text                           | Ensures role="text" is used on elements with no focusable descendants                                                                          | Good to Fix |
 | aria-treeitem-name                  | Ensures every ARIA treeitem node has an accessible name                                                                                        | Good to Fix |

package/README.md

@@ -571,7 +571,7 @@ The following URL and file validation error codes are provided to troubleshoot t
 | 11   | invalidUrl           | Invalid URL. Please check and try again.                                | • Ensure the URL starts with `http://` or `https://`.<br>• Check for typos in the URL. |
 | 12   | cannotBeResolved     | URL cannot be accessed. Please verify whether the website exists.     | • Confirm the domain name is correct.<br>• Check DNS resolution with `ping` or `nslookup`.<br>• Ensure the site is publicly accessible (not behind VPN/firewall). |
 | 14   | systemError          | Something went wrong when verifying the URL. Please try again in a few minutes. If this issue persists, please contact the Oobee team.          | • Retry after a few minutes.<br>• Check internet connection.<br>• If persistent, report as a system issue. |
-| 15   | notASitemap          | Invalid sitemap URL format. Please enter a valid sitemap URL ending with .XML e.g. https://www.example.com/sitemap.xml.                                                | • Ensure the URL points to a valid XML sitemap.<br>• View [Examples of sitemaps sitemaps.org - Protocol](https://www.sitemaps.org/protocol.html)<br>• Test the URL in a browser to confirm it returns XML. |
+| 15   | notASitemap          | Invalid sitemap URL format. Please enter a valid sitemap URL ending with .XML or .TXT e.g. https://www.example.com/sitemap.xml.                                                | • Ensure the URL points to a valid XML sitemap.<br>• View [Examples of sitemaps sitemaps.org - Protocol](https://www.sitemaps.org/protocol.html)<br>• Test the URL in a browser to confirm it returns XML. |
 | 16   | unauthorised         | Login required. Please enter your credentials and try again.                                       | • Check if the site requires username/password.<br>• Provide credentials in Oobee if supported. |
 | 17   | browserError         | Incompatible browser. Please ensure you are using Chrome or Edge browser. | • Install the latest version of Chrome or Edge.|
 | 18   | sslProtocolError     | SSL certificate  error. Please check the SSL configuration of your website and try again. | • Verify SSL certificate validity (not expired, issued by trusted CA).<br>• Check for mismatched TLS versions or cipher issues.<br>• Use an SSL checker tool (e.g., Qualys SSL Labs). |
@@ -666,3 +666,15 @@ We recommend looking at our **Technology Stack** to understand the usage of each
 Oobee uses third-party open-source tools that may be downloaded over the Internet during the installation process of Oobee. Users should be aware of the libraries used by examining `package.json`.
 
 Oobee may send information to the website or URL where the user chooses to initiate a Oobee scan. Limited user information such as e-mail address, name, and basic analytics is collected for the purpose of knowing our usage patterns better.
+
+## Development and Testing Helpers
+
+Useful helpers for Oobee developers
+### Generate new HTML
+When working on EJS (HTML, CSS, JS), use the following helper script to test your changes so that you do not need to run a full scan to get the new report in results directory.
+
+It uses the existing report *.json files for the embedded HTML dataset.
+
+```
+npx tsx dev/runGenerateJustHtmlReport.ts results/<report directory>
+```

package/S3_UPLOAD_README.md

@@ -0,0 +1,172 @@
+# S3 Upload Integration
+
+This document explains how to use the S3 upload feature in the Oobee engine.
+
+## Overview
+
+The Oobee engine can now automatically upload scan results to Amazon S3 after completing a scan. This enables integration with cloud-based workflows and event-driven architectures:
+
+1. Oobee runs and generates scan results locally
+2. Results are automatically uploaded to S3 with metadata
+3. S3 events can trigger downstream processing (e.g., serverless functions)
+4. External systems can process results and update databases, send notifications, etc.
+
+## Environment Variables
+
+To enable S3 upload, set the following environment variables before running Oobee:
+
+### Required Variables
+
+```bash
+# AWS Configuration
+export AWS_REGION="ap-southeast-1"                    # AWS region
+export AWS_ACCESS_KEY_ID="your-access-key"           # AWS credentials
+export AWS_SECRET_ACCESS_KEY="your-secret-key"       # AWS credentials
+export S3_BUCKET_NAME="oobee-scan-results"           # S3 bucket name
+
+# Scan Metadata (provided by your orchestration system)
+export OOBEE_SCAN_ID="uuid-scan-id"                  # Unique scan identifier
+export OOBEE_USER_ID="uuid-user-id"                  # User identifier
+export OOBEE_USER_EMAIL="user@example.com"           # User email
+
+# Optional Variables
+export CLOUDFRONT_BASE_URL="https://cdn.example.com" # CloudFront distribution URL
+```
+
+### S3 Upload Path
+
+Files will be uploaded to: `s3://{bucket}/users/{userId}/scans/{scanId}/`
+
+Example: `s3://oobee-scan-results/users/550e8400-e29b-41d4-a716-446655440000/scans/650e8400-e29b-41d4-a716-446655440001/`
+
+## Usage
+
+### 1. Automated Workflow (Containerized/Orchestrated)
+
+When running Oobee in a containerized or orchestrated environment, pass the required environment variables to the container:
+
+```bash
+# Docker example
+docker run -e OOBEE_SCAN_ID="650e8400-e29b-41d4-a716-446655440001" \
+           -e OOBEE_USER_ID="550e8400-e29b-41d4-a716-446655440000" \
+           -e OOBEE_USER_EMAIL="user@example.com" \
+           -e S3_BUCKET_NAME="oobee-scan-results" \
+           -e AWS_REGION="ap-southeast-1" \
+           oobee-engine:latest
+
+# Kubernetes example
+env:
+  - name: OOBEE_SCAN_ID
+    value: "650e8400-e29b-41d4-a716-446655440001"
+  - name: OOBEE_USER_ID
+    value: "550e8400-e29b-41d4-a716-446655440000"
+  - name: OOBEE_USER_EMAIL
+    value: "user@example.com"
+  - name: S3_BUCKET_NAME
+    value: "oobee-scan-results"
+  - name: AWS_REGION
+    value: "ap-southeast-1"
+```
+
+### 2. From CLI (Manual Testing)
+
+```bash
+# Set environment variables
+export AWS_REGION="ap-southeast-1"
+export AWS_ACCESS_KEY_ID="your-key"
+export AWS_SECRET_ACCESS_KEY="your-secret"
+### 3. Without S3 Upload (Default Behavior)
+
+If environment variables are not set, Oobee works as before:
+- Results are saved locally only
+- No S3 upload occurs
+- No errors (logs informational message)
+
+```bash
+# Run normally without S3 variables
+npm run cli -- -c website -u https://example.com
+```
+If environment variables are not set, Oobee will work as before:
+- Results are saved locally
+- No S3 upload occurs
+- No errors or warnings (except info log)
+
+```bash
+# Just run normally without S3 variables
+npm run cli -- -c website -u https://example.com
+```
+
+## Implementation Details
+
+### Files Uploaded
+
+The S3 uploader uploads all files from the results directory, but only tracks these file types in the response:
+- `.html` - HTML reports
+- `.csv` - CSV data files
+- `.pdf` - PDF reports
+- `.zip` - Zipped scan results
+
+All files are uploaded with S3 metadata containing:
+- `scanid` - Scan ID
+- `userid` - User ID
+- `useremail` - User email
+
+### S3 Metadata
+
+Each uploaded file includes metadata that the S3 Lambda processor uses:
+
+```javascript
+{
+  "scanid": "650e8400-e29b-41d4-a716-446655440001",
+  "userid": "550e8400-e29b-41d4-a716-446655440000",
+  "useremail": "user@example.com",
+}
+```
+
+### Error Handling
+
+- If S3 upload fails, the scan continues normally
+- Error is logged but doesn't fail the scan
+- Results are still available locally
+- User can retry upload manually if needed
+
+### Key Functions
+
+#### `isS3UploadEnabled()`
+Checks if all required environment variables are set.
+
+#### `getS3MetadataFromEnv()`
+Extracts scan metadata from environment variables.
+
+#### `getS3UploadPrefix()`
+Generates the S3 path: `users/{userId}/scans/{scanId}`
+
+#### `uploadFolderToS3(localPath, s3Prefix, metadata)`
+Uploads entire results folder to S3 with metadata.
+
+## Testing
+
+### Local Testing
+
+1. Create a test S3 bucket:
+```bash
+aws s3 mb s3://oobee-test-scan-results
+```
+
+2. Set test environment variables:
+```bash
+export S3_BUCKET_NAME="oobee-test-scan-results"
+export OOBEE_SCAN_ID="test-$(date +%s)"
+export OOBEE_USER_ID="test-user-123"
+export OOBEE_USER_EMAIL="test@example.com"
+```
+
+3. Run a test scan:
+```bash
+npm run cli -- -c website -u https://example.com -p 5
+```
+
+4. Verify upload:
+```bash
+aws s3 ls s3://oobee-test-scan-results/users/test-user-123/scans/
+```

package/dev/runGenerateJustHtmlReport.ts

@@ -0,0 +1,25 @@
+/* 
+For developemnt and testing purposes only. Run the report generator
+on an existing results directory.
+*/
+
+import path from 'path';
+import { generateHtmlReport } from '../src/generateHtmlReport.js';
+
+async function main() {
+  const dirArg = process.argv[2];
+
+  if (!dirArg) {
+    console.error('Usage: npx tsx src/dev-generate-from-existing.ts <results-dir>');
+    process.exit(1);
+  }
+
+  const resultDir = path.resolve(process.cwd(), dirArg);
+  const out = await generateHtmlReport(resultDir);
+  console.log('\nOpen:', out);
+}
+
+main().catch((e) => {
+  console.error(e);
+  process.exit(1);
+});

package/package.json

@@ -1,13 +1,15 @@
 {
   "name": "@govtechsg/oobee",
   "main": "dist/npmIndex.js",
-  "version": "0.10.69",
+  "version": "0.10.72",
   "type": "module",
   "author": "Government Technology Agency <info@tech.gov.sg>",
   "dependencies": {
+    "@aws-sdk/client-s3": "^3.948.0",
     "@json2csv/node": "^7.0.3",
     "@napi-rs/canvas": "^0.1.53",
     "@sentry/node": "^9.13.0",
+    "@types/aws-sdk": "^0.0.42",
     "axe-core": "^4.10.3",
     "axios": "^1.8.2",
     "base64-stream": "^1.0.0",
@@ -16,7 +18,7 @@
     "ejs": "^3.1.9",
     "file-type": "^19.5.0",
     "fs-extra": "^11.2.0",
-    "glob": "^10.3.10",
+    "glob": "^11.1.0",
     "https": "^1.0.0",
     "inquirer": "^9.2.12",
     "jsdom": "^21.1.2",

package/src/combine.ts

@@ -5,13 +5,19 @@ import crawlDomain from './crawlers/crawlDomain.js';
 import crawlLocalFile from './crawlers/crawlLocalFile.js';
 import crawlIntelligentSitemap from './crawlers/crawlIntelligentSitemap.js';
 import generateArtifacts from './mergeAxeResults.js';
-import { getHost, createAndUpdateResultsFolders, cleanUpAndExit } from './utils.js';
+import { getHost, createAndUpdateResultsFolders, cleanUpAndExit, getStoragePath } from './utils.js';
 import { ScannerTypes, UrlsCrawled } from './constants/constants.js';
 import { getBlackListedPatterns, submitForm } from './constants/common.js';
 import { consoleLogger, silentLogger } from './logs.js';
 import runCustom from './crawlers/runCustom.js';
 import { alertMessageOptions } from './constants/cliFunctions.js';
 import { Data } from './index.js';
+import {
+  isS3UploadEnabled,
+  getS3MetadataFromEnv,
+  getS3UploadPrefix,
+  uploadFolderToS3,
+} from './services/s3Uploader.js';
 
 // Class exports
 export class ViewportSettingsClass {
@@ -61,18 +67,18 @@ const combineRun = async (details: Data, deviceToScan: string) => {
     zip,
     ruleset, // Enable custom checks, Enable WCAG AAA: if checked, = 'enable-wcag-aaa')
     generateJsonFiles,
-    scanDuration
+    scanDuration,
   } = envDetails;
 
   process.env.CRAWLEE_LOG_LEVEL = 'ERROR';
   process.env.CRAWLEE_STORAGE_DIR = randomToken;
 
   if (process.env.CRAWLEE_SYSTEM_INFO_V2 === undefined) {
-  // Set the environment variable to enable system info v2
-  // Resolves issue with when wmic is not installed on Windows
+    // Set the environment variable to enable system info v2
+    // Resolves issue with when wmic is not installed on Windows
     process.env.CRAWLEE_SYSTEM_INFO_V2 = '1';
   }
-  
+
   const host = type === ScannerTypes.SITEMAP || type === ScannerTypes.LOCALFILE ? '' : getHost(url);
 
   let blacklistedPatterns: string[] | null = null;
@@ -121,19 +127,26 @@ const combineRun = async (details: Data, deviceToScan: string) => {
   );
 
   let urlsCrawledObj: UrlsCrawled;
+  let uiCustomFlowLabel: string | undefined;
+  let durationExceeded = false;
+
   switch (type) {
     case ScannerTypes.CUSTOM:
-      urlsCrawledObj = await runCustom(
+      const res = await runCustom(
         url,
         randomToken,
         viewportSettings,
         blacklistedPatterns,
         includeScreenshots,
+        customFlowLabel && customFlowLabel !== 'None' ? customFlowLabel : '',
       );
+
+      urlsCrawledObj = res.urlsCrawled;
+      uiCustomFlowLabel = res.customFlowLabel;
       break;
 
     case ScannerTypes.SITEMAP:
-        urlsCrawledObj = await crawlSitemap({
+      const sitemapResult = await crawlSitemap({
         sitemapUrl: url,
         randomToken,
         host,
@@ -148,10 +161,12 @@ const combineRun = async (details: Data, deviceToScan: string) => {
         extraHTTPHeaders,
         scanDuration,
       });
+      urlsCrawledObj = sitemapResult.urlsCrawled;
+      durationExceeded = sitemapResult.durationExceeded;
       break;
 
     case ScannerTypes.LOCALFILE:
-      urlsCrawledObj = await crawlLocalFile({
+      const localFileResult = await crawlLocalFile({
         url,
         randomToken,
         host,
@@ -166,10 +181,18 @@ const combineRun = async (details: Data, deviceToScan: string) => {
         extraHTTPHeaders,
         scanDuration,
       });
+      if (localFileResult) {
+        if ('urlsCrawled' in localFileResult) {
+          urlsCrawledObj = localFileResult.urlsCrawled;
+          durationExceeded = localFileResult.durationExceeded;
+        } else {
+          urlsCrawledObj = localFileResult;
+        }
+      }
       break;
 
     case ScannerTypes.INTELLIGENT:
-      urlsCrawledObj = await crawlIntelligentSitemap(
+      const intelligentResult = await crawlIntelligentSitemap(
         url,
         randomToken,
         host,
@@ -185,12 +208,14 @@ const combineRun = async (details: Data, deviceToScan: string) => {
         followRobots,
         extraHTTPHeaders,
         safeMode,
-        scanDuration
+        scanDuration,
       );
+      urlsCrawledObj = intelligentResult.urlsCrawled;
+      durationExceeded = intelligentResult.durationExceeded;
       break;
 
     case ScannerTypes.WEBSITE:
-      urlsCrawledObj = await crawlDomain({
+      const websiteResult = await crawlDomain({
         url,
         randomToken,
         host,
@@ -209,6 +234,8 @@ const combineRun = async (details: Data, deviceToScan: string) => {
         safeMode,
         ruleset,
       });
+      urlsCrawledObj = websiteResult.urlsCrawled;
+      durationExceeded = websiteResult.durationExceeded;
       break;
 
     default:
@@ -235,7 +262,9 @@ const combineRun = async (details: Data, deviceToScan: string) => {
         deviceToScan,
         urlsCrawledObj.scanned,
         pagesNotScanned,
-        customFlowLabel,
+        uiCustomFlowLabel && uiCustomFlowLabel.length > 0
+          ? uiCustomFlowLabel
+          : customFlowLabel || 'None',
         undefined,
         scanDetails,
         zip,
@@ -243,6 +272,36 @@ const combineRun = async (details: Data, deviceToScan: string) => {
       );
       const [name, email] = nameEmail.split(':');
 
+      // Upload results to S3 if environment variables are set
+      if (isS3UploadEnabled()) {
+        const siteName = (urlsCrawledObj.scanned[0]?.pageTitle ?? '')
+          .replace(/^\d+\s*:\s*/, '')
+          .trim();
+        const scanMetadata = getS3MetadataFromEnv(siteName, durationExceeded);
+        const s3Prefix = getS3UploadPrefix();
+
+        if (scanMetadata && s3Prefix) {
+          try {
+            const storagePath = getStoragePath(randomToken);
+            consoleLogger.info('Starting S3 upload...');
+            consoleLogger.info(`Upload path: ${s3Prefix}`);
+
+            const uploadedFiles = await uploadFolderToS3(storagePath, s3Prefix, scanMetadata);
+
+            consoleLogger.info(`Successfully uploaded ${uploadedFiles.length} files to S3`);
+          } catch (error) {
+            const errorMessage = error instanceof Error ? error.message : String(error);
+            consoleLogger.error(`Failed to upload results to S3: ${errorMessage}`);
+            // Don't fail the scan if S3 upload fails
+            consoleLogger.warn('Continuing without S3 upload...');
+          }
+        } else {
+          consoleLogger.warn('S3 upload enabled but metadata/prefix not available');
+        }
+      } else {
+        consoleLogger.info('S3 upload not enabled (missing environment variables)');
+      }
+
       await submitForm(
         browser,
         userDataDirectory,
@@ -258,13 +317,11 @@ const combineRun = async (details: Data, deviceToScan: string) => {
         metadata,
       );
     } else {
-
       // No page were scanned because the URL loaded does not meet the crawler requirements
       printMessage([`No pages were scanned.`], alertMessageOptions);
       cleanUpAndExit(1, randomToken, true);
     }
   } else {
-
     // No page were scanned because the URL loaded does not meet the crawler requirements
     printMessage([`No pages were scanned.`], alertMessageOptions);
     cleanUpAndExit(1, randomToken, true);