@akotliar/sitemap-qa 1.0.0-alpha.3 → 1.0.0-alpha.5

package/README.md

@@ -11,14 +11,39 @@ Sitemap-QA is a command-line tool that automatically discovers, parses, and anal
 
 ---
 
+## 📑 Table of Contents
+
+- [Why Sitemap-QA?](#-why-sitemap-qa)
+- [Quick Start](#-quick-start)
+  - [Installation](#installation)
+  - [Basic Usage](#basic-usage)
+- [Features](#-features)
+  - [Automatic Sitemap Discovery](#automatic-sitemap-discovery)
+  - [Risk Detection Patterns](#risk-detection-patterns)
+  - [Customizing Risks](#customizing-risks)
+  - [Output Formats](#output-formats)
+- [CLI Commands](#-cli-commands)
+  - [analyze](#analyze-command)
+  - [init](#init-command)
+- [Configuration](#-configuration)
+  - [Configuration Options](#configuration-options)
+
+- [License](#-license)
+- [Acknowledgments](#-acknowledgments)
+- [Support](#-support)
+
+---
+
 ## 🎯 Why Sitemap-QA?
 
-Unlike SEO-focused sitemap validators, Sitemap-QA is designed specifically for **QA validation and risk detection**:
+Unlike SEO-focused sitemap validators, Sitemap-QA is designed specifically for **QA validation and risk detection** using a **Policy-as-Code** approach:
 
 - ✅ **Detect environment leakage** — Find staging, dev, or test URLs that shouldn't be in production sitemaps
 - ✅ **Identify exposed admin paths** — Catch `/admin`, `/dashboard`, and internal routes in public indexes
-- ✅ **Flag sensitive parameters** — Detect API keys, tokens, or passwords in sitemap URLs
-- ✅ **Validate domain consistency** — Find protocol mismatches and subdomain issues
+- ✅ **Flag sensitive files** — Detect database backups, environment files, and archives
+- ✅ **Domain Consistency** — Automatically flag URLs that point to external or incorrect domains (handles `www.` normalization)
+- ✅ **Acceptable Patterns (Allowlist)** — Exclude known safe URLs from being flagged as risks
+- ✅ **Fully Customizable** — Define your own risk categories and patterns using Literal, Glob, or Regex matching
 - ✅ **Fast and automated** — Analyze thousands of URLs in seconds with detailed reports
 
 Perfect for CI/CD pipelines, pre-release validation, and security audits.
@@ -37,14 +62,17 @@ npm install -g @akotliar/sitemap-qa@alpha
 ### Basic Usage
 
 ```bash
-# Analyze a website's sitemap
+# Step 1: Initialize a configuration file (optional but recommended)
+sitemap-qa init
+
+# Step 2: Analyze a website's sitemap
 sitemap-qa analyze https://example.com
 
-# Generate JSON output for CI/CD
-sitemap-qa analyze https://example.com --output json > report.json
+# Generate JSON output only for CI/CD
+sitemap-qa analyze https://example.com --output json
 
-# Increase verbosity for debugging
-sitemap-qa analyze https://example.com --verbose
+# Use a custom configuration file
+sitemap-qa analyze https://example.com --config ./custom-config.yaml
 ```
 
 ---
@@ -55,67 +83,102 @@ sitemap-qa analyze https://example.com --verbose
 - Checks `robots.txt` for sitemap declarations
 - Tests standard paths (`/sitemap.xml`, `/sitemap_index.xml`, etc.)
 - Recursively follows sitemap indexes
-- Handles multiple sitemaps and formats
-- Detects and processes malformed sitemap indexes (sitemaps listed in `<url>` blocks instead of `<sitemap>` blocks)
+- Handles multiple sitemaps in supported formats (XML, compressed XML `.xml.gz`, and dynamically generated/PHP-based sitemaps)
 
 ### Risk Detection Patterns
 
-| Risk Category | Severity | Examples | Can Be Excluded |
-|--------------|----------|----------|-----------------|
-| **Environment Leakage** | High | `staging.example.com`, `/dev/`, `/test/` | ✅ Via patterns |
-| **Admin Paths** | High | `/admin`, `/dashboard`, `/config`, `/console` | ✅ Via patterns |
-| **Internal Content** | Medium | `/internal` paths | ✅ Via patterns |
-| **Sensitive Parameters** | High | `?token=`, `?apikey=`, `?password=` | ✅ Via patterns |
-| **Test Content** | Medium | `/test-`, `sample-`, `demo-` | ✅ Via patterns |
-| **Protocol Inconsistency** | Medium | HTTP URLs in HTTPS sitemaps | ❌ Always detected |
-| **Domain Mismatch** | Medium | Different domains in sitemap | ❌ Always detected |
+The tool comes with a set of default policies, but you can fully customize them in your `sitemap-qa.yaml`.
+
+| Risk Category | Description | Example Patterns |
+|--------------|-------------|------------------|
+| **Security & Admin** | Detects exposed administrative interfaces and sensitive configuration files. | `**/admin/**`, `**/.env*`, `/wp-admin` |
+| **Environment Leakage** | Finds staging or development URLs that shouldn't be in production sitemaps. | `**/staging.**`, `**/dev.**` |
+| **Sensitive Files** | Flags database backups, archives, and other sensitive file types. | `**/*.{sql,bak,zip,tar}`, `**/*.tar.gz` |
+| **Domain Consistency** | Detects URLs that don't match the target domain (ignoring `www.` differences). | `example.com` vs `other.com` |
+
+### Customizing Risks
+
+You can add your own categories and patterns to the `sitemap-qa.yaml` file. Patterns support `literal`, `glob`, and `regex` matching. See the [Configuration](#-configuration) section for details.
 
 
 ### Output Formats
 
 #### HTML Report (Interactive)
 The HTML report provides an interactive, visually appealing view with:
-- Expandable/collapsible sections by severity
-- Download buttons to export all URLs per category
+- Expandable/collapsible sections by category
+- Download buttons to export all URLs per finding
 - Clean, modern design with hover effects
 - Portable single-file format
 
 #### JSON Report (Machine-Readable)
 ```json
 {
-  "analysis_metadata": {
-    "base_url": "https://example.com",
-    "tool_version": "1.0.0",
-    "analysis_type": "rule-based analysis",
-    "analysis_timestamp": "2025-12-11T00:00:00.000Z",
-    "execution_time_ms": 4523
+  "metadata": {
+    "generatedAt": "2025-12-24T12:00:00.000Z",
+    "durationMs": 1240
+  },
+  "summary": {
+    "totalUrls": 895,
+    "totalRisks": 2,
+    "urlsWithRisksCount": 1,
+    "ignoredUrlsCount": 5
   },
-  "sitemaps_discovered": [
-    "https://example.com/sitemap.xml"
-  ],
-  "suspicious_groups": [
+  "findings": [
     {
-      "category": "environment_leakage",
-      "severity": "high",
-      "count": 3,
-      "rationale": "Production sitemap contains staging URLs",
-      "sample_urls": ["..."],
-      "recommended_action": "Verify sitemap generation excludes non-production environments"
+      "loc": "https://example.com/admin/login",
+      "risks": [
+        {
+          "category": "Security & Admin",
+          "pattern": "**/admin/**",
+          "type": "glob",
+          "reason": "Administrative interfaces should not be publicly indexed."
+        }
+      ]
     }
-  ],
-  "summary": {
-    "high_severity_count": 2,
-    "medium_severity_count": 1,
-    "low_severity_count": 0,
-    "total_risky_urls": 8,
-    "overall_status": "issues_found"
-  }
+  ]
 }
 ```
 
 ---
 
-## 🛠️ CLI Options
+## 🛠️ CLI Commands
+
+Sitemap-QA provides two main commands: `init` and `analyze`.
+
+
+### init Command
+
+Initialize a default `sitemap-qa.yaml` configuration file in the current directory.
+
+```
+Usage: sitemap-qa init [options]
+
+Initialize a default sitemap-qa.yaml configuration file
+
+Options:
+  -h, --help                   Display help for command
+```
+
+#### Example
+
+```bash
+# Create a default configuration file
+sitemap-qa init
+
+# This creates sitemap-qa.yaml with:
+# - Default risk policies (Security & Admin, Environment Leakage, Sensitive Files)
+# - Example acceptable patterns
+# - Default output settings
+```
+
+**Note:** The `init` command will fail if `sitemap-qa.yaml` already exists in the current directory to prevent accidental overwrites.
+
+---
+
+
+### analyze Command
+
+Analyze a website's sitemap for quality issues and security risks.
 
 ```
 Usage: sitemap-qa analyze <url> [options]
@@ -126,90 +189,84 @@ Arguments:
   url                          Base URL of the website to analyze
 
 Options:
-  --timeout <seconds>          HTTP request timeout in seconds (default: 30)
-  --output <format>            Output format: html or json (default: "html")
-  --output-dir <path>          Output directory for reports (default: "./sitemap-qa/report")
-  --output-file <path>         Custom output filename
-  --accepted-patterns <list>   Comma-separated patterns to exclude from risk detection
-  --verbose                    Enable verbose logging
+  -c, --config <path>          Path to sitemap-qa.yaml configuration file
+  -o, --output <format>        Output format: json, html, or all (default: "all")
+  -d, --out-dir <path>         Output directory for reports (default: ".")
   -h, --help                   Display help for command
 ```
 
-### Examples
+#### Examples
 
 ```bash
-# Basic analysis with HTML report (default)
+# Basic analysis with both HTML and JSON reports (default)
 sitemap-qa analyze https://example.com
 
-# JSON output for CI/CD integration
+# JSON output only
 sitemap-qa analyze https://example.com --output json
 
-# Custom output directory
-sitemap-qa analyze https://example.com --output-dir ./reports
+# HTML output only
+sitemap-qa analyze https://example.com --output html
 
-# Exclude specific URL patterns from detection
-sitemap-qa analyze https://example.com --accepted-patterns "internal-*,test-*"
+# Custom output directory
+sitemap-qa analyze https://example.com --out-dir ./reports
 
-# Increase timeout for slow servers
-sitemap-qa analyze https://example.com --timeout 60
+# Use a specific configuration file
+sitemap-qa analyze https://example.com --config ./custom-config.yaml
 
-# Verbose mode for debugging
-sitemap-qa analyze https://example.com --verbose
+# Combine options
+sitemap-qa analyze https://example.com --config ./custom-config.yaml --output json --out-dir ./reports
 ```
 
----
-
 ## 🔧 Configuration
 
-Create a `.sitemap-qa.config.json` file in your project root or `~/.sitemap-qa/config.json` for global settings:
-
-```json
-{
-  "timeout": 30,
-  "concurrency": 10,
-  "outputFormat": "html",
-  "outputDir": "./sitemap-qa/report",
-  "verbose": false,
-  "acceptedPatterns": [
-    "test-*",
-    "staging-*"
-  ]
-}
+Create a `sitemap-qa.yaml` file in your project root to define your monitoring policies and tool settings:
+
+```yaml
+# Tool Settings
+# Default outDir is "."; this example uses a custom reports directory
+outDir: "./sitemap-qa/report" # custom output directory
+outputFormat: "all" # Options: json, html, all
+enforceDomainConsistency: true # Flag URLs from other domains
+
+# Monitoring Policies
+acceptable_patterns:
+  - type: "literal"
+    value: "/acceptable-path"
+    reason: "Example of an acceptable path that should not be flagged."
+  - type: "glob"
+    value: "**/public-docs/**"
+    reason: "Public documentation is always acceptable."
+
+policies:
+  - category: "Security & Admin"
+    patterns:
+      - type: "glob"
+        value: "**/admin/**"
+        reason: "Administrative interfaces should not be publicly indexed."
+      - type: "literal"
+        value: "/wp-admin"
+        reason: "WordPress admin paths are common attack vectors."
+      - type: "regex"
+        value: ".*\\.php$"
+        reason: "PHP file detected"
 ```
 
 ### Configuration Options
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `timeout` | number | `30` | HTTP request timeout in seconds (1-300) |
-| `concurrency` | number | `10` | Number of concurrent HTTP requests |
-| `outputFormat` | string | `"html"` | Output format: `"html"` or `"json"` |
-| `outputDir` | string | `"./sitemap-qa/report"` | Directory for generated reports |
-| `verbose` | boolean | `false` | Enable detailed logging |
-| `acceptedPatterns` | string[] | `[]` | URL patterns to exclude from risk detection |
+| `outDir` | string | `"."` | Directory for generated reports (current working directory by default) |
+| `outputFormat` | string | `"all"` | Report types to generate: `json`, `html`, or `all` |
+| `enforceDomainConsistency` | boolean | `true` | If true, flags URLs that don't match the root sitemap domain (ignoring `www.`) |
+| `acceptable_patterns` | array | `[]` | List of patterns to exclude from risk analysis |
+| `policies` | array | `[]` | List of monitoring policies with patterns |
 
-### Accepted Patterns
 
-Exclude specific URLs from risk detection using wildcard patterns:
+**Priority:** CLI options > Project config (`sitemap-qa.yaml`) > Defaults
 
-```json
-{
-  "acceptedPatterns": [
-    "testing-*",                 // Matches: test-player, test-player-stats
-    "https://example.com/admin/*",  // Matches: any URL under /admin/
-    "/special-case"                 // Matches: exact path segment
-  ]
-}
-```
 
-**Pattern Syntax:**
-- Use `*` as a wildcard (matches any characters within a path segment)
-- Patterns are case-insensitive
-- Special characters are automatically escaped
-- Patterns match against the full URL
-- Full URLs or path fragments both work
 
-**Priority:** CLI options > Project config (`.sitemap-qa.config.json`) > Global config (`~/.sitemap-qa/config.json`) > Defaults
+---
 
 ## 📝 License
 
@@ -222,6 +279,10 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 Built with:
 - [Commander.js](https://github.com/tj/commander.js) - CLI framework
 - [Chalk](https://github.com/chalk/chalk) - Terminal styling
+- [Undici](https://github.com/nodejs/undici) - High-performance HTTP client
+- [Fast-XML-Parser](https://github.com/NaturalIntelligence/fast-xml-parser) - Fast XML parsing
+- [Zod](https://zod.dev/) - Schema validation
+- [Micromatch](https://github.com/micromatch/micromatch) - Glob pattern matching
 - [Vitest](https://vitest.dev/) - Testing framework
 - [TypeScript](https://www.typescriptlang.org/) - Type safety
 
@@ -229,7 +290,7 @@ Built with:
 
 ## 📧 Support
 
-- **Issues**: [GitHub Issues](https://github.com/akotliar/sitemap-qa/issues)-
+- **Issues**: [GitHub Issues](https://github.com/akotliar/sitemap-qa/issues)
 
 ---