@akotliar/sitemap-qa 1.0.0-alpha.4 → 1.0.0-alpha.6

package/README.md

@@ -11,6 +11,29 @@ Sitemap-QA is a command-line tool that automatically discovers, parses, and anal
 
 ---
 
+## 📑 Table of Contents
+
+- [Why Sitemap-QA?](#-why-sitemap-qa)
+- [Quick Start](#-quick-start)
+  - [Installation](#installation)
+  - [Basic Usage](#basic-usage)
+- [Features](#-features)
+  - [Automatic Sitemap Discovery](#automatic-sitemap-discovery)
+  - [Risk Detection Patterns](#risk-detection-patterns)
+  - [Customizing Risks](#customizing-risks)
+  - [Output Formats](#output-formats)
+- [CLI Commands](#-cli-commands)
+  - [analyze](#analyze-command)
+  - [init](#init-command)
+- [Configuration](#-configuration)
+  - [Configuration Options](#configuration-options)
+
+- [License](#-license)
+- [Acknowledgments](#-acknowledgments)
+- [Support](#-support)
+
+---
+
 ## 🎯 Why Sitemap-QA?
 
 Unlike SEO-focused sitemap validators, Sitemap-QA is designed specifically for **QA validation and risk detection** using a **Policy-as-Code** approach:
@@ -18,6 +41,8 @@ Unlike SEO-focused sitemap validators, Sitemap-QA is designed specifically for *
 - ✅ **Detect environment leakage** — Find staging, dev, or test URLs that shouldn't be in production sitemaps
 - ✅ **Identify exposed admin paths** — Catch `/admin`, `/dashboard`, and internal routes in public indexes
 - ✅ **Flag sensitive files** — Detect database backups, environment files, and archives
+- ✅ **Domain Consistency** — Automatically flag URLs that point to external or incorrect domains (handles `www.` normalization)
+- ✅ **Acceptable Patterns (Allowlist)** — Exclude known safe URLs from being flagged as risks
 - ✅ **Fully Customizable** — Define your own risk categories and patterns using Literal, Glob, or Regex matching
 - ✅ **Fast and automated** — Analyze thousands of URLs in seconds with detailed reports
 
@@ -37,11 +62,17 @@ npm install -g @akotliar/sitemap-qa@alpha
 ### Basic Usage
 
 ```bash
-# Analyze a website's sitemap
+# Step 1: Initialize a configuration file (optional but recommended)
+sitemap-qa init
+
+# Step 2: Analyze a website's sitemap
 sitemap-qa analyze https://example.com
 
-# Generate JSON output for CI/CD
-sitemap-qa analyze https://example.com --output json > report.json
+# Generate JSON output only for CI/CD
+sitemap-qa analyze https://example.com --output json
+
+# Use a custom configuration file
+sitemap-qa analyze https://example.com --config ./custom-config.yaml
 ```
 
 ---
@@ -63,19 +94,11 @@ The tool comes with a set of default policies, but you can fully customize them
 | **Security & Admin** | Detects exposed administrative interfaces and sensitive configuration files. | `**/admin/**`, `**/.env*`, `/wp-admin` |
 | **Environment Leakage** | Finds staging or development URLs that shouldn't be in production sitemaps. | `**/staging.**`, `**/dev.**` |
 | **Sensitive Files** | Flags database backups, archives, and other sensitive file types. | `**/*.{sql,bak,zip,tar}`, `**/*.tar.gz` |
+| **Domain Consistency** | Detects URLs that don't match the target domain (ignoring `www.` differences). | `example.com` vs `other.com` |
 
 ### Customizing Risks
 
-You can add your own categories and patterns to the `sitemap-qa.yaml` file. Patterns support `literal`, `glob`, and `regex` matching.
-
-```yaml
-policies:
-  - category: "Internal API"
-    patterns:
-      - type: "glob"
-        value: "**/api/v1/internal/**"
-        reason: "Internal API version 1 should not be exposed."
-```
+You can add your own categories and patterns to the `sitemap-qa.yaml` file. Patterns support `literal`, `glob`, and `regex` matching. See the [Configuration](#-configuration) section for details.
 
 
 ### Output Formats
@@ -97,7 +120,8 @@ The HTML report provides an interactive, visually appealing view with:
   "summary": {
     "totalUrls": 895,
     "totalRisks": 2,
-    "urlsWithRisksCount": 1
+    "urlsWithRisksCount": 1,
+    "ignoredUrlsCount": 5
   },
   "findings": [
     {
@@ -117,7 +141,44 @@ The HTML report provides an interactive, visually appealing view with:
 
 ---
 
-## 🛠️ CLI Options
+## 🛠️ CLI Commands
+
+Sitemap-QA provides two main commands: `init` and `analyze`.
+
+
+### init Command
+
+Initialize a default `sitemap-qa.yaml` configuration file in the current directory.
+
+```
+Usage: sitemap-qa init [options]
+
+Initialize a default sitemap-qa.yaml configuration file
+
+Options:
+  -h, --help                   Display help for command
+```
+
+#### Example
+
+```bash
+# Create a default configuration file
+sitemap-qa init
+
+# This creates sitemap-qa.yaml with:
+# - Default risk policies (Security & Admin, Environment Leakage, Sensitive Files)
+# - Example acceptable patterns
+# - Default output settings
+```
+
+**Note:** The `init` command will fail if `sitemap-qa.yaml` already exists in the current directory to prevent accidental overwrites.
+
+---
+
+
+### analyze Command
+
+Analyze a website's sitemap for quality issues and security risks.
 
 ```
 Usage: sitemap-qa analyze <url> [options]
@@ -128,13 +189,13 @@ Arguments:
   url                          Base URL of the website to analyze
 
 Options:
-  -c, --config <path>          Path to sitemap-qa.yaml
+  -c, --config <path>          Path to sitemap-qa.yaml configuration file
   -o, --output <format>        Output format: json, html, or all (default: "all")
   -d, --out-dir <path>         Output directory for reports (default: ".")
   -h, --help                   Display help for command
 ```
 
-### Examples
+#### Examples
 
 ```bash
 # Basic analysis with both HTML and JSON reports (default)
@@ -143,14 +204,18 @@ sitemap-qa analyze https://example.com
 # JSON output only
 sitemap-qa analyze https://example.com --output json
 
+# HTML output only
+sitemap-qa analyze https://example.com --output html
+
 # Custom output directory
 sitemap-qa analyze https://example.com --out-dir ./reports
 
 # Use a specific configuration file
 sitemap-qa analyze https://example.com --config ./custom-config.yaml
-```
 
----
+# Combine options
+sitemap-qa analyze https://example.com --config ./custom-config.yaml --output json --out-dir ./reports
+```
 
 ## 🔧 Configuration
 
@@ -161,8 +226,17 @@ Create a `sitemap-qa.yaml` file in your project root to define your monitoring p
 # Default outDir is "."; this example uses a custom reports directory
 outDir: "./sitemap-qa/report" # custom output directory
 outputFormat: "all" # Options: json, html, all
+enforceDomainConsistency: true # Flag URLs from other domains
 
 # Monitoring Policies
+acceptable_patterns:
+  - type: "literal"
+    value: "/acceptable-path"
+    reason: "Example of an acceptable path that should not be flagged."
+  - type: "glob"
+    value: "**/public-docs/**"
+    reason: "Public documentation is always acceptable."
+
 policies:
   - category: "Security & Admin"
     patterns:
@@ -183,35 +257,16 @@ policies:
 |--------|------|---------|-------------|
 | `outDir` | string | `"."` | Directory for generated reports (current working directory by default) |
 | `outputFormat` | string | `"all"` | Report types to generate: `json`, `html`, or `all` |
+| `enforceDomainConsistency` | boolean | `true` | If true, flags URLs that don't match the root sitemap domain (ignoring `www.`) |
+| `acceptable_patterns` | array | `[]` | List of patterns to exclude from risk analysis |
 | `policies` | array | `[]` | List of monitoring policies with patterns |
 
-> Note: The earlier `sitemap-qa.yaml` example sets `outDir: "./sitemap-qa/report"` as a recommended path. If you omit `outDir`, the default is `"."` (the current working directory).
-### Policy Patterns
 
-Define patterns to detect risks in your sitemaps:
+**Priority:** CLI options > Project config (`sitemap-qa.yaml`) > Defaults
 
-```yaml
-policies:
-  - category: "Custom Rules"
-    patterns:
-      - type: "literal"
-        value: "test"
-        reason: "Test URL found"
-      - type: "glob"
-        value: "**/internal/*"
-        reason: "Internal path exposed"
-      - type: "regex"
-        value: "api/v[0-9]/"
-        reason: "API versioning detected"
-```
 
-**Rule Types:**
-- `literal`: Exact string match
-- `glob`: Wildcard patterns (e.g., `**/admin/**`)
-- `regex`: Regular expression matching (patterns are YAML strings and must use proper escaping)
-  - When defining regex patterns in `sitemap-qa.yaml`, remember they are YAML strings, so you must escape backslashes (for example, `".*\\\\.php$"` in YAML corresponds to the regex `.*\.php$`).
 
-**Priority:** CLI options > Project config (`sitemap-qa.yaml`) > Defaults
+---
 
 ## 📝 License
 
@@ -235,7 +290,7 @@ Built with:
 
 ## 📧 Support
 
-- **Issues**: [GitHub Issues](https://github.com/akotliar/sitemap-qa/issues)-
+- **Issues**: [GitHub Issues](https://github.com/akotliar/sitemap-qa/issues)
 
 ---