@intentsolutionsio/penetration-tester 2.0.0 → 3.0.4

package/skills/detecting-directory-listing/SKILL.md

@@ -0,0 +1,206 @@
+---
+name: detecting-directory-listing
+description: |
+  Probe a target for directories that return auto-generated index
+  listings instead of denying or serving a specific file — exposes
+  the full file tree under any reachable directory, including files
+  the application never linked to.
+  Use when: post-deploy verification on a static-asset host, security
+  audit before SOC2, or following up on a finding from skill #6
+  (exposed-files) where a backup-file path returned 200 with HTML
+  body instead of the expected file content (suggests autoindex
+  serving a directory listing).
+  Threshold: any directory-shaped path returns 200 with HTML body
+  matching the framework-specific autoindex fingerprint (nginx
+  fancyindex, Apache mod_autoindex Index of/, Caddy browse, Lighttpd
+  mod_dirlisting, etc.).
+  Trigger with: "directory listing check", "autoindex detection",
+  "open directory scan".
+allowed-tools:
+  - Read
+  - Bash(python3:*)
+  - Bash(curl:*)
+disallowed-tools:
+  - Bash(rm:*)
+  - Edit(/etc/*)
+version: 3.0.0-dev
+author: Jeremy Longshore <jeremy@intentsolutions.io>
+license: MIT
+compatibility: Designed for Claude Code
+tags:
+  - security
+  - information-disclosure
+  - autoindex
+  - pentest
+---
+
+# Detecting Directory Listing
+
+## Overview
+
+Web servers can be configured to auto-generate an HTML index page
+when a request hits a directory without a matching default file
+(no `index.html` / `index.php` / etc.). The auto-generated page
+lists every file in the directory. This is by design for static-
+file servers and FTP-style file-sharing setups; it's a misconfiguration
+on application servers where the file tree was never meant to be
+public.
+
+The risk compounds in two ways:
+
+1. **File enumeration without prior knowledge.** Attackers find files
+   you didn't link to (backup files, log files, old uploads).
+2. **Chaining with exposed-files (#6).** A `/.git/` request that
+   returns an autoindex listing reveals every object file under
+   `.git/objects/` and confirms the entire repo is reachable for
+   GitDumper-style reconstruction.
+
+This skill probes commonly-vulnerable directory paths and grades
+each based on the framework-specific autoindex fingerprint.
+
+## When the skill produces findings
+
+| Finding | Severity | Threshold | Affected control |
+|---|---|---|---|
+| Application-root directory listing | **HIGH** | `/` returns autoindex page | OWASP A05:2021 |
+| Backup / upload / log directory listing | **HIGH** | `/backup/`, `/uploads/`, `/logs/`, `/dump/` autoindex | CWE-548 |
+| Asset directory listing (`/assets/`, `/static/`) | **MEDIUM** | autoindex on asset dirs (enables file enumeration) | CWE-548 |
+| Config directory listing | **CRITICAL** | `/config/`, `/conf/`, `/.config/` autoindex | NIST 800-53 SC-28 |
+| VCS subdir listing (chains with skill #6) | **CRITICAL** | `/.git/`, `/.svn/`, `/.hg/` autoindex | NIST 800-53 SC-28 |
+| Generic root reachable via autoindex | **MEDIUM** | `/` or arbitrary path autoindex on app server | OWASP A05:2021 |
+
+## Prerequisites
+
+- Python 3.9+ with `requests`
+- Authorization for non-local targets
+
+## Instructions
+
+### Step 1 — Confirm Authorization
+
+```text
+"Do you have authorization to perform directory-listing discovery on
+ this target? I need confirmation before proceeding."
+```
+
+### Step 2 — Run the scanner
+
+```bash
+python3 ${CLAUDE_PLUGIN_ROOT}/skills/detecting-directory-listing/scripts/probe_directory_listing.py \
+    https://target.example.com \
+    --authorized
+```
+
+Options:
+
+```
+Usage: probe_directory_listing.py URL [OPTIONS]
+
+Options:
+  --authorized       Attest authorization (required for non-local)
+  --output FILE      Write findings to FILE
+  --format FMT       json | jsonl | markdown (default: markdown)
+  --min-severity SEV (default: info)
+  --timeout SECS     Per-probe timeout (default: 10)
+  --paths-file FILE  Custom probe set (one path per line)
+```
+
+For each candidate directory path, the scanner appends a trailing
+slash and sends a GET. If the response is 200 and the HTML body
+matches one of the canonical autoindex fingerprints (Apache,
+nginx, Caddy, Lighttpd, IIS, Python http.server, Node serve, etc.),
+it's a finding.
+
+### Step 3 — Interpret findings
+
+CRITICAL = config or VCS directory listing → direct credential or
+source-code exposure when combined with skill #6's secret-file
+probe. Ship same-hour fix.
+
+HIGH = backup / upload / log / app-root listing → significant file
+enumeration. Often reveals backup files (`.bak`, `.swp`, `.orig`),
+log files with embedded credentials in URLs, and orphaned uploads
+from old releases. Ship within sprint.
+
+MEDIUM = asset directory listing → file enumeration but bounded
+risk if asset content is genuinely public. Still better to disable
+than to leave on.
+
+### Step 4 — Cross-skill chaining
+
+After this skill, suggest:
+
+- `detecting-exposed-secrets-files` (#6) — if any of the secret-
+  file paths (`.git/`, `.env`) return an autoindex page instead of
+  the expected file, that's the autoindex feature confirming repo
+  exposure.
+- `detecting-debug-endpoints` (#7) — directory listings under
+  framework paths (`/static/`, `/public/`) sometimes expose
+  framework-debug artifacts.
+
+## Examples
+
+### Example 1 — Static-asset host audit
+
+User: "We just spun up a new S3 + CloudFront. Verify autoindex isn't
+on by accident."
+
+```bash
+python3 ${CLAUDE_PLUGIN_ROOT}/skills/detecting-directory-listing/scripts/probe_directory_listing.py \
+    https://cdn.example.com --authorized --min-severity medium
+```
+
+S3 buckets configured with `ListBucket` permissions return XML
+directory listings; the scanner detects those too.
+
+### Example 2 — Following up on .git exposure
+
+User: "Skill #6 found .git/HEAD exposed. Check if the full directory
+is browseable."
+
+```bash
+python3 ${CLAUDE_PLUGIN_ROOT}/skills/detecting-directory-listing/scripts/probe_directory_listing.py \
+    https://app.example.com --authorized --paths-file <(echo .git/)
+```
+
+If the `.git/` probe returns an autoindex listing instead of the
+404 it should, the whole repository is reachable for reconstruction
+via GitDumper-style tools.
+
+### Example 3 — CI gate against autoindex regression
+
+```yaml
+- name: Directory-listing gate
+  run: |
+    python3 plugins/security/penetration-tester/skills/detecting-directory-listing/scripts/probe_directory_listing.py \
+        "${{ secrets.STAGING_URL }}" \
+        --authorized --min-severity high
+```
+
+Exit 1 fails the deploy if any HIGH or CRITICAL autoindex finding
+lands. Catches the regression where a new vhost gets deployed
+without the deny-autoindex directive.
+
+## Output
+
+JSON / JSONL / Markdown. Exit codes: 0 clean, 1 high/critical, 2 error.
+
+## Error Handling
+
+- **Target returns SPA index for every URL** → the scanner's
+  fingerprint check distinguishes a real autoindex from an SPA
+  catch-all. SPAs don't generate autoindex-shaped HTML.
+- **WAF blocks the scanner** → expected behavior on
+  Cloudflare-fronted hosts. The fingerprint will be of the CDN's
+  block page, not the origin.
+- **Connection error** → exit 2.
+
+## Resources
+
+- `references/THEORY.md` — Per-server autoindex behavior, fingerprint
+  patterns, S3 / GCS / Azure Blob considerations
+- `references/PLAYBOOK.md` — Per-server config to disable
+  autoindex (nginx `autoindex off`, Apache `Options -Indexes`,
+  Caddy disabling browse, etc.) + cloud-storage equivalents
+- `../analyzing-tls-config/references/AUTHORIZATION.md` — Active-scan
+  authorization pattern

package/skills/detecting-directory-listing/references/PLAYBOOK.md

@@ -0,0 +1,277 @@
+# Directory-Listing Remediation Playbook
+
+## Apache mod_autoindex
+
+### Disable autoindex globally (preferred)
+
+`httpd.conf` or vhost:
+
+```apache
+<Directory /var/www/html>
+    Options -Indexes
+    AllowOverride None
+    Require all granted
+</Directory>
+```
+
+The `Options -Indexes` line is the critical part. Combined with no
+`index.html` in a directory, requests return 403 (per Apache's
+default fall-through) instead of an autoindex page.
+
+### Per-directory override
+
+If autoindex is needed in one specific directory (rare):
+
+```apache
+<Directory /var/www/html/public-archive>
+    Options +Indexes
+    IndexOptions +FancyIndexing +SuppressDescription
+</Directory>
+```
+
+But this should require a deliberate decision, not a default.
+
+### .htaccess (if `AllowOverride` permits)
+
+```apache
+Options -Indexes
+```
+
+In every directory you want to deny. The `AllowOverride` in vhost
+must include `Options` for this to work.
+
+## nginx autoindex
+
+nginx defaults to autoindex off. The misconfiguration is when
+someone explicitly enabled it:
+
+```nginx
+# In vhost or location block, REMOVE these lines if present:
+autoindex on;
+autoindex_format html;
+autoindex_exact_size off;
+autoindex_localtime on;
+```
+
+To explicitly assert off:
+
+```nginx
+location / {
+    autoindex off;
+    try_files $uri $uri/ =404;
+}
+```
+
+The `try_files` directive ensures missing files return 404 instead
+of any fallback behavior.
+
+## Caddy file_server
+
+Don't use `browse`:
+
+```caddy
+example.com {
+    root * /srv/site
+    file_server         # NO browse keyword
+}
+```
+
+If browse was previously enabled, remove it:
+
+```caddy
+# BEFORE (vulnerable):
+example.com {
+    root * /srv/site
+    file_server browse
+}
+
+# AFTER:
+example.com {
+    root * /srv/site
+    file_server
+    try_files {path} {path}/index.html =404
+}
+```
+
+## Python http.server
+
+Don't use in production. Period. If you absolutely must serve static
+files with python in a constrained environment, use a real server
+behind it OR a subclass that overrides `list_directory`:
+
+```python
+from http.server import HTTPServer, SimpleHTTPRequestHandler
+
+class NoBrowse(SimpleHTTPRequestHandler):
+    def list_directory(self, path):
+        self.send_error(403, "Directory listing not permitted")
+        return None
+
+HTTPServer(('0.0.0.0', 8000), NoBrowse).serve_forever()
+```
+
+## Node serve / http-server
+
+```bash
+# serve — pass --single to disable directory listing
+npx serve --single ./build
+
+# http-server — pass -d false
+npx http-server -d false ./build
+```
+
+Both options return 404 for missing files instead of generating a
+listing page.
+
+## IIS
+
+`web.config`:
+
+```xml
+<system.webServer>
+    <directoryBrowse enabled="false" />
+    <defaultDocument>
+        <files>
+            <add value="index.html" />
+            <add value="default.htm" />
+        </files>
+    </defaultDocument>
+</system.webServer>
+```
+
+## AWS S3
+
+### Block public ListBucket
+
+S3 bucket policy — explicitly deny ListBucket to public:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "DenyPublicList",
+            "Effect": "Deny",
+            "Principal": "*",
+            "Action": "s3:ListBucket",
+            "Resource": "arn:aws:s3:::my-public-cdn-bucket"
+        },
+        {
+            "Sid": "AllowPublicRead",
+            "Effect": "Allow",
+            "Principal": "*",
+            "Action": "s3:GetObject",
+            "Resource": "arn:aws:s3:::my-public-cdn-bucket/*"
+        }
+    ]
+}
+```
+
+Even better: enable S3 Block Public Access at the account level:
+
+```bash
+aws s3api put-public-access-block --bucket my-bucket \
+    --public-access-block-configuration \
+    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
+```
+
+This blocks ALL public access at the bucket level. To then serve
+content publicly, do it via CloudFront with Origin Access Control,
+NOT via direct S3 URLs.
+
+## AWS CloudFront in front of S3
+
+When using CloudFront with Origin Access Control (recommended
+pattern):
+
+1. S3 bucket policy denies all public access.
+2. Only the OAC's CloudFront distribution can read objects.
+3. CloudFront's behavior settings determine what's reachable.
+
+```bash
+# Lock down the bucket
+aws s3api put-bucket-policy --bucket my-bucket --policy '{
+    "Version": "2012-10-17",
+    "Statement": [{
+        "Sid": "AllowCloudFrontOAC",
+        "Effect": "Allow",
+        "Principal": {"Service": "cloudfront.amazonaws.com"},
+        "Action": "s3:GetObject",
+        "Resource": "arn:aws:s3:::my-bucket/*",
+        "Condition": {
+            "StringEquals": {
+                "AWS:SourceArn": "arn:aws:cloudfront::ACCOUNT:distribution/DIST_ID"
+            }
+        }
+    }]
+}'
+```
+
+## GCS
+
+```bash
+# Remove allUsers from objectViewer role
+gsutil iam ch -d allUsers:objectViewer gs://my-bucket
+
+# Make individual objects publicly readable instead (per-object ACL)
+gsutil acl ch -u AllUsers:R gs://my-bucket/path/to/specific-file.png
+```
+
+Or use a Cloud CDN + signed URLs for true private serving.
+
+## Azure Blob
+
+```bash
+# Set container access level to Private (not Container)
+az storage container set-permission \
+    --account-name myaccount \
+    --name mycontainer \
+    --public-access off
+```
+
+Then use SAS tokens or Azure CDN with private origin for public-facing
+content.
+
+## CI integration
+
+```yaml
+- name: Directory-listing posture gate
+  run: |
+    python3 plugins/security/penetration-tester/skills/detecting-directory-listing/scripts/probe_directory_listing.py \
+        "${{ secrets.PROD_URL }}" \
+        --authorized \
+        --min-severity high \
+        --format json \
+        --output autoindex-report.json
+- run: |
+    if jq 'any(.severity == "critical" or .severity == "high")' autoindex-report.json | grep -q true; then
+      echo "::error::Directory listing detected"
+      exit 1
+    fi
+```
+
+## After remediation: assume enumeration occurred
+
+If autoindex was exposed for any period:
+
+- For `/backup/` listings: assume all backup files were enumerated.
+  Rotate any credentials referenced in those files. Investigate
+  whether any backup files contained PII that needs breach
+  notification.
+- For `/uploads/` listings: assume the file index was scraped.
+  If any uploaded files were private (e.g., user-uploaded
+  documents), assume those filenames are known to attackers
+  even if the file contents required separate auth to fetch.
+- For `/.git/` listings: assume the entire repository was
+  reconstructed. Rotate every credential ever committed to the
+  repo, including credentials in past commits that were later
+  removed.
+
+## Verification after remediation
+
+```bash
+python3 ${CLAUDE_PLUGIN_ROOT}/skills/detecting-directory-listing/scripts/probe_directory_listing.py \
+    https://example.com --authorized --min-severity info
+```
+
+Expected: exit 0, zero findings of any severity.

package/skills/detecting-directory-listing/references/THEORY.md

@@ -0,0 +1,203 @@
+# Directory-Listing Theory
+
+## The recurring pattern
+
+Web servers default-handle a directory request in one of three ways:
+
+1. **Serve the directory's `index` file** (`index.html`, `index.php`,
+   etc.) — the safe default.
+2. **Generate an autoindex** — list every file in the directory as
+   HTML. The dangerous default.
+3. **Return 403 / 404** — deny the listing entirely. The defensive
+   default.
+
+Which behavior applies depends on the server's configuration and
+whether an index file is present. The risk surface: a directory
+without an `index.html`, where the server is configured to
+fall back to autoindex.
+
+This was the default in older Apache configurations. Modern stacks
+(Caddy, nginx defaults) deny by default but legacy configs still
+have it enabled.
+
+## Per-server behavior
+
+### Apache mod_autoindex
+
+```apache
+Options +Indexes        # autoindex ENABLED — bad on app servers
+Options -Indexes        # autoindex DISABLED — safe default
+```
+
+The default in Apache 2.4 depends on the distribution: Debian /
+Ubuntu have `Indexes` enabled in the default `<Directory /var/www/>`
+block. Red Hat / CentOS disable it by default.
+
+Fingerprint: `<title>Index of /<path></title>` + an HTML table of
+file entries with mtime + size columns.
+
+### nginx autoindex
+
+```nginx
+location /uploads/ {
+    autoindex on;          # enabled — bad
+}
+location /uploads/ {
+    autoindex off;         # disabled — safe (default)
+}
+```
+
+nginx defaults to autoindex off; you have to explicitly enable it.
+When seen, it's a deliberate config someone added.
+
+Fingerprint: HTML body with `<pre>` formatting, lines like
+`<a href="filename">filename</a>` with size + mtime columns.
+
+### Caddy file_server browse
+
+```caddy
+example.com {
+    root * /srv/site
+    file_server browse    # autoindex enabled
+}
+```
+
+vs.
+
+```caddy
+example.com {
+    root * /srv/site
+    file_server           # no browse — disabled (default)
+}
+```
+
+Caddy v2 defaults to NOT browse. The browse mode generates a styled
+HTML directory listing.
+
+Fingerprint: `<table class="listing">` with sortable column headers.
+
+### Lighttpd mod_dirlisting
+
+```lighttpd
+dir-listing.activate = "enable"
+```
+
+Less common in modern deployments but still found in IoT firmware
+web UIs and embedded server setups.
+
+### Python http.server (dev only)
+
+`python3 -m http.server 8000` — the stdlib dev server defaults to
+autoindex. Should never be in production but sometimes runs on a
+forgotten port.
+
+Fingerprint: `<title>Directory listing for /<path></title>`.
+
+### Node `serve` / `http-server`
+
+`npx serve` and `http-server` both default to autoindex. Common in
+developer-facing tooling, hosting docs sites, or "I just need a
+quick static server" setups.
+
+### IIS
+
+IIS disables directory browsing by default. If enabled in
+`web.config`:
+
+```xml
+<system.webServer>
+    <directoryBrowse enabled="true" />
+</system.webServer>
+```
+
+Fingerprint: HTML body with "Directory browsing" title.
+
+### AWS S3 ListBucket
+
+S3 buckets aren't web servers in the conventional sense but expose
+similar functionality. A bucket policy that grants `s3:ListBucket`
+to the public lets unauthenticated requestors list every object.
+
+Fingerprint: XML response with `<ListBucketResult>` root element,
+listing every key in the bucket.
+
+### Azure Blob Storage list-blob
+
+Same pattern: `?restype=container&comp=list` on a public-list
+container returns XML enumeration of every blob.
+
+Fingerprint: `<EnumerationResults>` root + blob entries with
+properties.
+
+### GCS bucket "list" permission
+
+GCP Cloud Storage equivalent: `storage.objects.list` granted to
+`allUsers` lets anyone enumerate bucket contents.
+
+Fingerprint: similar XML / JSON enumeration depending on whether
+the request hit the JSON or XML API.
+
+## Why directory listings escalate other findings
+
+Directory listings rarely stand alone. They compound the impact of
+other findings:
+
+### Combined with `.git/` exposure
+
+Skill #6 detects `.git/HEAD` reachable. If the parent `.git/`
+directory ALSO has autoindex on, every object in `.git/objects/`
+is enumerable. GitDumper-style tools become unnecessary — you can
+just `wget -r` the directory.
+
+### Combined with backup-file exposure
+
+Skill #6 probes specific backup file paths (`backup.sql`,
+`dump.sql`). Autoindex on `/backup/` lets the attacker see every
+backup file the operator ever left, including ones with non-
+canonical names (`db-snapshot-2024-03-15.sql.bz2`,
+`pre-migration-dump.sql`).
+
+### Combined with upload directories
+
+If `/uploads/` lists, every user-uploaded file is enumerable
+including potentially-sensitive private uploads that the
+application's auth layer was supposed to gate.
+
+## Why static-asset hosts are a common pitfall
+
+A common pattern: front-end is hosted on a CDN (S3 plus CloudFront,
+GCS plus Cloud CDN). The bucket policy is set to "allow public read
+of specific objects" but the BUCKET listing permission is left at
+default, which on S3 with an explicit grant is "public ListBucket."
+
+The fix is to:
+
+1. Make individual objects publicly readable (`s3:GetObject`).
+2. NOT grant `s3:ListBucket` to `*`.
+
+Result: clients can fetch `https://cdn.example.com/assets/logo.png`
+if they know the path, but `https://cdn.example.com/assets/`
+returns AccessDenied.
+
+## Why the SPA case isn't autoindex
+
+Single-Page Applications using client-side routing return the app's
+`index.html` for any unknown route. That's NOT an autoindex page —
+it's a deliberate routing decision. The fingerprint check
+distinguishes:
+
+- Autoindex → HTML body contains "Index of /<path>" or framework
+  banner
+- SPA → HTML body is the application's own UI
+
+A 200 response with HTML body that doesn't match any autoindex
+fingerprint is an SPA catch-all and is NOT flagged.
+
+## Primary sources
+
+- [OWASP WSTG-CONF-04 — Review Old Backup and Unreferenced Files](https://owasp.org/www-project-web-security-testing-guide/v42/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/04-Review_Old_Backup_and_Unreferenced_Files_for_Sensitive_Information)
+- [CWE-548 — Exposure of Information Through Directory Listing](https://cwe.mitre.org/data/definitions/548.html)
+- [nginx autoindex module](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html)
+- [Apache mod_autoindex](https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html)
+- [Caddy file_server directive](https://caddyserver.com/docs/caddyfile/directives/file_server)
+- [AWS S3 — Blocking public access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html)