susbot 2.0.1__tar.gz

susbot-2.0.1/LICENSE.md

@@ -0,0 +1,75 @@
+# PolyForm Noncommercial License 1.0.0
+
+<https://polyformproject.org/licenses/noncommercial/1.0.0>
+
+## Acceptance
+
+In order to get any license under these terms, you must agree to them as both strict obligations and conditions to all your licenses.
+
+## Copyright License
+
+The licensor grants you a copyright license for the software to do everything you might do with the software that would otherwise infringe the licensor's copyright in it for any permitted purpose.  However, you may only distribute the software according to [Distribution License](#distribution-license) and make changes or new works based on the software according to [Changes and New Works License](#changes-and-new-works-license).
+
+## Distribution License
+
+The licensor grants you an additional copyright license to distribute copies of the software.  Your license to distribute covers distributing the software with changes and new works permitted by [Changes and New Works License](#changes-and-new-works-license).
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms or the URL for them above, as well as copies of any plain-text lines beginning with `Required Notice:` that the licensor provided with the software.  For example:
+
+> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
+
+## Changes and New Works License
+
+The licensor grants you an additional copyright license to make changes and new works based on the software for any permitted purpose.
+
+## Patent License
+
+The licensor grants you a patent license for the software that covers patent claims the licensor can license, or becomes able to license, that you would infringe by using the software.
+
+## Noncommercial Purposes
+
+Any noncommercial purpose is a permitted purpose.
+
+## Personal Uses
+
+Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, is use for a permitted purpose.
+
+## Noncommercial Organizations
+
+Use by any charitable organization, educational institution, public research organization, public safety or health organization, environmental protection organization, or government institution is use for a permitted purpose regardless of the source of funding or obligations resulting from the funding.
+
+## Fair Use
+
+You may have "fair use" rights for the software under the law. These terms do not limit them.
+
+## No Other Rights
+
+These terms do not allow you to sublicense or transfer any of your licenses to anyone else, or prevent the licensor from granting licenses to anyone else.  These terms do not imply any other licenses.
+
+## Patent Defense
+
+If you make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.
+
+## Violations
+
+The first time you are notified in writing that you have violated any of these terms, or done anything with the software not covered by your licenses, your licenses can nonetheless continue if you come into full compliance with these terms, and take practical steps to correct past violations, within 32 days of receiving notice.  Otherwise, all your licenses end immediately.
+
+## No Liability
+
+***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.***
+
+## Definitions
+
+The **licensor** is the individual or entity offering these terms, and the **software** is the software the licensor makes available under these terms.
+
+**You** refers to the individual or entity agreeing to these terms.
+
+**Your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization.  **Control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise.  Control can be direct or indirect.
+
+**Your licenses** are all the licenses granted to you for the software under these terms.
+
+**Use** means anything you do with the software requiring one of your licenses.
+
+Required Notice: Copyright 2026 Sitefig (https://sitefig.eu)

susbot-2.0.1/PKG-INFO

@@ -0,0 +1,74 @@
+Metadata-Version: 2.4
+Name: susbot
+Version: 2.0.1
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Rust
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Typing :: Typed
+License-File: LICENSE.md
+Summary: robots.txt checker: which search engines and AI crawlers a site lets in, lint findings, exposed paths and diffs (sus.bot). Python bindings and the susbot command.
+Keywords: robots.txt,crawler,seo,ai,gptbot,rfc9309,audit
+Home-Page: https://sus.bot/
+Author: Sitefig
+License-Expression: PolyForm-Noncommercial-1.0.0
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Homepage, https://sus.bot/
+Project-URL: Issues, https://github.com/sitefig/robots/issues
+Project-URL: Source, https://github.com/sitefig/robots
+
+# susbot
+
+The Python package of [sus.bot](https://sus.bot/), a robots.txt checker. It tells you which search engines and AI crawlers a site lets in, what is wrong with the file, and which paths it gives away. It ships the Rust engine behind sus.bot as Python bindings and the `susbot` command.
+
+```
+pip install susbot
+```
+
+Wheels are built for Linux, macOS and Windows on CPython 3.10 and later.
+
+## Command line
+
+```
+susbot https://example.com                       # audit a live site
+susbot robots.txt --format markdown --out audit.md
+susbot diff old.txt new.txt                      # what changed, crawler by crawler
+python -m susbot --help
+```
+
+It is the same program as `cargo install susbot`. Exit codes: 0 on success, 1 when a `--fail-on` or `--fail-on-security` threshold is met, 2 on errors.
+
+## Python
+
+```python
+import susbot
+
+text = open("robots.txt", encoding="utf-8").read()
+a = susbot.Analysis(text, site_url="https://example.com/")
+
+a.allowed("GPTBot", "/blog/post")        # False
+a.check_access("Googlebot", "/admin/")   # the verdict and the rule that decided it
+a.report["summary"]                      # counts, default policy, AI training, platform
+a.issues                                 # lint findings with stable ids
+a.security                               # disallowed paths that look sensitive
+a.markdown()                             # the client audit; also html(), csv_tabs(), tsv()
+
+old = susbot.Analysis(old_text, site_url="https://example.com/")
+susbot.diff(old, a)                      # crawler verdict flips, new sensitive paths, issues
+```
+
+`Analysis` takes `config=` for a TOML file merged over `susbot.default_config()`, and `locale=` with `lang=` for a translated report. The report follows the [published JSON schema](https://sus.bot/schema/report.schema.json). The package never fetches anything itself; the `susbot` command does.
+
+## Links
+
+- Website: [sus.bot](https://sus.bot/)
+- Source, GitHub Action and issues: [github.com/sitefig/robots](https://github.com/sitefig/robots)
+
+## Licence
+
+[PolyForm Noncommercial 1.0.0](LICENSE.md): free for noncommercial use. Commercial use needs a licence from [Sitefig](https://sitefig.eu/).
+

susbot-2.0.1/README.md

@@ -0,0 +1,50 @@
+# susbot
+
+The Python package of [sus.bot](https://sus.bot/), a robots.txt checker. It tells you which search engines and AI crawlers a site lets in, what is wrong with the file, and which paths it gives away. It ships the Rust engine behind sus.bot as Python bindings and the `susbot` command.
+
+```
+pip install susbot
+```
+
+Wheels are built for Linux, macOS and Windows on CPython 3.10 and later.
+
+## Command line
+
+```
+susbot https://example.com                       # audit a live site
+susbot robots.txt --format markdown --out audit.md
+susbot diff old.txt new.txt                      # what changed, crawler by crawler
+python -m susbot --help
+```
+
+It is the same program as `cargo install susbot`. Exit codes: 0 on success, 1 when a `--fail-on` or `--fail-on-security` threshold is met, 2 on errors.
+
+## Python
+
+```python
+import susbot
+
+text = open("robots.txt", encoding="utf-8").read()
+a = susbot.Analysis(text, site_url="https://example.com/")
+
+a.allowed("GPTBot", "/blog/post")        # False
+a.check_access("Googlebot", "/admin/")   # the verdict and the rule that decided it
+a.report["summary"]                      # counts, default policy, AI training, platform
+a.issues                                 # lint findings with stable ids
+a.security                               # disallowed paths that look sensitive
+a.markdown()                             # the client audit; also html(), csv_tabs(), tsv()
+
+old = susbot.Analysis(old_text, site_url="https://example.com/")
+susbot.diff(old, a)                      # crawler verdict flips, new sensitive paths, issues
+```
+
+`Analysis` takes `config=` for a TOML file merged over `susbot.default_config()`, and `locale=` with `lang=` for a translated report. The report follows the [published JSON schema](https://sus.bot/schema/report.schema.json). The package never fetches anything itself; the `susbot` command does.
+
+## Links
+
+- Website: [sus.bot](https://sus.bot/)
+- Source, GitHub Action and issues: [github.com/sitefig/robots](https://github.com/sitefig/robots)
+
+## Licence
+
+[PolyForm Noncommercial 1.0.0](LICENSE.md): free for noncommercial use. Commercial use needs a licence from [Sitefig](https://sitefig.eu/).

susbot-2.0.1/cli/Cargo.toml

@@ -0,0 +1,30 @@
+[package]
+name = "susbot-cli"
+description = "robots.txt checker for the command line and CI: which search engines and AI crawlers a site lets in, lint findings, exposed paths, diffs (sus.bot)"
+version = "2.0.1"
+edition = "2021"
+license = "PolyForm-Noncommercial-1.0.0"
+repository = "https://github.com/sitefig/robots"
+homepage = "https://sus.bot/"
+authors = ["Sitefig <https://sitefig.eu/>"]
+readme = "README.md"
+# Tests read files from the repository (examples, schema), so they stay out of the package.
+exclude = ["tests/"]
+keywords = ["robots-txt", "crawler", "seo", "ai", "audit"]
+categories = ["command-line-utilities", "web-programming", "development-tools"]
+
+[lib]
+path = "src/lib.rs"
+
+[[bin]]
+name = "susbot"
+path = "src/main.rs"
+
+[dependencies]
+susbot-core = { path = "../core", version = "2.0.1" }
+serde = { version = "^1", features = ["derive"] }
+serde_json = "^1"
+clap = { version = "4", features = ["derive"] }
+ureq = { version = "2", default-features = false, features = ["tls"] }
+url = "^2"
+flate2 = "1"

susbot-2.0.1/cli/LICENSE.md

@@ -0,0 +1,75 @@
+# PolyForm Noncommercial License 1.0.0
+
+<https://polyformproject.org/licenses/noncommercial/1.0.0>
+
+## Acceptance
+
+In order to get any license under these terms, you must agree to them as both strict obligations and conditions to all your licenses.
+
+## Copyright License
+
+The licensor grants you a copyright license for the software to do everything you might do with the software that would otherwise infringe the licensor's copyright in it for any permitted purpose.  However, you may only distribute the software according to [Distribution License](#distribution-license) and make changes or new works based on the software according to [Changes and New Works License](#changes-and-new-works-license).
+
+## Distribution License
+
+The licensor grants you an additional copyright license to distribute copies of the software.  Your license to distribute covers distributing the software with changes and new works permitted by [Changes and New Works License](#changes-and-new-works-license).
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms or the URL for them above, as well as copies of any plain-text lines beginning with `Required Notice:` that the licensor provided with the software.  For example:
+
+> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
+
+## Changes and New Works License
+
+The licensor grants you an additional copyright license to make changes and new works based on the software for any permitted purpose.
+
+## Patent License
+
+The licensor grants you a patent license for the software that covers patent claims the licensor can license, or becomes able to license, that you would infringe by using the software.
+
+## Noncommercial Purposes
+
+Any noncommercial purpose is a permitted purpose.
+
+## Personal Uses
+
+Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, is use for a permitted purpose.
+
+## Noncommercial Organizations
+
+Use by any charitable organization, educational institution, public research organization, public safety or health organization, environmental protection organization, or government institution is use for a permitted purpose regardless of the source of funding or obligations resulting from the funding.
+
+## Fair Use
+
+You may have "fair use" rights for the software under the law. These terms do not limit them.
+
+## No Other Rights
+
+These terms do not allow you to sublicense or transfer any of your licenses to anyone else, or prevent the licensor from granting licenses to anyone else.  These terms do not imply any other licenses.
+
+## Patent Defense
+
+If you make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.
+
+## Violations
+
+The first time you are notified in writing that you have violated any of these terms, or done anything with the software not covered by your licenses, your licenses can nonetheless continue if you come into full compliance with these terms, and take practical steps to correct past violations, within 32 days of receiving notice.  Otherwise, all your licenses end immediately.
+
+## No Liability
+
+***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.***
+
+## Definitions
+
+The **licensor** is the individual or entity offering these terms, and the **software** is the software the licensor makes available under these terms.
+
+**You** refers to the individual or entity agreeing to these terms.
+
+**Your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization.  **Control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise.  Control can be direct or indirect.
+
+**Your licenses** are all the licenses granted to you for the software under these terms.
+
+**Use** means anything you do with the software requiring one of your licenses.
+
+Required Notice: Copyright 2026 Sitefig (https://sitefig.eu)

susbot-2.0.1/cli/README.md

@@ -0,0 +1,46 @@
+# susbot
+
+The command-line version of [sus.bot](https://sus.bot/), a robots.txt checker. It reads a site's robots.txt and reports which search engines and AI crawlers may visit which pages, what is wrong with the file, and which paths it reveals to anyone who reads it. The same engine runs the website, so results match.
+
+## Install
+
+```
+cargo install susbot
+```
+
+This installs a program called `susbot`. `cargo install susbot-cli` installs the same program.
+
+## Use
+
+```
+susbot https://example.com                         # summary with issues, security findings and recommended actions
+susbot https://example.com --format markdown --out audit.md
+susbot https://example.com --format json           # the full report, see the schema in the repository
+susbot robots.txt --site-url https://example.com   # a local file
+susbot https://example.com --fail-on warning       # exit 1 in CI when there are warnings or errors
+susbot https://example.com --fail-on-security high # exit 1 when a high-risk path is exposed
+susbot https://example.com --access-check          # refetch with every crawler's user-agent
+susbot --print-default-config > my-rules.toml      # every rule, to adjust and pass with --config
+```
+
+Other commands:
+
+```
+susbot diff old.txt new.txt        # what changed for crawlers, new sensitive paths, issues, sitemaps
+susbot track --config sites.json --data-dir data   # keep snapshots of many sites and report changes
+susbot crawl --input domains.csv --out audit.jsonl.gz   # audit a list of domains into a dataset
+```
+
+Exit codes: 0 when all is well, 1 when findings reach `--fail-on` or `--fail-on-security`, 2 on fetch or configuration errors.
+
+Reports are in English by default. For another EU language, pass `--lang` with `--locale-dir` pointing at the `locales` folder of the [repository](https://github.com/sitefig/robots).
+
+## More
+
+- Website: [sus.bot](https://sus.bot/)
+- GitHub Action and source: [github.com/sitefig/robots](https://github.com/sitefig/robots)
+- The engine on its own: [susbot-core](https://crates.io/crates/susbot-core)
+
+## Licence
+
+[PolyForm Noncommercial 1.0.0](LICENSE.md): free for noncommercial use. Commercial use, including running it in a company's CI, needs a licence from [Sitefig](https://sitefig.eu/).

susbot-2.0.1/cli/src/audit.rs

@@ -0,0 +1,234 @@
+//! `susbot audit`: one file or URL, every check, any export.
+
+use crate::net::{fetch_robots, iso_now};
+use clap::{Args, ValueEnum};
+use std::io::Read;
+use std::path::PathBuf;
+use susbot_core::model::Level;
+use susbot_core::{Analysis, Options};
+
+#[derive(Clone, Copy, ValueEnum, Debug)]
+pub enum Format {
+    /// Machine-readable report (schema/report.schema.json)
+    Json,
+    /// Client audit in Markdown
+    Markdown,
+    /// The audit as a standalone HTML document
+    Html,
+    /// All spreadsheet tabs as one tagged CSV
+    Csv,
+    /// Recommended actions and issue counts, for logs
+    Summary,
+}
+
+#[derive(Clone, Copy, ValueEnum, Debug, PartialEq)]
+pub enum FailOn {
+    Never,
+    Error,
+    Warning,
+}
+
+#[derive(Clone, Copy, ValueEnum, Debug, PartialEq)]
+pub enum Severity {
+    Never,
+    High,
+    Medium,
+    Low,
+}
+
+#[derive(Args, Debug)]
+pub struct AuditArgs {
+    /// Site URL (https://example.com), a path to a robots.txt file, or "-" for stdin
+    #[arg(required_unless_present = "print_default_config", default_value = "")]
+    pub target: String,
+    /// TOML config merged over the built-in rules (see config/default.toml)
+    #[arg(short, long)]
+    pub config: Option<PathBuf>,
+    /// Print the built-in configuration and exit
+    #[arg(long)]
+    pub print_default_config: bool,
+    /// Output format
+    #[arg(short, long, value_enum, default_value = "summary")]
+    pub format: Format,
+    /// Write the output to a file instead of stdout
+    #[arg(short, long)]
+    pub out: Option<PathBuf>,
+    /// Language code for messages; loads <lang>.json from --locale-dir, ./locales or next to the binary
+    #[arg(long, default_value = "en")]
+    pub lang: String,
+    /// Directory holding <lang>.json dictionaries
+    #[arg(long)]
+    pub locale_dir: Option<PathBuf>,
+    /// Site URL assumed for a local file, enabling origin-dependent checks
+    #[arg(long)]
+    pub site_url: Option<String>,
+    /// Exit 1 when an issue at this level or worse is present
+    #[arg(long, value_enum, default_value = "never")]
+    pub fail_on: FailOn,
+    /// Exit 1 when a security finding at this severity or worse is present
+    #[arg(long, value_enum, default_value = "never")]
+    pub fail_on_security: Severity,
+    /// Also refetch the file with every crawler's real User-Agent and report differences
+    #[arg(long)]
+    pub access_check: bool,
+    /// Request timeout in seconds
+    #[arg(long, default_value = "15")]
+    pub timeout: u64,
+}
+
+pub fn load_locale(lang: &str, locale_dir: Option<&PathBuf>) -> Option<String> {
+    if lang == "en" {
+        return None;
+    }
+    let mut candidates = Vec::new();
+    if let Some(d) = locale_dir {
+        candidates.push(d.join(format!("{lang}.json")));
+    }
+    candidates.push(PathBuf::from("locales").join(format!("{lang}.json")));
+    if let Ok(exe) = std::env::current_exe() {
+        if let Some(dir) = exe.parent() {
+            candidates.push(dir.join("locales").join(format!("{lang}.json")));
+        }
+    }
+    let found = candidates.iter().find_map(|p| std::fs::read_to_string(p).ok());
+    if found.is_none() {
+        eprintln!("susbot: no dictionary for \"{lang}\" found (looked in {}); messages fall back to English", candidates.iter().map(|p| p.display().to_string()).collect::<Vec<_>>().join(", "));
+    }
+    found
+}
+
+fn access_check(origin: &str, analysis: &Analysis, timeout: u64) -> String {
+    let crawlers = &analysis.engine.config.crawlers;
+    let robots_url = format!("{origin}/robots.txt");
+    let client = ureq::AgentBuilder::new().timeout(std::time::Duration::from_secs(timeout)).build();
+    let get = |ua: &str| -> (u16, String) {
+        match client.get(&robots_url).set("User-Agent", ua).call() {
+            Ok(r) => {
+                let s = r.status();
+                (s, r.into_string().unwrap_or_default())
+            }
+            Err(ureq::Error::Status(s, r)) => (s, r.into_string().unwrap_or_default()),
+            Err(e) => (0, e.to_string()),
+        }
+    };
+    let (base_status, base_body) = get(&crawlers.browser_ua);
+    let mut lines = vec![format!("{:<28} {:>6}  {}", "user-agent", "status", "body vs browser"), format!("{:<28} {:>6}  -", "browser (baseline)", base_status)];
+    for c in crawlers.list.iter().filter(|c| c.ua.is_some()) {
+        let (s, body) = get(c.ua.as_deref().unwrap());
+        let diff = if s != base_status {
+            "status differs"
+        } else if body != base_body {
+            "body differs"
+        } else {
+            "same"
+        };
+        lines.push(format!("{:<28} {:>6}  {diff}", c.name, s));
+    }
+    lines.join("\n")
+}
+
+pub fn run(cli: &AuditArgs) -> Result<u8, String> {
+    if cli.print_default_config {
+        print!("{}", susbot_core::config::DEFAULT_TOML);
+        return Ok(0);
+    }
+    if cli.target.is_empty() {
+        return Err("a target is required: a URL, a file, or - for stdin".into());
+    }
+    let config = match &cli.config {
+        Some(p) => Some(std::fs::read_to_string(p).map_err(|e| format!("{}: {e}", p.display()))?),
+        None => None,
+    };
+    let locale = load_locale(&cli.lang, cli.locale_dir.as_ref());
+    let is_url = cli.target.starts_with("http://") || cli.target.starts_with("https://") || (!cli.target.contains('/') && cli.target.contains('.') && !std::path::Path::new(&cli.target).exists());
+    let (text, fetch, site_url, origin) = if cli.target == "-" {
+        let mut s = String::new();
+        std::io::stdin().read_to_string(&mut s).map_err(|e| e.to_string())?;
+        (s, None, cli.site_url.clone(), None)
+    } else if is_url {
+        let raw = if cli.target.contains("://") { cli.target.clone() } else { format!("https://{}", cli.target) };
+        let u = url::Url::parse(&raw).map_err(|e| format!("{raw}: {e}"))?;
+        let origin = u.origin().ascii_serialization();
+        let engine = susbot_core::Engine::from_toml(config.as_deref())?;
+        let f = fetch_robots(&origin, &engine.config.crawlers.browser_ua, cli.timeout)?;
+        let text = f.text.clone();
+        let site = f.final_url.clone();
+        (text, Some(f), Some(site), Some(origin))
+    } else {
+        let s = std::fs::read_to_string(&cli.target).map_err(|e| format!("{}: {e}", cli.target))?;
+        (s, None, cli.site_url.clone(), None)
+    };
+
+    let analysis = Analysis::new(&text, Options { site_url, fetch, config, lang: Some(cli.lang.clone()), locale, now: Some(iso_now()), schema_url: Some("https://sus.bot/schema/report.schema.json".into()) })?;
+    let r = &analysis.report;
+
+    let mut output = match cli.format {
+        Format::Json => analysis.report_json_pretty() + "\n",
+        Format::Markdown => analysis.markdown(),
+        Format::Html => analysis.html(),
+        Format::Csv => analysis.tagged_csv(),
+        Format::Summary => {
+            let mut s = format!(
+                "{}\n  errors {}, warnings {}, notes {}; security findings {}; AI training crawlers blocked {}/{}\n\n",
+                r.source.final_url.clone().or(r.source.site_url.clone()).unwrap_or_else(|| cli.target.clone()),
+                r.summary.issues.errors,
+                r.summary.issues.warnings,
+                r.summary.issues.notes,
+                r.summary.security_findings,
+                r.summary.ai_training.blocked,
+                r.summary.ai_training.total
+            );
+            let mut issues: Vec<_> = r.issues.iter().filter(|i| i.level != Level::Info).collect();
+            issues.sort_by(|a, b| a.level.cmp(&b.level).then(a.line.unwrap_or(u32::MAX).cmp(&b.line.unwrap_or(u32::MAX))));
+            for i in issues {
+                s.push_str(&format!("{:<7} {:>5}  {}\n", i.level.as_str(), i.line.map(|l| format!("L{l}")).unwrap_or_default(), i.message));
+            }
+            for f in &r.security {
+                s.push_str(&format!("{:<7} {:>5}  Disallow: {} ({}, {})\n", f.severity, format!("L{}", f.line), f.path, f.category, f.reason));
+            }
+            s.push_str("\nRecommended actions:\n");
+            for (i, a) in analysis.recommended_actions().iter().enumerate() {
+                s.push_str(&format!("{}. {a}\n", i + 1));
+            }
+            s
+        }
+    };
+    if cli.access_check {
+        if let Some(origin) = &origin {
+            let table = access_check(origin, &analysis, cli.timeout);
+            if matches!(cli.format, Format::Summary | Format::Markdown) {
+                output.push_str("\nReal access by user-agent:\n");
+                output.push_str(&table);
+                output.push('\n');
+            } else {
+                eprintln!("{table}");
+            }
+        } else {
+            eprintln!("susbot: --access-check needs a URL target");
+        }
+    }
+    match &cli.out {
+        Some(p) => std::fs::write(p, output).map_err(|e| format!("{}: {e}", p.display()))?,
+        None => print!("{output}"),
+    }
+
+    let worst_issue = r.issues.iter().map(|i| i.level).min();
+    let fail_issue = match cli.fail_on {
+        FailOn::Never => false,
+        FailOn::Error => worst_issue == Some(Level::Error),
+        FailOn::Warning => matches!(worst_issue, Some(Level::Error) | Some(Level::Warning)),
+    };
+    let rank = |s: &str| match s {
+        "high" => 0,
+        "medium" => 1,
+        _ => 2,
+    };
+    let worst_sec = r.security.iter().map(|f| rank(&f.severity)).min();
+    let fail_sec = match cli.fail_on_security {
+        Severity::Never => false,
+        Severity::High => worst_sec == Some(0),
+        Severity::Medium => matches!(worst_sec, Some(0) | Some(1)),
+        Severity::Low => worst_sec.is_some(),
+    };
+    Ok(if fail_issue || fail_sec { 1 } else { 0 })
+}