osintkit 0.4.0__tar.gz → 0.5.0__tar.gz

{osintkit-0.4.0 → osintkit-0.5.0}/CHANGELOG.md

@@ -6,6 +6,21 @@ All notable changes are recorded here. The format follows
 
 ## Unreleased
 
+## 0.5.0 — 2026-11-02
+
+### Added
+
+- breach rows are enriched with date, exposed fields, domain, and record count from the keyless
+  HIBP and XposedOrNot catalogues
+- `osintkit password` checks Pwned Passwords by SHA-1 prefix; no key, no plaintext sent
+- email lookups also collect the address domain's DNS, mail posture, and WHOIS, marked as derived
+- a failed synthesis now falls back to a locally built report instead of an empty one
+
+### Fixed
+
+- large prompts came back empty on reasoning models: token truncation is detected and retried,
+  and the prompt carries at most 15 breach rows
+
 ## 0.4.0 — 2026-11-02
 
 ### Added

{osintkit-0.4.0 → osintkit-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: osintkit
-Version: 0.4.0
+Version: 0.5.0
 Summary: Public-data OSINT aggregator for emails, usernames, domains, and companies
 Project-URL: Homepage, https://github.com/sjdsjadjajdjsa/osintkit
 Project-URL: Issues, https://github.com/sjdsjadjajdjsa/osintkit/issues
@@ -103,6 +103,7 @@ Optional keys: `HIBP_API_KEY` turns on Have I Been Pwned, `GITHUB_TOKEN` lifts G
 | `lookup` | One target, one report (`--email`, `--username`, `--domain`, `--company`) |
 | `batch` | A CSV of targets, run concurrently, reported in order |
 | `diff` | What changed between two archived runs |
+| `password` | Is a password in Pwned Passwords (free, no key, hash-prefix only) |
 | `history` | Past runs, replayable by id |
 | `doctor` | Key, endpoint, and directory checks (`--deep` verifies auth) |
 | `collectors` | Every collector and what it touches |
@@ -132,8 +133,11 @@ target -> type detection -> collectors (parallel, cached, retried)
 Collectors degrade to a logged gap, never a dead run; a report is produced even when sources or
 the model fail.
 
-- **Merged breaches.** Three indexes listing one incident collapse into one row with every
-  source and the earliest date.
+- **Merged, enriched breaches.** Records collapse into one row per incident, keeping every
+  source and the earliest date, then pick up dates and exposed-field lists from the public HIBP
+  and XposedOrNot catalogues. Those catalogues are free; only HIBP *account* checks need a key.
+- **Email domains follow the address.** An email lookup also checks the domain's DNS, mail
+  posture, and WHOIS, marked as derived.
 - **Risk floor.** Breach count and credential types set a floor the model may exceed but never
   undercut.
 - **Cached everything.** 24h default, shorter for volatile sources; repeat prompts cost nothing.
@@ -148,6 +152,17 @@ connect probes, GitHub (profile, commits, user search, org repos), psbdmp with a
 grep.app, Hacker News, Stack Exchange, and username presence. The full table is in
 [docs/REFERENCE.md](docs/REFERENCE.md#collectors) or via `osintkit collectors`.
 
+## Breach depth
+
+Breach name lists are free; breach *metadata* is what makes them useful. Every row is joined
+against the public HIBP and XposedOrNot catalogues for date, exposed fields, and record count, so
+the report can say "LinkedIn, 2021-06, emails and password hashes" instead of a bare name. Add a
+`HIBP_API_KEY` (about $4/month) to get per-account membership with the same metadata, plus pastes.
+
+Passwords are checked separately with `osintkit password`: Pwned Passwords covers hundreds of
+millions of leaked passwords, needs no key, and only the first five SHA-1 characters leave the
+machine.
+
 ## Safety and legality
 
 - Public data only. No logins, no scraping behind auth, no paid brokers.

{osintkit-0.4.0 → osintkit-0.5.0}/README.md

@@ -71,6 +71,7 @@ Optional keys: `HIBP_API_KEY` turns on Have I Been Pwned, `GITHUB_TOKEN` lifts G
 | `lookup` | One target, one report (`--email`, `--username`, `--domain`, `--company`) |
 | `batch` | A CSV of targets, run concurrently, reported in order |
 | `diff` | What changed between two archived runs |
+| `password` | Is a password in Pwned Passwords (free, no key, hash-prefix only) |
 | `history` | Past runs, replayable by id |
 | `doctor` | Key, endpoint, and directory checks (`--deep` verifies auth) |
 | `collectors` | Every collector and what it touches |
@@ -100,8 +101,11 @@ target -> type detection -> collectors (parallel, cached, retried)
 Collectors degrade to a logged gap, never a dead run; a report is produced even when sources or
 the model fail.
 
-- **Merged breaches.** Three indexes listing one incident collapse into one row with every
-  source and the earliest date.
+- **Merged, enriched breaches.** Records collapse into one row per incident, keeping every
+  source and the earliest date, then pick up dates and exposed-field lists from the public HIBP
+  and XposedOrNot catalogues. Those catalogues are free; only HIBP *account* checks need a key.
+- **Email domains follow the address.** An email lookup also checks the domain's DNS, mail
+  posture, and WHOIS, marked as derived.
 - **Risk floor.** Breach count and credential types set a floor the model may exceed but never
   undercut.
 - **Cached everything.** 24h default, shorter for volatile sources; repeat prompts cost nothing.
@@ -116,6 +120,17 @@ connect probes, GitHub (profile, commits, user search, org repos), psbdmp with a
 grep.app, Hacker News, Stack Exchange, and username presence. The full table is in
 [docs/REFERENCE.md](docs/REFERENCE.md#collectors) or via `osintkit collectors`.
 
+## Breach depth
+
+Breach name lists are free; breach *metadata* is what makes them useful. Every row is joined
+against the public HIBP and XposedOrNot catalogues for date, exposed fields, and record count, so
+the report can say "LinkedIn, 2021-06, emails and password hashes" instead of a bare name. Add a
+`HIBP_API_KEY` (about $4/month) to get per-account membership with the same metadata, plus pastes.
+
+Passwords are checked separately with `osintkit password`: Pwned Passwords covers hundreds of
+millions of leaked passwords, needs no key, and only the first five SHA-1 characters leave the
+machine.
+
 ## Safety and legality
 
 - Public data only. No logins, no scraping behind auth, no paid brokers.

{osintkit-0.4.0 → osintkit-0.5.0}/docs/REFERENCE.md

@@ -61,6 +61,19 @@ Explicit `--base-url` and `--model` beat both the provider and the prefix. Tempe
 Per-command: `--json`, `--markdown`, `-o/--out`, `--raw`, `--offline`, `--dry-run`, `--only`, `--skip`,
 `--ports`, `--fail-fast`, `-c/--concurrency`, `--limit`, `--show`, `--days`, `--clear`, `--deep`.
 
+## Breach catalogue enrichment
+
+Breach names from free indexes are joined against `haveibeenpwned.com/api/v3/breaches` (keyless
+catalogue) with an XposedOrNot catalogue fallback, cached a week, to add dates, exposed fields,
+domains, and record counts. At most 15 enriched rows go into the prompt; the true count stays in
+`signals`.
+
+## Password checks
+
+`osintkit password [VALUE]` hashes locally, sends only the five-character SHA-1 prefix to
+`api.pwnedpasswords.com`, and reports the count and a commonness rating. Exit code `1` means
+pwned. `--json` for machines. Nothing about the password beyond the prefix leaves the machine.
+
 ## Configuration file
 
 `osintkit init` writes `~/.config/osintkit/config.toml` (or `$XDG_CONFIG_HOME`) with mode 600 in

{osintkit-0.4.0 → osintkit-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "osintkit"
-version = "0.4.0"
+version = "0.5.0"
 description = "Public-data OSINT aggregator for emails, usernames, domains, and companies"
 readme = "README.md"
 requires-python = ">=3.12"

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/__init__.py

@@ -1,3 +1,3 @@
 """osintkit: a public-data OSINT aggregator."""
 
-__version__ = "0.4.0"
+__version__ = "0.5.0"

osintkit-0.5.0/src/osintkit/catalogue.py

@@ -0,0 +1,94 @@
+"""Public breach catalogues used to enrich bare breach names."""
+
+from __future__ import annotations
+
+import re
+from typing import Any
+
+from .errors import OsintKitError
+from .http import HttpClient, cache_hit, cache_put
+
+NAMESPACE = "catalogue"
+TTL = 7 * 24 * 3600
+KEY_RE = re.compile(r"[^a-z0-9]+")
+
+HIBP = "https://haveibeenpwned.com/api/v3/breaches"
+XPOSED = "https://api.xposedornot.com/v1/breaches"
+
+
+def _key(name: str) -> str:
+    return KEY_RE.sub("", name.lower())
+
+
+def _from_hibp(rows: Any) -> dict[str, dict[str, Any]]:
+    if not isinstance(rows, list):
+        return {}
+    out: dict[str, dict[str, Any]] = {}
+    for row in rows:
+        if not isinstance(row, dict):
+            continue
+        name = str(row.get("Name") or row.get("Title") or "").strip()
+        if not name:
+            continue
+        out[_key(name)] = {
+            "name": name,
+            "date": row.get("BreachDate"),
+            "data_types": [str(kind) for kind in row.get("DataClasses") or []],
+            "domain": row.get("Domain"),
+            "pwn_count": row.get("PwnCount"),
+        }
+    return out
+
+
+def _from_xposed(body: Any) -> dict[str, dict[str, Any]]:
+    rows = body.get("exposedBreaches") if isinstance(body, dict) else None
+    if not isinstance(rows, list):
+        return {}
+    out: dict[str, dict[str, Any]] = {}
+    for row in rows:
+        if not isinstance(row, dict):
+            continue
+        name = str(row.get("breachID") or "").strip()
+        if not name:
+            continue
+        out[_key(name)] = {
+            "name": name,
+            "date": (row.get("breachedDate") or "")[:10] or None,
+            "data_types": [str(kind) for kind in row.get("exposedData") or []],
+            "domain": row.get("domain"),
+            "pwn_count": row.get("exposedRecords"),
+        }
+    return out
+
+
+async def _download(http: HttpClient) -> dict[str, dict[str, Any]]:
+    try:
+        response = await http.get(HIBP, namespace=NAMESPACE, accept="application/json")
+        if response.status_code == 200:
+            data = _from_hibp(response.json())
+            if data:
+                return data
+    except (OsintKitError, ValueError):
+        pass
+
+    try:
+        response = await http.get(XPOSED, namespace=NAMESPACE, accept="application/json")
+    except OsintKitError:
+        return {}
+    if response.status_code != 200:
+        return {}
+    try:
+        return _from_xposed(response.json())
+    except ValueError:
+        return {}
+
+
+async def fetch(http: HttpClient) -> dict[str, dict[str, Any]]:
+    """Return breach metadata keyed by normalised name, cached for a week."""
+    cached = cache_hit(http.settings, NAMESPACE, "index", TTL)
+    if cached is not None:
+        return cached
+    data = await _download(http)
+    if data:
+        cache_put(http.settings, NAMESPACE, "index", data)
+    return data

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/cli.py

@@ -18,7 +18,7 @@ from rich.console import Console
 from rich.panel import Panel
 from rich.table import Table
 
-from . import __version__, diff, doctor, pipeline, render, retention, userconfig
+from . import __version__, diff, doctor, passwords, pipeline, render, retention, userconfig
 from .collectors import REGISTRY
 from .config import (
     DEFAULT_RESOLVERS,
@@ -74,7 +74,7 @@ def root(
     model: str | None = typer.Option(None, "--model", help="Model name (overrides the provider default)."),
     api_key: str | None = typer.Option(None, "--api-key", help="API key (or set API_KEY)."),
     temperature: float = typer.Option(0.3, "--temperature", min=0.0, max=2.0),
-    max_tokens: int = typer.Option(2400, "--max-tokens", min=256, max=16000),
+    max_tokens: int = typer.Option(4000, "--max-tokens", min=256, max=16000),
     timeout: float = typer.Option(30.0, "--timeout", help="HTTP timeout in seconds."),
     max_connections: int = typer.Option(16, "--max-connections", min=1, max=64),
     max_attempts: int = typer.Option(3, "--max-attempts", min=1, max=6),
@@ -627,6 +627,50 @@ def init(
     out.print("[dim]try: osintkit lookup --domain example.com[/dim]")
 
 
+@app.command()
+def password(
+    value: str | None = typer.Argument(None, help="Password to check; prompt if omitted."),
+    json_out: bool = typer.Option(False, "--json", help="JSON output."),
+) -> None:
+    """Check a password against Pwned Passwords, without sending it."""
+    out = _console()
+    secret = value
+    if not secret:
+        if not sys.stdin.isatty():
+            secret = sys.stdin.readline().rstrip("\n")
+        if not secret:
+            secret = typer.prompt("Password", hide_input=True)
+    if not secret:
+        out.print("[red]no password given[/red]")
+        raise typer.Exit(2)
+
+    settings: Settings = STATE["settings"]  # type: ignore[assignment]
+
+    async def run() -> dict[str, object]:
+        from .http import HttpClient
+
+        async with HttpClient(settings) as http:
+            return await passwords.check(http, secret)
+
+    try:
+        result = asyncio.run(run())
+    except OsintKitError as exc:
+        out.print(f"[red]{exc}[/red]")
+        raise typer.Exit(1) from exc
+
+    if json_out:
+        print(json.dumps(result, indent=2))
+        return
+    if result["pwned"]:
+        out.print(
+            f"[red]pwned[/red] {result['count']:,} times "
+            f"({result['rating']}); still only the first 5 SHA-1 characters left this machine"
+        )
+        raise typer.Exit(1)
+    out.print(f"[green]not seen[/green] in Pwned Passwords (checked by hash prefix {result['prefix']})")
+    out.print("[yellow]absence is not proof of strength; length and uniqueness still matter[/yellow]")
+
+
 @app.command("config")
 def config_cmd(
     reveal: bool = typer.Option(False, "--reveal", help="Show the key in full."),

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/config.py

@@ -110,7 +110,7 @@ class Settings:
     model: str = DEFAULT_MODEL
     api_key: str | None = None
     temperature: float = DEFAULT_TEMPERATURE
-    max_tokens: int = 2400
+    max_tokens: int = 4000
     timeout: float = DEFAULT_TIMEOUT
     max_connections: int = 16
     max_attempts: int = DEFAULT_MAX_ATTEMPTS

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/llm.py

@@ -34,7 +34,7 @@ class LLMClient:
         messages: Sequence[Mapping[str, str]],
         *,
         temperature: float | None = None,
-        max_tokens: int = 2400,
+        max_tokens: int = 4000,
         response_format: Mapping[str, Any] | None = None,
     ) -> str:
         api_key = self.settings.resolved_api_key()
@@ -51,6 +51,15 @@ class LLMClient:
         if response_format:
             payload["response_format"] = dict(response_format)
 
+        content, finish = await self._post(payload, api_key)
+        if finish == "length" and not content and max_tokens < 8000:
+            payload["max_tokens"] = min(8000, max_tokens * 2)
+            content, finish = await self._post(payload, api_key)
+        if finish == "length" and not content:
+            raise SynthesisError("model ran out of tokens before answering")
+        return content
+
+    async def _post(self, payload: dict[str, Any], api_key: str | None) -> tuple[str, str | None]:
         try:
             response = await self._http.post_json(
                 self.endpoint(),
@@ -69,7 +78,17 @@ class LLMClient:
             body = response.json()
         except ValueError as exc:
             raise SynthesisError("Model endpoint returned non-JSON") from exc
-        return extract_content(body)
+        return extract_content(body), extract_finish(body)
+
+
+def extract_finish(body: Any) -> str | None:
+    if not isinstance(body, dict):
+        return None
+    choices = body.get("choices")
+    if not isinstance(choices, list) or not choices:
+        return None
+    finish = choices[0].get("finish_reason")
+    return finish if isinstance(finish, str) else None
 
 
 def extract_content(body: Any) -> str:
@@ -86,6 +105,6 @@ def extract_content(body: Any) -> str:
     if isinstance(content, list):
         parts = [c.get("text", "") for c in content if isinstance(c, dict)]
         content = "".join(parts)
-    if not isinstance(content, str) or not content.strip():
-        raise SynthesisError("Model returned empty content")
+    if not isinstance(content, str):
+        return ""
     return content

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/normalise.py

@@ -11,6 +11,7 @@ BREACH_SOURCES = frozenset({"hibp_breaches", "xposed_or_not", "leakcheck_public"
 KEY_RE = re.compile(r"[^a-z0-9]+")
 CREDENTIAL_RE = re.compile(r"pass|hash|credential|secret|token|api[ _-]?key", re.I)
 SEVERITY = ("low", "medium", "high")
+MAX_PROMPT_BREACHES = 15
 
 
 def signals_for(merged: list[dict[str, Any]]) -> dict[str, Any]:
@@ -66,7 +67,10 @@ def _records(result: CollectorResult) -> list[dict[str, Any]]:
     return []
 
 
-def merge_breaches(results: list[CollectorResult]) -> tuple[list[CollectorResult], int]:
+def merge_breaches(
+    results: list[CollectorResult],
+    catalogue: dict[str, dict[str, Any]] | None = None,
+) -> tuple[list[CollectorResult], int]:
     """Merge duplicate breach names across sources.
 
     Returns the results to prompt with (breach sources replaced by one index)
@@ -88,18 +92,23 @@ def merge_breaches(results: list[CollectorResult]) -> tuple[list[CollectorResult
             if not name:
                 continue
             key = _key(name)
+            known = (catalogue or {}).get(key) or {}
+            types = set(_strings(entry.get("data_types"))) or set(known.get("data_types") or [])
+            date = entry.get("date") or known.get("date")
             existing = merged.get(key)
             if existing is None:
                 merged[key] = {
                     "name": name,
-                    "date": entry.get("date"),
-                    "data_types": set(_strings(entry.get("data_types"))),
+                    "date": date,
+                    "data_types": types,
                     "sources": {source},
+                    "domain": known.get("domain"),
+                    "pwn_count": known.get("pwn_count"),
                 }
                 continue
             duplicates += 1
-            if not existing["date"] and entry.get("date"):
-                existing["date"] = entry["date"]
+            if not existing["date"] and date:
+                existing["date"] = date
             existing["data_types"].update(_strings(entry.get("data_types")))
             existing["sources"].add(source)
 
@@ -110,13 +119,16 @@ def merge_breaches(results: list[CollectorResult]) -> tuple[list[CollectorResult
         {
             "name": item["name"],
             "date": item["date"],
-            "data_types": sorted(item["data_types"]),
+            "data_types": sorted(item["data_types"])[:8],
             "sources": sorted(item["sources"]),
             "source": ", ".join(sorted(item["sources"])),
+            "domain": item.get("domain"),
+            "pwn_count": item.get("pwn_count"),
         }
         for item in merged.values()
     ]
     rendered.sort(key=lambda item: (-len(item["data_types"]), item["name"].lower()))
+    rendered = rendered[:MAX_PROMPT_BREACHES]
 
     index = CollectorResult(
         collector="breach_index",

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/offline.py

@@ -142,7 +142,9 @@ def _recommendations(
     return recs
 
 
-def build(results: list[CollectorResult], report_notes: list[str], target: str) -> Analysis:
+def build(
+    results: list[CollectorResult], report_notes: list[str], target: str, *, narrated: bool = False
+) -> Analysis:
     breaches_raw = next((r.data for r in results if r.collector == "breach_index" and r.has_data), [])
     signals = signals_for(breaches_raw)
     accounts = _found_accounts(results)
@@ -158,8 +160,8 @@ def build(results: list[CollectorResult], report_notes: list[str], target: str)
     failed = [r.collector for r in results if not r.ok and not r.skipped]
 
     summary_parts = [
-        f"Collected public data for {target} from {len(collected)} of "
-        f"{len(results)} sources, with no model call."
+        f"Collected public data for {target} from {len(collected)} of {len(results)} sources"
+        + (", with no model call." if not narrated else "; the model returned no summary.")
     ]
     if signals["breach_count"]:
         summary_parts.append(
@@ -174,7 +176,8 @@ def build(results: list[CollectorResult], report_notes: list[str], target: str)
         summary_parts.append(f"Failed sources: {', '.join(failed)}.")
     if skipped:
         summary_parts.append(f"Skipped sources: {', '.join(skipped)}.")
-    summary_parts.append("Risk is computed from counts and policy checks, not narrated.")
+    if not narrated:
+        summary_parts.append("Risk is computed from counts and policy checks, not narrated.")
 
     risk = apply_floor("low", signals["risk_floor"])
     if risky or (wildcard and isinstance(mail, dict) and not mail.get("dmarc_present")):

osintkit-0.5.0/src/osintkit/passwords.py

@@ -0,0 +1,66 @@
+"""Pwned Passwords range check (k-anonymity, no key, no plaintext sent)."""
+
+from __future__ import annotations
+
+import hashlib
+
+from .errors import OsintKitError
+from .http import HttpClient, cache_hit, cache_put
+
+NAMESPACE = "pwnedpasswords"
+TTL = 24 * 3600
+RANGE = "https://api.pwnedpasswords.com/range/"
+
+
+def digest(password: str) -> tuple[str, str]:
+    sha1 = hashlib.sha1(password.encode("utf-8")).hexdigest().upper()
+    return sha1[:5], sha1[5:]
+
+
+def parse_range(text: str, suffix: str) -> int:
+    for line in text.splitlines():
+        hash_part, _, count = line.partition(":")
+        if hash_part.strip().upper() == suffix:
+            try:
+                return int(count.strip())
+            except ValueError:
+                return 0
+    return 0
+
+
+def rating(count: int) -> str:
+    if count == 0:
+        return "not seen"
+    if count < 100:
+        return "rare"
+    if count < 10_000:
+        return "common"
+    if count < 1_000_000:
+        return "very common"
+    return "extremely common"
+
+
+async def check(http: HttpClient, password: str) -> dict[str, object]:
+    prefix, suffix = digest(password)
+    key = f"{prefix}:{suffix}"
+    cached = cache_hit(http.settings, NAMESPACE, key, TTL)
+    if cached is not None:
+        count = int(cached)
+    else:
+        response = await http.get(
+            f"{RANGE}{prefix}",
+            namespace=NAMESPACE,
+            accept="text/plain",
+            headers={"Add-Padding": "true"},
+        )
+        if response.status_code != 200:
+            raise OsintKitError(f"pwned passwords returned HTTP {response.status_code}")
+        count = parse_range(response.text, suffix)
+        cache_put(http.settings, NAMESPACE, key, count)
+    return {
+        "pwned": count > 0,
+        "count": count,
+        "rating": rating(count),
+        "prefix": prefix,
+        "checked_by": "api.pwnedpasswords.com",
+    }

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/pipeline.py

@@ -14,7 +14,7 @@ from pathlib import Path
 
 from rich.console import Console
 
-from . import collectors, normalise, offline, render, retention, synthesis
+from . import catalogue, collectors, normalise, offline, render, retention, synthesis
 from .config import Settings
 from .errors import MissingAPIKey, OsintKitError, SynthesisError, TargetParseError
 from .http import HttpClient
@@ -158,9 +158,29 @@ async def execute(
     async with HttpClient(settings) as http:
         with _status(progress, settings, f"[cyan]running collectors for {target}…[/cyan]"):
             results = await collect(target, target_type, settings, logger, http, only, skip)
+            if target_type == "email" and "@" in target:
+                derived = target.split("@", 1)[1].lower()
+                domain_ctx = collectors.Context(
+                    target=derived, target_type="domain", settings=settings, http=http
+                )
+                for name in ("dns_records", "dns_mail_posture", "whois_registration"):
+                    spec = next((c for c in collectors.REGISTRY if c.name == name), None)
+                    if spec is None or (only and name not in only) or (skip and name in skip):
+                        continue
+                    extra = await collectors.run_collector(spec, domain_ctx)
+                    extra.note = "derived from the email domain"
+                    results.append(extra)
         report.collectors = results
 
-        prompt_results, duplicates = normalise.merge_breaches(results)
+        if target_type == "email":
+            try:
+                known = await catalogue.fetch(http)
+            except Exception:
+                known = {}
+        else:
+            known = {}
+
+        prompt_results, duplicates = normalise.merge_breaches(results, known)
         if duplicates:
             report.notes.append(f"{duplicates} duplicate breach records were merged across sources")
             logger.event("breaches_merged", duplicates=duplicates)
@@ -224,9 +244,19 @@ async def execute(
                     report.analysis.risk_level = normalise.apply_floor(
                         report.analysis.risk_level, signals.get("risk_floor", "low")
                     )
+                    if not report.analysis.summary.strip():
+                        fallback = offline.build(prompt_results, report.notes, target, narrated=True)
+                        report.analysis.summary = fallback.summary
+                        report.notes.append("summary filled locally: the model returned none")
         except (OsintKitError, SynthesisError) as exc:
             report.notes.append(f"synthesis failed: {exc}")
+            report.notes.append("report built locally instead")
             logger.event("run_error", error=str(exc))
+            report.model = None
+            report.analysis = offline.build(prompt_results, report.notes, target, narrated=True)
+            report.analysis.risk_level = normalise.apply_floor(
+                report.analysis.risk_level, signals.get("risk_floor", "low")
+            )
             report.elapsed_ms = int((time.perf_counter() - started) * 1000)
             path = archive_report(report.model_dump(mode="json"))
             return Outcome(report=report, error=str(exc), archived=path)

{osintkit-0.4.0 → osintkit-0.5.0}/src/osintkit/synthesis.py

@@ -20,7 +20,7 @@ authorised research.
 
 Rules:
 - Cite the supplied source name in every breach entry. Mark dates as found; use null when unknown.
-- If a source reports more breaches than it listed, say so and keep the 15 most material, ordering
+- If a source reports more breaches than it listed, say so and Take only the 15 most material, ordering
   credential-bearing and recent incidents first.
 - Distinguish confirmed findings from plausible inference in the summary and notes.
 - Risk level reflects exposure only: high means credential-bearing breaches or a wide,

osintkit-0.5.0/tests/test_catalogue.py

@@ -0,0 +1,70 @@
+import httpx
+import respx
+
+from osintkit import catalogue
+from osintkit.http import HttpClient
+
+HIBP = "https://haveibeenpwned.com/api/v3/breaches"
+XPOSED = "https://api.xposedornot.com/v1/breaches"
+
+
+@respx.mock
+async def test_hibp_catalogue_is_parsed(http):
+    respx.get(HIBP).mock(
+        return_value=httpx.Response(
+            200,
+            json=[
+                {
+                    "Name": "Acme Corp",
+                    "BreachDate": "2020-09-01",
+                    "DataClasses": ["Emails"],
+                    "Domain": "acme.test",
+                    "PwnCount": 100,
+                }
+            ],
+        )
+    )
+    data = await catalogue.fetch(http)
+    assert data["acmecorp"]["date"] == "2020-09-01"
+    assert data["acmecorp"]["data_types"] == ["Emails"]
+
+
+@respx.mock
+async def test_xposed_is_the_fallback(http):
+    respx.get(HIBP).mock(return_value=httpx.Response(503))
+    respx.get(XPOSED).mock(
+        return_value=httpx.Response(
+            200,
+            json={
+                "exposedBreaches": [
+                    {"breachID": "Acme", "breachedDate": "2019-05-05T00:00:00+00:00", "exposedData": ["Emails"]}
+                ]
+            },
+        )
+    )
+    data = await catalogue.fetch(http)
+    assert data["acme"]["date"] == "2019-05-05"
+
+
+@respx.mock
+async def test_both_failing_is_empty(http):
+    respx.get(HIBP).mock(return_value=httpx.Response(503))
+    respx.get(XPOSED).mock(return_value=httpx.Response(503))
+    assert await catalogue.fetch(http) == {}
+
+
+@respx.mock
+async def test_result_is_cached(settings):
+    settings.use_cache = True
+    async with HttpClient(settings) as client:
+        route = respx.get(HIBP).mock(
+            return_value=httpx.Response(200, json=[{"Name": "Acme", "BreachDate": "2020-01-01"}])
+        )
+        await catalogue.fetch(client)
+        await catalogue.fetch(client)
+        assert route.call_count == 1
+
+
+def test_null_rows_are_skipped():
+    assert catalogue._from_hibp([None, {}, {"Name": ""}]) == {}
+    assert catalogue._from_xposed({"exposedBreaches": [None, {"breachID": ""}]}) == {}

{osintkit-0.4.0 → osintkit-0.5.0}/tests/test_cli.py

@@ -618,3 +618,28 @@ def test_flag_beats_stored_config(monkeypatch, xdg):
     monkeypatch.setattr(pipeline, "LLMClient", FakeLLM)
     result = runner.invoke(app, ["--model", "flag-model", "lookup", "--domain", "a.com", "--json"])
     assert jsonish(result.output)["model"] == "flag-model"
+
+
+def test_password_command_hit_and_miss(monkeypatch):
+    import httpx
+    import respx
+
+    from osintkit import passwords
+
+    prefix, suffix = passwords.digest("hunter2")
+    miss_prefix = passwords.digest("hunter3")[0]
+    with respx.mock:
+        respx.get(f"https://api.pwnedpasswords.com/range/{prefix}").mock(
+            return_value=httpx.Response(200, text=f"{suffix}:5")
+        )
+        hit = runner.invoke(app, ["password", "hunter2"])
+    assert hit.exit_code == 1
+    assert "pwned" in hit.output
+
+    with respx.mock:
+        respx.get(f"https://api.pwnedpasswords.com/range/{miss_prefix}").mock(
+            return_value=httpx.Response(200, text="")
+        )
+        miss = runner.invoke(app, ["password", "hunter3", "--json"])
+    assert miss.exit_code == 0
+    assert jsonish(miss.output)["pwned"] is False

{osintkit-0.4.0 → osintkit-0.5.0}/tests/test_e2e.py

@@ -62,7 +62,7 @@ async def test_run_survives_wrapped_code_fence(settings, monkeypatch, console):
 def test_settings_defaults_match_spec():
     settings = Settings()
     assert settings.temperature == 0.3
-    assert settings.max_tokens == 2400
+    assert settings.max_tokens == 4000
     assert settings.base_url == "https://api.venice.ai/api/v1"
     assert settings.model == "deepseek-v4-1-flash"
     assert settings.max_attempts == 3

{osintkit-0.4.0 → osintkit-0.5.0}/tests/test_llm.py

@@ -74,12 +74,16 @@ def test_extract_content_surfaces_api_error():
         extract_content({"error": {"message": "quota exceeded"}})
 
 
-@pytest.mark.parametrize("body", [{}, {"choices": []}, {"choices": [{}]}, "nope"])
+@pytest.mark.parametrize("body", [{}, {"choices": []}, "nope"])
 def test_extract_content_rejects_bad_shapes(body):
     with pytest.raises(SynthesisError):
         extract_content(body)
 
 
+def test_extract_content_allows_empty_message():
+    assert extract_content({"choices": [{}]}) == ""
+
+
 @respx.mock
 async def test_rejected_key_becomes_actionable_error(settings):
     respx.post(URL).mock(return_value=httpx.Response(401))
@@ -109,3 +113,31 @@ async def test_keyless_provider_sends_no_auth_header(settings):
     async with LLMClient(settings) as llm:
         assert await llm.chat([{"role": "user", "content": "x"}]) == "ok"
     assert "authorization" not in route.calls[0].request.headers
+
+
+@respx.mock
+async def test_truncated_empty_reply_is_retried_with_more_tokens(settings):
+    route = respx.post(URL).mock(
+        side_effect=[
+            httpx.Response(200, json={"choices": [{"finish_reason": "length", "message": {"content": ""}}]}),
+            httpx.Response(200, json={"choices": [{"finish_reason": "stop", "message": {"content": "ok"}}]}),
+        ]
+    )
+    async with client(settings) as llm:
+        assert await llm.chat([{"role": "user", "content": "x"}], max_tokens=2400) == "ok"
+    import json as _json
+
+    budgets = [_json.loads(call.request.content)["max_tokens"] for call in route.calls]
+    assert budgets == [2400, 4800]
+
+
+@respx.mock
+async def test_truncation_without_content_gives_up(settings):
+    respx.post(URL).mock(
+        return_value=httpx.Response(
+            200, json={"choices": [{"finish_reason": "length", "message": {"content": ""}}]}
+        )
+    )
+    async with client(settings) as llm:
+        with pytest.raises(SynthesisError, match="ran out of tokens"):
+            await llm.chat([{"role": "user", "content": "x"}], max_tokens=8000)

{osintkit-0.4.0 → osintkit-0.5.0}/tests/test_offline.py

@@ -97,3 +97,9 @@ def test_empty_collection_still_yields_a_report():
     assert analysis.risk_level == "low"
     assert analysis.breaches == []
     assert analysis.recommendations
+
+
+def test_narrated_summary_wording():
+    analysis = offline.build([], [], "example.com", narrated=True)
+    assert "model returned no summary" in analysis.summary
+    assert "not narrated" not in analysis.summary

osintkit-0.5.0/tests/test_passwords.py

@@ -0,0 +1,58 @@
+import hashlib
+
+import httpx
+import respx
+
+from osintkit import passwords
+
+PASSWORD = "hunter2"
+SHA = hashlib.sha1(PASSWORD.encode()).hexdigest().upper()
+
+
+def test_digest_splits_the_hash():
+    prefix, suffix = passwords.digest(PASSWORD)
+    assert prefix == SHA[:5]
+    assert suffix == SHA[5:]
+
+
+def test_parse_range_matches_suffix_only():
+    text = f"{'A' * 35}:5\r\n{SHA[5:]}:42\r\n"
+    assert passwords.parse_range(text, SHA[5:]) == 42
+    assert passwords.parse_range(text, "F" * 35) == 0
+
+
+def test_rating_bands():
+    assert passwords.rating(0) == "not seen"
+    assert passwords.rating(10) == "rare"
+    assert passwords.rating(500) == "common"
+    assert passwords.rating(50_000) == "very common"
+    assert passwords.rating(9_000_000) == "extremely common"
+
+
+@respx.mock
+async def test_check_reports_hits(http):
+    respx.get(f"https://api.pwnedpasswords.com/range/{SHA[:5]}").mock(
+        return_value=httpx.Response(200, text=f"{SHA[5:]}:42\r\n{'B' * 35}:1\r\n")
+    )
+    result = await passwords.check(http, PASSWORD)
+    assert result["pwned"] is True
+    assert result["count"] == 42
+
+
+@respx.mock
+async def test_check_reports_misses(http):
+    respx.get(f"https://api.pwnedpasswords.com/range/{SHA[:5]}").mock(
+        return_value=httpx.Response(200, text=f"{'B' * 35}:1\r\n")
+    )
+    result = await passwords.check(http, PASSWORD)
+    assert result["pwned"] is False
+
+
+@respx.mock
+async def test_range_request_sends_only_the_prefix(http):
+    route = respx.get(f"https://api.pwnedpasswords.com/range/{SHA[:5]}").mock(
+        return_value=httpx.Response(200, text="")
+    )
+    await passwords.check(http, PASSWORD)
+    assert route.calls[0].request.headers["add-padding"] == "true"
+    assert PASSWORD not in str(route.calls[0].request.url)

{osintkit-0.4.0 → osintkit-0.5.0}/tests/test_pipeline.py

@@ -18,6 +18,14 @@ VALID = {
 }
 
 
+@pytest.fixture(autouse=True)
+async def no_catalogue(monkeypatch):
+    async def empty(_http):
+        return {}
+
+    monkeypatch.setattr(pipeline.catalogue, "fetch", empty)
+
+
 @pytest.fixture
 def fake_collector(monkeypatch):
     async def fake(ctx):
@@ -158,7 +166,7 @@ async def test_duplicate_breaches_are_merged_before_prompting(settings, monkeypa
     assert "breach_index" in prompt["user"]
     assert prompt["user"].count('"name": "Acme"') == 1
     assert '"sources": [' in prompt["user"]
-    assert [r.collector for r in outcome.report.collectors] == ["hibp_breaches", "xposed_or_not"]
+    assert [r.collector for r in outcome.report.collectors][:2] == ["hibp_breaches", "xposed_or_not"]
 
 
 async def test_risk_floor_overrides_a_timid_model(settings, monkeypatch, console):
@@ -194,3 +202,46 @@ async def test_offline_builds_analysis_without_model(settings, fake_collector, m
     assert outcome.report.analysis is not None
     assert outcome.report.model is None
     assert any("offline" in note for note in outcome.report.notes)
+
+
+async def test_empty_summary_is_backfilled(settings, fake_collector, monkeypatch, console):
+    class Empty(FakeLLM):
+        async def chat(self, messages, **kwargs):
+            return json.dumps({**VALID, "summary": "  "})
+
+    monkeypatch.setattr(pipeline, "LLMClient", Empty)
+    outcome = await pipeline.execute("example.com", settings, RunLogger("sum"), console)
+    assert outcome.report.analysis.summary
+    assert any("filled locally" in note for note in outcome.report.notes)
+
+
+async def test_failed_synthesis_still_yields_a_local_report(settings, fake_collector, monkeypatch, console):
+    class Broken(FakeLLM):
+        async def chat(self, messages, **kwargs):
+            raise SynthesisError("no dice")
+
+    monkeypatch.setattr(pipeline, "LLMClient", Broken)
+    outcome = await pipeline.execute("example.com", settings, RunLogger("fb"), console)
+    assert outcome.error == "no dice"
+    assert outcome.report.analysis is not None
+    assert any("built locally" in note for note in outcome.report.notes)
+
+
+async def test_email_pulls_its_domain(settings, monkeypatch, console):
+    calls = []
+
+    async def fake(ctx):
+        calls.append((ctx.target, ctx.target_type))
+        return {"A": ["1.2.3.4"]}
+
+    monkeypatch.setattr(
+        base,
+        "REGISTRY",
+        [base.Collector(name="dns_records", target_types=("domain",), func=fake)],
+    )
+    monkeypatch.setattr("osintkit.collectors.REGISTRY", base.REGISTRY)
+    monkeypatch.setattr(pipeline, "LLMClient", FakeLLM)
+    outcome = await pipeline.execute("alice@example.com", settings, RunLogger("dom"), console)
+    assert ("example.com", "domain") in calls
+    derived = [c for c in outcome.report.collectors if c.note == "derived from the email domain"]
+    assert derived and derived[0].ok