reconkit 0.1.0__tar.gz → 0.2.0__tar.gz

{reconkit-0.1.0 → reconkit-0.2.0}/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file.
 
 Format: loosely based on *Keep a Changelog*. Versioning: SemVer where practical.
 
+## [0.2.0] - 2026-11-04
+
+### Added
+- WARC 1.1 and WACZ 1.2 exports with an index, page list, SHA-256 fixity, and data-package digest.
+- Hash-linked chain-of-custody ledgers and Ed25519-signed manifests.
+- DFIR-IRIS v2 evidence-registration adapter.
+- Redirect-chain recording and conservative public-destination checks for web capture.
+
+### Changed
+- Robots rules are checked at every redirect destination and fail closed by default.
+
 ## [0.1.0] - 2026-02-09
 
 ### Added

reconkit-0.2.0/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.5
+Name: reconkit
+Version: 0.2.0
+Summary: Evidence collection, provenance, and preservation utilities for authorized investigations.
+Author: reconkit
+License: MIT
+License-File: LICENSE
+Keywords: forensics,osint,scraping,triage
+Requires-Python: >=3.10
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: cryptography>=42
+Requires-Dist: dnspython>=2.6
+Requires-Dist: exifread>=3.0
+Requires-Dist: filetype>=1.2
+Requires-Dist: httpx>=0.27
+Requires-Dist: ipwhois>=1.2
+Requires-Dist: lxml>=5.1
+Requires-Dist: python-whois>=0.9
+Requires-Dist: rich>=13.7
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Requires-Dist: twine>=5.1; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# ReconKit
+
+ReconKit is a small Python toolkit for preserving authorized web observations and local file findings as reviewable evidence records. It is a collection layer, not a scanner, case-management system, or source of truth.
+
+It complements reconnaissance tools such as SpiderFoot and BBOT, web archiving tools such as Webrecorder, and case-management platforms such as DFIR-IRIS. ReconKit focuses on the handoff between collection and review: what was collected, where it came from, what redirects occurred, what limits apply, and whether exported files have changed.
+
+## What it provides
+
+- Public-web capture with bounded size, rate limiting, redirect-chain recording, robots checks at every destination, and blocking of private, loopback, and reserved destinations by default.
+- Standards-oriented WARC 1.1 and WACZ 1.2 exports. WACZ files contain a WARC, CDXJ index, page list, file-fixity manifest, and data-package digest.
+- Append-only, hash-linked chain-of-custody ledgers.
+- Ed25519-signed manifests that can be verified offline.
+- A DFIR-IRIS v2 adapter that registers an exported evidence file with an existing case.
+- Local file hashes, magic-byte identification, EXIF extraction, printable-string extraction, and small DNS/WHOIS/RDAP observations.
+
+## What it does not claim
+
+ReconKit records a transport observation. It does not prove the identity of a publisher, the truth of content, legal chain of custody, or attribution. A SHA-256 digest proves that bytes have not changed relative to the recorded digest; it does not prove where they originated.
+
+Web collection is only for systems and data you own or are authorized to assess. The default public-destination restriction helps prevent accidental collection from local or private infrastructure; it is not a substitute for operational authorization.
+
+## Install
+
+```bash
+python -m pip install reconkit
+```
+
+For development:
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+## Capture and preserve a page
+
+```bash
+reconkit evidence archive-warc "https://example.com" --output evidence/example.warc
+reconkit evidence archive-wacz "https://example.com" --output evidence/example.wacz --title "Example capture"
+```
+
+The WACZ export follows the [WACZ 1.2 layout](https://specs.webrecorder.net/wacz/1.2.0/), so it can be validated and replayed with compatible web-archive tooling.
+Robots rules are respected by default; use `--no-obey-robots` only when you have clear authorization for the capture.
+
+## Sign a manifest and record custody
+
+```python
+from pathlib import Path
+from reconkit.evidence import CustodyLedger, ManifestSigner, verify_manifest, write_manifest
+
+evidence = Path("evidence/example.wacz")
+ledger = CustodyLedger("evidence/custody.jsonl")
+ledger.append(action="acquired", actor="analyst@example.org", artifact_path=evidence)
+ledger.append(action="exported", actor="analyst@example.org", artifact_path=evidence)
+assert ledger.verify()
+
+signer = ManifestSigner.generate()
+Path("keys/evidence-signing-key.pem").write_bytes(signer.private_pem())
+manifest = write_manifest([evidence], "evidence/manifest.json", signer, custody_ledger=ledger.path)
+assert verify_manifest(manifest)
+```
+
+Keep the private key outside the evidence directory and protect it with your organization’s key-management practices. The manifest embeds the corresponding public key so a reviewer can verify it without the private key.
+
+## Register exported evidence in DFIR-IRIS
+
+```python
+from reconkit.evidence import DFIRIRISAdapter
+
+iris = DFIRIRISAdapter("https://iris.example.org", api_key="your-api-key")
+iris.register(
+    case_identifier=42,
+    evidence_path="evidence/example.wacz",
+    description="Public web capture; signature and custody ledger stored alongside export.",
+)
+```
+
+The adapter registers file metadata, digest, and description through DFIR-IRIS’s documented v2 evidence endpoint. Upload and retention of the evidence bytes remain controlled by the case system’s own workflow.
+
+## Existing local commands
+
+```bash
+reconkit evidence acquire-web "https://example.com" --trust-boundary public-internet
+reconkit evidence derive-links "https://example.com"
+reconkit boundary dns example.com
+reconkit verify file-hash .\sample.bin --algo sha256
+```
+
+## Development
+
+```bash
+pytest
+python -m build
+twine check dist/*
+```
+
+## License
+
+MIT

reconkit-0.2.0/README.md

@@ -0,0 +1,98 @@
+# ReconKit
+
+ReconKit is a small Python toolkit for preserving authorized web observations and local file findings as reviewable evidence records. It is a collection layer, not a scanner, case-management system, or source of truth.
+
+It complements reconnaissance tools such as SpiderFoot and BBOT, web archiving tools such as Webrecorder, and case-management platforms such as DFIR-IRIS. ReconKit focuses on the handoff between collection and review: what was collected, where it came from, what redirects occurred, what limits apply, and whether exported files have changed.
+
+## What it provides
+
+- Public-web capture with bounded size, rate limiting, redirect-chain recording, robots checks at every destination, and blocking of private, loopback, and reserved destinations by default.
+- Standards-oriented WARC 1.1 and WACZ 1.2 exports. WACZ files contain a WARC, CDXJ index, page list, file-fixity manifest, and data-package digest.
+- Append-only, hash-linked chain-of-custody ledgers.
+- Ed25519-signed manifests that can be verified offline.
+- A DFIR-IRIS v2 adapter that registers an exported evidence file with an existing case.
+- Local file hashes, magic-byte identification, EXIF extraction, printable-string extraction, and small DNS/WHOIS/RDAP observations.
+
+## What it does not claim
+
+ReconKit records a transport observation. It does not prove the identity of a publisher, the truth of content, legal chain of custody, or attribution. A SHA-256 digest proves that bytes have not changed relative to the recorded digest; it does not prove where they originated.
+
+Web collection is only for systems and data you own or are authorized to assess. The default public-destination restriction helps prevent accidental collection from local or private infrastructure; it is not a substitute for operational authorization.
+
+## Install
+
+```bash
+python -m pip install reconkit
+```
+
+For development:
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+## Capture and preserve a page
+
+```bash
+reconkit evidence archive-warc "https://example.com" --output evidence/example.warc
+reconkit evidence archive-wacz "https://example.com" --output evidence/example.wacz --title "Example capture"
+```
+
+The WACZ export follows the [WACZ 1.2 layout](https://specs.webrecorder.net/wacz/1.2.0/), so it can be validated and replayed with compatible web-archive tooling.
+Robots rules are respected by default; use `--no-obey-robots` only when you have clear authorization for the capture.
+
+## Sign a manifest and record custody
+
+```python
+from pathlib import Path
+from reconkit.evidence import CustodyLedger, ManifestSigner, verify_manifest, write_manifest
+
+evidence = Path("evidence/example.wacz")
+ledger = CustodyLedger("evidence/custody.jsonl")
+ledger.append(action="acquired", actor="analyst@example.org", artifact_path=evidence)
+ledger.append(action="exported", actor="analyst@example.org", artifact_path=evidence)
+assert ledger.verify()
+
+signer = ManifestSigner.generate()
+Path("keys/evidence-signing-key.pem").write_bytes(signer.private_pem())
+manifest = write_manifest([evidence], "evidence/manifest.json", signer, custody_ledger=ledger.path)
+assert verify_manifest(manifest)
+```
+
+Keep the private key outside the evidence directory and protect it with your organization’s key-management practices. The manifest embeds the corresponding public key so a reviewer can verify it without the private key.
+
+## Register exported evidence in DFIR-IRIS
+
+```python
+from reconkit.evidence import DFIRIRISAdapter
+
+iris = DFIRIRISAdapter("https://iris.example.org", api_key="your-api-key")
+iris.register(
+    case_identifier=42,
+    evidence_path="evidence/example.wacz",
+    description="Public web capture; signature and custody ledger stored alongside export.",
+)
+```
+
+The adapter registers file metadata, digest, and description through DFIR-IRIS’s documented v2 evidence endpoint. Upload and retention of the evidence bytes remain controlled by the case system’s own workflow.
+
+## Existing local commands
+
+```bash
+reconkit evidence acquire-web "https://example.com" --trust-boundary public-internet
+reconkit evidence derive-links "https://example.com"
+reconkit boundary dns example.com
+reconkit verify file-hash .\sample.bin --algo sha256
+```
+
+## Development
+
+```bash
+pytest
+python -m build
+twine check dist/*
+```
+
+## License
+
+MIT

{reconkit-0.1.0 → reconkit-0.2.0}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "hatchling.build"
 
 [project]
 name = "reconkit"
-version = "0.1.0"
-description = "A small, safe-by-default toolkit for scraping, OSINT, and file triage."
+version = "0.2.0"
+description = "Evidence collection, provenance, and preservation utilities for authorized investigations."
 readme = "README.md"
 requires-python = ">=3.10"
 license = { text = "MIT" }
@@ -22,6 +22,7 @@ dependencies = [
   "ipwhois>=1.2",
   "exifread>=3.0",
   "filetype>=1.2",
+  "cryptography>=42",
 ]
 
 [project.optional-dependencies]
@@ -42,7 +43,7 @@ packages = ["src/reconkit"]
 
 [tool.pytest.ini_options]
 addopts = "-q"
-asyncio_mode = "auto"
+testpaths = ["tests"]
 
 [tool.ruff]
 line-length = 100

{reconkit-0.1.0 → reconkit-0.2.0}/src/reconkit/__init__.py

@@ -4,7 +4,7 @@ Public API: Noir Stack / Hexarch / Judicium surface lives under `reconkit.noir`.
 Internal modules remain available but are not the supported public interface.
 """
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 from reconkit.noir.primitives import (  # noqa: E402
     EvidenceArtifact,

{reconkit-0.1.0 → reconkit-0.2.0}/src/reconkit/cli.py

@@ -6,8 +6,10 @@ from typing import Optional
 import typer
 from rich.console import Console
 
+from reconkit.evidence.archives import export_wacz, export_warc
 from reconkit.noir import public_api
 from reconkit.noir.primitives import VerificationResult, json_dumps_audit
+from reconkit.scraping.fetch import fetch_url
 
 app = typer.Typer(add_completion=False, no_args_is_help=True)
 console = Console()
@@ -68,6 +70,33 @@ def evidence_acquire_web(
     )
 
 
+@evidence_app.command("archive-warc")
+def evidence_archive_warc(
+    url: str,
+    output: Path = typer.Option(..., "--output", "-o"),
+    max_bytes: int = typer.Option(2_000_000),
+    obey_robots: bool = typer.Option(True, "--obey-robots/--no-obey-robots"),
+):
+    """Capture one public URL and preserve its response in WARC 1.1."""
+    capture = fetch_url(url, max_bytes=max_bytes, obey_robots=obey_robots)
+    path = export_warc([capture], output)
+    typer.echo(json_dumps_audit({"format": "WARC 1.1", "path": str(path)}))
+
+
+@evidence_app.command("archive-wacz")
+def evidence_archive_wacz(
+    url: str,
+    output: Path = typer.Option(..., "--output", "-o"),
+    title: str = typer.Option("ReconKit collection"),
+    max_bytes: int = typer.Option(2_000_000),
+    obey_robots: bool = typer.Option(True, "--obey-robots/--no-obey-robots"),
+):
+    """Capture one public URL and create a portable WACZ 1.2 archive."""
+    capture = fetch_url(url, max_bytes=max_bytes, obey_robots=obey_robots)
+    path = export_wacz([capture], output, title=title)
+    typer.echo(json_dumps_audit({"format": "WACZ 1.2", "path": str(path)}))
+
+
 @evidence_app.command("derive-links")
 def evidence_derive_links(
     url: str,

reconkit-0.2.0/src/reconkit/evidence/__init__.py

@@ -0,0 +1,16 @@
+"""Portable evidence preservation, custody, signing, and case adapters."""
+
+from .archives import export_wacz, export_warc
+from .custody import CustodyLedger
+from .iris import DFIRIRISAdapter
+from .manifest import ManifestSigner, verify_manifest, write_manifest
+
+__all__ = [
+    "CustodyLedger",
+    "DFIRIRISAdapter",
+    "ManifestSigner",
+    "export_wacz",
+    "export_warc",
+    "verify_manifest",
+    "write_manifest",
+]

reconkit-0.2.0/src/reconkit/evidence/archives.py

@@ -0,0 +1,108 @@
+"""Standards-oriented WARC and WACZ exports for captured HTTP responses."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import zipfile
+from datetime import timezone
+from pathlib import Path
+from urllib.parse import urlsplit
+from uuid import uuid4
+
+from reconkit.scraping.fetch import FetchResult
+
+
+def _json(value: object) -> bytes:
+    return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
+
+
+def _sha256(data: bytes) -> str:
+    return hashlib.sha256(data).hexdigest()
+
+
+def _surt(url: str) -> str:
+    """Return a CDXJ-compatible SURT key for a captured URL."""
+    parsed = urlsplit(url)
+    hostname = (parsed.hostname or "").lower().strip(".")
+    authority = ",".join(reversed(hostname.split(".")))
+    if parsed.port and parsed.port not in {80, 443}:
+        authority = f"{authority}:{parsed.port}"
+    path = parsed.path or "/"
+    if parsed.query:
+        path = f"{path}?{parsed.query}"
+    return f"{authority}){path}"
+
+
+def _warc_record(headers: dict[str, str], body: bytes) -> bytes:
+    head = "WARC/1.1\r\n" + "".join(f"{key}: {value}\r\n" for key, value in headers.items()) + "\r\n"
+    return head.encode("utf-8") + body + b"\r\n\r\n"
+
+
+def _http_response(capture: FetchResult) -> bytes:
+    headers = {key: value for key, value in capture.headers.items() if key.lower() not in {"transfer-encoding", "content-length"}}
+    headers["Content-Length"] = str(len(capture.body))
+    status = f"HTTP/1.1 {capture.status_code}\r\n"
+    return status.encode("utf-8") + "".join(f"{key}: {value}\r\n" for key, value in headers.items()).encode("utf-8") + b"\r\n" + capture.body
+
+
+def warc_bytes(captures: list[FetchResult]) -> tuple[bytes, list[dict[str, object]]]:
+    """Create a WARC 1.1 stream and the information needed for a CDXJ index."""
+    output = bytearray()
+    info = _json({"software": "reconkit", "format": "WARC File Format 1.1"})
+    output.extend(_warc_record({
+        "WARC-Type": "warcinfo", "WARC-Date": captures[0].observed_at.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") if captures else "1970-01-01T00:00:00Z",
+        "WARC-Record-ID": f"<urn:uuid:{uuid4()}>", "Content-Type": "application/warc-fields", "Content-Length": str(len(info)),
+    }, info))
+    index: list[dict[str, object]] = []
+    for capture in captures:
+        offset = len(output)
+        payload = _http_response(capture)
+        digest = _sha256(payload)
+        output.extend(_warc_record({
+            "WARC-Type": "response", "WARC-Target-URI": capture.url,
+            "WARC-Date": capture.observed_at.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"),
+            "WARC-Record-ID": f"<urn:uuid:{uuid4()}>", "WARC-Block-Digest": f"sha256:{digest}",
+            "Content-Type": "application/http; msgtype=response", "Content-Length": str(len(payload)),
+        }, payload))
+        index.append({"url": capture.url, "timestamp": capture.observed_at.astimezone(timezone.utc).strftime("%Y%m%d%H%M%S"), "status": str(capture.status_code), "digest": f"sha256:{digest}", "offset": offset, "length": len(output) - offset})
+    return bytes(output), index
+
+
+def export_warc(captures: list[FetchResult], destination: str | Path) -> Path:
+    """Write a WARC 1.1 file for one or more captures."""
+    path = Path(destination)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    data, _ = warc_bytes(captures)
+    path.write_bytes(data)
+    return path
+
+
+def export_wacz(captures: list[FetchResult], destination: str | Path, *, title: str = "ReconKit collection") -> Path:
+    """Write a WACZ 1.2 archive with WARC, CDXJ, pages, and fixity manifests."""
+    if not captures:
+        raise ValueError("At least one capture is required for a WACZ export.")
+    path = Path(destination)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    warc, index = warc_bytes(captures)
+    cdxj = b"".join(
+        f"{_surt(str(entry['url']))} {entry['timestamp']} ".encode() + _json({
+            "url": entry["url"], "status": entry["status"], "digest": entry["digest"],
+            "offset": entry["offset"], "length": entry["length"], "filename": "data.warc",
+        }) + b"\n" for entry in index
+    )
+    pages = b"".join(_json({"url": capture.url, "ts": capture.observed_at.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")}) + b"\n" for capture in captures)
+    resources = [
+        {"name": "data.warc", "path": "archive/data.warc", "hash": f"sha256:{_sha256(warc)}", "bytes": len(warc)},
+        {"name": "index.cdxj", "path": "indexes/index.cdxj", "hash": f"sha256:{_sha256(cdxj)}", "bytes": len(cdxj)},
+        {"name": "pages.jsonl", "path": "pages/pages.jsonl", "hash": f"sha256:{_sha256(pages)}", "bytes": len(pages)},
+    ]
+    datapackage = _json({"profile": "wacz", "title": title, "resources": resources})
+    digest = _json({"path": "datapackage.json", "hash": f"sha256:{_sha256(datapackage)}"})
+    with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_DEFLATED) as archive:
+        archive.writestr("archive/data.warc", warc)
+        archive.writestr("indexes/index.cdxj", cdxj)
+        archive.writestr("pages/pages.jsonl", pages)
+        archive.writestr("datapackage.json", datapackage)
+        archive.writestr("datapackage-digest.json", digest)
+    return path

reconkit-0.2.0/src/reconkit/evidence/custody.py

@@ -0,0 +1,53 @@
+"""Append-only, hash-linked chain-of-custody ledgers."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+from uuid import uuid4
+
+
+def _canonical(value: Any) -> bytes:
+    return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
+
+
+class CustodyLedger:
+    """A JSONL ledger whose event hash commits to the previous event hash."""
+
+    def __init__(self, path: str | Path):
+        self.path = Path(path)
+
+    def append(self, *, action: str, actor: str, artifact_path: str | Path, notes: str = "") -> dict[str, Any]:
+        artifact = Path(artifact_path)
+        digest = hashlib.sha256(artifact.read_bytes()).hexdigest()
+        previous = self._last_hash()
+        event = {
+            "event_id": str(uuid4()), "recorded_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
+            "action": action, "actor": actor, "artifact": {"path": artifact.name, "sha256": digest, "bytes": artifact.stat().st_size},
+            "notes": notes, "previous_event_sha256": previous,
+        }
+        event["event_sha256"] = hashlib.sha256(_canonical(event)).hexdigest()
+        self.path.parent.mkdir(parents=True, exist_ok=True)
+        with self.path.open("a", encoding="utf-8", newline="\n") as handle:
+            handle.write(_canonical(event).decode("utf-8") + "\n")
+        return event
+
+    def verify(self) -> bool:
+        previous: str | None = None
+        if not self.path.exists():
+            return True
+        for line in self.path.read_text(encoding="utf-8").splitlines():
+            event = json.loads(line)
+            claimed = event.pop("event_sha256", None)
+            if claimed != hashlib.sha256(_canonical(event)).hexdigest() or event.get("previous_event_sha256") != previous:
+                return False
+            previous = claimed
+        return True
+
+    def _last_hash(self) -> str | None:
+        if not self.path.exists() or not self.path.stat().st_size:
+            return None
+        return json.loads(self.path.read_text(encoding="utf-8").splitlines()[-1])["event_sha256"]

reconkit-0.2.0/src/reconkit/evidence/iris.py

@@ -0,0 +1,37 @@
+"""Optional DFIR-IRIS v2 evidence registration adapter."""
+
+from __future__ import annotations
+
+import hashlib
+from pathlib import Path
+
+import httpx
+
+
+class DFIRIRISAdapter:
+    """Register a preserved file as evidence in an existing DFIR-IRIS case.
+
+    The adapter registers evidence metadata through the documented v2 endpoint.
+    Uploading the evidence bytes is intentionally left to the organization's IRIS
+    retention workflow.
+    """
+
+    def __init__(self, base_url: str, api_key: str, *, timeout: float = 20.0):
+        self.base_url = base_url.rstrip("/")
+        self.api_key = api_key
+        self.timeout = timeout
+
+    def register(self, case_identifier: int, evidence_path: str | Path, *, description: str = "", type_id: int | None = None) -> dict:
+        path = Path(evidence_path)
+        payload: dict[str, object] = {
+            "filename": path.name, "file_size": path.stat().st_size,
+            "file_hash": hashlib.sha256(path.read_bytes()).hexdigest(), "file_description": description,
+        }
+        if type_id is not None:
+            payload["type_id"] = type_id
+        response = httpx.post(
+            f"{self.base_url}/api/v2/cases/{case_identifier}/evidences", json=payload,
+            headers={"Authorization": f"Bearer {self.api_key}", "Accept": "application/json"}, timeout=self.timeout,
+        )
+        response.raise_for_status()
+        return response.json()

reconkit-0.2.0/src/reconkit/evidence/manifest.py

@@ -0,0 +1,98 @@
+"""Ed25519-signed manifests for evidence files."""
+
+from __future__ import annotations
+
+import base64
+import hashlib
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from cryptography.exceptions import InvalidSignature
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey
+
+
+def _canonical(value: Any) -> bytes:
+    return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
+
+
+class ManifestSigner:
+    """A local Ed25519 signing key. Keep the private PEM outside evidence exports."""
+
+    def __init__(self, private_key: Ed25519PrivateKey):
+        self._private_key = private_key
+
+    @classmethod
+    def generate(cls) -> ManifestSigner:
+        return cls(Ed25519PrivateKey.generate())
+
+    @classmethod
+    def load(cls, pem: bytes, password: bytes | None = None) -> ManifestSigner:
+        key = serialization.load_pem_private_key(pem, password=password)
+        if not isinstance(key, Ed25519PrivateKey):
+            raise TypeError("An Ed25519 private key is required.")
+        return cls(key)
+
+    def private_pem(self, password: bytes | None = None) -> bytes:
+        encryption = serialization.NoEncryption() if password is None else serialization.BestAvailableEncryption(password)
+        return self._private_key.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, encryption)
+
+    def public_pem(self) -> bytes:
+        return self._private_key.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)
+
+    def sign(self, data: bytes) -> str:
+        return base64.b64encode(self._private_key.sign(data)).decode("ascii")
+
+
+def write_manifest(files: list[str | Path], destination: str | Path, signer: ManifestSigner, *, custody_ledger: str | Path | None = None) -> Path:
+    items = []
+    for item in files:
+        path = Path(item)
+        contents = path.read_bytes()
+        items.append({"path": path.name, "bytes": len(contents), "sha256": hashlib.sha256(contents).hexdigest()})
+    manifest: dict[str, Any] = {"version": 1, "created_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), "files": items}
+    if custody_ledger is not None:
+        ledger = Path(custody_ledger)
+        manifest["custody_ledger"] = {"path": ledger.name, "sha256": hashlib.sha256(ledger.read_bytes()).hexdigest()}
+    payload = _canonical(manifest)
+    envelope = {"manifest": manifest, "algorithm": "Ed25519", "public_key_pem": signer.public_pem().decode("ascii"), "signature": signer.sign(payload)}
+    path = Path(destination)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_bytes(_canonical(envelope) + b"\n")
+    return path
+
+
+def verify_manifest(path: str | Path, *, evidence_dir: str | Path | None = None) -> bool:
+    """Verify a manifest signature and, optionally, the referenced evidence files.
+
+    Supplying ``evidence_dir`` verifies each listed file and custody ledger against
+    its recorded SHA-256. Without it, this function verifies the signed manifest
+    envelope only.
+    """
+    envelope = json.loads(Path(path).read_text(encoding="utf-8"))
+    public_key = serialization.load_pem_public_key(envelope["public_key_pem"].encode("ascii"))
+    if not isinstance(public_key, Ed25519PublicKey) or envelope.get("algorithm") != "Ed25519":
+        return False
+    try:
+        public_key.verify(base64.b64decode(envelope["signature"]), _canonical(envelope["manifest"]))
+    except (InvalidSignature, KeyError, TypeError, ValueError):
+        return False
+
+    if evidence_dir is None:
+        return True
+    root = Path(evidence_dir)
+    try:
+        for item in envelope["manifest"]["files"]:
+            artifact = root / item["path"]
+            if not artifact.is_file() or hashlib.sha256(artifact.read_bytes()).hexdigest() != item["sha256"]:
+                return False
+        ledger = envelope["manifest"].get("custody_ledger")
+        if ledger:
+            artifact = root / ledger["path"]
+            if not artifact.is_file() or hashlib.sha256(artifact.read_bytes()).hexdigest() != ledger["sha256"]:
+                return False
+        return True
+    except (KeyError, OSError, TypeError):
+        return False

{reconkit-0.1.0 → reconkit-0.2.0}/src/reconkit/noir/public_api.py

@@ -23,7 +23,7 @@ def acquire_web_artifact(
     timeout_s: float = 20.0,
     max_bytes: int = 2_000_000,
     obey_robots: bool = True,
-    user_agent: str = "reconkit/0.1 (+https://localhost)",
+    user_agent: str = "reconkit/0.2 (+https://github.com/no1rstack/reconkit)",
 ) -> EvidenceArtifact:
     """Acquire a web artifact as evidence.
 
@@ -57,7 +57,11 @@ def acquire_web_artifact(
         payload=result.body,
         media_type=media_type,
         encoding=result.encoding,
-        provenance=prov,
+        provenance=ProvenanceRecord(
+            source=result.url, acquisition_method="http-get", trust_boundary=trust_boundary,
+            observed_at=result.observed_at,
+            notes=prov.notes + [f"redirect_chain={list(result.redirect_chain)!r}"],
+        ),
     )
 
 

reconkit-0.2.0/src/reconkit/scraping/fetch.py

@@ -0,0 +1,112 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime, timezone
+import ipaddress
+import socket
+from typing import Optional
+from urllib.parse import urljoin, urlparse
+
+import httpx
+
+from reconkit.core.rate_limit import RateLimiter
+from reconkit.scraping import robots
+from reconkit.noir.errors import AcquisitionFailureError, EvidenceSizeLimitError, RobotsDisallowedError
+
+
+@dataclass
+class FetchResult:
+    requested_url: str
+    url: str
+    status_code: int
+    headers: dict[str, str]
+    body: bytes
+    encoding: Optional[str]
+    observed_at: datetime
+    redirect_chain: tuple[dict[str, object], ...] = ()
+
+
+def _validate_url(url: str, *, allow_private_hosts: bool) -> None:
+    parsed = urlparse(url)
+    if parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.username or parsed.password:
+        raise AcquisitionFailureError("Only absolute, credential-free http(s) URLs are permitted.")
+    if allow_private_hosts:
+        return
+    try:
+        addresses = {item[4][0] for item in socket.getaddrinfo(parsed.hostname, None)}
+    except OSError as exc:
+        raise AcquisitionFailureError(f"Could not resolve destination host: {parsed.hostname}") from exc
+    for address in addresses:
+        value = ipaddress.ip_address(address)
+        if not value.is_global:
+            raise AcquisitionFailureError("Private, loopback, link-local, and reserved destinations are blocked by default.")
+
+
+def fetch_url(
+    url: str,
+    *,
+    max_bytes: int = 2_000_000,
+    timeout_s: float = 20.0,
+    rps: float = 1.0,
+    obey_robots: bool = True,
+    user_agent: str = "reconkit/0.2 (+https://github.com/no1rstack/reconkit)",
+    retries: int = 2,
+    max_redirects: int = 10,
+    allow_private_hosts: bool = False,
+    robots_fail_closed: bool = True,
+) -> FetchResult:
+    """Fetch a URL with conservative defaults.
+
+    No anti-bot bypass behavior is implemented (no proxy rotation, no fingerprinting evasion).
+    """
+
+    _validate_url(url, allow_private_hosts=allow_private_hosts)
+    limiter = RateLimiter(requests_per_second=rps)
+
+    headers = {"User-Agent": user_agent, "Accept": "text/html,application/xhtml+xml,*/*;q=0.8"}
+    timeout = httpx.Timeout(timeout_s)
+
+    redirects: list[dict[str, object]] = []
+    current_url = url
+    with httpx.Client(follow_redirects=False, timeout=timeout, headers=headers) as client:
+        for _ in range(max_redirects + 1):
+            _validate_url(current_url, allow_private_hosts=allow_private_hosts)
+            limiter.wait()
+            if obey_robots and not robots.can_fetch(
+                current_url, user_agent=user_agent, timeout_s=min(10.0, timeout_s), client=client,
+                fail_closed=robots_fail_closed,
+            ):
+                raise RobotsDisallowedError("Acquisition denied because robots.txt does not authorize this URL.")
+            for attempt in range(retries + 1):
+                try:
+                    with client.stream("GET", current_url) as resp:
+                        location = resp.headers.get("location")
+                        if resp.status_code in {301, 302, 303, 307, 308} and location:
+                            next_url = urljoin(current_url, location)
+                            redirects.append({"from": current_url, "to": next_url, "status_code": resp.status_code})
+                            current_url = next_url
+                            break
+                        if resp.status_code >= 400:
+                            raise AcquisitionFailureError(f"Collection failed with HTTP status {resp.status_code}.")
+                        chunks: list[bytes] = []
+                        size = 0
+                        for chunk in resp.iter_bytes():
+                            size += len(chunk)
+                            if max_bytes and size > max_bytes:
+                                raise EvidenceSizeLimitError(f"Evidence size limit exceeded (max_bytes={max_bytes}).")
+                            chunks.append(chunk)
+                        return FetchResult(
+                            requested_url=url, url=str(resp.url), status_code=resp.status_code,
+                            headers={k: v for k, v in resp.headers.items()}, body=b"".join(chunks),
+                            encoding=resp.encoding, observed_at=datetime.now(timezone.utc),
+                            redirect_chain=tuple(redirects),
+                        )
+                except AcquisitionFailureError:
+                    raise
+                except (httpx.TimeoutException, httpx.NetworkError) as exc:
+                    if attempt >= retries:
+                        raise AcquisitionFailureError("Collection failed because the destination was unavailable.") from exc
+            else:
+                raise AcquisitionFailureError("Collection retry loop ended unexpectedly.")
+            continue
+    raise AcquisitionFailureError(f"Redirect limit exceeded ({max_redirects}).")

{reconkit-0.1.0 → reconkit-0.2.0}/src/reconkit/scraping/robots.py

@@ -19,12 +19,12 @@ def can_fetch(
     user_agent: str,
     timeout_s: float = 10.0,
     client: httpx.Client | None = None,
+    fail_closed: bool = True,
 ) -> bool:
     """Return whether `user_agent` is allowed to fetch `url` per robots.txt.
 
-    Safe defaults:
-    - If robots.txt cannot be fetched/parsed, default to allow.
-    - Caches robots parsers per (scheme, netloc).
+    When ``fail_closed`` is true, an unavailable or malformed robots.txt denies
+    collection. That is the conservative default for evidence collection.
     """
 
     parsed = urlparse(url)
@@ -43,18 +43,17 @@ def can_fetch(
                 resp = client.get(robots_txt)
 
             if resp.status_code >= 400:
-                # treat missing/blocked robots as allow
                 _CACHE[key] = rp
-                return True
+                return not fail_closed
 
             rp.parse(resp.text.splitlines())
         except Exception:
             _CACHE[key] = rp
-            return True
+            return not fail_closed
 
         _CACHE[key] = rp
 
     try:
         return bool(rp.can_fetch(user_agent, url))
     except Exception:
-        return True
+        return not fail_closed

reconkit-0.2.0/tests/test_evidence_exports.py

@@ -0,0 +1,48 @@
+from __future__ import annotations
+
+import json
+import zipfile
+from datetime import datetime, timezone
+from pathlib import Path
+
+from reconkit.evidence.archives import export_wacz, export_warc
+from reconkit.evidence.custody import CustodyLedger
+from reconkit.evidence.manifest import ManifestSigner, verify_manifest, write_manifest
+from reconkit.scraping.fetch import FetchResult
+
+
+def capture() -> FetchResult:
+    return FetchResult(
+        requested_url="https://example.com", url="https://example.com/final", status_code=200,
+        headers={"content-type": "text/plain"}, body=b"evidence", encoding="utf-8",
+        observed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
+        redirect_chain=({"from": "https://example.com", "to": "https://example.com/final", "status_code": 302},),
+    )
+
+
+def test_warc_and_wacz_include_standard_manifests(tmp_path: Path):
+    warc = export_warc([capture()], tmp_path / "capture.warc")
+    assert b"WARC/1.1" in warc.read_bytes()
+    wacz = export_wacz([capture()], tmp_path / "capture.wacz", title="Test evidence")
+    with zipfile.ZipFile(wacz) as archive:
+        assert set(archive.namelist()) == {"archive/data.warc", "indexes/index.cdxj", "pages/pages.jsonl", "datapackage.json", "datapackage-digest.json"}
+        manifest = json.loads(archive.read("datapackage.json"))
+        assert manifest["profile"] == "wacz"
+        assert len(manifest["resources"]) == 3
+
+
+def test_custody_hash_chain_and_signed_manifest_detect_tampering(tmp_path: Path):
+    evidence = tmp_path / "capture.warc"
+    evidence.write_bytes(b"captured bytes")
+    ledger = CustodyLedger(tmp_path / "custody.jsonl")
+    ledger.append(action="acquired", actor="analyst@example.test", artifact_path=evidence)
+    ledger.append(action="exported", actor="analyst@example.test", artifact_path=evidence)
+    assert ledger.verify()
+    signer = ManifestSigner.generate()
+    manifest = write_manifest([evidence], tmp_path / "manifest.json", signer, custody_ledger=ledger.path)
+    assert verify_manifest(manifest)
+    assert verify_manifest(manifest, evidence_dir=tmp_path)
+    contents = json.loads(manifest.read_text(encoding="utf-8"))
+    contents["manifest"]["files"][0]["bytes"] = 1
+    manifest.write_text(json.dumps(contents), encoding="utf-8")
+    assert not verify_manifest(manifest)

reconkit-0.1.0/PKG-INFO

@@ -1,108 +0,0 @@
-Metadata-Version: 2.4
-Name: reconkit
-Version: 0.1.0
-Summary: A small, safe-by-default toolkit for scraping, OSINT, and file triage.
-Author: reconkit
-License: MIT
-License-File: LICENSE
-Keywords: forensics,osint,scraping,triage
-Requires-Python: >=3.10
-Requires-Dist: beautifulsoup4>=4.12
-Requires-Dist: dnspython>=2.6
-Requires-Dist: exifread>=3.0
-Requires-Dist: filetype>=1.2
-Requires-Dist: httpx>=0.27
-Requires-Dist: ipwhois>=1.2
-Requires-Dist: lxml>=5.1
-Requires-Dist: python-whois>=0.9
-Requires-Dist: rich>=13.7
-Requires-Dist: typer>=0.12
-Provides-Extra: dev
-Requires-Dist: build>=1.2; extra == 'dev'
-Requires-Dist: mypy>=1.10; extra == 'dev'
-Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
-Requires-Dist: pytest>=8.0; extra == 'dev'
-Requires-Dist: ruff>=0.4; extra == 'dev'
-Requires-Dist: twine>=5.1; extra == 'dev'
-Description-Content-Type: text/markdown
-
-# reconkit (Judicium surface)
-
-`reconkit` is a **conservative, safe-by-default** toolkit that exposes a Noir Stack / Hexarch / Judicium-flavored public API.
-
-It is designed to produce **audit-grade structured outputs** that explicitly state:
-- what was observed (evidence)
-- how it was observed (provenance)
-- what is and is not being asserted (verification limits)
-
-This project is intended for lawful use on systems/data you own or have permission to assess.
-
-## Noir Stack primitives
-
-Public primitives:
-- `EvidenceArtifact`: immutable bytes + provenance
-- `EvidenceBundle`: a set of artifacts for joint assessment
-- `ProvenanceRecord`: acquisition/derivation metadata + trust boundary
-- `VerificationResult`: structured decision output
-
-All structured outputs include:
-- `decision_basis`
-- `verification_status`
-- `provenance_confidence`
-
-Because this is Judicium, not divination.
-
-## Guarantees
-
-What the system will reliably do:
-- Preserve evidence payload bytes as acquired/derived, and compute stable SHA-256 for artifacts.
-- Emit structured, JSON-serializable outputs suitable for logging and later review.
-- Respect declared acquisition constraints (e.g., `max_bytes`) and treat them as policy boundaries.
-- Respect `robots.txt` by default for web acquisition.
-
-## Assumptions
-
-What the system assumes unless you override it:
-- `trust_boundary` is declared explicitly for acquisitions. The tool will not invent one.
-- Web acquisition is transport-level observation, not identity/authenticity proof.
-- Public-internet boundary data (DNS/WHOIS/RDAP) is administrative metadata.
-
-## Non-goals
-
-What the system explicitly does not do:
-- No anti-bot bypassing, fingerprint evasion, proxy-rotation orchestration, or credentialed scraping.
-- No claims of attribution, intent, or identity. (You can do that in your report; the tool will not do it for you.)
-- No “automatic truth.” The system is already busy being correct.
-
-## Install
-
-```bash
-python -m pip install -e ".[dev]"
-```
-
-## CLI quickstart (Noir terminology)
-
-All commands emit audit-grade JSON.
-
-```bash
-# Evidence acquisition / derivation
-reconkit evidence acquire-web "https://example.com" --trust-boundary public-internet
-reconkit evidence derive-links "https://example.com"
-reconkit evidence derive-text "https://example.com"
-
-# Boundary observations (OSINT-ish, but we say the quiet part out loud)
-reconkit boundary dns example.com
-reconkit boundary whois example.com
-reconkit boundary ip 8.8.8.8
-
-# Verification (informational)
-reconkit verify file-hash .\somefile.bin --algo sha256
-reconkit verify file-type .\somefile.bin
-reconkit verify strings .\somefile.bin --min-len 6
-reconkit verify exif .\photo.jpg
-```
-
-## Notes
-
-- If acquisition is blocked by `robots.txt`, that is not a “bug.” It is a boundary condition.
-- If you need features that reduce accountability, you are in the wrong repository. Convenient.

reconkit-0.1.0/README.md

@@ -1,80 +0,0 @@
-# reconkit (Judicium surface)
-
-`reconkit` is a **conservative, safe-by-default** toolkit that exposes a Noir Stack / Hexarch / Judicium-flavored public API.
-
-It is designed to produce **audit-grade structured outputs** that explicitly state:
-- what was observed (evidence)
-- how it was observed (provenance)
-- what is and is not being asserted (verification limits)
-
-This project is intended for lawful use on systems/data you own or have permission to assess.
-
-## Noir Stack primitives
-
-Public primitives:
-- `EvidenceArtifact`: immutable bytes + provenance
-- `EvidenceBundle`: a set of artifacts for joint assessment
-- `ProvenanceRecord`: acquisition/derivation metadata + trust boundary
-- `VerificationResult`: structured decision output
-
-All structured outputs include:
-- `decision_basis`
-- `verification_status`
-- `provenance_confidence`
-
-Because this is Judicium, not divination.
-
-## Guarantees
-
-What the system will reliably do:
-- Preserve evidence payload bytes as acquired/derived, and compute stable SHA-256 for artifacts.
-- Emit structured, JSON-serializable outputs suitable for logging and later review.
-- Respect declared acquisition constraints (e.g., `max_bytes`) and treat them as policy boundaries.
-- Respect `robots.txt` by default for web acquisition.
-
-## Assumptions
-
-What the system assumes unless you override it:
-- `trust_boundary` is declared explicitly for acquisitions. The tool will not invent one.
-- Web acquisition is transport-level observation, not identity/authenticity proof.
-- Public-internet boundary data (DNS/WHOIS/RDAP) is administrative metadata.
-
-## Non-goals
-
-What the system explicitly does not do:
-- No anti-bot bypassing, fingerprint evasion, proxy-rotation orchestration, or credentialed scraping.
-- No claims of attribution, intent, or identity. (You can do that in your report; the tool will not do it for you.)
-- No “automatic truth.” The system is already busy being correct.
-
-## Install
-
-```bash
-python -m pip install -e ".[dev]"
-```
-
-## CLI quickstart (Noir terminology)
-
-All commands emit audit-grade JSON.
-
-```bash
-# Evidence acquisition / derivation
-reconkit evidence acquire-web "https://example.com" --trust-boundary public-internet
-reconkit evidence derive-links "https://example.com"
-reconkit evidence derive-text "https://example.com"
-
-# Boundary observations (OSINT-ish, but we say the quiet part out loud)
-reconkit boundary dns example.com
-reconkit boundary whois example.com
-reconkit boundary ip 8.8.8.8
-
-# Verification (informational)
-reconkit verify file-hash .\somefile.bin --algo sha256
-reconkit verify file-type .\somefile.bin
-reconkit verify strings .\somefile.bin --min-len 6
-reconkit verify exif .\photo.jpg
-```
-
-## Notes
-
-- If acquisition is blocked by `robots.txt`, that is not a “bug.” It is a boundary condition.
-- If you need features that reduce accountability, you are in the wrong repository. Convenient.

reconkit-0.1.0/src/reconkit/scraping/fetch.py

@@ -1,81 +0,0 @@
-from __future__ import annotations
-
-from dataclasses import dataclass
-from typing import Optional
-
-import httpx
-
-from reconkit.core.rate_limit import RateLimiter
-from reconkit.scraping import robots
-from reconkit.noir.errors import AcquisitionFailureError, EvidenceSizeLimitError, RobotsDisallowedError
-
-
-@dataclass
-class FetchResult:
-    url: str
-    status_code: int
-    headers: dict[str, str]
-    body: bytes
-    encoding: Optional[str]
-
-
-def fetch_url(
-    url: str,
-    *,
-    max_bytes: int = 2_000_000,
-    timeout_s: float = 20.0,
-    rps: float = 1.0,
-    obey_robots: bool = True,
-    user_agent: str = "reconkit/0.1 (+https://localhost)",
-    retries: int = 2,
-) -> FetchResult:
-    """Fetch a URL with conservative defaults.
-
-    No anti-bot bypass behavior is implemented (no proxy rotation, no fingerprinting evasion).
-    """
-
-    limiter = RateLimiter(requests_per_second=rps)
-    limiter.wait()
-
-    headers = {"User-Agent": user_agent, "Accept": "text/html,application/xhtml+xml,*/*;q=0.8"}
-    timeout = httpx.Timeout(timeout_s)
-
-    with httpx.Client(follow_redirects=True, timeout=timeout, headers=headers) as client:
-        if obey_robots and not robots.can_fetch(url, user_agent=user_agent, timeout_s=min(10.0, timeout_s), client=client):
-            raise RobotsDisallowedError(
-                "Acquisition denied by robots.txt policy. "
-                "Trust boundary: remote publisher policy; provenance cannot be improved by bypass."
-            )
-
-        for attempt in range(retries + 1):
-            try:
-                with client.stream("GET", url) as resp:
-                    resp.raise_for_status()
-                    chunks: list[bytes] = []
-                    size = 0
-                    for chunk in resp.iter_bytes():
-                        chunks.append(chunk)
-                        size += len(chunk)
-                        if max_bytes and size > max_bytes:
-                            raise EvidenceSizeLimitError(
-                                f"Evidence size limit exceeded (max_bytes={max_bytes}). "
-                                "Trust boundary: acquisition constraints; partial evidence is not emitted by default."
-                            )
-                    body = b"".join(chunks)
-
-                    return FetchResult(
-                        url=str(resp.url),
-                        status_code=resp.status_code,
-                        headers={k: v for k, v in resp.headers.items()},
-                        body=body,
-                        encoding=resp.encoding,
-                    )
-            except (httpx.TimeoutException, httpx.NetworkError):
-                if attempt >= retries:
-                    raise AcquisitionFailureError(
-                        "Acquisition failed within the declared trust boundary (network/timeout). "
-                        "Decision basis: transport failure; provenance confidence unchanged."
-                    )
-                limiter.wait()
-
-        raise RuntimeError("Unreachable")