paytrace 2.0.0__tar.gz

paytrace-2.0.0/.github/ISSUE_TEMPLATE/new_collector.md

@@ -0,0 +1,18 @@
+---
+name: New collector proposal
+about: Propose a new data source
+labels: collector
+---
+
+**Source** (URL and what it publishes)
+
+**Legal basis** — statutory registry / IAB spec / RFC protocol / mandated disclosure
+
+**Auth** — keyless, free key, or paid (paid sources need a strong argument)
+
+**Rate limit and robots.txt status**
+
+**Claims emitted** — predicate, reliability, and what constitutes one
+`correlation_group`
+
+**Selectivity** — roughly how many entities share a typical value?

paytrace-2.0.0/.gitignore

@@ -0,0 +1,150 @@
+# ─────────────────────────────────────────────────────────────────────────
+# INVESTIGATION ARTIFACTS — never commit
+#
+# These are the real leak risk in this project. Case files carry the
+# authorization reference (often an internal ticket number). Evidence
+# packages carry bytes captured from a live target. Audit logs and graphs
+# carry the identifiers of whoever was investigated. A corpus index carries
+# a crawl of real domains.
+#
+# None of it is method, all of it is operational, and once it is in git
+# history it is in every clone and fork forever.
+# ─────────────────────────────────────────────────────────────────────────
+
+# Case files (authorization refs, seeds, contact emails)
+case.yaml
+case-*.yaml
+case_*.yaml
+*.case.yaml
+!case.example.yaml
+!case.demo.yaml
+
+# Run output
+out/
+out-*/
+output/
+results/
+runs/
+
+# Evidence packages: preserved response bodies, manifests, declarations
+evidence/
+captures/
+evidence_manifest.json
+DECLARATION_DRAFT.md
+TIMESTAMP.md
+verify.py
+!src/**/verify.py
+*.tsq
+*.tsr
+
+# Graphs and reports
+investigation_graph.json
+investigation_graph.minimized.json
+entities.ftm.json
+graph.cypher
+attribution_report.md
+attribution_report.html
+verification_trail.md
+verification_trail.json
+portfolio.json
+handle_graph.json
+assessments.json
+
+# Audit logs
+audit.jsonl
+audit-*.jsonl
+*.audit.jsonl
+
+# Corpus indexes and crawl data
+*.sqlite
+*.sqlite3
+*.db
+!tests/**/*.sqlite
+paytrace.sqlite
+domains.txt
+tranco*.csv
+tranco*.txt
+
+# Operational blocklists — data, not method
+infra_blacklist*.txt
+*blacklist*.txt
+!tests/**/*blacklist*.txt
+
+# Imported third-party investigation exports
+investigations/
+spiderfoot*.db
+spiderfoot*.csv
+*.stix.json
+opencti*.json
+robin*.json
+
+# Handle observations (real handles from real cases)
+handles*.csv
+observations*.csv
+!examples/demo.csv
+
+# Calibration corpora (labelled pairs from closed cases)
+labelled_pairs.json
+pairs.json
+train.json
+test.json
+calibration_report.md
+corpus/
+
+# HTTP response cache
+.eae-cache/
+.cache/
+
+# ─────────────────────────────────────────────────────────────────────────
+# Secrets
+# ─────────────────────────────────────────────────────────────────────────
+.env
+.env.*
+!.env.example
+*.pem
+*.key
+!**/tests/**/*.key
+.pypirc
+credentials.json
+secrets.yaml
+
+# ─────────────────────────────────────────────────────────────────────────
+# Python
+# ─────────────────────────────────────────────────────────────────────────
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+dist/
+sdist/
+wheels/
+*.egg-info/
+*.egg
+MANIFEST
+.venv/
+venv/
+ENV/
+.tox/
+.nox/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+.hypothesis/
+
+# ─────────────────────────────────────────────────────────────────────────
+# Editors and OS
+# ─────────────────────────────────────────────────────────────────────────
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+.DS_Store
+Thumbs.db
+.verify-venv/

paytrace-2.0.0/AUDIT_RESPONSE.md

@@ -0,0 +1,167 @@
+# Response to external audit
+
+Twenty-five findings. I agree with twenty-three, partially disagree with one,
+and disagree with one. This records what changed, what did not, and why.
+
+**The audit's central point is correct and I have acted on it:** the system is
+an evidence-ranking engine and the API described it as a probabilistic
+attribution engine. My own methodology audit said "an ordering, not a measured
+frequency" and then shipped a field called `probability` and a band called
+`ATTRIBUTED`. That gap between documentation and surface is exactly the failure
+the auditor identified.
+
+---
+
+## Fixed — the four that were dangerous
+
+**1. CI failed open.** `|| true` on integration tests, examples and `pip-audit`.
+Documentation claimed the integration suite gated releases; GitHub did not.
+All three now fail the build. `pip-audit` is renamed "Dependency CVEs (fails the
+build)" because a scan that cannot fail is theatre.
+
+**2. `max_bytes` did not prevent OOM.** `r.content[:max_bytes]` materialises the
+whole body first, so a 5 GB response exhausted memory before the slice applied.
+The docstring claimed a protection the code did not provide, which is the worst
+class of bug. Now streams with `aiter_bytes()` and stops reading at the cap,
+after transfer decoding so decompression bombs are bounded by the same limit.
+
+**3. Egress did not control transport.** `EgressPool` sat beside a single
+`httpx.AsyncClient` that never used it; `self._clients` was dead. The manifest
+would have recorded a vantage point the request never used — fabricated
+provenance, worse than none. `Fetcher.client_for(label)` now builds one proxied
+client per egress, and a misconfigured egress raises rather than silently
+falling back to direct.
+
+**4. Cache key omitted the egress.** A US capture would have served a request
+that asked for a DE vantage point. Key now includes egress label and country.
+
+## Fixed — semantics
+
+**5. Stopped calling the output a probability.** `log(1/selectivity)` informs
+the *denominator* of a likelihood ratio; `P(evidence | same entity)` is not
+modelled and varies enormously between a shared GA4 ID and a shared company
+address. Multiplying by a subjective reliability coefficient does not restore
+the missing term. Output now carries `calibration_status: "unvalidated"` and a
+`probability_note` stating plainly what the number is.
+
+**23. Renamed the bands.** `ATTRIBUTED` / `PROBABLE` / `POSSIBLE` became
+`STRONG_EVIDENCE` / `MODERATE_EVIDENCE` / `LIMITED_EVIDENCE`, and ICD 203
+estimative language ("almost certainly") became evidence-strength phrasing
+("strongly supported by the evidence"). The auditor's argument is the right one:
+downstream users quote the strongest surface, and a disclaimer in a methodology
+file does not travel with a word pasted into a report.
+
+**7 & 21. Dependence is typed metadata, and the default is conservative.**
+`Claim.dependence_class` is declared by the collector and always wins; regex
+matching survives only as a fallback for older claims. Unrecognised evidence is
+now `UNKNOWN` and discounted at 0.20, not assumed independent. My original
+argument — that a wrong discount is harder to notice than a missing one — was
+worse than the auditor's: for a model whose dominant failure mode is
+overconfidence, unrecognised evidence should fail conservative.
+
+**9. Agent no longer reimplements domain logic.** The `sellers.json` tool
+hard-coded `ORG_NAME` while `classify_seller_name()` existed, so the agent path
+typed individual publishers as organisations and routed them to corporate
+registries that hold no record. It now delegates.
+
+**12. Version reads source truth.** `importlib.metadata.version()` returned a
+fallback for an uninstalled source tree, so a valid run reported a divergence
+that did not exist.
+
+**11. Test taxonomy.** Markers for `network`, `browser`, `integration`,
+`adversarial`, `slow`. Default `addopts` excludes network and browser, so
+`pytest` is green on a machine without DNS or Chromium — which is an environment
+fact, not a defect.
+
+**25. Exception taxonomy.** `EXPECTED_FAILURES` covers the world being
+uncooperative; anything else is our defect and raises under `strict=True`. A
+`TypeError` reporting as "tool failed" made production bugs indistinguishable
+from a source being down.
+
+---
+
+## Where I partially disagree — the monorepo
+
+The auditor is right about the operational cost: four CI files, four LICENSEs,
+four CHANGELOGs, synchronised versioning, and integration tests that hope
+siblings exist at `../`. That is a distributed monorepo, and it is fragile.
+
+But I do not think the answer is one repository *and* one package. The
+`attribution-graph` boundary is real and load-bearing: **it performs no network
+I/O**, which is precisely what makes its scoring auditable, and that guarantee
+is enforced by the package boundary rather than by convention. Someone who wants
+the inference model should not have to install an HTTP client and 36 collectors.
+
+So: **one repository, four published packages.** A workspace, not a merge. That
+takes the auditor's operational fix — one CI pipeline where integration failures
+are fatal, one release process, no duplicated workflow config — while keeping
+the boundary that carries a technical guarantee.
+
+This is staged rather than done. The repository layout change is mechanical but
+touches every path in `PUSH_INSTRUCTIONS.md`, and doing it half-way would be
+worse than either end state.
+
+---
+
+## Where I disagree
+
+**14, on cache versus evidence.** The auditor reads these as conflated. They are
+already separate objects: `EvidenceLog` writes content-addressed, hash-chained
+captures under `evidence/`, while `.eae-cache/` is a replaceable performance
+cache. The `net.py` comment saying "the cached body is the artifact" is wrong and
+has been corrected — but the architecture was right; only the comment was not.
+
+The auditor's underlying point about **capture completeness** is correct and is
+now staged: the evidence record should carry the redirect chain, request and
+response headers, TLS peer info, wire-body versus decoded-body hashes, and
+truncation state. That is a real gap.
+
+---
+
+## Staged, with reasons
+
+**3. DNS rebinding / TOCTOU.** Correct, and not yet fixed. Validation resolves
+the host; `httpx` resolves again when connecting. Closing it properly means a
+custom transport that pins the validated address while preserving Host header,
+SNI and certificate verification — worth doing carefully rather than quickly.
+**Documented as a known limitation in `SECURITY.md` and the audit** rather than
+left implied.
+
+**6. Candidate-generation context.** The auditor is right that a global prior
+applied to deliberately-selected candidates is selection bias, and that no amount
+of tuning fixes it. The fix is an `AssessmentContext` recording how a candidate
+entered the system, so calibration can be conditional on the selection rule.
+That is a modelling change that should land *with* the calibration study, not
+before it.
+
+**8. `Reliability` conflates three concepts** — observation fidelity, source
+independence, binding strength. This is the deepest finding after #5, and the
+auditor is right that `AUTHORITATIVE` on a subject-controlled page is
+misleading. Splitting it is a breaking change across every collector, and it
+should be done once, deliberately.
+
+**13. Capture completeness.** Above.
+
+**24. Predicate policy table.** Behaviour currently spread across `model.py`,
+`scoring.py`, `dependence.py` and collectors. Consolidating is correct and
+mechanical.
+
+---
+
+## What the audit changed about the project's claim
+
+From:
+
+> determine which legal entity or person operates a domain, with a confidence
+> figure
+
+To:
+
+> build an auditable evidence graph for entity-attribution investigations, rank
+> candidate relationships by evidence strength, and preserve the provenance
+> necessary for independent review
+
+The auditor's closing observation is the one worth keeping: narrowing the claim
+makes the project **more** credible, not less. The narrow claim is supported by
+what exists. The broad one is not, and would not have survived first contact
+with a reviewer who checked.

paytrace-2.0.0/AUDIT_RESPONSE_2.md

@@ -0,0 +1,130 @@
+# Response to re-audit (v1.6.0 → v1.7.0)
+
+**Every finding I checked was reproducible.** I verified the five criticals
+against the code before changing anything, and each behaved exactly as the
+report described. No pushback on any P0 item.
+
+The report's framing is the right one and I am adopting it: the problem is
+**assurance mismatch**. Good local implementations existed while the
+user-facing execution path did not compose their guarantees end-to-end. Fixing
+that was this pass; no features were added.
+
+---
+
+## Verified before fixing
+
+| Finding | Reproduced |
+|---|---|
+| EA-01 entry hash omits headers/collector/note/body_path | yes — all five unhashed |
+| EA-02 suffix deletion verifies | yes — deleted a capture, verifier said PASSED |
+| EA-03 runner never wires EvidenceLog | yes — `Fetcher(...)` built with no log |
+| EA-04 definitional self-grant | yes — `subject_html` claim merged evil.example into Victim Corp |
+| EA-05 unique emails corroborate | yes — 6 different emails → 7 points, HIGH, durable=True |
+| EA-09 budget bypassed by redirects | yes — `max_requests` not re-checked per hop |
+| EA-10 declared dependence ignored | yes — never gathered in `assess()`, never serialised |
+| EA-16 VERIFY.sh absent from repos | yes — existed only at workspace root |
+
+---
+
+## P0 — fixed
+
+**EA-01.** The entry hash now covers a versioned canonical envelope: request
+headers, response headers, collector, note, body_path, egress, outcome. Bumping
+`ENTRY_SCHEMA` is required to change the field set, so a future field cannot
+fall outside it by omission. Four tamper cases are permanent tests.
+
+**EA-02.** The generated verifier recomputes the chain head and compares it to
+the declared `manifest_hash`, and reconciles `capture_count`. Deleting the final
+capture now fails with an explicit message. The verifier also states what it
+**cannot** prove — internal consistency is not provenance, and only an external
+anchor establishes that the package was not regenerated wholesale.
+
+**EA-03 / EA-06.** Capture recording and policy evaluation are now transport
+invariants: the layer performing the request records it, including refusals and
+errors. `run_case()` passes the log and the policy engine into `Fetcher`, and
+reconciles fetch count against capture count — a non-empty run producing an
+empty package now fails verification rather than reporting `evidence_verified=True`.
+
+**EA-04.** `is_definitional()` requires an engine-recognised registry collector
+or source class, matched literally. Reliability is a scalar the claim author
+supplies, so keying the model's strongest privilege on it made the carve-out
+self-service. The hostile claim now scores UNSUPPORTED and does not merge;
+`gleif` retains the exemption.
+
+**EA-05.** A correlation point must connect observations. Identifiers are keyed
+by value and counted only when the same normalised value appears on two or more
+distinct profiles; singletons are reported in the caveat rather than silently
+dropped. The reproduced case is now a permanent test: six profiles, six
+different emails → **1 point, INSUFFICIENT**, down from 7/HIGH.
+
+One existing test had encoded the bug — five profiles with five *different*
+names and domains, asserting MODERATE. Rewritten to share the identifiers so it
+tests the cap it was written for.
+
+**EA-07.** `Fetcher` is constructed from `EgressPool.from_case(scope.egress)`.
+
+**EA-09.** Budget is checked before every physical request including redirect
+hops. `max_requests=1` with a 302→200 chain now raises rather than completing at
+count=2.
+
+**EA-10.** `assess()` gathers declared classes per canonical group, rejects
+conflicting declarations within a group, and passes them through. Serialised in
+`Claim.to_dict()`.
+
+**EA-11.** Merge policy moved out of probability space:
+`MERGE_LLR_THRESHOLD` in nats, applied to `log_odds`. Renaming the display while
+keeping a 0.90 cut on the same unvalidated number would have been cosmetic.
+
+**EA-15.** CI clones sibling repositories before the integration step, so the
+gate can actually pass rather than being structurally red.
+
+**EA-16.** `VERIFY.sh` ships in all four repositories. It existed only at the
+workspace root while the docs told users to run it.
+
+**EA-17 / EA-18.** Screenshot tests use the declared `browser` marker rather
+than a `skipif` alias, so default `addopts` actually excludes them.
+`available_renderer()` resolves and stats the Chromium executable instead of
+treating package importability as availability. `Fetcher` accepts an injected
+resolver, making the transport tests deterministic — mocking HTTP was not
+enough, because validation performed real DNS first.
+
+**EA-22.** Stale `net.py` comment corrected. Test counts reconciled: **723
+collected, 714 in the deterministic default.**
+
+---
+
+## Accepted, staged, with the reason
+
+**EA-12 — screenshot is a second network stack outside the SSRF boundary.**
+Correct, and it also breaks the "no network I/O in the core" argument I used to
+justify the package split. `screenshot.py` should move to a capture package
+behind the same policy boundary, and browser subresource requests need route
+interception. That is a package boundary change plus a sandboxing design, and
+doing it badly would be worse than the current documented state.
+
+**EA-13 — `parent_body_sha256` is caller-declared.** Right: Playwright
+navigates the live URL independently, so a page changing between fetch and
+screenshot yields screenshot B labelled as rendering body A. The fix is to
+render the preserved body or hash the browser's actual main-document response.
+
+**EA-14 — screenshot CLI flags are no-ops.** Confirmed. Rather than leave a
+flag that does nothing, the flags are staged with the capture path.
+
+**EA-19 / EA-20 / EA-21.** Exception taxonomy narrowing in `Engine`, audit-log
+minimisation at the serializer boundary, and operator/hostname metadata as an
+explicit choice. All correct, none load-bearing for the P0 assurance story.
+
+---
+
+## On the release framing
+
+I accept the recommendation. This is **not** "ready to publish". `DESIGN.md`
+and `PUBLICATION_PLAN.md` now describe it as a **pre-release experimental
+research codebase**, and the evidence package is no longer described as
+tamper-evident without the qualifier that deletion resistance requires an
+external anchor.
+
+The two things that would change that assessment, in order: replace the
+LICENSE/OWNER placeholders, and run the calibration study. Everything else on
+the P1/P2 list is quality work on a system whose assurances now at least
+compose.

paytrace-2.0.0/AUDIT_RESPONSE_3.md

@@ -0,0 +1,138 @@
+# Response to release-readiness audit (v1.7.0 → v1.7.1)
+
+**Decision accepted: HOLD was correct.** Four of the five mandatory blockers are
+fixed; the fifth is licensing, which needs your decision rather than a code
+change.
+
+One finding I have to report before the substance.
+
+---
+
+## A discrepancy in the artifacts I shipped
+
+The auditor tested the v1.7.0 tarballs. Reproducing their findings, I discovered
+the working tree contained fixes for RB-01, RA-02 and RA-03 that were **not in
+the tarballs I shipped**. I did not make those edits during the session that
+produced v1.7.0.
+
+I cannot explain the divergence, and for a tool whose entire purpose is evidence
+integrity that is worth stating plainly rather than quietly reconciling. What I
+have done:
+
+- treated the **tarballs** as authoritative, since that is what was audited;
+- re-run every one of the auditor's reproductions against the current tree
+  rather than assuming any fix was real;
+- verified each remaining fix myself before claiming it.
+
+Concretely: RB-01, RA-02 and RA-03 were already correct in the tree and I
+confirmed them. RB-02, RB-03, RB-04, RA-01 and RA-04 were **not** fixed and I
+fixed them in this pass.
+
+---
+
+## RB-02 — my own regression, and worse than reported
+
+The auditor found `MERGE_LLR_THRESHOLD = 13.71` nats compared against
+`a.probability`, which cannot exceed 1.0. Correct, and it was my fix from the
+previous round: I renamed the constant's units without changing what it was
+compared to, converting a tuning question into a correctness bug that failed
+silently in the quiet direction.
+
+The constant was also wrong on its own terms. `Assessment.log_odds` already
+includes the prior, so adding `|log(PRIOR_ODDS)|` double-counted it. The
+posterior threshold preserving the old 0.90 boundary is `log(0.9/0.1) = 2.197`.
+
+**But fixing the threshold did not restore merging**, and this is the more
+consequential finding. The resolver only scored directly-claimed
+subject→object edges. Two domains sharing an identifier were **never assessed
+as a pair at all** — the toolkit's canonical case. Each leg had one correlation
+group, the corroboration rule correctly refused to merge on one group, and the
+two-group evidence that actually existed was never pooled.
+
+So no threshold could have helped: inferential merging had never worked.
+
+`resolve()` now generates co-reference candidates — subjects sharing an object —
+and pools both legs, which is the standard record-linkage construction.
+Measured: two domains sharing two identifiers now merge at log_odds 8.89 with
+two independent groups; sharing one identifier still does not merge.
+
+## RB-03 — the constraint passed for the wrong reason
+
+The transitive must-not-link check appeared to pass in my first run. It passed
+because nothing merged at all. With merging restored, the real behaviour is
+correct: A~B merges, and B~C is **rejected** because the union would place A and
+C in one cluster against an explicit contradiction.
+
+That is a lesson worth recording: a constraint test that passes while the
+mechanism it constrains is disabled proves nothing.
+
+## RB-04 — verifier no longer executes package code
+
+`attribution verify` ran the `verify.py` inside the evidence directory. An
+evidence package is untrusted input by definition. New
+`attribution_graph.verify` reads the manifest as data, refuses `body_path`
+values escaping the package, and the CLI uses it. Tested with a planted script
+that writes a marker file: exit 0, marker absent.
+
+The generated standalone `verify.py` still ships — a dependency-free checker a
+reader can inspect in full has real value — and the CLI now prints a note saying
+it was *not* executed and why.
+
+## RB-05 — licensing
+
+Unresolved, and not something I should decide. All four LICENSE files remain
+placeholders while `pyproject.toml` asserts Apache-2.0, so the built metadata
+makes a claim the shipped file does not support. `VERIFY.sh` fails on this by
+design. ~20 files still carry `github.com/tusharkarumudi/`.
+
+---
+
+## RA items
+
+**RA-01 — evidence completeness.** 404s and redirect hops were already recorded
+in the tree; I verified both. The cache case was not: a second run consumed
+retrieved content and produced a package with zero captures that verified
+clean. Cache hits now emit a capture with `outcome="cache"` carrying the body,
+so the digest still commits to what was used and a reviewer can see the bytes
+were not fetched during that run.
+
+**RA-04 — provider/network mismatch.** `provider: oxylabs` with
+`network: datacenter` was accepted and routed to `pr.oxylabs.io`, the profile's
+own documented *residential* endpoint, while recording
+`consent_sensitive: false`. Provenance and safeguard wrong at once. Profiles now
+declare which networks they serve and a contradicting label is rejected.
+
+**RA-06 / 8.3 / 8.4 — screenshots.** Taken off the release surface.
+`--screenshots` now exits 2 with an explanation rather than being silently
+accepted and doing nothing. Kept as an explicit error rather than removed so a
+user with it in a script learns why.
+
+---
+
+## Phase D gate
+
+All six tests the auditor specified now exist and pass:
+
+1. real `PolicyEngine` + real `Fetcher` across respect / record / ignore
+2. resolver positive merge — two independent groups, non-definitional
+3. resolver cluster constraint — A~B, B~C, A!~C
+4. verifier safety — planted script not executed
+5. evidence completeness — 404, redirect, cache
+6. case egress — survives `load()`, reaches transport, mismatch rejected
+
+Writing (1) surfaced a further defect in my own test: a shared cache directory
+served mode 2 from mode 1's fetch, so the assertion passed for the wrong reason.
+Each mode now gets its own cache.
+
+**739 tests collected, 739 passing in the deterministic default.**
+
+---
+
+## Still open, deliberately
+
+DNS rebinding through remote proxies; moving `screenshot.py` out of the
+inference package; collector-by-collector `dependence_class` adoption; the
+calibration study. All are on the auditor's own defer list, and the claims
+around them are restrained accordingly.
+
+The remaining release gate is licensing, which is yours.