pagecheck 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pagecheck-0.1.0/LICENSE +21 -0
- pagecheck-0.1.0/PKG-INFO +67 -0
- pagecheck-0.1.0/README.md +43 -0
- pagecheck-0.1.0/pyproject.toml +34 -0
- pagecheck-0.1.0/setup.cfg +4 -0
- pagecheck-0.1.0/src/pagecheck/__init__.py +0 -0
- pagecheck-0.1.0/src/pagecheck/app.py +106 -0
- pagecheck-0.1.0/src/pagecheck/cli.py +120 -0
- pagecheck-0.1.0/src/pagecheck/engine.py +835 -0
- pagecheck-0.1.0/src/pagecheck.egg-info/PKG-INFO +67 -0
- pagecheck-0.1.0/src/pagecheck.egg-info/SOURCES.txt +13 -0
- pagecheck-0.1.0/src/pagecheck.egg-info/dependency_links.txt +1 -0
- pagecheck-0.1.0/src/pagecheck.egg-info/entry_points.txt +2 -0
- pagecheck-0.1.0/src/pagecheck.egg-info/requires.txt +6 -0
- pagecheck-0.1.0/src/pagecheck.egg-info/top_level.txt +1 -0
pagecheck-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Anaum Pandit
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
pagecheck-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pagecheck
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Pre-launch verdict for a landing page: will it actually capture leads and attribution, or is it silently losing them?
|
|
5
|
+
Author-email: Anaum Pandit <anaump7@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/panaum/pagecheck
|
|
8
|
+
Project-URL: Issues, https://github.com/panaum/pagecheck/issues
|
|
9
|
+
Keywords: qa,testing,playwright,landing-page,attribution,lead-capture
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
14
|
+
Classifier: Topic :: Software Development :: Testing
|
|
15
|
+
Requires-Python: >=3.10
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
License-File: LICENSE
|
|
18
|
+
Requires-Dist: playwright>=1.40
|
|
19
|
+
Provides-Extra: web
|
|
20
|
+
Requires-Dist: fastapi>=0.110; extra == "web"
|
|
21
|
+
Requires-Dist: uvicorn>=0.27; extra == "web"
|
|
22
|
+
Requires-Dist: pydantic>=2.0; extra == "web"
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
# pagecheck
|
|
26
|
+
|
|
27
|
+
A landing page can return HTTP 200 on every request and still be losing every
|
|
28
|
+
lead. The form posts to an endpoint that no longer exists. The attribution
|
|
29
|
+
parameters are stripped by a redirect before anything records them. The submit
|
|
30
|
+
button has no destination at all.
|
|
31
|
+
|
|
32
|
+
`pagecheck` gives a single page a pre-launch verdict: will it actually capture
|
|
33
|
+
leads and attribution, or is it silently losing them?
|
|
34
|
+
|
|
35
|
+
## Install
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
pip install pagecheck
|
|
39
|
+
playwright install chromium
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Use
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
pagecheck https://example.com/landing-page
|
|
46
|
+
pagecheck --file urls.txt --json
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Exit code is 0 when nothing failed, 1 when at least one check failed.
|
|
50
|
+
|
|
51
|
+
## What it does
|
|
52
|
+
|
|
53
|
+
Loads the page with test attribution parameters attached, reads the forms back,
|
|
54
|
+
and reports on lead capture, attribution survival, tracking presence and
|
|
55
|
+
destination integrity.
|
|
56
|
+
|
|
57
|
+
## What it does not do
|
|
58
|
+
|
|
59
|
+
It is read-only. It never submits a form, never clicks anything that would
|
|
60
|
+
create a record, and never writes to the page it is checking.
|
|
61
|
+
|
|
62
|
+
A check it cannot prove is reported as a warning, not a failure. For a tool
|
|
63
|
+
whose output goes to a client, a false alarm costs more than a soft warning.
|
|
64
|
+
|
|
65
|
+
## Licence
|
|
66
|
+
|
|
67
|
+
MIT
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# pagecheck
|
|
2
|
+
|
|
3
|
+
A landing page can return HTTP 200 on every request and still be losing every
|
|
4
|
+
lead. The form posts to an endpoint that no longer exists. The attribution
|
|
5
|
+
parameters are stripped by a redirect before anything records them. The submit
|
|
6
|
+
button has no destination at all.
|
|
7
|
+
|
|
8
|
+
`pagecheck` gives a single page a pre-launch verdict: will it actually capture
|
|
9
|
+
leads and attribution, or is it silently losing them?
|
|
10
|
+
|
|
11
|
+
## Install
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
pip install pagecheck
|
|
15
|
+
playwright install chromium
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Use
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
pagecheck https://example.com/landing-page
|
|
22
|
+
pagecheck --file urls.txt --json
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Exit code is 0 when nothing failed, 1 when at least one check failed.
|
|
26
|
+
|
|
27
|
+
## What it does
|
|
28
|
+
|
|
29
|
+
Loads the page with test attribution parameters attached, reads the forms back,
|
|
30
|
+
and reports on lead capture, attribution survival, tracking presence and
|
|
31
|
+
destination integrity.
|
|
32
|
+
|
|
33
|
+
## What it does not do
|
|
34
|
+
|
|
35
|
+
It is read-only. It never submits a form, never clicks anything that would
|
|
36
|
+
create a record, and never writes to the page it is checking.
|
|
37
|
+
|
|
38
|
+
A check it cannot prove is reported as a warning, not a failure. For a tool
|
|
39
|
+
whose output goes to a client, a false alarm costs more than a soft warning.
|
|
40
|
+
|
|
41
|
+
## Licence
|
|
42
|
+
|
|
43
|
+
MIT
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
[project]
|
|
2
|
+
name = "pagecheck"
|
|
3
|
+
version = "0.1.0"
|
|
4
|
+
description = "Pre-launch verdict for a landing page: will it actually capture leads and attribution, or is it silently losing them?"
|
|
5
|
+
readme = "README.md"
|
|
6
|
+
requires-python = ">=3.10"
|
|
7
|
+
license = "MIT"
|
|
8
|
+
authors = [{ name = "Anaum Pandit", email = "anaump7@gmail.com" }]
|
|
9
|
+
keywords = ["qa", "testing", "playwright", "landing-page", "attribution", "lead-capture"]
|
|
10
|
+
classifiers = [
|
|
11
|
+
"Development Status :: 4 - Beta",
|
|
12
|
+
"Intended Audience :: Developers",
|
|
13
|
+
"Programming Language :: Python :: 3",
|
|
14
|
+
"Topic :: Software Development :: Quality Assurance",
|
|
15
|
+
"Topic :: Software Development :: Testing",
|
|
16
|
+
]
|
|
17
|
+
dependencies = ["playwright>=1.40"]
|
|
18
|
+
|
|
19
|
+
[project.optional-dependencies]
|
|
20
|
+
web = ["fastapi>=0.110", "uvicorn>=0.27", "pydantic>=2.0"]
|
|
21
|
+
|
|
22
|
+
[project.scripts]
|
|
23
|
+
pagecheck = "pagecheck.cli:run"
|
|
24
|
+
|
|
25
|
+
[project.urls]
|
|
26
|
+
Homepage = "https://github.com/panaum/pagecheck"
|
|
27
|
+
Issues = "https://github.com/panaum/pagecheck/issues"
|
|
28
|
+
|
|
29
|
+
[build-system]
|
|
30
|
+
requires = ["setuptools>=68", "wheel"]
|
|
31
|
+
build-backend = "setuptools.build_meta"
|
|
32
|
+
|
|
33
|
+
[tool.setuptools.packages.find]
|
|
34
|
+
where = ["src"]
|
|
File without changes
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
"""Pagecheck — web app.
|
|
2
|
+
|
|
3
|
+
Serves the single-page UI at / and runs checks. Playwright's sync API cannot
|
|
4
|
+
run inside the event loop, so every check runs in a worker thread; progress is
|
|
5
|
+
pushed onto a queue and streamed to the browser as it happens.
|
|
6
|
+
|
|
7
|
+
The engine is engine.check_page(). This file adds no checking logic.
|
|
8
|
+
"""
|
|
9
|
+
|
|
10
|
+
from __future__ import annotations
|
|
11
|
+
|
|
12
|
+
import asyncio
|
|
13
|
+
import json
|
|
14
|
+
import queue
|
|
15
|
+
import threading
|
|
16
|
+
from pathlib import Path
|
|
17
|
+
|
|
18
|
+
from fastapi import FastAPI, Query
|
|
19
|
+
from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
|
|
20
|
+
from pydantic import BaseModel
|
|
21
|
+
|
|
22
|
+
from .engine import check_page
|
|
23
|
+
|
|
24
|
+
app = FastAPI(title="Pagecheck", docs_url=None, redoc_url=None)
|
|
25
|
+
HERE = Path(__file__).parent
|
|
26
|
+
MAX_URLS = 25
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
class CheckRequest(BaseModel):
|
|
30
|
+
urls: list[str] = []
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
def _clean(urls: list[str]) -> list[str]:
|
|
34
|
+
out: list[str] = []
|
|
35
|
+
for u in urls:
|
|
36
|
+
for part in str(u).splitlines():
|
|
37
|
+
part = part.strip()
|
|
38
|
+
if part and not part.startswith("#") and part not in out:
|
|
39
|
+
out.append(part)
|
|
40
|
+
return out[:MAX_URLS]
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
@app.get("/")
|
|
44
|
+
def index() -> FileResponse:
|
|
45
|
+
return FileResponse(HERE / "index.html")
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
@app.post("/check")
|
|
49
|
+
async def check(req: CheckRequest) -> JSONResponse:
|
|
50
|
+
"""Blocking JSON check — one or many URLs. The streaming route below is
|
|
51
|
+
what the UI uses; this is the plain contract for anything else."""
|
|
52
|
+
urls = _clean(req.urls)
|
|
53
|
+
if not urls:
|
|
54
|
+
return JSONResponse({"error": "give at least one url"}, status_code=400)
|
|
55
|
+
results = [await asyncio.to_thread(check_page, u) for u in urls]
|
|
56
|
+
return JSONResponse({"results": results})
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
@app.get("/check/stream")
|
|
60
|
+
async def check_stream(u: list[str] = Query(default=[])) -> StreamingResponse:
|
|
61
|
+
"""Server-sent events: a `progress` event per step, a `result` event per
|
|
62
|
+
URL, then `done`. The UI never shows a spinner with nothing behind it."""
|
|
63
|
+
urls = _clean(list(u or []))
|
|
64
|
+
|
|
65
|
+
async def events():
|
|
66
|
+
if not urls:
|
|
67
|
+
yield _sse("error", {"message": "give at least one url"})
|
|
68
|
+
return
|
|
69
|
+
|
|
70
|
+
for index, url in enumerate(urls):
|
|
71
|
+
q: queue.Queue = queue.Queue()
|
|
72
|
+
|
|
73
|
+
def run(url=url, q=q):
|
|
74
|
+
try:
|
|
75
|
+
result = check_page(url, on_progress=lambda m: q.put(("progress", m)))
|
|
76
|
+
q.put(("result", result))
|
|
77
|
+
except Exception as exc: # noqa: BLE001 — a bad page is a result
|
|
78
|
+
q.put(("result", {"url": url, "outcome": "load_failed",
|
|
79
|
+
"error": f"{type(exc).__name__}: {exc}",
|
|
80
|
+
"checks": [], "failed": True,
|
|
81
|
+
"counts": {"PASS": 0, "FAIL": 1, "WARN": 0, "INFO": 0}}))
|
|
82
|
+
|
|
83
|
+
threading.Thread(target=run, daemon=True).start()
|
|
84
|
+
|
|
85
|
+
while True:
|
|
86
|
+
try:
|
|
87
|
+
kind, payload = await asyncio.to_thread(q.get, True, 1.0)
|
|
88
|
+
except queue.Empty:
|
|
89
|
+
yield ": keep-alive\n\n" # proxies drop a silent stream
|
|
90
|
+
continue
|
|
91
|
+
if kind == "progress":
|
|
92
|
+
yield _sse("progress", {"url": url, "index": index,
|
|
93
|
+
"total": len(urls), "message": payload})
|
|
94
|
+
else:
|
|
95
|
+
yield _sse("result", {"index": index, "total": len(urls),
|
|
96
|
+
"report": payload})
|
|
97
|
+
break
|
|
98
|
+
yield _sse("done", {"total": len(urls)})
|
|
99
|
+
|
|
100
|
+
return StreamingResponse(events(), media_type="text/event-stream",
|
|
101
|
+
headers={"Cache-Control": "no-cache",
|
|
102
|
+
"X-Accel-Buffering": "no"})
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
def _sse(event: str, data: dict) -> str:
|
|
106
|
+
return f"event: {event}\ndata: {json.dumps(data)}\n\n"
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Pagecheck — command line.
|
|
3
|
+
|
|
4
|
+
pagecheck https://example.com/lp
|
|
5
|
+
pagecheck --file urls.txt --json
|
|
6
|
+
pagecheck https://a.test https://b.test
|
|
7
|
+
|
|
8
|
+
Argument parsing and printing only; the checking is engine.check_page().
|
|
9
|
+
Exit codes: 0 = nothing failed, 1 = at least one FAIL, 2 = usage problem.
|
|
10
|
+
"""
|
|
11
|
+
|
|
12
|
+
from __future__ import annotations
|
|
13
|
+
|
|
14
|
+
import argparse
|
|
15
|
+
import json
|
|
16
|
+
import sys
|
|
17
|
+
from pathlib import Path
|
|
18
|
+
|
|
19
|
+
from .engine import check_page, DEFAULT_TIMEOUT_MS, LATE_DELAY_S # noqa: E402
|
|
20
|
+
|
|
21
|
+
MARK = {"PASS": "PASS", "FAIL": "FAIL", "WARN": "WARN", "INFO": "····"}
|
|
22
|
+
OUTCOME = {"load_failed": "PAGE DID NOT LOAD", "timeout": "TIMED OUT",
|
|
23
|
+
"no_form": "NO FORM FOUND"}
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
def render(rep: dict) -> None:
|
|
27
|
+
print()
|
|
28
|
+
print("─" * 78)
|
|
29
|
+
print(rep["url"])
|
|
30
|
+
if rep.get("platform") and rep["platform"] != "unknown":
|
|
31
|
+
note = rep.get("platform_note") or ""
|
|
32
|
+
print(f"{rep['platform']}{f' · {note}' if note else ''}")
|
|
33
|
+
if rep.get("outcome") not in ("ok", None):
|
|
34
|
+
line = OUTCOME.get(rep["outcome"], rep["outcome"].upper())
|
|
35
|
+
print(f" {line}" + (f" — {rep['error']}" if rep.get("error") else ""))
|
|
36
|
+
if rep["outcome"] != "no_form":
|
|
37
|
+
return
|
|
38
|
+
print("─" * 78)
|
|
39
|
+
for c in rep.get("checks", []):
|
|
40
|
+
print(f" {MARK.get(c['status'], c['status']):<4} {c['name']:<22} {c['detail']}")
|
|
41
|
+
for ev in c.get("evidence", [])[:6]:
|
|
42
|
+
print(f" {ev}")
|
|
43
|
+
|
|
44
|
+
diff = (rep.get("consent_diff") or {}).get("fields_differ") or {}
|
|
45
|
+
if diff:
|
|
46
|
+
print("\n consent divergence")
|
|
47
|
+
print(f" {'field':<18} {'accepted':<26} ignored")
|
|
48
|
+
for field, v in diff.items():
|
|
49
|
+
print(f" {field:<18} {(v['accepted'] or '(empty)'):<26} {v['ignored'] or '(empty)'}")
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def summarise(reports: list[dict]) -> None:
|
|
53
|
+
bad = [r for r in reports if r.get("failed")]
|
|
54
|
+
print()
|
|
55
|
+
print("─" * 78)
|
|
56
|
+
if not bad:
|
|
57
|
+
print(f" {len(reports)} page(s) checked · nothing failed")
|
|
58
|
+
else:
|
|
59
|
+
print(f" {len(reports)} page(s) checked · {len(bad)} with failures:")
|
|
60
|
+
for r in bad:
|
|
61
|
+
why = OUTCOME.get(r.get("outcome")) or next(
|
|
62
|
+
(c["detail"] for c in r.get("checks", []) if c["status"] == "FAIL"), "failed")
|
|
63
|
+
print(f" {r['url']} — {why}")
|
|
64
|
+
print()
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
def main() -> int:
|
|
68
|
+
ap = argparse.ArgumentParser(
|
|
69
|
+
prog="pagecheck",
|
|
70
|
+
description="Will this landing page actually capture leads and attribution?")
|
|
71
|
+
ap.add_argument("urls", nargs="*", help="page URL(s)")
|
|
72
|
+
ap.add_argument("--file", help="file of URLs, one per line (# comments allowed)")
|
|
73
|
+
ap.add_argument("--json", action="store_true", dest="as_json", help="emit JSON")
|
|
74
|
+
ap.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT_MS // 1000,
|
|
75
|
+
help="page load timeout in seconds (default 30)")
|
|
76
|
+
ap.add_argument("--delay", type=float, default=LATE_DELAY_S,
|
|
77
|
+
help="seconds before the second read (default 3)")
|
|
78
|
+
args = ap.parse_args()
|
|
79
|
+
|
|
80
|
+
urls = list(args.urls)
|
|
81
|
+
if args.file:
|
|
82
|
+
try:
|
|
83
|
+
urls += [ln.strip() for ln in Path(args.file).read_text(encoding="utf-8").splitlines()
|
|
84
|
+
if ln.strip() and not ln.strip().startswith("#")]
|
|
85
|
+
except OSError as exc:
|
|
86
|
+
print(f"could not read {args.file}: {exc}", file=sys.stderr)
|
|
87
|
+
return 2
|
|
88
|
+
if not urls:
|
|
89
|
+
ap.print_usage(sys.stderr)
|
|
90
|
+
print("give at least one URL, or --file", file=sys.stderr)
|
|
91
|
+
return 2
|
|
92
|
+
|
|
93
|
+
try:
|
|
94
|
+
import playwright # noqa: F401
|
|
95
|
+
except ImportError:
|
|
96
|
+
print("playwright is not installed. pip install playwright && playwright install chromium",
|
|
97
|
+
file=sys.stderr)
|
|
98
|
+
return 2
|
|
99
|
+
|
|
100
|
+
reports = []
|
|
101
|
+
for url in urls:
|
|
102
|
+
rep = check_page(url, args.timeout * 1000, args.delay)
|
|
103
|
+
reports.append(rep)
|
|
104
|
+
if not args.as_json:
|
|
105
|
+
render(rep)
|
|
106
|
+
|
|
107
|
+
if args.as_json:
|
|
108
|
+
print(json.dumps({"results": reports}, indent=2))
|
|
109
|
+
else:
|
|
110
|
+
summarise(reports)
|
|
111
|
+
return 1 if any(r.get("failed") for r in reports) else 0
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
if __name__ == "__main__":
|
|
115
|
+
sys.exit(main())
|
|
116
|
+
|
|
117
|
+
|
|
118
|
+
def run() -> None:
|
|
119
|
+
"""Console-script entry point: main() returns an exit code, setuptools ignores it."""
|
|
120
|
+
raise SystemExit(main())
|
|
@@ -0,0 +1,835 @@
|
|
|
1
|
+
"""Pagecheck — the check engine.
|
|
2
|
+
|
|
3
|
+
Answers one question about a landing page: will it actually capture leads and
|
|
4
|
+
their attribution, or is it silently losing them?
|
|
5
|
+
|
|
6
|
+
The whole engine lives here. The web app and the CLI are both thin callers of
|
|
7
|
+
`check_page()` — one implementation, never two.
|
|
8
|
+
|
|
9
|
+
Read-only. It never submits a form.
|
|
10
|
+
"""
|
|
11
|
+
|
|
12
|
+
from __future__ import annotations
|
|
13
|
+
|
|
14
|
+
import re
|
|
15
|
+
from dataclasses import dataclass, field as dc_field, asdict
|
|
16
|
+
from typing import Any, Callable, Iterable
|
|
17
|
+
from urllib.parse import urlencode, urlparse, urlunparse, parse_qsl
|
|
18
|
+
|
|
19
|
+
# ── Probe parameters ────────────────────────────────────────────────────────
|
|
20
|
+
TEST_PARAMS: dict[str, str] = {
|
|
21
|
+
"utm_source": "qa", "utm_medium": "qa", "utm_campaign": "qa",
|
|
22
|
+
"utm_term": "qa", "utm_content": "qa",
|
|
23
|
+
"gclid": "QA_TEST_GCLID", "fbclid": "QA_TEST_FBCLID",
|
|
24
|
+
}
|
|
25
|
+
LATE_DELAY_S = 3.0
|
|
26
|
+
DEFAULT_TIMEOUT_MS = 30_000
|
|
27
|
+
NETWORK_IDLE_MS = 8_000
|
|
28
|
+
UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
|
29
|
+
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")
|
|
30
|
+
|
|
31
|
+
PASS, FAIL, WARN, INFO = "PASS", "FAIL", "WARN", "INFO"
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
# ── Platform fingerprints ───────────────────────────────────────────────────
|
|
35
|
+
# Infrastructure hosts only, never a brand word. A page that SAYS "we build
|
|
36
|
+
# gohighlevel funnels" is not a GoHighLevel page, and the wrong platform means
|
|
37
|
+
# every quirk below it is interpreted wrongly.
|
|
38
|
+
PLATFORMS: list[tuple[str, tuple[str, ...]]] = [
|
|
39
|
+
("Unbounce", ("lp-pom", "ub-emb", "unbouncepages.com", "ubembed.com")),
|
|
40
|
+
("ClickFunnels", ("clickfunnels", "containerwrapper", "data-page-element")),
|
|
41
|
+
("GoHighLevel", ("leadconnectorhq", "msgsndr", "app.gohighlevel.com")),
|
|
42
|
+
("Kajabi", ("kajabi-cdn", "kajabi-theme", "kjb-")),
|
|
43
|
+
("HubSpot", ("hs-scripts", "hsforms.net", "hubspotusercontent")),
|
|
44
|
+
("Webflow", ("data-wf-page", "data-wf-site", "website-files.com")),
|
|
45
|
+
("Squarespace", ("squarespace-cdn", "static1.squarespace")),
|
|
46
|
+
("Shopify", ("cdn.shopify", "myshopify.com", "/cdn/shop/")),
|
|
47
|
+
("WordPress", ("/wp-content/", "/wp-includes/", "wp-json")),
|
|
48
|
+
]
|
|
49
|
+
WP_PLUGINS: list[tuple[str, tuple[str, ...]]] = [
|
|
50
|
+
("Gravity Forms", ("gform_", "gravityforms")),
|
|
51
|
+
("WPForms", ("wpforms-", "wpforms_")),
|
|
52
|
+
("Elementor Forms", ("elementor-field", "elementor-form")),
|
|
53
|
+
("Contact Form 7", ("wpcf7",)),
|
|
54
|
+
("Ninja Forms", ("nf-form", "ninja-forms")),
|
|
55
|
+
("Formidable", ("frm_form", "formidable")),
|
|
56
|
+
]
|
|
57
|
+
|
|
58
|
+
# ── Attribution field naming ────────────────────────────────────────────────
|
|
59
|
+
CANONICAL = ["utm_source", "utm_medium", "utm_campaign", "utm_term",
|
|
60
|
+
"utm_content", "gclid", "fbclid"]
|
|
61
|
+
ALIASES: dict[str, tuple[str, ...]] = {
|
|
62
|
+
"utm_source": ("utmsource", "source", "trafficsource", "ubutmsource"),
|
|
63
|
+
"utm_medium": ("utmmedium", "medium", "ubutmmedium"),
|
|
64
|
+
"utm_campaign": ("utmcampaign", "campaign", "campaignname", "ubutmcampaign"),
|
|
65
|
+
"utm_term": ("utmterm", "term", "keyword", "ubutmterm"),
|
|
66
|
+
"utm_content": ("utmcontent", "content", "adcontent", "ubutmcontent"),
|
|
67
|
+
"gclid": ("googleclickid", "gclidfield"),
|
|
68
|
+
"fbclid": ("facebookclickid", "fbc"),
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
# Platforms that attribute a lead by cookie, with no hidden field needed.
|
|
72
|
+
COOKIE_ATTRIBUTION: dict[str, tuple[str, ...]] = {
|
|
73
|
+
"HubSpot": ("hubspotutk", "__hstc"),
|
|
74
|
+
"Google Ads": ("_gcl_aw", "gcl_aw_p", "_gac_"),
|
|
75
|
+
"Meta": ("_fbc",),
|
|
76
|
+
"Google Analytics": ("_ga",),
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
# ── Non-production signals ──────────────────────────────────────────────────
|
|
80
|
+
STAGING_MARKERS = ("staging", "-stage", "test", "dev.", ".dev", "localhost",
|
|
81
|
+
"127.0.0.1", "sandbox", "preprod", "uat")
|
|
82
|
+
# A builder's own collector means the lead may never reach the client's stack.
|
|
83
|
+
DEFAULT_COLLECTORS = ("formspree.io", "unbouncepages.com", "getform.io",
|
|
84
|
+
"formsubmit.co", "usebasin.com", "mailerlite.com",
|
|
85
|
+
"eocampaign1.com", "emailoctopus.com", "list-manage.com",
|
|
86
|
+
"activehosted.com", "hsforms.com", "jotform.com",
|
|
87
|
+
"typeform.com", "tally.so", "convertkit.com")
|
|
88
|
+
PLACEHOLDERS = ("lorem ipsum", "headline here", "your text here", "your headline",
|
|
89
|
+
"insert text", "todo", "tbd", "coming soon…", "sample text",
|
|
90
|
+
"example@example.com", "test@test.com", "john@doe.com")
|
|
91
|
+
CAPTCHAS: list[tuple[str, tuple[str, ...], str]] = [
|
|
92
|
+
("reCAPTCHA", ("google.com/recaptcha", "gstatic.com/recaptcha", "g-recaptcha"), "data-sitekey"),
|
|
93
|
+
("hCaptcha", ("hcaptcha.com", "h-captcha"), "data-sitekey"),
|
|
94
|
+
("Turnstile", ("challenges.cloudflare.com", "cf-turnstile"), "data-sitekey"),
|
|
95
|
+
]
|
|
96
|
+
|
|
97
|
+
# Consent banners: the accept control, in rough order of specificity.
|
|
98
|
+
CONSENT_ACCEPT_SELECTORS = [
|
|
99
|
+
"#onetrust-accept-btn-handler",
|
|
100
|
+
"#CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll",
|
|
101
|
+
"#CybotCookiebotDialogBodyButtonAccept",
|
|
102
|
+
".cc-allow", ".cky-btn-accept", "#hs-eu-confirmation-button",
|
|
103
|
+
"[data-cky-tag='accept-button']", "#truste-consent-button",
|
|
104
|
+
"button[aria-label*='Accept' i]", "button[title*='Accept' i]",
|
|
105
|
+
]
|
|
106
|
+
CONSENT_ACCEPT_TEXT = ["accept all", "allow all", "accept cookies", "i agree",
|
|
107
|
+
"agree and close", "accept", "allow", "got it", "ok"]
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
def _norm(name: str) -> str:
|
|
111
|
+
return re.sub(r"[^a-z0-9]", "", (name or "").lower())
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
def canonical_for(field_name: str) -> str | None:
|
|
115
|
+
"""Which attribution parameter a field is meant to hold, if any."""
|
|
116
|
+
n = _norm(field_name)
|
|
117
|
+
if not n:
|
|
118
|
+
return None
|
|
119
|
+
for canon in CANONICAL:
|
|
120
|
+
if n == _norm(canon):
|
|
121
|
+
return canon
|
|
122
|
+
if any(n == _norm(a) for a in ALIASES.get(canon, ())):
|
|
123
|
+
return canon
|
|
124
|
+
for canon in CANONICAL:
|
|
125
|
+
if _norm(canon) in n:
|
|
126
|
+
return canon
|
|
127
|
+
return None
|
|
128
|
+
|
|
129
|
+
|
|
130
|
+
def build_test_url(url: str) -> str:
|
|
131
|
+
if "://" not in url:
|
|
132
|
+
url = "https://" + url
|
|
133
|
+
p = urlparse(url)
|
|
134
|
+
q = [(k, v) for k, v in parse_qsl(p.query, keep_blank_values=True) if k not in TEST_PARAMS]
|
|
135
|
+
q.extend(TEST_PARAMS.items())
|
|
136
|
+
return urlunparse(p._replace(query=urlencode(q)))
|
|
137
|
+
|
|
138
|
+
|
|
139
|
+
def detect_platform(html: str) -> tuple[str, str]:
|
|
140
|
+
low = html.lower()
|
|
141
|
+
found = [n for n, markers in PLATFORMS if any(m in low for m in markers)]
|
|
142
|
+
if not found:
|
|
143
|
+
return "unknown", ""
|
|
144
|
+
primary, note = found[0], ""
|
|
145
|
+
if "WordPress" in found:
|
|
146
|
+
primary = "WordPress"
|
|
147
|
+
plugins = [p for p, m in WP_PLUGINS if any(x in low for x in m)]
|
|
148
|
+
note = ("form plugin: " + ", ".join(plugins)) if plugins else \
|
|
149
|
+
"no form plugin recognised — hidden field naming is unknown"
|
|
150
|
+
others = [f for f in found if f != primary]
|
|
151
|
+
if others:
|
|
152
|
+
note = (note + "; " if note else "") + "also present: " + ", ".join(others)
|
|
153
|
+
return primary, note
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
def _is_placeholder_id(tag_id: str) -> bool:
|
|
157
|
+
body = tag_id.split("-", 1)[-1]
|
|
158
|
+
return len(set(body)) <= 1 or "XXXX" in body.upper()
|
|
159
|
+
|
|
160
|
+
|
|
161
|
+
def find_tracking(html: str, requests: list[str] | None = None) -> dict[str, list[str]]:
|
|
162
|
+
"""Every tracking ID on the page, read from the source AND the network.
|
|
163
|
+
|
|
164
|
+
Source alone is not enough: a tag injected by GTM or a dynamic loader never
|
|
165
|
+
appears in the HTML, so an HTML-only scan reported "no analytics" on a page
|
|
166
|
+
that was demonstrably loading the Meta pixel. Duplicates matter too — two
|
|
167
|
+
GA4 ids double-count conversions and halve reported cost per lead."""
|
|
168
|
+
wire = " ".join(requests or [])
|
|
169
|
+
both = html + " " + wire
|
|
170
|
+
|
|
171
|
+
def clean(xs: Iterable[str]) -> list[str]:
|
|
172
|
+
return sorted({x for x in xs if not _is_placeholder_id(x)})
|
|
173
|
+
|
|
174
|
+
pixels = set(re.findall(r"fbq\(\s*['\"]init['\"]\s*,\s*['\"](\d{6,})['\"]", html))
|
|
175
|
+
pixels |= set(re.findall(r"facebook\.com/tr[/?][^\s\"']*?\bid=(\d{6,})", both))
|
|
176
|
+
# The pixel SCRIPT loading without an id is its own finding: the tag is on
|
|
177
|
+
# the page but may never fire.
|
|
178
|
+
pixel_script = "connect.facebook.net" in both and "fbevents" in both
|
|
179
|
+
return {
|
|
180
|
+
"gtm": clean(re.findall(r"GTM-[A-Z0-9]{4,}", both)),
|
|
181
|
+
"ga4": clean(re.findall(r"\bG-[A-Z0-9]{6,}\b", both)),
|
|
182
|
+
"meta_pixel": sorted(pixels),
|
|
183
|
+
"meta_script_only": [] if pixels or not pixel_script else ["connect.facebook.net"],
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
|
|
187
|
+
def detect_captcha(html: str) -> dict[str, Any] | None:
|
|
188
|
+
low = html.lower()
|
|
189
|
+
for name, markers, key_attr in CAPTCHAS:
|
|
190
|
+
if any(m in low for m in markers):
|
|
191
|
+
has_key = bool(
|
|
192
|
+
re.search(key_attr + r"\s*=\s*['\"][^'\"]{8,}", html, re.I)
|
|
193
|
+
# v3/invisible: the key is in the script URL or the execute call.
|
|
194
|
+
or re.search(r"recaptcha/api\.js\?[^'\"]*render=([\w-]{20,})", html, re.I)
|
|
195
|
+
or re.search(r"grecaptcha\.(?:execute|render)\s*\(\s*['\"]([\w-]{20,})", html, re.I)
|
|
196
|
+
or re.search(r"turnstile\.render\s*\([^)]*sitekey", html, re.I))
|
|
197
|
+
return {"kind": name, "site_key_present": has_key}
|
|
198
|
+
return None
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
def find_placeholders(text: str) -> list[str]:
|
|
202
|
+
low = text.lower()
|
|
203
|
+
hits = [p for p in PLACEHOLDERS if p in low]
|
|
204
|
+
if re.search(r"\b555[-.\s]?\d{3}[-.\s]?\d{4}\b", text):
|
|
205
|
+
hits.append("555 phone number")
|
|
206
|
+
return sorted(set(hits))
|
|
207
|
+
|
|
208
|
+
|
|
209
|
+
def classify_action(action: str | None, page_url: str) -> tuple[str, str] | None:
|
|
210
|
+
"""(severity, reason) when a form's destination looks wrong.
|
|
211
|
+
|
|
212
|
+
The staging test reads the destination HOST, not the whole URL: a live page
|
|
213
|
+
posting to staging.acme.com is exactly the failure worth catching, and a
|
|
214
|
+
naive "is the site's domain in the string" guard silently excused it
|
|
215
|
+
(staging.acme.com contains acme.com). Path segments are excluded too, or
|
|
216
|
+
every /test-drive/ form would be flagged."""
|
|
217
|
+
if not action:
|
|
218
|
+
return None
|
|
219
|
+
a = action.lower()
|
|
220
|
+
host = (urlparse(a if "://" in a else f"https://{a}").hostname or "")
|
|
221
|
+
page_host = (urlparse(page_url).hostname or "").lower()
|
|
222
|
+
|
|
223
|
+
if host and host != page_host:
|
|
224
|
+
labels = host.split(".")
|
|
225
|
+
if any(m.strip(".") in labels or host.startswith(m) for m in STAGING_MARKERS):
|
|
226
|
+
return (FAIL, f"posts to {host}, which looks like a staging or test endpoint")
|
|
227
|
+
for c in DEFAULT_COLLECTORS:
|
|
228
|
+
if c in host:
|
|
229
|
+
return (WARN, f"posts to {c}, a builder's default collector rather than "
|
|
230
|
+
"the client's own system")
|
|
231
|
+
return None
|
|
232
|
+
|
|
233
|
+
|
|
234
|
+
def detect_cookie_attribution(cookies: list[dict[str, Any]]) -> list[str]:
|
|
235
|
+
names = [str(c.get("name", "")).lower() for c in cookies or []]
|
|
236
|
+
return [p for p, markers in COOKIE_ATTRIBUTION.items()
|
|
237
|
+
if any(any(m in n for n in names) for m in markers)]
|
|
238
|
+
|
|
239
|
+
|
|
240
|
+
# ── Browser probes ──────────────────────────────────────────────────────────
|
|
241
|
+
COLLECT_JS = r"""
|
|
242
|
+
() => {
|
|
243
|
+
const readValue = (el) => { try { return el.value == null ? "" : String(el.value); } catch (e) { return ""; } };
|
|
244
|
+
const isHidden = (el) => {
|
|
245
|
+
try {
|
|
246
|
+
if ((el.type || "").toLowerCase() === "hidden") return true;
|
|
247
|
+
const s = window.getComputedStyle(el);
|
|
248
|
+
if (s.display === "none" || s.visibility === "hidden" || s.opacity === "0") return true;
|
|
249
|
+
const r = el.getBoundingClientRect();
|
|
250
|
+
return r.width === 0 && r.height === 0;
|
|
251
|
+
} catch (e) { return false; }
|
|
252
|
+
};
|
|
253
|
+
const describe = (el) => ({
|
|
254
|
+
name: el.name || el.id || "", type: ((el.type || el.tagName || "").toLowerCase()),
|
|
255
|
+
hidden: isHidden(el), value: readValue(el),
|
|
256
|
+
});
|
|
257
|
+
const SEL = "input,select,textarea";
|
|
258
|
+
const forms = Array.prototype.slice.call(document.querySelectorAll("form")).map((f, i) => ({
|
|
259
|
+
index: i, id: f.id || null, action: f.getAttribute("action") || null,
|
|
260
|
+
method: (f.getAttribute("method") || "get").toLowerCase(),
|
|
261
|
+
className: (typeof f.className === "string" ? f.className : "") || "",
|
|
262
|
+
fields: Array.prototype.slice.call(f.querySelectorAll(SEL)).map(describe),
|
|
263
|
+
}));
|
|
264
|
+
const orphans = Array.prototype.slice.call(document.querySelectorAll(SEL))
|
|
265
|
+
.filter((el) => !el.closest("form")).map(describe);
|
|
266
|
+
return { frameUrl: location.href, forms: forms, orphans: orphans, text: document.body ? document.body.innerText : "" };
|
|
267
|
+
}
|
|
268
|
+
"""
|
|
269
|
+
|
|
270
|
+
STORAGE_JS = r"""
|
|
271
|
+
() => {
|
|
272
|
+
const dump = (s) => { const o = {}; try { for (let i=0;i<s.length;i++){const k=s.key(i); o[k]=s.getItem(k);} } catch(e){} return o; };
|
|
273
|
+
let ub = null;
|
|
274
|
+
try { if (window.ub && window.ub.page) ub = { variantId: window.ub.page.variantId || null }; } catch (e) {}
|
|
275
|
+
return { local: dump(window.localStorage), session: dump(window.sessionStorage), ub: ub };
|
|
276
|
+
}
|
|
277
|
+
"""
|
|
278
|
+
|
|
279
|
+
ROBOTS_JS = r"""
|
|
280
|
+
() => {
|
|
281
|
+
const m = document.querySelector('meta[name="robots"], meta[name="googlebot"]');
|
|
282
|
+
return m ? (m.getAttribute("content") || "") : "";
|
|
283
|
+
}
|
|
284
|
+
"""
|
|
285
|
+
|
|
286
|
+
|
|
287
|
+
# ── Result model ────────────────────────────────────────────────────────────
|
|
288
|
+
@dataclass
|
|
289
|
+
class Check:
|
|
290
|
+
id: str
|
|
291
|
+
name: str
|
|
292
|
+
status: str
|
|
293
|
+
detail: str
|
|
294
|
+
# Evidence is the product: field values, the console error, the dead
|
|
295
|
+
# request. A "field empty" line alone is not actionable.
|
|
296
|
+
evidence: list[str] = dc_field(default_factory=list)
|
|
297
|
+
|
|
298
|
+
|
|
299
|
+
@dataclass
|
|
300
|
+
class Report:
|
|
301
|
+
url: str
|
|
302
|
+
tested_url: str = ""
|
|
303
|
+
outcome: str = "ok" # ok | load_failed | timeout | no_form
|
|
304
|
+
error: str = ""
|
|
305
|
+
platform: str = "unknown"
|
|
306
|
+
platform_note: str = ""
|
|
307
|
+
checks: list[Check] = dc_field(default_factory=list)
|
|
308
|
+
forms: list[dict[str, Any]] = dc_field(default_factory=list)
|
|
309
|
+
tracking: dict[str, list[str]] = dc_field(default_factory=dict)
|
|
310
|
+
cookie_attribution: list[str] = dc_field(default_factory=list)
|
|
311
|
+
storage_hits: list[str] = dc_field(default_factory=list)
|
|
312
|
+
consent_diff: dict[str, Any] = dc_field(default_factory=dict)
|
|
313
|
+
console_errors: list[str] = dc_field(default_factory=list)
|
|
314
|
+
failed_requests: list[str] = dc_field(default_factory=list)
|
|
315
|
+
|
|
316
|
+
def add(self, cid: str, name: str, status: str, detail: str,
|
|
317
|
+
evidence: list[str] | None = None) -> None:
|
|
318
|
+
self.checks.append(Check(cid, name, status, detail, evidence or []))
|
|
319
|
+
|
|
320
|
+
@property
|
|
321
|
+
def failed(self) -> bool:
|
|
322
|
+
return any(c.status == FAIL for c in self.checks) or self.outcome != "ok"
|
|
323
|
+
|
|
324
|
+
@property
|
|
325
|
+
def counts(self) -> dict[str, int]:
|
|
326
|
+
out = {PASS: 0, FAIL: 0, WARN: 0, INFO: 0}
|
|
327
|
+
for c in self.checks:
|
|
328
|
+
out[c.status] = out.get(c.status, 0) + 1
|
|
329
|
+
return out
|
|
330
|
+
|
|
331
|
+
|
|
332
|
+
# ── One page load ───────────────────────────────────────────────────────────
|
|
333
|
+
@dataclass
|
|
334
|
+
class PassResult:
|
|
335
|
+
"""Everything one load of the page yielded."""
|
|
336
|
+
html: str = ""
|
|
337
|
+
forms: list[dict[str, Any]] = dc_field(default_factory=list)
|
|
338
|
+
forms_late: list[dict[str, Any]] = dc_field(default_factory=list)
|
|
339
|
+
storage: dict[str, Any] = dc_field(default_factory=dict)
|
|
340
|
+
cookies: list[dict[str, Any]] = dc_field(default_factory=list)
|
|
341
|
+
text: str = ""
|
|
342
|
+
robots: str = ""
|
|
343
|
+
console_errors: list[str] = dc_field(default_factory=list)
|
|
344
|
+
failed_requests: list[str] = dc_field(default_factory=list)
|
|
345
|
+
requests: list[str] = dc_field(default_factory=list)
|
|
346
|
+
consent_clicked: bool = False
|
|
347
|
+
error: str = ""
|
|
348
|
+
outcome: str = "ok"
|
|
349
|
+
|
|
350
|
+
|
|
351
|
+
# Frames that never host a lead form. Skipping them is a speedup, but the
|
|
352
|
+
# reason it exists is correctness: an ad or analytics iframe is exactly the
|
|
353
|
+
# kind that stalls, and evaluate() has no timeout to save us.
|
|
354
|
+
SKIP_FRAME_HOSTS = (
|
|
355
|
+
"googletagmanager.com", "google-analytics.com", "doubleclick.net",
|
|
356
|
+
"googlesyndication.com", "google.com/recaptcha", "gstatic.com",
|
|
357
|
+
"facebook.com", "facebook.net", "connect.facebook", "hotjar",
|
|
358
|
+
"player.vimeo.com", "youtube.com/embed", "youtube-nocookie",
|
|
359
|
+
"clarity.ms", "service_worker",
|
|
360
|
+
)
|
|
361
|
+
|
|
362
|
+
|
|
363
|
+
def _skip_frame(frame, is_main: bool) -> bool:
|
|
364
|
+
if is_main:
|
|
365
|
+
return False
|
|
366
|
+
try:
|
|
367
|
+
if frame.is_detached():
|
|
368
|
+
return True
|
|
369
|
+
url = (frame.url or "").strip()
|
|
370
|
+
except Exception:
|
|
371
|
+
return True
|
|
372
|
+
# A frame with no URL has no execution context and never will — calling
|
|
373
|
+
# evaluate() on one blocks FOREVER, which hung whole runs on real pages.
|
|
374
|
+
if not url or url == "about:blank":
|
|
375
|
+
return True
|
|
376
|
+
return any(h in url for h in SKIP_FRAME_HOSTS)
|
|
377
|
+
|
|
378
|
+
|
|
379
|
+
def _collect_frames(page) -> tuple[list[dict[str, Any]], str]:
|
|
380
|
+
"""Every form in every frame. Cross-origin iframes are the whole point —
|
|
381
|
+
a GoHighLevel form lives in one and same-document JS cannot read it, so
|
|
382
|
+
cross-origin frames are traversed; only frames that cannot hold a form
|
|
383
|
+
are skipped."""
|
|
384
|
+
out: list[dict[str, Any]] = []
|
|
385
|
+
text_parts: list[str] = []
|
|
386
|
+
for frame in page.frames:
|
|
387
|
+
is_main = frame is page.main_frame
|
|
388
|
+
if _skip_frame(frame, is_main):
|
|
389
|
+
continue
|
|
390
|
+
try:
|
|
391
|
+
data = frame.evaluate(COLLECT_JS)
|
|
392
|
+
except Exception:
|
|
393
|
+
continue
|
|
394
|
+
where = "main" if is_main else (data.get("frameUrl") or "iframe")
|
|
395
|
+
for form in data.get("forms", []):
|
|
396
|
+
form["frame"], form["in_iframe"] = where, not is_main
|
|
397
|
+
out.append(form)
|
|
398
|
+
if data.get("orphans"):
|
|
399
|
+
out.append({"index": -1, "id": None, "action": None, "method": "",
|
|
400
|
+
"className": "", "fields": data["orphans"], "frame": where,
|
|
401
|
+
"in_iframe": not is_main, "orphan_group": True})
|
|
402
|
+
if data.get("text"):
|
|
403
|
+
text_parts.append(data["text"])
|
|
404
|
+
return out, "\n".join(text_parts)
|
|
405
|
+
|
|
406
|
+
|
|
407
|
+
ACCEPT_JS = r"""
|
|
408
|
+
(labels) => {
|
|
409
|
+
// One evaluation instead of dozens of locator round-trips: on a large DOM
|
|
410
|
+
// that difference is minutes, and it was hanging the run outright.
|
|
411
|
+
const KNOWN = ["#onetrust-accept-btn-handler",
|
|
412
|
+
"#CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll",
|
|
413
|
+
"#CybotCookiebotDialogBodyButtonAccept", ".cc-allow", ".cky-btn-accept",
|
|
414
|
+
"#hs-eu-confirmation-button", "[data-cky-tag='accept-button']",
|
|
415
|
+
"#truste-consent-button"];
|
|
416
|
+
const visible = (el) => {
|
|
417
|
+
const r = el.getBoundingClientRect();
|
|
418
|
+
if (!r.width || !r.height) return false;
|
|
419
|
+
const s = getComputedStyle(el);
|
|
420
|
+
return s.visibility !== "hidden" && s.display !== "none" && s.opacity !== "0";
|
|
421
|
+
};
|
|
422
|
+
for (const sel of KNOWN) {
|
|
423
|
+
const el = document.querySelector(sel);
|
|
424
|
+
if (el && visible(el)) { el.click(); return sel; }
|
|
425
|
+
}
|
|
426
|
+
const clickable = document.querySelectorAll("button,[role=button],a.cc-btn,input[type=button]");
|
|
427
|
+
for (const el of clickable) {
|
|
428
|
+
const t = (el.innerText || el.value || "").trim().toLowerCase();
|
|
429
|
+
if (!t || t.length > 24) continue;
|
|
430
|
+
if (labels.includes(t) && visible(el)) { el.click(); return t; }
|
|
431
|
+
}
|
|
432
|
+
return null;
|
|
433
|
+
}
|
|
434
|
+
"""
|
|
435
|
+
|
|
436
|
+
|
|
437
|
+
def _accept_consent(page, wait_ms: int = 5000) -> bool:
|
|
438
|
+
"""Click the consent banner's accept control. Best-effort and read-only —
|
|
439
|
+
a consent click is not a form submission.
|
|
440
|
+
|
|
441
|
+
Polls, because consent banners are injected by a third-party script and
|
|
442
|
+
commonly render AFTER network idle; a single look reported "no cookie
|
|
443
|
+
banner" on pages that visibly had one, silently disabling the whole
|
|
444
|
+
consent-divergence comparison. Each poll is ONE evaluation — the earlier
|
|
445
|
+
per-label locator queries were slow enough to hang a large page."""
|
|
446
|
+
remaining, step = wait_ms, 700
|
|
447
|
+
while remaining > 0:
|
|
448
|
+
try:
|
|
449
|
+
if page.evaluate(ACCEPT_JS, CONSENT_ACCEPT_TEXT):
|
|
450
|
+
page.wait_for_timeout(700)
|
|
451
|
+
return True
|
|
452
|
+
except Exception:
|
|
453
|
+
pass
|
|
454
|
+
page.wait_for_timeout(step)
|
|
455
|
+
remaining -= step
|
|
456
|
+
return False
|
|
457
|
+
|
|
458
|
+
|
|
459
|
+
def _one_pass(browser, url: str, accept_consent: bool, timeout_ms: int,
|
|
460
|
+
delay_s: float) -> PassResult:
|
|
461
|
+
from playwright.sync_api import Error as PWError, TimeoutError as PWTimeout
|
|
462
|
+
|
|
463
|
+
res = PassResult()
|
|
464
|
+
context = browser.new_context(user_agent=UA, viewport={"width": 1280, "height": 900})
|
|
465
|
+
page = context.new_page()
|
|
466
|
+
|
|
467
|
+
page.on("request", lambda r: res.requests.append(r.url[:300])
|
|
468
|
+
if len(res.requests) < 400 else None)
|
|
469
|
+
page.on("console", lambda m: res.console_errors.append(
|
|
470
|
+
f"{m.text[:180]}"[:200]) if m.type == "error" else None)
|
|
471
|
+
page.on("requestfailed", lambda r: res.failed_requests.append(
|
|
472
|
+
f"{r.url[:160]} ({(r.failure or {}).get('errorText', 'failed') if isinstance(r.failure, dict) else 'failed'})"))
|
|
473
|
+
page.on("response", lambda r: res.failed_requests.append(f"{r.url[:160]} (HTTP {r.status})")
|
|
474
|
+
if r.status >= 400 else None)
|
|
475
|
+
|
|
476
|
+
try:
|
|
477
|
+
try:
|
|
478
|
+
page.goto(url, wait_until="domcontentloaded", timeout=timeout_ms)
|
|
479
|
+
except PWTimeout:
|
|
480
|
+
res.outcome, res.error = "timeout", f"page did not load within {timeout_ms // 1000}s"
|
|
481
|
+
return res
|
|
482
|
+
except PWError as e:
|
|
483
|
+
res.outcome = "load_failed"
|
|
484
|
+
res.error = f"{type(e).__name__}: {str(e).splitlines()[0][:200]}"
|
|
485
|
+
return res
|
|
486
|
+
|
|
487
|
+
try:
|
|
488
|
+
page.wait_for_load_state("networkidle", timeout=NETWORK_IDLE_MS)
|
|
489
|
+
except Exception:
|
|
490
|
+
pass
|
|
491
|
+
|
|
492
|
+
if accept_consent:
|
|
493
|
+
res.consent_clicked = _accept_consent(page)
|
|
494
|
+
try:
|
|
495
|
+
page.wait_for_load_state("networkidle", timeout=4000)
|
|
496
|
+
except Exception:
|
|
497
|
+
pass
|
|
498
|
+
|
|
499
|
+
res.forms, res.text = _collect_frames(page)
|
|
500
|
+
page.wait_for_timeout(int(delay_s * 1000))
|
|
501
|
+
res.forms_late, late_text = _collect_frames(page)
|
|
502
|
+
res.text = res.text or late_text
|
|
503
|
+
|
|
504
|
+
try:
|
|
505
|
+
res.html = page.content()
|
|
506
|
+
except Exception:
|
|
507
|
+
pass
|
|
508
|
+
try:
|
|
509
|
+
res.storage = page.evaluate(STORAGE_JS)
|
|
510
|
+
except Exception:
|
|
511
|
+
res.storage = {"local": {}, "session": {}, "ub": None}
|
|
512
|
+
try:
|
|
513
|
+
res.robots = page.evaluate(ROBOTS_JS) or ""
|
|
514
|
+
except Exception:
|
|
515
|
+
pass
|
|
516
|
+
try:
|
|
517
|
+
res.cookies = context.cookies()
|
|
518
|
+
except Exception:
|
|
519
|
+
pass
|
|
520
|
+
return res
|
|
521
|
+
finally:
|
|
522
|
+
try:
|
|
523
|
+
context.close()
|
|
524
|
+
except Exception:
|
|
525
|
+
pass
|
|
526
|
+
|
|
527
|
+
|
|
528
|
+
# ── Analysis ────────────────────────────────────────────────────────────────
|
|
529
|
+
def attribution_map(forms: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
|
|
530
|
+
found: dict[str, list[dict[str, Any]]] = {}
|
|
531
|
+
for form in forms:
|
|
532
|
+
for f in form.get("fields", []):
|
|
533
|
+
canon = canonical_for(f.get("name", ""))
|
|
534
|
+
if canon:
|
|
535
|
+
found.setdefault(canon, []).append({**f, "frame": form.get("frame", "main")})
|
|
536
|
+
return found
|
|
537
|
+
|
|
538
|
+
|
|
539
|
+
def search_storage(storage: dict[str, Any], cookies: list[dict[str, Any]]) -> list[str]:
|
|
540
|
+
hits, sentinels = [], set(TEST_PARAMS.values())
|
|
541
|
+
for scope in ("local", "session"):
|
|
542
|
+
for k, v in (storage.get(scope) or {}).items():
|
|
543
|
+
if any(s in f"{v}" for s in sentinels):
|
|
544
|
+
hits.append(f"{scope}Storage[{k}]")
|
|
545
|
+
for c in cookies or []:
|
|
546
|
+
if any(s in str(c.get("value", "")) for s in sentinels):
|
|
547
|
+
hits.append(f"cookie[{c.get('name')}]")
|
|
548
|
+
return sorted(set(hits))
|
|
549
|
+
|
|
550
|
+
|
|
551
|
+
def _related_evidence(res: PassResult, *needles: str) -> list[str]:
|
|
552
|
+
"""Console errors and dead requests that mention any needle. This is what
|
|
553
|
+
turns "field empty" into "field empty, and utm-capture.js 404s"."""
|
|
554
|
+
out: list[str] = []
|
|
555
|
+
for item in res.console_errors + res.failed_requests:
|
|
556
|
+
low = item.lower()
|
|
557
|
+
if any(n and n.lower() in low for n in needles):
|
|
558
|
+
out.append(item)
|
|
559
|
+
return out[:5]
|
|
560
|
+
|
|
561
|
+
|
|
562
|
+
def check_ssl(url: str, page_loaded: bool) -> tuple[str, str]:
|
|
563
|
+
"""Chromium refuses to load a page whose certificate does not verify, so a
|
|
564
|
+
successful https load IS the check. An independent Python check only tests
|
|
565
|
+
whichever CA bundle this machine happens to have, which produced a false
|
|
566
|
+
failure for two sites with perfectly valid certificates."""
|
|
567
|
+
if urlparse(url).scheme != "https":
|
|
568
|
+
return FAIL, "the page is served over plain http"
|
|
569
|
+
if page_loaded:
|
|
570
|
+
return PASS, "served over https with a certificate the browser accepts"
|
|
571
|
+
return WARN, "could not confirm the certificate — the page did not load"
|
|
572
|
+
|
|
573
|
+
|
|
574
|
+
def _evaluate(rep: Report, main: PassResult, other: PassResult | None) -> None:
|
|
575
|
+
"""Turn the passes into the report. `main` is the consent-accepted load."""
|
|
576
|
+
forms_now, forms_late = main.forms, main.forms_late or main.forms
|
|
577
|
+
|
|
578
|
+
# 1 — platform
|
|
579
|
+
rep.platform, rep.platform_note = detect_platform(main.html)
|
|
580
|
+
rep.add("platform", "Platform", INFO,
|
|
581
|
+
rep.platform + (f" ({rep.platform_note})" if rep.platform_note else ""))
|
|
582
|
+
if rep.platform == "Unbounce":
|
|
583
|
+
variant = ((main.storage or {}).get("ub") or {}).get("variantId")
|
|
584
|
+
rep.add("variant", "Unbounce variant", WARN,
|
|
585
|
+
(f"served variant {variant} — " if variant else "") +
|
|
586
|
+
"A/B variants are served per visit; only the variant served now was checked")
|
|
587
|
+
|
|
588
|
+
# 2 — forms
|
|
589
|
+
real = [f for f in forms_late if not f.get("orphan_group")]
|
|
590
|
+
iframed = [f for f in forms_late if f.get("in_iframe")]
|
|
591
|
+
orphan_fields = sum(len(f["fields"]) for f in forms_late if f.get("orphan_group"))
|
|
592
|
+
# A page with no form is a real finding, not a dead end: it may still be
|
|
593
|
+
# noindexed, double-tagged or full of placeholder text, and the reader
|
|
594
|
+
# needs those. Only the form-dependent checks are skipped.
|
|
595
|
+
if not forms_late:
|
|
596
|
+
rep.outcome = "no_form"
|
|
597
|
+
rep.add("forms", "Forms found", FAIL,
|
|
598
|
+
"no form or input anywhere on the page, in any frame — nothing here can "
|
|
599
|
+
"capture a lead",
|
|
600
|
+
_related_evidence(main, "form", "iframe"))
|
|
601
|
+
if forms_late:
|
|
602
|
+
bits = [f"{len(real)} form{'' if len(real) == 1 else 's'}"]
|
|
603
|
+
if iframed:
|
|
604
|
+
bits.append(f"{len(iframed)} inside an embedded widget")
|
|
605
|
+
if orphan_fields:
|
|
606
|
+
bits.append(f"{orphan_fields} field{'' if orphan_fields == 1 else 's'} outside any form")
|
|
607
|
+
rep.add("forms", "Forms found", PASS, ", ".join(bits))
|
|
608
|
+
# Only meaningful when the page HAS forms but none of them is embedded —
|
|
609
|
+
# on a page with no form at all the earlier FAIL already said everything.
|
|
610
|
+
if forms_late and rep.platform == "GoHighLevel" and not iframed:
|
|
611
|
+
rep.add("ghl_iframe", "Embedded form", WARN,
|
|
612
|
+
"no embedded form found — a GoHighLevel form is normally a cross-origin "
|
|
613
|
+
"iframe, so the embed may not have rendered")
|
|
614
|
+
|
|
615
|
+
# 3/4 — attribution fields and their values
|
|
616
|
+
map_now, map_late = attribution_map(forms_now), attribution_map(forms_late)
|
|
617
|
+
rep.cookie_attribution = detect_cookie_attribution(main.cookies)
|
|
618
|
+
rep.storage_hits = search_storage(main.storage, main.cookies)
|
|
619
|
+
|
|
620
|
+
if not forms_late:
|
|
621
|
+
pass # no form to carry attribution; the FAIL above already says so
|
|
622
|
+
elif not map_late:
|
|
623
|
+
by_cookie = rep.cookie_attribution
|
|
624
|
+
if by_cookie:
|
|
625
|
+
rep.add("attribution", "Attribution capture", WARN,
|
|
626
|
+
"no attribution field on any form, but " + ", ".join(by_cookie) +
|
|
627
|
+
" identify the visitor by cookie and resolve the source themselves — "
|
|
628
|
+
"a lead routed anywhere else carries nothing",
|
|
629
|
+
[f"cookie attribution: {', '.join(by_cookie)}"])
|
|
630
|
+
else:
|
|
631
|
+
rep.add("attribution", "Attribution capture", FAIL,
|
|
632
|
+
"no attribution field on any form, and no ad or analytics platform is "
|
|
633
|
+
"tracking the visit — nothing is recording where these leads come from",
|
|
634
|
+
_related_evidence(main, "utm", "gclid", "attribution"))
|
|
635
|
+
else:
|
|
636
|
+
empty, late = [], []
|
|
637
|
+
for canon, fields in map_late.items():
|
|
638
|
+
want = TEST_PARAMS[canon]
|
|
639
|
+
if any(want in (f.get("value") or "") for f in map_now.get(canon, [])):
|
|
640
|
+
continue
|
|
641
|
+
(late if any(want in (f.get("value") or "") for f in fields) else empty).append(canon)
|
|
642
|
+
ev = [f"{f['name']} = {f.get('value') or '(empty)'}"
|
|
643
|
+
for fs in map_late.values() for f in fs][:12]
|
|
644
|
+
if empty:
|
|
645
|
+
rep.add("attribution", "Attribution capture", FAIL,
|
|
646
|
+
f"field(s) exist but stayed empty: {', '.join(sorted(empty))} — the form "
|
|
647
|
+
"submits, leads arrive, and the campaign is lost",
|
|
648
|
+
ev + _related_evidence(main, "utm", "gclid", "capture"))
|
|
649
|
+
else:
|
|
650
|
+
missing = [c for c in CANONICAL if c not in map_late]
|
|
651
|
+
rep.add("attribution", "Attribution capture", WARN if missing else PASS,
|
|
652
|
+
(f"captured; not present: {', '.join(missing)}" if missing
|
|
653
|
+
else f"all {len(map_late)} attribution field(s) captured"), ev)
|
|
654
|
+
if late:
|
|
655
|
+
rep.add("timing", "Population timing", WARN,
|
|
656
|
+
f"{', '.join(sorted(late))} was empty on load and filled by the "
|
|
657
|
+
f"{LATE_DELAY_S:g}s re-read — a fast submitter loses it", ev)
|
|
658
|
+
|
|
659
|
+
# 5 — storage fallback
|
|
660
|
+
if rep.storage_hits and not map_late:
|
|
661
|
+
rep.add("storage", "Stored in the browser", INFO,
|
|
662
|
+
"the campaign values were stored in the browser, so some setups attach them "
|
|
663
|
+
"at submit time — which a read-only check cannot confirm", rep.storage_hits[:8])
|
|
664
|
+
|
|
665
|
+
# 6 — where the form posts
|
|
666
|
+
for form in (real if forms_late else []):
|
|
667
|
+
verdict = classify_action(form.get("action"), rep.url)
|
|
668
|
+
if verdict:
|
|
669
|
+
sev, why = verdict
|
|
670
|
+
rep.add("endpoint", "Form destination", sev, why, [str(form.get("action"))])
|
|
671
|
+
break
|
|
672
|
+
else:
|
|
673
|
+
actions = [f.get("action") for f in real if f.get("action")]
|
|
674
|
+
page_host = (urlparse(rep.url).hostname or "").lower()
|
|
675
|
+
offsite = sorted({h for a in actions
|
|
676
|
+
if (h := (urlparse(a if "://" in a else f"https://{a}").hostname or "").lower())
|
|
677
|
+
and h != page_host})
|
|
678
|
+
if offsite:
|
|
679
|
+
rep.add("endpoint", "Form destination", WARN,
|
|
680
|
+
"submits to " + ", ".join(offsite) + " rather than the site's own domain — "
|
|
681
|
+
"confirm the client actually controls that destination",
|
|
682
|
+
[str(a) for a in actions][:4])
|
|
683
|
+
else:
|
|
684
|
+
rep.add("endpoint", "Form destination", PASS if actions else INFO,
|
|
685
|
+
"posts to the site's own domain" if actions
|
|
686
|
+
else "no form action attribute — the page submits with its own script",
|
|
687
|
+
[str(a) for a in actions][:4])
|
|
688
|
+
|
|
689
|
+
# 7 — anti-spam
|
|
690
|
+
cap = detect_captcha(main.html)
|
|
691
|
+
if cap:
|
|
692
|
+
ok = cap["site_key_present"]
|
|
693
|
+
rep.add("captcha", "Anti-spam", PASS if ok else FAIL,
|
|
694
|
+
f"{cap['kind']} present with a site key" if ok else
|
|
695
|
+
f"{cap['kind']} is on the page but no site key was found — this can block "
|
|
696
|
+
"every submission while the page looks fine",
|
|
697
|
+
_related_evidence(main, "recaptcha", "hcaptcha", "turnstile"))
|
|
698
|
+
else:
|
|
699
|
+
rep.add("captcha", "Anti-spam", INFO, "no captcha on this page")
|
|
700
|
+
|
|
701
|
+
# 8 — tracking inventory
|
|
702
|
+
rep.tracking = find_tracking(main.html, main.requests)
|
|
703
|
+
script_only = rep.tracking.pop("meta_script_only", [])
|
|
704
|
+
dupes = [f"{len(v)}× {k.upper()}" for k, v in rep.tracking.items() if len(v) > 1]
|
|
705
|
+
ids = [f"{k.upper()}: {', '.join(v)}" for k, v in rep.tracking.items() if v]
|
|
706
|
+
if dupes:
|
|
707
|
+
rep.add("tracking", "Tracking tags", WARN,
|
|
708
|
+
"more than one of the same tag is installed (" + ", ".join(dupes) +
|
|
709
|
+
") — duplicates double-count conversions and halve reported cost per lead", ids)
|
|
710
|
+
elif ids:
|
|
711
|
+
rep.add("tracking", "Tracking tags", PASS, "; ".join(ids))
|
|
712
|
+
elif script_only:
|
|
713
|
+
rep.add("tracking", "Tracking tags", WARN,
|
|
714
|
+
"the Meta pixel script loads but no pixel ID was ever set — the tag is on the "
|
|
715
|
+
"page and may never actually record anything", script_only)
|
|
716
|
+
else:
|
|
717
|
+
rep.add("tracking", "Tracking tags", WARN, "no analytics or ad tag found on this page")
|
|
718
|
+
|
|
719
|
+
# 9 — consent divergence: the signature finding
|
|
720
|
+
if other is not None:
|
|
721
|
+
diff = _consent_diff(main, other)
|
|
722
|
+
rep.consent_diff = diff
|
|
723
|
+
if diff.get("unavailable"):
|
|
724
|
+
rep.add("consent", "Consent divergence", INFO, diff["unavailable"])
|
|
725
|
+
elif diff["fields_differ"] or diff["tags_differ"]:
|
|
726
|
+
what = []
|
|
727
|
+
if diff["fields_differ"]:
|
|
728
|
+
what.append(f"{len(diff['fields_differ'])} attribution field(s)")
|
|
729
|
+
if diff["tags_differ"]:
|
|
730
|
+
what.append("tracking tags")
|
|
731
|
+
rep.add("consent", "Consent divergence", FAIL,
|
|
732
|
+
"this page behaves differently before the cookie banner is accepted — " +
|
|
733
|
+
" and ".join(what) + " change — so every visitor who ignores the banner "
|
|
734
|
+
"arrives with different (or missing) data",
|
|
735
|
+
[f"{k}: accepted={v['accepted'] or '(empty)'} · ignored={v['ignored'] or '(empty)'}"
|
|
736
|
+
for k, v in diff["fields_differ"].items()][:8])
|
|
737
|
+
else:
|
|
738
|
+
rep.add("consent", "Consent divergence", PASS,
|
|
739
|
+
"attribution and tags behave the same whether or not the banner is accepted")
|
|
740
|
+
|
|
741
|
+
# 10 — placeholders
|
|
742
|
+
ph = find_placeholders(main.text or "")
|
|
743
|
+
rep.add("placeholder", "Placeholder content", FAIL if ph else PASS,
|
|
744
|
+
"unreplaced placeholder text is visible on the page" if ph
|
|
745
|
+
else "no placeholder text found", ph)
|
|
746
|
+
|
|
747
|
+
# 11 — production hygiene
|
|
748
|
+
robots = (main.robots or "").lower()
|
|
749
|
+
if "noindex" in robots or "nofollow" in robots:
|
|
750
|
+
rep.add("robots", "Search visibility", FAIL,
|
|
751
|
+
f"this live page asks search engines to stay away (robots: {robots})", [robots])
|
|
752
|
+
else:
|
|
753
|
+
rep.add("robots", "Search visibility", PASS, "the page is indexable")
|
|
754
|
+
ssl_status, ssl_detail = check_ssl(rep.tested_url or rep.url, bool(main.html))
|
|
755
|
+
rep.add("ssl", "Certificate", ssl_status, ssl_detail)
|
|
756
|
+
mixed = [r for r in main.failed_requests if r.startswith("http://")]
|
|
757
|
+
if mixed:
|
|
758
|
+
rep.add("mixed", "Mixed content", WARN,
|
|
759
|
+
"the page loads resources over plain http, which browsers may block", mixed[:5])
|
|
760
|
+
|
|
761
|
+
# 12 — diagnostics, kept as their own line as well as attached above
|
|
762
|
+
rep.console_errors = main.console_errors[:20]
|
|
763
|
+
rep.failed_requests = sorted(set(main.failed_requests))[:20]
|
|
764
|
+
if rep.console_errors or rep.failed_requests:
|
|
765
|
+
rep.add("diagnostics", "Errors during load",
|
|
766
|
+
WARN if rep.console_errors else INFO,
|
|
767
|
+
f"{len(rep.console_errors)} console error(s), "
|
|
768
|
+
f"{len(rep.failed_requests)} failed request(s) while loading",
|
|
769
|
+
(rep.console_errors + rep.failed_requests)[:8])
|
|
770
|
+
|
|
771
|
+
|
|
772
|
+
def _consent_diff(accepted: PassResult, ignored: PassResult) -> dict[str, Any]:
|
|
773
|
+
"""What changes between accepting and ignoring the cookie banner. Invisible
|
|
774
|
+
to any single-pass checker, and a real, common, undiagnosed failure."""
|
|
775
|
+
if not accepted.consent_clicked:
|
|
776
|
+
return {"unavailable": "no cookie banner was found, so there is nothing to diverge",
|
|
777
|
+
"fields_differ": {}, "tags_differ": False}
|
|
778
|
+
a_fields = {k: (v[0].get("value") or "") for k, v in
|
|
779
|
+
attribution_map(accepted.forms_late or accepted.forms).items() if v}
|
|
780
|
+
i_fields = {k: (v[0].get("value") or "") for k, v in
|
|
781
|
+
attribution_map(ignored.forms_late or ignored.forms).items() if v}
|
|
782
|
+
differ = {k: {"accepted": a_fields.get(k, ""), "ignored": i_fields.get(k, "")}
|
|
783
|
+
for k in set(a_fields) | set(i_fields)
|
|
784
|
+
if a_fields.get(k, "") != i_fields.get(k, "")}
|
|
785
|
+
a_tags = find_tracking(accepted.html, accepted.requests)
|
|
786
|
+
i_tags = find_tracking(ignored.html, ignored.requests)
|
|
787
|
+
return {"fields_differ": differ, "tags_differ": a_tags != i_tags,
|
|
788
|
+
"tags_accepted": a_tags, "tags_ignored": i_tags,
|
|
789
|
+
"banner_found": accepted.consent_clicked}
|
|
790
|
+
|
|
791
|
+
|
|
792
|
+
# ── Entry point ─────────────────────────────────────────────────────────────
|
|
793
|
+
def check_page(url: str, timeout_ms: int = DEFAULT_TIMEOUT_MS,
|
|
794
|
+
delay_s: float = LATE_DELAY_S,
|
|
795
|
+
on_progress: Callable[[str], None] | None = None) -> dict[str, Any]:
|
|
796
|
+
"""Check one page. Owns its browser; never raises for a bad page — a load
|
|
797
|
+
failure, a timeout and a page with no form are each reported outcomes."""
|
|
798
|
+
from playwright.sync_api import sync_playwright
|
|
799
|
+
|
|
800
|
+
say = on_progress or (lambda _m: None)
|
|
801
|
+
rep = Report(url=url, tested_url=build_test_url(url))
|
|
802
|
+
|
|
803
|
+
try:
|
|
804
|
+
with sync_playwright() as pw:
|
|
805
|
+
browser = pw.chromium.launch(headless=True)
|
|
806
|
+
try:
|
|
807
|
+
say("Loading the page and accepting the cookie banner")
|
|
808
|
+
accepted = _one_pass(browser, rep.tested_url, True, timeout_ms, delay_s)
|
|
809
|
+
if accepted.outcome != "ok":
|
|
810
|
+
rep.outcome, rep.error = accepted.outcome, accepted.error
|
|
811
|
+
return _finish(rep)
|
|
812
|
+
|
|
813
|
+
say("Loading again, ignoring the cookie banner")
|
|
814
|
+
ignored = _one_pass(browser, rep.tested_url, False, timeout_ms, delay_s)
|
|
815
|
+
|
|
816
|
+
say("Comparing the two loads")
|
|
817
|
+
_evaluate(rep, accepted, ignored if ignored.outcome == "ok" else None)
|
|
818
|
+
rep.forms = accepted.forms_late or accepted.forms
|
|
819
|
+
finally:
|
|
820
|
+
try:
|
|
821
|
+
browser.close()
|
|
822
|
+
except Exception:
|
|
823
|
+
pass
|
|
824
|
+
except Exception as e: # noqa: BLE001 — a bad page is a result, not a crash
|
|
825
|
+
rep.outcome = "load_failed"
|
|
826
|
+
rep.error = f"{type(e).__name__}: {str(e).splitlines()[0][:200]}"
|
|
827
|
+
|
|
828
|
+
return _finish(rep)
|
|
829
|
+
|
|
830
|
+
|
|
831
|
+
def _finish(rep: Report) -> dict[str, Any]:
|
|
832
|
+
out = asdict(rep)
|
|
833
|
+
out["failed"] = rep.failed
|
|
834
|
+
out["counts"] = rep.counts
|
|
835
|
+
return out
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pagecheck
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Pre-launch verdict for a landing page: will it actually capture leads and attribution, or is it silently losing them?
|
|
5
|
+
Author-email: Anaum Pandit <anaump7@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/panaum/pagecheck
|
|
8
|
+
Project-URL: Issues, https://github.com/panaum/pagecheck/issues
|
|
9
|
+
Keywords: qa,testing,playwright,landing-page,attribution,lead-capture
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
14
|
+
Classifier: Topic :: Software Development :: Testing
|
|
15
|
+
Requires-Python: >=3.10
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
License-File: LICENSE
|
|
18
|
+
Requires-Dist: playwright>=1.40
|
|
19
|
+
Provides-Extra: web
|
|
20
|
+
Requires-Dist: fastapi>=0.110; extra == "web"
|
|
21
|
+
Requires-Dist: uvicorn>=0.27; extra == "web"
|
|
22
|
+
Requires-Dist: pydantic>=2.0; extra == "web"
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
# pagecheck
|
|
26
|
+
|
|
27
|
+
A landing page can return HTTP 200 on every request and still be losing every
|
|
28
|
+
lead. The form posts to an endpoint that no longer exists. The attribution
|
|
29
|
+
parameters are stripped by a redirect before anything records them. The submit
|
|
30
|
+
button has no destination at all.
|
|
31
|
+
|
|
32
|
+
`pagecheck` gives a single page a pre-launch verdict: will it actually capture
|
|
33
|
+
leads and attribution, or is it silently losing them?
|
|
34
|
+
|
|
35
|
+
## Install
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
pip install pagecheck
|
|
39
|
+
playwright install chromium
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Use
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
pagecheck https://example.com/landing-page
|
|
46
|
+
pagecheck --file urls.txt --json
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Exit code is 0 when nothing failed, 1 when at least one check failed.
|
|
50
|
+
|
|
51
|
+
## What it does
|
|
52
|
+
|
|
53
|
+
Loads the page with test attribution parameters attached, reads the forms back,
|
|
54
|
+
and reports on lead capture, attribution survival, tracking presence and
|
|
55
|
+
destination integrity.
|
|
56
|
+
|
|
57
|
+
## What it does not do
|
|
58
|
+
|
|
59
|
+
It is read-only. It never submits a form, never clicks anything that would
|
|
60
|
+
create a record, and never writes to the page it is checking.
|
|
61
|
+
|
|
62
|
+
A check it cannot prove is reported as a warning, not a failure. For a tool
|
|
63
|
+
whose output goes to a client, a false alarm costs more than a soft warning.
|
|
64
|
+
|
|
65
|
+
## Licence
|
|
66
|
+
|
|
67
|
+
MIT
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
src/pagecheck/__init__.py
|
|
5
|
+
src/pagecheck/app.py
|
|
6
|
+
src/pagecheck/cli.py
|
|
7
|
+
src/pagecheck/engine.py
|
|
8
|
+
src/pagecheck.egg-info/PKG-INFO
|
|
9
|
+
src/pagecheck.egg-info/SOURCES.txt
|
|
10
|
+
src/pagecheck.egg-info/dependency_links.txt
|
|
11
|
+
src/pagecheck.egg-info/entry_points.txt
|
|
12
|
+
src/pagecheck.egg-info/requires.txt
|
|
13
|
+
src/pagecheck.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
pagecheck
|