@analitikisler/ai-tag-debugger 0.6.1 → 0.6.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -0
- package/README.md +1 -1
- package/README.tr.md +1 -1
- package/package.json +1 -1
- package/src/browser.js +71 -6
- package/src/checks.js +33 -4
- package/src/mcp.js +1 -1
- package/src/parsers.js +8 -4
- package/src/redact.js +58 -19
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,18 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.6.3 (2026-10-12)
|
|
4
|
+
|
|
5
|
+
- The audited site's own email address in a `mailto:` link or a link field (`link_url`, `link_text`, `gtm.elementUrl`, `gtm.elementText`), such as the contact link in a GTM `gtm.linkClick` push, is no longer reported as unhashed user data or redacted. An own-domain address anywhere else, and every named user data field (`email`, `user_data`, `em`, …), is still redacted and reported.
|
|
6
|
+
|
|
7
|
+
## 0.6.2 (2026-10-12)
|
|
8
|
+
|
|
9
|
+
Fixes from Bahadır's audit of analitikisler.com.
|
|
10
|
+
|
|
11
|
+
- GA4 resends (`gaf=1`, which gtag.js sends when it thinks the first send was lost) are not counted as hits on any host. Copies to `www.google.com` are labelled "Google Analytics (resend)" instead of "server-side".
|
|
12
|
+
- The duplicate check counts per page load (GA4's `_p`, or the page address). A step that opens a new page no longer adds the previous page's last `page_view` to the new one.
|
|
13
|
+
- A `page_view` sent while its page unloads is filed under the step that loaded that page, not the next step. A `page_view` for the page that is still open (for example GTM resending it after consent) stays in the step that sent it.
|
|
14
|
+
- dataLayer entries holding DOM elements or objects that refer to themselves (such as GTM click events) are kept, with the element shown as `<tag#id>` and only true cycles cut, instead of becoming "[object Object]".
|
|
15
|
+
|
|
3
16
|
## 0.6.1 (2026-10-12)
|
|
4
17
|
|
|
5
18
|
- GA4 events were reported as "sent 2 times" on sites with Google signals or ads features. gtag.js also sends a copy of those events to `www.google.com/g/collect` (or `doubleclick.net`), which GA4 doesn't count. Those copies are now still answered locally and listed under network requests, but no longer counted as hits. Requests to `analytics.google.com` still count: it is a primary GA4 endpoint.
|
package/README.md
CHANGED
|
@@ -316,7 +316,7 @@ A hosted server, so you don't need a tunnel, is part of the [Analitik İşler](h
|
|
|
316
316
|
- The browser stays on the audited site and its subdomains, plus any `allowed_hosts`. It opens only `http` and `https` pages.
|
|
317
317
|
- The model is told that page content is data, never instructions. A prompt rule can't guarantee a model is never misled, so the navigation limits above contain what a misled model can do.
|
|
318
318
|
- The browser runs without your API keys and other secrets in its environment.
|
|
319
|
-
- **User data sent for ad matching is redacted** before it is stored in `capture.json` or shown in the report: emails, phone numbers, names, addresses and user ids in GA4 and Google Ads enhanced conversions (`em`, `ph`, `uid`, …), Meta advanced matching (`ud[*]`, `udff[*]`), TikTok `context.user`, and the `user_data` the site pushes to the dataLayer for enhanced conversions. Any value that looks like an email address is redacted whatever its field is called. Each value becomes a marker such as `[redacted: 64 chars, looks hashed]`, and a value that wasn't hashed is reported as a personal data risk. `--keep-bodies` (`"keep_bodies": true`) stores requests as sent, for debugging.
|
|
319
|
+
- **User data sent for ad matching is redacted** before it is stored in `capture.json` or shown in the report: emails, phone numbers, names, addresses and user ids in GA4 and Google Ads enhanced conversions (`em`, `ph`, `uid`, …), Meta advanced matching (`ud[*]`, `udff[*]`), TikTok `context.user`, and the `user_data` the site pushes to the dataLayer for enhanced conversions. Any value that looks like an email address is redacted whatever its field is called, except the audited site's own address in a `mailto:` link or a link field (`link_url`, `link_text`, `gtm.elementUrl`, …), such as a contact link in a GTM click event, which is not visitor data. Each value becomes a marker such as `[redacted: 64 chars, looks hashed]`, and a value that wasn't hashed is reported as a personal data risk. `--keep-bodies` (`"keep_bodies": true`) stores requests as sent, for debugging.
|
|
320
320
|
- Stored request bodies are capped at 32 KB per request and 2 MB per journey. Past that, the report says the body wasn't stored.
|
|
321
321
|
- Only run it against sites you own or are authorized to test.
|
|
322
322
|
- Hosting the HTTP server for other people? Also block private and cloud metadata address ranges at the network level (an egress firewall or an isolated network), as defense in depth.
|
package/README.tr.md
CHANGED
|
@@ -318,7 +318,7 @@ Tünele gerek kalmayan, hazır bir adresle barındırılan sunucu, [Analitik İ
|
|
|
318
318
|
- Tarayıcı test edilen sitede ve alt alan adlarında, ayrıca `allowed_hosts` içindeki adreslerde kalır. Sadece `http` ve `https` sayfalarını açar.
|
|
319
319
|
- Modele, sayfa içeriğinin talimat değil veri olduğu söylenir. Bir prompt kuralı, modelin hiçbir zaman yanıltılmayacağını garanti edemez; bu yüzden yukarıdaki gezinme sınırları, yanıltılmış bir modelin yapabileceklerini kısıtlar.
|
|
320
320
|
- Tarayıcı, ortamında API anahtarlarınız ve diğer gizli bilgiler olmadan çalışır.
|
|
321
|
-
- **Reklam eşleştirmesi için gönderilen kullanıcı verileri**, `capture.json` dosyasına kaydedilmeden ya da raporda gösterilmeden önce **gizlenir**: GA4 ve Google Ads enhanced conversions (`em`, `ph`, `uid`, …), Meta advanced matching (`ud[*]`, `udff[*]`), TikTok `context.user` içindeki e-postalar, telefon numaraları, adlar, adresler ve kullanıcı kimlikleri ile sitenin enhanced conversions için dataLayer'a gönderdiği `user_data`. E-posta adresine benzeyen her değer, alanın adı ne olursa olsun gizlenir. Her değer `[redacted: 64 chars, looks hashed]` gibi bir işarete dönüşür; hash'lenmemiş bir değer kişisel veri riski olarak raporlanır. `--keep-bodies` (`"keep_bodies": true`) hata ayıklama için istekleri gönderildiği haliyle saklar.
|
|
321
|
+
- **Reklam eşleştirmesi için gönderilen kullanıcı verileri**, `capture.json` dosyasına kaydedilmeden ya da raporda gösterilmeden önce **gizlenir**: GA4 ve Google Ads enhanced conversions (`em`, `ph`, `uid`, …), Meta advanced matching (`ud[*]`, `udff[*]`), TikTok `context.user` içindeki e-postalar, telefon numaraları, adlar, adresler ve kullanıcı kimlikleri ile sitenin enhanced conversions için dataLayer'a gönderdiği `user_data`. E-posta adresine benzeyen her değer, alanın adı ne olursa olsun gizlenir. Denetlenen sitenin kendi adresi bir `mailto:` bağlantısında ya da bir bağlantı alanında (`link_url`, `link_text`, `gtm.elementUrl`, …) geçiyorsa, örneğin bir GTM tıklama olayındaki iletişim bağlantısında, ziyaretçi verisi olmadığı için bu kuralın dışında kalır. Her değer `[redacted: 64 chars, looks hashed]` gibi bir işarete dönüşür; hash'lenmemiş bir değer kişisel veri riski olarak raporlanır. `--keep-bodies` (`"keep_bodies": true`) hata ayıklama için istekleri gönderildiği haliyle saklar.
|
|
322
322
|
- Saklanan istek gövdeleri istek başına 32 KB ve senaryo başına 2 MB ile sınırlıdır. Bu sınırın üstünde rapor, gövdenin saklanmadığını belirtir.
|
|
323
323
|
- Sadece size ait ya da test etme yetkiniz olan sitelerde çalıştırın.
|
|
324
324
|
- HTTP sunucusunu başkaları için mi barındırıyorsunuz? Ek bir güvenlik katmanı olarak özel ağ ve bulut metadata adres aralıklarını ağ seviyesinde de engelleyin (bir egress firewall ya da izole bir ağ ile).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@analitikisler/ai-tag-debugger",
|
|
3
|
-
"version": "0.6.
|
|
3
|
+
"version": "0.6.3",
|
|
4
4
|
"description": "Open-source AI agent that QA-tests your website's analytics, ad and consent tracking (GA4, GTM, Google Ads, Meta, TikTok). Bring your own model (Claude, OpenAI, Gemini) or use it from your chat app over MCP.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
package/src/browser.js
CHANGED
|
@@ -20,13 +20,51 @@ export const VIEWPORTS = {
|
|
|
20
20
|
// Runs in the page before any site script: reports every dataLayer push
|
|
21
21
|
// (including gtag consent commands) to Node through an exposed binding, so the
|
|
22
22
|
// log survives page navigations.
|
|
23
|
+
/** A page address without its hash, for matching page_location to navigations. */
|
|
24
|
+
const pageKey = (url) => String(url ?? "").split("#")[0];
|
|
25
|
+
/** The host of a URL, or null. */
|
|
26
|
+
const hostOf = (url) => {
|
|
27
|
+
try {
|
|
28
|
+
return new URL(url).hostname || null;
|
|
29
|
+
} catch {
|
|
30
|
+
return null;
|
|
31
|
+
}
|
|
32
|
+
};
|
|
33
|
+
|
|
23
34
|
const INIT_SCRIPT = `(() => {
|
|
24
35
|
const record = (entry) => {
|
|
25
36
|
let value = entry;
|
|
26
37
|
try {
|
|
27
38
|
// gtag() pushes an arguments object; store it as a plain array.
|
|
28
39
|
if (entry && typeof entry === "object" && typeof entry.length === "number" && !Array.isArray(entry)) value = Array.from(entry);
|
|
29
|
-
|
|
40
|
+
// GTM click and form events carry DOM elements and objects that refer to themselves.
|
|
41
|
+
// Copied by hand so only true cycles (an object inside itself) are cut, not shared objects.
|
|
42
|
+
const copy = (v, parents) => {
|
|
43
|
+
if (typeof v === "bigint") return String(v);
|
|
44
|
+
if (typeof v === "function" || typeof v === "symbol") return undefined;
|
|
45
|
+
if (!v || typeof v !== "object") return v;
|
|
46
|
+
if (typeof Element !== "undefined" && v instanceof Element) return "<" + v.tagName.toLowerCase() + (v.id ? "#" + v.id : "") + ">";
|
|
47
|
+
if (typeof Node !== "undefined" && v instanceof Node) return "[node]";
|
|
48
|
+
if (v === window) return "[window]";
|
|
49
|
+
if (parents.includes(v)) return "[circular]";
|
|
50
|
+
if (parents.length > 20) return "[too deep]";
|
|
51
|
+
if (typeof v.toJSON === "function") { try { return copy(v.toJSON(), parents); } catch { return String(v); } }
|
|
52
|
+
parents.push(v);
|
|
53
|
+
let out;
|
|
54
|
+
if (Array.isArray(v)) out = v.map((x) => { const c = copy(x, parents); return c === undefined ? null : c; });
|
|
55
|
+
else {
|
|
56
|
+
out = {};
|
|
57
|
+
for (const k of Object.keys(v)) {
|
|
58
|
+
let x;
|
|
59
|
+
try { x = v[k]; } catch { continue; }
|
|
60
|
+
const c = copy(x, parents);
|
|
61
|
+
if (c !== undefined) out[k] = c;
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
parents.pop();
|
|
65
|
+
return out;
|
|
66
|
+
};
|
|
67
|
+
value = JSON.parse(JSON.stringify(copy(value, [])));
|
|
30
68
|
} catch { value = String(entry); }
|
|
31
69
|
if (window.__atdRecord) window.__atdRecord({ url: location.href, value });
|
|
32
70
|
};
|
|
@@ -183,6 +221,8 @@ export class AuditSession {
|
|
|
183
221
|
this.network = [];
|
|
184
222
|
this.providers = resolveProviders(this.opts.providers);
|
|
185
223
|
this.skipped = {}; // step index -> { vendor label: requests }
|
|
224
|
+
this.loadedIn = new Map(); // page address -> the last step whose main-frame navigation reached it
|
|
225
|
+
this.leaving = null; // the page address being navigated away from, until the next page commits
|
|
186
226
|
this.bodyBytes = 0;
|
|
187
227
|
this.bodyBudgetStep = null; // the step at which POST bodies stopped being stored
|
|
188
228
|
this.startedAt = Date.now();
|
|
@@ -207,6 +247,15 @@ export class AuditSession {
|
|
|
207
247
|
await this.context.addInitScript(INIT_SCRIPT);
|
|
208
248
|
this.page = await this.context.newPage();
|
|
209
249
|
this.page.on("pageerror", (err) => this.errors.push({ step: this.currentStep, message: err.message }));
|
|
250
|
+
this.page.on("framenavigated", (frame) => {
|
|
251
|
+
if (frame !== this.page.mainFrame()) return;
|
|
252
|
+
this.loadedIn.set(pageKey(frame.url()), this.currentStep);
|
|
253
|
+
this.leaving = null;
|
|
254
|
+
});
|
|
255
|
+
// From the moment the main frame starts loading another document, the open page is leaving.
|
|
256
|
+
this.page.on("request", (request) => {
|
|
257
|
+
if (request.isNavigationRequest() && request.frame() === this.page.mainFrame()) this.leaving = pageKey(this.page.url());
|
|
258
|
+
});
|
|
210
259
|
|
|
211
260
|
const onRequest = (request) => {
|
|
212
261
|
const hits = parseTrackingRequest(request.url(), request.postData());
|
|
@@ -214,7 +263,7 @@ export class AuditSession {
|
|
|
214
263
|
const recorded = [];
|
|
215
264
|
for (const hit of hits) {
|
|
216
265
|
if (!this.providers.includes(hit.platform)) continue;
|
|
217
|
-
const entry = { ...this.redactHit(hit), step: this.
|
|
266
|
+
const entry = { ...this.redactHit(hit), step: this.stepForHit(hit), pageUrl: this.page.url(), t: Date.now() };
|
|
218
267
|
if (this.record("hits", entry)) recorded.push(entry);
|
|
219
268
|
}
|
|
220
269
|
if (recorded.length) hitsOf.set(request, recorded);
|
|
@@ -296,12 +345,27 @@ export class AuditSession {
|
|
|
296
345
|
this.beginStep("start");
|
|
297
346
|
}
|
|
298
347
|
|
|
348
|
+
/**
|
|
349
|
+
* The step a hit belongs to. A page_view sent as the page unloads arrives after the next
|
|
350
|
+
* step has started, so it goes to the step that loaded its page, when that was earlier.
|
|
351
|
+
*/
|
|
352
|
+
stepForHit(hit) {
|
|
353
|
+
if (hit.name !== "page_view" || !hit.params?.page_location) return this.currentStep;
|
|
354
|
+
// Only when its page is going or gone: a page_view for the page still open (GTM resending
|
|
355
|
+
// it after consent, a virtual page view) belongs to the current step.
|
|
356
|
+
const page = pageKey(hit.params.page_location);
|
|
357
|
+
if (page === pageKey(this.page.url()) && page !== this.leaving) return this.currentStep;
|
|
358
|
+
const loaded = this.loadedIn.get(page);
|
|
359
|
+
return loaded !== undefined && loaded < this.currentStep ? loaded : this.currentStep;
|
|
360
|
+
}
|
|
361
|
+
|
|
299
362
|
/** The hit with user data for ad matching redacted, and userData: { fields, plain } when there was any. */
|
|
300
363
|
redactHit({ userFields, ...hit }) {
|
|
301
364
|
const found = { fields: [], plain: [] };
|
|
302
|
-
const
|
|
303
|
-
const
|
|
304
|
-
redactParams(
|
|
365
|
+
const site = hostOf(hit.params?.page_location) ?? hostOf(this.page.url());
|
|
366
|
+
const url = redactUrl(hit.url, found, site).url;
|
|
367
|
+
const params = redactParams(hit.params, found, site);
|
|
368
|
+
redactParams(userFields, found, site); // fields from the POST body that the hit doesn't keep
|
|
305
369
|
if (!found.fields.length) return hit;
|
|
306
370
|
const userData = { fields: [...new Set(found.fields)], plain: [...new Set(found.plain)] };
|
|
307
371
|
return this.opts.keepBodies ? { ...hit, userData } : { ...hit, url, params, userData };
|
|
@@ -309,7 +373,7 @@ export class AuditSession {
|
|
|
309
373
|
|
|
310
374
|
/** A dataLayer entry with user data redacted, and userData: { fields, plain } when there was any. */
|
|
311
375
|
redactEntry(entry) {
|
|
312
|
-
const found = redactPush(entry.value);
|
|
376
|
+
const found = redactPush(entry.value, undefined, hostOf(this.page.url()));
|
|
313
377
|
if (!found.fields.length) return entry;
|
|
314
378
|
const userData = { fields: [...new Set(found.fields)], plain: [...new Set(found.plain)] };
|
|
315
379
|
return this.opts.keepBodies ? { ...entry, userData } : { ...entry, value: found.value, userData };
|
|
@@ -366,6 +430,7 @@ export class AuditSession {
|
|
|
366
430
|
}
|
|
367
431
|
|
|
368
432
|
beginStep(label, extra = {}) {
|
|
433
|
+
this.leaving = null; // a navigation that never committed (a download, a 204) ends with its step
|
|
369
434
|
this.steps.push({ index: this.steps.length, label, t: Date.now(), ...extra });
|
|
370
435
|
}
|
|
371
436
|
|
package/src/checks.js
CHANGED
|
@@ -2,9 +2,10 @@
|
|
|
2
2
|
// They run without any API call; the AI model only explains the results.
|
|
3
3
|
|
|
4
4
|
import { messages } from "./i18n.js";
|
|
5
|
+
import { isOwnAddress } from "./redact.js";
|
|
5
6
|
|
|
6
7
|
const NON_GOOGLE_AD_PLATFORMS = new Set(["meta", "tiktok"]);
|
|
7
|
-
const EMAIL = /[A-Z0-9._%+-]+(@|%40)[A-Z0-9.-]+\.[A-Z]{2,}/
|
|
8
|
+
const EMAIL = /[A-Z0-9._%+-]+(@|%40)[A-Z0-9.-]+\.[A-Z]{2,}/gi;
|
|
8
9
|
const SEVERITY_ORDER = { broken: 0, risk: 1, warning: 2, info: 3, ok: 4 };
|
|
9
10
|
|
|
10
11
|
/**
|
|
@@ -15,6 +16,33 @@ const SEVERITY_ORDER = { broken: 0, risk: 1, warning: 2, info: 3, ok: 4 };
|
|
|
15
16
|
|
|
16
17
|
const key = (platform, name) => `${platform}:${name}`;
|
|
17
18
|
|
|
19
|
+
/** The page load a hit came from: GA4's page load id (_p) when it has one, else the page address. */
|
|
20
|
+
function pageOf(hit) {
|
|
21
|
+
try {
|
|
22
|
+
const p = new URL(hit.url).searchParams.get("_p");
|
|
23
|
+
if (p) return p;
|
|
24
|
+
} catch { /* no URL */ }
|
|
25
|
+
return hit.params?.page_location ?? hit.pageUrl ?? "";
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
// Just before an address: a mailto: link or a link field (link_url=..., "link_text":"...").
|
|
29
|
+
const LINK_BEFORE = /(mailto:|\b(link_url|link_text|click_url|click_text)(=|":"))$/i;
|
|
30
|
+
|
|
31
|
+
/**
|
|
32
|
+
* Whether text holds an email address that could be a visitor's. The site's own address in a
|
|
33
|
+
* mailto: link or a link field is a contact link it shows; anywhere else it still counts.
|
|
34
|
+
*/
|
|
35
|
+
function visitorEmailIn(text, hit) {
|
|
36
|
+
let site = null;
|
|
37
|
+
try {
|
|
38
|
+
site = new URL(hit.params?.page_location ?? hit.pageUrl).hostname;
|
|
39
|
+
} catch { /* no page address */ }
|
|
40
|
+
return [...text.matchAll(EMAIL)].some((m) => {
|
|
41
|
+
const link = LINK_BEFORE.test(text.slice(Math.max(0, m.index - 16), m.index));
|
|
42
|
+
return !(link && isOwnAddress(m[0].replace(/%40/i, "@"), site));
|
|
43
|
+
});
|
|
44
|
+
}
|
|
45
|
+
|
|
18
46
|
function expectedIn(planEvent, journeyId) {
|
|
19
47
|
return !planEvent.journeys || planEvent.journeys.includes(journeyId);
|
|
20
48
|
}
|
|
@@ -95,10 +123,11 @@ export function checkRun(plan, run, { lang = "en" } = {}) {
|
|
|
95
123
|
}
|
|
96
124
|
}
|
|
97
125
|
|
|
98
|
-
// The same event sent more than once by a single action.
|
|
126
|
+
// The same event sent more than once by a single action on one page. A step that loads a
|
|
127
|
+
// new page can also carry the previous page's last hits, so the page is part of the key.
|
|
99
128
|
const perStep = new Map();
|
|
100
129
|
for (const hit of run.hits) {
|
|
101
|
-
const k = `${hit.step}|${key(hit.platform, hit.name)}`;
|
|
130
|
+
const k = `${hit.step}|${key(hit.platform, hit.name)}|${pageOf(hit)}`;
|
|
102
131
|
perStep.set(k, [...(perStep.get(k) ?? []), hit]);
|
|
103
132
|
}
|
|
104
133
|
for (const [, hits] of perStep) {
|
|
@@ -162,7 +191,7 @@ export function checkRun(plan, run, { lang = "en" } = {}) {
|
|
|
162
191
|
if (hit.userData?.plain?.length) {
|
|
163
192
|
piiSeen.add(key(hit.platform, hit.name));
|
|
164
193
|
findings.push({ ...base, severity: "risk", check: "pii", platform: hit.platform, event: hit.name, step: stepLabel(hit.step), detail: d.piiUnhashed(hit.platform, hit.name, hit.userData.plain) });
|
|
165
|
-
} else if (
|
|
194
|
+
} else if (visitorEmailIn(haystack, hit)) {
|
|
166
195
|
piiSeen.add(key(hit.platform, hit.name));
|
|
167
196
|
findings.push({ ...base, severity: "risk", check: "pii", platform: hit.platform, event: hit.name, step: stepLabel(hit.step), detail: d.pii(hit.platform, hit.name) });
|
|
168
197
|
}
|
package/src/mcp.js
CHANGED
|
@@ -22,7 +22,7 @@ import { loadConfig, readJson, runAudit, writeReports } from "./audit.js";
|
|
|
22
22
|
import { DEFAULT_PROVIDERS, PROVIDERS as TRACKING_PROVIDERS } from "./parsers.js";
|
|
23
23
|
import { expectEvents, filterFindings, JOURNEY_PRESETS, onlyEvents, presetFor, withPresetEvents } from "./journeys.js";
|
|
24
24
|
|
|
25
|
-
const VERSION = "0.6.
|
|
25
|
+
const VERSION = "0.6.3";
|
|
26
26
|
const MAX_OPEN_AUDITS = 4;
|
|
27
27
|
const IDLE_MS = 15 * 60 * 1000; // an audit untouched this long is closed
|
|
28
28
|
const MAX_BODY_BYTES = 1024 * 1024;
|
package/src/parsers.js
CHANGED
|
@@ -35,9 +35,9 @@ export function parseGa4(url, postData) {
|
|
|
35
35
|
// Matched on the GA4 protocol rather than the host, so server-side GTM
|
|
36
36
|
// endpoints on first-party domains are captured too.
|
|
37
37
|
if (!u.pathname.endsWith("/g/collect") || u.searchParams.get("v") !== "2") return null;
|
|
38
|
-
// gtag.js
|
|
39
|
-
//
|
|
40
|
-
if (GOOGLE_COMPANION_HOST.test(u.hostname)) return null;
|
|
38
|
+
// gtag.js resends a hit to www.google.com/g/collect (marked gaf=1) when the first send
|
|
39
|
+
// seems lost, and copies some to doubleclick.net. GA4 doesn't count these twice, so they aren't hits.
|
|
40
|
+
if (GOOGLE_COMPANION_HOST.test(u.hostname) || u.searchParams.has("gaf")) return null;
|
|
41
41
|
const base = new URLSearchParams(u.search);
|
|
42
42
|
const lines = postData ? postData.split("\n").filter(Boolean) : [""];
|
|
43
43
|
return lines.map((line) => {
|
|
@@ -236,6 +236,10 @@ export function resolveProviders(list, plan) {
|
|
|
236
236
|
export function vendorOf(url) {
|
|
237
237
|
const p = providerOf(url);
|
|
238
238
|
if (!p) return null;
|
|
239
|
-
if (p.key === "ga4")
|
|
239
|
+
if (p.key === "ga4") {
|
|
240
|
+
const host = new URL(url).hostname;
|
|
241
|
+
if (/(^|\.)(google-analytics|analytics\.google|googletagmanager)\.com$/.test(host)) return "Google Analytics";
|
|
242
|
+
return GOOGLE_COMPANION_HOST.test(host) ? "Google Analytics (resend)" : "Google Analytics (server-side)";
|
|
243
|
+
}
|
|
240
244
|
return p.label;
|
|
241
245
|
}
|
package/src/redact.js
CHANGED
|
@@ -14,9 +14,48 @@ const USER_KEYS = new Set([
|
|
|
14
14
|
const PUSH_KEYS = new Set(["email", "email_address", "phone", "phone_number", "sha256_email_address", "sha256_phone_number", "sha256_email", "sha256_phone"]);
|
|
15
15
|
const USER_OBJECTS = /(^|\.)(user_data|user_provided_data|enhanced_conversion_data)(\.|$)/;
|
|
16
16
|
|
|
17
|
-
const EMAIL = /^[^\s@]+@[^\s
|
|
18
|
-
|
|
19
|
-
|
|
17
|
+
const EMAIL = /^(mailto:)?[^\s@]+@([^\s@?]+\.[^\s@?]+)(\?.*)?$/i;
|
|
18
|
+
|
|
19
|
+
// Second levels that are shared under a country code (shop.com.tr, shop.co.uk), and hosting
|
|
20
|
+
// suffixes whose subdomains belong to different owners (shop.myshopify.com).
|
|
21
|
+
const SHARED_SECOND_LEVELS = new Set(["com", "co", "org", "net", "gov", "edu", "ac", "gen", "web", "bel", "k12", "or", "ne", "go"]);
|
|
22
|
+
const SHARED_SUFFIXES = ["myshopify.com", "github.io", "vercel.app", "pages.dev", "netlify.app", "herokuapp.com", "web.app", "firebaseapp.com", "appspot.com", "wixsite.com", "blogspot.com", "azurewebsites.net", "cloudfront.net", "workers.dev"];
|
|
23
|
+
|
|
24
|
+
/** "www.shop.example.com.tr" -> "example.com.tr", "a.shop.myshopify.com" -> "shop.myshopify.com": the part one owner controls. */
|
|
25
|
+
function siteOf(host) {
|
|
26
|
+
const labels = String(host).toLowerCase().replace(/\.$/, "").split(".");
|
|
27
|
+
const name = labels.join(".");
|
|
28
|
+
const shared = SHARED_SUFFIXES.find((s) => name === s || name.endsWith(`.${s}`));
|
|
29
|
+
let n = 2;
|
|
30
|
+
if (shared) n = shared.split(".").length + 1;
|
|
31
|
+
else if (labels.length >= 3 && labels.at(-1).length === 2 && SHARED_SECOND_LEVELS.has(labels.at(-2))) n = 3;
|
|
32
|
+
return labels.slice(-n).join(".");
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/** Whether an email address is on the audited site's own domain, like hello@shop.com on a link at shop.com. */
|
|
36
|
+
export function isOwnAddress(email, siteHost) {
|
|
37
|
+
if (!siteHost) return false;
|
|
38
|
+
const domain = String(email).trim().match(EMAIL)?.[2];
|
|
39
|
+
return Boolean(domain) && siteOf(domain) === siteOf(siteHost);
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
// Fields that hold a link or its text. The site's own address there is a contact link it shows.
|
|
43
|
+
const LINK_KEYS = new Set(["link_url", "link_text", "click_url", "click_text"]);
|
|
44
|
+
const isLinkKey = (key) => {
|
|
45
|
+
const k = String(key ?? "").toLowerCase();
|
|
46
|
+
return LINK_KEYS.has(k.split(".").pop()) || /(^|\.)gtm\.element(url|text)$/.test(k);
|
|
47
|
+
};
|
|
48
|
+
|
|
49
|
+
/**
|
|
50
|
+
* A value that is an email address (or a mailto: link to one), whatever its field is called.
|
|
51
|
+
* The site's own address in a mailto: link or a link field is a contact link it shows, not
|
|
52
|
+
* visitor data. Anywhere else even an own-domain address could be a visitor's (staff, B2B).
|
|
53
|
+
*/
|
|
54
|
+
const isEmail = (v, site, key) => {
|
|
55
|
+
if (typeof v !== "string" || !EMAIL.test(v.trim())) return false;
|
|
56
|
+
const link = /^mailto:/i.test(v.trim()) || isLinkKey(key);
|
|
57
|
+
return !(link && isOwnAddress(v, site));
|
|
58
|
+
};
|
|
20
59
|
|
|
21
60
|
/** Whether a request parameter (or dotted JSON path) carries user data for ad matching. */
|
|
22
61
|
export function isUserDataKey(key) {
|
|
@@ -49,34 +88,34 @@ function note(found, key, value) {
|
|
|
49
88
|
}
|
|
50
89
|
|
|
51
90
|
/** Redacts form-encoded parameters (a URL query or a GA4 body line). Returns null when nothing needed it. */
|
|
52
|
-
function redactParamString(text, found) {
|
|
91
|
+
function redactParamString(text, found, site) {
|
|
53
92
|
const params = new URLSearchParams(text);
|
|
54
|
-
if (![...params].some(([k, v]) => isUserDataKey(k) || isEmail(v))) return null;
|
|
93
|
+
if (![...params].some(([k, v]) => isUserDataKey(k) || isEmail(v, site, k))) return null;
|
|
55
94
|
const out = new URLSearchParams();
|
|
56
|
-
for (const [k, v] of params) out.append(k, isUserDataKey(k) || isEmail(v) ? note(found, k, v) : v);
|
|
95
|
+
for (const [k, v] of params) out.append(k, isUserDataKey(k) || isEmail(v, site, k) ? note(found, k, v) : v);
|
|
57
96
|
return out.toString();
|
|
58
97
|
}
|
|
59
98
|
|
|
60
|
-
/** Redacts user data in a URL's query. Returns the URL unchanged when there is none. */
|
|
61
|
-
export function redactUrl(url, found = emptyFound()) {
|
|
99
|
+
/** Redacts user data in a URL's query. Returns the URL unchanged when there is none. site: see isOwnAddress. */
|
|
100
|
+
export function redactUrl(url, found = emptyFound(), site = null) {
|
|
62
101
|
let u;
|
|
63
102
|
try {
|
|
64
103
|
u = new URL(url);
|
|
65
104
|
} catch {
|
|
66
105
|
return { url, ...found };
|
|
67
106
|
}
|
|
68
|
-
const query = redactParamString(u.search, found);
|
|
107
|
+
const query = redactParamString(u.search, found, site);
|
|
69
108
|
if (query === null) return { url, ...found };
|
|
70
109
|
u.search = query;
|
|
71
110
|
return { url: u.href, ...found };
|
|
72
111
|
}
|
|
73
112
|
|
|
74
|
-
function redactJson(value, path, found, isKey) {
|
|
75
|
-
if (Array.isArray(value)) return value.map((v) => redactJson(v, path, found, isKey));
|
|
113
|
+
function redactJson(value, path, found, isKey, site = null) {
|
|
114
|
+
if (Array.isArray(value)) return value.map((v) => redactJson(v, path, found, isKey, site));
|
|
76
115
|
if (value && typeof value === "object") {
|
|
77
|
-
return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, redactJson(v, path ? `${path}.${k}` : k, found, isKey)]));
|
|
116
|
+
return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, redactJson(v, path ? `${path}.${k}` : k, found, isKey, site)]));
|
|
78
117
|
}
|
|
79
|
-
if ((typeof value === "string" || typeof value === "number") && ((path && isKey(path)) || isEmail(value))) return note(found, path || "(value)", value);
|
|
118
|
+
if ((typeof value === "string" || typeof value === "number") && ((path && isKey(path)) || isEmail(value, site, path))) return note(found, path || "(value)", value);
|
|
80
119
|
return value;
|
|
81
120
|
}
|
|
82
121
|
|
|
@@ -110,21 +149,21 @@ export function redactBody(body, found = emptyFound()) {
|
|
|
110
149
|
return { text, ...found };
|
|
111
150
|
}
|
|
112
151
|
|
|
113
|
-
/** Redacts user-data entries in parsed hit parameters. */
|
|
114
|
-
export function redactParams(params, found = emptyFound()) {
|
|
152
|
+
/** Redacts user-data entries in parsed hit parameters. site: see isOwnAddress. */
|
|
153
|
+
export function redactParams(params, found = emptyFound(), site = null) {
|
|
115
154
|
const out = {};
|
|
116
|
-
for (const [k, v] of Object.entries(params ?? {})) out[k] = isUserDataKey(k) || isEmail(v) ? note(found, k, v) : v;
|
|
155
|
+
for (const [k, v] of Object.entries(params ?? {})) out[k] = isUserDataKey(k) || isEmail(v, site, k) ? note(found, k, v) : v;
|
|
117
156
|
return out;
|
|
118
157
|
}
|
|
119
158
|
|
|
120
159
|
/**
|
|
121
160
|
* Redacts user data in a dataLayer push (GTM objects or gtag.js arguments):
|
|
122
|
-
* email and phone fields, anything under user_data, and email-like values.
|
|
161
|
+
* email and phone fields, anything under user_data, and email-like values not on the site's own domain.
|
|
123
162
|
*/
|
|
124
|
-
export function redactPush(value, found = emptyFound()) {
|
|
163
|
+
export function redactPush(value, found = emptyFound(), site = null) {
|
|
125
164
|
// gtag("set", "user_data", {...}) has no user_data key: the objects after the name are the user data.
|
|
126
165
|
if (Array.isArray(value) && value.includes("user_data")) {
|
|
127
166
|
return { value: value.map((v) => (v && typeof v === "object" ? redactJson(v, "user_data", found, isPushUserKey) : v)), ...found };
|
|
128
167
|
}
|
|
129
|
-
return { value: redactJson(value, "", found, isPushUserKey), ...found };
|
|
168
|
+
return { value: redactJson(value, "", found, isPushUserKey, site), ...found };
|
|
130
169
|
}
|