urlformat-audit 0.1.0__tar.gz

urlformat_audit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 [Author names]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

urlformat_audit-0.1.0/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.5
+Name: urlformat-audit
+Version: 0.1.0
+Summary: Detect class-conditional URL formatting (shortcut risk) in labelled phishing/malicious URL datasets
+Project-URL: Homepage, https://github.com/MIT-Istiak/Learning-the-Template-Not-the-Threat-Format-Shortcuts-in-Cross-Dataset-Phishing-URL-Detection
+Author: Md Istiak Tanvir - eruddro@gmail.com
+License-Expression: MIT
+License-File: LICENSE
+Keywords: dataset audit,machine learning,phishing,security,shortcut learning,url
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Security
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.21
+Requires-Dist: pandas>=1.3
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == 'test'
+Description-Content-Type: text/markdown
+
+# urlformat-audit
+
+Check a labelled phishing / malicious URL dataset for **format shortcuts**: URL formatting
+(scheme, `www.`, trailing slash, path) that differs between the classes and lets a model
+predict the label without learning anything about the threat.
+
+Example: in some datasets every legitimate URL is written `https://www.<host>` and phishing
+URLs are not. A model trained on such data learns "legitimate = this template". It scores
+well in-dataset and fails on any other dataset.
+
+# Author Details
+
+Md Istiak Tanvir - eruddro@gmail.com
+Msc in CSE - MIST
+Dhaka, Bangladesh
+
+`urlformat-audit` reports:
+
+| Output | Meaning |
+|---|---|
+| **Class table** | % of URLs per class with each format property: `https`, `http`, no scheme, `www.`, trailing slash, path, query, port, IP host, uppercase, median length |
+| **Format cells** | the 24 cells scheme (https/http/none) × `www.` × trailing slash × path, with class counts and purity |
+| **Format-only rule** | predicts the majority label of each cell. Its MCC says how much of the label format alone explains. MCC ≥ 0.5 is flagged. |
+| **Template rule** | "legitimate iff the URL matches `^https://www\.[a-z0-9.-]+$`" |
+| **Trivial baseline** | predict the positive class everywhere |
+| **NMI** | I(cell; label) / H(label): share of label entropy explained by the format cell (0 = none, 1 = all) |
+| **Flags** | properties whose prevalence differs by ≥ 30 percentage points between the classes, and large single-class cells |
+| **compare** | fit the format-only rule on dataset A, evaluate it on dataset B. This shows whether format cues transfer, fail, or invert. |
+| **factorial** | writes the 24 format variants of every URL, for testing a trained model's sensitivity to formatting |
+
+Requires Python ≥ 3.9 with numpy and pandas. Nothing else.
+
+## Install
+
+```bash
+pip install urlformat-audit
+```
+
+## Command line
+
+```bash
+# one file with a label column (the positive class is the phishing/malicious label)
+urlformat-audit audit data.csv --positive phishing
+urlformat-audit audit data.csv --url-col URL --label-col label --positive 1 --json report.json
+
+# one file per class, without a label column (CSV/TSV with a url column, or a JSON list of URLs)
+urlformat-audit audit legit.json phish.json --label 0 1 --name "my dataset"
+
+# does a format-only rule fitted on A transfer to B?
+urlformat-audit compare A.csv B.csv --positive 1
+
+# 24 format variants of every URL (columns: id = row of the input, cell, url_variant)
+urlformat-audit factorial test.csv -o test_factorial.csv
+```
+
+Example output (dataset of Sahingoz et al., 2019; legitimate = 0, phishing = 1), abridged:
+
+```
+Format-only rule (majority per cell):   accuracy=0.756  balanced_accuracy=0.756  f1_positive=0.771  mcc=0.515  tpr=0.812  fpr=0.300
+Template rule (legit iff ^https://www.<host>$): accuracy=0.505  ...  mcc=0.000
+Normalized mutual information I(cell;label)/H(label) = 0.266
+
+Shortcut flags:
+  - scheme=https: 36.8% of negatives vs 1.8% of positives
+  - scheme=http: 63.2% of negatives vs 97.7% of positives
+  - format-only rule (4 cues, 24 cells) reaches MCC 0.515 / balanced accuracy 0.756 on this dataset
+```
+
+## Python API
+
+```python
+from urlformat_audit import audit, compare, factorial_variants, format_features, FormatRule
+
+r = audit(urls, labels, name="my data")   # labels: 1 = phishing/malicious, 0 = legitimate
+print(r.format_rule["mcc"], r.nmi_cell_label)
+print(r.class_table)       # pandas DataFrame
+print(r.cell_table.head())
+for f in r.flags: print(f)
+r.to_dict()                # JSON-serialisable
+
+res = compare(train_urls, train_labels, test_urls, test_labels)
+res["format_rule_transfer"]["mcc"], res["share_target_in_unseen_cells"]
+
+factorial_variants("http://example.org/login/")   # [(cell_name, url), ...] x 24
+format_features(urls)                              # per-URL format columns + cell
+```
+
+## How to read the result
+
+- **Format-only rule MCC near 1** (or a NMI near 1): format alone separates the classes, so any
+  model that sees the raw URL can learn the shortcut. Cross-dataset results on this data need
+  format-controlled checks: factorial edits, host-only input, or a scheme-neutral representation.
+- **High in-dataset MCC but low or negative `compare` MCC**: the format cue is specific to that dataset.
+  Expect transfer failures.
+- **Low MCC**: format is not a major shortcut. This says nothing about other artifacts (domains, TLDs,
+  collection time).
+
+The audit is descriptive. It cannot tell you whether a format difference is a collection
+artifact or a real property of phishing URLs; it only tells you that a model can use it.
+
+## Development
+
+```bash
+pip install -e ".[test]"
+pytest
+python -m build        # sdist + wheel in dist/
+```
+
+## Citation
+
+If you use this tool, please cite the accompanying paper:
+*Learning the Template, Not the Threat: Format Shortcuts in Cross-Dataset Phishing URL Detection* (authors and venue to be added).
+
+## License
+
+MIT

urlformat_audit-0.1.0/README.md

@@ -0,0 +1,116 @@
+# urlformat-audit
+
+Check a labelled phishing / malicious URL dataset for **format shortcuts**: URL formatting
+(scheme, `www.`, trailing slash, path) that differs between the classes and lets a model
+predict the label without learning anything about the threat.
+
+Example: in some datasets every legitimate URL is written `https://www.<host>` and phishing
+URLs are not. A model trained on such data learns "legitimate = this template". It scores
+well in-dataset and fails on any other dataset.
+
+# Author Details
+
+Md Istiak Tanvir - eruddro@gmail.com
+Msc in CSE - MIST
+Dhaka, Bangladesh
+
+`urlformat-audit` reports:
+
+| Output | Meaning |
+|---|---|
+| **Class table** | % of URLs per class with each format property: `https`, `http`, no scheme, `www.`, trailing slash, path, query, port, IP host, uppercase, median length |
+| **Format cells** | the 24 cells scheme (https/http/none) × `www.` × trailing slash × path, with class counts and purity |
+| **Format-only rule** | predicts the majority label of each cell. Its MCC says how much of the label format alone explains. MCC ≥ 0.5 is flagged. |
+| **Template rule** | "legitimate iff the URL matches `^https://www\.[a-z0-9.-]+$`" |
+| **Trivial baseline** | predict the positive class everywhere |
+| **NMI** | I(cell; label) / H(label): share of label entropy explained by the format cell (0 = none, 1 = all) |
+| **Flags** | properties whose prevalence differs by ≥ 30 percentage points between the classes, and large single-class cells |
+| **compare** | fit the format-only rule on dataset A, evaluate it on dataset B. This shows whether format cues transfer, fail, or invert. |
+| **factorial** | writes the 24 format variants of every URL, for testing a trained model's sensitivity to formatting |
+
+Requires Python ≥ 3.9 with numpy and pandas. Nothing else.
+
+## Install
+
+```bash
+pip install urlformat-audit
+```
+
+## Command line
+
+```bash
+# one file with a label column (the positive class is the phishing/malicious label)
+urlformat-audit audit data.csv --positive phishing
+urlformat-audit audit data.csv --url-col URL --label-col label --positive 1 --json report.json
+
+# one file per class, without a label column (CSV/TSV with a url column, or a JSON list of URLs)
+urlformat-audit audit legit.json phish.json --label 0 1 --name "my dataset"
+
+# does a format-only rule fitted on A transfer to B?
+urlformat-audit compare A.csv B.csv --positive 1
+
+# 24 format variants of every URL (columns: id = row of the input, cell, url_variant)
+urlformat-audit factorial test.csv -o test_factorial.csv
+```
+
+Example output (dataset of Sahingoz et al., 2019; legitimate = 0, phishing = 1), abridged:
+
+```
+Format-only rule (majority per cell):   accuracy=0.756  balanced_accuracy=0.756  f1_positive=0.771  mcc=0.515  tpr=0.812  fpr=0.300
+Template rule (legit iff ^https://www.<host>$): accuracy=0.505  ...  mcc=0.000
+Normalized mutual information I(cell;label)/H(label) = 0.266
+
+Shortcut flags:
+  - scheme=https: 36.8% of negatives vs 1.8% of positives
+  - scheme=http: 63.2% of negatives vs 97.7% of positives
+  - format-only rule (4 cues, 24 cells) reaches MCC 0.515 / balanced accuracy 0.756 on this dataset
+```
+
+## Python API
+
+```python
+from urlformat_audit import audit, compare, factorial_variants, format_features, FormatRule
+
+r = audit(urls, labels, name="my data")   # labels: 1 = phishing/malicious, 0 = legitimate
+print(r.format_rule["mcc"], r.nmi_cell_label)
+print(r.class_table)       # pandas DataFrame
+print(r.cell_table.head())
+for f in r.flags: print(f)
+r.to_dict()                # JSON-serialisable
+
+res = compare(train_urls, train_labels, test_urls, test_labels)
+res["format_rule_transfer"]["mcc"], res["share_target_in_unseen_cells"]
+
+factorial_variants("http://example.org/login/")   # [(cell_name, url), ...] x 24
+format_features(urls)                              # per-URL format columns + cell
+```
+
+## How to read the result
+
+- **Format-only rule MCC near 1** (or a NMI near 1): format alone separates the classes, so any
+  model that sees the raw URL can learn the shortcut. Cross-dataset results on this data need
+  format-controlled checks: factorial edits, host-only input, or a scheme-neutral representation.
+- **High in-dataset MCC but low or negative `compare` MCC**: the format cue is specific to that dataset.
+  Expect transfer failures.
+- **Low MCC**: format is not a major shortcut. This says nothing about other artifacts (domains, TLDs,
+  collection time).
+
+The audit is descriptive. It cannot tell you whether a format difference is a collection
+artifact or a real property of phishing URLs; it only tells you that a model can use it.
+
+## Development
+
+```bash
+pip install -e ".[test]"
+pytest
+python -m build        # sdist + wheel in dist/
+```
+
+## Citation
+
+If you use this tool, please cite the accompanying paper:
+*Learning the Template, Not the Threat: Format Shortcuts in Cross-Dataset Phishing URL Detection* (authors and venue to be added).
+
+## License
+
+MIT

urlformat_audit-0.1.0/pyproject.toml

@@ -0,0 +1,34 @@
+[build-system]
+requires = ["hatchling>=1.18"]
+build-backend = "hatchling.build"
+
+[project]
+name = "urlformat-audit"
+version = "0.1.0"
+description = "Detect class-conditional URL formatting (shortcut risk) in labelled phishing/malicious URL datasets"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.9"
+authors = [{ name = "Md Istiak Tanvir - eruddro@gmail.com" }]
+keywords = ["phishing", "url", "dataset audit", "shortcut learning", "machine learning", "security"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Operating System :: OS Independent",
+  "Intended Audience :: Science/Research",
+  "Topic :: Security",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["numpy>=1.21", "pandas>=1.3"]
+
+[project.optional-dependencies]
+test = ["pytest>=7"]
+
+[project.scripts]
+urlformat-audit = "urlformat_audit.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/MIT-Istiak/Learning-the-Template-Not-the-Threat-Format-Shortcuts-in-Cross-Dataset-Phishing-URL-Detection"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/urlformat_audit"]

urlformat_audit-0.1.0/src/urlformat_audit/__init__.py

@@ -0,0 +1,6 @@
+"""urlformat-audit: detect class-conditional URL formatting (shortcut risk) in labelled URL datasets."""
+__version__ = "0.1.0"
+
+from .parse import url_parts, format_features, cell_of  # noqa: F401
+from .audit import audit, compare, binary_metrics, FormatRule, TEMPLATE_REGEX  # noqa: F401
+from .factorial import factorial_variants, variant, expand, CELLS  # noqa: F401

urlformat_audit-0.1.0/src/urlformat_audit/audit.py

@@ -0,0 +1,150 @@
+"""Class-conditional format audit and format-only baselines for labelled URL datasets."""
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass, field
+from typing import Dict, Iterable, Optional, Sequence
+
+import numpy as np
+import pandas as pd
+
+from .parse import format_features
+
+TEMPLATE_REGEX = r"^https://www\.[a-z0-9.-]+$"
+
+
+# ----------------------------------------------------------------------------- metrics
+def binary_metrics(y_true: Sequence[int], y_pred: Sequence[int]) -> Dict[str, float]:
+    """Accuracy, balanced accuracy, F1 of the positive class (1) and Matthews correlation."""
+    y = np.asarray(y_true).astype(int)
+    p = np.asarray(y_pred).astype(int)
+    tp = float(np.sum((p == 1) & (y == 1))); tn = float(np.sum((p == 0) & (y == 0)))
+    fp = float(np.sum((p == 1) & (y == 0))); fn = float(np.sum((p == 0) & (y == 1)))
+    den = math.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))
+    tpr = tp / (tp + fn) if tp + fn else float("nan")
+    tnr = tn / (tn + fp) if tn + fp else float("nan")
+    return {
+        "accuracy": (tp + tn) / max(len(y), 1),
+        "balanced_accuracy": (tpr + tnr) / 2,
+        "f1_positive": 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) else 0.0,
+        "mcc": (tp * tn - fp * fn) / den if den > 0 else 0.0,
+        "tpr": tpr, "fpr": 1 - tnr,
+    }
+
+
+def normalized_mutual_information(a: Sequence, b: Sequence) -> float:
+    """I(A;B) / H(B): share of the label entropy explained by the format cell (0 = none, 1 = all)."""
+    t = pd.crosstab(pd.Series(a, name="a"), pd.Series(b, name="b")).to_numpy(dtype=float)
+    n = t.sum()
+    if n == 0:
+        return float("nan")
+    pab = t / n; pa = pab.sum(1, keepdims=True); pb = pab.sum(0, keepdims=True)
+    nz = pab > 0
+    mi = float(np.sum(pab[nz] * np.log(pab[nz] / (pa @ pb)[nz])))
+    hb = float(-np.sum(pb[pb > 0] * np.log(pb[pb > 0])))
+    return mi / hb if hb > 0 else float("nan")
+
+
+# ----------------------------------------------------------------------------- format-only rule
+@dataclass
+class FormatRule:
+    """Predicts the majority label of each format cell seen in training (fallback: overall majority)."""
+    cell_label: Dict[str, int] = field(default_factory=dict)
+    default: int = 1
+
+    @classmethod
+    def fit(cls, urls: Iterable[str], labels: Sequence[int]) -> "FormatRule":
+        f = format_features(urls)
+        y = np.asarray(labels).astype(int)
+        d = pd.DataFrame({"cell": f["cell"].values, "y": y})
+        maj = d.groupby("cell")["y"].mean()
+        return cls({c: int(v >= 0.5) for c, v in maj.items()}, int(y.mean() >= 0.5))
+
+    def predict(self, urls: Iterable[str]) -> np.ndarray:
+        f = format_features(urls)
+        return np.array([self.cell_label.get(c, self.default) for c in f["cell"]], dtype=int)
+
+
+# ----------------------------------------------------------------------------- audit
+@dataclass
+class AuditResult:
+    name: str
+    n: int
+    class_table: pd.DataFrame          # % of URLs per format property, per class
+    cell_table: pd.DataFrame           # rows per format cell: counts per class and purity
+    format_rule: Dict[str, float]      # in-sample metrics of the majority-per-cell rule
+    template_rule: Dict[str, float]    # metrics of 'legitimate iff URL matches TEMPLATE_REGEX'
+    trivial: Dict[str, float]          # metrics of predicting the positive class everywhere
+    nmi_cell_label: float
+    flags: list
+
+    def to_dict(self) -> dict:
+        return {"name": self.name, "n": self.n, "format_rule": self.format_rule, "template_rule": self.template_rule,
+                "trivial": self.trivial, "nmi_cell_label": self.nmi_cell_label, "flags": self.flags,
+                "class_table": self.class_table.reset_index().to_dict(orient="records"),
+                "cell_table": self.cell_table.reset_index().to_dict(orient="records")}
+
+
+def audit(urls: Sequence[str], labels: Sequence[int], name: str = "dataset", flag_gap: float = 30.0) -> AuditResult:
+    """Audit a labelled URL dataset. ``labels``: 1 = positive (e.g. phishing), 0 = negative (legitimate).
+
+    flag_gap: a format property is flagged when its prevalence differs between the classes by at least this many
+    percentage points.
+    """
+    urls = [str(u) for u in urls]
+    y = np.asarray(labels).astype(int)
+    if set(np.unique(y)) - {0, 1}:
+        raise ValueError("labels must be 0/1 (1 = positive class)")
+    f = format_features(urls)
+    f["label"] = y
+    props = ["scheme=https", "scheme=http", "scheme=none", "scheme=other", "www", "trailing_slash", "path", "query", "port", "ip_host", "uppercase"]
+    rows = {}
+    for cls_, nm in [(0, "negative (0)"), (1, "positive (1)")]:
+        g = f[f.label == cls_]
+        r = {"n": len(g)}
+        for p in props:
+            if p.startswith("scheme="):
+                r[p] = 100 * float((g["scheme"] == p.split("=")[1]).mean()) if len(g) else float("nan")
+            else:
+                r[p] = 100 * float(g[p].mean()) if len(g) else float("nan")
+        r["median_length"] = float(g["length"].median()) if len(g) else float("nan")
+        rows[nm] = r
+    class_table = pd.DataFrame(rows).T
+
+    ct = pd.crosstab(f["cell"], f["label"]).reindex(columns=[0, 1], fill_value=0)
+    ct.columns = ["n_negative", "n_positive"]
+    ct["n"] = ct.sum(axis=1)
+    ct["share_positive"] = ct["n_positive"] / ct["n"]
+    ct["purity"] = np.maximum(ct["share_positive"], 1 - ct["share_positive"])
+    ct = ct.sort_values("n", ascending=False)
+
+    rule = FormatRule.fit(urls, y)
+    fmt = binary_metrics(y, rule.predict(urls))
+    tpl_pred = (~pd.Series(urls).str.strip().str.match(TEMPLATE_REGEX)).astype(int).values
+    tpl = binary_metrics(y, tpl_pred)
+    triv = binary_metrics(y, np.ones(len(y), int))
+    nmi = normalized_mutual_information(f["cell"], y)
+
+    flags = []
+    for p in props:
+        gap = abs(class_table.loc["positive (1)", p] - class_table.loc["negative (0)", p])
+        if gap >= flag_gap:
+            flags.append(f"{p}: {class_table.loc['negative (0)', p]:.1f}% of negatives vs {class_table.loc['positive (1)', p]:.1f}% of positives")
+    if fmt["mcc"] >= 0.5:
+        flags.append(f"format-only rule (4 cues, 24 cells) reaches MCC {fmt['mcc']:.3f} / balanced accuracy {fmt['balanced_accuracy']:.3f} on this dataset")
+    pure = ct[(ct["purity"] >= 0.99) & (ct["n"] >= 100)]
+    if len(pure):
+        flags.append(f"{len(pure)} format cell(s) with >=100 URLs are >=99% one class, covering {100 * pure['n'].sum() / len(y):.1f}% of the data")
+    return AuditResult(name, len(y), class_table, ct, fmt, tpl, triv, nmi, flags)
+
+
+def compare(train_urls: Sequence[str], train_labels: Sequence[int], test_urls: Sequence[str], test_labels: Sequence[int]) -> Dict[str, object]:
+    """Transfer of the format-only rule from one dataset to another, and cell coverage of the target."""
+    rule = FormatRule.fit(train_urls, train_labels)
+    y = np.asarray(test_labels).astype(int)
+    res = binary_metrics(y, rule.predict(test_urls))
+    tr_cells = set(rule.cell_label)
+    te = format_features(test_urls)
+    return {"format_rule_transfer": res, "trivial_target": binary_metrics(y, np.ones(len(y), int)),
+            "share_target_in_unseen_cells": float((~te["cell"].isin(tr_cells)).mean()),
+            "train_cell_majority": rule.cell_label}

urlformat_audit-0.1.0/src/urlformat_audit/cli.py

@@ -0,0 +1,103 @@
+"""Command-line interface: urlformat-audit {audit,compare,factorial}."""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+import numpy as np
+import pandas as pd
+
+from . import __version__
+from .audit import audit, compare
+from .factorial import expand
+from .io import binarize, read_table, resolve_columns
+
+
+def _load(path, url_col, label_col, positive, extra=None):
+    df = read_table(path)
+    u, l = resolve_columns(df, url_col, label_col)
+    urls = df[u].astype(str).tolist()
+    if l is None:
+        if extra is None:
+            raise ValueError(f"{path}: no label column; pass --label-col (or --label for single-class files)")
+        y = np.full(len(urls), int(extra))
+    else:
+        y = binarize(df[l], positive)
+    keep = y >= 0
+    if (~keep).any():
+        print(f"note: {path}: {(~keep).sum()} row(s) without a label were skipped", file=sys.stderr)
+    return [x for x, k in zip(urls, keep) if k], y[keep]
+
+
+def _fmt(d):
+    return "  ".join(f"{k}={v:.3f}" for k, v in d.items() if isinstance(v, float))
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="urlformat-audit", description="Audit class-conditional URL formatting (shortcut risk) in labelled URL datasets.")
+    ap.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    def common(p):
+        p.add_argument("--url-col"); p.add_argument("--label-col")
+        p.add_argument("--positive", help="label value of the positive class (e.g. phishing, bad, 1)")
+
+    a = sub.add_parser("audit", help="format audit of one labelled dataset")
+    a.add_argument("path", nargs="+", help="one CSV/TSV/JSON file, or one file per class with --label")
+    a.add_argument("--label", nargs="+", type=int, help="class (0/1) of each file when files have no label column")
+    a.add_argument("--name", default=None); a.add_argument("--json", help="write the full result as JSON")
+    a.add_argument("--cells", type=int, default=10, help="number of format cells to print")
+    common(a)
+
+    c = sub.add_parser("compare", help="transfer of the format-only rule from TRAIN to TEST")
+    c.add_argument("train"); c.add_argument("test"); common(c)
+
+    f = sub.add_parser("factorial", help="write the 24 factorial format variants of every URL")
+    f.add_argument("path"); f.add_argument("-o", "--out", required=True); f.add_argument("--url-col")
+
+    args = ap.parse_args(argv)
+    if args.cmd == "audit":
+        if len(args.path) > 1 or args.label:
+            labs = args.label or []
+            if len(labs) != len(args.path):
+                ap.error("give one --label per file")
+            parts = [_load(p, args.url_col, args.label_col, args.positive, extra=l) for p, l in zip(args.path, labs)]
+            urls = sum([p[0] for p in parts], []); y = np.concatenate([p[1] for p in parts])
+        else:
+            urls, y = _load(args.path[0], args.url_col, args.label_col, args.positive)
+        r = audit(urls, y, name=args.name or args.path[0])
+        pd.set_option("display.width", 200)
+        print(f"# {r.name}: {r.n:,} URLs ({100 * y.mean():.1f}% positive)\n")
+        print("Format properties (% of URLs per class):")
+        print(r.class_table.round(1).to_string(), "\n")
+        print(f"Format cells (scheme|www|slash|path), top {args.cells}:")
+        print(r.cell_table.head(args.cells).round(3).to_string(), "\n")
+        print("Format-only rule (majority per cell):  ", _fmt(r.format_rule))
+        print("Template rule (legit iff ^https://www.<host>$):", _fmt(r.template_rule))
+        print("Trivial (all positive):                ", _fmt(r.trivial))
+        print(f"Normalized mutual information I(cell;label)/H(label) = {r.nmi_cell_label:.3f}\n")
+        print("Shortcut flags:" if r.flags else "No shortcut flags.")
+        for fl in r.flags:
+            print("  -", fl)
+        if args.json:
+            with open(args.json, "w") as fh:
+                json.dump(r.to_dict(), fh, indent=1, default=float)
+    elif args.cmd == "compare":
+        tu, ty = _load(args.train, args.url_col, args.label_col, args.positive)
+        su, sy = _load(args.test, args.url_col, args.label_col, args.positive)
+        r = compare(tu, ty, su, sy)
+        print("Format-only rule fitted on TRAIN, applied to TEST:", _fmt(r["format_rule_transfer"]))
+        print("Trivial (all positive) on TEST:                   ", _fmt(r["trivial_target"]))
+        print(f"Share of TEST URLs in format cells unseen in TRAIN: {100 * r['share_target_in_unseen_cells']:.1f}%")
+    elif args.cmd == "factorial":
+        df = read_table(args.path)
+        u = args.url_col or next(c for c in ("url", "URL") if c in df.columns)
+        out = expand(df[u].astype(str).tolist())
+        out.to_csv(args.out, index=False)
+        print(f"wrote {len(out):,} rows ({len(df):,} URLs x 24 cells) to {args.out}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

urlformat_audit-0.1.0/src/urlformat_audit/factorial.py

@@ -0,0 +1,41 @@
+"""Factorial format edits: rebuild a URL in every combination of scheme x www x trailing slash x path."""
+from __future__ import annotations
+
+from itertools import product
+from typing import Iterable, List, Tuple
+
+import pandas as pd
+
+from .parse import url_parts
+
+FACTOR_SCHEMES = ("https", "http", "none")
+CELLS: List[Tuple[str, bool, bool, bool]] = [(s, w, sl, p) for s, w, sl, p in product(FACTOR_SCHEMES, (True, False), (True, False), (True, False))]
+
+
+def cell_name(scheme: str, www: bool, slash: bool, path: bool) -> str:
+    return "|".join([scheme, "www" if www else "nowww", "slash" if slash else "noslash", "path" if path else "nopath"])
+
+
+def variant(url: str, scheme: str, www: bool, slash: bool, path: bool) -> str:
+    """Rebuild ``url`` with the given format; the path is everything after the host.
+
+    >>> variant("http://x.org/a/b/", "https", True, False, False)
+    'https://www.x.org'
+    """
+    p = url_parts(url)
+    head = "" if scheme == "none" else f"{scheme}://"
+    body = ("www." if www else "") + p.host + (p.tail if path else "")
+    return head + body + ("/" if slash else "")
+
+
+def factorial_variants(url: str) -> List[Tuple[str, str]]:
+    """All 24 (cell name, edited URL) pairs for one URL."""
+    return [(cell_name(*c), variant(url, *c)) for c in CELLS]
+
+
+def expand(urls: Iterable[str], ids: Iterable = None) -> pd.DataFrame:
+    """Long table (id, cell, url_variant) for scoring with any model."""
+    urls = list(urls)
+    ids = list(ids) if ids is not None else list(range(len(urls)))
+    rows = [(i, c, v) for i, u in zip(ids, urls) for c, v in factorial_variants(u)]
+    return pd.DataFrame(rows, columns=["id", "cell", "url_variant"])

urlformat_audit-0.1.0/src/urlformat_audit/io.py

@@ -0,0 +1,49 @@
+"""Reading labelled URL datasets from CSV/JSON with flexible column and label handling."""
+from __future__ import annotations
+
+import json
+from typing import Optional, Tuple
+
+import numpy as np
+import pandas as pd
+
+URL_COLS = ("url", "URL", "Url", "uri")
+LABEL_COLS = ("label", "Label", "class", "Class", "ClassLabel", "type", "status", "result")
+
+
+def read_table(path: str) -> pd.DataFrame:
+    if path.endswith(".json"):
+        with open(path) as fh:
+            data = json.load(fh)
+        return pd.DataFrame({"url": data}) if isinstance(data, list) else pd.DataFrame(data)
+    sep = "\t" if path.endswith((".tsv", ".txt")) else ","
+    return pd.read_csv(path, sep=sep, dtype=str, keep_default_na=False, encoding="utf-8-sig")
+
+
+def resolve_columns(df: pd.DataFrame, url_col: Optional[str], label_col: Optional[str]) -> Tuple[str, Optional[str]]:
+    u = url_col or next((c for c in URL_COLS if c in df.columns), None)
+    if u is None:
+        raise ValueError(f"no URL column found; pass --url-col (columns: {list(df.columns)})")
+    l = label_col or next((c for c in LABEL_COLS if c in df.columns), None)
+    return u, l
+
+
+def binarize(values: pd.Series, positive: Optional[str]) -> np.ndarray:
+    """Map raw labels to 0/1. ``positive`` names the positive (e.g. phishing) value; empty labels -> -1."""
+    v = values.astype(str).str.strip()
+    nonempty = sorted(set(v) - {""})
+    if positive is None:
+        if set(nonempty) <= {"0", "1", "0.0", "1.0"}:
+            positive = "1"
+        else:
+            raise ValueError(f"cannot infer the positive label from {nonempty[:10]}; pass --positive")
+
+    def norm(x: str) -> str:
+        x = x.lower()
+        try:
+            return str(float(x))
+        except ValueError:
+            return x
+
+    p = norm(positive)
+    return np.array([-1 if x == "" else int(norm(x) == p) for x in v], dtype=int)

urlformat_audit-0.1.0/src/urlformat_audit/parse.py

@@ -0,0 +1,88 @@
+"""Parsing of the format properties of URL strings (no network access, no external parsers)."""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from typing import Iterable, Optional
+
+import pandas as pd
+
+_SCHEME = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*)://")
+_IPV4 = re.compile(r"^\d{1,3}(?:\.\d{1,3}){3}$")
+
+SCHEMES = ("https", "http", "none")  # 'other' schemes (ftp, ...) are mapped to 'other' in features
+
+
+@dataclass(frozen=True)
+class URLParts:
+    scheme: Optional[str]   # lower-cased scheme or None
+    host: str               # host[:port] without a leading 'www.' (case kept)
+    www: bool               # host started with 'www.'
+    tail: str               # everything after the host (path, query, fragment) without a trailing '/'
+    trailing_slash: bool    # the string ended with '/' after something non-empty
+
+    @property
+    def has_path(self) -> bool:
+        return len(self.tail) > 0
+
+
+def _split_host(rest: str) -> tuple[str, str]:
+    cuts = [i for i in (rest.find("/"), rest.find("?"), rest.find("#")) if i >= 0]
+    cut = min(cuts) if cuts else len(rest)
+    return rest[:cut], rest[cut:]
+
+
+def url_parts(url: str) -> URLParts:
+    """Split a URL string into the parts used by the audit.
+
+    >>> url_parts("http://www.Example.org/a/b/")
+    URLParts(scheme='http', host='Example.org', www=True, tail='/a/b', trailing_slash=True)
+    """
+    u = str(url).strip()
+    m = _SCHEME.match(u)
+    scheme = m.group(1).lower() if m else None
+    rest = u[m.end():] if m else u
+    host, tail = _split_host(rest)
+    www = host.lower().startswith("www.")
+    if www:
+        host = host[4:]
+    slash = (host + tail).endswith("/") and len((host + tail).rstrip("/")) > 0
+    if slash:
+        tail = tail.rstrip("/")
+    return URLParts(scheme, host, www, tail, slash)
+
+
+def scheme_group(scheme: Optional[str]) -> str:
+    if scheme is None:
+        return "none"
+    return scheme if scheme in ("http", "https") else "other"
+
+
+def cell_of(url: str) -> str:
+    """Format cell 'scheme|www|slash|path' used by the audit and the factorial edits."""
+    p = url_parts(url)
+    return "|".join([scheme_group(p.scheme), "www" if p.www else "nowww",
+                     "slash" if p.trailing_slash else "noslash", "path" if p.has_path else "nopath"])
+
+
+def format_features(urls: Iterable[str]) -> pd.DataFrame:
+    """One row per URL with boolean/numeric format features and its format cell."""
+    rows = []
+    for u in urls:
+        s = str(u).strip()
+        p = url_parts(s)
+        hostname = p.host.split(":")[0]
+        rows.append({
+            "scheme": scheme_group(p.scheme),
+            "www": p.www,
+            "trailing_slash": p.trailing_slash,
+            "path": p.has_path,
+            "query": "?" in p.tail,
+            "port": bool(re.search(r":\d+$", p.host)),
+            "ip_host": bool(_IPV4.match(hostname)) or hostname.startswith("["),
+            "uppercase": any(c.isupper() for c in s),
+            "length": len(s),
+            "cell": "|".join([scheme_group(p.scheme), "www" if p.www else "nowww",
+                              "slash" if p.trailing_slash else "noslash", "path" if p.has_path else "nopath"]),
+        })
+    return pd.DataFrame(rows)

urlformat_audit-0.1.0/tests/test_basic.py

@@ -0,0 +1,52 @@
+import numpy as np
+
+from urlformat_audit import audit, compare, factorial_variants, url_parts, variant, FormatRule, format_features
+from urlformat_audit.cli import main
+
+
+def test_url_parts():
+    p = url_parts("http://www.Example.org/a/b/")
+    assert (p.scheme, p.host, p.www, p.tail, p.trailing_slash) == ("http", "Example.org", True, "/a/b", True)
+    p = url_parts("example.org")
+    assert p.scheme is None and not p.has_path and not p.trailing_slash
+    assert url_parts("https://a.com/").trailing_slash and not url_parts("https://a.com/").has_path
+
+
+def test_factorial():
+    v = factorial_variants("http://x.org/a/")
+    assert len(v) == 24 and len({c for c, _ in v}) == 24
+    assert variant("http://x.org/a/b/", "https", True, False, False) == "https://www.x.org"
+    assert variant("x.org", "none", False, True, True) == "x.org/"
+
+
+def test_audit_detects_template_shortcut():
+    legit = [f"https://www.site{i}.com" for i in range(300)]
+    phish = [f"http://bad{i}.xyz/login/" for i in range(300)]
+    r = audit(legit + phish, [0] * 300 + [1] * 300)
+    assert r.format_rule["mcc"] == 1.0 and r.template_rule["mcc"] == 1.0
+    assert any("format-only rule" in f for f in r.flags)
+    assert abs(r.nmi_cell_label - 1.0) < 1e-9
+
+
+def test_compare_and_rule():
+    tr = ["https://www.a.com", "http://b.com/x/"] * 50
+    ytr = [0, 1] * 50
+    te = ["http://www.c.edu/", "https://www.d.org"]
+    res = compare(tr, ytr, te, [0, 0])
+    assert 0 <= res["share_target_in_unseen_cells"] <= 1
+    assert list(FormatRule.fit(tr, ytr).predict(["https://www.z.com"])) == [0]
+
+
+def test_features_columns():
+    f = format_features(["https://1.2.3.4:8080/x?q=1"])
+    assert bool(f.ip_host[0]) and bool(f.port[0]) and bool(f["query"][0])
+
+
+def test_cli(tmp_path, capsys):
+    p = tmp_path / "d.csv"
+    p.write_text("url,label\nhttps://www.a.com,good\nhttp://b.com/x/,bad\nhttps://www.c.com,good\nhttp://d.com/,bad\n")
+    assert main(["audit", str(p), "--positive", "bad"]) == 0
+    assert "Format-only rule" in capsys.readouterr().out
+    out = tmp_path / "f.csv"
+    assert main(["factorial", str(p), "-o", str(out)]) == 0
+    assert out.read_text().count("\n") == 4 * 24 + 1