sameness 0.1.0__tar.gz

sameness-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tsuruta Lab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sameness-0.1.0/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.4
+Name: sameness
+Version: 0.1.0
+Summary: Do the pages of one site look like one template? Compare content skeletons, shells, fonts, palettes and copy across pages. No dependencies.
+Author: Tsuruta Lab
+License: MIT
+Project-URL: Homepage, https://github.com/tsurutanmen/sameness
+Project-URL: Issues, https://github.com/tsurutanmen/sameness/issues
+Keywords: design,ai slop,template,website audit,html,consistency
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: WWW/HTTP :: Site Management
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Dynamic: license-file
+
+# sameness
+
+Do the pages of one site look like one template?
+
+Detectors for the AI look score one page at a time: the purple gradient, Inter, three feature cards.
+But when people are asked what gives a machine-made site away, the two answers at the top are not
+features. In a Reddit corpus of 3,033 on-topic comments, "screams AI" (6.4%) and "they all look the
+same" (6.1%) each beat every specific feature; purple, gradients, animations and glass together add
+up to about 7%. Sameness is a property of a *set* of pages. `sameness` compares the pages of one
+site with each other.
+
+```
+pip install git+https://github.com/tsurutanmen/sameness
+```
+
+No dependencies. Python 3.9 or later.
+
+## Thirty seconds
+
+```
+$ sameness tests/fixtures/clone_*.html
+
+4 pages
+
+sameness    1.00   mean similarity of the content skeleton across page pairs (1 = one template)
+shells      1      distinct header/nav/footer structures (1 = one site)
+palette     1.00   mean overlap of hex colours between pages (1 = one palette)
+
+near-clones (content skeleton >= 90% similar):
+  clone_0.html  ~  clone_1.html  ~  clone_2.html  ~  clone_3.html
+
+same meta description on several pages:
+    4 page(s)  "One platform to elevate your workflow."
+
+headings that appear on several pages (outside header/nav/footer):
+    4x  Ready?
+
+page           words emoji grad   sym  repeated blocks
+clone_0.html      36     0    0   3x3  cardx3
+...
+note: pages share one content skeleton; a reader will see one template with the words swapped
+note: 4 pages reuse a meta description; search results will show the same sentence for each
+```
+
+Or crawl a live site, same host only, linked stylesheets included:
+
+```
+sameness --site https://example.com --max 20
+```
+
+## What it measures
+
+| line | meaning |
+|---|---|
+| `sameness` | Mean similarity of the *content* skeleton (the tag sequence of the page with header, nav and footer removed) over all page pairs. 1.00 means every page is the same template. |
+| `near-clones` | Groups of pages whose content skeletons are at least 90% similar. Pages of one kind (articles, product pages) are expected to share a skeleton; the question is whether pages of different kinds do too. |
+| `shells` | Number of distinct header/nav/footer structures. A site has one. When it has four, the pages do not read as one place, which is the opposite failure and just as visible. |
+| `font sets`, `palette` | Fonts declared and hex colours used, per page and their overlap across pages. |
+| `same meta description`, `same <title>` | Copy reused across pages, usually a default that was never replaced. |
+| `sym` | Perfectly symmetric blocks on a page: `3x3` means three lists of exactly three items. |
+| `--pairs` | The full pairwise table: skeleton similarity, shared headings, shared repeated block classes. |
+
+Everything comes from the markup and the stylesheet text. Nothing is rendered, so layout that only
+exists after JavaScript runs is not seen.
+
+## Two real sites
+
+A 15-page site of ours that had been built in four sittings. No page-level detector complained; the
+Tell Score was A or B on every page. `sameness` reported:
+
+```
+sameness    0.26
+shells      8      distinct header/nav/footer structures (1 = one site)
+palette     0.26
+font sets:  5
+note: 8 different shells on one site: the pages do not read as one place
+```
+
+That was the finding a reader had already made ("it looks like four different sites"), now as a
+number. The other direction, a small research site with one shell:
+
+```
+sameness    0.51
+shells      1
+palette     1.00
+near-clones: the six research notes share one article skeleton
+same meta description on several pages:  2 page(s)  "aaaaaaaaaaaaaa"
+```
+
+The near-clones are articles and are supposed to look alike. The placeholder description was real
+and had been missed.
+
+## Python
+
+```python
+import sameness
+
+pages = [sameness.extract(html, name=url) for url, html in docs]
+rep = sameness.compare(pages)
+print(rep)
+rep.to_dict()                      # everything, JSON-serialisable
+
+for p in rep.pairs:                # pairwise numbers
+    print(p.a, p.b, p.skeleton, p.headings, p.classes)
+
+for url, html, css in sameness.crawl("https://example.com", max_pages=20):
+    ...
+```
+
+## Claude Code skill
+
+`skill/sameness/SKILL.md` tells Claude Code to run this after building or editing more than one page
+of a site, and how to read the result. Install by copying the folder:
+
+```
+cp -r skill/sameness ~/.claude/skills/sameness
+```
+
+## Related tools
+
+Per-page detectors, which this complements rather than replaces:
+[ai-design-tells](https://github.com/hankimis/ai-design-tells) (Tell Score, 27 tells),
+[avoid-ai-design](https://github.com/funboy322/avoid-ai-design) (a rewrite skill),
+[vibecoded-audit](https://github.com/KreshBack/vibecoded-audit) (49 tells, crawls but scores per page).
+The Reddit corpus is from
+[vibecoded-design-tells](https://github.com/JCarterJohnson/vibecoded-design-tells).
+
+## License
+
+MIT.

sameness-0.1.0/README.md

@@ -0,0 +1,130 @@
+# sameness
+
+Do the pages of one site look like one template?
+
+Detectors for the AI look score one page at a time: the purple gradient, Inter, three feature cards.
+But when people are asked what gives a machine-made site away, the two answers at the top are not
+features. In a Reddit corpus of 3,033 on-topic comments, "screams AI" (6.4%) and "they all look the
+same" (6.1%) each beat every specific feature; purple, gradients, animations and glass together add
+up to about 7%. Sameness is a property of a *set* of pages. `sameness` compares the pages of one
+site with each other.
+
+```
+pip install git+https://github.com/tsurutanmen/sameness
+```
+
+No dependencies. Python 3.9 or later.
+
+## Thirty seconds
+
+```
+$ sameness tests/fixtures/clone_*.html
+
+4 pages
+
+sameness    1.00   mean similarity of the content skeleton across page pairs (1 = one template)
+shells      1      distinct header/nav/footer structures (1 = one site)
+palette     1.00   mean overlap of hex colours between pages (1 = one palette)
+
+near-clones (content skeleton >= 90% similar):
+  clone_0.html  ~  clone_1.html  ~  clone_2.html  ~  clone_3.html
+
+same meta description on several pages:
+    4 page(s)  "One platform to elevate your workflow."
+
+headings that appear on several pages (outside header/nav/footer):
+    4x  Ready?
+
+page           words emoji grad   sym  repeated blocks
+clone_0.html      36     0    0   3x3  cardx3
+...
+note: pages share one content skeleton; a reader will see one template with the words swapped
+note: 4 pages reuse a meta description; search results will show the same sentence for each
+```
+
+Or crawl a live site, same host only, linked stylesheets included:
+
+```
+sameness --site https://example.com --max 20
+```
+
+## What it measures
+
+| line | meaning |
+|---|---|
+| `sameness` | Mean similarity of the *content* skeleton (the tag sequence of the page with header, nav and footer removed) over all page pairs. 1.00 means every page is the same template. |
+| `near-clones` | Groups of pages whose content skeletons are at least 90% similar. Pages of one kind (articles, product pages) are expected to share a skeleton; the question is whether pages of different kinds do too. |
+| `shells` | Number of distinct header/nav/footer structures. A site has one. When it has four, the pages do not read as one place, which is the opposite failure and just as visible. |
+| `font sets`, `palette` | Fonts declared and hex colours used, per page and their overlap across pages. |
+| `same meta description`, `same <title>` | Copy reused across pages, usually a default that was never replaced. |
+| `sym` | Perfectly symmetric blocks on a page: `3x3` means three lists of exactly three items. |
+| `--pairs` | The full pairwise table: skeleton similarity, shared headings, shared repeated block classes. |
+
+Everything comes from the markup and the stylesheet text. Nothing is rendered, so layout that only
+exists after JavaScript runs is not seen.
+
+## Two real sites
+
+A 15-page site of ours that had been built in four sittings. No page-level detector complained; the
+Tell Score was A or B on every page. `sameness` reported:
+
+```
+sameness    0.26
+shells      8      distinct header/nav/footer structures (1 = one site)
+palette     0.26
+font sets:  5
+note: 8 different shells on one site: the pages do not read as one place
+```
+
+That was the finding a reader had already made ("it looks like four different sites"), now as a
+number. The other direction, a small research site with one shell:
+
+```
+sameness    0.51
+shells      1
+palette     1.00
+near-clones: the six research notes share one article skeleton
+same meta description on several pages:  2 page(s)  "aaaaaaaaaaaaaa"
+```
+
+The near-clones are articles and are supposed to look alike. The placeholder description was real
+and had been missed.
+
+## Python
+
+```python
+import sameness
+
+pages = [sameness.extract(html, name=url) for url, html in docs]
+rep = sameness.compare(pages)
+print(rep)
+rep.to_dict()                      # everything, JSON-serialisable
+
+for p in rep.pairs:                # pairwise numbers
+    print(p.a, p.b, p.skeleton, p.headings, p.classes)
+
+for url, html, css in sameness.crawl("https://example.com", max_pages=20):
+    ...
+```
+
+## Claude Code skill
+
+`skill/sameness/SKILL.md` tells Claude Code to run this after building or editing more than one page
+of a site, and how to read the result. Install by copying the folder:
+
+```
+cp -r skill/sameness ~/.claude/skills/sameness
+```
+
+## Related tools
+
+Per-page detectors, which this complements rather than replaces:
+[ai-design-tells](https://github.com/hankimis/ai-design-tells) (Tell Score, 27 tells),
+[avoid-ai-design](https://github.com/funboy322/avoid-ai-design) (a rewrite skill),
+[vibecoded-audit](https://github.com/KreshBack/vibecoded-audit) (49 tells, crawls but scores per page).
+The Reddit corpus is from
+[vibecoded-design-tells](https://github.com/JCarterJohnson/vibecoded-design-tells).
+
+## License
+
+MIT.

sameness-0.1.0/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sameness"
+version = "0.1.0"
+description = "Do the pages of one site look like one template? Compare content skeletons, shells, fonts, palettes and copy across pages. No dependencies."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Tsuruta Lab" }]
+keywords = ["design", "ai slop", "template", "website audit", "html", "consistency"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Internet :: WWW/HTTP :: Site Management",
+]
+dependencies = []
+
+[project.optional-dependencies]
+test = ["pytest>=7"]
+
+[project.urls]
+Homepage = "https://github.com/tsurutanmen/sameness"
+Issues = "https://github.com/tsurutanmen/sameness/issues"
+
+[project.scripts]
+sameness = "sameness.cli:main"
+
+[tool.setuptools.packages.find]
+include = ["sameness*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

sameness-0.1.0/sameness/__init__.py

@@ -0,0 +1,18 @@
+"""sameness: do the pages of one site look like one template?
+
+Per-page detectors catch the purple gradient and the Inter font.  The two
+things people actually say about machine-made sites are "it screams AI" and
+"they all look the same", and those are properties of a *set* of pages.
+This package compares pages of one site with each other.
+
+    >>> import sameness
+    >>> pages = [sameness.extract(html, name) for name, html in docs]
+    >>> print(sameness.compare(pages))
+"""
+
+from .extract import extract, Page
+from .compare import compare, Report, skeleton_similarity
+from .fetch import crawl
+
+__version__ = "0.1.0"
+__all__ = ["extract", "Page", "compare", "Report", "skeleton_similarity", "crawl"]

sameness-0.1.0/sameness/cli.py

@@ -0,0 +1,77 @@
+"""sameness PAGE.html PAGE.html ...   |   sameness --site https://example.com --max 20"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import os
+import sys
+import urllib.parse
+
+from .extract import extract
+from .compare import compare
+from .fetch import crawl, read_file, load_local_css
+
+
+def _short(url: str) -> str:
+    u = urllib.parse.urlparse(url)
+    path = u.path or "/"
+    if u.query:
+        path += "?" + u.query[:80]
+    return path
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(
+        prog="sameness",
+        description="Do the pages of one site look like one template? Compare content skeletons, shells, fonts, palettes and copy across pages.")
+    ap.add_argument("pages", nargs="*", help="HTML files (globs allowed)")
+    ap.add_argument("--site", help="crawl this URL, same host only")
+    ap.add_argument("--max", type=int, default=20, help="pages to crawl (default 20)")
+    ap.add_argument("--no-css", action="store_true", help="do not fetch linked stylesheets")
+    ap.add_argument("--json", help="write the report as JSON here")
+    ap.add_argument("--pairs", action="store_true", help="print the pairwise similarity table")
+    a = ap.parse_args(argv)
+    try:                                   # Windows consoles default to a legacy code page
+        sys.stdout.reconfigure(encoding="utf-8", errors="replace")
+    except (AttributeError, ValueError):
+        pass
+
+    pages = []
+    if a.site:
+        for url, html, css in crawl(a.site, max_pages=a.max, fetch_css=not a.no_css):
+            pages.append(extract(html, name=_short(url), extra_css=css))
+    files = []
+    for pat in a.pages:
+        files.extend(sorted(glob.glob(pat)) or [pat])
+    for f in files:
+        if not os.path.isfile(f):
+            print(f"  skip {f}: not a file", file=sys.stderr)
+            continue
+        name, html = read_file(f)
+        p = extract(html, name=name)
+        if not a.no_css and p.stylesheet_hrefs:
+            css = load_local_css(f, p.stylesheet_hrefs)
+            if css:
+                p = extract(html, name=name, extra_css=css)
+        pages.append(p)
+    if not pages:
+        ap.error("give HTML files or --site URL")
+
+    rep = compare(pages)
+    print(rep)
+    if a.pairs:
+        print()
+        print(f"{'a':24s} {'b':24s} {'skeleton':>8s} {'headings':>8s} {'blocks':>7s}")
+        for p in sorted(rep.pairs, key=lambda p: -p.skeleton):
+            print(f"{p.a[:24]:24s} {p.b[:24]:24s} {p.skeleton:8.2f} {p.headings:8.2f} {p.classes:7.2f}")
+    if a.json:
+        with open(a.json, "w", encoding="utf-8") as f:
+            json.dump(rep.to_dict(), f, ensure_ascii=False, indent=1)
+        print(f"\nwrote {a.json}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

sameness-0.1.0/sameness/compare.py

@@ -0,0 +1,213 @@
+"""Compare pages of one site: do they look like one template, and is the shell one shell?"""
+
+from __future__ import annotations
+
+import difflib
+import math
+import re
+from collections import Counter
+from dataclasses import dataclass, field
+from typing import Dict, List, Sequence, Tuple
+
+from .extract import Page
+
+CLONE_THRESHOLD = 0.9
+
+
+def skeleton_similarity(a: Sequence[str], b: Sequence[str]) -> float:
+    """Sequence similarity of two tag skeletons, 0..1."""
+    if not a and not b:
+        return 1.0
+    return difflib.SequenceMatcher(None, list(a), list(b), autojunk=False).ratio()
+
+
+def jaccard(a, b) -> float:
+    a, b = set(a), set(b)
+    if not a and not b:
+        return 1.0
+    return len(a & b) / len(a | b)
+
+
+def symmetry(list_sizes: Sequence[int], class_counts: Dict[str, int]) -> Dict[str, object]:
+    """Perfectly symmetric blocks: N cards each with the same number of bullets, 3x3 and so on."""
+    cnt = Counter(list_sizes)
+    equal_runs = [(k, v) for k, v in cnt.items() if v >= 3 and k >= 2]     # v lists of exactly k items
+    repeated = {k: v for k, v in class_counts.items() if v >= 3 and 3 <= v <= 12}
+    return {"equal_list_runs": sorted(equal_runs), "repeated_blocks": repeated}
+
+
+@dataclass
+class Pair:
+    a: str
+    b: str
+    skeleton: float
+    headings: float
+    classes: float
+
+
+@dataclass
+class Report:
+    pages: List[str]
+    pairs: List[Pair] = field(default_factory=list)
+    clone_groups: List[List[str]] = field(default_factory=list)
+    sameness: float = 0.0                  # mean pairwise content-skeleton similarity
+    shell_count: int = 0                   # distinct header/nav/footer skeletons
+    shells: Dict[str, List[str]] = field(default_factory=dict)
+    font_sets: Dict[str, List[str]] = field(default_factory=dict)
+    palette_overlap: float = 1.0           # mean pairwise jaccard of hex colours
+    duplicate_descriptions: List[Tuple[str, List[str]]] = field(default_factory=list)
+    duplicate_titles: List[Tuple[str, List[str]]] = field(default_factory=list)
+    shared_headings: List[Tuple[str, int]] = field(default_factory=list)
+    per_page: Dict[str, Dict[str, object]] = field(default_factory=dict)
+    notes: List[str] = field(default_factory=list)
+
+    def to_dict(self) -> dict:
+        d = self.__dict__.copy()
+        d["pairs"] = [p.__dict__ for p in self.pairs]
+        return d
+
+    def __str__(self) -> str:
+        n = len(self.pages)
+        L = [f"{n} pages", ""]
+        L.append(f"sameness    {self.sameness:.2f}   mean similarity of the content skeleton across page pairs (1 = one template)")
+        L.append(f"shells      {self.shell_count}      distinct header/nav/footer structures (1 = one site)")
+        L.append(f"palette     {self.palette_overlap:.2f}   mean overlap of hex colours between pages (1 = one palette)")
+        if self.clone_groups:
+            L.append("")
+            L.append(f"near-clones (content skeleton >= {CLONE_THRESHOLD:.0%} similar):")
+            for g in self.clone_groups:
+                L.append("  " + "  ~  ".join(g))
+        if self.shell_count > 1:
+            L.append("")
+            L.append("shells:")
+            for key, names in self.shells.items():
+                L.append(f"  {len(names):3d} page(s)  {', '.join(names[:6])}{' ...' if len(names) > 6 else ''}")
+        if len(self.font_sets) > 1:
+            L.append("")
+            L.append("font sets:")
+            for key, names in self.font_sets.items():
+                L.append(f"  {key or '(none declared)':40s} {', '.join(names[:5])}{' ...' if len(names) > 5 else ''}")
+        if self.duplicate_descriptions:
+            L.append("")
+            L.append("same meta description on several pages:")
+            for desc, names in self.duplicate_descriptions:
+                L.append(f"  {len(names):3d} page(s)  \"{desc[:70]}\"")
+        if self.duplicate_titles:
+            L.append("")
+            L.append("same <title> on several pages:")
+            for t, names in self.duplicate_titles:
+                L.append(f"  {len(names):3d} page(s)  \"{t[:70]}\"")
+        if self.shared_headings:
+            L.append("")
+            L.append("headings that appear on several pages (outside header/nav/footer):")
+            for h, c in self.shared_headings[:12]:
+                L.append(f"  {c:3d}x  {h[:70]}")
+        L.append("")
+        w = min(48, max(len(n) for n in self.pages) + 1)
+        L.append(f"{'page':{w}s} {'words':>6s} {'emoji':>5s} {'grad':>4s} {'sym':>5s}  repeated blocks")
+        for name in self.pages:
+            pp = self.per_page[name]
+            sym = pp["symmetry"]
+            blocks = ", ".join(f"{k[:24]}x{v}" for k, v in list(sym["repeated_blocks"].items())[:3])
+            runs = " ".join(f"{v}x{k}" for k, v in sym["equal_list_runs"][:2])
+            L.append(f"{name[:w]:{w}s} {pp['words']:6d} {pp['emoji']:5d} {pp['gradients']:4d} {runs:>5s}  {blocks}")
+        if self.notes:
+            L.append("")
+            for w in self.notes:
+                L.append(f"note: {w}")
+        return "\n".join(L)
+
+
+def _group_duplicates(items: List[Tuple[str, str]]) -> List[Tuple[str, List[str]]]:
+    by: Dict[str, List[str]] = {}
+    for name, val in items:
+        if val:
+            by.setdefault(val, []).append(name)
+    return sorted([(v, n) for v, n in by.items() if len(n) > 1], key=lambda x: -len(x[1]))
+
+
+def compare(pages: Sequence[Page]) -> Report:
+    names = [p.name for p in pages]
+    rep = Report(pages=names)
+    if len(pages) < 2:
+        rep.notes.append("need at least two pages to measure sameness")
+        for p in pages:
+            rep.per_page[p.name] = _per_page(p)
+        return rep
+
+    sims, pal = [], []
+    for i in range(len(pages)):
+        for j in range(i + 1, len(pages)):
+            a, b = pages[i], pages[j]
+            s = skeleton_similarity(a.skeleton, b.skeleton)
+            h = jaccard([t for _, t in a.headings], [t for _, t in b.headings])
+            c = jaccard(a.repeated_classes().keys(), b.repeated_classes().keys())
+            rep.pairs.append(Pair(a.name, b.name, s, h, c))
+            sims.append(s)
+            pal.append(jaccard(a.colors, b.colors) if (a.colors or b.colors) else 1.0)
+    rep.sameness = float(sum(sims) / len(sims))
+    rep.palette_overlap = float(sum(pal) / len(pal))
+
+    # clone groups: union-find over pairs above threshold
+    parent = {n: n for n in names}
+
+    def find(x):
+        while parent[x] != x:
+            parent[x] = parent[parent[x]]
+            x = parent[x]
+        return x
+
+    for p in rep.pairs:
+        if p.skeleton >= CLONE_THRESHOLD:
+            parent[find(p.a)] = find(p.b)
+    groups: Dict[str, List[str]] = {}
+    for n in names:
+        groups.setdefault(find(n), []).append(n)
+    rep.clone_groups = sorted([g for g in groups.values() if len(g) > 1], key=lambda g: -len(g))
+
+    # shells
+    shells: Dict[str, List[str]] = {}
+    for p in pages:
+        shells.setdefault(" ".join(p.shell), []).append(p.name)
+    rep.shells = dict(sorted(shells.items(), key=lambda kv: -len(kv[1])))
+    rep.shell_count = len(shells)
+
+    fonts: Dict[str, List[str]] = {}
+    for p in pages:
+        fonts.setdefault(", ".join(p.fonts), []).append(p.name)
+    rep.font_sets = dict(sorted(fonts.items(), key=lambda kv: -len(kv[1])))
+
+    rep.duplicate_descriptions = _group_duplicates([(p.name, p.description) for p in pages])
+    rep.duplicate_titles = _group_duplicates([(p.name, p.title) for p in pages])
+    hc = Counter()
+    for p in pages:
+        for _, t in set(p.headings):
+            hc[t] += 1
+    rep.shared_headings = [(t, c) for t, c in hc.most_common() if c > 1]
+
+    for p in pages:
+        rep.per_page[p.name] = _per_page(p)
+
+    # notes
+    if rep.sameness >= 0.85:
+        rep.notes.append("pages share one content skeleton; a reader will see one template with the words swapped")
+    if rep.clone_groups:
+        rep.notes.append("near-clones: pages of one kind (articles, product pages) are expected to share a skeleton; "
+                         "the question is whether pages of different kinds do too")
+    if rep.shell_count > 1:
+        rep.notes.append(f"{rep.shell_count} different shells on one site: the pages do not read as one place")
+    if len(rep.font_sets) > 1:
+        rep.notes.append(f"{len(rep.font_sets)} different font sets; one is usual")
+    if rep.duplicate_descriptions:
+        n = sum(len(v) for _, v in rep.duplicate_descriptions)
+        rep.notes.append(f"{n} pages reuse a meta description; search results will show the same sentence for each")
+    return rep
+
+
+def _per_page(p: Page) -> Dict[str, object]:
+    return {
+        "title": p.title, "words": p.word_count, "emoji": p.emoji, "gradients": p.gradients,
+        "headings": len(p.headings), "skeleton_len": len(p.skeleton),
+        "symmetry": symmetry(p.list_sizes, p.class_counts),
+        "fonts": p.fonts, "n_colors": len(p.colors),
+    }

sameness-0.1.0/sameness/extract.py

@@ -0,0 +1,175 @@
+"""Turn one HTML document into the features the comparison needs.
+
+No rendering, no dependencies.  Everything comes from the markup and the
+inline or linked stylesheet text the caller hands over.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from html.parser import HTMLParser
+from typing import Dict, List, Optional, Tuple
+
+SKELETON_TAGS = {
+    "header", "nav", "main", "section", "article", "aside", "footer",
+    "h1", "h2", "h3", "h4", "h5", "h6", "ul", "ol", "table", "form", "img",
+    "button", "figure", "blockquote", "pre",
+}
+SHELL_TAGS = {"header", "nav", "footer"}
+HEADING_TAGS = {"h1", "h2", "h3", "h4", "h5", "h6"}
+SKIP_TEXT = {"script", "style", "noscript", "template", "svg"}
+
+EMOJI = re.compile(
+    "[\U0001F000-\U0001FAFF\U00002600-\U000027BF\U00002B00-\U00002BFF\U0001F1E6-\U0001F1FF\U0000FE0F]"
+)
+HEX = re.compile(r"#(?:[0-9a-fA-F]{3}){1,2}\b")
+FONT = re.compile(r"font-family\s*:\s*([^;}]+)", re.I)
+RADIUS = re.compile(r"border-radius\s*:\s*([0-9.]+)(px|rem|em|%)", re.I)
+GRID = re.compile(r"grid-template-columns\s*:\s*([^;}]+)", re.I)
+GRADIENT = re.compile(r"(linear|radial|conic)-gradient", re.I)
+
+
+@dataclass
+class Page:
+    name: str
+    title: str = ""
+    description: str = ""
+    lang: str = ""
+    skeleton: List[str] = field(default_factory=list)          # tag sequence, content region
+    shell: List[str] = field(default_factory=list)             # tag sequence inside header/nav/footer
+    headings: List[Tuple[int, str]] = field(default_factory=list)
+    list_sizes: List[int] = field(default_factory=list)        # items per ul/ol in content
+    class_counts: Dict[str, int] = field(default_factory=dict) # class value -> occurrences
+    text: str = ""
+    word_count: int = 0
+    emoji: int = 0
+    css_text: str = ""
+    fonts: List[str] = field(default_factory=list)
+    colors: List[str] = field(default_factory=list)
+    radii: List[str] = field(default_factory=list)
+    grids: List[str] = field(default_factory=list)
+    gradients: int = 0
+    stylesheet_hrefs: List[str] = field(default_factory=list)
+    n_tags: int = 0
+
+    def repeated_classes(self, min_count: int = 3) -> Dict[str, int]:
+        return {k: v for k, v in self.class_counts.items() if v >= min_count}
+
+
+class _Parser(HTMLParser):
+    def __init__(self):
+        super().__init__(convert_charrefs=True)
+        self.page = Page(name="")
+        self._stack: List[str] = []
+        self._skip = 0
+        self._in_title = False
+        self._heading: Optional[Tuple[int, List[str]]] = None
+        self._list_stack: List[int] = []
+        self._text: List[str] = []
+        self._style: List[str] = []
+        self._in_style = False
+
+    # --- helpers
+    def _in_shell(self) -> bool:
+        return any(t in SHELL_TAGS for t in self._stack)
+
+    def handle_starttag(self, tag, attrs):
+        a = dict(attrs)
+        self.page.n_tags += 1
+        if tag == "html" and a.get("lang"):
+            self.page.lang = a["lang"]
+        if tag == "meta" and (a.get("name") or "").lower() == "description":
+            self.page.description = (a.get("content") or "").strip()
+        if tag == "link" and "stylesheet" in (a.get("rel") or "") and a.get("href"):
+            self.page.stylesheet_hrefs.append(a["href"])
+        if tag == "title":
+            self._in_title = True
+        if tag == "style":
+            self._in_style = True
+        if tag in SKIP_TEXT:
+            self._skip += 1
+        if a.get("style"):
+            self._style.append(a["style"])
+        cls = (a.get("class") or "").strip()
+        if cls and not self._in_shell():
+            self.page.class_counts[cls] = self.page.class_counts.get(cls, 0) + 1
+        if tag in SKELETON_TAGS:
+            (self.page.shell if self._in_shell() or tag in SHELL_TAGS else self.page.skeleton).append(tag)
+        if tag in HEADING_TAGS and not self._in_shell():
+            self._heading = (int(tag[1]), [])
+        if tag in ("ul", "ol") and not self._in_shell():
+            self._list_stack.append(0)
+        if tag == "li" and self._list_stack:
+            self._list_stack[-1] += 1
+        self._stack.append(tag)
+
+    def handle_endtag(self, tag):
+        if tag == "title":
+            self._in_title = False
+        if tag == "style":
+            self._in_style = False
+        if tag in SKIP_TEXT and self._skip:
+            self._skip -= 1
+        if tag in HEADING_TAGS and self._heading is not None:
+            lvl, parts = self._heading
+            txt = re.sub(r"\s+", " ", "".join(parts)).strip()
+            if txt:
+                self.page.headings.append((lvl, txt))
+            self._heading = None
+        if tag in ("ul", "ol") and self._list_stack:
+            n = self._list_stack.pop()
+            if n:
+                self.page.list_sizes.append(n)
+        # pop to the matching open tag if present
+        if tag in self._stack:
+            while self._stack and self._stack.pop() != tag:
+                pass
+
+    def handle_data(self, data):
+        if self._in_title:
+            self.page.title += data
+        if self._in_style:
+            self._style.append(data)
+            return
+        if self._skip:
+            return
+        if self._heading is not None:
+            self._heading[1].append(data)
+        if not self._in_shell():
+            self._text.append(data)
+
+    def finish(self, name: str, extra_css: str = "") -> Page:
+        p = self.page
+        p.name = name
+        p.title = re.sub(r"\s+", " ", p.title).strip()
+        p.text = re.sub(r"\s+", " ", " ".join(self._text)).strip()
+        p.word_count = _count_words(p.text)
+        p.emoji = len(EMOJI.findall(p.text))
+        css = "\n".join(self._style) + "\n" + extra_css
+        p.css_text = css
+        p.fonts = sorted({_first_font(m) for m in FONT.findall(css)} - {""})
+        p.colors = sorted({c.lower() for c in HEX.findall(css)})
+        p.radii = sorted({f"{v}{u}" for v, u in RADIUS.findall(css)})
+        p.grids = sorted({re.sub(r"\s+", " ", g).strip()[:60] for g in GRID.findall(css)})
+        p.gradients = len(GRADIENT.findall(css))
+        return p
+
+
+def _first_font(decl: str) -> str:
+    first = decl.split(",")[0].strip().strip("'\"")
+    return first if first and not first.startswith("var(") else ""
+
+
+def _count_words(text: str) -> int:
+    """Words for space-separated scripts; characters for CJK, where spaces are rare."""
+    cjk = len(re.findall(r"[぀-ヿ㐀-鿿]", text))
+    latin = len(re.findall(r"[A-Za-z0-9]+", text))
+    return cjk + latin
+
+
+def extract(html: str, name: str = "", extra_css: str = "") -> Page:
+    p = _Parser()
+    p.feed(html)
+    p.close()
+    return p.finish(name, extra_css)

sameness-0.1.0/sameness/fetch.py

@@ -0,0 +1,107 @@
+"""Read pages from files or from a site.  Standard library only."""
+
+from __future__ import annotations
+
+import os
+import re
+import sys
+import urllib.parse
+import urllib.request
+from html.parser import HTMLParser
+from typing import Dict, List, Optional, Tuple
+
+UA = "sameness/0.1 (+https://github.com/tsurutanmen/sameness)"
+
+
+def read_file(path: str) -> Tuple[str, str]:
+    with open(path, "r", encoding="utf-8", errors="replace") as f:
+        return os.path.basename(path), f.read()
+
+
+def get(url: str, timeout: float = 15.0) -> Optional[str]:
+    req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "text/html,text/css;q=0.9,*/*;q=0.5"})
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as r:
+            ctype = r.headers.get("Content-Type", "")
+            raw = r.read()
+    except Exception as exc:  # noqa: BLE001
+        print(f"  skip {url}: {exc}", file=sys.stderr)
+        return None
+    if "html" not in ctype and "css" not in ctype and not url.endswith(".css"):
+        return None
+    m = re.search(r"charset=([\w-]+)", ctype)
+    enc = m.group(1) if m else "utf-8"
+    try:
+        return raw.decode(enc, errors="replace")
+    except LookupError:
+        return raw.decode("utf-8", errors="replace")
+
+
+class _Links(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.hrefs: List[str] = []
+        self.css: List[str] = []
+
+    def handle_starttag(self, tag, attrs):
+        a = dict(attrs)
+        if tag == "a" and a.get("href"):
+            self.hrefs.append(a["href"])
+        if tag == "link" and "stylesheet" in (a.get("rel") or "") and a.get("href"):
+            self.css.append(a["href"])
+
+
+def _same_site(u: str, base: str) -> bool:
+    a, b = urllib.parse.urlparse(u), urllib.parse.urlparse(base)
+    return a.netloc == b.netloc and a.scheme in ("http", "https")
+
+
+def _clean(u: str) -> str:
+    u, _ = urllib.parse.urldefrag(u)
+    return u
+
+
+def crawl(start: str, max_pages: int = 20, fetch_css: bool = True) -> List[Tuple[str, str, str]]:
+    """Breadth-first over same-host links.  Returns (url, html, css_text) triples."""
+    seen, queue, out = set(), [_clean(start)], []
+    css_cache: Dict[str, str] = {}
+    while queue and len(out) < max_pages:
+        url = queue.pop(0)
+        if url in seen:
+            continue
+        seen.add(url)
+        html = get(url)
+        if html is None:
+            continue
+        lp = _Links()
+        try:
+            lp.feed(html)
+        except Exception:  # noqa: BLE001
+            pass
+        css_text = ""
+        if fetch_css:
+            for href in lp.css[:6]:
+                cu = urllib.parse.urljoin(url, href)
+                if cu not in css_cache:
+                    css_cache[cu] = get(cu) or ""
+                css_text += "\n" + css_cache[cu]
+        out.append((url, html, css_text))
+        for href in lp.hrefs:
+            nu = _clean(urllib.parse.urljoin(url, href))
+            if _same_site(nu, start) and nu not in seen and not re.search(r"\.(png|jpe?g|gif|svg|pdf|zip|css|js|ico|webp|mp4)$", nu, re.I):
+                queue.append(nu)
+    return out
+
+
+def load_local_css(html_path: str, hrefs: List[str]) -> str:
+    """Resolve relative stylesheet hrefs next to a local HTML file."""
+    base = os.path.dirname(os.path.abspath(html_path))
+    parts = []
+    for h in hrefs:
+        if h.startswith(("http://", "https://", "//")):
+            continue
+        p = os.path.normpath(os.path.join(base, h.split("?")[0].lstrip("/")))
+        if os.path.exists(p):
+            with open(p, "r", encoding="utf-8", errors="replace") as f:
+                parts.append(f.read())
+    return "\n".join(parts)

sameness-0.1.0/sameness.egg-info/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.4
+Name: sameness
+Version: 0.1.0
+Summary: Do the pages of one site look like one template? Compare content skeletons, shells, fonts, palettes and copy across pages. No dependencies.
+Author: Tsuruta Lab
+License: MIT
+Project-URL: Homepage, https://github.com/tsurutanmen/sameness
+Project-URL: Issues, https://github.com/tsurutanmen/sameness/issues
+Keywords: design,ai slop,template,website audit,html,consistency
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: WWW/HTTP :: Site Management
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Dynamic: license-file
+
+# sameness
+
+Do the pages of one site look like one template?
+
+Detectors for the AI look score one page at a time: the purple gradient, Inter, three feature cards.
+But when people are asked what gives a machine-made site away, the two answers at the top are not
+features. In a Reddit corpus of 3,033 on-topic comments, "screams AI" (6.4%) and "they all look the
+same" (6.1%) each beat every specific feature; purple, gradients, animations and glass together add
+up to about 7%. Sameness is a property of a *set* of pages. `sameness` compares the pages of one
+site with each other.
+
+```
+pip install git+https://github.com/tsurutanmen/sameness
+```
+
+No dependencies. Python 3.9 or later.
+
+## Thirty seconds
+
+```
+$ sameness tests/fixtures/clone_*.html
+
+4 pages
+
+sameness    1.00   mean similarity of the content skeleton across page pairs (1 = one template)
+shells      1      distinct header/nav/footer structures (1 = one site)
+palette     1.00   mean overlap of hex colours between pages (1 = one palette)
+
+near-clones (content skeleton >= 90% similar):
+  clone_0.html  ~  clone_1.html  ~  clone_2.html  ~  clone_3.html
+
+same meta description on several pages:
+    4 page(s)  "One platform to elevate your workflow."
+
+headings that appear on several pages (outside header/nav/footer):
+    4x  Ready?
+
+page           words emoji grad   sym  repeated blocks
+clone_0.html      36     0    0   3x3  cardx3
+...
+note: pages share one content skeleton; a reader will see one template with the words swapped
+note: 4 pages reuse a meta description; search results will show the same sentence for each
+```
+
+Or crawl a live site, same host only, linked stylesheets included:
+
+```
+sameness --site https://example.com --max 20
+```
+
+## What it measures
+
+| line | meaning |
+|---|---|
+| `sameness` | Mean similarity of the *content* skeleton (the tag sequence of the page with header, nav and footer removed) over all page pairs. 1.00 means every page is the same template. |
+| `near-clones` | Groups of pages whose content skeletons are at least 90% similar. Pages of one kind (articles, product pages) are expected to share a skeleton; the question is whether pages of different kinds do too. |
+| `shells` | Number of distinct header/nav/footer structures. A site has one. When it has four, the pages do not read as one place, which is the opposite failure and just as visible. |
+| `font sets`, `palette` | Fonts declared and hex colours used, per page and their overlap across pages. |
+| `same meta description`, `same <title>` | Copy reused across pages, usually a default that was never replaced. |
+| `sym` | Perfectly symmetric blocks on a page: `3x3` means three lists of exactly three items. |
+| `--pairs` | The full pairwise table: skeleton similarity, shared headings, shared repeated block classes. |
+
+Everything comes from the markup and the stylesheet text. Nothing is rendered, so layout that only
+exists after JavaScript runs is not seen.
+
+## Two real sites
+
+A 15-page site of ours that had been built in four sittings. No page-level detector complained; the
+Tell Score was A or B on every page. `sameness` reported:
+
+```
+sameness    0.26
+shells      8      distinct header/nav/footer structures (1 = one site)
+palette     0.26
+font sets:  5
+note: 8 different shells on one site: the pages do not read as one place
+```
+
+That was the finding a reader had already made ("it looks like four different sites"), now as a
+number. The other direction, a small research site with one shell:
+
+```
+sameness    0.51
+shells      1
+palette     1.00
+near-clones: the six research notes share one article skeleton
+same meta description on several pages:  2 page(s)  "aaaaaaaaaaaaaa"
+```
+
+The near-clones are articles and are supposed to look alike. The placeholder description was real
+and had been missed.
+
+## Python
+
+```python
+import sameness
+
+pages = [sameness.extract(html, name=url) for url, html in docs]
+rep = sameness.compare(pages)
+print(rep)
+rep.to_dict()                      # everything, JSON-serialisable
+
+for p in rep.pairs:                # pairwise numbers
+    print(p.a, p.b, p.skeleton, p.headings, p.classes)
+
+for url, html, css in sameness.crawl("https://example.com", max_pages=20):
+    ...
+```
+
+## Claude Code skill
+
+`skill/sameness/SKILL.md` tells Claude Code to run this after building or editing more than one page
+of a site, and how to read the result. Install by copying the folder:
+
+```
+cp -r skill/sameness ~/.claude/skills/sameness
+```
+
+## Related tools
+
+Per-page detectors, which this complements rather than replaces:
+[ai-design-tells](https://github.com/hankimis/ai-design-tells) (Tell Score, 27 tells),
+[avoid-ai-design](https://github.com/funboy322/avoid-ai-design) (a rewrite skill),
+[vibecoded-audit](https://github.com/KreshBack/vibecoded-audit) (49 tells, crawls but scores per page).
+The Reddit corpus is from
+[vibecoded-design-tells](https://github.com/JCarterJohnson/vibecoded-design-tells).
+
+## License
+
+MIT.

sameness-0.1.0/sameness.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE
+README.md
+pyproject.toml
+sameness/__init__.py
+sameness/cli.py
+sameness/compare.py
+sameness/extract.py
+sameness/fetch.py
+sameness.egg-info/PKG-INFO
+sameness.egg-info/SOURCES.txt
+sameness.egg-info/dependency_links.txt
+sameness.egg-info/entry_points.txt
+sameness.egg-info/requires.txt
+sameness.egg-info/top_level.txt
+tests/test_sameness.py

sameness-0.1.0/sameness.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

sameness-0.1.0/sameness.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+sameness = sameness.cli:main

sameness-0.1.0/sameness.egg-info/requires.txt

@@ -0,0 +1,3 @@
+
+[test]
+pytest>=7

sameness-0.1.0/sameness.egg-info/top_level.txt

@@ -0,0 +1 @@
+sameness

sameness-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sameness-0.1.0/tests/test_sameness.py

@@ -0,0 +1,92 @@
+import glob
+import os
+import subprocess
+import sys
+
+import pytest
+
+import sameness
+from sameness.extract import extract
+from sameness.compare import compare
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+FIX = os.path.join(HERE, "fixtures")
+
+
+@pytest.fixture(scope="module", autouse=True)
+def fixtures():
+    if not os.path.isdir(FIX):
+        subprocess.check_call([sys.executable, os.path.join(HERE, "make_fixtures.py")])
+
+
+def load(pattern):
+    pages = []
+    for f in sorted(glob.glob(os.path.join(FIX, pattern))):
+        with open(f, encoding="utf-8") as fh:
+            pages.append(extract(fh.read(), name=os.path.basename(f)))
+    return pages
+
+
+def test_extract_basics():
+    p = load("japanese.html")[0]
+    assert p.title == "学習サイト" and p.description == "説明" and p.lang == "ja"
+    assert p.headings == [(1, "今日の学習 🐢")]
+    assert p.list_sizes == [2]
+    assert p.emoji == 1
+    assert p.word_count > 5                      # CJK counted by character
+    assert "nav" in p.shell and "h1" in p.skeleton and "h1" not in p.shell
+
+
+def test_clones_are_one_template():
+    rep = compare(load("clone_*.html"))
+    assert rep.sameness > 0.95
+    assert rep.clone_groups and len(rep.clone_groups[0]) == 4
+    assert rep.shell_count == 1
+    assert rep.palette_overlap == 1.0
+    assert rep.duplicate_descriptions and len(rep.duplicate_descriptions[0][1]) == 4
+    assert any("one template" in n for n in rep.notes)
+    sym = rep.per_page["clone_0.html"]["symmetry"]
+    assert (3, 3) in sym["equal_list_runs"]      # three lists of exactly three items
+    assert sym["repeated_blocks"].get("card") == 3
+
+
+def test_diverse_pages_are_not_clones():
+    rep = compare(load("diverse_*.html"))
+    assert rep.sameness < 0.6
+    assert not rep.clone_groups
+    assert rep.shell_count == 1
+    assert not rep.duplicate_descriptions
+
+
+def test_forked_shells_are_counted():
+    rep = compare(load("fork_*.html"))
+    assert rep.shell_count == 2
+    assert len(rep.font_sets) == 2
+    assert rep.palette_overlap < 0.5
+    assert rep.pairs[0].skeleton > 0.95          # same content skeleton under two shells
+    assert any("different shells" in n for n in rep.notes)
+
+
+def test_headings_shared_across_pages():
+    rep = compare(load("clone_*.html"))
+    shared = dict(rep.shared_headings)
+    assert shared.get("Ready?") == 4
+
+
+def test_single_page_is_refused_gently():
+    rep = compare(load("japanese.html"))
+    assert any("at least two" in n for n in rep.notes)
+
+
+def test_skeleton_similarity_bounds():
+    assert sameness.skeleton_similarity([], []) == 1.0
+    assert sameness.skeleton_similarity(["h1", "p"], ["table"]) == 0.0
+
+
+def test_cli_and_json(tmp_path, capsys):
+    from sameness.cli import main
+    out = tmp_path / "r.json"
+    rc = main([os.path.join(FIX, "clone_*.html"), "--json", str(out), "--pairs"])
+    assert rc == 0
+    text = capsys.readouterr().out
+    assert "sameness" in text and "near-clones" in text and out.exists()