semlint 0.1.0__tar.gz

semlint-0.1.0/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.3
+Name: semlint
+Version: 0.1.0
+Summary: Find semantic redundancy in agent instruction corpora.
+Author: Karlis Vagalis
+Author-email: Karlis Vagalis <karlis.vagalis@gmail.com>
+License: MIT
+Requires-Dist: markdown-it-py[plugins]>=4.2.0
+Requires-Dist: pydantic>=2.13.5
+Requires-Dist: rich>=15.0.0
+Requires-Dist: semhash>=0.4.1
+Requires-Dist: typer>=0.27.2
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+
+# semlint
+
+`semlint` is a small, local Python CLI for finding redundant information in agent instruction corpora: `SKILL.md`, `AGENTS.md`, prompts, rules, and other Markdown/text files.
+
+It answers: **how many tokens are consuming context without adding unique information, and where are they?**
+
+## Quick start
+
+```bash
+uv sync
+uv run semlint ./skills
+uv run semlint ./skills ./rules/AGENTS.md
+uv run semlint './skills/**/*.md' './prompts/*.txt'
+uv run semlint ./skills --threshold 0.90 --min-block-tokens 10
+uv run semlint ./skills --split markdown
+uv run semlint ./skills --output json > report.json
+uv run semlint ./skills --limit 10
+uv run semlint ./skills -s asc similarity
+uv run semlint ./skills --fail-above 0.10
+uv run semlint self list
+```
+
+The command exits with `1` when `--fail-above` is exceeded and `2` for invalid input/errors.
+
+### Stronger semantic matching
+
+SemHash with local Model2Vec embeddings is the default semantic engine. It loads its small CPU model on first run (and caches it locally). For an explicit alternate Model2Vec model, use:
+
+```bash
+uv run semlint ./skills --model minishlab/potion-base-8M --threshold 0.90
+```
+
+SemHash and Model2Vec run locally on CPU. No API, LLM, vector database, server, or GPU is needed.
+
+## Agent capability discovery
+
+`semlint self list` prints one full command synopsis and help description per line. Optional arguments and options are bracketed inline. The listing is generated by inspecting the registered Typer command definitions, so it stays aligned with the CLI.
+
+## What it scans
+
+- Accepts one or more file paths, directories, or glob patterns; recursively scans `.md` and `.txt` files and de-duplicates overlapping matches.
+- By default (`--split auto`), uses Markdown block boundaries for `.md` files and blank-line paragraph splitting for other supported text files. Markdown blocks include paragraphs (including list and blockquote paragraphs), fenced/indented code, HTML blocks, and tables. Headings and front matter are not treated as instruction blocks.
+- Use `--split paragraph` to force blank-line splitting, or `--split markdown` to force Markdown parsing.
+- All modes preserve original source text and line ranges; none infer sections or rule IDs.
+- Ignores blocks below `--min-block-tokens`.
+- Clusters likely duplicates without modifying source files.
+
+Similarity is intentionally configurable because generic phrases can be false positives. Review clusters before replacing instructions.
+
+## Output formats
+
+Text is the default and uses Rich-formatted summary metrics, cluster separators, cluster summaries, then a blank line followed by blank-line-separated `path:start-end` locations and numbered source lines. By default, clusters sort by estimated redundant tokens (largest first); `-s|--sort <asc|desc> <occurrences|similarity|estimated-savings>` selects a different order. `--limit N` displays only the first N clusters while keeping corpus-wide metrics unchanged. Linters commonly use this compact location notation with a source code frame; Rich provides readable terminal rendering without introducing a separate diagnostic framework.
+
+Choose JSON with `--output json`. Top-level fields include `files_scanned`, `blocks_scanned`, `total_tokens`, `exact_duplicate_percentage`, `semantic_redundancy_percentage`, `estimated_unique_tokens`, `estimated_redundant_tokens`, `estimated_context_token_savings`, and `clusters`. Clusters include similarity, all source blocks, and token/savings estimates—no canonical or action suggestions.
+
+The savings estimate assumes retaining the shortest occurrence in each cluster; review the source blocks yourself before deciding what to consolidate. `context_redundancy` is token-weighted:
+
+```text
+estimated_redundant_tokens / total_tokens
+```
+
+This is an estimate, not a tokenizer-specific bill. The initial version counts word tokens for transparent, consistent reporting.
+
+## Development
+
+`just build` uses the installed `doxxer` binary to inject the next Git-derived version into `pyproject.toml`, runs `uv build`, then restores the `0.0.0` source placeholder. Install `doxxer` before building.
+
+```bash
+just build
+```
+
+Install the project and development tools with `uv sync`, then run static checks:
+
+```bash
+uv sync
+uv run ruff check src
+uv run ty check src
+```
+
+The project uses Python 3.13+, Pydantic `BaseModel`s for domain and report models, and a modular `src/semlint` package (parser/scanner, SemHash clustering, metrics, reporting, and Typer CLI). Structured SemHash inputs and JSON report output are validated with Pydantic. It does not automatically edit files.

semlint-0.1.0/README.md

@@ -0,0 +1,80 @@
+# semlint
+
+`semlint` is a small, local Python CLI for finding redundant information in agent instruction corpora: `SKILL.md`, `AGENTS.md`, prompts, rules, and other Markdown/text files.
+
+It answers: **how many tokens are consuming context without adding unique information, and where are they?**
+
+## Quick start
+
+```bash
+uv sync
+uv run semlint ./skills
+uv run semlint ./skills ./rules/AGENTS.md
+uv run semlint './skills/**/*.md' './prompts/*.txt'
+uv run semlint ./skills --threshold 0.90 --min-block-tokens 10
+uv run semlint ./skills --split markdown
+uv run semlint ./skills --output json > report.json
+uv run semlint ./skills --limit 10
+uv run semlint ./skills -s asc similarity
+uv run semlint ./skills --fail-above 0.10
+uv run semlint self list
+```
+
+The command exits with `1` when `--fail-above` is exceeded and `2` for invalid input/errors.
+
+### Stronger semantic matching
+
+SemHash with local Model2Vec embeddings is the default semantic engine. It loads its small CPU model on first run (and caches it locally). For an explicit alternate Model2Vec model, use:
+
+```bash
+uv run semlint ./skills --model minishlab/potion-base-8M --threshold 0.90
+```
+
+SemHash and Model2Vec run locally on CPU. No API, LLM, vector database, server, or GPU is needed.
+
+## Agent capability discovery
+
+`semlint self list` prints one full command synopsis and help description per line. Optional arguments and options are bracketed inline. The listing is generated by inspecting the registered Typer command definitions, so it stays aligned with the CLI.
+
+## What it scans
+
+- Accepts one or more file paths, directories, or glob patterns; recursively scans `.md` and `.txt` files and de-duplicates overlapping matches.
+- By default (`--split auto`), uses Markdown block boundaries for `.md` files and blank-line paragraph splitting for other supported text files. Markdown blocks include paragraphs (including list and blockquote paragraphs), fenced/indented code, HTML blocks, and tables. Headings and front matter are not treated as instruction blocks.
+- Use `--split paragraph` to force blank-line splitting, or `--split markdown` to force Markdown parsing.
+- All modes preserve original source text and line ranges; none infer sections or rule IDs.
+- Ignores blocks below `--min-block-tokens`.
+- Clusters likely duplicates without modifying source files.
+
+Similarity is intentionally configurable because generic phrases can be false positives. Review clusters before replacing instructions.
+
+## Output formats
+
+Text is the default and uses Rich-formatted summary metrics, cluster separators, cluster summaries, then a blank line followed by blank-line-separated `path:start-end` locations and numbered source lines. By default, clusters sort by estimated redundant tokens (largest first); `-s|--sort <asc|desc> <occurrences|similarity|estimated-savings>` selects a different order. `--limit N` displays only the first N clusters while keeping corpus-wide metrics unchanged. Linters commonly use this compact location notation with a source code frame; Rich provides readable terminal rendering without introducing a separate diagnostic framework.
+
+Choose JSON with `--output json`. Top-level fields include `files_scanned`, `blocks_scanned`, `total_tokens`, `exact_duplicate_percentage`, `semantic_redundancy_percentage`, `estimated_unique_tokens`, `estimated_redundant_tokens`, `estimated_context_token_savings`, and `clusters`. Clusters include similarity, all source blocks, and token/savings estimates—no canonical or action suggestions.
+
+The savings estimate assumes retaining the shortest occurrence in each cluster; review the source blocks yourself before deciding what to consolidate. `context_redundancy` is token-weighted:
+
+```text
+estimated_redundant_tokens / total_tokens
+```
+
+This is an estimate, not a tokenizer-specific bill. The initial version counts word tokens for transparent, consistent reporting.
+
+## Development
+
+`just build` uses the installed `doxxer` binary to inject the next Git-derived version into `pyproject.toml`, runs `uv build`, then restores the `0.0.0` source placeholder. Install `doxxer` before building.
+
+```bash
+just build
+```
+
+Install the project and development tools with `uv sync`, then run static checks:
+
+```bash
+uv sync
+uv run ruff check src
+uv run ty check src
+```
+
+The project uses Python 3.13+, Pydantic `BaseModel`s for domain and report models, and a modular `src/semlint` package (parser/scanner, SemHash clustering, metrics, reporting, and Typer CLI). Structured SemHash inputs and JSON report output are validated with Pydantic. It does not automatically edit files.

semlint-0.1.0/pyproject.toml

@@ -0,0 +1,48 @@
+[project]
+name = "semlint"
+version = "0.1.0"
+description = "Find semantic redundancy in agent instruction corpora."
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "markdown-it-py[plugins]>=4.2.0",
+    "pydantic>=2.13.5",
+    "rich>=15.0.0",
+    "semhash>=0.4.1",
+    "typer>=0.27.2",
+]
+
+[project.license]
+text = "MIT"
+
+[[project.authors]]
+name = "Karlis Vagalis"
+email = "karlis.vagalis@gmail.com"
+
+[project.scripts]
+semlint = "semlint.cli:main"
+
+[dependency-groups]
+dev = [
+    "ruff>=0.16.8",
+    "ty>=0.0.83",
+]
+
+[build-system]
+requires = ["uv_build>=0.12.18,<0.13"]
+build-backend = "uv_build"
+
+[tool.ruff]
+line-length = 100
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+    "UP",
+    "B",
+]
+
+[tool.ty.environment]
+python-version = "3.13"

semlint-0.1.0/pyproject.toml.orig

@@ -0,0 +1,37 @@
+[project]
+name = "semlint"
+version = "0.1.0"
+description = "Find semantic redundancy in agent instruction corpora."
+readme = "README.md"
+requires-python = ">=3.13"
+license = { text = "MIT" }
+authors = [{ name = "Karlis Vagalis", email = "karlis.vagalis@gmail.com" }]
+dependencies = [
+    "markdown-it-py[plugins]>=4.2.0",
+    "pydantic>=2.13.5",
+    "rich>=15.0.0",
+    "semhash>=0.4.1",
+    "typer>=0.27.2",
+]
+
+[project.scripts]
+semlint = "semlint.cli:main"
+
+[dependency-groups]
+dev = [
+    "ruff>=0.16.8",
+    "ty>=0.0.83",
+]
+
+[build-system]
+requires = ["uv_build>=0.12.18,<0.13"]
+build-backend = "uv_build"
+
+[tool.ruff]
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[tool.ty.environment]
+python-version = "3.13"

semlint-0.1.0/src/semlint/__init__.py

@@ -0,0 +1,8 @@
+"""Semantic redundancy analysis for agent instruction files."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("semlint")
+except PackageNotFoundError:
+    __version__ = "0.0.0"

semlint-0.1.0/src/semlint/cli.py

@@ -0,0 +1,225 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Annotated, Any, Literal, cast
+
+import typer
+from typer._click import Context
+from typer.core import TyperArgument, TyperCommand, TyperGroup, TyperOption
+from typer.main import get_command
+
+from .core import (
+    AnalysisConfig,
+    SortCategory,
+    SortOrder,
+    SplitMode,
+    analyze,
+)
+from .report import render, render_json
+
+app = typer.Typer(
+    help="Find redundant information in agent instruction files.", add_completion=False
+)
+discovery_app = typer.Typer(help="Agent-facing CLI capability discovery.", add_completion=False)
+self_app = typer.Typer(help="CLI capability discovery for agent clients.", add_completion=False)
+
+
+def _parameter_type(parameter: TyperArgument | TyperOption) -> str:
+    choices = getattr(parameter.type, "choices", None)
+    if choices is not None:
+        return "|".join(str(choice) for choice in choices)
+    type_name = getattr(parameter.type, "name", None)
+    return str(type_name or parameter.type)
+
+
+def _command_entry(
+    command: TyperCommand | TyperGroup,
+    command_path: str,
+) -> dict[str, Any]:
+    arguments = [param for param in command.params if isinstance(param, TyperArgument)]
+    options = [param for param in command.params if isinstance(param, TyperOption)]
+    usage_parts = [command_path]
+    for parameter in arguments:
+        name = (parameter.name or "ARG").upper().replace("_", "-")
+        if parameter.name == "paths":
+            name = "PATH"
+        part = f"<{name}>"
+        if parameter.nargs == -1:
+            part += "..."
+        usage_parts.append(part if parameter.required else f"[{part}]")
+    for parameter in options:
+        long_flag = next((option for option in parameter.opts if option.startswith("--")), None)
+        short_flag = next(
+            (
+                option
+                for option in parameter.opts
+                if option.startswith("-") and not option.startswith("--")
+            ),
+            None,
+        )
+        flag = (
+            f"{short_flag}/{long_flag}"
+            if short_flag and long_flag
+            else long_flag or parameter.opts[0]
+        )
+        value_types = getattr(parameter.type, "types", None)
+        if value_types:
+            value_parts = []
+            for index, value_type in enumerate(value_types):
+                choices = getattr(value_type, "choices", None)
+                label = "|".join(str(choice) for choice in choices) if choices else ""
+                if not label:
+                    label = "order" if index == 0 else "category"
+                value_parts.append(f"<{label}>")
+            value = " " + " ".join(value_parts)
+        elif parameter.is_flag:
+            value = ""
+        else:
+            value = f" <{(parameter.name or 'VALUE').replace('_', '-')}>"
+        part = f"{flag}{value}"
+        usage_parts.append(part if parameter.required else f"[{part}]")
+    usage = " ".join(usage_parts)
+    params: list[dict[str, Any]] = []
+    for param in command.params:
+        record: dict[str, Any] = {
+            "name": param.name,
+            "kind": "option" if isinstance(param, TyperOption) else "argument",
+            "required": param.required,
+            "multiple": isinstance(param, TyperArgument) and param.nargs == -1,
+            "help": getattr(param, "help", None),
+        }
+        if isinstance(param, TyperOption):
+            record["flags"] = list(param.opts)
+            record["type"] = _parameter_type(param)
+            choices = getattr(param.type, "choices", None)
+            if choices is not None:
+                record["choices"] = list(choices)
+            if param.default is not None:
+                record["default"] = param.default
+        elif isinstance(param, TyperArgument):
+            record["type"] = _parameter_type(param)
+        params.append(record)
+    return {
+        "command": usage,
+        "description": command.help or "",
+        "parameters": params,
+    }
+
+
+def discover_commands() -> list[dict[str, Any]]:
+    """Describe executable commands from the registered Typer/Click tree."""
+    scan_command = get_command(app)
+    entries = [_command_entry(cast(TyperCommand | TyperGroup, scan_command), "semlint")]
+    root = get_command(discovery_app)
+    root_context = Context(root, info_name="semlint")
+
+    def visit(group: TyperGroup, prefix: str, parent: Context) -> None:
+        for name, command in group.commands.items():
+            command_path = f"{prefix} {name}"
+            context = Context(command, info_name=command_path, parent=parent)
+            if isinstance(command, TyperGroup):
+                visit(command, command_path, context)
+            elif isinstance(command, TyperCommand):
+                entries.append(_command_entry(command, command_path))
+
+    if isinstance(root, TyperGroup):
+        visit(root, "semlint", root_context)
+    return entries
+
+
+@self_app.command("list")
+def list_commands() -> None:
+    """List each available command and its usage in a single line."""
+    for command in discover_commands():
+        description = " ".join(command["description"].split())
+        suffix = f" - {description}" if description else ""
+        typer.echo(f"{command['command']}{suffix}")
+
+
+discovery_app.add_typer(self_app, name="self")
+
+
+@app.command()
+def scan(
+    paths: Annotated[
+        list[Path],
+        typer.Argument(help="One or more files, directories, or glob patterns to scan."),
+    ],
+    threshold: Annotated[
+        float,
+        typer.Option(
+            "--threshold",
+            "-t",
+            min=0.0,
+            max=1.0,
+            help="SemHash similarity threshold.",
+        ),
+    ] = 0.90,
+    min_block_tokens: Annotated[
+        int, typer.Option("--min-block-tokens", min=1, help="Ignore shorter blocks.")
+    ] = 10,
+    split_mode: Annotated[
+        SplitMode,
+        typer.Option("--split", help="Block splitting mode: auto, paragraph, or markdown."),
+    ] = SplitMode.AUTO,
+    limit: Annotated[
+        int | None, typer.Option("--limit", min=1, help="Maximum number of clusters to display.")
+    ] = None,
+    fail_above: Annotated[
+        float | None,
+        typer.Option(
+            "--fail-above",
+            min=0.0,
+            max=1.0,
+            help="Exit 1 when context redundancy exceeds this ratio.",
+        ),
+    ] = None,
+    output: Annotated[
+        Literal["text", "json"],
+        typer.Option("--output", help="Output format: text or json."),
+    ] = "text",
+    model: Annotated[str | None, typer.Option("--model", help="Model2Vec model name/path.")] = None,
+    sort: Annotated[
+        tuple[SortOrder, SortCategory],
+        typer.Option(
+            "--sort",
+            "-s",
+            help="Sort clusters by asc/desc and occurrences, similarity, or estimated-savings.",
+        ),
+    ] = (SortOrder.DESC, SortCategory.ESTIMATED_SAVINGS),
+) -> None:
+    """Scan paths, directories, or glob patterns for redundant instruction blocks."""
+    try:
+        report = analyze(
+            paths,
+            AnalysisConfig(
+                threshold=threshold,
+                min_block_tokens=min_block_tokens,
+                split_mode=split_mode,
+                model=model,
+            ),
+        )
+    except (OSError, RuntimeError, ImportError) as exc:
+        typer.echo(f"semlint: {exc}", err=True)
+        raise typer.Exit(2) from exc
+    typer.echo(
+        render_json(report, limit=limit, sort=sort)
+        if output == "json"
+        else render(report, limit=limit, sort=sort)
+    )
+    if fail_above is not None and report.context_redundancy > fail_above:
+        raise typer.Exit(1)
+
+
+def main() -> None:
+    import sys
+
+    args = sys.argv[1:]
+    if args and args[0] == "self":
+        discovery_app(args=args)
+    else:
+        app(args=args)
+
+
+if __name__ == "__main__":
+    main()

semlint-0.1.0/src/semlint/core.py

@@ -0,0 +1,331 @@
+from __future__ import annotations
+
+import glob
+import os
+import re
+from collections import defaultdict
+from collections.abc import Sequence
+from enum import StrEnum
+from pathlib import Path
+from typing import Any, cast
+
+from markdown_it import MarkdownIt
+from mdit_py_plugins.front_matter import front_matter_plugin
+from pydantic import BaseModel, ConfigDict, Field
+
+WORD_RE = re.compile(r"[\w][\w'-]*", re.UNICODE)
+
+
+class Block(BaseModel):
+    model_config = ConfigDict(extra="forbid", frozen=True)
+
+    path: str = Field(min_length=1)
+    start_line: int = Field(ge=1)
+    end_line: int = Field(ge=1)
+    text: str = Field(min_length=1)
+    tokens: int = Field(gt=0)
+
+    @property
+    def location(self) -> str:
+        return f"{self.path}:{self.start_line}-{self.end_line}"
+
+
+class SortOrder(StrEnum):
+    ASC = "asc"
+    DESC = "desc"
+
+
+class SortCategory(StrEnum):
+    OCCURRENCES = "occurrences"
+    SIMILARITY = "similarity"
+    ESTIMATED_SAVINGS = "estimated-savings"
+
+
+class SplitMode(StrEnum):
+    AUTO = "auto"
+    PARAGRAPH = "paragraph"
+    MARKDOWN = "markdown"
+
+
+class AnalysisConfig(BaseModel):
+    model_config = ConfigDict(extra="forbid", frozen=True)
+
+    threshold: float = Field(default=0.90, ge=0.0, le=1.0)
+    min_block_tokens: int = Field(default=10, ge=1)
+    split_mode: SplitMode = SplitMode.AUTO
+    extensions: frozenset[str] = frozenset({".md", ".txt"})
+    model: str | None = None
+
+
+class Cluster(BaseModel):
+    model_config = ConfigDict(extra="forbid", frozen=True)
+
+    blocks: list[Block] = Field(min_length=2)
+    similarity: float = Field(ge=0.0, le=1.0)
+    exact: bool = False
+
+    @property
+    def total_tokens(self) -> int:
+        return sum(block.tokens for block in self.blocks)
+
+    @property
+    def redundant_tokens(self) -> int:
+        return self.total_tokens - min(block.tokens for block in self.blocks)
+
+    @property
+    def estimated_saving(self) -> float:
+        return self.redundant_tokens / self.total_tokens if self.total_tokens else 0.0
+
+
+class Report(BaseModel):
+    model_config = ConfigDict(extra="forbid", frozen=True)
+
+    files_scanned: int = Field(ge=0)
+    blocks_scanned: int = Field(ge=0)
+    total_tokens: int = Field(ge=0)
+    exact_duplicate_tokens: int = Field(ge=0)
+    clusters: list[Cluster] = Field(default_factory=list)
+
+    @property
+    def redundant_tokens(self) -> int:
+        return sum(cluster.redundant_tokens for cluster in self.clusters)
+
+    @property
+    def unique_tokens(self) -> int:
+        return max(0, self.total_tokens - self.redundant_tokens)
+
+    @property
+    def context_redundancy(self) -> float:
+        return self.redundant_tokens / self.total_tokens if self.total_tokens else 0.0
+
+    @property
+    def exact_duplicate_percentage(self) -> float:
+        return self.exact_duplicate_tokens / self.total_tokens if self.total_tokens else 0.0
+
+    @property
+    def semantic_redundancy_percentage(self) -> float:
+        return self.context_redundancy
+
+
+def normalize(text: str) -> str:
+    return " ".join(WORD_RE.findall(text.casefold()))
+
+
+def token_count(text: str) -> int:
+    return len(WORD_RE.findall(text))
+
+
+def scan_paths(roots: Sequence[Path], extensions: frozenset[str]) -> tuple[list[Path], Path]:
+    if not roots:
+        raise ValueError("provide at least one file, directory, or glob pattern")
+    discovered: dict[Path, Path] = {}
+    display_bases: list[Path] = []
+    for root in roots:
+        raw = os.fspath(root)
+        has_glob = glob.has_magic(raw)
+        matches = [Path(value) for value in glob.glob(raw, recursive=True)] if has_glob else [root]
+        if not matches or any(not match.exists() for match in matches):
+            raise FileNotFoundError(f"input path or glob did not match: {root}")
+
+        if has_glob:
+            prefix = raw[: min(raw.find(char) for char in "*?[" if char in raw)]
+            base = (
+                Path(prefix).resolve() if prefix.endswith(os.sep) else Path(prefix).parent.resolve()
+            )
+        elif len(matches) == 1 and matches[0].is_dir():
+            base = matches[0].resolve()
+        else:
+            base = matches[0].resolve().parent
+        display_bases.append(base)
+
+        for match in matches:
+            candidates = match.rglob("*") if match.is_dir() else [match]
+            for candidate in candidates:
+                if candidate.is_file() and candidate.suffix.lower() in extensions:
+                    resolved = candidate.resolve()
+                    discovered[resolved] = resolved
+
+    display_root = (
+        display_bases[0] if len(display_bases) == 1 else Path(os.path.commonpath(display_bases))
+    )
+    return sorted(discovered), display_root
+
+
+def parse_file(
+    path: Path,
+    root: Path,
+    min_tokens: int,
+    split_mode: SplitMode = SplitMode.AUTO,
+) -> list[Block]:
+    """Split a file into blocks using blank lines or Markdown block boundaries."""
+    source = path.read_text(encoding="utf-8", errors="replace")
+    lines = source.splitlines()
+    if split_mode is SplitMode.MARKDOWN or (
+        split_mode is SplitMode.AUTO and path.suffix.lower() == ".md"
+    ):
+        return parse_markdown(path, root, lines, source, min_tokens)
+
+    blocks: list[Block] = []
+    pending: list[str] = []
+    start = 0
+
+    def flush(end: int) -> None:
+        nonlocal pending
+        text = "\n".join(pending).strip()
+        pending = []
+        if not text or token_count(text) < min_tokens:
+            return
+        blocks.append(
+            Block(
+                path=str(path.relative_to(root)),
+                start_line=start + 1,
+                end_line=end + 1,
+                text=text,
+                tokens=token_count(text),
+            )
+        )
+
+    for index, line in enumerate(lines):
+        if not line.strip():
+            flush(index - 1)
+        else:
+            if not pending:
+                start = index
+            pending.append(line)
+    flush(len(lines) - 1)
+    return blocks
+
+
+def parse_markdown(
+    path: Path,
+    root: Path,
+    lines: list[str],
+    source: str,
+    min_tokens: int,
+) -> list[Block]:
+    """Extract Markdown leaf blocks while retaining parser-provided source line ranges."""
+    parser = MarkdownIt("default").use(front_matter_plugin)
+    block_types = {"paragraph_open", "fence", "code_block", "html_block", "table_open"}
+    blocks: list[Block] = []
+    for token in parser.parse(source):
+        if token.type not in block_types or token.map is None:
+            continue
+        start, end = token.map
+        text = "\n".join(lines[start:end]).strip()
+        tokens = token_count(text)
+        if not text or tokens < min_tokens:
+            continue
+        blocks.append(
+            Block(
+                path=str(path.relative_to(root)),
+                start_line=start + 1,
+                end_line=end,
+                text=text,
+                tokens=tokens,
+            )
+        )
+    return blocks
+
+
+class SemHashRecord(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    text: str
+    index: int
+
+
+def sort_clusters(
+    clusters: Sequence[Cluster],
+    order: SortOrder = SortOrder.DESC,
+    category: SortCategory = SortCategory.ESTIMATED_SAVINGS,
+) -> list[Cluster]:
+    tie_sorted = sorted(
+        clusters,
+        key=lambda cluster: (
+            -cluster.redundant_tokens,
+            -cluster.similarity,
+            -len(cluster.blocks),
+            cluster.blocks[0].path,
+            cluster.blocks[0].start_line,
+        ),
+    )
+    value = {
+        SortCategory.OCCURRENCES: lambda cluster: len(cluster.blocks),
+        SortCategory.SIMILARITY: lambda cluster: cluster.similarity,
+        SortCategory.ESTIMATED_SAVINGS: lambda cluster: cluster.redundant_tokens,
+    }[category]
+    return sorted(tie_sorted, key=value, reverse=order is SortOrder.DESC)
+
+
+def analyze(roots: Path | Sequence[Path], config: AnalysisConfig) -> Report:
+    inputs = [roots] if isinstance(roots, Path) else list(roots)
+    paths, display_root = scan_paths(inputs, config.extensions)
+    blocks = [
+        block
+        for path in paths
+        for block in parse_file(path, display_root, config.min_block_tokens, config.split_mode)
+    ]
+    clusters: list[Cluster] = []
+    exact_tokens = 0
+    if blocks:
+        from semhash import SemHash
+
+        model: Any = None
+        if config.model:
+            from model2vec import StaticModel
+
+            model = cast(Any, StaticModel.from_pretrained(config.model))
+        records = [
+            SemHashRecord(text=block.text, index=index).model_dump()
+            for index, block in enumerate(blocks)
+        ]
+        semhash = SemHash.from_records(records=records, columns=["text"], model=model)
+        neighbors = semhash.index.query_threshold(semhash.index.vectors, threshold=config.threshold)
+        graph: dict[int, list[tuple[int, float]]] = defaultdict(list)
+        indexed_items = semhash.index.items
+        for row_number, row in enumerate(neighbors):
+            if not row:
+                continue
+            source_record = indexed_items[row_number][0]
+            source = SemHashRecord.model_validate(source_record).index
+            for record, score in row:
+                target = SemHashRecord.model_validate(record).index
+                if source == target:
+                    continue
+                graph[source].append((target, float(score)))
+
+        visited: set[int] = set()
+        for index in range(len(blocks)):
+            if index in visited or index not in graph:
+                continue
+            stack, members, scores = [index], set(), []
+            while stack:
+                current = stack.pop()
+                if current in members:
+                    continue
+                members.add(current)
+                for neighbor, score in graph[current]:
+                    scores.append(score)
+                    if neighbor not in members:
+                        stack.append(neighbor)
+            visited.update(members)
+            if len(members) < 2:
+                continue
+            matched = [blocks[item] for item in sorted(members)]
+            exact = len({normalize(item.text) for item in matched}) == 1
+            if exact:
+                exact_tokens += sum(item.tokens for item in matched[1:])
+            clusters.append(
+                Cluster(
+                    blocks=matched,
+                    similarity=min(scores, default=1.0),
+                    exact=exact,
+                )
+            )
+    return Report(
+        files_scanned=len(paths),
+        blocks_scanned=len(blocks),
+        total_tokens=sum(block.tokens for block in blocks),
+        exact_duplicate_tokens=exact_tokens,
+        clusters=clusters,
+    )

semlint-0.1.0/src/semlint/report.py

@@ -0,0 +1,187 @@
+from __future__ import annotations
+
+import json
+from io import StringIO
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict
+from rich.console import Console
+from rich.rule import Rule
+from rich.table import Table
+from rich.text import Text
+
+from .core import Cluster, Report, SortCategory, SortOrder, sort_clusters
+
+
+class BlockOutput(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    file: str
+    start_line: int
+    end_line: int
+    text: str
+    tokens: int
+
+
+class ClusterOutput(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    number: int
+    similarity: float
+    exact: bool
+    total_tokens: int
+    estimated_redundant_tokens: int
+    estimated_saving: float
+    blocks: list[BlockOutput]
+
+
+class ReportOutput(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    files_scanned: int
+    blocks_scanned: int
+    total_tokens: int
+    exact_duplicate_percentage: float
+    semantic_redundancy_percentage: float
+    context_redundancy: float
+    estimated_unique_tokens: int
+    estimated_redundant_tokens: int
+    estimated_context_token_savings: int
+    clusters: list[ClusterOutput]
+
+
+def cluster_output(cluster: Cluster, number: int) -> ClusterOutput:
+    total_tokens = cluster.total_tokens
+    redundant_tokens = cluster.redundant_tokens
+    return ClusterOutput(
+        number=number,
+        similarity=round(cluster.similarity, 4),
+        exact=cluster.exact,
+        total_tokens=total_tokens,
+        estimated_redundant_tokens=redundant_tokens,
+        estimated_saving=redundant_tokens / total_tokens if total_tokens else 0,
+        blocks=[
+            BlockOutput(
+                file=block.path,
+                start_line=block.start_line,
+                end_line=block.end_line,
+                text=block.text,
+                tokens=block.tokens,
+            )
+            for block in cluster.blocks
+        ],
+    )
+
+
+def output_model(
+    report: Report,
+    limit: int | None = None,
+    sort: tuple[SortOrder, SortCategory] = (
+        SortOrder.DESC,
+        SortCategory.ESTIMATED_SAVINGS,
+    ),
+) -> ReportOutput:
+    ordered_clusters = sort_clusters(report.clusters, order=sort[0], category=sort[1])
+    if limit is not None:
+        ordered_clusters = ordered_clusters[:limit]
+    return ReportOutput(
+        files_scanned=report.files_scanned,
+        blocks_scanned=report.blocks_scanned,
+        total_tokens=report.total_tokens,
+        exact_duplicate_percentage=report.exact_duplicate_percentage,
+        semantic_redundancy_percentage=report.semantic_redundancy_percentage,
+        context_redundancy=report.context_redundancy,
+        estimated_unique_tokens=report.unique_tokens,
+        estimated_redundant_tokens=report.redundant_tokens,
+        estimated_context_token_savings=report.redundant_tokens,
+        clusters=[
+            cluster_output(cluster, index) for index, cluster in enumerate(ordered_clusters, 1)
+        ],
+    )
+
+
+def as_dict(
+    report: Report,
+    limit: int | None = None,
+    sort: tuple[SortOrder, SortCategory] = (
+        SortOrder.DESC,
+        SortCategory.ESTIMATED_SAVINGS,
+    ),
+) -> dict[str, Any]:
+    return output_model(report, limit=limit, sort=sort).model_dump(mode="json")
+
+
+def render(
+    report: Report,
+    limit: int | None = None,
+    sort: tuple[SortOrder, SortCategory] = (
+        SortOrder.DESC,
+        SortCategory.ESTIMATED_SAVINGS,
+    ),
+) -> str:
+    data = output_model(report, limit=limit, sort=sort)
+    stream = StringIO()
+    console = Console(
+        file=stream,
+        force_terminal=False,
+        color_system=None,
+        width=100,
+        highlight=False,
+    )
+    console.print(Rule("semlint · instruction redundancy report", style="bright_blue"))
+
+    summary = Table(show_header=False, box=None, padding=(0, 2))
+    summary.add_column("Metric", style="cyan")
+    summary.add_column("Value", justify="right")
+    summary.add_row("Files scanned", str(data.files_scanned))
+    summary.add_row("Blocks scanned", str(data.blocks_scanned))
+    summary.add_row("Total word tokens", f"{data.total_tokens:,}")
+    summary.add_row("Exact duplicate share", f"{data.exact_duplicate_percentage:.1%}")
+    summary.add_row("Semantic redundancy", f"{data.semantic_redundancy_percentage:.1%}")
+    summary.add_row("Estimated unique tokens", f"{data.estimated_unique_tokens:,}")
+    summary.add_row("Estimated redundant tokens", f"{data.estimated_redundant_tokens:,}")
+    summary.add_row("Potential context-token savings", f"{data.estimated_context_token_savings:,}")
+    console.print(summary)
+
+    for cluster in data.clusters:
+        console.print()
+        console.print(
+            Rule(
+                f"Cluster #{cluster.number} · similarity {cluster.similarity:.2f}"
+                f" · {'exact' if cluster.exact else 'semantic'}",
+                style="bright_blue",
+            )
+        )
+        cluster_stats = Table(show_header=False, box=None, padding=(0, 2))
+        cluster_stats.add_column("Metric", style="dim")
+        cluster_stats.add_column("Value", justify="right")
+        cluster_stats.add_row("Occurrences", str(len(cluster.blocks)))
+        cluster_stats.add_row("Tokens in cluster", str(cluster.total_tokens))
+        cluster_stats.add_row("Estimated redundant tokens", str(cluster.estimated_redundant_tokens))
+        cluster_stats.add_row("Estimated savings", f"{cluster.estimated_saving:.1%}")
+        console.print(cluster_stats)
+        console.print()
+
+        for index, block in enumerate(cluster.blocks):
+            if index:
+                console.print()
+            if block.start_line == block.end_line:
+                location = f"{block.file}:{block.start_line}"
+            else:
+                location = f"{block.file}:{block.start_line}-{block.end_line}"
+            console.print(Text(location, style="bold cyan"))
+            for offset, line in enumerate(block.text.splitlines()):
+                console.print(Text(f"{block.start_line + offset:>5} │ {line}"))
+
+    return stream.getvalue().rstrip()
+
+
+def render_json(
+    report: Report,
+    limit: int | None = None,
+    sort: tuple[SortOrder, SortCategory] = (
+        SortOrder.DESC,
+        SortCategory.ESTIMATED_SAVINGS,
+    ),
+) -> str:
+    return json.dumps(as_dict(report, limit=limit, sort=sort), indent=2, ensure_ascii=False)