jevkit-lint 0.1.0__tar.gz

jevkit_lint-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.coverage
+htmlcov/
+.ruff_cache/
+uv.lock

jevkit_lint-0.1.0/PKG-INFO

@@ -0,0 +1,135 @@
+Metadata-Version: 2.5
+Name: jevkit-lint
+Version: 0.1.0
+Summary: Static linter for TypeSafe Jev questions. Catches the documented jev-1.13 failure modes before you spend a token. No API key required.
+Project-URL: Homepage, https://github.com/pjdurden/jevkit-py
+Project-URL: Issues, https://github.com/pjdurden/jevkit-py/issues
+Author: Prajjwal Chittori
+License-Expression: MIT
+Keywords: jev,lint,linter,static-analysis,system-one,typesafe
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.10
+Requires-Dist: jevkit-core>=0.1.0
+Description-Content-Type: text/markdown
+
+# jevkit-lint
+
+Static linter for [TypeSafe](https://typesafe.ai) Jev questions.
+
+TypeSafe publishes a [list of failure modes](https://docs.typesafe.ai/model-jaggedness/jev-1.13)
+for `jev-1.13`: it reads instructions literally, it cannot count, it reads dates
+as text rather than ordered quantities, it loses accuracy on indirection. Most
+of those are visible in your question definitions before you send anything.
+
+`jevkit-lint` reads the definitions and tells you. It never calls the API, so it
+needs no key and costs nothing to run in CI.
+
+> Unofficial and unaffiliated with TypeSafe.
+
+## Install
+
+```bash
+pip install jevkit-lint
+```
+
+## Use
+
+```python
+from jevkit_lint import lint
+
+result = lint({
+    "urgency":  {"type": "noul", "instructions": "How many days has the customer waited?"},
+    "team":     {"type": "choice", "instructions": "Which team should handle this",
+                 "criteria": {"billing": "Payment issues", "technical": "Bugs"}},
+})
+
+print(result.format())
+```
+
+```
+urgency: error [JEV001] Question asks the model to count or do arithmetic.
+    found: How many
+    hint:  jev-1.13 is not a calculator and does not count reliably. Iterate the
+           candidates in code, ask one Noul per item, and sum the answers yourself.
+team: warning [JEV008] Choice has no no-match option.
+    found: billing, technical
+    hint:  A Choice always returns one of its options. With nothing meaning 'none of
+           these', a state that fits no option still produces a confident-looking
+           answer. Add an explicit none/unknown option, or gate on a separate
+           presence Noul.
+```
+
+`result.ok` is `True` when nothing rose to an error, so it drops straight into a
+guard:
+
+```python
+if not lint(questions, state).ok:
+    raise ValueError("refusing to send a request that will not answer what we meant")
+```
+
+## CLI
+
+```bash
+jevkit-lint request.json           # a {state, questions} object, or bare questions
+jevkit-lint cassette.jevl          # lint every recorded request
+jevkit-lint - < request.json       # stdin
+jevkit-lint request.json --strict  # exit non-zero on warnings too
+jevkit-lint request.json --format json
+jevkit-lint --list-rules
+```
+
+Exit codes: `0` clean, `1` findings, `2` bad usage.
+
+## Rules
+
+Each rule names the documented failure mode it comes from.
+
+| Code | Rule | Failure mode |
+| --- | --- | --- |
+| JEV001 | math-and-counting | Math and Numbers |
+| JEV002 | date-comparison | Date and time comparison |
+| JEV003 | generation-request | Generation |
+| JEV004 | negation | Literal reading |
+| JEV005 | vague-scoping | Literal reading |
+| JEV006 | indirection | Indirection |
+| JEV007 | noul-polarity | Contradictory instructions and criteria |
+| JEV008 | choice-no-match | Common-sense structural invariants |
+| JEV009 | choice-arity | Common-sense structural invariants |
+| JEV010 | option-descriptions | Literal reading |
+| JEV011 | score-levels | Literal reading |
+| JEV012 | instructions-present | Literal reading |
+| JEV013 | question-id-reference | Indirection |
+| JEV014 | state-budget | Large state full of irrelevant detail |
+| JEV015 | total-budget | Large state full of irrelevant detail |
+| JEV016 | state-noise-ratio | Large state full of irrelevant detail |
+| JEV017 | numeric-representation | Math and Numbers |
+| JEV018 | duplicate-questions | Common-sense structural invariants |
+
+Select or suppress by code:
+
+```python
+lint(questions, select=["JEV001", "JEV002"])
+lint(questions, ignore=["JEV008"])
+```
+
+## What it cannot do
+
+Adversarial content is a documented failure mode and is **not** linted. Whether
+a state is hostile depends on the content at runtime, not on the question
+definition, so a static rule would be theatre. Screen untrusted state at
+request time instead.
+
+Token counts are estimates. jevkit deliberately does not bundle a tokenizer:
+TypeSafe does not publish which one Jev uses, and a confidently wrong count is
+worse than an honest approximation. The estimator errs conservative, so leave
+headroom near the limits.
+
+## License
+
+MIT

jevkit_lint-0.1.0/README.md

@@ -0,0 +1,115 @@
+# jevkit-lint
+
+Static linter for [TypeSafe](https://typesafe.ai) Jev questions.
+
+TypeSafe publishes a [list of failure modes](https://docs.typesafe.ai/model-jaggedness/jev-1.13)
+for `jev-1.13`: it reads instructions literally, it cannot count, it reads dates
+as text rather than ordered quantities, it loses accuracy on indirection. Most
+of those are visible in your question definitions before you send anything.
+
+`jevkit-lint` reads the definitions and tells you. It never calls the API, so it
+needs no key and costs nothing to run in CI.
+
+> Unofficial and unaffiliated with TypeSafe.
+
+## Install
+
+```bash
+pip install jevkit-lint
+```
+
+## Use
+
+```python
+from jevkit_lint import lint
+
+result = lint({
+    "urgency":  {"type": "noul", "instructions": "How many days has the customer waited?"},
+    "team":     {"type": "choice", "instructions": "Which team should handle this",
+                 "criteria": {"billing": "Payment issues", "technical": "Bugs"}},
+})
+
+print(result.format())
+```
+
+```
+urgency: error [JEV001] Question asks the model to count or do arithmetic.
+    found: How many
+    hint:  jev-1.13 is not a calculator and does not count reliably. Iterate the
+           candidates in code, ask one Noul per item, and sum the answers yourself.
+team: warning [JEV008] Choice has no no-match option.
+    found: billing, technical
+    hint:  A Choice always returns one of its options. With nothing meaning 'none of
+           these', a state that fits no option still produces a confident-looking
+           answer. Add an explicit none/unknown option, or gate on a separate
+           presence Noul.
+```
+
+`result.ok` is `True` when nothing rose to an error, so it drops straight into a
+guard:
+
+```python
+if not lint(questions, state).ok:
+    raise ValueError("refusing to send a request that will not answer what we meant")
+```
+
+## CLI
+
+```bash
+jevkit-lint request.json           # a {state, questions} object, or bare questions
+jevkit-lint cassette.jevl          # lint every recorded request
+jevkit-lint - < request.json       # stdin
+jevkit-lint request.json --strict  # exit non-zero on warnings too
+jevkit-lint request.json --format json
+jevkit-lint --list-rules
+```
+
+Exit codes: `0` clean, `1` findings, `2` bad usage.
+
+## Rules
+
+Each rule names the documented failure mode it comes from.
+
+| Code | Rule | Failure mode |
+| --- | --- | --- |
+| JEV001 | math-and-counting | Math and Numbers |
+| JEV002 | date-comparison | Date and time comparison |
+| JEV003 | generation-request | Generation |
+| JEV004 | negation | Literal reading |
+| JEV005 | vague-scoping | Literal reading |
+| JEV006 | indirection | Indirection |
+| JEV007 | noul-polarity | Contradictory instructions and criteria |
+| JEV008 | choice-no-match | Common-sense structural invariants |
+| JEV009 | choice-arity | Common-sense structural invariants |
+| JEV010 | option-descriptions | Literal reading |
+| JEV011 | score-levels | Literal reading |
+| JEV012 | instructions-present | Literal reading |
+| JEV013 | question-id-reference | Indirection |
+| JEV014 | state-budget | Large state full of irrelevant detail |
+| JEV015 | total-budget | Large state full of irrelevant detail |
+| JEV016 | state-noise-ratio | Large state full of irrelevant detail |
+| JEV017 | numeric-representation | Math and Numbers |
+| JEV018 | duplicate-questions | Common-sense structural invariants |
+
+Select or suppress by code:
+
+```python
+lint(questions, select=["JEV001", "JEV002"])
+lint(questions, ignore=["JEV008"])
+```
+
+## What it cannot do
+
+Adversarial content is a documented failure mode and is **not** linted. Whether
+a state is hostile depends on the content at runtime, not on the question
+definition, so a static rule would be theatre. Screen untrusted state at
+request time instead.
+
+Token counts are estimates. jevkit deliberately does not bundle a tokenizer:
+TypeSafe does not publish which one Jev uses, and a confidently wrong count is
+worse than an honest approximation. The estimator errs conservative, so leave
+headroom near the limits.
+
+## License
+
+MIT

jevkit_lint-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[project]
+name = "jevkit-lint"
+version = "0.1.0"
+description = "Static linter for TypeSafe Jev questions. Catches the documented jev-1.13 failure modes before you spend a token. No API key required."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+authors = [{ name = "Prajjwal Chittori" }]
+keywords = ["jev", "typesafe", "system-one", "lint", "linter", "static-analysis"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Software Development :: Quality Assurance",
+]
+dependencies = ["jevkit-core>=0.1.0"]
+
+[project.scripts]
+jevkit-lint = "jevkit_lint.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/pjdurden/jevkit-py"
+Issues = "https://github.com/pjdurden/jevkit-py/issues"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/jevkit_lint"]

jevkit_lint-0.1.0/src/jevkit_lint/__init__.py

@@ -0,0 +1,14 @@
+"""Static linter for TypeSafe Jev questions.
+
+Catches the failure modes TypeSafe documents for jev-1.13 before you spend a
+token on them. Needs no API key.
+"""
+
+from .diagnostic import Diagnostic, Severity
+from .linter import LintResult, lint
+from .rules import RULES, Rule, all_codes
+
+__version__ = "0.1.0"
+
+__all__ = ["lint", "LintResult", "Diagnostic", "Severity", "Rule", "RULES",
+           "all_codes", "__version__"]

jevkit_lint-0.1.0/src/jevkit_lint/cli.py

@@ -0,0 +1,140 @@
+"""``jevkit-lint`` command line interface."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from typing import Any
+
+from jevkit_core import RecordFormatError, read_records
+
+from .diagnostic import Severity
+from .linter import lint
+from .rules import RULES
+
+EXIT_OK = 0
+EXIT_FINDINGS = 1
+EXIT_USAGE = 2
+
+
+def _load_payload(path: str) -> list[tuple[str, Any, dict[str, Any]]]:
+    """Return (label, state, questions) triples from a file.
+
+    Accepts a `.jevl` record file, a request object with `state` and
+    `questions`, or a bare questions object.
+    """
+    if path == "-":
+        text = sys.stdin.read()
+        source = "<stdin>"
+    else:
+        with open(path, "r", encoding="utf-8") as fh:
+            text = fh.read()
+        source = path
+
+    if path.endswith(".jevl"):
+        import io
+        out = []
+        for i, record in enumerate(read_records(io.StringIO(text))):
+            out.append((f"{source}#{i}", record.state, record.questions))
+        return out
+
+    data = json.loads(text)
+    if not isinstance(data, dict):
+        raise ValueError(f"{source}: expected a JSON object at the top level")
+
+    if "questions" in data and isinstance(data["questions"], dict):
+        return [(source, data.get("state", ""), data["questions"])]
+    return [(source, "", data)]
+
+
+def _print_rules() -> None:
+    width = max(len(r.name) for r in RULES)
+    print("code    name" + " " * (width - 4) + "  documented failure mode")
+    print("-" * (8 + width + 2 + 40))
+    for rule in RULES:
+        print(f"{rule.code}  {rule.name:<{width}}  {rule.mode}")
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="jevkit-lint",
+        description="Statically lint TypeSafe Jev questions against the documented "
+                    "jev-1.13 failure modes. Never calls the API.",
+    )
+    parser.add_argument("files", nargs="*",
+                        help="JSON request files, .jevl record files, or - for stdin")
+    parser.add_argument("--select", metavar="CODES",
+                        help="comma-separated rule codes to run exclusively")
+    parser.add_argument("--ignore", metavar="CODES",
+                        help="comma-separated rule codes to skip")
+    parser.add_argument("--format", choices=("text", "json"), default="text")
+    parser.add_argument("--strict", action="store_true",
+                        help="exit non-zero on warnings too, not just errors")
+    parser.add_argument("--list-rules", action="store_true", help="list rules and exit")
+    parser.add_argument("--no-color", action="store_true")
+    args = parser.parse_args(argv)
+
+    if args.list_rules:
+        _print_rules()
+        return EXIT_OK
+
+    if not args.files:
+        parser.error("no input files (use - to read stdin, or --list-rules)")
+
+    select = args.select.split(",") if args.select else None
+    ignore = args.ignore.split(",") if args.ignore else None
+    color = sys.stdout.isatty() and not args.no_color
+
+    payloads: list[tuple[str, Any, dict[str, Any]]] = []
+    for path in args.files:
+        try:
+            payloads.extend(_load_payload(path))
+        except (OSError, ValueError, RecordFormatError) as exc:
+            print(f"jevkit-lint: {exc}", file=sys.stderr)
+            return EXIT_USAGE
+
+    reports = []
+    worst = Severity.INFO
+    any_error = any_warning = False
+
+    for label, state, questions in payloads:
+        try:
+            result = lint(questions, state, select=select, ignore=ignore)
+        except ValueError as exc:
+            print(f"jevkit-lint: {label}: {exc}", file=sys.stderr)
+            return EXIT_USAGE
+        any_error = any_error or bool(result.errors)
+        any_warning = any_warning or bool(result.warnings)
+        reports.append((label, result))
+
+    if args.format == "json":
+        print(json.dumps(
+            {"results": [{"source": label, **result.to_dict()} for label, result in reports]},
+            indent=2,
+        ))
+    else:
+        for label, result in reports:
+            if len(payloads) > 1:
+                print(f"== {label}")
+            print(result.format(color=color))
+            if len(payloads) > 1:
+                print()
+        total = {"error": 0, "warning": 0, "info": 0}
+        for _, result in reports:
+            for key, value in result.counts().items():
+                total[key] += value
+        if sum(total.values()):
+            print(f"\n{total['error']} error(s), {total['warning']} warning(s), "
+                  f"{total['info']} info")
+
+    del worst
+    if any_error:
+        return EXIT_FINDINGS
+    if args.strict and any_warning:
+        return EXIT_FINDINGS
+    return EXIT_OK
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

jevkit_lint-0.1.0/src/jevkit_lint/diagnostic.py

@@ -0,0 +1,57 @@
+"""Diagnostics produced by the linter."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from enum import Enum
+from typing import Any
+
+__all__ = ["Severity", "Diagnostic"]
+
+
+class Severity(str, Enum):
+    ERROR = "error"      # Will very likely produce wrong answers or be rejected.
+    WARNING = "warning"  # A documented failure mode is in play.
+    INFO = "info"        # Worth a look; may be intentional.
+
+    def __str__(self) -> str:  # pragma: no cover - display only
+        return self.value
+
+
+_ORDER = {Severity.ERROR: 0, Severity.WARNING: 1, Severity.INFO: 2}
+
+
+@dataclass(frozen=True)
+class Diagnostic:
+    code: str
+    severity: Severity
+    question_id: str | None
+    message: str
+    hint: str
+    evidence: str = ""
+
+    @property
+    def sort_key(self) -> tuple[int, str, str]:
+        return (_ORDER[self.severity], self.code, self.question_id or "")
+
+    def format(self, *, color: bool = False) -> str:
+        loc = self.question_id or "<request>"
+        head = f"{loc}: {self.severity.value} [{self.code}] {self.message}"
+        if color:
+            tint = {"error": "\033[31m", "warning": "\033[33m", "info": "\033[36m"}
+            head = f"{tint[self.severity.value]}{head}\033[0m"
+        lines = [head]
+        if self.evidence:
+            lines.append(f"    found: {self.evidence}")
+        lines.append(f"    hint:  {self.hint}")
+        return "\n".join(lines)
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "code": self.code,
+            "severity": self.severity.value,
+            "question_id": self.question_id,
+            "message": self.message,
+            "hint": self.hint,
+            "evidence": self.evidence,
+        }

jevkit_lint-0.1.0/src/jevkit_lint/linter.py

@@ -0,0 +1,92 @@
+"""The lint entry point."""
+
+from __future__ import annotations
+
+from typing import Any, Iterable
+
+from jevkit_core import normalize_questions
+
+from .diagnostic import Diagnostic, Severity
+from .rules import LintContext, rules_for
+
+__all__ = ["lint", "LintResult"]
+
+
+class LintResult:
+    """Diagnostics for one request, ordered most severe first."""
+
+    def __init__(self, diagnostics: list[Diagnostic]) -> None:
+        self.diagnostics = sorted(diagnostics, key=lambda d: d.sort_key)
+
+    def __iter__(self):
+        return iter(self.diagnostics)
+
+    def __len__(self) -> int:
+        return len(self.diagnostics)
+
+    def __bool__(self) -> bool:
+        return bool(self.diagnostics)
+
+    def by_severity(self, severity: Severity) -> list[Diagnostic]:
+        return [d for d in self.diagnostics if d.severity is severity]
+
+    @property
+    def errors(self) -> list[Diagnostic]:
+        return self.by_severity(Severity.ERROR)
+
+    @property
+    def warnings(self) -> list[Diagnostic]:
+        return self.by_severity(Severity.WARNING)
+
+    @property
+    def infos(self) -> list[Diagnostic]:
+        return self.by_severity(Severity.INFO)
+
+    @property
+    def ok(self) -> bool:
+        """True when nothing rose to an error."""
+        return not self.errors
+
+    def counts(self) -> dict[str, int]:
+        return {
+            "error": len(self.errors),
+            "warning": len(self.warnings),
+            "info": len(self.infos),
+        }
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "ok": self.ok,
+            "counts": self.counts(),
+            "diagnostics": [d.to_dict() for d in self.diagnostics],
+        }
+
+    def format(self, *, color: bool = False) -> str:
+        if not self.diagnostics:
+            return "No problems found."
+        return "\n".join(d.format(color=color) for d in self.diagnostics)
+
+
+def lint(
+    questions: dict[str, Any],
+    state: Any = "",
+    *,
+    select: Iterable[str] | None = None,
+    ignore: Iterable[str] | None = None,
+) -> LintResult:
+    """Lint a jev request without calling the API.
+
+    ``questions`` accepts SDK objects or plain dicts. ``state`` is optional: omit
+    it to check the questions alone, though the budget rules can only report
+    meaningfully when the real state is supplied.
+    """
+    normalized = normalize_questions(questions)
+    ctx = LintContext.build(state, normalized)
+    ignored = {c.upper() for c in (ignore or ())}
+
+    found: list[Diagnostic] = []
+    for rule in rules_for(select):
+        if rule.code in ignored:
+            continue
+        found.extend(rule.check(ctx))
+    return LintResult(found)