mostlyright-data 0.9.0__py3-none-any.whl

mostlyright/data_harness/fast_delimited.py

@@ -0,0 +1,239 @@
+"""One certified fast row splitter for delimited text, and the closed subset it is certified over.
+
+Two places in the harness turn delimited bytes into rows: the Reader family's
+``decode_delimited_stream``, which normalizes a publisher's file into the canonical sealed
+artifact, and the build kernel's ``_stream_csv_rows``, which reads that artifact back.  Both
+reach the stdlib ``csv`` module, whose tokenizer is already C.  What costs the time at
+year scale is not the tokenizing but the per-cell Python objects on either side of it, and
+what costs the memory is decoding a whole year-scale file into one Python ``str`` before the
+first row is looked at.
+
+This module addresses both without moving a single sealed byte, by doing less rather than by
+doing something different.
+
+The certified subset
+--------------------
+``certifies`` answers one question about exact bytes: is this input inside a subset where
+splitting on line terminators and then on the delimiter is *provably* the same tokenizing the
+stdlib parser performs?  The subset is closed and each member of it is a refusal the stdlib
+parser would otherwise have had to reason about:
+
+* **No quote character anywhere.**  A quote is the only byte that makes the stdlib tokenizer's
+  state machine leave its plain-field state -- it opens a quoted field, doubles into a literal
+  quote, or ends one.  With no quote in the input, the machine never leaves ``IN_FIELD``, so a
+  field ends at the delimiter or at the line terminator and nowhere else.
+* **No carriage return anywhere.**  A lone carriage return terminates a record for the stdlib
+  reader, and a carriage return inside a quoted value survives into the value.  Excluding the
+  byte removes both cases, leaving the newline as the one record terminator.
+* **No NUL byte anywhere.**  Both call sites refuse a NUL; excluding it here means the
+  per-cell scan each of them performs is provably answering "no", so it does not have to run.
+* **No empty record.**  The stdlib reader yields an empty row for a blank line, which is not
+  what splitting produces, and which the callers refuse on width.  An input with a blank line
+  is left to the stdlib reader so that it refuses it in its own words, at its own row.
+* **Strict UTF-8 over the whole input, and a non-empty input.**  Validity is settled here,
+  over every byte, before a single row is yielded -- so an input whose last byte is invalid is
+  never certified, and is refused by the existing path exactly as early as it always was.
+
+Anything outside the subset is not approximated and not repaired: ``certifies`` returns
+``False`` and the caller runs its existing stdlib path, untouched, from the beginning.  There
+is no partial fast lane and no mid-input hand-over, so the only inputs this module has an
+opinion about are inputs whose tokenizing has one possible answer.
+
+Why this is byte-equivalence rather than a close approximation
+--------------------------------------------------------------
+For an input in the subset the claim is small enough to check by reading it.  The stdlib
+tokenizer starts each record in ``START_FIELD``; with no quote and no escape character
+configured, the only transitions available are "delimiter ends this field", "line terminator
+ends this field and this record", and "any other character joins this field".  That is the
+definition of splitting the record on the delimiter.  The records themselves are the input
+split on the newline, with the terminator of the final record dropped when the input ends
+with one.  The differential tests compare the two implementations over random inputs drawn
+from the subset and its boundary rather than trusting this paragraph.
+
+The one refusal the splitter has to make for itself
+---------------------------------------------------
+Splitting removes every refusal the state machine could raise except one: the stdlib parser
+refuses a field longer than ``csv.field_size_limit()`` characters, and splitting has no such
+bound.  That refusal is *reproduced* rather than dropped or approximated.  ``certified_rows``
+raises the same ``csv.Error`` the stdlib parser raises, in the same place in the iteration --
+before the row carrying the oversized field is yielded -- so the callers' existing
+``except csv.Error`` handlers turn it into the refusal they always turned it into.
+
+A block shorter than the limit cannot contain a field longer than it, so in the ordinary case
+the per-cell comparison is skipped for the whole block rather than run and passed.
+
+The limit is read at parse time rather than at import, because it is process-global and
+mutable, and once per block rather than once per row.  That granularity is stated rather than
+hidden: a caller that changes the limit *while rows are being pulled* has it take effect at
+the next block instead of at the next row, where the stdlib parser would apply it at the next
+character.  Reading it per row instead costs about a sixth of the splitting at year scale, to
+buy exactness in a case that is already racy -- the limit is one process-global integer, and
+nothing in this repository moves it except a test that restores it afterwards.
+
+Bounded memory
+--------------
+Rows are produced from blocks cut at record terminators, so the decoded text alive at any
+moment is one block rather than the whole input.  A block always ends at a newline (or at the
+end of the input), which is never inside a multi-byte UTF-8 sequence, so decoding block by
+block admits exactly what decoding the whole input admits.
+
+The bound is the block, not one row: a block is split into its lines at once, so the live
+cost is one ``str`` per line of one block.  For short records that per-line overhead is
+several times the block's own bytes, which is why the block is sized in tens of kilobytes
+rather than megabytes.  Measured at the worst record shape this can be given -- two-byte
+records, which is one ``str`` object per two bytes of input -- the transient is about 1.2 MB
+at 64 KiB blocks, against 76 MB at 4 MiB blocks.  It does not grow with the input.
+"""
+
+from __future__ import annotations
+
+import codecs
+import csv
+from collections.abc import Iterator
+
+__all__ = [
+    "CERTIFIED_BLOCK_BYTES",
+    "certified_rows",
+    "certifies",
+]
+
+# The decoded text alive while rows are pulled.  Two things pin it rather than throughput: the
+# per-line ``str`` overhead of splitting a block is a multiple of the block's own size, and a
+# block below ``csv.field_size_limit()``'s default cannot carry an over-limit field, so the
+# ordinary input skips the per-cell comparison entirely.
+CERTIFIED_BLOCK_BYTES = 64 * 1024
+
+# The window the up-front validity pass feeds the incremental decoder.  Separate from the row
+# block size because this pass keeps nothing it decodes; it only has to answer whether every
+# byte of the input is strict UTF-8.
+_VALIDATION_WINDOW_BYTES = 1 * 1024 * 1024
+
+# The bytes whose presence takes an input out of the subset, each with the state-machine
+# transition it is here to rule out.
+_QUOTE_BYTE = b'"'
+_CARRIAGE_RETURN_BYTE = b"\r"
+_NUL_BYTE = b"\x00"
+_EMPTY_RECORD = b"\n\n"
+_RECORD_TERMINATOR = b"\n"
+
+
+def certifies(content: bytes, *, delimiter: str) -> bool:
+    """Answer whether these exact bytes lie inside the certified subset.
+
+    The scans are the whole of the decision and each one is a single pass the interpreter
+    performs in C.  Validity of the text is settled last because it is the only pass whose
+    cost is proportional to the work rather than to the search, and because an input that
+    fails a cheaper test never reaches it.
+
+    A delimiter this module cannot reason about -- one that is not a single character, or one
+    that is itself a byte the subset excludes -- is not certified rather than special-cased.
+    The caller's own delimiter contract is stricter, so this is a second lock on the same
+    door rather than the only one.
+    """
+
+    if not isinstance(content, (bytes, bytearray)):
+        return False
+    if not isinstance(delimiter, str) or len(delimiter) != 1 or delimiter in '"\r\n\x00':
+        return False
+    if not content:
+        return False
+    if content.startswith(_RECORD_TERMINATOR):
+        return False
+    if _QUOTE_BYTE in content:
+        return False
+    if _CARRIAGE_RETURN_BYTE in content:
+        return False
+    if _NUL_BYTE in content:
+        return False
+    if _EMPTY_RECORD in content:
+        return False
+    return _is_strict_utf8(content)
+
+
+def _is_strict_utf8(content: bytes) -> bool:
+    """Decide strict UTF-8 validity over every byte without holding the decoded text.
+
+    The incremental decoder carries the partial sequence across window boundaries, so a
+    multi-byte character split by a window is decoded rather than refused, and the final call
+    refuses an input that ends mid-sequence.  Nothing it returns is kept: the question is
+    whether the whole input decodes, and the answer is a boolean.
+    """
+
+    decoder = codecs.getincrementaldecoder("utf-8")("strict")
+    view = memoryview(content)
+    try:
+        for start in range(0, len(content), _VALIDATION_WINDOW_BYTES):
+            decoder.decode(view[start : start + _VALIDATION_WINDOW_BYTES])
+        decoder.decode(b"", final=True)
+    except UnicodeDecodeError:
+        return False
+    return True
+
+
+def certified_rows(
+    content: bytes,
+    *,
+    delimiter: str,
+    block_bytes: int = CERTIFIED_BLOCK_BYTES,
+) -> Iterator[list[str]]:
+    """Yield each record of a certified input as its list of cells.
+
+    The caller has already had ``certifies`` answer ``True`` for these exact bytes; this
+    function does not re-decide it, because a second, differently worded copy of the subset
+    is exactly how two statements of one rule drift apart.  It performs none of the caller's
+    admission -- no width, no budget, no NUL -- so that every refusal an input can draw is
+    raised by the caller that always raised it, in the caller's own vocabulary, at the
+    caller's own row.
+
+    The one exception is the stdlib parser's own field-size refusal, which belongs to the
+    tokenizing rather than to any caller, and is therefore reproduced here as the same
+    ``csv.Error``, before the row carrying the oversized field is yielded.
+    """
+
+    if block_bytes < 1:
+        # A block that cannot hold a byte cannot advance the cursor, and the loop below would
+        # never terminate.  Refused rather than clamped, because a caller asking for it has a
+        # bug rather than a preference.
+        raise ValueError("block_bytes must be at least one byte")
+    view = memoryview(content)
+    total = len(content)
+    position = 0
+    while position < total:
+        window = min(position + block_bytes, total)
+        cut = content.rfind(_RECORD_TERMINATOR, position, window)
+        if cut < 0:
+            # One record is longer than a block.  Take it whole rather than splitting a record
+            # across two blocks: a block that ends anywhere but a record terminator would need
+            # state carried between blocks, which is the complication this module exists
+            # without.  The caller's field and row budgets are what bound such a record.
+            cut = content.find(_RECORD_TERMINATOR, window)
+        stop = total if cut < 0 else cut + 1
+        block = str(view[position:stop], "utf-8")
+        position = stop
+        lines = block.split("\n")
+        if lines and lines[-1] == "":
+            # The terminator of the block's final record, which is not itself a record.  Only
+            # the last block of an input that does not end with a terminator keeps its tail.
+            lines.pop()
+        # Read per block rather than once: the limit is process-global and mutable, and the
+        # stdlib parser consults it as it goes rather than remembering what it was.
+        limit = csv.field_size_limit()
+        # A block no longer than the limit cannot contain a field longer than it, so the
+        # comparison below is already answered for every cell in it.
+        checked = len(block) > limit
+        # The block's own text is redundant once it has been split into separately allocated
+        # lines, and the generator's locals outlive each yield -- so it is released here rather
+        # than being held alive while the next block is cut.
+        block = ""
+        if checked:
+            for line in lines:
+                cells = line.split(delimiter)
+                for cell in cells:
+                    if len(cell) > limit:
+                        raise csv.Error(f"field larger than field limit ({limit})")
+                yield cells
+        else:
+            for line in lines:
+                yield line.split(delimiter)
+        # Likewise the lines, which are all spent by now.
+        lines = []

mostlyright/data_harness/fleet.py

@@ -0,0 +1,237 @@
+"""Scan a directory of runs once and report which runs need operator action.
+
+The command binds no port. ``--html`` writes a self-contained document to stdout. State publication
+uses ``os.replace``, so the scan reads without taking the coordinator lock.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import stat
+from collections.abc import Mapping, Sequence
+from pathlib import Path
+from typing import Any
+
+from mostlyright.data_harness import events
+from mostlyright.data_harness.nbrender.parse import esc
+from mostlyright.data_harness.watch import stylesheet
+
+FLEET_SCHEMA_VERSION = "mr-fleet.v1"
+
+# A root is walked one level deep and no further, and no more than this many children are read. A
+# directory of runs is a working directory, not a filesystem to crawl (T-30-35, T-30-36).
+MAX_FLEET_RUNS = 500
+
+_STATE_PATH = (".mr-data", "state.json")
+_MAX_STATE_BYTES = 4 * 1024 * 1024
+
+# Display labels for persisted run phases.
+_PLAIN_STATE: Mapping[str, str] = {
+    "initialized": "not started",
+    "planned": "ready to build",
+    "building": "building",
+    "candidate_built": "built",
+    "review_recorded": "reviewed",
+    "failed": "stopped",
+}
+
+_WAITING = "waiting for a person to look at it"
+_RIGHTS_REASON = "stopped: the rights position needs a decision"
+# Keep completed reviews that require fixes distinct from reviews that have not started.
+_FIXES_REASON = "reviewed: the review asked for fixes"
+
+
+def _read_json(name: str | os.PathLike[str], *, dir_fd: int | None = None) -> Any:
+    """Read bounded JSON through one nonblocking, no-follow descriptor.
+
+    ``dir_fd`` anchors every path component beneath the validated child directory. Disk and decode
+    failures return ``None``.
+    """
+
+    try:
+        return json.loads(
+            events.read_regular_bytes(name, dir_fd=dir_fd, max_bytes=_MAX_STATE_BYTES).decode(
+                "utf-8"
+            )
+        )
+    except (OSError, ValueError, RecursionError):
+        return None
+
+
+def _feed_dirs(child: Path) -> list[Path]:
+    """The feed directories that could belong to this child, newest-run-shape first.
+
+    ``events.run_feed_dir`` scopes a feed by the run's own name, so a workspace's feed is at
+    ``<child>/.mr-events/result/`` and a bare run directory's is at
+    ``<root>/.mr-events/<child name>/``. A child can only be one of the two, and a directory that
+    does not exist simply yields no feed.
+    """
+
+    found: list[Path] = []
+    for run_dir in (child / "result", child):
+        try:
+            found.append(events.run_feed_dir(run_dir))
+        except ValueError:
+            continue
+    return found
+
+
+def _terminal_record(child: Path) -> Mapping[str, Any] | None:
+    """Return the terminal record selected by the shared attempt selector, if present."""
+
+    for feed_dir in _feed_dirs(child):
+        _path, _size, records = events.select_narrated_feed(feed_dir)
+        if not records:
+            continue
+        last = records[-1]
+        if last.get("event") in events.TERMINAL_EVENT_NAMES:
+            return last
+        return None
+    return None
+
+
+def _run_directory(child: Path) -> Path | None:
+    """Return the directory holding ``candidate/`` for this child, or ``None``.
+
+    ``lstat``, for the reason ``scan_fleet`` already gives about a symlinked child: a sealed
+    candidate directory is one ``build_candidate`` renamed into place, never a link, and a link at
+    that name would let anything on this disk answer "this run built" for a run that did not.
+    """
+
+    for candidate in (child / "result", child):
+        try:
+            if stat.S_ISDIR(os.lstat(candidate / "candidate").st_mode):
+                return candidate
+        except OSError:
+            continue
+    return None
+
+
+def _failure_reason(record: Mapping[str, Any]) -> str:
+    facts = record.get("facts")
+    facts = facts if isinstance(facts, Mapping) else {}
+    if record.get("event") == "run_interrupted":
+        return "stopped: the build was interrupted before it finished"
+    finding = facts.get("finding_id")
+    if isinstance(finding, str) and finding.startswith("RIGHTS_"):
+        # Kept distinct on purpose: a rights escalation is a decision a person owes the run, and
+        # burying it in a generic failure line is how it gets missed.
+        return _RIGHTS_REASON
+    message = facts.get("message")
+    if isinstance(message, str) and message:
+        return f"stopped: {message}"
+    return "stopped: the build did not finish"
+
+
+def scan_fleet(root: str | os.PathLike[str]) -> list[dict[str, Any]]:
+    """Scan ``root``'s IMMEDIATE children and return one row per run, in sorted name order.
+
+    A child that is neither a workspace nor a run directory is skipped silently -- a directory of
+    runs routinely holds notes, archives and half-finished experiments, and a scan that complained
+    about each one would be a scan nobody runs twice. A symlinked child is skipped without being
+    followed.
+    """
+
+    base = Path(root)
+    rows: list[dict[str, Any]] = []
+    try:
+        names = sorted(entry.name for entry in os.scandir(base))
+    except OSError:
+        return rows
+
+    for name in names[:MAX_FLEET_RUNS]:
+        child = base / name
+        try:
+            info = os.lstat(child)
+        except OSError:
+            continue
+        if not stat.S_ISDIR(info.st_mode):
+            continue  # never follow a symlinked child, never read a file as a run
+
+        # Anchor the state read to the validated child descriptor so no component follows a link.
+        child_fd = None
+        try:
+            child_fd = events.open_directory(child)
+            state = _read_json(os.path.join(*_STATE_PATH), dir_fd=child_fd)
+        except OSError:
+            state = None
+        finally:
+            if child_fd is not None:
+                os.close(child_fd)
+        run_dir = _run_directory(child)
+        terminal = _terminal_record(child)
+        if state is None and run_dir is None:
+            continue
+
+        phase = state.get("phase") if isinstance(state, Mapping) else None
+        if isinstance(phase, str):
+            plain = _PLAIN_STATE.get(phase, "state not recorded")
+        else:
+            plain = "built" if run_dir is not None else "state not recorded"
+
+        # A sealed candidate outranks a terminal feed record. Persisted ``failed`` state remains
+        # authoritative when no sealed candidate exists.
+        if run_dir is not None:
+            terminal = None
+
+        # `phase == "review_recorded"` covers both review verdicts. Read `verified_status` from the
+        # review record.
+        review = state.get("review") if isinstance(state, Mapping) else None
+        verdict = review.get("verified_status") if isinstance(review, Mapping) else None
+
+        reasons: list[str] = []
+        if phase == "failed" or terminal is not None:
+            plain = "stopped"
+            reasons.append(
+                _failure_reason(terminal) if terminal is not None else "stopped: the build failed"
+            )
+        elif verdict == "fixes_required":
+            # Ahead of the waiting check on purpose: a recorded verdict is durable state, and it
+            # outranks "is there a review directory" whether or not one was ever written.
+            reasons.append(_FIXES_REASON)
+        elif run_dir is not None and not (run_dir / "review").is_dir():
+            reasons.append(_WAITING)
+
+        rows.append(
+            {
+                # Sanitize only the reported name. Filesystem calls above use the original bytes.
+                "name": events.writable_text(name),
+                "state": plain,
+                "needs_person": bool(reasons),
+                "reasons": reasons,
+            }
+        )
+    return rows
+
+
+def render_fleet_html(rows: Sequence[Mapping[str, Any]]) -> str:
+    """Render the scan as one self-contained page. Printed to stdout, never served."""
+
+    waiting = sum(1 for row in rows if row.get("needs_person"))
+    body = "".join(
+        "<tr>"
+        f"<td>{esc(row.get('name', ''))}</td>"
+        f"<td>{esc(row.get('state', ''))}</td>"
+        f"<td>{esc('yes' if row.get('needs_person') else 'no')}</td>"
+        f"<td>{esc('; '.join(str(reason) for reason in row.get('reasons') or []))}</td>"
+        "</tr>"
+        for row in rows
+    )
+    return (
+        "<!doctype html>\n"
+        '<html lang="en"><head><meta charset="utf-8">'
+        '<meta name="viewport" content="width=device-width, initial-scale=1">'
+        "<title>Runs</title>"
+        f"{stylesheet()}"
+        "</head><body>\n"
+        '<div class="mr-watch-page">'
+        '<h1 class="mr-watch-title">Runs</h1>'
+        f'<p class="mr-watch-run">{esc(len(rows))} runs. '
+        f"{esc(waiting)} need a person.</p>"
+        '<table class="mr-watch-table"><thead><tr>'
+        "<th>Run</th><th>State</th><th>Needs a person</th><th>Why</th>"
+        f"</tr></thead><tbody>{body}</tbody></table>"
+        "</div>\n"
+        "</body></html>\n"
+    )