cr-proc 0.1.0__tar.gz

cr_proc-0.1.0/PKG-INFO

@@ -0,0 +1,142 @@
+Metadata-Version: 2.4
+Name: cr_proc
+Version: 0.1.0
+Summary: 
+Author: Ethan Dye
+Author-email: mrtops03@gmail.com
+Requires-Python: >=3.14
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: py-jsonl (>=1.3.22,<2.0.0)
+Description-Content-Type: text/markdown
+
+# `code_recorder_processor`
+
+[![CI](https://github.com/BYU-CS-Course-Ops/code_recorder_processor/actions/workflows/ci.yml/badge.svg)](https://github.com/BYU-CS-Course-Ops/code_recorder_processor/actions/workflows/ci.yml)
+
+This contains code to process and verify the `*.recorder.jsonl.gz` files that
+are produced by the
+[jetbrains-recorder](https://github.com/BYU-CS-Course-Ops/jetbrains-recorder).
+
+## Installation
+
+Install the package and its dependencies using Poetry:
+
+```bash
+poetry install
+```
+
+## Usage
+
+The processor can be run using the `cr_proc` command with two arguments:
+
+```bash
+poetry run cr_proc <path-to-jsonl-file> <path-to-template-file>
+```
+
+### Arguments
+
+- `<path-to-jsonl-file>`: Path to the compressed JSONL file
+  (`*.recorder.jsonl.gz`) produced by the jetbrains-recorder
+- `<path-to-template-file>`: Path to the initial template file that was recorded
+
+### Options
+
+- `--time-limit MINUTES`: (Optional) Maximum allowed time in minutes between the
+  first and last edit in the recording. If the elapsed time exceeds this limit,
+  the recording is flagged as suspicious. Useful for detecting unusually long
+  work sessions or potential external assistance.
+
+### Example
+
+```bash
+poetry run cr_proc homework0.recording.jsonl.gz homework0.py
+```
+
+With time limit flag:
+
+```bash
+poetry run cr_proc homework0.recording.jsonl.gz homework0.py --time-limit 30
+```
+
+This will flag the recording if more than 30 minutes elapsed between the first and last edit.
+
+The processor will:
+
+1. Load the recorded events from the JSONL file
+2. Verify that the initial event matches the template (allowances for newline
+   differences are made)
+3. Reconstruct the final file state by applying all recorded events
+4. Output the reconstructed file contents to stdout
+
+### Output
+
+The reconstructed file is printed to stdout. Any warnings or errors are printed
+to stderr, including:
+
+- The document path being processed
+- Suspicious copy-paste and AI activity indicators
+
+### Suspicious Activity Detection
+
+The processor automatically detects and reports three types of suspicious activity
+patterns:
+
+#### 1. Time Limit Exceeded
+
+When the `--time-limit` flag is specified, the processor flags recordings where
+the elapsed time between the first and last edit exceeds the specified limit.
+This can indicate unusually long work sessions or potential external assistance.
+
+**Example warning:**
+
+```
+Time limit exceeded!
+  Limit: 30 minutes
+  Elapsed: 45.5 minutes
+  First edit: 2025-01-15T10:00:00+00:00
+  Last edit: 2025-01-15T10:45:30+00:00
+```
+
+#### 2. External Copy-Paste (Multi-line Pastes)
+
+The processor flags multi-line additions (more than one line) that do not appear
+to be copied from within the document itself. These indicate content pasted from
+external sources.
+
+**Example warning:**
+
+```
+Event #15 (multi-line external paste): 5 lines, 156 chars - newFragment: def helper_function():...
+```
+
+#### 3. Rapid One-line Pastes (AI Indicator)
+
+When 3 or more single-line pastes occur within a 1-second window, this is
+flagged as a potential AI activity indicator. Human typing does not typically
+produce this pattern; rapid sequential pastes suggest automated code generation.
+
+**Example warning:**
+
+```
+Events #42-#44 (rapid one-line pastes (AI indicator)): 3 lines, 89 chars
+```
+
+### Error Handling
+
+If verification fails (the recorded initial state doesn't match the template),
+the processor will:
+
+- Print an error message to stderr
+- Display a diff showing the differences
+- Exit with status code 1
+
+If file loading or processing errors occur, the processor will:
+
+- Print a descriptive error message to stderr
+- Exit with status code 1
+
+## Future Ideas
+
+- Check for odd typing behavior
+

cr_proc-0.1.0/README.md

@@ -0,0 +1,129 @@
+# `code_recorder_processor`
+
+[![CI](https://github.com/BYU-CS-Course-Ops/code_recorder_processor/actions/workflows/ci.yml/badge.svg)](https://github.com/BYU-CS-Course-Ops/code_recorder_processor/actions/workflows/ci.yml)
+
+This contains code to process and verify the `*.recorder.jsonl.gz` files that
+are produced by the
+[jetbrains-recorder](https://github.com/BYU-CS-Course-Ops/jetbrains-recorder).
+
+## Installation
+
+Install the package and its dependencies using Poetry:
+
+```bash
+poetry install
+```
+
+## Usage
+
+The processor can be run using the `cr_proc` command with two arguments:
+
+```bash
+poetry run cr_proc <path-to-jsonl-file> <path-to-template-file>
+```
+
+### Arguments
+
+- `<path-to-jsonl-file>`: Path to the compressed JSONL file
+  (`*.recorder.jsonl.gz`) produced by the jetbrains-recorder
+- `<path-to-template-file>`: Path to the initial template file that was recorded
+
+### Options
+
+- `--time-limit MINUTES`: (Optional) Maximum allowed time in minutes between the
+  first and last edit in the recording. If the elapsed time exceeds this limit,
+  the recording is flagged as suspicious. Useful for detecting unusually long
+  work sessions or potential external assistance.
+
+### Example
+
+```bash
+poetry run cr_proc homework0.recording.jsonl.gz homework0.py
+```
+
+With time limit flag:
+
+```bash
+poetry run cr_proc homework0.recording.jsonl.gz homework0.py --time-limit 30
+```
+
+This will flag the recording if more than 30 minutes elapsed between the first and last edit.
+
+The processor will:
+
+1. Load the recorded events from the JSONL file
+2. Verify that the initial event matches the template (allowances for newline
+   differences are made)
+3. Reconstruct the final file state by applying all recorded events
+4. Output the reconstructed file contents to stdout
+
+### Output
+
+The reconstructed file is printed to stdout. Any warnings or errors are printed
+to stderr, including:
+
+- The document path being processed
+- Suspicious copy-paste and AI activity indicators
+
+### Suspicious Activity Detection
+
+The processor automatically detects and reports three types of suspicious activity
+patterns:
+
+#### 1. Time Limit Exceeded
+
+When the `--time-limit` flag is specified, the processor flags recordings where
+the elapsed time between the first and last edit exceeds the specified limit.
+This can indicate unusually long work sessions or potential external assistance.
+
+**Example warning:**
+
+```
+Time limit exceeded!
+  Limit: 30 minutes
+  Elapsed: 45.5 minutes
+  First edit: 2025-01-15T10:00:00+00:00
+  Last edit: 2025-01-15T10:45:30+00:00
+```
+
+#### 2. External Copy-Paste (Multi-line Pastes)
+
+The processor flags multi-line additions (more than one line) that do not appear
+to be copied from within the document itself. These indicate content pasted from
+external sources.
+
+**Example warning:**
+
+```
+Event #15 (multi-line external paste): 5 lines, 156 chars - newFragment: def helper_function():...
+```
+
+#### 3. Rapid One-line Pastes (AI Indicator)
+
+When 3 or more single-line pastes occur within a 1-second window, this is
+flagged as a potential AI activity indicator. Human typing does not typically
+produce this pattern; rapid sequential pastes suggest automated code generation.
+
+**Example warning:**
+
+```
+Events #42-#44 (rapid one-line pastes (AI indicator)): 3 lines, 89 chars
+```
+
+### Error Handling
+
+If verification fails (the recorded initial state doesn't match the template),
+the processor will:
+
+- Print an error message to stderr
+- Display a diff showing the differences
+- Exit with status code 1
+
+If file loading or processing errors occur, the processor will:
+
+- Print a descriptive error message to stderr
+- Exit with status code 1
+
+## Future Ideas
+
+- Check for odd typing behavior

cr_proc-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "cr_proc"
+version = "0.1.0"
+description = ""
+authors = [
+    {name = "Ethan Dye",email = "mrtops03@gmail.com"}
+]
+readme = "README.md"
+requires-python = ">=3.14"
+dependencies = [
+    "py-jsonl (>=1.3.22,<2.0.0)"
+]
+
+[tool.poetry]
+packages = [{include = "code_recorder_processor", from = "src"}]
+
+[tool.poetry.scripts]
+cr_proc = "code_recorder_processor.cli:main"
+test = "pytest:main"
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[dependency-groups]
+dev = [
+    "mdformat (>=1.0.0,<2.0.0)",
+    "pytest (>=9.0.2,<10.0.0)"
+]

cr_proc-0.1.0/src/code_recorder_processor/api/build.py

@@ -0,0 +1,212 @@
+from datetime import datetime
+from typing import  Any, Optional
+
+
+def _parse_ts(ts: str) -> datetime:
+    """Parse ISO 8601 timestamps that end with 'Z' (UTC)."""
+    # Convert trailing Z to +00:00 for fromisoformat
+    return datetime.fromisoformat(ts.replace("Z", "+00:00"))
+
+
+def _normalize_newlines(text: str) -> str:
+    """Normalize CRLF to LF for consistent offset handling."""
+    return text.replace("\r\n", "\n")
+
+
+def _apply_event_exact_text(doc: str, offset: int, old: str, new: str) -> str:
+    """Apply an event assuming Python string indices (Unicode code points)."""
+    return doc[:offset] + new + doc[offset + len(old):]
+
+
+def _apply_event_fuzzy_text(doc: str, offset: int, old: str, new: str, window: int) -> str:
+    """
+    Apply an event with a fuzzy fallback in Python string space:
+    - If old == '', it's a pure insertion at offset (clamped).
+    - If exact match fails, search for old around the offset within ±window chars and
+      replace the nearest occurrence.
+    """
+    if old == "":
+        off = max(0, min(offset, len(doc)))
+        return doc[:off] + new + doc[off:]
+
+    # Try exact match first
+    if doc[offset:offset + len(old)] == old:
+        return _apply_event_exact_text(doc, offset, old, new)
+
+    # Fuzzy search around the offset
+    start = max(0, offset - window)
+    end = min(len(doc), offset + window)
+    best_pos, best_dist = None, None
+
+    i = start
+    while True:
+        i = doc.find(old, i, end)
+        if i == -1:
+            break
+        dist = abs(i - offset)
+        if best_dist is None or dist < best_dist:
+            best_pos, best_dist = i, dist
+        i += 1
+
+    if best_pos is not None:
+        return _apply_event_exact_text(doc, best_pos, old, new)
+
+    raise ValueError(
+        f"Old fragment not found near offset {offset}. old={old!r}, new={new!r}"
+    )
+
+
+def _apply_event_exact_utf16(doc_bytes: bytes, offset_units: int, old: str, new: str) -> bytes:
+    """
+    Apply an event in UTF-16-LE byte space (JetBrains Document uses UTF-16 code units):
+    offset_units is measured in code units; each unit is 2 bytes in UTF-16-LE.
+    """
+    # Convert offset in code units to byte index
+    bidx = offset_units * 2
+    old_b = old.encode("utf-16-le")
+    new_b = new.encode("utf-16-le")
+    return doc_bytes[:bidx] + new_b + doc_bytes[bidx + len(old_b):]
+
+
+def _apply_event_fuzzy_utf16(doc_bytes: bytes, offset_units: int, old: str, new: str, window_units: int) -> bytes:
+    """
+    Fuzzy apply in UTF-16-LE byte space:
+    - If old == '', pure insertion at offset_units.
+    - Else try exact match, otherwise search for old within ±window_units and replace nearest occurrence.
+    """
+    bidx = offset_units * 2
+    old_b = old.encode("utf-16-le")
+    new_b = new.encode("utf-16-le")
+
+    if old == "":
+        # Insertion: clamp to bounds
+        bidx = max(0, min(bidx, len(doc_bytes)))
+        return doc_bytes[:bidx] + new_b + doc_bytes[bidx:]
+
+    # Exact match
+    if doc_bytes[bidx:bidx + len(old_b)] == old_b:
+        return _apply_event_exact_utf16(doc_bytes, offset_units, old, new)
+
+    # Fuzzy search around offset
+    start_b = max(0, bidx - window_units * 2)
+    end_b = min(len(doc_bytes), bidx + window_units * 2)
+
+    best_pos_b, best_dist_units = None, None
+    i = start_b
+    while True:
+        i = doc_bytes.find(old_b, i, end_b)
+        if i == -1:
+            break
+        dist_units = abs((i // 2) - offset_units)
+        if best_dist_units is None or dist_units < best_dist_units:
+            best_pos_b, best_dist_units = i, dist_units
+        i += 1
+
+    if best_pos_b is not None:
+        return doc_bytes[:best_pos_b] + new_b + doc_bytes[best_pos_b + len(old_b):]
+
+    raise ValueError(
+        f"Old fragment not found near offset {offset_units} (UTF-16 units). "
+        f"old={old!r}, new={new!r}"
+    )
+
+
+def reconstruct_file_from_events(
+    events: tuple[dict[str, Any], ...],
+    template: str,
+    document_path: Optional[str] = None,
+    *,
+    utf16_mode: bool = False,
+    window: int = 200,
+    normalize_newlines: bool = True,
+) -> str:
+    """
+    Reconstruct the final document by replaying PyCharm/IntelliJ edit events.
+
+    Parameters
+    ----------
+    events : tuple of dict
+        Each dict should contain:
+          - 'timestamp': ISO 8601 string, e.g., '2026-01-13T22:40:44.137341Z'
+          - 'document': absolute path string of the edited file
+          - 'offset': integer offset (JetBrains Document uses UTF-16 code units)
+          - 'oldFragment': string being replaced/removed at offset
+          - 'newFragment': string inserted at offset
+    template_path : Path
+        Path to the template file content to use as the starting point.
+    document_path : Optional[str]
+        If provided, only events matching this path will be applied. If None,
+        the function tries to infer:
+          - If any event matches str(template_path), use those.
+          - If there's only one distinct document in events, use that.
+          - Otherwise raises ValueError.
+    utf16_mode : bool (default: False)
+        If True, treat offsets as UTF-16 code units and apply changes in UTF-16-LE
+        byte space (safer for emoji/astral chars). If False, operate on Python
+        string indices (fine for ASCII/BMP-only source).
+    window : int (default: 200)
+        Fuzzy search window radius around the intended offset. Used when exact
+        match of oldFragment at offset fails; the nearest occurrence within the
+        window is replaced.
+    normalize_newlines : bool (default: True)
+        If True, normalize CRLF to LF in the starting template content. This
+        prevents offset drift if events were recorded using LF.
+
+    Returns
+    -------
+    str
+        The reconstructed final document content.
+
+    Raises
+    ------
+    ValueError
+        - If the target document cannot be determined.
+        - If an edit cannot be applied (oldFragment not found near offset).
+    """
+    # Read template content
+    if normalize_newlines:
+        template = _normalize_newlines(template)
+
+    # Decide which document's events to replay
+    docs = {e.get("document") for e in events}
+    target_doc = document_path
+    template_doc_str = str(template)
+
+    if target_doc is None:
+        if template_doc_str in docs:
+            target_doc = template_doc_str
+        elif len(docs) == 1:
+            target_doc = next(iter(docs))
+        else:
+            raise ValueError(
+                "Ambiguous target document: provide document_path explicitly. "
+                f"Found documents: {sorted(d for d in docs if d is not None)}"
+            )
+
+    # Filter events to the target document and sort by timestamp
+    evs = [e for e in events if e.get("document") == target_doc]
+    evs.sort(key=lambda e: _parse_ts(e["timestamp"]))
+
+    if not evs:
+        # No events for target_doc; return template unchanged
+        return template
+
+    if utf16_mode:
+        # Work in UTF-16-LE byte space
+        doc_bytes = template.encode("utf-16-le")
+        for e in evs:
+            offset_units = int(e["offset"])
+            old = e.get("oldFragment", "")
+            new = e.get("newFragment", "")
+            doc_bytes = _apply_event_fuzzy_utf16(doc_bytes, offset_units, old, new, window_units=window)
+        # Decode back to text
+        return doc_bytes.decode("utf-16-le")
+    else:
+        # Work in Python string space
+        doc = template
+        for e in evs:
+            offset = int(e["offset"])
+            old = e.get("oldFragment", "")
+            new = e.get("newFragment", "")
+            doc = _apply_event_fuzzy_text(doc, offset, old, new, window=window)
+        return doc

cr_proc-0.1.0/src/code_recorder_processor/api/load.py

@@ -0,0 +1,100 @@
+import jsonl
+import zlib
+from gzip import BadGzipFile, open as gzip_open
+from io import StringIO
+from pathlib import Path
+from typing import Any
+
+
+def load_jsonl(file: Path) -> tuple[dict[str, Any], ...]:
+    """
+    Load JSONL data from a file (gzip compressed or plain text).
+
+    Parameters
+    ----------
+    file : Path
+        Path to the JSONL file (can be .jsonl or .jsonl.gz)
+
+    Returns
+    -------
+    tuple[dict[str, Any], ...]
+        Tuple of dictionaries parsed from the JSONL file
+
+    Raises
+    ------
+    FileNotFoundError
+        If the file does not exist
+    IOError
+        If there's an error reading the file
+    ValueError
+        If the file contains invalid JSONL data
+    """
+    if not file.exists():
+        raise FileNotFoundError(f"File not found: {file}")
+
+    if not file.is_file():
+        raise ValueError(f"Path is not a file: {file}")
+
+    def _load_jsonl(source: Any) -> tuple[dict[str, Any], ...]:
+        return tuple(jsonl.load(source))
+
+    def _read_magic(path: Path) -> bytes:
+        try:
+            with path.open("rb") as fh:
+                return fh.read(2)
+        except FileNotFoundError:
+            raise
+        except OSError:
+            return b""
+
+    try:
+        data = _load_jsonl(file)
+    except BadGzipFile:
+        magic = _read_magic(file)
+        looks_gzip = magic == b"\x1f\x8b"
+
+        # If it looks like gzip, try an explicit gzip open before giving up.
+        if looks_gzip:
+            try:
+                with gzip_open(file, "rt", encoding="utf-8") as gz:
+                    data = _load_jsonl(gz)
+            except (BadGzipFile, OSError):
+                data = None
+        else:
+            data = None
+
+        if data is None:
+            # If gzip stream is broken, attempt a lenient zlib decompress to salvage content.
+            try:
+                raw = file.read_bytes()
+                dobj = zlib.decompressobj(16 + zlib.MAX_WBITS)
+                text_bytes = dobj.decompress(raw) + dobj.flush()
+                text = text_bytes.decode("utf-8", errors="replace")
+                data = _load_jsonl(StringIO(text))
+            except Exception:
+                data = None
+
+        if data is None:
+            # Fall back to plain text even if the header hinted gzip.
+            try:
+                with file.open("r", encoding="utf-8", errors="replace") as plain_file:
+                    data = _load_jsonl(plain_file)
+            except FileNotFoundError as e:
+                raise FileNotFoundError(f"Error reading file {file}: {e}")
+            except ValueError as e:
+                raise ValueError(f"Invalid JSONL format in {file} (plain read fallback): {e}")
+            except Exception as e:
+                raise IOError(
+                    f"Error loading JSONL file {file} without compression (magic={magic!r}): {type(e).__name__}: {e}"
+                )
+    except FileNotFoundError as e:
+        raise FileNotFoundError(f"Error reading file {file}: {e}")
+    except ValueError as e:
+        raise ValueError(f"Invalid JSONL format in {file}: {e}")
+    except Exception as e:
+        raise IOError(f"Error loading JSONL file {file}: {type(e).__name__}: {e}")
+
+    if not data:
+        raise ValueError(f"JSONL file is empty: {file}")
+
+    return data

cr_proc-0.1.0/src/code_recorder_processor/api/verify.py

@@ -0,0 +1,412 @@
+from typing import Any
+from datetime import datetime
+import difflib
+
+
+def _normalize_newlines(text: str) -> str:
+    """Normalize CRLF to LF to avoid offset and diff noise."""
+    return text.replace("\r\n", "\n")
+
+
+def is_only_whitespace_differences(template: str, actual: str) -> bool:
+    """
+    Return True if `actual` can be derived from `template` by changing only
+    whitespace (spaces, tabs, newlines). All non-whitespace characters must
+    appear in the same order with no additions, deletions, or substitutions.
+    """
+    t = _normalize_newlines(template)
+    a = _normalize_newlines(actual)
+
+    lt, la = len(t), len(a)
+    i = j = 0
+
+    while True:
+        # Skip any whitespace on both sides
+        while i < lt and t[i].isspace():
+            i += 1
+        while j < la and a[j].isspace():
+            j += 1
+
+        if i >= lt or j >= la:
+            break
+
+        if t[i] != a[j]:
+            return False
+
+        i += 1
+        j += 1
+
+    # Ensure no remaining non-whitespace characters on either side
+    while i < lt:
+        if not t[i].isspace():
+            return False
+        i += 1
+
+    while j < la:
+        if not a[j].isspace():
+            return False
+        j += 1
+
+    return True
+
+
+def verify_template(template: str, jsonData: tuple[dict[str, Any], ...]) -> str:
+    """
+    Verify the initial event is a faithful snapshot of the template,
+    allowing only whitespace differences in the event's newFragment.
+
+    Also validates that the recorder wrote an initial snapshot
+    (oldFragment == newFragment on the first record).
+
+    Returns the verified fragment text.
+    Raises ValueError if verification fails.
+    """
+    if not jsonData:
+        raise ValueError("jsonData is empty")
+
+    first = jsonData[0]
+    new_frag = _normalize_newlines(first["newFragment"])
+    old_frag = _normalize_newlines(first["oldFragment"])
+    temp_norm = _normalize_newlines(template)
+
+    # Recorder must have written an initial full snapshot:
+    if new_frag != old_frag:
+        raise ValueError("oldFragment does not match newFragment (no initial snapshot)")
+
+    # Accept exact match OR match that differs only by whitespace
+    if new_frag == temp_norm:
+        return new_frag
+
+    if is_only_whitespace_differences(temp_norm, new_frag):
+        return new_frag
+
+    raise ValueError("newFragment does not match template (differs by more than whitespace)")
+
+
+def template_diff(template: str, jsonData: tuple[dict[str, Any], ...]) -> str:
+    """
+    Produce a unified diff between the given template and the first event's newFragment.
+    If the only differences are whitespace, return ''.
+    """
+    if not jsonData:
+        return ""
+
+    template_norm = _normalize_newlines(template)
+    actual_norm = _normalize_newlines(jsonData[0]["newFragment"])
+
+    # Suppress diff if only whitespace differences
+    if is_only_whitespace_differences(template_norm, actual_norm):
+        return ""
+
+    # Generate a proper unified diff (headers on separate lines)
+    t_lines = template_norm.splitlines(keepends=True)
+    a_lines = actual_norm.splitlines(keepends=True)
+
+    diff_iter = difflib.unified_diff(
+        t_lines,
+        a_lines,
+        fromfile="template",
+        tofile="actual",
+        n=3,
+        lineterm="\n",
+    )
+    return "".join(diff_iter)
+
+
+def _detect_multiline_external_pastes(jsonData: tuple[dict[str, Any], ...]) -> list[dict[str, Any]]:
+    """
+    Detect multi-line copy-paste events from external sources.
+
+    Flags newFragments that are significant in length (more than one line)
+    and do not appear to be copied from within the document itself.
+
+    Returns a list of suspicious multi-line paste events.
+    """
+    suspicious_events = []
+
+    # Build a history of all document content seen so far
+    document_history = set()
+
+    for idx, event in enumerate(jsonData):
+        old_frag = _normalize_newlines(event.get("oldFragment", ""))
+        new_frag = _normalize_newlines(event.get("newFragment", ""))
+
+        # Skip if no actual change
+        if new_frag == old_frag or new_frag.strip() == "":
+            continue
+
+        # Only check multi-line content (more than 2 lines means at least 2 actual lines)
+        new_lines = new_frag.split("\n")
+        if len(new_lines) <= 2:  # Single line or line + empty
+            continue
+
+        # Check if the new content appears to be from within the document
+        is_internal_copy = False
+
+        # Check if new_frag content was present in any previous fragments
+        for hist_content in document_history:
+            # Ignore tiny fragments; they appear everywhere and cause false positives
+            if len(hist_content) < 20:
+                continue
+
+            # Require substantial overlap in size to count as an internal copy
+            similar_length = (
+                len(hist_content) >= 0.8 * len(new_frag)
+                and len(hist_content) <= 1.25 * len(new_frag)
+            )
+
+            if new_frag == hist_content:
+                is_internal_copy = True
+                break
+
+            if new_frag in hist_content and similar_length:
+                is_internal_copy = True
+                break
+
+            if hist_content in new_frag and similar_length:
+                is_internal_copy = True
+                break
+
+        # Also check if it's in the old fragment (internal move/copy)
+        if not is_internal_copy and old_frag and (new_frag in old_frag or old_frag in new_frag):
+            is_internal_copy = True
+
+        if not is_internal_copy:
+            suspicious_events.append({
+                "event_index": idx,
+                "line_count": len(new_lines),
+                "char_count": len(new_frag),
+                "reason": "multi-line external paste",
+                "newFragment": new_frag[:100] + ("..." if len(new_frag) > 100 else ""),
+            })
+
+        # Update history after analysis so the current fragment cannot mask itself
+        if len(old_frag) > 1:
+            document_history.add(old_frag)
+        if len(new_frag) > 1:
+            document_history.add(new_frag)
+
+    return suspicious_events
+
+
+def _detect_rapid_paste_sequences(jsonData: tuple[dict[str, Any], ...]) -> list[dict[str, Any]]:
+    """
+    Detect rapid sequences of one-line pastes (AI assistance indicator).
+
+    Identifies clusters of 3+ one-line paste events occurring within 1 second,
+    which may indicate AI-assisted code generation.
+
+    Returns a list of suspicious rapid-paste events.
+    """
+    suspicious_events = []
+
+    # Track one-line paste events for rapid-paste detection
+    one_line_pastes = []
+
+    for idx, event in enumerate(jsonData):
+        new_frag = _normalize_newlines(event.get("newFragment", ""))
+        old_frag = _normalize_newlines(event.get("oldFragment", ""))
+        timestamp = event.get("timestamp")
+
+        # Skip if no timestamp or no change
+        if not timestamp or new_frag == old_frag or new_frag.strip() == "":
+            continue
+
+        # Check if newFragment is a single line (2 elements = 1 line + trailing \n)
+        new_lines = new_frag.split("\n")
+        if len(new_lines) == 2:
+            # Heuristic: if it's more than a few characters, it might be pasted
+            if len(new_frag.strip()) > 5:
+                one_line_pastes.append({
+                    "event_index": idx,
+                    "timestamp": timestamp,
+                    "content": new_frag
+                })
+
+    # Analyze one-line pastes for rapid clusters
+    if not one_line_pastes:
+        return suspicious_events
+
+    def parse_ts(ts_str: str) -> datetime:
+        return datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
+
+    i = 0
+    while i < len(one_line_pastes):
+        cluster = [one_line_pastes[i]]
+        cluster_start = parse_ts(one_line_pastes[i]["timestamp"])
+
+        # Look ahead for more pastes within 1 second
+        j = i + 1
+        while j < len(one_line_pastes):
+            current_time = parse_ts(one_line_pastes[j]["timestamp"])
+            if (current_time - cluster_start).total_seconds() <= 1.0:
+                cluster.append(one_line_pastes[j])
+                j += 1
+            else:
+                break
+
+        # If we found 3+ one-line pastes within 1 second, flag it
+        if len(cluster) >= 3:
+            event_indices = [p["event_index"] for p in cluster]
+            suspicious_events.append({
+                "event_index": event_indices[0],
+                "event_indices": event_indices,
+                "line_count": len(cluster),
+                "char_count": sum(len(p["content"]) for p in cluster),
+                "reason": "rapid one-line pastes (AI indicator)",
+                "newFragment": f"{len(cluster)} one-line pastes in 1 second",
+            })
+
+        i = j if j > i + 1 else i + 1
+
+    return suspicious_events
+
+
+def detect_external_copypaste(jsonData: tuple[dict[str, Any], ...]) -> list[dict[str, Any]]:
+    """
+    Detect copy-paste events from external sources and AI-assisted coding patterns.
+
+    Combines detection of:
+    1. Multi-line external paste events (content not from within document)
+    2. Rapid one-line paste sequences (potential AI assistance indicator)
+
+    Returns a list of all suspicious events with metadata.
+    """
+    suspicious_events = []
+
+    # Detect multi-line external pastes
+    suspicious_events.extend(_detect_multiline_external_pastes(jsonData))
+
+    # Detect rapid one-line paste sequences (AI indicator)
+    suspicious_events.extend(_detect_rapid_paste_sequences(jsonData))
+
+    return suspicious_events
+
+
+def check_time_limit(jsonData: tuple[dict[str, Any], ...], time_limit_minutes: int | None) -> dict[str, Any] | None:
+    """
+    Check if the time between first and last edit exceeds the specified time limit.
+
+    Tracks elapsed editing time across sessions by summing actual editing time within
+    each session (excluding gaps between sessions). For the time limit check, compares
+    the span from the first timestamp to the last timestamp overall.
+
+    Parameters
+    ----------
+    jsonData : tuple[dict[str, Any], ...]
+        The event data from the JSONL file
+    time_limit_minutes : int | None
+        Maximum allowed time in minutes between first and last overall edit.
+        If None, no time limit is enforced.
+
+    Returns
+    -------
+    dict[str, Any] | None
+        A dictionary with time limit and elapsed time info.
+        Contains 'exceeds_limit' flag and always includes 'minutes_elapsed'.
+    """
+    if not jsonData:
+        return None
+
+    def parse_ts(ts_str: str) -> datetime:
+        return datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
+
+    # Identify session boundaries: sessions start at indices where oldFragment == newFragment (non-empty)
+    session_starts = [0]  # First session always starts at index 0
+    for idx in range(1, len(jsonData)):
+        old_frag = jsonData[idx].get("oldFragment", "")
+        new_frag = jsonData[idx].get("newFragment", "")
+        if old_frag == new_frag and old_frag.strip() != "":
+            session_starts.append(idx)
+
+    # Add sentinel to mark end of last session
+    session_starts.append(len(jsonData))
+
+    # Find first and last timestamps overall
+    first_timestamp_overall = None
+    last_timestamp_overall = None
+
+    for event in jsonData:
+        if event.get("timestamp"):
+            if first_timestamp_overall is None:
+                first_timestamp_overall = event["timestamp"]
+            last_timestamp_overall = event["timestamp"]
+
+    if first_timestamp_overall is None or last_timestamp_overall is None:
+        # Not enough events with timestamps
+        return None
+
+    # Calculate elapsed time by summing editing time within each session
+    total_minutes_elapsed = 0.0
+
+    for i in range(len(session_starts) - 1):
+        session_start = session_starts[i]
+        session_end = session_starts[i + 1]
+
+        # Find first and last events with timestamps in this session
+        first_event = None
+        last_event = None
+
+        for event in jsonData[session_start:session_end]:
+            if event.get("timestamp"):
+                if first_event is None:
+                    first_event = event
+                last_event = event
+
+        # If this session has timestamped events, add its elapsed time
+        if first_event is not None and last_event is not None:
+            try:
+                first_time = parse_ts(first_event["timestamp"])
+                last_time = parse_ts(last_event["timestamp"])
+                session_diff = last_time - first_time
+                total_minutes_elapsed += session_diff.total_seconds() / 60
+            except (ValueError, KeyError):
+                # Timestamp parsing failed for this session, skip it
+                continue
+
+    # For time limit check, use the span from first to last timestamp overall
+    try:
+        first_time_overall = parse_ts(first_timestamp_overall)
+        last_time_overall = parse_ts(last_timestamp_overall)
+        overall_span_minutes = (last_time_overall - first_time_overall).total_seconds() / 60
+    except (ValueError, KeyError):
+        # Timestamp parsing failed
+        return None
+
+    result = {
+        "time_limit_minutes": time_limit_minutes,
+        "minutes_elapsed": round(total_minutes_elapsed, 2),
+        "first_timestamp": first_timestamp_overall,
+        "last_timestamp": last_timestamp_overall,
+    }
+
+    # For time limit check, compare the overall span (first to last timestamp) against the limit
+    if time_limit_minutes is not None:
+        result["exceeds_limit"] = overall_span_minutes > time_limit_minutes
+    else:
+        result["exceeds_limit"] = False
+
+    return result
+
+
+def verify(template: str, jsonData: tuple[dict[str, Any], ...]) -> tuple[str, list[dict[str, Any]]]:
+    """
+    Comprehensive verification of recorded code events.
+
+    Performs:
+    1. Template verification (initial snapshot matches template)
+    2. External copy-paste detection
+
+    Returns:
+        tuple: (verified_template_text, list_of_suspicious_copypaste_events)
+
+    Raises:
+        ValueError: If template verification fails
+    """
+    # Verify template
+    verified_template = verify_template(template, jsonData)
+
+    # Detect external copy-paste events
+    suspicious_events = detect_external_copypaste(jsonData)
+
+    return verified_template, suspicious_events

cr_proc-0.1.0/src/code_recorder_processor/cli.py

@@ -0,0 +1,183 @@
+import argparse
+import sys
+import json
+from datetime import datetime
+from pathlib import Path
+from .api.load import load_jsonl
+from .api.verify import verify, template_diff, check_time_limit
+from .api.build import reconstruct_file_from_events
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Process and verify code recorder JSONL files"
+    )
+    parser.add_argument(
+        "jsonl_file",
+        type=Path,
+        help="Path to the compressed JSONL file (*.recording.jsonl.gz)",
+    )
+    parser.add_argument(
+        "template_file",
+        type=Path,
+        help="Path to the initial template file that was recorded",
+    )
+    parser.add_argument(
+        "--time-limit",
+        type=int,
+        default=None,
+        help="Maximum allowed time in minutes between first and last edit. If exceeded, recording is flagged.",
+    )
+    parser.add_argument(
+        "--document",
+        type=str,
+        default=None,
+        help=("Document path or filename to process from the recording. "
+              "Defaults to the document whose extension matches the template file."),
+    )
+    parser.add_argument(
+        "--output-json",
+        type=Path,
+        default=None,
+        help="Path to output JSON file with verification results (time info and suspicious events).",
+    )
+
+    args = parser.parse_args()
+
+    # Load JSONL file first to get document path
+    try:
+        jsonData = load_jsonl(args.jsonl_file)
+    except FileNotFoundError as e:
+        print(f"Error: {e}", file=sys.stderr)
+        return 1
+    except (ValueError, IOError) as e:
+        print(f"Error loading JSONL file: {e}", file=sys.stderr)
+        return 1
+
+    # Decide which recorded document to process
+    documents = {e.get("document") for e in jsonData if "document" in e and e.get("document") is not None}
+    recorded_docs = sorted([d for d in documents if d is not None])
+
+    def resolve_document(docs: list[str], template_path: Path, override: str | None) -> str | None:
+        if not docs:
+            return None
+
+        if override:
+            matches = [d for d in docs if d.endswith(override) or Path(d).name == override]
+            if not matches:
+                raise ValueError(
+                    f"No document in recording matches '{override}'. Available: {docs}"
+                )
+            if len(matches) > 1:
+                raise ValueError(
+                    f"Ambiguous document override '{override}'. Matches: {matches}"
+                )
+            return matches[0]
+
+        template_ext = template_path.suffix
+        ext_matches = [d for d in docs if Path(d).suffix == template_ext]
+        if len(ext_matches) == 1:
+            return ext_matches[0]
+        if len(ext_matches) > 1:
+            raise ValueError(
+                f"Multiple documents share extension '{template_ext}': {ext_matches}. "
+                "Use --document to choose one."
+            )
+
+        if len(docs) == 1:
+            return docs[0]
+
+        raise ValueError(
+            "Could not determine document to process. Use --document to select one. "
+            f"Available documents: {docs}"
+        )
+
+    try:
+        target_document = resolve_document(recorded_docs, args.template_file, args.document)
+    except ValueError as e:
+        print(f"Error determining document: {e}", file=sys.stderr)
+        return 1
+
+    if target_document:
+        doc_events = tuple(e for e in jsonData if e.get("document") == target_document)
+        if not doc_events:
+            print(f"Error: No events found for document '{target_document}'", file=sys.stderr)
+            return 1
+    else:
+        doc_events = jsonData
+
+    print(f"Processing: {target_document or args.template_file}", file=sys.stderr)
+
+    # Read template file
+    try:
+        templateData = args.template_file.read_text()
+    except FileNotFoundError:
+        print(f"Error: Template file not found: {args.template_file}", file=sys.stderr)
+        return 1
+    except Exception as e:
+        print(f"Error reading template file: {e}", file=sys.stderr)
+        return 1
+
+    # Check time limit and display elapsed time
+    time_info = check_time_limit(doc_events, args.time_limit)
+    if time_info:
+        print(f"Elapsed editing time: {time_info['minutes_elapsed']} minutes", file=sys.stderr)
+        print(f"Time span (first to last edit): {(datetime.fromisoformat(time_info['last_timestamp'].replace('Z', '+00:00')) - datetime.fromisoformat(time_info['first_timestamp'].replace('Z', '+00:00'))).total_seconds() / 60:.2f} minutes", file=sys.stderr)
+        if time_info['exceeds_limit']:
+            print(f"\nTime limit exceeded!", file=sys.stderr)
+            print(f"  Limit: {time_info['time_limit_minutes']} minutes", file=sys.stderr)
+            print(f"  First edit: {time_info['first_timestamp']}", file=sys.stderr)
+            print(f"  Last edit: {time_info['last_timestamp']}", file=sys.stderr)
+
+    # Verify and process
+    try:
+        templateData, suspicious_events = verify(templateData, doc_events)
+        print(reconstruct_file_from_events(doc_events, templateData, document_path=target_document))
+
+        # Prepare results for JSON output
+        results = {
+            "document": target_document or str(args.template_file),
+            "time_info": time_info,
+            "suspicious_events": suspicious_events,
+        }
+
+        if suspicious_events:
+            print("\nSuspicious copy-paste events detected:", file=sys.stderr)
+            for ev in suspicious_events:
+                reason = ev.get('reason', 'unknown')
+                indices = ev.get('event_indices', [ev['event_index']])
+                if len(indices) > 1:
+                    print(f"  Events #{indices[0]}-#{indices[-1]} ({reason}): "
+                          f"{ev['line_count']} lines, {ev['char_count']} chars", file=sys.stderr)
+                else:
+                    print(f"  Event #{ev['event_index']} ({reason}): "
+                          f"{ev['line_count']} lines, {ev['char_count']} chars - "
+                          f"newFragment: {ev['newFragment']}", file=sys.stderr)
+        else:
+            print("Success! No suspicious events detected.", file=sys.stderr)
+
+        # Write JSON output if requested
+        if args.output_json:
+            try:
+                args.output_json.parent.mkdir(parents=True, exist_ok=True)
+                with open(args.output_json, 'w') as f:
+                    json.dump(results, f, indent=2)
+                print(f"Results written to {args.output_json}", file=sys.stderr)
+            except Exception as e:
+                print(f"Error writing JSON output: {e}", file=sys.stderr)
+                return 1
+    except ValueError as e:
+        print("File failed verification from template!", file=sys.stderr)
+        print(str(e), file=sys.stderr)
+        try:
+            print(template_diff(templateData, doc_events), file=sys.stderr)
+        except Exception:
+            pass
+        return 1
+    except Exception as e:
+        print(f"Error processing file: {type(e).__name__}: {e}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())