sawmill-parser 0.3.2__tar.gz → 0.4.0__tar.gz

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/PKG-INFO

@@ -1,10 +1,10 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: sawmill-parser
-Version: 0.3.2
+Version: 0.4.0
 Summary: A configurable TUI for post-processing and filtering structured tool logs
-Project-URL: Homepage, https://github.com/sawmill/sawmill
-Project-URL: Documentation, https://github.com/sawmill/sawmill#readme
-Project-URL: Repository, https://github.com/sawmill/sawmill
+Project-URL: Homepage, https://github.com/sjalloq/sawmill
+Project-URL: Documentation, https://sjalloq.github.io/sawmill
+Project-URL: Repository, https://github.com/sjalloq/sawmill
 Author: Sawmill Contributors
 License: MIT
 Keywords: eda,filter,log,terminal,tui

sawmill_parser-0.4.0/SAWMILL_STREAMING_SPEC.md

@@ -0,0 +1,66 @@
+# Sawmill performance: findings and plan
+
+Working notes for making chip-level logs (100 MB+, 1M+ messages) usable.
+Supersedes the earlier streaming spec, whose diagnosis was wrong.
+
+## 1. What was measured (2026-11-04)
+
+Synthetic Xcelium log, 200k messages, 24 MB, Python 3.14, Textual DataTable.
+
+| Step | Time |
+|---|---|
+| Plugin parse alone | 0.7 s |
+| TUI load until table drawn | 9.4 s |
+| Filter logic excluding table rebuild | 0.14 s |
+| Clear filter / sort / severity toggle | 6 to 9 s |
+| First scroll after rebuild | 5 s |
+| RSS after parse / after table populated | 219 MB / 404 MB |
+
+- About 90 % of load time and 95 % of every interactive action is spent
+  inside `DataTable.add_row` and the deferred `_update_dimensions` pass that
+  measures every cell. DataTable virtualises rendering, not population.
+- Parse is bound by Python object creation (dict per match, Message, FileRef,
+  metadata dict), not regex (under 20 %) or I/O (about 6 %).
+- Memory per Message was about 820 B: 184 B metadata dict, 167 B raw_text,
+  104 B slots object, 86 B content (always a substring of raw_text), plus
+  unshared copies of message_id, category and file path.
+
+## 2. Done
+
+- Plugin hook `load_and_parse` returns `Iterator[Message]` (plugin API 2).
+  `sawmill.core.plugin.load_messages()` wraps list-returning plugins.
+  Plugins open the file eagerly so errors surface at call time.
+- `Message` and `FileRef` are `@dataclass(slots=True)` with int coercion.
+- Xcelium plugin: single regex, no per-match dict, interned strings, shared
+  per-tool metadata dict. 820 B to 520 B per message, output identical.
+- Benchmarks in `tools/bench/` report current and peak RSS and go through
+  the real filter helpers.
+
+## 3. In progress
+
+- Windowed DataTable: the table holds a window of rows around the cursor;
+  `_filtered_messages` stays the full list. Branch `perf/windowed-table`.
+- One waiver match per message; redundant list copies removed.
+
+## 4. Next: store investigation
+
+After the view is fixed, measure a real 130 MB log. If memory is still the
+complaint, compare on a branch:
+
+- Intern-only (current design, about 500 B/message, no new dependency).
+- DuckDB: columnar, dictionary-encoded, vectorised filter/sort/group-by,
+  native regex, spill to disk, `LIMIT/OFFSET` paging. Roughly 150 to 250 MB
+  per 1M messages. Cost: 50 MB wheel, 100 ms import, SQL in the app, waiver
+  and suppression logic expressed as predicates.
+- Polars: same idea, dataframe API, lighter, no spill.
+- Hand-rolled columnar (int arrays plus byte offsets, text re-read from
+  mmap): about 40 B/message but a third plugin contract and binary-mode
+  offsets in every plugin. Only if the above are insufficient.
+
+Decide with numbers from the real log, not estimates.
+
+## 5. Dropped
+
+Background parse thread, stats-only mode, Rust hot loop and FSM preamble
+matcher are parked. Parse is under 10 % of load; none of these move the
+user-visible numbers until the view and store are settled.

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/docs/index.rst

@@ -33,6 +33,14 @@ parse each tool's output.
    plugin-guide/pre-defined-filters
    plugin-guide/writing-a-plugin
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Performance
+
+   performance/overview
+   performance/plugin-api-2
+   performance/store-investigation
+
 .. toctree::
    :maxdepth: 2
    :caption: Reference

sawmill_parser-0.4.0/docs/performance/overview.rst

@@ -0,0 +1,103 @@
+Performance overview
+====================
+
+This page records what was measured, what was changed, and the current
+numbers.
+
+What was measured
+-----------------
+
+Measurements were taken on 2026-11-04 with a synthetic Xcelium log from
+``tools/bench/gen_xcelium_log.py`` (200k messages, 24 MB).
+
+.. list-table:: Before
+   :header-rows: 1
+
+   * - Step
+     - Before
+   * - Plugin parse alone
+     - 0.7 s
+   * - TUI load until table drawn
+     - 9.4 s
+   * - Filter logic excluding table rebuild
+     - 0.14 s
+   * - Clear filter / sort / severity toggle
+     - 6 to 9 s
+   * - First scroll after rebuild
+     - 5 s
+   * - RSS after parse / after table drawn
+     - 219 MB / 404 MB
+
+Diagnosis
+---------
+
+- About 90% of load and 95% of each interactive action was Textual
+  ``DataTable.add_row`` plus its deferred ``_update_dimensions`` cell
+  measurement. ``DataTable`` virtualises rendering, not population.
+- Parse was bound by Python object creation (about 40%: per-match dict,
+  ``Message``, ``FileRef``, metadata dict). Regex was under 20%, I/O
+  about 6%.
+- Memory per ``Message`` was about 820 B:
+
+  - metadata dict: 184 B
+  - ``raw_text``: 167 B
+  - slots object: 104 B
+  - ``content``: 86 B (always a substring of ``raw_text``)
+  - unshared ``message_id`` / ``category`` / path strings: about 144 B
+
+What was changed
+----------------
+
+- **Plugin API 2**: ``load_and_parse`` returns ``Iterator[Message]``.
+  API 1 plugins returning lists keep working. See :doc:`plugin-api-2`.
+- **Message model**: ``Message`` and ``FileRef`` are
+  ``@dataclass(slots=True)`` (``FileRef`` is frozen).
+  ``__post_init__`` coerces ``start_line``, ``end_line`` and ``line`` to
+  ``int``. No Pydantic on the hot path.
+- **Windowed table**: the TUI populates only a window of rows around the
+  cursor, ``max(3 × visible rows, 200)``, and re-windows centred on the
+  cursor as it moves. The subtitle shows ``row i of N``.
+- **Waivers**: matched once per message; redundant list copies removed.
+
+Benchmarks
+----------
+
+- ``tools/bench/gen_xcelium_log.py``: synthetic log generator.
+- ``tools/bench/run_bench.py``: size sweep, fresh subprocess per size,
+  current RSS from ``/proc/self/statm`` plus peak.
+- ``tools/bench/bench_message.py``: ``Message`` construction cost.
+
+Current numbers
+---------------
+
+.. list-table:: 200k messages
+   :header-rows: 1
+
+   * - Action
+     - Before
+     - After
+   * - Load
+     - 9.7 s
+     - 0.24 s
+   * - Clear filter
+     - 8.87 s
+     - 0.12 s
+   * - Sort
+     - 8.93 s
+     - 0.26 s
+   * - Severity toggle
+     - 6.66 s
+     - 0.18 s
+   * - RSS after table drawn
+     - 332 MB
+     - 158 MB
+
+At 1M messages:
+
+- parse: 3.5 s
+- load: 0.49 s
+- filter / sort / toggles: 0.36 to 0.70 s
+- RSS: about 550 MB after parse, 660 MB at end
+
+A column store was also evaluated and not adopted; see
+:doc:`store-investigation`.

sawmill_parser-0.4.0/docs/performance/plugin-api-2.rst

@@ -0,0 +1,97 @@
+Plugin API 2
+============
+
+``SAWMILL_PLUGIN_API`` is ``"2"``. The change is to ``load_and_parse``,
+which now streams messages.
+
+The contract
+------------
+
+.. code-block:: python
+
+   @hookspec
+   def load_and_parse(self, path: Path) -> Iterator[Message]: ...
+
+- Plugins should be generators reading the file line by line.
+- Yield each ``Message`` as soon as it is complete.
+- The ``SawmillPlugin`` base class default is a generator that yields
+  nothing.
+- ``[tool.sawmill] plugin_api`` in a plugin's ``pyproject.toml`` is
+  informational; nothing enforces it.
+
+Migrating from API 1
+--------------------
+
+API 1 returned ``list[Message]``. To migrate:
+
+- Replace ``messages.append(msg)`` with ``yield msg``.
+- Drop the final ``return messages``.
+- Flush any in-flight multi-line message after the loop with a final
+  ``yield``.
+
+The ``load_messages`` shim
+--------------------------
+
+The base app calls plugins through ``sawmill.core.plugin.load_messages``:
+
+.. code-block:: python
+
+   def load_messages(plugin, path) -> Iterator[Message]:
+       return iter(plugin.load_and_parse(path))
+
+A legacy list is wrapped with ``iter()``, so API 1 plugins keep working
+unchanged. They do not get the memory benefit.
+
+Eager-open pattern
+------------------
+
+A plain generator does not run until the first ``next()``, so a missing
+file would raise deep inside the consumer. Open the file in an outer
+function and return an inner generator over the handle:
+
+.. code-block:: python
+
+   @hookimpl
+   def load_and_parse(self, path):
+       f = open(path, errors="replace")   # raises here if missing
+       return self._iter_messages(f)
+
+   def _iter_messages(self, f):
+       try:
+           current = None
+           for line_num, line in enumerate(f, start=1):
+               ...  # build or extend current; yield when complete
+           if current is not None:
+               yield current             # flush the in-flight message
+       finally:
+           f.close()
+
+Caveat: a generator that is never iterated keeps its file handle open
+until garbage collection.
+
+Allocation guidance
+-------------------
+
+Parse time is dominated by Python object creation, not regex or I/O.
+
+- Read with ``for line in f``, never ``readlines()``.
+- Use one combined regex per line rather than two tried in sequence.
+- Use ``m.groups()`` or a ``NamedTuple``, not a dict per line.
+- Intern repeated strings (tool, rule / message ID, file path) through a
+  small cache dict.
+- Share one ``metadata`` dict per constant value, or omit ``metadata``.
+
+Measured effect on an Xcelium plugin at 200k messages, output identical:
+
+- memory: 820 → 520 B per message
+- parse: 0.67 → 0.63 s
+
+The remaining per-message cost is mostly the ``Message`` object itself.
+The realistic Python ceiling is about 2 µs and 400 B per message.
+
+Generated plugins
+-----------------
+
+The plugin generator template ``plugin.py.j2`` (under
+``tools/sawmill-plugin-generator/``) emits a generator-shaped
+``load_and_parse``.

sawmill_parser-0.4.0/docs/performance/store-investigation.rst

@@ -0,0 +1,127 @@
+Column-store investigation
+==========================
+
+A spike tested whether a column store would beat a list of ``Message``
+objects. The code lives on branch ``spike/duckdb-store``
+(``tools/bench/store_spike.py``).
+
+Stores compared
+---------------
+
+Three stores behind one query interface:
+
+- **List**: list of ``Message`` (today).
+- **DuckDB**: in-memory table, ingested in 50k-row batches via pyarrow.
+  ``executemany`` was unusable (25 s per 50k rows).
+- **Polars**: DataFrame built from dicts of columns per batch.
+
+Pandas was ruled out up front: per-row string handling is Python
+objects anyway, and the wheel is heavy.
+
+Results
+-------
+
+1M messages, 128 MB log. "DuckDB file" is file-backed with a 256 MB
+memory limit and late materialisation.
+
+.. list-table::
+   :header-rows: 1
+
+   * -
+     - List
+     - DuckDB in-memory
+     - DuckDB file
+     - Polars
+   * - Ingest
+     - 3.56 s
+     - 4.03 s
+     - 3.87 s
+     - 2.33 s
+   * - Time to first batch
+     - 0.15 s
+     - 0.16 s
+     - 0.18 s
+     - 0.12 s
+   * - RSS after ingest
+     - 531 MB
+     - 495 MB
+     - 294 MB
+     - 423 MB
+   * - Severity filter
+     - 22 ms
+     - 4 ms
+     - 3 ms
+     - 6 ms
+   * - ID glob
+     - 200 ms
+     - 1 ms
+     - 1 ms
+     - 6 ms
+   * - Regex (common pattern)
+     - 439 ms
+     - 19 ms
+     - 22 ms
+     - 7 ms
+   * - ID counts (stats)
+     - 46 ms
+     - 3 ms
+     - 1 ms
+     - 1 ms
+   * - Sort by ID count + page of 200
+     - 398 ms
+     - 113 ms
+     - 124 ms
+     - 32 ms
+   * - Combined keystroke (filter + sort + page)
+     - 389 ms
+     - 35 ms
+     - 36 ms
+     - 12 ms
+
+Other findings
+--------------
+
+- With a 256 MB ``memory_limit``, a full-row ``ORDER BY`` raised
+  ``OutOfMemoryException`` even at 200k. Sorting only key columns, then
+  joining back the 200 page rows (late materialisation), fixed it and
+  was 5 to 10× faster generally.
+- Without a limit and without late materialisation, RSS reached 2.4 GB.
+- Parsing to ``Message`` costs about 0.8 s per 1M over yielding tuples
+  (2.58 vs 1.78 s). Ingest only improves if plugins yield tuples.
+- Import time (warm): duckdb 45 ms, polars 48 ms, pyarrow 12 ms.
+- Installed size: duckdb about 65 MB, pyarrow about 165 MB (needed for
+  DuckDB ingest), polars about 180 MB.
+
+Awkward points
+--------------
+
+- Regex dialect changes from Python ``re`` to RE2 (DuckDB) or Rust regex
+  (Polars): no backreferences or lookaround. Saved filters and waiver
+  content patterns could break.
+- Waivers would become a table joined on ``message_id`` with
+  ``regexp_matches(raw_text, p, 's')``.
+- Suppressions become ``message_id IN (...)`` and
+  ``NOT regexp_matches``.
+- The TUI, waiver generation and export hold ``Message`` objects and
+  would move to row indexes.
+- ``metadata`` has no column.
+- A tuple-yielding plugin hook is needed to realise the ingest gain.
+
+Recommendation
+--------------
+
+Not adopted.
+
+- The windowed view removed the user-visible problem.
+- A store buys 10× faster keystrokes on any backend, but only
+  file-backed DuckDB with a limit saves memory (about 45%).
+- Revisit only if a real 130 MB log shows memory is still the complaint,
+  or sub-50 ms filtering becomes a feature. File-backed DuckDB is then
+  the path.
+
+Parked alternatives:
+
+- background parse thread
+- Rust inner loop
+- hand-rolled columnar arrays with byte offsets (about 40 B per message,
+  but a third plugin contract)

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/docs/plugin-guide/what-is-a-plugin.rst

@@ -22,10 +22,10 @@ interface:
    Detection --- examine a file and return a confidence score (0.0 to
    1.0) indicating how well this plugin can parse it.
 
-``load_and_parse(path) -> list[Message]``
-   Parsing --- open the file, parse every line, group multi-line
-   messages, extract severity and message IDs, and return a flat list
-   of ``Message`` objects.
+``load_and_parse(path) -> Iterator[Message]``
+   Parsing --- open the file, parse it line by line, group multi-line
+   messages, extract severity and message IDs, and yield each
+   ``Message`` as it is completed (plugin API 2).
 
 ``get_severity_levels() -> list[dict]`` (**required**)
    Define the tool's severity levels with ``id``, ``name``, ``level``

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/docs/plugin-guide/writing-a-plugin.rst

@@ -88,11 +88,13 @@ extension, or first few lines of the file:
            pass
        return 0.0
 
-``load_and_parse(path) -> list[Message]`` (optional)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+``load_and_parse(path) -> Iterator[Message]`` (optional)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Open the file, parse every line, group multi-line messages, and return
-a flat list of ``Message`` objects:
+Open the file, parse it line by line, group multi-line messages, and
+``yield`` each ``Message`` as soon as it is complete. Since plugin API 2
+this hook is a generator; returning a list still works but forces the
+whole log into memory. See :doc:`/performance/plugin-api-2`.
 
 .. code-block:: python
 
@@ -100,20 +102,24 @@ a flat list of ``Message`` objects:
 
    @hookimpl
    def load_and_parse(self, path):
-       messages = []
-       with open(path, errors="replace") as f:
+       f = open(path, errors="replace")  # open eagerly: errors surface here
+       return self._iter_messages(f)
+
+   def _iter_messages(self, f):
+       try:
            for line_num, line in enumerate(f, start=1):
                severity, msg_id, content = self._parse_line(line)
                if severity:
-                   messages.append(Message(
+                   yield Message(
                        start_line=line_num,
                        end_line=line_num,
                        raw_text=line.rstrip(),
                        content=content,
                        severity=severity,
                        message_id=msg_id,
-                   ))
-       return messages
+                   )
+       finally:
+           f.close()
 
 ``get_severity_levels() -> list[dict]`` (**required**)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -235,10 +241,21 @@ The typical workflow is:
    message ID format, and multi-line grouping rules.
 2. Write regex patterns in ``patterns.py``.
 3. Implement ``can_handle()`` to detect the tool's log files.
-4. Implement ``load_and_parse()`` to parse lines into messages.
+4. Implement ``load_and_parse()`` as a generator that yields messages.
 5. Implement ``get_severity_levels()`` with the correct level mapping.
 6. Optionally add ``get_filters()`` for common categories.
 
+Performance checklist
+---------------------
+
+- Make ``load_and_parse()`` a generator; read with ``for line in f``,
+  never ``readlines()``.
+- Open the file eagerly so missing-file errors are raised at the call.
+- Use one combined regex per line; avoid a dict per match.
+- Intern repeated strings and share or omit ``metadata`` dicts.
+
+Details and measurements: :doc:`/performance/plugin-api-2`.
+
 Testing
 -------
 

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/pyproject.toml

@@ -56,9 +56,9 @@ docs = [
 sawmill = "sawmill.__main__:cli"
 
 [project.urls]
-Homepage = "https://github.com/sawmill/sawmill"
-Documentation = "https://github.com/sawmill/sawmill#readme"
-Repository = "https://github.com/sawmill/sawmill"
+Homepage = "https://github.com/sjalloq/sawmill"
+Documentation = "https://sjalloq.github.io/sawmill"
+Repository = "https://github.com/sjalloq/sawmill"
 
 [tool.hatch.version]
 source = "vcs"

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/sawmill/cli/commands.py

@@ -27,6 +27,53 @@ from sawmill.models.waiver import Waiver
 
 click.rich_click.TEXT_MARKUP = "rich"
 click.rich_click.SHOW_ARGUMENTS = True
+click.rich_click.OPTION_GROUPS = {
+    "sawmill": [
+        {
+            "name": "General",
+            "options": [
+                "--version",
+                "--list-plugins",
+                "--plugin",
+                "--show-info",
+                "--batch",
+                "--help",
+            ],
+        },
+        {
+            "name": "Filtering (works in both TUI and batch mode)",
+            "options": [
+                "--severity",
+                "--filter",
+                "--suppress",
+                "--suppress-id",
+                "--id",
+                "--category",
+                "--waivers",
+            ],
+        },
+        {
+            "name": "Batch mode (implies --batch)",
+            "options": [
+                "--check",
+                "--fail-on",
+                "--show-waived",
+                "--report-unused",
+                "--report",
+                "--generate-waivers",
+                "--waiver-level",
+                "--summary",
+                "--group-by",
+                "--top",
+                "--format",
+            ],
+        },
+        {
+            "name": "Display",
+            "options": ["--list-severity", "--list-groupings"],
+        },
+    ],
+}
 
 
 def _get_implemented_hooks(plugin) -> list[str]:
@@ -169,7 +216,7 @@ def _get_implemented_hooks(plugin) -> list[str]:
 @click.option(
     "--batch",
     is_flag=True,
-    help="Run in batch mode (no TUI). Implied by any output/filter/check flags.",
+    help="Force batch mode (no TUI).",
 )
 @click.pass_context
 def cli(
@@ -404,19 +451,13 @@ def cli(
         return
 
     # Determine if batch mode is needed.
-    # Explicit --batch flag, or any output/filter/check flag implies batch.
+    # Explicit --batch flag, or batch-only flags imply batch.
+    # Filtering flags (--severity, --filter, --suppress, etc.) work in both modes.
     is_batch = batch or any(
         [
-            severity is not None,
-            filter_pattern is not None,
-            suppress_patterns,
-            suppress_ids,
-            id_patterns,
-            categories,
             generate_waivers,
             check,
             fail_on is not None,
-            waivers is not None,
             show_waived,
             report_unused,
             report_file is not None,

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/sawmill/cli/processing.py

@@ -17,6 +17,7 @@ from sawmill.core.plugin import (
     NoPluginFoundError,
     PluginConflictError,
     get_plugin_manager,
+    load_messages,
     select_plugin,
 )
 from sawmill.core.severity import (
@@ -116,8 +117,11 @@ def process_log_file(
                 console.print("\nUse --list-severity to see all available levels.")
                 ctx.exit(1)
 
-    # Load and parse the file using the plugin
-    messages = plugin.load_and_parse(path)
+    # Load and parse the file using the plugin.
+    # PR1: the plugin now streams via Iterator[Message]; the base app
+    # still consumes eagerly. PR3 will move this onto a background
+    # worker and drop the list() collapse.
+    messages = list(load_messages(plugin, path))
 
     # Apply severity filter
     if severity:
@@ -217,8 +221,9 @@ def generate_waivers(
         stderr_console.print(f"[red]Error:[/red] {e}")
         ctx.exit(1)
 
-    # Load and parse the file using the plugin
-    messages = plugin.load_and_parse(path)
+    # Load and parse the file using the plugin (eager collapse — see note in
+    # process_file for the streaming migration plan).
+    messages = list(load_messages(plugin, path))
 
     # Get severity levels from plugin
     severity_levels = get_severity_levels(plugin)

{sawmill_parser-0.3.2 → sawmill_parser-0.4.0}/sawmill/core/plugin.py

@@ -6,20 +6,30 @@ via Python entry points and registration with pluggy.
 
 from __future__ import annotations
 
+from collections.abc import Iterator
 from pathlib import Path
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 import pluggy
 
 from sawmill.plugin import SawmillHookSpec, SawmillPlugin
 
 if TYPE_CHECKING:
-    pass
+    from sawmill.models.message import Message
 
 # Entry point group name for sawmill plugins
 ENTRY_POINT_GROUP = "sawmill.plugins"
 
 
+def load_messages(plugin: Any, path: Path) -> Iterator[Message]:
+    """Call ``plugin.load_and_parse(path)`` and return an iterator.
+
+    Plugin API 2 returns an iterator; legacy (API 1) plugins return a list.
+    Wrapping in ``iter()`` lets both work and passes iterators through unchanged.
+    """
+    return iter(plugin.load_and_parse(path))
+
+
 class PluginError(Exception):
     """Base exception for plugin-related errors."""