sawmill-parser 0.2.4__tar.gz → 0.4.0__tar.gz

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/.github/workflows/docs.yml

@@ -28,9 +28,7 @@ jobs:
           python-version: "3.12"
 
       - name: Install dependencies
-        run: |
-          pip install -e .
-          pip install -r docs/requirements.txt
+        run: pip install -e ".[docs]"
 
       - name: Build docs
         run: sphinx-build -b html docs docs/_build/html

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/.gitignore

@@ -58,6 +58,7 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+uv.lock
 
 # IDE
 .idea/

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/PKG-INFO

@@ -1,10 +1,10 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: sawmill-parser
-Version: 0.2.4
+Version: 0.4.0
 Summary: A configurable TUI for post-processing and filtering structured tool logs
-Project-URL: Homepage, https://github.com/sawmill/sawmill
-Project-URL: Documentation, https://github.com/sawmill/sawmill#readme
-Project-URL: Repository, https://github.com/sawmill/sawmill
+Project-URL: Homepage, https://github.com/sjalloq/sawmill
+Project-URL: Documentation, https://sjalloq.github.io/sawmill
+Project-URL: Repository, https://github.com/sjalloq/sawmill
 Author: Sawmill Contributors
 License: MIT
 Keywords: eda,filter,log,terminal,tui
@@ -33,6 +33,11 @@ Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
 Requires-Dist: pytest>=7.0.0; extra == 'dev'
 Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: furo; extra == 'docs'
+Requires-Dist: sphinx; extra == 'docs'
+Requires-Dist: sphinx-autodoc-typehints; extra == 'docs'
+Requires-Dist: sphinx-copybutton; extra == 'docs'
 Description-Content-Type: text/markdown
 
 <p align="center">
@@ -125,11 +130,12 @@ sawmill vivado.log --suppress-id "Synth 8-7129"
 
 Sawmill uses a plugin architecture to support different log formats. Plugins are auto-detected based on log content.
 
-### Built-in Plugins
+### Available Plugins
 
-| Plugin | Description |
-|--------|-------------|
-| `vivado` | Xilinx Vivado synthesis and implementation logs |
+| Plugin | Repo | Description |
+|--------|--------|-------------|
+| `vivado` | https://github.com/sjalloq/sawmill-plugin-vivado | Xilinx Vivado synthesis and implementation logs |
+| `slang` | https://github.com/sjalloq/sawmill-plugin-slang | Slang System Verilog Compiler |
 
 ### External Plugins
 

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/README.md

@@ -88,11 +88,12 @@ sawmill vivado.log --suppress-id "Synth 8-7129"
 
 Sawmill uses a plugin architecture to support different log formats. Plugins are auto-detected based on log content.
 
-### Built-in Plugins
+### Available Plugins
 
-| Plugin | Description |
-|--------|-------------|
-| `vivado` | Xilinx Vivado synthesis and implementation logs |
+| Plugin | Repo | Description |
+|--------|--------|-------------|
+| `vivado` | https://github.com/sjalloq/sawmill-plugin-vivado | Xilinx Vivado synthesis and implementation logs |
+| `slang` | https://github.com/sjalloq/sawmill-plugin-slang | Slang System Verilog Compiler |
 
 ### External Plugins
 

sawmill_parser-0.4.0/SAWMILL_STREAMING_SPEC.md

@@ -0,0 +1,66 @@
+# Sawmill performance: findings and plan
+
+Working notes for making chip-level logs (100 MB+, 1M+ messages) usable.
+Supersedes the earlier streaming spec, whose diagnosis was wrong.
+
+## 1. What was measured (2026-11-04)
+
+Synthetic Xcelium log, 200k messages, 24 MB, Python 3.14, Textual DataTable.
+
+| Step | Time |
+|---|---|
+| Plugin parse alone | 0.7 s |
+| TUI load until table drawn | 9.4 s |
+| Filter logic excluding table rebuild | 0.14 s |
+| Clear filter / sort / severity toggle | 6 to 9 s |
+| First scroll after rebuild | 5 s |
+| RSS after parse / after table populated | 219 MB / 404 MB |
+
+- About 90 % of load time and 95 % of every interactive action is spent
+  inside `DataTable.add_row` and the deferred `_update_dimensions` pass that
+  measures every cell. DataTable virtualises rendering, not population.
+- Parse is bound by Python object creation (dict per match, Message, FileRef,
+  metadata dict), not regex (under 20 %) or I/O (about 6 %).
+- Memory per Message was about 820 B: 184 B metadata dict, 167 B raw_text,
+  104 B slots object, 86 B content (always a substring of raw_text), plus
+  unshared copies of message_id, category and file path.
+
+## 2. Done
+
+- Plugin hook `load_and_parse` returns `Iterator[Message]` (plugin API 2).
+  `sawmill.core.plugin.load_messages()` wraps list-returning plugins.
+  Plugins open the file eagerly so errors surface at call time.
+- `Message` and `FileRef` are `@dataclass(slots=True)` with int coercion.
+- Xcelium plugin: single regex, no per-match dict, interned strings, shared
+  per-tool metadata dict. 820 B to 520 B per message, output identical.
+- Benchmarks in `tools/bench/` report current and peak RSS and go through
+  the real filter helpers.
+
+## 3. In progress
+
+- Windowed DataTable: the table holds a window of rows around the cursor;
+  `_filtered_messages` stays the full list. Branch `perf/windowed-table`.
+- One waiver match per message; redundant list copies removed.
+
+## 4. Next: store investigation
+
+After the view is fixed, measure a real 130 MB log. If memory is still the
+complaint, compare on a branch:
+
+- Intern-only (current design, about 500 B/message, no new dependency).
+- DuckDB: columnar, dictionary-encoded, vectorised filter/sort/group-by,
+  native regex, spill to disk, `LIMIT/OFFSET` paging. Roughly 150 to 250 MB
+  per 1M messages. Cost: 50 MB wheel, 100 ms import, SQL in the app, waiver
+  and suppression logic expressed as predicates.
+- Polars: same idea, dataframe API, lighter, no spill.
+- Hand-rolled columnar (int arrays plus byte offsets, text re-read from
+  mmap): about 40 B/message but a third plugin contract and binary-mode
+  offsets in every plugin. Only if the above are insufficient.
+
+Decide with numbers from the real log, not estimates.
+
+## 5. Dropped
+
+Background parse thread, stats-only mode, Rust hot loop and FSM preamble
+matcher are parked. Parse is under 10 % of load; none of these move the
+user-visible numbers until the view and store are settled.

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/docs/index.rst

@@ -33,6 +33,14 @@ parse each tool's output.
    plugin-guide/pre-defined-filters
    plugin-guide/writing-a-plugin
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Performance
+
+   performance/overview
+   performance/plugin-api-2
+   performance/store-investigation
+
 .. toctree::
    :maxdepth: 2
    :caption: Reference

sawmill_parser-0.4.0/docs/performance/overview.rst

@@ -0,0 +1,103 @@
+Performance overview
+====================
+
+This page records what was measured, what was changed, and the current
+numbers.
+
+What was measured
+-----------------
+
+Measurements were taken on 2026-11-04 with a synthetic Xcelium log from
+``tools/bench/gen_xcelium_log.py`` (200k messages, 24 MB).
+
+.. list-table:: Before
+   :header-rows: 1
+
+   * - Step
+     - Before
+   * - Plugin parse alone
+     - 0.7 s
+   * - TUI load until table drawn
+     - 9.4 s
+   * - Filter logic excluding table rebuild
+     - 0.14 s
+   * - Clear filter / sort / severity toggle
+     - 6 to 9 s
+   * - First scroll after rebuild
+     - 5 s
+   * - RSS after parse / after table drawn
+     - 219 MB / 404 MB
+
+Diagnosis
+---------
+
+- About 90% of load and 95% of each interactive action was Textual
+  ``DataTable.add_row`` plus its deferred ``_update_dimensions`` cell
+  measurement. ``DataTable`` virtualises rendering, not population.
+- Parse was bound by Python object creation (about 40%: per-match dict,
+  ``Message``, ``FileRef``, metadata dict). Regex was under 20%, I/O
+  about 6%.
+- Memory per ``Message`` was about 820 B:
+
+  - metadata dict: 184 B
+  - ``raw_text``: 167 B
+  - slots object: 104 B
+  - ``content``: 86 B (always a substring of ``raw_text``)
+  - unshared ``message_id`` / ``category`` / path strings: about 144 B
+
+What was changed
+----------------
+
+- **Plugin API 2**: ``load_and_parse`` returns ``Iterator[Message]``.
+  API 1 plugins returning lists keep working. See :doc:`plugin-api-2`.
+- **Message model**: ``Message`` and ``FileRef`` are
+  ``@dataclass(slots=True)`` (``FileRef`` is frozen).
+  ``__post_init__`` coerces ``start_line``, ``end_line`` and ``line`` to
+  ``int``. No Pydantic on the hot path.
+- **Windowed table**: the TUI populates only a window of rows around the
+  cursor, ``max(3 × visible rows, 200)``, and re-windows centred on the
+  cursor as it moves. The subtitle shows ``row i of N``.
+- **Waivers**: matched once per message; redundant list copies removed.
+
+Benchmarks
+----------
+
+- ``tools/bench/gen_xcelium_log.py``: synthetic log generator.
+- ``tools/bench/run_bench.py``: size sweep, fresh subprocess per size,
+  current RSS from ``/proc/self/statm`` plus peak.
+- ``tools/bench/bench_message.py``: ``Message`` construction cost.
+
+Current numbers
+---------------
+
+.. list-table:: 200k messages
+   :header-rows: 1
+
+   * - Action
+     - Before
+     - After
+   * - Load
+     - 9.7 s
+     - 0.24 s
+   * - Clear filter
+     - 8.87 s
+     - 0.12 s
+   * - Sort
+     - 8.93 s
+     - 0.26 s
+   * - Severity toggle
+     - 6.66 s
+     - 0.18 s
+   * - RSS after table drawn
+     - 332 MB
+     - 158 MB
+
+At 1M messages:
+
+- parse: 3.5 s
+- load: 0.49 s
+- filter / sort / toggles: 0.36 to 0.70 s
+- RSS: about 550 MB after parse, 660 MB at end
+
+A column store was also evaluated and not adopted; see
+:doc:`store-investigation`.

sawmill_parser-0.4.0/docs/performance/plugin-api-2.rst

@@ -0,0 +1,97 @@
+Plugin API 2
+============
+
+``SAWMILL_PLUGIN_API`` is ``"2"``. The change is to ``load_and_parse``,
+which now streams messages.
+
+The contract
+------------
+
+.. code-block:: python
+
+   @hookspec
+   def load_and_parse(self, path: Path) -> Iterator[Message]: ...
+
+- Plugins should be generators reading the file line by line.
+- Yield each ``Message`` as soon as it is complete.
+- The ``SawmillPlugin`` base class default is a generator that yields
+  nothing.
+- ``[tool.sawmill] plugin_api`` in a plugin's ``pyproject.toml`` is
+  informational; nothing enforces it.
+
+Migrating from API 1
+--------------------
+
+API 1 returned ``list[Message]``. To migrate:
+
+- Replace ``messages.append(msg)`` with ``yield msg``.
+- Drop the final ``return messages``.
+- Flush any in-flight multi-line message after the loop with a final
+  ``yield``.
+
+The ``load_messages`` shim
+--------------------------
+
+The base app calls plugins through ``sawmill.core.plugin.load_messages``:
+
+.. code-block:: python
+
+   def load_messages(plugin, path) -> Iterator[Message]:
+       return iter(plugin.load_and_parse(path))
+
+A legacy list is wrapped with ``iter()``, so API 1 plugins keep working
+unchanged. They do not get the memory benefit.
+
+Eager-open pattern
+------------------
+
+A plain generator does not run until the first ``next()``, so a missing
+file would raise deep inside the consumer. Open the file in an outer
+function and return an inner generator over the handle:
+
+.. code-block:: python
+
+   @hookimpl
+   def load_and_parse(self, path):
+       f = open(path, errors="replace")   # raises here if missing
+       return self._iter_messages(f)
+
+   def _iter_messages(self, f):
+       try:
+           current = None
+           for line_num, line in enumerate(f, start=1):
+               ...  # build or extend current; yield when complete
+           if current is not None:
+               yield current             # flush the in-flight message
+       finally:
+           f.close()
+
+Caveat: a generator that is never iterated keeps its file handle open
+until garbage collection.
+
+Allocation guidance
+-------------------
+
+Parse time is dominated by Python object creation, not regex or I/O.
+
+- Read with ``for line in f``, never ``readlines()``.
+- Use one combined regex per line rather than two tried in sequence.
+- Use ``m.groups()`` or a ``NamedTuple``, not a dict per line.
+- Intern repeated strings (tool, rule / message ID, file path) through a
+  small cache dict.
+- Share one ``metadata`` dict per constant value, or omit ``metadata``.
+
+Measured effect on an Xcelium plugin at 200k messages, output identical:
+
+- memory: 820 → 520 B per message
+- parse: 0.67 → 0.63 s
+
+The remaining per-message cost is mostly the ``Message`` object itself.
+The realistic Python ceiling is about 2 µs and 400 B per message.
+
+Generated plugins
+-----------------
+
+The plugin generator template ``plugin.py.j2`` (under
+``tools/sawmill-plugin-generator/``) emits a generator-shaped
+``load_and_parse``.

sawmill_parser-0.4.0/docs/performance/store-investigation.rst

@@ -0,0 +1,127 @@
+Column-store investigation
+==========================
+
+A spike tested whether a column store would beat a list of ``Message``
+objects. The code lives on branch ``spike/duckdb-store``
+(``tools/bench/store_spike.py``).
+
+Stores compared
+---------------
+
+Three stores behind one query interface:
+
+- **List**: list of ``Message`` (today).
+- **DuckDB**: in-memory table, ingested in 50k-row batches via pyarrow.
+  ``executemany`` was unusable (25 s per 50k rows).
+- **Polars**: DataFrame built from dicts of columns per batch.
+
+Pandas was ruled out up front: per-row string handling is Python
+objects anyway, and the wheel is heavy.
+
+Results
+-------
+
+1M messages, 128 MB log. "DuckDB file" is file-backed with a 256 MB
+memory limit and late materialisation.
+
+.. list-table::
+   :header-rows: 1
+
+   * -
+     - List
+     - DuckDB in-memory
+     - DuckDB file
+     - Polars
+   * - Ingest
+     - 3.56 s
+     - 4.03 s
+     - 3.87 s
+     - 2.33 s
+   * - Time to first batch
+     - 0.15 s
+     - 0.16 s
+     - 0.18 s
+     - 0.12 s
+   * - RSS after ingest
+     - 531 MB
+     - 495 MB
+     - 294 MB
+     - 423 MB
+   * - Severity filter
+     - 22 ms
+     - 4 ms
+     - 3 ms
+     - 6 ms
+   * - ID glob
+     - 200 ms
+     - 1 ms
+     - 1 ms
+     - 6 ms
+   * - Regex (common pattern)
+     - 439 ms
+     - 19 ms
+     - 22 ms
+     - 7 ms
+   * - ID counts (stats)
+     - 46 ms
+     - 3 ms
+     - 1 ms
+     - 1 ms
+   * - Sort by ID count + page of 200
+     - 398 ms
+     - 113 ms
+     - 124 ms
+     - 32 ms
+   * - Combined keystroke (filter + sort + page)
+     - 389 ms
+     - 35 ms
+     - 36 ms
+     - 12 ms
+
+Other findings
+--------------
+
+- With a 256 MB ``memory_limit``, a full-row ``ORDER BY`` raised
+  ``OutOfMemoryException`` even at 200k. Sorting only key columns, then
+  joining back the 200 page rows (late materialisation), fixed it and
+  was 5 to 10× faster generally.
+- Without a limit and without late materialisation, RSS reached 2.4 GB.
+- Parsing to ``Message`` costs about 0.8 s per 1M over yielding tuples
+  (2.58 vs 1.78 s). Ingest only improves if plugins yield tuples.
+- Import time (warm): duckdb 45 ms, polars 48 ms, pyarrow 12 ms.
+- Installed size: duckdb about 65 MB, pyarrow about 165 MB (needed for
+  DuckDB ingest), polars about 180 MB.
+
+Awkward points
+--------------
+
+- Regex dialect changes from Python ``re`` to RE2 (DuckDB) or Rust regex
+  (Polars): no backreferences or lookaround. Saved filters and waiver
+  content patterns could break.
+- Waivers would become a table joined on ``message_id`` with
+  ``regexp_matches(raw_text, p, 's')``.
+- Suppressions become ``message_id IN (...)`` and
+  ``NOT regexp_matches``.
+- The TUI, waiver generation and export hold ``Message`` objects and
+  would move to row indexes.
+- ``metadata`` has no column.
+- A tuple-yielding plugin hook is needed to realise the ingest gain.
+
+Recommendation
+--------------
+
+Not adopted.
+
+- The windowed view removed the user-visible problem.
+- A store buys 10× faster keystrokes on any backend, but only
+  file-backed DuckDB with a limit saves memory (about 45%).
+- Revisit only if a real 130 MB log shows memory is still the complaint,
+  or sub-50 ms filtering becomes a feature. File-backed DuckDB is then
+  the path.
+
+Parked alternatives:
+
+- background parse thread
+- Rust inner loop
+- hand-rolled columnar arrays with byte offsets (about 40 B per message,
+  but a third plugin contract)

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/docs/plugin-guide/what-is-a-plugin.rst

@@ -22,10 +22,10 @@ interface:
    Detection --- examine a file and return a confidence score (0.0 to
    1.0) indicating how well this plugin can parse it.
 
-``load_and_parse(path) -> list[Message]``
-   Parsing --- open the file, parse every line, group multi-line
-   messages, extract severity and message IDs, and return a flat list
-   of ``Message`` objects.
+``load_and_parse(path) -> Iterator[Message]``
+   Parsing --- open the file, parse it line by line, group multi-line
+   messages, extract severity and message IDs, and yield each
+   ``Message`` as it is completed (plugin API 2).
 
 ``get_severity_levels() -> list[dict]`` (**required**)
    Define the tool's severity levels with ``id``, ``name``, ``level``

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/docs/plugin-guide/writing-a-plugin.rst

@@ -88,11 +88,13 @@ extension, or first few lines of the file:
            pass
        return 0.0
 
-``load_and_parse(path) -> list[Message]`` (optional)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+``load_and_parse(path) -> Iterator[Message]`` (optional)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Open the file, parse every line, group multi-line messages, and return
-a flat list of ``Message`` objects:
+Open the file, parse it line by line, group multi-line messages, and
+``yield`` each ``Message`` as soon as it is complete. Since plugin API 2
+this hook is a generator; returning a list still works but forces the
+whole log into memory. See :doc:`/performance/plugin-api-2`.
 
 .. code-block:: python
 
@@ -100,20 +102,24 @@ a flat list of ``Message`` objects:
 
    @hookimpl
    def load_and_parse(self, path):
-       messages = []
-       with open(path, errors="replace") as f:
+       f = open(path, errors="replace")  # open eagerly: errors surface here
+       return self._iter_messages(f)
+
+   def _iter_messages(self, f):
+       try:
            for line_num, line in enumerate(f, start=1):
                severity, msg_id, content = self._parse_line(line)
                if severity:
-                   messages.append(Message(
+                   yield Message(
                        start_line=line_num,
                        end_line=line_num,
                        raw_text=line.rstrip(),
                        content=content,
                        severity=severity,
                        message_id=msg_id,
-                   ))
-       return messages
+                   )
+       finally:
+           f.close()
 
 ``get_severity_levels() -> list[dict]`` (**required**)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -235,10 +241,21 @@ The typical workflow is:
    message ID format, and multi-line grouping rules.
 2. Write regex patterns in ``patterns.py``.
 3. Implement ``can_handle()`` to detect the tool's log files.
-4. Implement ``load_and_parse()`` to parse lines into messages.
+4. Implement ``load_and_parse()`` as a generator that yields messages.
 5. Implement ``get_severity_levels()`` with the correct level mapping.
 6. Optionally add ``get_filters()`` for common categories.
 
+Performance checklist
+---------------------
+
+- Make ``load_and_parse()`` a generator; read with ``for line in f``,
+  never ``readlines()``.
+- Open the file eagerly so missing-file errors are raised at the call.
+- Use one combined regex per line; avoid a dict per match.
+- Intern repeated strings and share or omit ``metadata`` dicts.
+
+Details and measurements: :doc:`/performance/plugin-api-2`.
+
 Testing
 -------
 

{sawmill_parser-0.2.4 → sawmill_parser-0.4.0}/pyproject.toml

@@ -45,21 +45,24 @@ dev = [
     "ruff>=0.4.0",
     "pre-commit>=3.0.0",
 ]
+docs = [
+    "sphinx",
+    "furo",
+    "sphinx-autodoc-typehints",
+    "sphinx-copybutton",
+]
 
 [project.scripts]
 sawmill = "sawmill.__main__:cli"
 
 [project.urls]
-Homepage = "https://github.com/sawmill/sawmill"
-Documentation = "https://github.com/sawmill/sawmill#readme"
-Repository = "https://github.com/sawmill/sawmill"
+Homepage = "https://github.com/sjalloq/sawmill"
+Documentation = "https://sjalloq.github.io/sawmill"
+Repository = "https://github.com/sjalloq/sawmill"
 
 [tool.hatch.version]
 source = "vcs"
 
-[tool.hatch.metadata]
-allow-direct-references = true
-
 [tool.hatch.build.targets.wheel]
 packages = ["sawmill"]