pySigma-backend-sqlite 2.0.0__tar.gz → 2.1.0__tar.gz

pysigma_backend_sqlite-2.1.0/PKG-INFO

@@ -0,0 +1,176 @@
+Metadata-Version: 2.4
+Name: pySigma-backend-sqlite
+Version: 2.1.0
+Summary: pySigma sqlite backend
+License-Expression: LGPL-3.0-only
+License-File: LICENSE
+Author: wagga
+Requires-Python: >=3.11,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Requires-Dist: pysigma (>=2.0.0,<3.0)
+Project-URL: Repository, https://github.com/SigmaHQ/pySigma-backend-sqlite
+Description-Content-Type: text/markdown
+
+![Tests](https://github.com/wagga40/pySigma-backend-sqlite/actions/workflows/test.yml/badge.svg)
+![Coverage Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/wagga40/2ec45ded898fa11f2c42bcb9d2b163cf/raw/test.json)
+
+# pySigma SQLite Backend
+
+Converts [Sigma](https://github.com/SigmaHQ/sigma) rules into SQLite queries. The
+package `sigma.backends.sqlite` provides `sqliteBackend`, registered as the
+sigma-cli target `sqlite`. It is the converter behind
+[Zircolite](https://github.com/wagga40/Zircolite), and its output runs on any
+SQLite table of flattened events.
+
+Maintained by [wagga](https://github.com/wagga40/).
+
+## Install
+
+```shell
+pip install pysigma-backend-sqlite
+```
+
+Requires Python 3.11+ and [pySigma](https://github.com/SigmaHQ/pySigma) 2.x.
+Backend 2.0.0 is the last release for pySigma 1.5. Correlation output needs
+SQLite 3.38 or newer.
+
+## Quick start
+
+```python
+from sigma.collection import SigmaCollection
+from sigma.backends.sqlite import sqliteBackend
+
+rules = SigmaCollection.from_yaml(r"""
+title: Whoami execution
+status: test
+logsource:
+    category: process_creation
+    product: windows
+detection:
+    selection:
+        Image|endswith: '\whoami.exe'
+    condition: selection
+""")
+
+print(sqliteBackend().convert(rules)[0])
+# SELECT * FROM logs WHERE Image LIKE '%\\whoami.exe' ESCAPE '\'
+```
+
+Processing pipelines map a logsource to concrete fields. With
+`sqliteBackend(sysmon_pipeline() + windows_logsource_pipeline())` the same rule becomes:
+
+```sql
+SELECT * FROM logs WHERE Channel='Microsoft-Windows-Sysmon/Operational' AND (EventID=1 AND Image LIKE '%\\whoami.exe' ESCAPE '\')
+```
+
+Those pipeline packages still declare `pysigma<2`; until they are updated, install
+them with `pip install --no-deps pysigma-pipeline-sysmon pysigma-pipeline-windows`.
+With sigma-cli:
+
+```shell
+sigma convert -t sqlite -f zircolite -p sysmon -p windows-logsources rules/
+```
+
+sigma-cli 3.1 still requires pySigma 1.x, so it installs backend 2.0.0.
+
+## Output formats
+
+| Format | Output |
+|---|---|
+| `default` | One `SELECT * FROM logs WHERE ...` statement per rule |
+| `zircolite` | A JSON list with one entry per rule: Sigma metadata, the SQL, and what a consumer needs to run it. See [output format](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/output-format.md) |
+
+## Options
+
+Pass options as constructor keywords, or from sigma-cli with `-O name=value`.
+Options are read at construction; assigning attributes later skips validation
+and does not change the timestamp expression.
+
+| Option | Default | Purpose |
+|---|---|---|
+| `table` | `logs` | Table to query. A pipeline `setState` on `table` takes precedence |
+| `collate_nocase` | `false` | Emit `field='value' COLLATE NOCASE` for tables without `NOCASE` columns |
+| `max_flat_operands` | `100` | Longer AND/OR chains are nested to stay under SQLite's expression depth limit |
+| `fulltext_field` | none | Column of searchable event text; enables [keyword search](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/keyword-search.md) |
+| `timestamp_field` | `timestamp` | Event time column used by correlations |
+| `timestamp_format` | `iso` | `iso`, `unix`, `unix_ms` or `unix_us` |
+| `timestamp_seconds_expression` | ISO parser | SQL turning `{field}` into Unix seconds; overrides `timestamp_format` |
+| `event_id_field` | `rowid` | Non-NULL, per-table unique event identifier for correlation evidence |
+| `observation_end` | latest event | Unix seconds up to which absence windows can expire |
+
+## Generated SQL
+
+| Modifier | SQL |
+|---|---|
+| `contains`, `startswith`, `endswith` | `LIKE '%value%' ESCAPE '\'` and its prefix/suffix forms |
+| `all` | `AND` of the values |
+| `re` | `REGEXP` (needs a registered function, see [Limitations](#limitations)) |
+| `cidr` | `=` or `LIKE` prefixes of the expanded network |
+| `cased` | `GLOB`; literal `*`, `?`, `[` become `[*]`, `[?]`, `[[]` |
+| `fieldref` | `field1=field2`, or `LIKE` with the second field's `%` and `_` escaped |
+| `exists` | `IS NOT NULL` / `IS NULL` |
+| `gt`, `gte`, `lt`, `lte`, `neq` | `>`, `>=`, `<`, `<=`, negated equality |
+| `minute`, `hour`, `day`, `week`, `month`, `year` | `strftime()` part |
+| `base64`, `base64offset`, `wide`, `utf16*`, `windash` | Expanded by pySigma into ordinary matches |
+
+The SQL keeps Sigma's meaning without help from the caller:
+
+- **Negation.** Sigma reads a condition on a missing field as false; SQLite reads
+  it as `NULL`, and `NOT NULL` drops the row. Every `NOT` therefore wraps its
+  operand: `EventID=1 AND (NOT COALESCE((CommandLine LIKE '%foo%' ESCAPE '\'), 0))`.
+- **Case.** `LIKE` ignores case. `=` does only on `NOCASE` columns, which is how
+  Zircolite builds its table; otherwise set `collate_nocase`. `cased` uses `GLOB`.
+- **Booleans.** `Flag: true` matches text or integer storage: `(Flag='true' OR Flag=1)`.
+- **Long lists.** Chains above `max_flat_operands` are regrouped, so rules with
+  thousands of values don't fail with "Expression tree is too large".
+- **Field names.** SQLite keywords and names with characters other than letters,
+  digits and `_` are quoted with backticks. `field: null` becomes `field IS NULL`.
+
+## Correlations
+
+All Sigma correlation types run in SQLite: `event_count`, `value_count`,
+`value_sum`, `value_avg`, `value_median`, `value_percentile`, `temporal` and
+`temporal_ordered`, including chains, aliases and extended temporal conditions.
+Each result row is one alert occurrence with its window, metric and contributing
+event IDs. Semantics are described in [correlations](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/correlations.md);
+upgrading from backend 1.x is covered in the [migration guide](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/migration-v2.md).
+
+## Keyword search
+
+Keyword (fieldless) selections need a column of searchable event text:
+`sqliteBackend(fulltext_field="_sigma_fulltext")`. Without it, keyword rules raise
+an error rather than guess. See [keyword search](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/keyword-search.md).
+
+## Limitations
+
+- **REGEXP.** SQLite has no built-in `REGEXP`. Register one that is false for `NULL`:
+  `connection.create_function("regexp", 2, lambda p, v: v is not None and re.search(p, str(v)) is not None)`.
+  pySigma 2 validates `|re` patterns with RE2, so rules with lookarounds or
+  backreferences are rejected when parsed.
+- **CIDR.** SQLite has no network type. Prefix patterns match IPv6 addresses only
+  in the spellings pySigma expands to.
+- **Case folding** (`NOCASE`, `LIKE`) is ASCII only.
+- **Timestamp parts** use `strftime()` and need ISO-8601 text; a numeric epoch is
+  read as a Julian day. `%W` can return week 0.
+- **Field existence.** SQL `NULL` cannot tell a missing field from an explicit null.
+- **Multi-field `value_count`** (a list in `condition.field`) is rejected.
+- **Unknown columns.** A query naming a column the table lacks fails to prepare.
+  Add missing columns as `NULL` first; see [output format](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/output-format.md).
+
+## Development
+
+```shell
+poetry install
+poetry run pytest
+SQLITE_MIN_BINARY=/path/to/sqlite-3.38 poetry run pytest tests/test_sqlite_minimum.py
+```
+
+`tools/audit_corpus.py` converts and prepares a SigmaHQ checkout per profile
+(coverage, not detection accuracy). `tools/benchmark_correlations.py` times
+sparse and dense correlation windows.
+

pysigma_backend_sqlite-2.1.0/README.md

@@ -0,0 +1,157 @@
+![Tests](https://github.com/wagga40/pySigma-backend-sqlite/actions/workflows/test.yml/badge.svg)
+![Coverage Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/wagga40/2ec45ded898fa11f2c42bcb9d2b163cf/raw/test.json)
+
+# pySigma SQLite Backend
+
+Converts [Sigma](https://github.com/SigmaHQ/sigma) rules into SQLite queries. The
+package `sigma.backends.sqlite` provides `sqliteBackend`, registered as the
+sigma-cli target `sqlite`. It is the converter behind
+[Zircolite](https://github.com/wagga40/Zircolite), and its output runs on any
+SQLite table of flattened events.
+
+Maintained by [wagga](https://github.com/wagga40/).
+
+## Install
+
+```shell
+pip install pysigma-backend-sqlite
+```
+
+Requires Python 3.11+ and [pySigma](https://github.com/SigmaHQ/pySigma) 2.x.
+Backend 2.0.0 is the last release for pySigma 1.5. Correlation output needs
+SQLite 3.38 or newer.
+
+## Quick start
+
+```python
+from sigma.collection import SigmaCollection
+from sigma.backends.sqlite import sqliteBackend
+
+rules = SigmaCollection.from_yaml(r"""
+title: Whoami execution
+status: test
+logsource:
+    category: process_creation
+    product: windows
+detection:
+    selection:
+        Image|endswith: '\whoami.exe'
+    condition: selection
+""")
+
+print(sqliteBackend().convert(rules)[0])
+# SELECT * FROM logs WHERE Image LIKE '%\\whoami.exe' ESCAPE '\'
+```
+
+Processing pipelines map a logsource to concrete fields. With
+`sqliteBackend(sysmon_pipeline() + windows_logsource_pipeline())` the same rule becomes:
+
+```sql
+SELECT * FROM logs WHERE Channel='Microsoft-Windows-Sysmon/Operational' AND (EventID=1 AND Image LIKE '%\\whoami.exe' ESCAPE '\')
+```
+
+Those pipeline packages still declare `pysigma<2`; until they are updated, install
+them with `pip install --no-deps pysigma-pipeline-sysmon pysigma-pipeline-windows`.
+With sigma-cli:
+
+```shell
+sigma convert -t sqlite -f zircolite -p sysmon -p windows-logsources rules/
+```
+
+sigma-cli 3.1 still requires pySigma 1.x, so it installs backend 2.0.0.
+
+## Output formats
+
+| Format | Output |
+|---|---|
+| `default` | One `SELECT * FROM logs WHERE ...` statement per rule |
+| `zircolite` | A JSON list with one entry per rule: Sigma metadata, the SQL, and what a consumer needs to run it. See [output format](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/output-format.md) |
+
+## Options
+
+Pass options as constructor keywords, or from sigma-cli with `-O name=value`.
+Options are read at construction; assigning attributes later skips validation
+and does not change the timestamp expression.
+
+| Option | Default | Purpose |
+|---|---|---|
+| `table` | `logs` | Table to query. A pipeline `setState` on `table` takes precedence |
+| `collate_nocase` | `false` | Emit `field='value' COLLATE NOCASE` for tables without `NOCASE` columns |
+| `max_flat_operands` | `100` | Longer AND/OR chains are nested to stay under SQLite's expression depth limit |
+| `fulltext_field` | none | Column of searchable event text; enables [keyword search](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/keyword-search.md) |
+| `timestamp_field` | `timestamp` | Event time column used by correlations |
+| `timestamp_format` | `iso` | `iso`, `unix`, `unix_ms` or `unix_us` |
+| `timestamp_seconds_expression` | ISO parser | SQL turning `{field}` into Unix seconds; overrides `timestamp_format` |
+| `event_id_field` | `rowid` | Non-NULL, per-table unique event identifier for correlation evidence |
+| `observation_end` | latest event | Unix seconds up to which absence windows can expire |
+
+## Generated SQL
+
+| Modifier | SQL |
+|---|---|
+| `contains`, `startswith`, `endswith` | `LIKE '%value%' ESCAPE '\'` and its prefix/suffix forms |
+| `all` | `AND` of the values |
+| `re` | `REGEXP` (needs a registered function, see [Limitations](#limitations)) |
+| `cidr` | `=` or `LIKE` prefixes of the expanded network |
+| `cased` | `GLOB`; literal `*`, `?`, `[` become `[*]`, `[?]`, `[[]` |
+| `fieldref` | `field1=field2`, or `LIKE` with the second field's `%` and `_` escaped |
+| `exists` | `IS NOT NULL` / `IS NULL` |
+| `gt`, `gte`, `lt`, `lte`, `neq` | `>`, `>=`, `<`, `<=`, negated equality |
+| `minute`, `hour`, `day`, `week`, `month`, `year` | `strftime()` part |
+| `base64`, `base64offset`, `wide`, `utf16*`, `windash` | Expanded by pySigma into ordinary matches |
+
+The SQL keeps Sigma's meaning without help from the caller:
+
+- **Negation.** Sigma reads a condition on a missing field as false; SQLite reads
+  it as `NULL`, and `NOT NULL` drops the row. Every `NOT` therefore wraps its
+  operand: `EventID=1 AND (NOT COALESCE((CommandLine LIKE '%foo%' ESCAPE '\'), 0))`.
+- **Case.** `LIKE` ignores case. `=` does only on `NOCASE` columns, which is how
+  Zircolite builds its table; otherwise set `collate_nocase`. `cased` uses `GLOB`.
+- **Booleans.** `Flag: true` matches text or integer storage: `(Flag='true' OR Flag=1)`.
+- **Long lists.** Chains above `max_flat_operands` are regrouped, so rules with
+  thousands of values don't fail with "Expression tree is too large".
+- **Field names.** SQLite keywords and names with characters other than letters,
+  digits and `_` are quoted with backticks. `field: null` becomes `field IS NULL`.
+
+## Correlations
+
+All Sigma correlation types run in SQLite: `event_count`, `value_count`,
+`value_sum`, `value_avg`, `value_median`, `value_percentile`, `temporal` and
+`temporal_ordered`, including chains, aliases and extended temporal conditions.
+Each result row is one alert occurrence with its window, metric and contributing
+event IDs. Semantics are described in [correlations](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/correlations.md);
+upgrading from backend 1.x is covered in the [migration guide](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/migration-v2.md).
+
+## Keyword search
+
+Keyword (fieldless) selections need a column of searchable event text:
+`sqliteBackend(fulltext_field="_sigma_fulltext")`. Without it, keyword rules raise
+an error rather than guess. See [keyword search](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/keyword-search.md).
+
+## Limitations
+
+- **REGEXP.** SQLite has no built-in `REGEXP`. Register one that is false for `NULL`:
+  `connection.create_function("regexp", 2, lambda p, v: v is not None and re.search(p, str(v)) is not None)`.
+  pySigma 2 validates `|re` patterns with RE2, so rules with lookarounds or
+  backreferences are rejected when parsed.
+- **CIDR.** SQLite has no network type. Prefix patterns match IPv6 addresses only
+  in the spellings pySigma expands to.
+- **Case folding** (`NOCASE`, `LIKE`) is ASCII only.
+- **Timestamp parts** use `strftime()` and need ISO-8601 text; a numeric epoch is
+  read as a Julian day. `%W` can return week 0.
+- **Field existence.** SQL `NULL` cannot tell a missing field from an explicit null.
+- **Multi-field `value_count`** (a list in `condition.field`) is rejected.
+- **Unknown columns.** A query naming a column the table lacks fails to prepare.
+  Add missing columns as `NULL` first; see [output format](https://github.com/wagga40/pySigma-backend-sqlite/blob/main/docs/output-format.md).
+
+## Development
+
+```shell
+poetry install
+poetry run pytest
+SQLITE_MIN_BINARY=/path/to/sqlite-3.38 poetry run pytest tests/test_sqlite_minimum.py
+```
+
+`tools/audit_corpus.py` converts and prepares a SigmaHQ checkout per profile
+(coverage, not detection accuracy). `tools/benchmark_correlations.py` times
+sparse and dense correlation windows.

{pysigma_backend_sqlite-2.0.0 → pysigma_backend_sqlite-2.1.0}/pyproject.toml

@@ -1,8 +1,8 @@
 [project]
 name = "pySigma-backend-sqlite"
-version = "2.0.0"
-requires-python = ">=3.10,<4.0"
-dependencies = ["pysigma>=1.5.1,<2.0"]
+version = "2.1.0"
+requires-python = ">=3.11,<4.0"
+dependencies = ["pysigma>=2.0.0,<3.0"]
 description = "pySigma sqlite backend"
 readme = "README.md"
 authors = [
@@ -29,12 +29,6 @@ filterwarnings = [
     "ignore:'parseString' deprecated:DeprecationWarning",
 ]
 
-[tool.pdm]
-[tool.pdm.dev-dependencies]
-dev = []
-
-[tool.pdm.build]
-includes = ["sigma"]
 [build-system]
 requires = ["poetry-core>=2.0.0"]
 build-backend = "poetry.core.masonry.api"

{pysigma_backend_sqlite-2.0.0 → pysigma_backend_sqlite-2.1.0}/sigma/backends/sqlite/correlation.py

@@ -1,7 +1,7 @@
 """Relational correlation compiler shared by SQL and materialized execution.
 
-Every stage is a SELECT. The portable form combines stages as CTEs; the
-Zircolite executor materializes the same stages, with explicit indexes. No
+Every stage is a SELECT. The portable form combines stages as CTEs;
+runtime.execute_plan materializes the same stages, with explicit indexes. No
 finalized query or output-format dictionary is ever used as a predicate.
 """
 
@@ -28,6 +28,7 @@ def literal(value: str) -> str:
 
 
 def quote(value: str) -> str:
+    # Backticks, not double quotes: SQLite reads an unknown "name" as a string literal.
     return "`" + value.replace("`", "``") + "`"
 
 
@@ -114,6 +115,15 @@ class CorrelationCompiler:
         self.source_tables = {}
         self.diagnostic_queries = []
         self._discover(root)
+        fulltext = {
+            (item["table"], item["field"])
+            for rule in self.base_rules.values()
+            for item in self.backend.fulltext_requirements(rule)
+        }
+        self.fulltext_requirements = [
+            {"table": table, "field": field, "representation": "source-values-v1"}
+            for table, field in sorted(fulltext)
+        ]
         self.field_ids = {f: f"f{i}" for i, f in enumerate(sorted(self.fields))}
         # Stable identifiers make saved plans reproducible. Tables live in temp.
         identity = json.dumps(root.to_dict(), sort_keys=True, default=str)
@@ -133,6 +143,10 @@ class CorrelationCompiler:
         if isinstance(rule, SigmaCorrelationRule):
             self.fields.update(rule.group_by or [])
             field = getattr(rule.condition, "fieldref", None)
+            if isinstance(field, list):
+                raise SigmaFeatureNotSupportedByBackendError(
+                    f"Correlation condition field list {field} is not supported"
+                )
             if field:
                 self.fields.add(field)
             for alias in rule.aliases:
@@ -181,6 +195,8 @@ class CorrelationCompiler:
             self.required_by_table.setdefault(table, set()).update(
                 self.backend.required_fields(rule)
             )
+        for item in self.fulltext_requirements:
+            self.required_by_table.setdefault(item["table"], set()).add(item["field"])
         for i, (table, rules) in enumerate(by_table.items()):
             self.source_tables[str(i)] = table
             fields = self.required_by_table[table]
@@ -598,7 +614,7 @@ class CorrelationCompiler:
         ctes = ", ".join(f"{s.name} AS ({s.select})" for s in self.stages)
         diagnostics = " UNION ALL ".join(self.diagnostic_queries)
         plan = {
-            "version": 2,
+            "version": 3 if self.fulltext_requirements else 2,
             "sqlite_min_version": "3.38.0",
             "prepare": [
                 dict(name=s.name, select=s.select, indexes=s.indexes)
@@ -613,4 +629,6 @@ class CorrelationCompiler:
             "event_id_field": self.backend.event_id_field,
             "source_tables": self.source_tables,
         }
+        if self.fulltext_requirements:
+            plan["fulltext"] = self.fulltext_requirements
         return "WITH " + ctes + " " + query, plan

{pysigma_backend_sqlite-2.0.0 → pysigma_backend_sqlite-2.1.0}/sigma/backends/sqlite/runtime.py

@@ -1,4 +1,4 @@
-"""Execute a version-2 correlation plan on a consumer-owned SQLite connection."""
+"""Execute correlation plans on a consumer-owned SQLite connection."""
 
 from __future__ import annotations
 
@@ -9,6 +9,57 @@ import sqlite3
 from .correlation import quote
 
 
+_TABLE_NAME = re.compile(
+    r"(?:[A-Za-z_][A-Za-z0-9_]*|`(?:[^`]|``)+`)(?:\.(?:[A-Za-z_][A-Za-z0-9_]*|`(?:[^`]|``)+`))*\Z"
+)
+
+
+def validate_fulltext(connection, requirements):
+    """Require a populated source-values-v1 TEXT column for every source row."""
+    if not isinstance(requirements, list) or not requirements:
+        raise ValueError("Keyword search requires nonempty fulltext requirements")
+    for item in requirements:
+        if not isinstance(item, dict) or set(item) != {
+            "table",
+            "field",
+            "representation",
+        }:
+            raise ValueError(
+                "Invalid fulltext requirement; re-ingest events for keyword search"
+            )
+        table, field = item["table"], item["field"]
+        if (
+            not isinstance(table, str)
+            or not _TABLE_NAME.fullmatch(table)
+            or not isinstance(field, str)
+            or not field
+            or item["representation"] != "source-values-v1"
+        ):
+            raise ValueError(
+                "Invalid fulltext requirement; re-ingest events for keyword search"
+            )
+        try:
+            cursor = connection.execute(f"SELECT * FROM {table} LIMIT 0")
+            existing = {column[0].lower() for column in cursor.description}
+            cursor.close()
+            if field.lower() not in existing:
+                raise ValueError(
+                    f"Keyword text column {field!r} is missing from {table}; re-ingest events"
+                )
+            missing = connection.execute(
+                f"SELECT 1 FROM {table} WHERE {quote(field)} IS NULL "
+                f"OR typeof({quote(field)}) != 'text' LIMIT 1"
+            ).fetchone()
+        except sqlite3.Error as error:
+            raise ValueError(
+                f"Keyword text is unavailable in {table}; re-ingest events"
+            ) from error
+        if missing:
+            raise ValueError(
+                f"Keyword text in {table}.{field} is incomplete; re-ingest events"
+            )
+
+
 def ensure_fields(connection, requirements):
     """Materialize absent event fields as NULL, using compiler metadata."""
     for table, fields in requirements.items():
@@ -23,15 +74,25 @@ def ensure_fields(connection, requirements):
                 existing.add(field.lower())
 
 
-def execute_plan(connection, plan, *, limit=None, include_events=True):
+def execute_plan(
+    connection, plan, *, limit=None, include_events=True, fulltext_preflight=None
+):
     """Return (alert summaries, diagnostic counts), cleaning temporary relations.
 
     A limit truncates retrieval at limit+1 so the consumer can retain its
     discard-noisy-rule policy. The executor never commits the caller's work.
-    Event IDs are scoped by source table; source filenames accompany evidence.
+    Event IDs are scoped by source table; evidence carries each full source row.
+    A caller may supply a cache-aware fulltext_preflight(requirements) hook;
+    without one, version-3 plans validate on every execution.
     """
-    if plan.get("version") != 2:
+    if plan.get("version") not in (2, 3):
         raise ValueError("Unsupported correlation plan version")
+    if plan["version"] == 3:
+        requirements = plan.get("fulltext")
+        if fulltext_preflight is None:
+            validate_fulltext(connection, requirements)
+        else:
+            fulltext_preflight(requirements)
     if sqlite3.sqlite_version_info < (3, 38, 0):
         raise RuntimeError(
             "Correlation plans require SQLite >= 3.38 with JSON functions"
@@ -72,6 +133,9 @@ def execute_plan(connection, plan, *, limit=None, include_events=True):
             for field in ("group_keys", "event_ids", "child_alert_ids"):
                 row[field] = json.loads(row[field])
         if include_events:
+            internal_fields = {
+                item["field"].lower() for item in plan.get("fulltext", [])
+            }
             evidence = {}
             ids_by_table = {}
             for row in rows:
@@ -96,7 +160,9 @@ def execute_plan(connection, plan, *, limit=None, include_events=True):
                         "event_id": identity,
                         "source_table": table,
                         "event": {
-                            k: v for k, v in zip(columns, event[1:]) if v is not None
+                            k: v
+                            for k, v in zip(columns, event[1:])
+                            if v is not None and k.lower() not in internal_fields
                         },
                     }
                 cursor.close()