audit-framework-jsonl 0.1.0__tar.gz

audit_framework_jsonl-0.1.0/.gitignore

@@ -0,0 +1,35 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+.eggs/
+build/
+dist/
+wheels/
+*.egg
+
+# Test / coverage
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+.tox/
+.cache/
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+
+# Tooling / SAST artifacts
+*.sarif
+bandit.sarif
+semgrep.sarif
+
+# Editors / OS
+.idea/
+.vscode/
+*.swp
+.DS_Store

audit_framework_jsonl-0.1.0/PKG-INFO

@@ -0,0 +1,84 @@
+Metadata-Version: 2.4
+Name: audit-framework-jsonl
+Version: 0.1.0
+Summary: Append-only JSONL file ExternalSink for audit-framework — the reference sink plugin.
+Project-URL: Homepage, https://github.com/vanmarkic/audit-logger
+Project-URL: Repository, https://github.com/vanmarkic/audit-logger
+License-Expression: MIT
+Keywords: audit,audit-log,jsonl,plugin,siem,sink
+Requires-Python: >=3.11
+Requires-Dist: audit-framework<0.2,>=0.1
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# audit-framework-jsonl
+
+An append-only **JSON-Lines file sink** for
+[`audit-framework`](../audit-framework) — and the **reference implementation**
+every other `ExternalSink` (Splunk HEC, Elasticsearch, syslog, …) can be copied
+from.
+
+It implements the `ExternalSink` port: one compact JSON object per line,
+appended to a file. Writes are serialised and offloaded off the event loop, with
+optional rotation by date and/or size.
+
+## Install
+
+```bash
+pip install audit-framework-jsonl
+```
+
+## Use
+
+```python
+from audit_framework_jsonl.sink import JsonlFileSink
+from audit_framework.core.middlewares.sink_fanout import SinkFanOutMiddleware
+
+sink = JsonlFileSink("/var/log/audit/audit.jsonl", daily=True, max_bytes=50_000_000)
+pipeline.use(SinkFanOutMiddleware([sink]))   # fan events out to this sink
+```
+
+Or wire it by configuration through the registry — it advertises itself under
+the `audit_framework.plugins` entry point as the `file_jsonl` external sink:
+
+```python
+registry.discover_entrypoints()                 # finds file_jsonl
+SinkClass = registry.get("external_sink", "file_jsonl")
+sink = SinkClass("/var/log/audit/audit.jsonl")
+```
+
+Each emitted line is `event.to_dict()` serialised compactly, e.g.:
+
+```json
+{"actor_id":"alice","action":"DELETE","resource_type":"contract","resource_id":"c-42",...}
+```
+
+### Options
+
+| Param | Effect |
+|---|---|
+| `name` | The `sink_name` used for per-policy sink filtering (default `"file_jsonl"`). |
+| `daily` | Write to a date-stamped file `audit-YYYY-MM-DD.jsonl` (one per UTC day). |
+| `max_bytes` | Roll the current file aside (`audit.<timestamp>.jsonl`) before it exceeds this size. Composes with `daily`. |
+| `clock` | Injectable time source for rotation stamps (testing). |
+
+## Writing your own sink
+
+`sink.py` is deliberately tiny — copy it and change four things:
+
+1. a stable `sink_name` (matched against `AuditPolicy.sinks`),
+2. `async emit(event, context)` — forward `event.to_dict()` to your platform (best-effort; raise on permanent failure so the pipeline records it),
+3. `async health_check()` — report reachability,
+4. a `register(registry)` that calls `registry.register("external_sink", "<name>", YourSink)` (+ an `audit_framework.plugins` entry point in `pyproject.toml`).
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest        # 9 stdlib-only tests (tmp files; no infrastructure)
+```
+
+## License
+
+MIT

audit_framework_jsonl-0.1.0/README.md

@@ -0,0 +1,70 @@
+# audit-framework-jsonl
+
+An append-only **JSON-Lines file sink** for
+[`audit-framework`](../audit-framework) — and the **reference implementation**
+every other `ExternalSink` (Splunk HEC, Elasticsearch, syslog, …) can be copied
+from.
+
+It implements the `ExternalSink` port: one compact JSON object per line,
+appended to a file. Writes are serialised and offloaded off the event loop, with
+optional rotation by date and/or size.
+
+## Install
+
+```bash
+pip install audit-framework-jsonl
+```
+
+## Use
+
+```python
+from audit_framework_jsonl.sink import JsonlFileSink
+from audit_framework.core.middlewares.sink_fanout import SinkFanOutMiddleware
+
+sink = JsonlFileSink("/var/log/audit/audit.jsonl", daily=True, max_bytes=50_000_000)
+pipeline.use(SinkFanOutMiddleware([sink]))   # fan events out to this sink
+```
+
+Or wire it by configuration through the registry — it advertises itself under
+the `audit_framework.plugins` entry point as the `file_jsonl` external sink:
+
+```python
+registry.discover_entrypoints()                 # finds file_jsonl
+SinkClass = registry.get("external_sink", "file_jsonl")
+sink = SinkClass("/var/log/audit/audit.jsonl")
+```
+
+Each emitted line is `event.to_dict()` serialised compactly, e.g.:
+
+```json
+{"actor_id":"alice","action":"DELETE","resource_type":"contract","resource_id":"c-42",...}
+```
+
+### Options
+
+| Param | Effect |
+|---|---|
+| `name` | The `sink_name` used for per-policy sink filtering (default `"file_jsonl"`). |
+| `daily` | Write to a date-stamped file `audit-YYYY-MM-DD.jsonl` (one per UTC day). |
+| `max_bytes` | Roll the current file aside (`audit.<timestamp>.jsonl`) before it exceeds this size. Composes with `daily`. |
+| `clock` | Injectable time source for rotation stamps (testing). |
+
+## Writing your own sink
+
+`sink.py` is deliberately tiny — copy it and change four things:
+
+1. a stable `sink_name` (matched against `AuditPolicy.sinks`),
+2. `async emit(event, context)` — forward `event.to_dict()` to your platform (best-effort; raise on permanent failure so the pipeline records it),
+3. `async health_check()` — report reachability,
+4. a `register(registry)` that calls `registry.register("external_sink", "<name>", YourSink)` (+ an `audit_framework.plugins` entry point in `pyproject.toml`).
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest        # 9 stdlib-only tests (tmp files; no infrastructure)
+```
+
+## License
+
+MIT

audit_framework_jsonl-0.1.0/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["hatchling", "hatch-vcs"]
+build-backend = "hatchling.build"
+
+[project]
+name = "audit-framework-jsonl"
+dynamic = ["version"]
+description = "Append-only JSONL file ExternalSink for audit-framework — the reference sink plugin."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.11"
+keywords = ["audit", "audit-log", "sink", "jsonl", "siem", "plugin"]
+dependencies = ["audit-framework>=0.1,<0.2"]
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0"]
+
+[project.urls]
+Homepage = "https://github.com/vanmarkic/audit-logger"
+Repository = "https://github.com/vanmarkic/audit-logger"
+
+# Discovered by audit-framework's PluginRegistry.discover_entrypoints().
+[project.entry-points."audit_framework.plugins"]
+file_jsonl = "audit_framework_jsonl.plugin:register"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/audit_framework_jsonl"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+# Resolve the sibling core package without an install during local dev.
+pythonpath = ["src", "../audit-framework/src"]
+python_files = ["*_test.py", "test_*.py"]
+
+[tool.hatch.version]
+source = "vcs"
+raw-options = { tag_regex = "^audit-framework-jsonl-v(?P<version>.+)$", fallback_version = "0.1.0" }

audit_framework_jsonl-0.1.0/src/audit_framework_jsonl/__init__.py

@@ -0,0 +1,18 @@
+"""audit-framework-jsonl — append-only JSONL file sink for audit-framework.
+
+The reference :class:`~audit_framework.core.ports.ExternalSink` implementation:
+copy ``sink.py`` as the template for your own sink (Splunk HEC, Elasticsearch,
+syslog, …).
+"""
+
+from audit_framework_jsonl.plugin import register
+from audit_framework_jsonl.sink import JsonlFileSink
+
+import importlib.metadata as _md
+
+try:
+    __version__ = _md.version("audit-framework-jsonl")
+except _md.PackageNotFoundError:  # running from source without an install
+    __version__ = "0.0.0+unknown"
+
+__all__ = ["JsonlFileSink", "register", "__version__"]

audit_framework_jsonl-0.1.0/src/audit_framework_jsonl/plugin.py

@@ -0,0 +1,20 @@
+"""Plugin registration for the JSONL file sink.
+
+The core never imports this module directly; it is reached either via the
+``audit_framework.plugins`` entry point declared in ``pyproject.toml`` (picked
+up by :meth:`PluginRegistry.discover_entrypoints`) or by an explicit module
+path passed to :meth:`PluginRegistry.load_from_config`.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from audit_framework_jsonl.sink import JsonlFileSink
+
+__all__ = ["register"]
+
+
+def register(registry: Any) -> None:
+    """Register :class:`JsonlFileSink` as the ``file_jsonl`` external sink."""
+    registry.register("external_sink", "file_jsonl", JsonlFileSink)

audit_framework_jsonl-0.1.0/src/audit_framework_jsonl/sink.py

@@ -0,0 +1,135 @@
+"""JsonlFileSink — an append-only JSON-Lines :class:`ExternalSink`.
+
+This is the **reference sink implementation** for the audit-framework plugin
+system: the simplest possible adapter, written so a customer can copy it as the
+template for their own sink (Splunk HEC, Elasticsearch, syslog, …). The recipe
+every sink follows:
+
+1.  Expose a stable :pyattr:`sink_name` (matched against ``AuditPolicy.sinks``).
+2.  Implement ``async emit(event, context)`` — forward one event, best-effort.
+3.  Implement ``async health_check()`` — report whether the downstream is usable.
+4.  Register the class in a ``register(registry)`` function (see ``plugin.py``).
+
+This sink appends one compact JSON object per line to a file. Writes are
+serialised with an :class:`asyncio.Lock` and offloaded with
+:func:`asyncio.to_thread`, so concurrent ``emit`` calls neither interleave nor
+block the event loop. Optional rotation is supported by date (one file per UTC
+day) and/or by size (roll the current file aside once it would exceed a byte
+threshold).
+
+Only the standard library is used here; the ``audit_framework`` import is for
+type hints and is part of this plugin's declared dependency.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Callable, Optional
+
+from audit_framework.core.models import AuditEvent, PipelineContext
+
+__all__ = ["JsonlFileSink"]
+
+
+def _utc_now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+class JsonlFileSink:
+    """Appends each audit event as one JSON line to a file.
+
+    Parameters
+    ----------
+    path:
+        Destination file (its parent directories are created on demand).
+    name:
+        The :pyattr:`sink_name` used for per-policy sink filtering.
+    daily:
+        When True, write to a date-stamped file ``<stem>-YYYY-MM-DD<suffix>``
+        so each UTC day gets its own file.
+    max_bytes:
+        When set, roll the current file aside (``<stem>.<timestamp><suffix>``)
+        before a write that would push it past this size. Composes with
+        ``daily``.
+    clock:
+        Injectable time source (returns an aware :class:`datetime`); used for
+        rotation stamps. Defaults to ``datetime.now(timezone.utc)``.
+    """
+
+    def __init__(
+        self,
+        path: str | os.PathLike[str],
+        *,
+        name: str = "file_jsonl",
+        daily: bool = False,
+        max_bytes: Optional[int] = None,
+        clock: Callable[[], datetime] = _utc_now,
+    ) -> None:
+        self._base = Path(path)
+        self._name = name
+        self._daily = daily
+        self._max_bytes = max_bytes
+        self._clock = clock
+        self._lock = asyncio.Lock()
+
+    @property
+    def sink_name(self) -> str:
+        """Stable identifier matched against ``AuditPolicy.sinks``."""
+        return self._name
+
+    async def emit(self, event: AuditEvent, context: PipelineContext) -> None:
+        """Append ``event`` (as one JSON line) to the current target file.
+
+        Best-effort and serialised: a raised OSError propagates so the
+        ``SinkFanOutMiddleware`` can record the failure, but it never corrupts a
+        concurrent write (the lock guarantees one writer at a time).
+        """
+        line = json.dumps(
+            event.to_dict(), separators=(",", ":"), ensure_ascii=False, default=str
+        )
+        async with self._lock:
+            await asyncio.to_thread(self._write, line)
+
+    async def health_check(self) -> bool:
+        """Return True if the target directory exists (or can be) and is writable."""
+        return await asyncio.to_thread(self._check_writable)
+
+    # ----------------------------------------------------------------- #
+    # Blocking helpers (run inside asyncio.to_thread)                    #
+    # ----------------------------------------------------------------- #
+    def _write(self, line: str) -> None:
+        path = self._target_path()
+        path.parent.mkdir(parents=True, exist_ok=True)
+        if self._max_bytes is not None and path.exists():
+            projected = path.stat().st_size + len(line.encode("utf-8")) + 1
+            if projected > self._max_bytes:
+                self._rollover(path)
+        with path.open("a", encoding="utf-8") as fh:
+            fh.write(line + "\n")
+
+    def _check_writable(self) -> bool:
+        try:
+            parent = self._base.parent
+            parent.mkdir(parents=True, exist_ok=True)
+            return os.access(parent, os.W_OK)
+        except OSError:
+            return False
+
+    def _target_path(self) -> Path:
+        if not self._daily:
+            return self._base
+        stamp = self._clock().strftime("%Y-%m-%d")
+        return self._base.with_name(f"{self._base.stem}-{stamp}{self._base.suffix}")
+
+    def _rollover(self, path: Path) -> None:
+        stamp = self._clock().strftime("%Y%m%dT%H%M%S")
+        target = path.with_name(f"{path.stem}.{stamp}{path.suffix}")
+        suffix = 0
+        while target.exists():  # never clobber an existing rolled file
+            suffix += 1
+            target = path.with_name(f"{path.stem}.{stamp}.{suffix}{path.suffix}")
+        path.rename(target)

audit_framework_jsonl-0.1.0/tests/sink_test.py

@@ -0,0 +1,155 @@
+"""Tests for JsonlFileSink — stdlib-only, no infrastructure (uses tmp_path)."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from datetime import datetime, timezone
+
+import pytest
+
+from audit_framework.core.models import AuditEvent, PipelineContext
+from audit_framework.core.ports import ExternalSink
+from audit_framework.core.plugin_registry import PluginRegistry
+
+from audit_framework_jsonl.plugin import register
+from audit_framework_jsonl.sink import JsonlFileSink
+
+
+def _event(resource_id: str = "c-1", action: str = "DELETE") -> AuditEvent:
+    return AuditEvent(
+        actor_id="alice",
+        action=action,
+        resource_type="contract",
+        resource_id=resource_id,
+        timestamp="2026-06-26T00:00:00+00:00",
+        request_id="req-1",
+        changes={"amount": {"old": 1, "new": 2}},
+    )
+
+
+def _ctx(event: AuditEvent) -> PipelineContext:
+    return PipelineContext(event=event)
+
+
+def test_satisfies_external_sink_protocol(tmp_path) -> None:
+    sink = JsonlFileSink(tmp_path / "audit.jsonl")
+    assert isinstance(sink, ExternalSink)  # structural (runtime_checkable) check
+    assert sink.sink_name == "file_jsonl"
+
+
+def test_emit_appends_one_json_line_per_event(tmp_path) -> None:
+    path = tmp_path / "audit.jsonl"
+    sink = JsonlFileSink(path)
+
+    asyncio.run(sink.emit(_event("c-1"), _ctx(_event("c-1"))))
+    asyncio.run(sink.emit(_event("c-2"), _ctx(_event("c-2"))))
+
+    lines = path.read_text(encoding="utf-8").splitlines()
+    assert len(lines) == 2
+    first = json.loads(lines[0])
+    assert first["action"] == "DELETE"
+    assert first["resource_id"] == "c-1"
+    assert first["changes"] == {"amount": {"old": 1, "new": 2}}
+    assert json.loads(lines[1])["resource_id"] == "c-2"
+
+
+def test_creates_parent_directories(tmp_path) -> None:
+    path = tmp_path / "nested" / "dir" / "audit.jsonl"
+    sink = JsonlFileSink(path)
+    ev = _event()
+    asyncio.run(sink.emit(ev, _ctx(ev)))
+    assert path.exists()
+
+
+def test_daily_rotation_uses_date_stamped_file(tmp_path) -> None:
+    clock = lambda: datetime(2026, 6, 26, 12, 0, tzinfo=timezone.utc)
+    sink = JsonlFileSink(tmp_path / "audit.jsonl", daily=True, clock=clock)
+    ev = _event()
+
+    asyncio.run(sink.emit(ev, _ctx(ev)))
+
+    assert (tmp_path / "audit-2026-06-26.jsonl").exists()
+    assert not (tmp_path / "audit.jsonl").exists()
+
+
+def test_size_rollover_moves_current_file_aside(tmp_path) -> None:
+    path = tmp_path / "audit.jsonl"
+    # tiny threshold so the second write triggers a rollover of the first
+    sink = JsonlFileSink(path, max_bytes=10)
+    ev = _event()
+
+    asyncio.run(sink.emit(ev, _ctx(ev)))  # creates the file
+    asyncio.run(sink.emit(ev, _ctx(ev)))  # rolls the full file aside, writes fresh
+
+    rolled = list(tmp_path.glob("audit.*.jsonl"))
+    assert len(rolled) == 1  # exactly one rolled-aside file
+    assert len(rolled[0].read_text(encoding="utf-8").splitlines()) == 1
+    assert len(path.read_text(encoding="utf-8").splitlines()) == 1  # fresh current file
+
+
+def test_health_check_true_for_writable_dir(tmp_path) -> None:
+    sink = JsonlFileSink(tmp_path / "sub" / "audit.jsonl")
+    assert asyncio.run(sink.health_check()) is True
+
+
+def test_concurrent_emits_do_not_interleave(tmp_path) -> None:
+    path = tmp_path / "audit.jsonl"
+    sink = JsonlFileSink(path)
+
+    async def emit_many() -> None:
+        events = [_event(f"c-{i}") for i in range(25)]
+        await asyncio.gather(*(sink.emit(e, _ctx(e)) for e in events))
+
+    asyncio.run(emit_many())
+
+    lines = path.read_text(encoding="utf-8").splitlines()
+    assert len(lines) == 25
+    # every line is intact JSON (no corruption) and all events are present
+    ids = sorted(json.loads(line)["resource_id"] for line in lines)
+    assert ids == sorted(f"c-{i}" for i in range(25))
+
+
+def test_register_wires_sink_into_registry() -> None:
+    registry = PluginRegistry()
+    register(registry)
+    assert registry.get("external_sink", "file_jsonl") is JsonlFileSink
+    assert "file_jsonl" in registry.list_providers("external_sink")
+
+
+def test_end_to_end_through_the_pipeline(tmp_path) -> None:
+    # Prove the reference sink composes with the real core: an event audited by
+    # the pipeline is fanned out to the JSONL file by SinkFanOutMiddleware.
+    from audit_framework.core.middlewares.audit_policy import AuditPolicyMiddleware
+    from audit_framework.core.middlewares.sink_fanout import SinkFanOutMiddleware
+    from audit_framework.core.models import AuditPolicy
+    from audit_framework.core.pipeline import Pipeline
+
+    class _PolicyStore:
+        def get_audit_policies(self):
+            return [AuditPolicy(name="all", match={})]
+
+        def get_broadcast_policies(self):
+            return []
+
+        def reload(self):
+            pass
+
+    path = tmp_path / "audit.jsonl"
+    sink = JsonlFileSink(path)
+    pipeline = (
+        Pipeline()
+        .use(AuditPolicyMiddleware(_PolicyStore()))
+        .use(SinkFanOutMiddleware([sink]))
+    )
+
+    ctx = asyncio.run(pipeline.execute(_event("c-9")))
+
+    assert "sink_failures" not in ctx.metadata
+    lines = path.read_text(encoding="utf-8").splitlines()
+    assert len(lines) == 1
+    assert json.loads(lines[0])["resource_id"] == "c-9"
+
+
+if __name__ == "__main__":
+    raise SystemExit(pytest.main([__file__, "-v"]))