fluidattacks-agent 0.1.1__tar.gz

fluidattacks_agent-0.1.1/.gitignore

@@ -0,0 +1,8 @@
+# Rust build artifacts (top-level crate and the ebpf/ sub-crates)
+target/
+
+# Python tooling caches from the probe's lint and test targets
+__pycache__/
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/

fluidattacks_agent-0.1.1/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.5
+Name: fluidattacks-agent
+Version: 0.1.1
+Summary: In-process probe reporting what a Python workload imports and runs
+Project-URL: Homepage, https://fluidattacks.com
+Project-URL: Source, https://gitlab.com/fluidattacks/universe/-/tree/trunk/watches/agents/python
+Author-email: Development <development@fluidattacks.com>
+License: MPL-2.0
+Keywords: dependencies,observability,reachability,runtime,sbom
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# fluidattacks-agent
+
+Reports what a running Python workload actually imports and runs, so that a
+dependency inventory can say which of its findings are reachable at runtime and
+which are not.
+
+It is a library, not a service. It observes the interpreter it is installed in,
+writes what it saw, and does nothing else. It takes no dependencies: the
+standard library only, because it is installed into workloads we do not own.
+
+## Installing
+
+```
+pip install fluidattacks-agent
+```
+
+That is the whole setup. A `.pth` file at the root of the wheel starts the probe
+at interpreter start, before the workload's own program runs, so nothing has to
+be imported or called by hand.
+
+## Turning it off
+
+```
+FLUIDATTACKS_AGENT=off
+```
+
+Also `0`, `false`, `no`, `disabled` or `none`. A workload that says no pays for
+reading the setting and for nothing above it. Saying nothing is taken for yes,
+because installing the package is the consent.
+
+## What it observes
+
+- distributions whose modules were imported, and which modules
+- functions that were executed, where the interpreter offers that
+- how often, in windows, and when a symbol was first reached
+
+Only what an installed distribution owns is attributed. The standard library and
+a workload's own first-party code produce no records.
+
+## Where reports go
+
+By default, one file per report under `/tmp/.watches-exec`, for a collector to
+drain. Naming an endpoint sends them instead:
+
+| | |
+|---|---|
+| `FLUIDATTACKS_AGENT_ENDPOINT` | where reports are posted, `https://` only |
+| `FLUIDATTACKS_AGENT_TOKEN_FILE` | a file holding the credential, preferred |
+| `FLUIDATTACKS_AGENT_TOKEN` | the credential itself, read only if no file is named |
+| `FLUIDATTACKS_AGENT_GROUP` | what the reports are filed under |
+| `FLUIDATTACKS_AGENT_WORKLOAD` | what this workload is called |
+
+A file is preferred over a variable because a file can be mode 400, while an
+environment variable is readable by any process of the same user.
+
+An endpoint named without enough beside it to reach is a misconfiguration, not a
+reason to fall back: the probe then holds nothing and counts every report it
+refused, so a half-configured deployment is loud rather than a directory filling
+up where nobody drains it.
+
+## What travels, and what does not
+
+Reports are gzipped and signed with a key derived from the credential; the
+credential itself never travels, appears in no record, and is in no exception.
+The far end must prove who it is — certificate chain and hostname both — and no
+redirect is followed.
+
+What a report contains is distribution names, versions, module and function
+names, and counts. No arguments, no return values, no file contents, no
+environment.
+
+## Licence
+
+MPL-2.0

fluidattacks_agent-0.1.1/README.md

@@ -0,0 +1,74 @@
+# fluidattacks-agent
+
+Reports what a running Python workload actually imports and runs, so that a
+dependency inventory can say which of its findings are reachable at runtime and
+which are not.
+
+It is a library, not a service. It observes the interpreter it is installed in,
+writes what it saw, and does nothing else. It takes no dependencies: the
+standard library only, because it is installed into workloads we do not own.
+
+## Installing
+
+```
+pip install fluidattacks-agent
+```
+
+That is the whole setup. A `.pth` file at the root of the wheel starts the probe
+at interpreter start, before the workload's own program runs, so nothing has to
+be imported or called by hand.
+
+## Turning it off
+
+```
+FLUIDATTACKS_AGENT=off
+```
+
+Also `0`, `false`, `no`, `disabled` or `none`. A workload that says no pays for
+reading the setting and for nothing above it. Saying nothing is taken for yes,
+because installing the package is the consent.
+
+## What it observes
+
+- distributions whose modules were imported, and which modules
+- functions that were executed, where the interpreter offers that
+- how often, in windows, and when a symbol was first reached
+
+Only what an installed distribution owns is attributed. The standard library and
+a workload's own first-party code produce no records.
+
+## Where reports go
+
+By default, one file per report under `/tmp/.watches-exec`, for a collector to
+drain. Naming an endpoint sends them instead:
+
+| | |
+|---|---|
+| `FLUIDATTACKS_AGENT_ENDPOINT` | where reports are posted, `https://` only |
+| `FLUIDATTACKS_AGENT_TOKEN_FILE` | a file holding the credential, preferred |
+| `FLUIDATTACKS_AGENT_TOKEN` | the credential itself, read only if no file is named |
+| `FLUIDATTACKS_AGENT_GROUP` | what the reports are filed under |
+| `FLUIDATTACKS_AGENT_WORKLOAD` | what this workload is called |
+
+A file is preferred over a variable because a file can be mode 400, while an
+environment variable is readable by any process of the same user.
+
+An endpoint named without enough beside it to reach is a misconfiguration, not a
+reason to fall back: the probe then holds nothing and counts every report it
+refused, so a half-configured deployment is loud rather than a directory filling
+up where nobody drains it.
+
+## What travels, and what does not
+
+Reports are gzipped and signed with a key derived from the credential; the
+credential itself never travels, appears in no record, and is in no exception.
+The far end must prove who it is — certificate chain and hostname both — and no
+redirect is followed.
+
+What a report contains is distribution names, versions, module and function
+names, and counts. No arguments, no return values, no file contents, no
+environment.
+
+## Licence
+
+MPL-2.0

fluidattacks_agent-0.1.1/fluidattacks_agent/batch.py

@@ -0,0 +1,161 @@
+"""Pack a report into a batch the centre can order, verify and decompress."""
+
+import os
+import threading
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from typing import Final
+
+from fluidattacks_agent.settings import Delivery, Secret
+
+BATCH_TAG: Final = "watches-batch/1"
+
+# what the centre keys on. The whitepaper's unit is a machine that outlives a
+# reboot; ours is an interpreter that may live for seconds, so what names one is
+# minted per start and never reused
+INSTANCE_BYTES: Final = 16
+
+# a probe's own health is not a workload's evidence and must not be filed as it
+EXEC: Final = "exec"
+
+# what orders one instance's batches. Wrapping would land on a key already
+# ingested and be discarded as a duplicate, so the counter refuses instead
+MAX_SEQ: Final = 1 << 53
+
+# a flush runs inside a host callback, so this is the workload's own latency:
+# measured on a mebibyte of real report text, level one costs 4.5 ms for 3.1x
+# and level six costs 18.6 ms for 3.7x. Fourteen milliseconds of somebody
+# else's request is not worth sixty kibibytes
+COMPRESSION: Final = 1
+
+
+def packed(text: str) -> bytes:
+    """Compress a report, the same way every time it holds the same thing."""
+    # here and not above: a flush is what needs this, and two milliseconds of
+    # interpreter start is not owed by a workload that never reaches one
+    import gzip  # noqa: PLC0415
+
+    # mtime zero because a gzip header otherwise carries a clock, and a probe
+    # that asserts no phase of its own should not assert one here either
+    return gzip.compress(text.encode(), compresslevel=COMPRESSION, mtime=0)
+
+
+@dataclass(frozen=True, slots=True)
+class Batch:
+    """One delivery: what names it, what orders it, and what proves it."""
+
+    instance: str
+    seq: int
+    stream: str
+    workload: str
+    group: str
+    body: bytes
+    signature: str
+
+
+def minted() -> str:
+    """Name this interpreter, once, out of the kernel's own entropy."""
+    # not secrets.token_hex: the same source at the same 0.8 us a call, but
+    # importing it costs 3.5 ms of a start this package spent months shortening
+    return os.urandom(INSTANCE_BYTES).hex()
+
+
+def _stated(*parts: str) -> bytes:
+    # length-prefixed over the encoded bytes, so that no two different batches
+    # can spell alike whatever the values hold: being unambiguous is a property
+    # of this form rather than a promise made somewhere else
+    return b"\n".join(b"%d:%s" % (len(spelt), spelt) for spelt in map(str.encode, parts))
+
+
+def derived(token: Secret) -> bytes:
+    """Make a key that signs, and is not the credential that authorises."""
+    import hmac  # noqa: PLC0415
+
+    # so a signature cannot be replayed as a credential, and a credential
+    # cannot be read back out of a signature
+    return hmac.digest(token.held.encode(), BATCH_TAG.encode(), "sha256")
+
+
+def batched(
+    delivery: Delivery,
+    instance: str,
+    seq: int,
+    stream: str,
+    text: str,
+) -> Batch | None:
+    """Name a report, order it, pack it, and prove where it came from."""
+    if not 0 < seq < MAX_SEQ:
+        return None
+    import hmac  # noqa: PLC0415
+
+    body = packed(text)
+    workload = delivery.workload or ""
+    group = delivery.group or ""
+    proof = hmac.new(derived(delivery.token), digestmod="sha256")
+    # what travels beside the body is signed with it, because a proxy that can
+    # rewrite which instance or which tenant sent this misfiles it without
+    # touching a byte of the evidence
+    proof.update(_stated(BATCH_TAG, group, workload, instance, str(seq), stream))
+    proof.update(body)
+    return Batch(
+        instance=instance,
+        seq=seq,
+        stream=stream,
+        workload=workload,
+        group=group,
+        body=body,
+        signature=proof.hexdigest(),
+    )
+
+
+@dataclass
+class Batcher:
+    """One instance's batches, in the order that instance made them."""
+
+    instance: str = ""
+    seq: int = 0
+    # two host callbacks can be inside a flush at the same time, because what
+    # keeps a second one out is a plain attribute and not a mutex: measured,
+    # eight threads asking for a place got the same one in all three hundred
+    # rounds, and two batches sharing a place have one of them discarded
+    guard: threading.Lock = field(default_factory=threading.Lock, repr=False)
+
+    def named(self) -> str:
+        """Say what names this interpreter, minting it the first time asked."""
+        if not self.instance:
+            self.instance = minted()
+        return self.instance
+
+    def next(
+        self,
+        delivery: Delivery,
+        text: str,
+        taken: Callable[[Batch], bool],
+        stream: str = EXEC,
+    ) -> Batch | None:
+        """Build the next batch, and spend its place only once it is held."""
+        # taking the place, building against it and offering it are one step, so
+        # that no two batches can be built against the same place and no place
+        # is spent on a batch that was turned away. Whoever takes it is asked
+        # from in here for that reason, and never asks anything back of this
+        with self.guard:
+            held = self.seq + 1
+            batch = batched(delivery, self.named(), held, stream, text)
+            if batch is None or not taken(batch):
+                return None
+            # a place spent on a batch nothing is holding would leave a hole,
+            # and a hole is how evidence lost in flight is meant to look
+            self.seq = held
+            return batch
+
+    def forked(self) -> None:
+        """Forget what named this instance, because a child of it is another."""
+        # a lock held by another thread at the moment of the fork is held in the
+        # child forever, and a fresh one cannot be: the child has no second
+        # thread, so there is nothing for this one to be held against
+        self.guard = threading.Lock()
+        # a child inherits the name and the count and goes on emitting the keys
+        # its parent is emitting: measured across a fork, every key after it
+        # collided, and ingest discards a collision without saying so
+        self.instance = ""
+        self.seq = 0

fluidattacks_agent-0.1.1/fluidattacks_agent/deliver.py

@@ -0,0 +1,158 @@
+"""Carry what a workload reported out of the process it was reported in."""
+
+import contextlib
+import errno
+import os
+import threading
+import time
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from typing import Final
+
+from fluidattacks_agent.batch import Batch, Batcher
+from fluidattacks_agent.outbox import Outbox
+from fluidattacks_agent.patience import TOPMOST, fraction, splay, standoff
+from fluidattacks_agent.post import TIMEOUT, Verdict, posted
+from fluidattacks_agent.settings import Delivery
+from fluidattacks_agent.sink import Stalled
+
+# how old the centre's picture may be while nothing new is being found
+INTERVAL: Final = 15.0
+
+# what a workload's own exit may be delayed by while what is held goes out
+PARTING: Final = 2.0
+
+NAME: Final = "fluidattacks-agent"
+
+
+@dataclass
+class Deliverer:
+    """A sink that puts a report on its way out rather than in a directory."""
+
+    told: Delivery
+    batcher: Batcher = field(default_factory=Batcher)
+    outbox: Outbox = field(default_factory=Outbox)
+    carry: Callable[[Delivery, Batch, float], Verdict] = posted
+    # kept in hand rather than put back, so its place stays its place
+    holding: Batch | None = field(default=None, repr=False)
+    interval: float = INTERVAL
+    share: Callable[[], float] = fraction
+    # a wake says there is something to carry, never that it is welcome
+    due: float = 0.0
+    tries: int = 0
+    woken: threading.Event = field(default_factory=threading.Event, repr=False)
+    thread: threading.Thread | None = field(default=None, repr=False)
+    # the thread field alone, and never held across anything that can block
+    guard: threading.Lock = field(default_factory=threading.Lock, repr=False)
+    # one pass at a time: what is taken but not delivered lives in one place
+    passing: threading.Lock = field(default_factory=threading.Lock, repr=False)
+    leaving: bool = False
+    delivered: int = 0
+
+    def write(self, text: str) -> None:
+        """Take a report for delivery, or refuse and be counted for it."""
+        if self.batcher.next(self.told, text, self.outbox.put) is None:
+            raise Stalled(errno.ENOSPC, "nothing taken", len(text))
+        self._rouse()
+
+    def _rouse(self) -> None:
+        """Say there is something to carry, and find a carrier if there is none."""
+        with self.guard:
+            # a carrier that stopped takes delivery with it, and one is not
+            # worth starting for a process already on its way out
+            if not self.leaving and (self.thread is None or not self.thread.is_alive()):
+                # replicas start in the same second, so this is drawn; and
+                # once, so that it can never overwrite a standoff
+                if not self.due:
+                    self.due = time.monotonic() + splay(self.interval, self.share())
+                # nothing here may wait on the carrier: a thread importing what
+                # a post needs blocks while its starter is mid-import
+                self.thread = threading.Thread(target=self._serving, name=NAME, daemon=True)
+                self.thread.start()
+        self.woken.set()
+
+    def _serving(self) -> None:
+        while not self.leaving:
+            # a thread that dies takes delivery with it and tells nobody
+            with contextlib.suppress(Exception):
+                self.woken.wait(self._waiting())
+                self.woken.clear()
+                if time.monotonic() >= self.due:
+                    self.offering()
+
+    def parting(self) -> None:
+        """Give whatever is held one bounded chance to travel, then give up."""
+        self.leaving = True
+        self.woken.set()
+        limit = time.monotonic() + PARTING
+        # a carrier mid-pass is doing this work, so the wait is that budget
+        if not self.passing.acquire(timeout=PARTING):
+            return
+        try:
+            with contextlib.suppress(Exception):
+                # bounded by what is left, or the bound is only the moment a
+                # last attempt may begin at
+                while (left := limit - time.monotonic()) > 0 and self._offered(left):
+                    pass
+        finally:
+            self.passing.release()
+
+    def forked(self) -> None:
+        """Leave a child holding none of its parent's, and owing itself a carrier."""
+        # a lock or event held at the moment of the fork is held in the child
+        # forever, so each is made again rather than reset
+        self.guard = threading.Lock()
+        self.passing = threading.Lock()
+        self.woken = threading.Event()
+        self.thread = None
+        self.holding = None
+        # every worker of a pre-forking server was born in the same second
+        self.due = 0.0
+        self.tries = 0
+        # a child is not on its way out merely because its parent was
+        self.leaving = False
+        self.batcher.forked()
+        self.outbox.forked()
+
+    def registered(self) -> None:
+        """Ask the interpreter to say when this process has become two."""
+        # cannot be taken back once given, so exactly one per process
+        if hasattr(os, "register_at_fork"):
+            os.register_at_fork(after_in_child=self.forked)
+
+    def offering(self) -> None:
+        """Offer what is waiting, one pass at a time, stopping at a refusal."""
+        if not self.passing.acquire(blocking=False):
+            return
+        try:
+            for _ in range(self.outbox.limit):
+                if not self._offered(TIMEOUT):
+                    return
+        finally:
+            self.passing.release()
+
+    def _offered(self, bound: float) -> bool:
+        """Offer one batch, and say whether offering another is worth trying."""
+        batch = self.holding or self.outbox.take()
+        if batch is None:
+            return False
+        # held before the attempt, so a cut-off leaves it findable
+        self.holding = batch
+        if (verdict := self.carry(self.told, batch, bound)) is not Verdict.DELIVERED:
+            self._stood_off(verdict)
+            return False
+        self.holding = None
+        self.due = 0.0
+        self.tries = 0
+        self.delivered += 1
+        return True
+
+    def _stood_off(self, verdict: Verdict) -> None:
+        """Wait longer for each refusal, and longest for one about nothing of ours."""
+        self.tries = TOPMOST if verdict is Verdict.STOP else self.tries + 1
+        self.due = time.monotonic() + standoff(self.tries, self.share())
+
+    def _waiting(self) -> float:
+        """Wait out a standoff if there is one, and otherwise a pass."""
+        left = self.due - time.monotonic()
+        return left if left > 0 else self.interval

fluidattacks_agent-0.1.1/fluidattacks_agent/distributions.py

@@ -0,0 +1,80 @@
+"""Map an imported module to the distribution that installed it."""
+
+from dataclasses import dataclass
+
+from fluidattacks_agent.report import Confidence, Name, readable
+
+
+@dataclass(frozen=True)
+class Dist:
+    """The distribution behind a module, and how well its version is known."""
+
+    name: Name
+    version: str | None
+    confidence: Confidence
+
+
+@dataclass(frozen=True)
+class Declared:
+    """The versions read off installed metadata, and the ones that would not."""
+
+    versions: dict[Name, str]
+    unreadable: int
+
+
+def declared() -> Declared:
+    """Read the version every installed distribution declares for itself."""
+    # imported here and not above: reading metadata costs twenty milliseconds of
+    # interpreter start, and it is a flush that needs it, not an import
+    from importlib.metadata import distributions  # noqa: PLC0415
+
+    found: dict[Name, str] = {}
+    unreadable = 0
+    for dist in distributions():
+        # metadata is a file the workload owns: one unreadable distribution
+        # must not cost the map, since without it the probe reports nothing
+        try:
+            metadata = dist.metadata
+            name, version = metadata["Name"], metadata["Version"]
+        except Exception:  # noqa: BLE001
+            unreadable += 1
+            continue
+        # the type keys it the way it will be looked up, which also folds two
+        # spellings of one project together, since PEP 503 says they are one
+        if name is not None and version is not None:
+            found.setdefault(Name(name), version)
+    return Declared(versions=found, unreadable=unreadable)
+
+
+def resolve(names: list[str], versions: dict[Name, str]) -> Dist:
+    """Choose the distribution behind a module, and say how sure that is."""
+    name = Name(names[0])
+    version = versions.get(name)
+    # a version the reader would refuse is one we could not read, and dropping
+    # the record over it would lose the package instead of the version
+    if version is None or not readable(version):
+        return Dist(name=name, version=None, confidence=Confidence.UNKNOWN)
+    known = Confidence.EXACT if len(names) == 1 else Confidence.INFERRED
+    return Dist(name=name, version=version, confidence=known)
+
+
+def distribution_map() -> dict[str, Dist]:
+    """
+    Read which distribution installed each top-level module, and its version.
+
+    This walks every installed distribution's metadata, so it is taken once per
+    report rather than once per module. The probe spends the workload's own cpu.
+    """
+    from importlib.metadata import packages_distributions  # noqa: PLC0415
+
+    versions = declared().versions
+    return {
+        module: resolve(names, versions)
+        for module, names in packages_distributions().items()
+        if names
+    }
+
+
+def distribution_of(module: str, installed: dict[str, Dist]) -> Dist | None:
+    """Name the distribution a module came from, submodules included."""
+    return installed.get(module.split(".", maxsplit=1)[0])