aggregate_api 1.0.0__py3-none-any.whl

aggregate_api/bounds.py

@@ -0,0 +1,331 @@
+"""Pricing bounds: the range of prices consistent with one calibration.
+
+Three questions, three classes in ``aggregate.bounds``, and this module is the
+thin layer that asks them.
+
+Ordinary pricing picks a distortion and reports the number it gives. That
+number is only as firm as the choice of distortion, and the choice is a
+judgment. Bounds asks the other question: hold the calibration fixed, let the
+distortion range over everything consistent with it, and report how wide the
+answer can be. A narrow range means the calibration decided the price; a wide
+one means the distortion did.
+
+* :class:`aggregate.bounds.Bounds` is the picture. Every distortion pricing the
+  risk to the target premium is a point in the (s, g(s)) plane, and the envelope
+  is the band they sweep.
+* :class:`aggregate.bounds.PricingBounds` carries the calibration across to a
+  *second* risk: given that some distortion prices X to P, what can it say about
+  Y? That is the question behind quoting a new line off an existing book.
+* :class:`aggregate.bounds.AllocationBounds` turns it inward, on the units of
+  one portfolio: the range of natural allocation consistent with the total
+  premium.
+
+Notes
+-----
+**The cost worry recorded in the plan was unfounded**, and this is the
+measurement that settled it. Constructing a ``Bounds`` is 0.01 s, ``cloud_df``
+is 0.12 s, and the fifty-resample envelope figure is 0.5 to 0.8 s, on the
+fixtures the tests use. The resamples are overplotted columns drawn from a frame
+that is already computed, so asking for fifty rather than none costs the drawing
+and nothing else.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from aggregate import Aggregate, Portfolio
+from aggregate import charts as agg_charts
+from aggregate.bounds import AllocationBounds, Bounds, PricingBounds
+
+#: How many bracket columns the envelope overplots. Fifty is enough to read the
+#: band as a band rather than a boundary, and cheap for the reason in the module
+#: notes.
+DEFAULT_RESAMPLES = 50
+
+#: Gone at a60, with the figure it grouped: ``ENVELOPE_PANELS`` split the five
+#: named distortions two-and-three because the matplotlib compositor drew three
+#: panels, and a52 built that list here to work around ``plot_envelope``
+#: silently ignoring the ``'space'`` shorthand the api had been passing. Both
+#: problems left with the figure. The emitter puts all five on one band, which
+#: is the comparison the panel is for, and the api no longer has an opinion
+#: about panel contents at all.
+
+
+def _document(df, formats: str = "price") -> dict | None:
+    """One frame as a table document, or None. Best effort, as in ``pricing``."""
+    from .tables import frame_document_dict
+
+    try:
+        return frame_document_dict(df, formats=formats)
+    except Exception:  # noqa: BLE001 -- a static table is never worth a 500
+        return None
+
+
+def _calibrate_for_envelope(obj: Any, premium: float, assets: float | None) -> bool:
+    """Calibrate the named distortions onto ``obj``, for the envelope's panel 2.
+
+    Calibrated to **the premium the request already gave**, which is the whole
+    point of the exhibit: panel 1 is every distortion consistent with that
+    premium, and panel 2 names the ones the calibration produces, so they have
+    to be the same premium or the two panels answer different questions.
+
+    Returns ``True`` when the object came away carrying a calibration, which is
+    what decides whether the emitter has a second panel to draw. It does not
+    build or group anything itself: which distortions go on which panel is the
+    emitter's business now, and through a59 this function split them across two
+    panels because the matplotlib compositor drew three.
+
+    ``calibrate_distortions`` takes a cost-of-capital rather than a premium, so
+    the premium is resolved through :func:`aggregate_api.pricing._coc_for_premium`,
+    which completes the pentagon and reads the cost of capital off it.
+
+    Through 1.0.0a100 that identity was written out here::
+
+        L = E[min(X, a)]        the limited expected loss
+        M = premium - L         the margin
+        Q = a - premium         the capital
+        coc = M / Q
+
+    which was this repo deciding what a price means, and it predated
+    ``price_pentagon`` being reachable from the api. The pentagon is one library
+    call and the identity stays upstream where it belongs.
+
+    **That hand form was also wrong, in a small way.** ``prob_loss_assets``
+    snaps the asset level to the loss grid and reports ``L`` there, while ``Q``
+    was computed from the caller's raw request, so the margin and the capital
+    came off two different asset levels and the cost of capital they implied
+    belonged to no consistent pentagon. On a 50 claim lognormal book at
+    ``log2=13``, a request for 15625.068 snaps to 15624.0 and the two readings
+    are 0.13334437 against 0.13335957, about 1.1e-4 relative; it scales with the
+    distance from a grid point. Panel 2 is calibrated on one asset level now,
+    the one the band is drawn at.
+
+    Returns ``False`` rather than raising, in three cases, and each leaves the
+    document honestly one-panelled instead of falsely two:
+
+    * **no asset cap.** Capital is unbounded, so there is no cost of capital to
+      calibrate to. The envelope in panel 1 is still meaningful.
+    * **degenerate margin or capital.** ``Bounds`` already refuses a premium
+      below the mean or above the cap; this catches the boundary where premium
+      equals the cap and capital is zero, which now reaches this function as the
+      library's own refusal out of the pentagon rather than as a comparison here.
+    * **the calibration itself declines**, which it does on a book where a mass
+      distortion cannot be fitted.
+
+    All three are rarer than they were. Since a100 the app carries a real
+    calibration into this pane rather than opening on ``mean * 1.25``, and an
+    implied ``(P, a)`` has ``M > 0`` and ``Q > 0`` by construction, so panel 2
+    draws where it used to vanish. They stay because a hand typed premium can
+    still hit them.
+
+    Notes
+    -----
+    This **mutates** the cached object: ``calibrate_distortions`` writes
+    ``distortions``, ``distortion_df`` and ``calibration_df`` onto it. That is
+    the library's contract for the method and already how the pricing runners
+    use it, so the object is no more shared-mutable than before; it is worth
+    knowing that a Bounds request leaves a calibration behind.
+    """
+    import math
+
+    from .pricing import _coc_for_premium
+
+    if assets is None or not math.isfinite(float(assets)):
+        return False
+    assets = float(assets)
+    try:
+        coc = _coc_for_premium(obj, {"a": assets}, float(premium), None)
+    except Exception:  # noqa: BLE001 -- an object that cannot answer gets one panel
+        return False
+    # A premium at or above the cap is negative capital and a premium below the
+    # limited expected loss is a negative margin. The library reports either as
+    # a non-positive cost of capital, so one test covers both boundaries.
+    if not (coc is not None and coc > 0):
+        return False
+    try:
+        obj.calibrate_distortions(coc, a=assets)
+    except Exception:  # noqa: BLE001 -- reported by the document having one panel
+        return False
+    return bool(getattr(obj, "distortions", None))
+
+
+def _require_risk(obj: Any) -> Any:
+    """Resolve to the risk the bounds classes take, or reject.
+
+    The library's own accepted set, not a kind list of ours: ``Bounds`` takes a
+    ``Portfolio``, an ``Aggregate``, a Series or a DataFrame, and the two the
+    api can hold are the first two.
+
+    A P&L resolves to its wrapped engine here, mirroring the library's own
+    unwrap (aggregate 1.0.0a375, [Bounds-PnL-Engine]). Resolving at the door
+    rather than leaning on the library's is deliberate: the envelope's second
+    panel calibrates onto and reads off the object handed to ``Bounds``
+    (``bounds._obj.distortions``), and a P&L carries no
+    ``calibrate_distortions`` of its own, so the engine has to be the object
+    the whole route works with. A kernel P&L wraps no engine and is rejected
+    like any other wrong kind.
+    """
+    engine = getattr(obj, "engine", None)
+    if not isinstance(obj, (Aggregate, Portfolio)) and \
+            isinstance(engine, (Aggregate, Portfolio)):
+        obj = engine
+    if not isinstance(obj, (Aggregate, Portfolio)):
+        raise ValueError("pricing bounds apply to an Aggregate or a Portfolio")
+    return obj
+
+
+def run_envelope(
+    obj: Any,
+    *,
+    premium: float,
+    assets: float | None = None,
+    n_resamples: int = DEFAULT_RESAMPLES,
+) -> tuple[bytes, str]:
+    """The envelope as a chart document: canonical bytes and their hash.
+
+    Panel one is the band of admissible prices with the bracketing cloud inside
+    it. Panel two puts the named distortions, calibrated to this request's own
+    premium, on the same band, so you can see which part of the feasible space
+    each one occupies; it is absent rather than empty when the calibration does
+    not come off, and :func:`_calibrate_for_envelope` gives the three cases.
+
+    Notes
+    -----
+    **Nothing is drawn here any more.** Through a59 this rendered a matplotlib
+    figure and shipped SVG or PNG bytes, and it was the last thing in the api
+    importing matplotlib. The emitter ``charts.chart_envelope`` now publishes
+    the same picture as a document, so the api serves semantics and the browser
+    realizes them, which is the same split every other chart already keeps.
+
+    Two consequences worth stating. The reader can zoom and read values off the
+    band rather than squinting at a fixed raster. And the figure is no longer
+    two pictures maintained apart: the matplotlib compositor and the browser
+    now draw the same document, so they cannot disagree about what the envelope
+    is.
+
+    The document arrives with **two panels where the figure had three**. That is
+    upstream's decision and the right one: the five calibrated distortions used
+    to be split across the last two panels, which was an accident of the order
+    they were added rather than a reading anyone wants, since the question is
+    how the five compare and five curves on one band answer it.
+
+    Parameters
+    ----------
+    obj : Aggregate or Portfolio
+        The risk.
+    premium : float
+        The target premium the distortions are held to. Must sit above the
+        expected loss and at or below the asset cap, which the library enforces.
+    assets : float, optional
+        Asset cap; the class then bounds prices of ``min(X, a)``. Unbounded when
+        omitted.
+    n_resamples : int
+        Bracketing curves drawn inside the band.
+
+    Returns
+    -------
+    (bytes, str)
+        Canonical document JSON and its 12-hex content hash, the second so a
+        caller can set an ETag without parsing the body back.
+
+    Raises
+    ------
+    ValueError
+        Wrong kind of object, or a premium the library will not accept.
+    """
+    obj = _require_risk(obj)
+
+    kwargs = {"premium": float(premium)}
+    if assets is not None:
+        kwargs["a"] = float(assets)
+    bounds = Bounds(obj, **kwargs)
+
+    # Before the document, not after: the emitter reads the calibration off the
+    # priced object, so panel two exists only if this has already run.
+    _calibrate_for_envelope(obj, premium, assets)
+
+    doc = agg_charts.build_chart_doc(bounds, "envelope",
+                                     n_resamples=int(n_resamples))
+    return agg_charts.canonical_json(doc), doc.hash
+
+
+def run_allocation(obj: Any, *, premium: float, assets: float | None = None,
+                   ir: bool = False) -> dict:
+    """Per-unit natural-allocation ranges consistent with a total premium.
+
+    Portfolio only, and not by our choice: the calculation reads the
+    ``exeqa_*`` columns, which are what a portfolio's density frame carries and
+    a single aggregate has no analogue of.
+
+    Returns one row per unit with ``lower``, ``upper`` and ``width``. The width
+    is the reading: it is how much of each unit's price is decided by the choice
+    of distortion rather than by the total premium.
+
+    Parameters
+    ----------
+    obj : Portfolio
+    premium : float
+        Total premium for the book.
+    assets : float, optional
+        Asset cap.
+    ir : bool
+        Also return the table document for the static view.
+
+    Returns
+    -------
+    dict
+        Matches :class:`aggregate_api.models.BoundsResponse`.
+    """
+    from .serializers import frame_to_payload, reset_index_safe
+
+    if not isinstance(obj, Portfolio):
+        raise ValueError("allocation bounds apply to a Portfolio")
+    engine = AllocationBounds(obj, **({"a": float(assets)} if assets else {}))
+    frame = engine.bounds(float(premium))
+    return {
+        "premium": float(premium),
+        "table": frame_to_payload(reset_index_safe(frame)),
+        "ir": {"table": _document(frame)} if ir else None,
+    }
+
+
+def run_pricing_bounds(obj: Any, *, premium: float, targets: dict,
+                       assets: float | None = None, ir: bool = False) -> dict:
+    """Price ranges for other risks, given this one priced to ``premium``.
+
+    The cross-pricing question. Some distortion prices this object to the
+    target; every such distortion also prices anything else, and this reports
+    how wide that second price can be.
+
+    Parameters
+    ----------
+    obj : Aggregate or Portfolio
+        The reference risk carrying the pricing constraint.
+    premium : float
+        What the reference is priced to.
+    targets : dict of str to object
+        The risks whose ranges are wanted, by display name.
+    assets : float, optional
+        Asset cap, applied to both sides.
+    ir : bool
+        Also return the table document for the static view.
+
+    Returns
+    -------
+    dict
+        Matches :class:`aggregate_api.models.BoundsResponse`.
+    """
+    from .serializers import frame_to_payload, reset_index_safe
+
+    obj = _require_risk(obj)
+    if not targets:
+        raise ValueError("name a risk to price against this one")
+    engine = PricingBounds(obj, targets,
+                           **({"a": float(assets)} if assets else {}))
+    frame = engine.bounds(float(premium))
+    return {
+        "premium": float(premium),
+        "table": frame_to_payload(reset_index_safe(frame)),
+        "ir": {"table": _document(frame)} if ir else None,
+    }

aggregate_api/cache.py

@@ -0,0 +1,319 @@
+"""In-memory LRU cache for built ``Aggregate`` / ``Portfolio`` objects.
+
+The api's "Option X" cache design: a single, bounded, LRU dict
+keyed by *content hash* of the DecL program. Same DecL +
+``log2`` + ``bs`` → same id → same cached object. Building is
+idempotent for the cache lifetime of one server process.
+
+Why bother
+----------
+
+Building a moderately sized portfolio takes seconds; FFTs over
+2**18 points across many lines aren't free. The cache lets the
+SPA's "per-button-fetch UX" (info, summary, stats_df, plot,
+kappa, pricing) all run as O(1) lookups against the prebuilt
+object, with the heavy lift paid only once per (decl, log2, bs).
+
+Why not lru_cache
+-----------------
+
+``functools.lru_cache`` is per-function and doesn't expose the
+inspection / list / delete operations the api needs
+(``GET /v1/objects``, ``DELETE /v1/objects/{id}``). An
+``OrderedDict`` does, and the move-to-end / popitem(last=False)
+pair gives plain LRU semantics in ~10 lines.
+
+Thread-safety
+-------------
+
+A single ``threading.Lock`` wraps every mutation. The cache is
+small (≤50 entries by default), so coarse-grained locking is
+cheaper than any per-entry alternative.
+
+Why not weakref
+---------------
+
+We want explicit bounded retention, not "alive until nobody
+holds a reference" -- the SPA holds the id, not the object,
+so weakref would have no live referents and evict immediately.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import re
+import threading
+from collections import OrderedDict
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any
+
+
+# DecL comments run from ``#`` to end-of-line. We strip them before
+# hashing so trivially commented-out / annotated variants of the same
+# program produce the same object id.
+_COMMENT = re.compile(r"#[^\n]*")
+
+
+def canonicalize_decl(decl: str) -> str:
+    """Strip comments and trailing whitespace for content hashing.
+
+    Note that interior whitespace is *preserved* -- two programs
+    that differ only in indentation hash differently. A stronger
+    normalization (whitespace-insensitive via Lark tree round-trip)
+    is flagged in the plan as a future enhancement.
+
+    Parameters
+    ----------
+    decl : str
+        Raw DecL source.
+
+    Returns
+    -------
+    str
+        Canonicalized form suitable for hashing.
+    """
+    no_comments = _COMMENT.sub("", decl)
+    return no_comments.rstrip()
+
+
+def object_id(decl: str, log2: int, bs: float) -> str:
+    """Compute the cache id for a (decl, log2, bs) triple.
+
+    16-hex-char prefix of SHA-256. Collision probability is
+    negligible at our scale (target ≤ 10k builds per session).
+
+    Parameters
+    ----------
+    decl : str
+        Already-canonicalized DecL.
+    log2, bs : int, float
+        Build knobs that affect the resulting object.
+    """
+    # ``bs!r`` because bs is a float; repr() pins exact bit pattern
+    # so e.g. 0.1 and 0.1000000000001 hash differently (they would
+    # build differently too).
+    payload = f"{decl}|{log2}|{bs!r}".encode("utf-8")
+    return hashlib.sha256(payload).hexdigest()[:16]
+
+
+def qualified_object_id(session_id: str, decl: str, log2: int, bs: float) -> str:
+    """The cache id for a program private to one session.
+
+    Same hash, one more field. A program that resolved a name its own session
+    declared does not mean the same thing to anyone else, so it cannot share a
+    slot with the identical text typed by a different session: Alice's
+    ``port Book agg.Line`` and Bob's are the same eight words over different
+    inners, and one of them would be served the other's answer.
+
+    Parameters
+    ----------
+    session_id : str
+        The caller's session, from ``aggregate_api.sessions``.
+    decl : str
+        Already-canonicalized DecL.
+    log2, bs : int, float
+        Build knobs, as in :func:`object_id`.
+
+    Returns
+    -------
+    str
+
+    Notes
+    -----
+    Which of the two builders a request uses is decided by what its parse
+    resolved, not by anything in the text: see ``_cache_key`` in
+    ``routes/objects.py``. A self-contained program, or one leaning only on
+    library entries, keeps the shared key and is built once for everybody, which
+    is the case a room on one hero example is made of.
+    """
+    payload = f"{session_id}|{decl}|{log2}|{bs!r}".encode("utf-8")
+    return hashlib.sha256(payload).hexdigest()[:16]
+
+
+@dataclass
+class CacheEntry:
+    """One slot in the LRU.
+
+    ``obj`` is the live ``Aggregate`` or ``Portfolio`` -- the SPA
+    never reaches it directly, but the api's per-button endpoints
+    pull data off it on demand.
+
+    The other fields are metadata returned by
+    ``GET /v1/objects/{id}`` and used by the build endpoint to
+    fill out the response without consulting the underlying object.
+
+    Thread-safety of ``obj``
+    ------------------------
+
+    ``lock`` serializes *reads of the built object*, which sounds unnecessary
+    and is not. An ``Aggregate`` and a ``Portfolio`` materialize several of
+    their frames lazily and cache them on the instance, so a "read" is a write
+    the first time. Two requests that touch the same object concurrently can
+    therefore see a half-built frame.
+
+    That is not hypothetical. Fetching ``unit_density_df`` and ``tail_df`` for
+    one Portfolio at the same moment (which the Overview exhibit does, in a
+    single ``Promise.all``) raised
+    ``KeyError: "['F', 'S'] not in index"`` from inside
+    ``Portfolio.unit_density_df`` roughly half the time on a cold object, and
+    never once the frames were warm. FastAPI runs sync handlers in a thread
+    pool, so those two requests really are two threads on one object.
+
+    The lock is per entry rather than global so unrelated objects still serve
+    in parallel, and contention is confined to the first access of each frame:
+    afterwards every read is a cache hit and the critical section is trivial.
+    """
+
+    obj: Any
+    decl: str
+    log2: int
+    bs: float
+    kind: str
+    name: str
+    created_at: datetime
+    # What the library said while building this object, at WARNING and above.
+    # Stored here rather than returned once and forgotten, because a warning
+    # ("this splice is coarse", "the grid clips the tail") describes the
+    # OBJECT, not the request that happened to build it. Kept on the entry, a
+    # cache hit reports the same warnings as the miss that made it, so
+    # rebuilding a program does not silently lose the reason to worry about it.
+    notes: list[str] = field(default_factory=list)
+    # Guards reads *of the object*, not of this dataclass. See below.
+    lock: threading.Lock = field(default_factory=threading.Lock, repr=False,
+                                 compare=False)
+
+
+class ObjectCache:
+    """Bounded LRU keyed by content hash.
+
+    Methods are coarsely thread-safe via a single lock. Reads
+    move the entry to MRU; writes evict the LRU when full. The
+    ``__contains__`` check does *not* move-to-MRU (peek, not
+    promote) -- the standard idiom for "is this in cache" before
+    a separate get/put decision.
+    """
+
+    def __init__(self, max_entries: int = 50) -> None:
+        self._max = max_entries
+        self._store: OrderedDict[str, CacheEntry] = OrderedDict()
+        self._lock = threading.Lock()
+        # Instrumentation for GET /v1/status. Incremented under the same lock
+        # that guards the store, so a reading of them is consistent with the
+        # entry list read in the same call and no second lock is introduced.
+        # They count this instance, and ``_get_cache`` replaces the instance
+        # when ``cache_max`` changes, so the page reports them against the
+        # process start time and that is honest for every deployment which does
+        # not edit its config while running.
+        self.hits = 0
+        self.misses = 0
+        self.puts = 0
+        self.evictions = 0
+        self.deletes = 0
+
+    def get(self, oid: str) -> CacheEntry | None:
+        """Return the cached entry for ``oid`` (moving it to MRU) or None."""
+        with self._lock:
+            entry = self._store.get(oid)
+            if entry is None:
+                self.misses += 1
+                return None
+            self.hits += 1
+            # ``move_to_end`` is the OrderedDict primitive that makes
+            # LRU work; entries closest to the front are evicted first.
+            self._store.move_to_end(oid)
+            return entry
+
+    def put(self, oid: str, entry: CacheEntry) -> None:
+        """Insert or refresh ``entry`` under ``oid``, evicting LRU if full."""
+        with self._lock:
+            self.puts += 1
+            if oid in self._store:
+                self._store.move_to_end(oid)
+                self._store[oid] = entry
+                return
+            self._store[oid] = entry
+            while len(self._store) > self._max:
+                # ``last=False`` pops the *least* recently used (front).
+                self._store.popitem(last=False)
+                self.evictions += 1
+
+    def delete(self, oid: str) -> bool:
+        """Remove ``oid`` from the cache; return True if it was present."""
+        with self._lock:
+            dropped = self._store.pop(oid, None) is not None
+            if dropped:
+                self.deletes += 1
+            return dropped
+
+    def list(self) -> list[CacheEntry]:
+        """Return a snapshot of cache contents, MRU last.
+
+        Returned list is a copy of the internal values -- safe to
+        iterate without holding the lock.
+        """
+        with self._lock:
+            return list(self._store.values())
+
+    def items(self) -> list[tuple[str, CacheEntry]]:
+        """Snapshot of ``(id, entry)`` pairs, **LRU first**.
+
+        Notes
+        -----
+        LRU first because the caller that wants ids as well as entries is
+        ``GET /v1/status``, whose cache table answers "what will I lose next",
+        and the front of the ``OrderedDict`` is exactly that. :meth:`list`
+        keeps its MRU-last ordering, which is what its callers already read.
+
+        Exists so callers stop reaching into ``_store`` directly, which two of
+        them in ``routes/objects.py`` do, unlocked. The pairs are a copy, so
+        iterating them is safe without the lock; the entries themselves are the
+        live objects, as :meth:`list` also returns.
+        """
+        with self._lock:
+            return list(self._store.items())
+
+    def clear(self) -> None:
+        """Drop every entry. Used by tests."""
+        with self._lock:
+            self._store.clear()
+
+    def __contains__(self, oid: str) -> bool:
+        with self._lock:
+            return oid in self._store
+
+    def __len__(self) -> int:
+        with self._lock:
+            return len(self._store)
+
+    def stats(self) -> dict:
+        """Counters and occupancy, read in one critical section.
+
+        Returns
+        -------
+        dict
+            ``entries``, ``max``, ``hits``, ``misses``, ``hit_rate``, ``puts``,
+            ``evictions`` and ``deletes``. ``hit_rate`` is ``None`` rather than
+            zero before the first lookup, because "no requests yet" and "every
+            request missed" are different facts and a status page that renders
+            them the same way is misreporting a cold start as a failure.
+
+        Notes
+        -----
+        One call rather than eight attribute reads so the numbers a caller
+        prints together were true together. An eviction landing between two
+        reads would otherwise show a hit count that does not reconcile with the
+        entry list.
+        """
+        with self._lock:
+            looks = self.hits + self.misses
+            return {
+                "entries": len(self._store),
+                "max": self._max,
+                "hits": self.hits,
+                "misses": self.misses,
+                "hit_rate": round(self.hits / looks, 4) if looks else None,
+                "puts": self.puts,
+                "evictions": self.evictions,
+                "deletes": self.deletes,
+            }