uhttp-workers 1.5.0__tar.gz → 1.6.0__tar.gz

{uhttp_workers-1.5.0/uhttp_workers.egg-info → uhttp_workers-1.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: uhttp-workers
-Version: 1.5.0
+Version: 1.6.0
 Summary: Multi-process worker dispatcher built on uhttp-server
 Author-email: Pavel Revak <pavelrevak@gmail.com>
 License-Expression: MIT
@@ -565,6 +565,7 @@ Reason is one of:
 | `PENDING_DISCONNECTED` | Client disconnected mid-stream; worker was notified via control queue (race possible). |
 | `PENDING_STREAM_CLOSED` | Worker ended the SSE stream cleanly. |
 | `PENDING_SHUTDOWN` | Dispatcher is shutting down; client got 503. |
+| `PENDING_WORKER_DIED` | Worker process died/was killed while owning the request; client got 500. `on_worker_died()` runs first. |
 
 The hook is invoked after the client-facing action (respond / disconnect / control queue put)
 so dispatcher state is finalized when it runs. Exceptions raised by the hook are logged at
@@ -575,6 +576,51 @@ Override `on_pending_removed()` if you need exactly-once cleanup. Overriding bot
 but discouraged — for the `PENDING_COMPLETED` reason, `on_response()` is called immediately
 before `on_pending_removed()`.
 
+## Worker Death Hook
+
+Workers die — segfault in a C extension, OOM-kill, or the dispatcher kills them after
+`stuck_timeout`. Override `on_worker_died()` to capture which requests they had in-flight
+(useful for forensics when a malformed payload reproduces a crash):
+
+```python
+class MyDispatcher(_workers.Dispatcher):
+    def on_worker_died(
+            self, pool, worker_id, reason, exitcode, victims):
+        # `victims` is a list of (request_id, _PendingRequest) for all
+        # requests this worker had claimed but not completed.
+        # `exitcode` is None for stuck workers, otherwise the process exit
+        # code (negative = signal: -9 OOM, -11 SIGSEGV).
+        for rid, pending in victims:
+            c = pending.client
+            self._crash_queue.append({
+                'reason': reason,
+                'exitcode': exitcode,
+                'method': c.method,
+                'path': c.path,
+                'address': c.address,
+                'body': c.body,  # raw bytes — replay this to reproduce
+            })
+        # Default impl responds 500 to victims and fires
+        # on_pending_removed(PENDING_WORKER_DIED). Call it after capture.
+        super().on_worker_died(
+            pool, worker_id, reason, exitcode, victims)
+```
+
+What's a victim: any request the worker had claimed via `MSG_HEARTBEAT`
+(`pending.worker_id == worker_id`). Requests still in the queue (`worker_id is None`)
+are **not** victims — other workers in the pool will pick them up after restart.
+
+Default behavior (if you don't override) is to log the death + each victim, respond
+500 (or close the stream for SSE/NDJSON), and fire `on_pending_removed` for each.
+Override only if you want to persist payloads or customize the response status/body.
+
+**500 vs 503:** a victim of a crashed worker gets **500** (processing started, then
+the server failed). A new request arriving while the pool has zero alive workers
+gets **503 + `Retry-After: 1`** (rejected before processing — try again shortly).
+A request to a pool that has exceeded `max_restarts` in `restart_window` gets **503**
+permanently (`pool.is_degraded`). `pool.alive_count` is exposed for monitoring and
+also appears in `pool.status()`.
+
 ## Dispatcher Idle Hook
 
 Override `on_idle()` on the dispatcher for periodic background tasks — called on each `select()` timeout (every `SELECT_TIMEOUT` seconds, default 1s):

{uhttp_workers-1.5.0 → uhttp_workers-1.6.0}/README.md

@@ -551,6 +551,7 @@ Reason is one of:
 | `PENDING_DISCONNECTED` | Client disconnected mid-stream; worker was notified via control queue (race possible). |
 | `PENDING_STREAM_CLOSED` | Worker ended the SSE stream cleanly. |
 | `PENDING_SHUTDOWN` | Dispatcher is shutting down; client got 503. |
+| `PENDING_WORKER_DIED` | Worker process died/was killed while owning the request; client got 500. `on_worker_died()` runs first. |
 
 The hook is invoked after the client-facing action (respond / disconnect / control queue put)
 so dispatcher state is finalized when it runs. Exceptions raised by the hook are logged at
@@ -561,6 +562,51 @@ Override `on_pending_removed()` if you need exactly-once cleanup. Overriding bot
 but discouraged — for the `PENDING_COMPLETED` reason, `on_response()` is called immediately
 before `on_pending_removed()`.
 
+## Worker Death Hook
+
+Workers die — segfault in a C extension, OOM-kill, or the dispatcher kills them after
+`stuck_timeout`. Override `on_worker_died()` to capture which requests they had in-flight
+(useful for forensics when a malformed payload reproduces a crash):
+
+```python
+class MyDispatcher(_workers.Dispatcher):
+    def on_worker_died(
+            self, pool, worker_id, reason, exitcode, victims):
+        # `victims` is a list of (request_id, _PendingRequest) for all
+        # requests this worker had claimed but not completed.
+        # `exitcode` is None for stuck workers, otherwise the process exit
+        # code (negative = signal: -9 OOM, -11 SIGSEGV).
+        for rid, pending in victims:
+            c = pending.client
+            self._crash_queue.append({
+                'reason': reason,
+                'exitcode': exitcode,
+                'method': c.method,
+                'path': c.path,
+                'address': c.address,
+                'body': c.body,  # raw bytes — replay this to reproduce
+            })
+        # Default impl responds 500 to victims and fires
+        # on_pending_removed(PENDING_WORKER_DIED). Call it after capture.
+        super().on_worker_died(
+            pool, worker_id, reason, exitcode, victims)
+```
+
+What's a victim: any request the worker had claimed via `MSG_HEARTBEAT`
+(`pending.worker_id == worker_id`). Requests still in the queue (`worker_id is None`)
+are **not** victims — other workers in the pool will pick them up after restart.
+
+Default behavior (if you don't override) is to log the death + each victim, respond
+500 (or close the stream for SSE/NDJSON), and fire `on_pending_removed` for each.
+Override only if you want to persist payloads or customize the response status/body.
+
+**500 vs 503:** a victim of a crashed worker gets **500** (processing started, then
+the server failed). A new request arriving while the pool has zero alive workers
+gets **503 + `Retry-After: 1`** (rejected before processing — try again shortly).
+A request to a pool that has exceeded `max_restarts` in `restart_window` gets **503**
+permanently (`pool.is_degraded`). `pool.alive_count` is exposed for monitoring and
+also appears in `pool.status()`.
+
 ## Dispatcher Idle Hook
 
 Override `on_idle()` on the dispatcher for periodic background tasks — called on each `select()` timeout (every `SELECT_TIMEOUT` seconds, default 1s):

{uhttp_workers-1.5.0 → uhttp_workers-1.6.0}/tests/test_dispatcher.py

@@ -13,7 +13,7 @@ from uhttp.workers import (
     MSG_SSE_OPEN, MSG_SSE_EVENT, MSG_SSE_CLOSE, MSG_NDJSON,
     CTL_DISCONNECT,
     PENDING_COMPLETED, PENDING_TIMEOUT, PENDING_DISCONNECTED,
-    PENDING_STREAM_CLOSED, PENDING_SHUTDOWN,
+    PENDING_STREAM_CLOSED, PENDING_SHUTDOWN, PENDING_WORKER_DIED,
     LOG_ERROR,
     _PendingRequest,
 )
@@ -29,13 +29,16 @@ class MockClient:
     """Mock HttpConnection for testing dispatcher logic."""
 
     def __init__(self, method='GET', path='/', query=None, data=None,
-            headers=None, content_type=None):
+            headers=None, content_type=None, body=None,
+            address='127.0.0.1'):
         self.method = method
         self.path = path
         self.query = query
         self.data = data
         self.headers = headers or {}
         self.content_type = content_type
+        self.body = body
+        self.address = address
         self.responded = False
         self.response_data = None
         self.response_status = None
@@ -188,6 +191,8 @@ class TestDispatcherDoCheck(unittest.TestCase):
 
     def test_pass_check(self):
         pool = WorkerPool(DummyWorker, routes=['/api/**'])
+        # fake a live worker so alive_count > 0 without starting processes
+        pool.workers = [type('W', (), {'is_alive': lambda self: True})()]
         d = Dispatcher.__new__(Dispatcher)
         d._sync_routes = []
         d._static_routes = {}
@@ -317,6 +322,48 @@ class TestDispatcherPoolRouting(unittest.TestCase):
         self.assertTrue(client.responded)
         self.assertEqual(client.response_status, 503)
 
+    def test_dispatch_no_alive_workers_returns_503(self):
+        """Transient: pool has workers but none currently alive."""
+        # pool has dead workers (not degraded yet)
+        self.pool_default.workers = [
+            type('W', (), {'is_alive': lambda self: False})()]
+        d = Dispatcher.__new__(Dispatcher)
+        d._sync_routes = []
+        d._static_routes = {}
+        d._pools = [self.pool_default]
+        d._pending = {}
+        d._max_pending = 1000
+        d._next_request_id = 0
+
+        client = MockClient('GET', '/test')
+        d._dispatch_to_pool(client)
+        self.assertTrue(client.responded)
+        self.assertEqual(client.response_status, 503)
+        self.assertEqual(
+            client.response_data['error'], 'No workers available')
+        self.assertEqual(
+            client.response_headers.get('Retry-After'), '1')
+        # request was NOT enqueued
+        self.assertEqual(d._pending, {})
+
+    def test_dispatch_empty_workers_returns_503(self):
+        """Pool was never started (empty workers list)."""
+        # self.pool_default.workers is [] from __init__
+        d = Dispatcher.__new__(Dispatcher)
+        d._sync_routes = []
+        d._static_routes = {}
+        d._pools = [self.pool_default]
+        d._pending = {}
+        d._max_pending = 1000
+        d._next_request_id = 0
+
+        client = MockClient('GET', '/test')
+        d._dispatch_to_pool(client)
+        self.assertTrue(client.responded)
+        self.assertEqual(client.response_status, 503)
+        self.assertEqual(
+            client.response_data['error'], 'No workers available')
+
 
 class TestDispatcherProcessResponse(unittest.TestCase):
 
@@ -769,5 +816,207 @@ class TestDispatcherPendingRemoved(unittest.TestCase):
         self.assertTrue(client.responded)
 
 
+class TestDispatcherWorkerDied(unittest.TestCase):
+    """Tests for on_worker_died hook and victim handling."""
+
+    def _make_dispatcher(self, dispatcher_cls=Dispatcher):
+        # queue_warning=0 disables the queue-size check (which would
+        # otherwise touch pool.pending_count → request_queue.qsize()).
+        pool = WorkerPool(
+            DummyWorker, routes=['/api/**'], queue_warning=0)
+        # Mock check_workers so we control what it returns without
+        # actually starting processes.
+        pool._fake_restarted = []
+        pool.check_workers = lambda: pool._fake_restarted
+        d = dispatcher_cls.__new__(dispatcher_cls)
+        d._sync_routes = []
+        d._static_routes = {}
+        d._pools = [pool]
+        d._pending = {}
+        d._max_pending = 1000
+        d._next_request_id = 0
+        d._response_queue = mp.Queue()
+        d._log_is_tty = False
+        d.log_calls = []
+        d.on_log = lambda name, level, msg: d.log_calls.append(
+            (name, level, msg))
+        d.recorded_removed = []
+        return d, pool
+
+    def test_single_victim_gets_500(self):
+
+        class RecordingDispatcher(Dispatcher):
+            def on_pending_removed(self, request_id, pending, reason):
+                self.recorded_removed.append((request_id, reason))
+
+        d, pool = self._make_dispatcher(RecordingDispatcher)
+        client = MockClient(
+            'POST', '/api/scan', body=b'\x00\x01bad', address='10.0.0.7')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        d._pending[42] = pending
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        # client got 500
+        self.assertTrue(client.responded)
+        self.assertEqual(client.response_status, 500)
+        self.assertEqual(client.response_data['error'], 'Worker crashed')
+        self.assertIn('exit=-11', client.response_data['reason'])
+        # removed from pending + hook fired
+        self.assertNotIn(42, d._pending)
+        self.assertEqual(
+            d.recorded_removed, [(42, PENDING_WORKER_DIED)])
+
+    def test_multiple_victims_all_handled(self):
+        d, pool = self._make_dispatcher()
+        c1 = MockClient('GET', '/api/a', address='1.1.1.1')
+        c2 = MockClient('GET', '/api/b', address='2.2.2.2')
+        c3 = MockClient('GET', '/api/c', address='3.3.3.3')
+        for rid, c in [(1, c1), (2, c2), (3, c3)]:
+            p = _PendingRequest(c, pool)
+            p.worker_id = 0
+            d._pending[rid] = p
+        pool._fake_restarted = [(0, 'stuck', None)]
+        d._check_all_workers()
+        for c in (c1, c2, c3):
+            self.assertTrue(c.responded)
+            self.assertEqual(c.response_status, 500)
+        self.assertEqual(d._pending, {})
+
+    def test_streaming_victim_gets_stream_end(self):
+        d, pool = self._make_dispatcher()
+        client = MockClient('GET', '/api/events')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        pending.streaming = True
+        d._pending[1] = pending
+        pool._fake_restarted = [(0, 'died exit=-9', -9)]
+        d._check_all_workers()
+        # stream ended, NOT respond()
+        self.assertTrue(getattr(client, 'stream_ended', False))
+        self.assertFalse(client.responded)
+        self.assertNotIn(1, d._pending)
+
+    def test_queued_request_not_a_victim(self):
+        """Request with worker_id=None is still in queue — not a victim."""
+        d, pool = self._make_dispatcher()
+        # request belonging to dying worker
+        in_flight = MockClient('GET', '/api/active')
+        p1 = _PendingRequest(in_flight, pool)
+        p1.worker_id = 0
+        d._pending[1] = p1
+        # request still in queue, no worker claimed it
+        queued = MockClient('GET', '/api/queued')
+        p2 = _PendingRequest(queued, pool)
+        # p2.worker_id stays None
+        d._pending[2] = p2
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        # in-flight responded
+        self.assertTrue(in_flight.responded)
+        self.assertNotIn(1, d._pending)
+        # queued untouched
+        self.assertFalse(queued.responded)
+        self.assertIn(2, d._pending)
+
+    def test_other_worker_not_affected(self):
+        """Only victims of THIS worker are handled; other workers stay."""
+        d, pool = self._make_dispatcher()
+        c1 = MockClient('GET', '/api/a')
+        c2 = MockClient('GET', '/api/b')
+        p1 = _PendingRequest(c1, pool)
+        p1.worker_id = 0
+        p2 = _PendingRequest(c2, pool)
+        p2.worker_id = 1
+        d._pending[1] = p1
+        d._pending[2] = p2
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        self.assertTrue(c1.responded)
+        self.assertNotIn(1, d._pending)
+        self.assertFalse(c2.responded)
+        self.assertIn(2, d._pending)
+
+    def test_late_response_after_victim_cleanup_dropped(self):
+        """MSG_RESPONSE from dead worker arriving after victim removal is dropped."""
+        d, pool = self._make_dispatcher()
+        client = MockClient('GET', '/api/test')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        d._pending[1] = pending
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        # request already gone; client already got 500
+        self.assertEqual(client.response_status, 500)
+        # late response from before-death — must not break or double-respond
+        client.response_status = None
+        late = Response(request_id=1, data={'ok': True}, status=200)
+        d._process_response((MSG_RESPONSE, 1, late))
+        # silently dropped
+        self.assertIsNone(client.response_status)
+
+    def test_no_victims_just_logs(self):
+        """Worker died while idle — restarted but no pending requests."""
+        d, pool = self._make_dispatcher()
+        pool._fake_restarted = [(0, 'died exit=0', 0)]
+        d._check_all_workers()
+        # no crash, no pending changes
+        self.assertEqual(d._pending, {})
+        # should have logged
+        error_logs = [
+            msg for _, level, msg in d.log_calls if level == LOG_ERROR]
+        self.assertEqual(len(error_logs), 1)
+        self.assertIn('victims=0', error_logs[0])
+
+    def test_override_can_persist_payload(self):
+        """User override can capture victim payload before super() responds."""
+        captured = []
+
+        class ForensicDispatcher(Dispatcher):
+            def on_worker_died(
+                    self, pool, worker_id, reason, exitcode, victims):
+                for rid, pending in victims:
+                    captured.append({
+                        'rid': rid,
+                        'address': pending.client.address,
+                        'body': pending.client.body,
+                        'reason': reason,
+                        'exitcode': exitcode})
+                super().on_worker_died(
+                    pool, worker_id, reason, exitcode, victims)
+
+        d, pool = self._make_dispatcher(ForensicDispatcher)
+        client = MockClient(
+            'POST', '/api/process',
+            body=b'\xff\xfecorrupted', address='9.9.9.9')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        d._pending[7] = pending
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        self.assertEqual(len(captured), 1)
+        self.assertEqual(captured[0]['address'], '9.9.9.9')
+        self.assertEqual(captured[0]['body'], b'\xff\xfecorrupted')
+        self.assertEqual(captured[0]['exitcode'], -11)
+        # super() still ran
+        self.assertEqual(client.response_status, 500)
+
+    def test_hook_exception_does_not_crash_dispatcher(self):
+
+        class BrokenDispatcher(Dispatcher):
+            def on_worker_died(self, *args, **kwargs):
+                raise RuntimeError('boom')
+
+        d, pool = self._make_dispatcher(BrokenDispatcher)
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        # must not propagate
+        d._check_all_workers()
+        error_logs = [
+            msg for _, level, msg in d.log_calls if level == LOG_ERROR]
+        # one error log about the hook failure
+        self.assertTrue(any('on_worker_died' in m for m in error_logs))
+        self.assertTrue(any('boom' in m for m in error_logs))
+
+
 if __name__ == '__main__':
     unittest.main()

{uhttp_workers-1.5.0 → uhttp_workers-1.6.0}/tests/test_worker_pool.py

@@ -120,6 +120,27 @@ class TestWorkerPoolStatus(unittest.TestCase):
         pool = WorkerPool(DummyWorker)
         self.assertFalse(pool.is_degraded)
 
+    def test_alive_count_empty(self):
+        pool = WorkerPool(DummyWorker, num_workers=2)
+        # not started yet
+        self.assertEqual(pool.alive_count, 0)
+
+    def test_alive_count_running(self):
+        pool = WorkerPool(DummyWorker, num_workers=2)
+        response_queue = mp.Queue()
+        pool.start(response_queue)
+        time.sleep(0.2)
+        self.assertEqual(pool.alive_count, 2)
+        pool.shutdown(timeout=3)
+
+    def test_alive_count_in_status(self):
+        pool = WorkerPool(DummyWorker, num_workers=2)
+        response_queue = mp.Queue()
+        pool.start(response_queue)
+        time.sleep(0.2)
+        self.assertEqual(pool.status()['alive_count'], 2)
+        pool.shutdown(timeout=3)
+
 
 class TestWorkerPoolCheckWorkers(unittest.TestCase):
 

{uhttp_workers-1.5.0 → uhttp_workers-1.6.0}/uhttp/workers.py

@@ -36,6 +36,7 @@ PENDING_TIMEOUT = 'TIMEOUT'
 PENDING_DISCONNECTED = 'DISCONNECTED'
 PENDING_STREAM_CLOSED = 'STREAM_CLOSED'
 PENDING_SHUTDOWN = 'SHUTDOWN'
+PENDING_WORKER_DIED = 'WORKER_DIED'
 
 # Sentinel for deferred response
 DEFERRED = object()
@@ -879,7 +880,10 @@ class WorkerPool:
         """Check worker health, restart dead or stuck workers.
 
         Returns:
-            List of (worker_id, reason) tuples for restarted workers.
+            List of (worker_id, reason, exitcode) tuples for restarted
+            workers. exitcode is None for stuck workers (dispatcher killed
+            them), otherwise the process exit code (negative = signal:
+            -9 OOM, -11 SIGSEGV, -15 SIGTERM, etc.).
         """
         restarted = []
         now = _time.time()
@@ -889,8 +893,10 @@ class WorkerPool:
             if now - t < self.restart_window]
         for i, worker in enumerate(self.workers):
             reason = None
+            exitcode = None
             if not worker.is_alive():
-                reason = f"died exit={worker.exitcode}"
+                exitcode = worker.exitcode
+                reason = f"died exit={exitcode}"
             elif now - self._last_seen.get(i, 0) > self.stuck_timeout:
                 reason = "stuck"
                 worker.kill()
@@ -904,7 +910,7 @@ class WorkerPool:
                 if len(self._restart_times) >= self.max_restarts:
                     self._degraded = True
                 self._start_worker(i)
-                restarted.append((i, reason))
+                restarted.append((i, reason, exitcode))
         return restarted
 
     def matches(self, path):
@@ -959,6 +965,11 @@ class WorkerPool:
     def is_degraded(self):
         return self._degraded
 
+    @property
+    def alive_count(self):
+        """Number of worker processes currently alive."""
+        return sum(1 for w in self.workers if w.is_alive())
+
     @property
     def pending_count(self):
         try:
@@ -976,6 +987,7 @@ class WorkerPool:
         return {
             'name': self.name,
             'degraded': self._degraded,
+            'alive_count': self.alive_count,
             'queue_size': self.pending_count,
             'workers': [
                 {
@@ -1127,6 +1139,9 @@ class Dispatcher:
                                     notified via control queue (race possible).
             PENDING_STREAM_CLOSED - worker ended the SSE stream cleanly.
             PENDING_SHUTDOWN      - dispatcher is shutting down; client got 503.
+            PENDING_WORKER_DIED   - worker process died/was killed while owning
+                                    this request; client got 500. on_worker_died()
+                                    runs first.
 
         Args:
             request_id: The request id being removed.
@@ -1201,6 +1216,11 @@ class Dispatcher:
             client.respond(
                 {'error': 'Service unavailable'}, status=503)
             return
+        if pool.alive_count == 0:
+            client.respond(
+                {'error': 'No workers available'}, status=503,
+                headers={'Retry-After': '1'})
+            return
         if len(self._pending) >= self._max_pending:
             client.respond(
                 {'error': 'Too many requests'}, status=503)
@@ -1353,8 +1373,18 @@ class Dispatcher:
         """Check health of all worker pools and queue sizes."""
         for pool in self._pools:
             restarted = pool.check_workers()
-            for worker_id, reason in restarted:
-                self._on_worker_restarted(pool, worker_id, reason)
+            for worker_id, reason, exitcode in restarted:
+                victims = [
+                    (rid, p) for rid, p in self._pending.items()
+                    if p.pool is pool and p.worker_id == worker_id]
+                try:
+                    self.on_worker_died(
+                        pool, worker_id, reason, exitcode, victims)
+                except Exception:
+                    self.on_log(
+                        pool.name, LOG_ERROR,
+                        f"on_worker_died() failed:\n"
+                        f"{_traceback.format_exc()}")
             if pool.queue_warning:
                 qsize = pool.pending_count
                 if qsize >= pool.queue_warning:
@@ -1390,14 +1420,57 @@ class Dispatcher:
             print(f"{prefix}{level_name:8s} {name:20s} {message}",
                 file=_sys.stderr)
 
-    def _on_worker_restarted(self, pool, worker_id, reason):
-        """Called when a worker is restarted.
+    def on_worker_died(self, pool, worker_id, reason, exitcode, victims):
+        """Called when a worker process died or was killed by the dispatcher.
+
+        Default behavior:
+          1. Log restart reason + each victim (request id, client address,
+             method, path, body size).
+          2. Respond 500 to every victim's client (or response_stream_end()
+             for streams), remove them from _pending, and fire
+             on_pending_removed(PENDING_WORKER_DIED) for each.
 
-        Default logs the event. Override to customize.
+        Override to capture victim payloads (e.g., persist to disk for
+        post-mortem) BEFORE calling super(). pending.client gives access
+        to method, path, headers, body, address.
+
+        Args:
+            pool: WorkerPool the worker belonged to.
+            worker_id: Index of the restarted worker.
+            reason: 'stuck' or 'died exit=N' (string from check_workers).
+            exitcode: Process exit code (int) or None for stuck workers.
+                Negative values are signals: -9 OOM, -11 SIGSEGV, etc.
+            victims: List of (request_id, _PendingRequest) tuples — requests
+                this worker had claimed (via MSG_HEARTBEAT) but never
+                completed. May be empty if worker died while idle.
         """
         self.on_log(
             f'{pool.name}[{worker_id}]', LOG_ERROR,
-            f"worker restarted: {reason}")
+            f"worker restarted: {reason}, "
+            f"victims={len(victims)}")
+        for request_id, pending in victims:
+            c = pending.client
+            body_len = len(c.body) if c.body is not None else 0
+            self.on_log(
+                pool.name, LOG_ERROR,
+                f"  victim rid={request_id} from={c.address} "
+                f"{c.method} {c.path} body={body_len}B")
+            del self._pending[request_id]
+            if pending.streaming:
+                try:
+                    pending.client.response_stream_end()
+                except Exception:
+                    pass
+            else:
+                try:
+                    pending.client.respond(
+                        {'error': 'Worker crashed',
+                         'reason': reason},
+                        status=500)
+                except Exception:
+                    pass
+            self._notify_pending_removed(
+                request_id, pending, PENDING_WORKER_DIED)
 
     def _sigterm(self, _signo, _stack_frame):
         self._running = False

{uhttp_workers-1.5.0 → uhttp_workers-1.6.0/uhttp_workers.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: uhttp-workers
-Version: 1.5.0
+Version: 1.6.0
 Summary: Multi-process worker dispatcher built on uhttp-server
 Author-email: Pavel Revak <pavelrevak@gmail.com>
 License-Expression: MIT
@@ -565,6 +565,7 @@ Reason is one of:
 | `PENDING_DISCONNECTED` | Client disconnected mid-stream; worker was notified via control queue (race possible). |
 | `PENDING_STREAM_CLOSED` | Worker ended the SSE stream cleanly. |
 | `PENDING_SHUTDOWN` | Dispatcher is shutting down; client got 503. |
+| `PENDING_WORKER_DIED` | Worker process died/was killed while owning the request; client got 500. `on_worker_died()` runs first. |
 
 The hook is invoked after the client-facing action (respond / disconnect / control queue put)
 so dispatcher state is finalized when it runs. Exceptions raised by the hook are logged at
@@ -575,6 +576,51 @@ Override `on_pending_removed()` if you need exactly-once cleanup. Overriding bot
 but discouraged — for the `PENDING_COMPLETED` reason, `on_response()` is called immediately
 before `on_pending_removed()`.
 
+## Worker Death Hook
+
+Workers die — segfault in a C extension, OOM-kill, or the dispatcher kills them after
+`stuck_timeout`. Override `on_worker_died()` to capture which requests they had in-flight
+(useful for forensics when a malformed payload reproduces a crash):
+
+```python
+class MyDispatcher(_workers.Dispatcher):
+    def on_worker_died(
+            self, pool, worker_id, reason, exitcode, victims):
+        # `victims` is a list of (request_id, _PendingRequest) for all
+        # requests this worker had claimed but not completed.
+        # `exitcode` is None for stuck workers, otherwise the process exit
+        # code (negative = signal: -9 OOM, -11 SIGSEGV).
+        for rid, pending in victims:
+            c = pending.client
+            self._crash_queue.append({
+                'reason': reason,
+                'exitcode': exitcode,
+                'method': c.method,
+                'path': c.path,
+                'address': c.address,
+                'body': c.body,  # raw bytes — replay this to reproduce
+            })
+        # Default impl responds 500 to victims and fires
+        # on_pending_removed(PENDING_WORKER_DIED). Call it after capture.
+        super().on_worker_died(
+            pool, worker_id, reason, exitcode, victims)
+```
+
+What's a victim: any request the worker had claimed via `MSG_HEARTBEAT`
+(`pending.worker_id == worker_id`). Requests still in the queue (`worker_id is None`)
+are **not** victims — other workers in the pool will pick them up after restart.
+
+Default behavior (if you don't override) is to log the death + each victim, respond
+500 (or close the stream for SSE/NDJSON), and fire `on_pending_removed` for each.
+Override only if you want to persist payloads or customize the response status/body.
+
+**500 vs 503:** a victim of a crashed worker gets **500** (processing started, then
+the server failed). A new request arriving while the pool has zero alive workers
+gets **503 + `Retry-After: 1`** (rejected before processing — try again shortly).
+A request to a pool that has exceeded `max_restarts` in `restart_window` gets **503**
+permanently (`pool.is_degraded`). `pool.alive_count` is exposed for monitoring and
+also appears in `pool.status()`.
+
 ## Dispatcher Idle Hook
 
 Override `on_idle()` on the dispatcher for periodic background tasks — called on each `select()` timeout (every `SELECT_TIMEOUT` seconds, default 1s):