uhttp-workers 1.4.0__tar.gz → 1.6.0__tar.gz

{uhttp_workers-1.4.0/uhttp_workers.egg-info → uhttp_workers-1.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: uhttp-workers
-Version: 1.4.0
+Version: 1.6.0
 Summary: Multi-process worker dispatcher built on uhttp-server
 Author-email: Pavel Revak <pavelrevak@gmail.com>
 License-Expression: MIT
@@ -10,7 +10,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Operating System :: POSIX
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
-Requires-Dist: uhttp-server
+Requires-Dist: uhttp-server>=2.5.2
 
 # uhttp-workers
 
@@ -384,6 +384,31 @@ Available streaming methods on `Request`:
 
 Streaming requests are excluded from dispatcher timeout expiration. When the client disconnects, the dispatcher notifies the worker via `on_disconnect(request_id)`.
 
+### NDJSON Streaming
+
+Stream JSON objects line-by-line (`application/x-ndjson`) — one JSON value per line, terminated by `\n`. Useful for incremental APIs that aren't event-shaped (long lists, log tails, progress updates):
+
+```python
+class MyWorker(_workers.Worker):
+    @_workers.api('/devices/scan', 'GET')
+    def scan(self, request):
+        request.response_ndjson()
+        for device in self.discover_devices():
+            request.send_ndjson({'id': device.id, 'name': device.name})
+        request.response_stream_end()
+        return _workers.DEFERRED
+```
+
+NDJSON methods on `Request`:
+
+| Method | Description |
+|--------|-------------|
+| `response_ndjson(headers, cookies)` | Start NDJSON stream (wrapper over `response_stream` with `application/x-ndjson`) |
+| `send_ndjson(obj)` | Send one JSON-serializable value as a line |
+| `response_stream_end()` | End stream and close connection (shared with SSE) |
+
+Same lifecycle as SSE: excluded from timeout expiration, client disconnect triggers `on_disconnect(request_id)`.
+
 ### Flow Control
 
 Workers can stop accepting new requests when busy. Requests stay in the shared pool queue for other workers to pick up:
@@ -540,6 +565,7 @@ Reason is one of:
 | `PENDING_DISCONNECTED` | Client disconnected mid-stream; worker was notified via control queue (race possible). |
 | `PENDING_STREAM_CLOSED` | Worker ended the SSE stream cleanly. |
 | `PENDING_SHUTDOWN` | Dispatcher is shutting down; client got 503. |
+| `PENDING_WORKER_DIED` | Worker process died/was killed while owning the request; client got 500. `on_worker_died()` runs first. |
 
 The hook is invoked after the client-facing action (respond / disconnect / control queue put)
 so dispatcher state is finalized when it runs. Exceptions raised by the hook are logged at
@@ -550,6 +576,51 @@ Override `on_pending_removed()` if you need exactly-once cleanup. Overriding bot
 but discouraged — for the `PENDING_COMPLETED` reason, `on_response()` is called immediately
 before `on_pending_removed()`.
 
+## Worker Death Hook
+
+Workers die — segfault in a C extension, OOM-kill, or the dispatcher kills them after
+`stuck_timeout`. Override `on_worker_died()` to capture which requests they had in-flight
+(useful for forensics when a malformed payload reproduces a crash):
+
+```python
+class MyDispatcher(_workers.Dispatcher):
+    def on_worker_died(
+            self, pool, worker_id, reason, exitcode, victims):
+        # `victims` is a list of (request_id, _PendingRequest) for all
+        # requests this worker had claimed but not completed.
+        # `exitcode` is None for stuck workers, otherwise the process exit
+        # code (negative = signal: -9 OOM, -11 SIGSEGV).
+        for rid, pending in victims:
+            c = pending.client
+            self._crash_queue.append({
+                'reason': reason,
+                'exitcode': exitcode,
+                'method': c.method,
+                'path': c.path,
+                'address': c.address,
+                'body': c.body,  # raw bytes — replay this to reproduce
+            })
+        # Default impl responds 500 to victims and fires
+        # on_pending_removed(PENDING_WORKER_DIED). Call it after capture.
+        super().on_worker_died(
+            pool, worker_id, reason, exitcode, victims)
+```
+
+What's a victim: any request the worker had claimed via `MSG_HEARTBEAT`
+(`pending.worker_id == worker_id`). Requests still in the queue (`worker_id is None`)
+are **not** victims — other workers in the pool will pick them up after restart.
+
+Default behavior (if you don't override) is to log the death + each victim, respond
+500 (or close the stream for SSE/NDJSON), and fire `on_pending_removed` for each.
+Override only if you want to persist payloads or customize the response status/body.
+
+**500 vs 503:** a victim of a crashed worker gets **500** (processing started, then
+the server failed). A new request arriving while the pool has zero alive workers
+gets **503 + `Retry-After: 1`** (rejected before processing — try again shortly).
+A request to a pool that has exceeded `max_restarts` in `restart_window` gets **503**
+permanently (`pool.is_degraded`). `pool.alive_count` is exposed for monitoring and
+also appears in `pool.status()`.
+
 ## Dispatcher Idle Hook
 
 Override `on_idle()` on the dispatcher for periodic background tasks — called on each `select()` timeout (every `SELECT_TIMEOUT` seconds, default 1s):

{uhttp_workers-1.4.0 → uhttp_workers-1.6.0}/README.md

@@ -370,6 +370,31 @@ Available streaming methods on `Request`:
 
 Streaming requests are excluded from dispatcher timeout expiration. When the client disconnects, the dispatcher notifies the worker via `on_disconnect(request_id)`.
 
+### NDJSON Streaming
+
+Stream JSON objects line-by-line (`application/x-ndjson`) — one JSON value per line, terminated by `\n`. Useful for incremental APIs that aren't event-shaped (long lists, log tails, progress updates):
+
+```python
+class MyWorker(_workers.Worker):
+    @_workers.api('/devices/scan', 'GET')
+    def scan(self, request):
+        request.response_ndjson()
+        for device in self.discover_devices():
+            request.send_ndjson({'id': device.id, 'name': device.name})
+        request.response_stream_end()
+        return _workers.DEFERRED
+```
+
+NDJSON methods on `Request`:
+
+| Method | Description |
+|--------|-------------|
+| `response_ndjson(headers, cookies)` | Start NDJSON stream (wrapper over `response_stream` with `application/x-ndjson`) |
+| `send_ndjson(obj)` | Send one JSON-serializable value as a line |
+| `response_stream_end()` | End stream and close connection (shared with SSE) |
+
+Same lifecycle as SSE: excluded from timeout expiration, client disconnect triggers `on_disconnect(request_id)`.
+
 ### Flow Control
 
 Workers can stop accepting new requests when busy. Requests stay in the shared pool queue for other workers to pick up:
@@ -526,6 +551,7 @@ Reason is one of:
 | `PENDING_DISCONNECTED` | Client disconnected mid-stream; worker was notified via control queue (race possible). |
 | `PENDING_STREAM_CLOSED` | Worker ended the SSE stream cleanly. |
 | `PENDING_SHUTDOWN` | Dispatcher is shutting down; client got 503. |
+| `PENDING_WORKER_DIED` | Worker process died/was killed while owning the request; client got 500. `on_worker_died()` runs first. |
 
 The hook is invoked after the client-facing action (respond / disconnect / control queue put)
 so dispatcher state is finalized when it runs. Exceptions raised by the hook are logged at
@@ -536,6 +562,51 @@ Override `on_pending_removed()` if you need exactly-once cleanup. Overriding bot
 but discouraged — for the `PENDING_COMPLETED` reason, `on_response()` is called immediately
 before `on_pending_removed()`.
 
+## Worker Death Hook
+
+Workers die — segfault in a C extension, OOM-kill, or the dispatcher kills them after
+`stuck_timeout`. Override `on_worker_died()` to capture which requests they had in-flight
+(useful for forensics when a malformed payload reproduces a crash):
+
+```python
+class MyDispatcher(_workers.Dispatcher):
+    def on_worker_died(
+            self, pool, worker_id, reason, exitcode, victims):
+        # `victims` is a list of (request_id, _PendingRequest) for all
+        # requests this worker had claimed but not completed.
+        # `exitcode` is None for stuck workers, otherwise the process exit
+        # code (negative = signal: -9 OOM, -11 SIGSEGV).
+        for rid, pending in victims:
+            c = pending.client
+            self._crash_queue.append({
+                'reason': reason,
+                'exitcode': exitcode,
+                'method': c.method,
+                'path': c.path,
+                'address': c.address,
+                'body': c.body,  # raw bytes — replay this to reproduce
+            })
+        # Default impl responds 500 to victims and fires
+        # on_pending_removed(PENDING_WORKER_DIED). Call it after capture.
+        super().on_worker_died(
+            pool, worker_id, reason, exitcode, victims)
+```
+
+What's a victim: any request the worker had claimed via `MSG_HEARTBEAT`
+(`pending.worker_id == worker_id`). Requests still in the queue (`worker_id is None`)
+are **not** victims — other workers in the pool will pick them up after restart.
+
+Default behavior (if you don't override) is to log the death + each victim, respond
+500 (or close the stream for SSE/NDJSON), and fire `on_pending_removed` for each.
+Override only if you want to persist payloads or customize the response status/body.
+
+**500 vs 503:** a victim of a crashed worker gets **500** (processing started, then
+the server failed). A new request arriving while the pool has zero alive workers
+gets **503 + `Retry-After: 1`** (rejected before processing — try again shortly).
+A request to a pool that has exceeded `max_restarts` in `restart_window` gets **503**
+permanently (`pool.is_degraded`). `pool.alive_count` is exposed for monitoring and
+also appears in `pool.status()`.
+
 ## Dispatcher Idle Hook
 
 Override `on_idle()` on the dispatcher for periodic background tasks — called on each `select()` timeout (every `SELECT_TIMEOUT` seconds, default 1s):

{uhttp_workers-1.4.0 → uhttp_workers-1.6.0}/pyproject.toml

@@ -13,7 +13,7 @@ authors = [
 license = "MIT"
 requires-python = ">=3.10"
 dependencies = [
-    "uhttp-server",
+    "uhttp-server>=2.5.2",
 ]
 classifiers = [
     "Programming Language :: Python :: 3",

{uhttp_workers-1.4.0 → uhttp_workers-1.6.0}/tests/test_dispatcher.py

@@ -10,10 +10,10 @@ from uhttp.workers import (
     Dispatcher, Worker, WorkerPool, Request, Response,
     api, sync, RejectRequest,
     MSG_RESPONSE, MSG_HEARTBEAT,
-    MSG_SSE_OPEN, MSG_SSE_EVENT, MSG_SSE_CLOSE,
+    MSG_SSE_OPEN, MSG_SSE_EVENT, MSG_SSE_CLOSE, MSG_NDJSON,
     CTL_DISCONNECT,
     PENDING_COMPLETED, PENDING_TIMEOUT, PENDING_DISCONNECTED,
-    PENDING_STREAM_CLOSED, PENDING_SHUTDOWN,
+    PENDING_STREAM_CLOSED, PENDING_SHUTDOWN, PENDING_WORKER_DIED,
     LOG_ERROR,
     _PendingRequest,
 )
@@ -29,13 +29,16 @@ class MockClient:
     """Mock HttpConnection for testing dispatcher logic."""
 
     def __init__(self, method='GET', path='/', query=None, data=None,
-            headers=None, content_type=None):
+            headers=None, content_type=None, body=None,
+            address='127.0.0.1'):
         self.method = method
         self.path = path
         self.query = query
         self.data = data
         self.headers = headers or {}
         self.content_type = content_type
+        self.body = body
+        self.address = address
         self.responded = False
         self.response_data = None
         self.response_status = None
@@ -78,6 +81,12 @@ class MockClient:
         self._chunks.append(data)
         return getattr(self, '_connected', True)
 
+    def send_ndjson(self, obj):
+        if not hasattr(self, '_ndjson'):
+            self._ndjson = []
+        self._ndjson.append(obj)
+        return getattr(self, '_connected', True)
+
     def response_stream_end(self):
         self.stream_ended = True
 
@@ -182,6 +191,8 @@ class TestDispatcherDoCheck(unittest.TestCase):
 
     def test_pass_check(self):
         pool = WorkerPool(DummyWorker, routes=['/api/**'])
+        # fake a live worker so alive_count > 0 without starting processes
+        pool.workers = [type('W', (), {'is_alive': lambda self: True})()]
         d = Dispatcher.__new__(Dispatcher)
         d._sync_routes = []
         d._static_routes = {}
@@ -311,6 +322,48 @@ class TestDispatcherPoolRouting(unittest.TestCase):
         self.assertTrue(client.responded)
         self.assertEqual(client.response_status, 503)
 
+    def test_dispatch_no_alive_workers_returns_503(self):
+        """Transient: pool has workers but none currently alive."""
+        # pool has dead workers (not degraded yet)
+        self.pool_default.workers = [
+            type('W', (), {'is_alive': lambda self: False})()]
+        d = Dispatcher.__new__(Dispatcher)
+        d._sync_routes = []
+        d._static_routes = {}
+        d._pools = [self.pool_default]
+        d._pending = {}
+        d._max_pending = 1000
+        d._next_request_id = 0
+
+        client = MockClient('GET', '/test')
+        d._dispatch_to_pool(client)
+        self.assertTrue(client.responded)
+        self.assertEqual(client.response_status, 503)
+        self.assertEqual(
+            client.response_data['error'], 'No workers available')
+        self.assertEqual(
+            client.response_headers.get('Retry-After'), '1')
+        # request was NOT enqueued
+        self.assertEqual(d._pending, {})
+
+    def test_dispatch_empty_workers_returns_503(self):
+        """Pool was never started (empty workers list)."""
+        # self.pool_default.workers is [] from __init__
+        d = Dispatcher.__new__(Dispatcher)
+        d._sync_routes = []
+        d._static_routes = {}
+        d._pools = [self.pool_default]
+        d._pending = {}
+        d._max_pending = 1000
+        d._next_request_id = 0
+
+        client = MockClient('GET', '/test')
+        d._dispatch_to_pool(client)
+        self.assertTrue(client.responded)
+        self.assertEqual(client.response_status, 503)
+        self.assertEqual(
+            client.response_data['error'], 'No workers available')
+
 
 class TestDispatcherProcessResponse(unittest.TestCase):
 
@@ -547,6 +600,42 @@ class TestDispatcherSSE(unittest.TestCase):
         # non-streaming should be expired
         self.assertNotIn(1, d._pending)
 
+    def test_ndjson_send(self):
+        d, pool = self._make_dispatcher()
+        client = MockClient('GET', '/api/stream')
+        pending = _PendingRequest(client, pool)
+        pending.streaming = True
+        d._pending[1] = pending
+        d._process_response(
+            (MSG_NDJSON, 1, {'devices': [1, 2, 3]}))
+        self.assertEqual(len(client._ndjson), 1)
+        self.assertEqual(client._ndjson[0], {'devices': [1, 2, 3]})
+        # still pending — stream open
+        self.assertIn(1, d._pending)
+
+    def test_ndjson_client_disconnect(self):
+        d, pool = self._make_dispatcher()
+        pool.start(d._response_queue)
+        client = MockClient('GET', '/api/stream')
+        client._connected = False
+        pending = _PendingRequest(client, pool)
+        pending.streaming = True
+        pending.worker_id = 0
+        d._pending[1] = pending
+        d._process_response((MSG_NDJSON, 1, {'x': 1}))
+        # removed from pending
+        self.assertNotIn(1, d._pending)
+        # CTL_DISCONNECT sent to worker's control queue
+        msg = pool._control_queues[0].get(timeout=1)
+        self.assertEqual(msg, (CTL_DISCONNECT, 1))
+        pool.shutdown(timeout=2)
+
+    def test_ndjson_ignored_after_close(self):
+        d, pool = self._make_dispatcher()
+        # no pending request with id 99
+        d._process_response((MSG_NDJSON, 99, {'x': 1}))
+        # should not raise, just ignore
+
     def test_sse_event_ignored_after_close(self):
         d, pool = self._make_dispatcher()
         # no pending request with id 99
@@ -727,5 +816,207 @@ class TestDispatcherPendingRemoved(unittest.TestCase):
         self.assertTrue(client.responded)
 
 
+class TestDispatcherWorkerDied(unittest.TestCase):
+    """Tests for on_worker_died hook and victim handling."""
+
+    def _make_dispatcher(self, dispatcher_cls=Dispatcher):
+        # queue_warning=0 disables the queue-size check (which would
+        # otherwise touch pool.pending_count → request_queue.qsize()).
+        pool = WorkerPool(
+            DummyWorker, routes=['/api/**'], queue_warning=0)
+        # Mock check_workers so we control what it returns without
+        # actually starting processes.
+        pool._fake_restarted = []
+        pool.check_workers = lambda: pool._fake_restarted
+        d = dispatcher_cls.__new__(dispatcher_cls)
+        d._sync_routes = []
+        d._static_routes = {}
+        d._pools = [pool]
+        d._pending = {}
+        d._max_pending = 1000
+        d._next_request_id = 0
+        d._response_queue = mp.Queue()
+        d._log_is_tty = False
+        d.log_calls = []
+        d.on_log = lambda name, level, msg: d.log_calls.append(
+            (name, level, msg))
+        d.recorded_removed = []
+        return d, pool
+
+    def test_single_victim_gets_500(self):
+
+        class RecordingDispatcher(Dispatcher):
+            def on_pending_removed(self, request_id, pending, reason):
+                self.recorded_removed.append((request_id, reason))
+
+        d, pool = self._make_dispatcher(RecordingDispatcher)
+        client = MockClient(
+            'POST', '/api/scan', body=b'\x00\x01bad', address='10.0.0.7')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        d._pending[42] = pending
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        # client got 500
+        self.assertTrue(client.responded)
+        self.assertEqual(client.response_status, 500)
+        self.assertEqual(client.response_data['error'], 'Worker crashed')
+        self.assertIn('exit=-11', client.response_data['reason'])
+        # removed from pending + hook fired
+        self.assertNotIn(42, d._pending)
+        self.assertEqual(
+            d.recorded_removed, [(42, PENDING_WORKER_DIED)])
+
+    def test_multiple_victims_all_handled(self):
+        d, pool = self._make_dispatcher()
+        c1 = MockClient('GET', '/api/a', address='1.1.1.1')
+        c2 = MockClient('GET', '/api/b', address='2.2.2.2')
+        c3 = MockClient('GET', '/api/c', address='3.3.3.3')
+        for rid, c in [(1, c1), (2, c2), (3, c3)]:
+            p = _PendingRequest(c, pool)
+            p.worker_id = 0
+            d._pending[rid] = p
+        pool._fake_restarted = [(0, 'stuck', None)]
+        d._check_all_workers()
+        for c in (c1, c2, c3):
+            self.assertTrue(c.responded)
+            self.assertEqual(c.response_status, 500)
+        self.assertEqual(d._pending, {})
+
+    def test_streaming_victim_gets_stream_end(self):
+        d, pool = self._make_dispatcher()
+        client = MockClient('GET', '/api/events')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        pending.streaming = True
+        d._pending[1] = pending
+        pool._fake_restarted = [(0, 'died exit=-9', -9)]
+        d._check_all_workers()
+        # stream ended, NOT respond()
+        self.assertTrue(getattr(client, 'stream_ended', False))
+        self.assertFalse(client.responded)
+        self.assertNotIn(1, d._pending)
+
+    def test_queued_request_not_a_victim(self):
+        """Request with worker_id=None is still in queue — not a victim."""
+        d, pool = self._make_dispatcher()
+        # request belonging to dying worker
+        in_flight = MockClient('GET', '/api/active')
+        p1 = _PendingRequest(in_flight, pool)
+        p1.worker_id = 0
+        d._pending[1] = p1
+        # request still in queue, no worker claimed it
+        queued = MockClient('GET', '/api/queued')
+        p2 = _PendingRequest(queued, pool)
+        # p2.worker_id stays None
+        d._pending[2] = p2
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        # in-flight responded
+        self.assertTrue(in_flight.responded)
+        self.assertNotIn(1, d._pending)
+        # queued untouched
+        self.assertFalse(queued.responded)
+        self.assertIn(2, d._pending)
+
+    def test_other_worker_not_affected(self):
+        """Only victims of THIS worker are handled; other workers stay."""
+        d, pool = self._make_dispatcher()
+        c1 = MockClient('GET', '/api/a')
+        c2 = MockClient('GET', '/api/b')
+        p1 = _PendingRequest(c1, pool)
+        p1.worker_id = 0
+        p2 = _PendingRequest(c2, pool)
+        p2.worker_id = 1
+        d._pending[1] = p1
+        d._pending[2] = p2
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        self.assertTrue(c1.responded)
+        self.assertNotIn(1, d._pending)
+        self.assertFalse(c2.responded)
+        self.assertIn(2, d._pending)
+
+    def test_late_response_after_victim_cleanup_dropped(self):
+        """MSG_RESPONSE from dead worker arriving after victim removal is dropped."""
+        d, pool = self._make_dispatcher()
+        client = MockClient('GET', '/api/test')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        d._pending[1] = pending
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        # request already gone; client already got 500
+        self.assertEqual(client.response_status, 500)
+        # late response from before-death — must not break or double-respond
+        client.response_status = None
+        late = Response(request_id=1, data={'ok': True}, status=200)
+        d._process_response((MSG_RESPONSE, 1, late))
+        # silently dropped
+        self.assertIsNone(client.response_status)
+
+    def test_no_victims_just_logs(self):
+        """Worker died while idle — restarted but no pending requests."""
+        d, pool = self._make_dispatcher()
+        pool._fake_restarted = [(0, 'died exit=0', 0)]
+        d._check_all_workers()
+        # no crash, no pending changes
+        self.assertEqual(d._pending, {})
+        # should have logged
+        error_logs = [
+            msg for _, level, msg in d.log_calls if level == LOG_ERROR]
+        self.assertEqual(len(error_logs), 1)
+        self.assertIn('victims=0', error_logs[0])
+
+    def test_override_can_persist_payload(self):
+        """User override can capture victim payload before super() responds."""
+        captured = []
+
+        class ForensicDispatcher(Dispatcher):
+            def on_worker_died(
+                    self, pool, worker_id, reason, exitcode, victims):
+                for rid, pending in victims:
+                    captured.append({
+                        'rid': rid,
+                        'address': pending.client.address,
+                        'body': pending.client.body,
+                        'reason': reason,
+                        'exitcode': exitcode})
+                super().on_worker_died(
+                    pool, worker_id, reason, exitcode, victims)
+
+        d, pool = self._make_dispatcher(ForensicDispatcher)
+        client = MockClient(
+            'POST', '/api/process',
+            body=b'\xff\xfecorrupted', address='9.9.9.9')
+        pending = _PendingRequest(client, pool)
+        pending.worker_id = 0
+        d._pending[7] = pending
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        d._check_all_workers()
+        self.assertEqual(len(captured), 1)
+        self.assertEqual(captured[0]['address'], '9.9.9.9')
+        self.assertEqual(captured[0]['body'], b'\xff\xfecorrupted')
+        self.assertEqual(captured[0]['exitcode'], -11)
+        # super() still ran
+        self.assertEqual(client.response_status, 500)
+
+    def test_hook_exception_does_not_crash_dispatcher(self):
+
+        class BrokenDispatcher(Dispatcher):
+            def on_worker_died(self, *args, **kwargs):
+                raise RuntimeError('boom')
+
+        d, pool = self._make_dispatcher(BrokenDispatcher)
+        pool._fake_restarted = [(0, 'died exit=-11', -11)]
+        # must not propagate
+        d._check_all_workers()
+        error_logs = [
+            msg for _, level, msg in d.log_calls if level == LOG_ERROR]
+        # one error log about the hook failure
+        self.assertTrue(any('on_worker_died' in m for m in error_logs))
+        self.assertTrue(any('boom' in m for m in error_logs))
+
+
 if __name__ == '__main__':
     unittest.main()

{uhttp_workers-1.4.0 → uhttp_workers-1.6.0}/tests/test_worker.py

@@ -8,7 +8,7 @@ from uhttp.workers import (
     Worker, Request, Response, api, RejectRequest, DEFERRED,
     Logger, LOG_DEBUG, LOG_INFO, LOG_WARNING, LOG_ERROR,
     MSG_RESPONSE, MSG_HEARTBEAT,
-    MSG_SSE_OPEN, MSG_SSE_EVENT, MSG_SSE_CLOSE,
+    MSG_SSE_OPEN, MSG_SSE_EVENT, MSG_SSE_CLOSE, MSG_NDJSON,
     CTL_DISCONNECT)
 
 
@@ -520,6 +520,38 @@ class TestSSERequest(unittest.TestCase):
         self.assertIsNone(msg[4])
         self.assertIsNone(msg[5])
 
+    def test_response_ndjson(self):
+        self.req.response_ndjson(
+            headers={'X-Custom': '1'},
+            cookies={'sid': 'abc'})
+        msg = self.queue.get(timeout=1)
+        self.assertEqual(msg[0], MSG_SSE_OPEN)
+        self.assertEqual(msg[1], 10)
+        self.assertEqual(msg[2], 'application/x-ndjson')
+        self.assertEqual(msg[3], {'X-Custom': '1'})
+        self.assertEqual(msg[4], {'sid': 'abc'})
+
+    def test_response_ndjson_defaults(self):
+        self.req.response_ndjson()
+        msg = self.queue.get(timeout=1)
+        self.assertEqual(msg[0], MSG_SSE_OPEN)
+        self.assertEqual(msg[2], 'application/x-ndjson')
+        self.assertIsNone(msg[3])
+        self.assertIsNone(msg[4])
+
+    def test_send_ndjson(self):
+        self.req.send_ndjson({'devices': [1, 2, 3]})
+        msg = self.queue.get(timeout=1)
+        self.assertEqual(msg[0], MSG_NDJSON)
+        self.assertEqual(msg[1], 10)
+        self.assertEqual(msg[2], {'devices': [1, 2, 3]})
+
+    def test_send_ndjson_keepalive(self):
+        self.req.send_ndjson({})
+        msg = self.queue.get(timeout=1)
+        self.assertEqual(msg[0], MSG_NDJSON)
+        self.assertEqual(msg[2], {})
+
     def test_response_stream_end(self):
         self.req.response_stream_end()
         msg = self.queue.get(timeout=1)

{uhttp_workers-1.4.0 → uhttp_workers-1.6.0}/tests/test_worker_pool.py

@@ -120,6 +120,27 @@ class TestWorkerPoolStatus(unittest.TestCase):
         pool = WorkerPool(DummyWorker)
         self.assertFalse(pool.is_degraded)
 
+    def test_alive_count_empty(self):
+        pool = WorkerPool(DummyWorker, num_workers=2)
+        # not started yet
+        self.assertEqual(pool.alive_count, 0)
+
+    def test_alive_count_running(self):
+        pool = WorkerPool(DummyWorker, num_workers=2)
+        response_queue = mp.Queue()
+        pool.start(response_queue)
+        time.sleep(0.2)
+        self.assertEqual(pool.alive_count, 2)
+        pool.shutdown(timeout=3)
+
+    def test_alive_count_in_status(self):
+        pool = WorkerPool(DummyWorker, num_workers=2)
+        response_queue = mp.Queue()
+        pool.start(response_queue)
+        time.sleep(0.2)
+        self.assertEqual(pool.status()['alive_count'], 2)
+        pool.shutdown(timeout=3)
+
 
 class TestWorkerPoolCheckWorkers(unittest.TestCase):
 

{uhttp_workers-1.4.0 → uhttp_workers-1.6.0}/uhttp/workers.py

@@ -23,6 +23,7 @@ MSG_LOG = 'LOG'
 MSG_SSE_OPEN = 'SSE_OPEN'
 MSG_SSE_EVENT = 'SSE_EVENT'
 MSG_SSE_CLOSE = 'SSE_CLOSE'
+MSG_NDJSON = 'NDJSON'
 
 # Worker control messages
 CTL_STOP = 'STOP'
@@ -35,6 +36,7 @@ PENDING_TIMEOUT = 'TIMEOUT'
 PENDING_DISCONNECTED = 'DISCONNECTED'
 PENDING_STREAM_CLOSED = 'STREAM_CLOSED'
 PENDING_SHUTDOWN = 'SHUTDOWN'
+PENDING_WORKER_DIED = 'WORKER_DIED'
 
 # Sentinel for deferred response
 DEFERRED = object()
@@ -282,6 +284,25 @@ class Request:
             (MSG_SSE_EVENT, self.request_id,
              data, event, event_id, retry))
 
+    def response_ndjson(self, headers=None, cookies=None):
+        """Start NDJSON streaming response (application/x-ndjson).
+
+        Thin wrapper over response_stream(). Use with DEFERRED — call from
+        handler, then send_ndjson() later. Call response_stream_end() to finish.
+        """
+        self._response_queue.put(
+            (MSG_SSE_OPEN, self.request_id,
+             'application/x-ndjson', headers, cookies))
+
+    def send_ndjson(self, obj):
+        """Send one JSON-serializable object as an NDJSON line.
+
+        Args:
+            obj: any JSON-serializable value (dict/list/str/int/float/bool/None)
+        """
+        self._response_queue.put(
+            (MSG_NDJSON, self.request_id, obj))
+
     def response_stream_end(self):
         """End streaming response and close connection."""
         self._response_queue.put(
@@ -859,7 +880,10 @@ class WorkerPool:
         """Check worker health, restart dead or stuck workers.
 
         Returns:
-            List of (worker_id, reason) tuples for restarted workers.
+            List of (worker_id, reason, exitcode) tuples for restarted
+            workers. exitcode is None for stuck workers (dispatcher killed
+            them), otherwise the process exit code (negative = signal:
+            -9 OOM, -11 SIGSEGV, -15 SIGTERM, etc.).
         """
         restarted = []
         now = _time.time()
@@ -869,8 +893,10 @@ class WorkerPool:
             if now - t < self.restart_window]
         for i, worker in enumerate(self.workers):
             reason = None
+            exitcode = None
             if not worker.is_alive():
-                reason = f"died exit={worker.exitcode}"
+                exitcode = worker.exitcode
+                reason = f"died exit={exitcode}"
             elif now - self._last_seen.get(i, 0) > self.stuck_timeout:
                 reason = "stuck"
                 worker.kill()
@@ -884,7 +910,7 @@ class WorkerPool:
                 if len(self._restart_times) >= self.max_restarts:
                     self._degraded = True
                 self._start_worker(i)
-                restarted.append((i, reason))
+                restarted.append((i, reason, exitcode))
         return restarted
 
     def matches(self, path):
@@ -939,6 +965,11 @@ class WorkerPool:
     def is_degraded(self):
         return self._degraded
 
+    @property
+    def alive_count(self):
+        """Number of worker processes currently alive."""
+        return sum(1 for w in self.workers if w.is_alive())
+
     @property
     def pending_count(self):
         try:
@@ -956,6 +987,7 @@ class WorkerPool:
         return {
             'name': self.name,
             'degraded': self._degraded,
+            'alive_count': self.alive_count,
             'queue_size': self.pending_count,
             'workers': [
                 {
@@ -1107,6 +1139,9 @@ class Dispatcher:
                                     notified via control queue (race possible).
             PENDING_STREAM_CLOSED - worker ended the SSE stream cleanly.
             PENDING_SHUTDOWN      - dispatcher is shutting down; client got 503.
+            PENDING_WORKER_DIED   - worker process died/was killed while owning
+                                    this request; client got 500. on_worker_died()
+                                    runs first.
 
         Args:
             request_id: The request id being removed.
@@ -1181,6 +1216,11 @@ class Dispatcher:
             client.respond(
                 {'error': 'Service unavailable'}, status=503)
             return
+        if pool.alive_count == 0:
+            client.respond(
+                {'error': 'No workers available'}, status=503,
+                headers={'Retry-After': '1'})
+            return
         if len(self._pending) >= self._max_pending:
             client.respond(
                 {'error': 'Too many requests'}, status=503)
@@ -1253,6 +1293,13 @@ class Dispatcher:
                         event_id=event_id, retry=retry)
                 if not ok:
                     self._stream_disconnected(request_id, pending)
+        elif msg_type == MSG_NDJSON:
+            _, request_id, obj = msg
+            pending = self._pending.get(request_id)
+            if pending is not None:
+                ok = pending.client.send_ndjson(obj)
+                if not ok:
+                    self._stream_disconnected(request_id, pending)
         elif msg_type == MSG_SSE_CLOSE:
             _, request_id = msg
             pending = self._pending.pop(request_id, None)
@@ -1326,8 +1373,18 @@ class Dispatcher:
         """Check health of all worker pools and queue sizes."""
         for pool in self._pools:
             restarted = pool.check_workers()
-            for worker_id, reason in restarted:
-                self._on_worker_restarted(pool, worker_id, reason)
+            for worker_id, reason, exitcode in restarted:
+                victims = [
+                    (rid, p) for rid, p in self._pending.items()
+                    if p.pool is pool and p.worker_id == worker_id]
+                try:
+                    self.on_worker_died(
+                        pool, worker_id, reason, exitcode, victims)
+                except Exception:
+                    self.on_log(
+                        pool.name, LOG_ERROR,
+                        f"on_worker_died() failed:\n"
+                        f"{_traceback.format_exc()}")
             if pool.queue_warning:
                 qsize = pool.pending_count
                 if qsize >= pool.queue_warning:
@@ -1363,14 +1420,57 @@ class Dispatcher:
             print(f"{prefix}{level_name:8s} {name:20s} {message}",
                 file=_sys.stderr)
 
-    def _on_worker_restarted(self, pool, worker_id, reason):
-        """Called when a worker is restarted.
+    def on_worker_died(self, pool, worker_id, reason, exitcode, victims):
+        """Called when a worker process died or was killed by the dispatcher.
 
-        Default logs the event. Override to customize.
+        Default behavior:
+          1. Log restart reason + each victim (request id, client address,
+             method, path, body size).
+          2. Respond 500 to every victim's client (or response_stream_end()
+             for streams), remove them from _pending, and fire
+             on_pending_removed(PENDING_WORKER_DIED) for each.
+
+        Override to capture victim payloads (e.g., persist to disk for
+        post-mortem) BEFORE calling super(). pending.client gives access
+        to method, path, headers, body, address.
+
+        Args:
+            pool: WorkerPool the worker belonged to.
+            worker_id: Index of the restarted worker.
+            reason: 'stuck' or 'died exit=N' (string from check_workers).
+            exitcode: Process exit code (int) or None for stuck workers.
+                Negative values are signals: -9 OOM, -11 SIGSEGV, etc.
+            victims: List of (request_id, _PendingRequest) tuples — requests
+                this worker had claimed (via MSG_HEARTBEAT) but never
+                completed. May be empty if worker died while idle.
         """
         self.on_log(
             f'{pool.name}[{worker_id}]', LOG_ERROR,
-            f"worker restarted: {reason}")
+            f"worker restarted: {reason}, "
+            f"victims={len(victims)}")
+        for request_id, pending in victims:
+            c = pending.client
+            body_len = len(c.body) if c.body is not None else 0
+            self.on_log(
+                pool.name, LOG_ERROR,
+                f"  victim rid={request_id} from={c.address} "
+                f"{c.method} {c.path} body={body_len}B")
+            del self._pending[request_id]
+            if pending.streaming:
+                try:
+                    pending.client.response_stream_end()
+                except Exception:
+                    pass
+            else:
+                try:
+                    pending.client.respond(
+                        {'error': 'Worker crashed',
+                         'reason': reason},
+                        status=500)
+                except Exception:
+                    pass
+            self._notify_pending_removed(
+                request_id, pending, PENDING_WORKER_DIED)
 
     def _sigterm(self, _signo, _stack_frame):
         self._running = False

{uhttp_workers-1.4.0 → uhttp_workers-1.6.0/uhttp_workers.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: uhttp-workers
-Version: 1.4.0
+Version: 1.6.0
 Summary: Multi-process worker dispatcher built on uhttp-server
 Author-email: Pavel Revak <pavelrevak@gmail.com>
 License-Expression: MIT
@@ -10,7 +10,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Operating System :: POSIX
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
-Requires-Dist: uhttp-server
+Requires-Dist: uhttp-server>=2.5.2
 
 # uhttp-workers
 
@@ -384,6 +384,31 @@ Available streaming methods on `Request`:
 
 Streaming requests are excluded from dispatcher timeout expiration. When the client disconnects, the dispatcher notifies the worker via `on_disconnect(request_id)`.
 
+### NDJSON Streaming
+
+Stream JSON objects line-by-line (`application/x-ndjson`) — one JSON value per line, terminated by `\n`. Useful for incremental APIs that aren't event-shaped (long lists, log tails, progress updates):
+
+```python
+class MyWorker(_workers.Worker):
+    @_workers.api('/devices/scan', 'GET')
+    def scan(self, request):
+        request.response_ndjson()
+        for device in self.discover_devices():
+            request.send_ndjson({'id': device.id, 'name': device.name})
+        request.response_stream_end()
+        return _workers.DEFERRED
+```
+
+NDJSON methods on `Request`:
+
+| Method | Description |
+|--------|-------------|
+| `response_ndjson(headers, cookies)` | Start NDJSON stream (wrapper over `response_stream` with `application/x-ndjson`) |
+| `send_ndjson(obj)` | Send one JSON-serializable value as a line |
+| `response_stream_end()` | End stream and close connection (shared with SSE) |
+
+Same lifecycle as SSE: excluded from timeout expiration, client disconnect triggers `on_disconnect(request_id)`.
+
 ### Flow Control
 
 Workers can stop accepting new requests when busy. Requests stay in the shared pool queue for other workers to pick up:
@@ -540,6 +565,7 @@ Reason is one of:
 | `PENDING_DISCONNECTED` | Client disconnected mid-stream; worker was notified via control queue (race possible). |
 | `PENDING_STREAM_CLOSED` | Worker ended the SSE stream cleanly. |
 | `PENDING_SHUTDOWN` | Dispatcher is shutting down; client got 503. |
+| `PENDING_WORKER_DIED` | Worker process died/was killed while owning the request; client got 500. `on_worker_died()` runs first. |
 
 The hook is invoked after the client-facing action (respond / disconnect / control queue put)
 so dispatcher state is finalized when it runs. Exceptions raised by the hook are logged at
@@ -550,6 +576,51 @@ Override `on_pending_removed()` if you need exactly-once cleanup. Overriding bot
 but discouraged — for the `PENDING_COMPLETED` reason, `on_response()` is called immediately
 before `on_pending_removed()`.
 
+## Worker Death Hook
+
+Workers die — segfault in a C extension, OOM-kill, or the dispatcher kills them after
+`stuck_timeout`. Override `on_worker_died()` to capture which requests they had in-flight
+(useful for forensics when a malformed payload reproduces a crash):
+
+```python
+class MyDispatcher(_workers.Dispatcher):
+    def on_worker_died(
+            self, pool, worker_id, reason, exitcode, victims):
+        # `victims` is a list of (request_id, _PendingRequest) for all
+        # requests this worker had claimed but not completed.
+        # `exitcode` is None for stuck workers, otherwise the process exit
+        # code (negative = signal: -9 OOM, -11 SIGSEGV).
+        for rid, pending in victims:
+            c = pending.client
+            self._crash_queue.append({
+                'reason': reason,
+                'exitcode': exitcode,
+                'method': c.method,
+                'path': c.path,
+                'address': c.address,
+                'body': c.body,  # raw bytes — replay this to reproduce
+            })
+        # Default impl responds 500 to victims and fires
+        # on_pending_removed(PENDING_WORKER_DIED). Call it after capture.
+        super().on_worker_died(
+            pool, worker_id, reason, exitcode, victims)
+```
+
+What's a victim: any request the worker had claimed via `MSG_HEARTBEAT`
+(`pending.worker_id == worker_id`). Requests still in the queue (`worker_id is None`)
+are **not** victims — other workers in the pool will pick them up after restart.
+
+Default behavior (if you don't override) is to log the death + each victim, respond
+500 (or close the stream for SSE/NDJSON), and fire `on_pending_removed` for each.
+Override only if you want to persist payloads or customize the response status/body.
+
+**500 vs 503:** a victim of a crashed worker gets **500** (processing started, then
+the server failed). A new request arriving while the pool has zero alive workers
+gets **503 + `Retry-After: 1`** (rejected before processing — try again shortly).
+A request to a pool that has exceeded `max_restarts` in `restart_window` gets **503**
+permanently (`pool.is_degraded`). `pool.alive_count` is exposed for monitoring and
+also appears in `pool.status()`.
+
 ## Dispatcher Idle Hook
 
 Override `on_idle()` on the dispatcher for periodic background tasks — called on each `select()` timeout (every `SELECT_TIMEOUT` seconds, default 1s):

uhttp_workers-1.6.0/uhttp_workers.egg-info/requires.txt

@@ -0,0 +1 @@
+uhttp-server>=2.5.2

uhttp_workers-1.4.0/uhttp_workers.egg-info/requires.txt

@@ -1 +0,0 @@
-uhttp-server