naeural-core 7.7.330__tar.gz → 7.7.332__tar.gz

{naeural_core-7.7.330 → naeural_core-7.7.332}/AGENTS.md

@@ -397,3 +397,5 @@ Latest:
 - `2026-10-09`: GiB-scale memory validation is opt-in via `plugin_memory_testbed/run_load.py` (3 GiB) and `run_e2e.py --scenario memory-growth` (4 GiB, both no swap). Kernel/RSS checks verify resident growth, 32K nested records and release; full overflow-handler timing includes queue scans and lock waits. Native RAM alerts measure host memory, not the container limit. Preserve timing failures even when repeats pass; the exclusion fix does not guarantee bounded scan latency.
 - `2026-10-13`: `EE_SHM_DIAGNOSTICS` is opt-in JSON with at most two actual serving worker targets and an absolute expiry within 600 seconds. Diagnostic identities resolve covered aliases; routing/envelopes stay unchanged. Output is capped at one MiB per interpreter, with twelve event-state keys, fixed suppressed-failure summaries, 64 scalar pending PREDICT requests per target and up to 64 image-count entries per request. Independently sampled timeout snapshots precede cleanup and retain bounded suppressed context; collection cannot prove slot ownership or pixel integrity. See `docs/20261012_shm_diagnostics.md` and run both `naeural_core/utils/test_shm_diagnostic_evidence.py` and `naeural_core/utils/test_shm_diagnostic_transport.py` alongside existing diagnostics tests.
 - `2026-10-13`: Covered aliases copy `COMM`, `SERVER` and `PID`, not `SHM`; their payloads can use pipe serialization while direct requests use the physical worker's segment. Configuration diagnostics must resolve the physical registry entry to avoid relogging on each alias switch. `business/test_framework/shm_testbed/run_e2e.py` exercises this behavior with a real isolated Edge node, generated frames and a deterministic CPU Torch model; broker traffic stays in a loopback-only namespace and cleanup checks preserve pre-existing service identities.
+
+- `2026-10-19`: Plugins receive `BCWrapper`, not the raw SDK engine. New plugin-facing blockchain methods and verification options must be forwarded explicitly in `naeural_core/utils/plugins_base/bc_wrapper.py`; dAuth reconciliation requires SDK 3.5.58 and must be verified through this wrapper. Run `python -m unittest naeural_core.utils.plugins_base.test_bc_wrapper_dauth` for these contracts.

{naeural_core-7.7.330 → naeural_core-7.7.332}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: naeural_core
-Version: 7.7.330
+Version: 7.7.332
 Summary: Ratio1 Core is the backbone of the Ratio1 Edge Protocol.
 Project-URL: Homepage, https://github.com/Ratio1/naeural_core
 Project-URL: Bug Tracker, https://github.com/Ratio1/naeural_core/issues
@@ -30,7 +30,7 @@ Requires-Dist: pydantic
 Requires-Dist: pymssql
 Requires-Dist: pynvml>=11.4
 Requires-Dist: pyyaml>=6.0
-Requires-Dist: ratio1>=3.5.50
+Requires-Dist: ratio1>=3.5.58
 Requires-Dist: scikit-image
 Requires-Dist: scikit-learn
 Requires-Dist: seaborn

{naeural_core-7.7.330 → naeural_core-7.7.332}/docs/20261014_serving_shm_lifecycle.md

@@ -11,6 +11,8 @@ generation. Logical model identities and inference routing do not change.
   counter and a bounded model label. The worker receives that exact name through
   its existing startup arguments. Only collisions receive up to three attempts;
   other allocation failures use the existing pipe fallback before child startup.
+  A failed cleanup of an owned candidate blocks retries and pipe fallback until
+  those resources are released; fallback must not hide a leaked allocation.
 - Each successfully created object is owned independently. Partial initialization
   rolls back only owned objects. Both objects receive cleanup attempts despite
   close/unlink failures. Failed closes remain retryable; attachers never unlink.
@@ -21,6 +23,12 @@ generation. Logical model identities and inference routing do not change.
   attachment fails cannot reach READY. Cleanup preserves the original failure.
   All-worker shutdown attempts the remaining workers before rethrowing the first
   teardown failure; one failed STOP cannot strand the next worker's allocation.
+- Incomplete cleanup is retained separately from inference routing. The manager
+  keeps only unfinished process, endpoint and SHM operations, retries them before
+  replacing that worker, and includes them in final shutdown. That final call
+  also makes one bounded retry for newly failed cleanup, preserving the original
+  exception even when the retry succeeds. Persistent failures remain explicit;
+  retries are bounded and cannot guarantee recovery from an OS-level failure.
 - Effective serving capacity is `max(100, configured SHM_MAX_LEN)`; omitted means
   100. The generic utility's default remains unchanged for non-serving callers.
   External capture producers keep their explicitly assigned names.
@@ -56,6 +64,11 @@ known legacy model names. Candidates cover the `r1s_` namespace and exact known
 legacy data/control names; this is not a complete host-wide orphan inventory.
 Truncation, stat errors and unavailable access are explicit. Unavailable counts
 are `unknown`. No discovery attaches, registers or deletes a foreign object.
+The summary also reports pending-cleanup worker count and at most 16 identities,
+with remaining process/endpoint/segment ownership. Pending allocations are
+recognized as manager-owned rather than classified as possible foreign leftovers.
+The ownership registry is never truncated like diagnostic history; blocking a
+replacement prevents repeated failed generations for the same worker.
 
 Failure history retains 32 scalar records, shows the most recent eight and counts
 dropped history. Immediate manager lifecycle reports are limited to one per ten
@@ -90,6 +103,27 @@ rejects the 101st admission and returns occupancy to zero. This does not replace
 GPU-model or 100-camera load acceptance. The retained POSIX backend is exercised
 through an API adapter with real shared-memory file descriptors and mmap when
 `posix_ipc` is unavailable.
+Cleanup regressions also use a real spawned worker interrupted by SIGINT during
+join, and real tiny SHM with injected unlink failure. Recovery must come through
+the manager's shutdown/retry path, not an extra test-only reference to the object.
+
+The accompanying hotfix corrects a regression in `9f31d28`: independent cleanup
+attempts were followed by unconditional registry removal even when resources
+remained. The preceding implementation retained its active entry when cleanup
+raised, but stopped attempting subsequent resources/workers. Both shortcomings
+are addressed by retaining unfinished ownership outside routing and continuing
+independent cleanup. Silent Creator/User role switching and unchecked inference
+indexing predate the reviewed three commits. They must not be confused with this
+new cleanup regression; this patch does not change the inference indexing API.
+
+A pre-existing constructor boundary remains outside this manager hotfix: if the
+SHM utility constructor raises before returning its object and its own rollback
+also fails, the manager never receives the handle to retain. An interrupted
+constructor log combined with an unlink failure reproduces this on both old and
+current code. Manager retention covers allocations returned to it, including
+failed-initialization objects; it does not guarantee recovery from that earlier
+boundary. A separate utility ownership-transfer change and focused review would
+be needed. Persistent OS cleanup failures and hard kills also remain limitations.
 
 Publish core before building E2 with the 100-slot template. Check the effective
 configuration and aggregate SHM budget, then observe an authorized E2-only

{naeural_core-7.7.330 → naeural_core-7.7.332}/naeural_core/business/base/cv_plugin_executor.py

@@ -957,7 +957,16 @@ class CVPluginExecutor(BasePluginExecutor,
         try:
           raw_inferences = self.dataapi_inferences()
           startup_params = self.cfg_startup_ai_engine_params
-          for model in self._get_detector_ai_engines():
+          # BusinessManager routes lowercase names, including list entries.
+          # Select diagnostics from that same configuration without changing
+          # the case-sensitive selectors used by inference filtering/tracking.
+          configured_engines = self.cfg_ai_engine
+          if not isinstance(configured_engines, list):
+            configured_engines = [configured_engines]
+          normalized_engines = {engine.lower() for engine in configured_engines}
+          diagnostic_detectors = [engine for engine in AI_ENGINES
+                                  if 'detector' in engine and engine in normalized_engines]
+          for model in diagnostic_detectors:
             # Match BusinessManager's single-engine flat/keyed and multi-engine
             # keyed startup config, retaining the actual worker instance.
             engine = model.lower()

{naeural_core-7.7.330 → naeural_core-7.7.332}/naeural_core/main/ver.py

@@ -1,5 +1,4 @@
-__VER__ = '7.7.330'
-
+__VER__ = '7.7.332'
 
 
 

{naeural_core-7.7.330 → naeural_core-7.7.332}/naeural_core/serving/serving_manager.py

@@ -16,6 +16,7 @@ windows: set PYTHONPATH=%PYTHONPATH%;C:\work\
 """
 
 import os
+import itertools
 import shutil
 import threading
 import traceback
@@ -181,6 +182,9 @@ class ServingManager(Manager):
     self.log_timeouts_last_ts = None
     self.log_timeouts_period = log_timeouts_period
     self._servers = {}
+    # Failed teardown is not routable, but its process/handles remain owned.
+    # A new generation of the same worker is blocked until these are released.
+    self._pending_cleanup = {}
     # Allocation identities must survive neither a worker restart nor a new
     # manager session. Logical model names remain unchanged for routing.
     self._shm_session = uuid4().hex
@@ -321,7 +325,16 @@ class ServingManager(Manager):
     return self._timeouts[server_name]
 
   def _assign_server(self, replaced_server, main_server):
+    """Assign coverage only after the old physical worker's resources retire.
+
+    Parameters
+    ----------
+    replaced_server, main_server : str
+      Normalized alias and physical serving identities. An alias must not hide
+      unfinished cleanup belonging to a former worker with the same name.
+    """
     assert main_server in self._servers
+    self._retry_pending_cleanup(replaced_server)
     if replaced_server not in self._servers:
       self._servers[replaced_server] = {}
     for prop in SMConst.BASE_PROPS:
@@ -530,6 +543,10 @@ class ServingManager(Manager):
       shm = entry.get(SMConst.SHM)
       if shm is not None:
         owned.update((shm._mem_name, shm._mem_name + '_additional'))
+    for entry in self._pending_cleanup.values():
+      shm = entry['shm']
+      if shm is not None:
+        owned.update((shm._mem_name, shm._mem_name + '_additional'))
     legacy = set(self._shm_legacy_names)
     legacy.update(name + '_additional' for name in self._shm_legacy_names)
     candidates, count, total_bytes, truncated, errors = [], 0, 0, False, 0
@@ -564,7 +581,14 @@ class ServingManager(Manager):
         len(self._shm_failures), max(0, self._shm_failure_count - len(self._shm_failures))),
       "SHM namespace: status={} scanned_limit=256 truncated={} stat_errors={} possible_leftovers={} sampled_bytes={} shown={}".format(
         status, truncated, errors, visible_count, visible_bytes, len(candidates)),
+      "SHM pending cleanup: workers={} shown={}".format(
+        len(self._pending_cleanup), min(16, len(self._pending_cleanup))),
     ]
+    for name, entry in itertools.islice(self._pending_cleanup.items(), 16):
+      shm = entry['shm']
+      lines.append("  Pending cleanup: server={} process={} comms={} segment={}".format(
+        name, entry['server'] is not None, len(entry['comms']),
+        None if shm is None else shm._mem_name))
     if candidates:
       lines.append("  Possible leftovers (ownership unproven): " + ', '.join(candidates))
     for failure in list(self._shm_failures)[-8:]:
@@ -589,6 +613,7 @@ class ServingManager(Manager):
       Allocation, exact child attachment kwargs, and effective transport.
       Only name collisions receive up to three fresh-identity attempts.
     """
+    self._retry_pending_cleanup(server_name)
     self._shm_legacy_names[server_name] = None
     self._shm_legacy_names.move_to_end(server_name)
     if len(self._shm_legacy_names) > 64:
@@ -626,7 +651,11 @@ class ServingManager(Manager):
         self._release_server_resources(server_name, None, (), shm)
         raise
       self._record_shm_failure(server_name, 'create', error_type, kwargs['mem_name'])
-      self._release_server_resources(server_name, None, (), shm)
+      cleanup_error = self._release_server_resources(server_name, None, (), shm)
+      if cleanup_error is not None:
+        # Do not allocate another generation or hide an owned failed cleanup
+        # behind pipe fallback. The pending entry is the retry authority.
+        raise cleanup_error
       if error_type != 'FileExistsError':
         break
     self.P("SHM unavailable for '{}'; using pipe before worker startup".format(server_name), color='error')
@@ -634,7 +663,7 @@ class ServingManager(Manager):
 
   def _release_server_resources(self, server_name, server, comm_engs, npy_shm,
                                 wait=False, terminate=False):
-    """Attempt independent cleanup steps and retain the first runtime error.
+    """Release independent resources, retaining every unfinished operation.
 
     Parameters
     ----------
@@ -654,9 +683,25 @@ class ServingManager(Manager):
     BaseException or None
       First failure for the caller to propagate, unless it is already handling
       an earlier error. Subsequent errors remain visible in bounded diagnostics.
+      Unfinished resources remain in ``_pending_cleanup`` outside inference
+      routing. Successful steps are removed individually so retries cannot
+      repeat an in-process model shutdown or a successfully closed endpoint.
     """
     first_error = None
     segment = None if npy_shm is None else npy_shm._mem_name
+    pending = self._pending_cleanup.setdefault(
+      server_name, dict(server=None, comms=(), shm=None))
+    # Register ownership before teardown. Nested startup cleanup may supply
+    # only endpoints after an allocation helper has retained its failed SHM.
+    # Merge by identity so that neither call can overwrite the other's handles.
+    if server is not None:
+      pending['server'] = server
+    comm_engs = tuple(comm for comm in comm_engs if comm is not None)
+    for comm in comm_engs:
+      if not any(comm is retained for retained in pending['comms']):
+        pending['comms'] += (comm,)
+    if npy_shm is not None:
+      pending['shm'] = npy_shm
     if server is not None:
       try:
         if server.inprocess:
@@ -671,18 +716,22 @@ class ServingManager(Manager):
               server.join(5)
               if server.is_alive():
                 raise RuntimeError('Serving process remains alive after kill')
+          if server.is_alive():
+            raise RuntimeError('Serving process cleanup incomplete')
       except BaseException as exc:
         first_error = exc
         self._record_shm_failure(server_name, 'process_cleanup', type(exc).__name__, segment)
+      else:
+        pending['server'] = None
     for comm in comm_engs:
-      if comm is None:
-        continue
       try:
         comm.close()
       except BaseException as exc:
         if first_error is None:
           first_error = exc
         self._record_shm_failure(server_name, 'comm_close', type(exc).__name__, segment)
+      else:
+        pending['comms'] = tuple(retained for retained in pending['comms'] if retained is not comm)
     if npy_shm is not None:
       self._shm_cleanup_attempts += 1
       previous_errors = len(npy_shm.cleanup_errors) + npy_shm.cleanup_errors_dropped
@@ -699,8 +748,41 @@ class ServingManager(Manager):
       current_errors = len(npy_shm.cleanup_errors) + npy_shm.cleanup_errors_dropped
       if current_errors > previous_errors and first_error is None:
         first_error = RuntimeError('Shared memory cleanup incomplete')
+      # The utility retains a handle/mapping when close or owned unlink fails.
+      # Historical errors alone do not mean resources remain after a retry.
+      if all(handle is None for handle in (
+          npy_shm._shm, npy_shm._shm_additional,
+          npy_shm._mmap_obj, npy_shm._mmap_obj_additional)):
+        pending['shm'] = None
+      elif first_error is None:
+        first_error = RuntimeError('Shared memory cleanup incomplete')
+    if pending['server'] is None and not pending['comms'] and pending['shm'] is None:
+      del self._pending_cleanup[server_name]
     return first_error
 
+  def _retry_pending_cleanup(self, server_name):
+    """Retry only manager-owned failed resources before shutdown or replacement.
+
+    Parameters
+    ----------
+    server_name : str
+      Physical worker identity. No namespace lookup or foreign unlink occurs.
+
+    Raises
+    ------
+    BaseException
+      First retry failure. The remaining ownership is retained for a later
+      attempt, and callers must not start a replacement generation.
+    """
+    pending = self._pending_cleanup.get(server_name)
+    if pending is None:
+      return
+    error = self._release_server_resources(
+      server_name, pending['server'], pending['comms'], pending['shm'],
+      wait=True, terminate=True)
+    if error is not None:
+      raise error
+
   def _server_wait_time(self, server_name):
     server_name = self.get_server_name(server_name)
     if server_name not in self._servers:
@@ -729,6 +811,7 @@ class ServingManager(Manager):
       Runtime exceptions propagate after all owned cleanup has been attempted.
     """
     upstream_config = upstream_config or {}
+    self._retry_pending_cleanup(server_name)
     if server_name in self._servers:
       self.P("Serving process '{}' already created".format(server_name), color='r')
       return server_name
@@ -1097,7 +1180,8 @@ class ServingManager(Manager):
     wait, terminate : bool
       Preserve the existing bounded process stop policy. Communication and SHM
       cleanup still run when process teardown raises, and the first error is
-      propagated after retiring the registry entries.
+      propagated after retiring the routing entries. Unfinished resources stay
+      manager-owned in the separate cleanup registry for shutdown/retry.
     """
     server_name = self.get_server_name(server_name)
     entry = self._servers.get(server_name)
@@ -1434,6 +1518,8 @@ class ServingManager(Manager):
       cleanup failures are retained separately in the bounded SHM summary.
     """
     server_name = self.get_server_name(server_name)
+    if server_name in self._pending_cleanup:
+      self._retry_pending_cleanup(server_name)
     server = self._get_server(server_name)
     if server is None:
       return
@@ -1475,6 +1561,10 @@ class ServingManager(Manager):
       self.P("GPU0 free mem before shutdown: {} GB".format(gpu_info1[0]['FREE_MEM']), color='g')
     server_names = [name for name, entry in self._servers.items()
                     if entry.get(SMConst.COVERED_BY) is None]
+    previously_pending = set(self._pending_cleanup)
+    # Dedupe against scheduled physical workers, never covered aliases: an
+    # alias with the same logical name cannot suppress an old owner's cleanup.
+    server_names.extend(name for name in previously_pending if name not in server_names)
     first_error = None
     for server_name in server_names:
       try:
@@ -1484,6 +1574,17 @@ class ServingManager(Manager):
         # failed STOP or close must not strand the next worker's allocation.
         if first_error is None:
           first_error = exc
+    # The launcher may exit immediately after this final shutdown. Give newly
+    # interrupted/failed cleanup one bounded retry here, rather than assuming
+    # another stop_all call. Never loop until success on persistent failures.
+    for server_name in list(self._pending_cleanup):
+      if server_name in previously_pending:
+        continue
+      try:
+        self._retry_pending_cleanup(server_name)
+      except BaseException as exc:
+        if first_error is None:
+          first_error = exc
     if first_error is not None:
       raise first_error
     gpu_info2 = self.log.gpu_info()

{naeural_core-7.7.330 → naeural_core-7.7.332}/naeural_core/utils/plugins_base/bc_wrapper.py

@@ -75,6 +75,7 @@ class BCWrapper:
     str_signature: str = None, 
     sender_address: str = None,
     return_full_info : bool = True,
+    log_hash_sign_fails: bool = True,
   ) -> bool:
     """
     Verifies a signature using the public key of the signer
@@ -92,7 +93,11 @@ class BCWrapper:
         Default `None` will be taken from the data
         
     return_full_info: bool, optional
-        whether to return the full verification info. Default `True`      
+        whether to return the full verification info. Default `True`
+
+    log_hash_sign_fails: bool, optional
+        Whether the engine may log signature failures. Default `True`. Pass
+        `False` when the signed message can contain secret values.
     
 
     Returns
@@ -105,6 +110,7 @@ class BCWrapper:
       signature=str_signature, 
       sender_address=sender_address,
       return_full_info=return_full_info,
+      log_hash_sign_fails=log_hash_sign_fails,
     )
 
   def encrypt_str(self, str_data : str, str_recipient : str):
@@ -670,6 +676,31 @@ class BCWrapper:
     return self.__bc.get_network_data(network)
   
   
+  def reconcile_dauth_job_secrets(self, payload, network=None, request_timeout=(10, 60)):
+    """
+    Reconcile job secrets on dAuth through the SDK client.
+
+    Parameters
+    ----------
+    payload : dict
+        Job ID, expected pipeline CID, prepared configs and optional scale-up
+        identity aliases. The SDK signs the request with this node's identity.
+    network : str, optional
+        Network whose dAuth endpoint and signer registry are used. Defaults to
+        the underlying blockchain engine's network.
+    request_timeout : tuple or float, optional
+        HTTP timeout forwarded unchanged to the SDK, default ``(10, 60)``.
+
+    Returns
+    -------
+    dict
+        SDK-authenticated result containing redacted configs and their CID.
+        Secret bundles are not returned. SDK errors propagate unchanged.
+    """
+    return self.__bc.reconcile_dauth_job_secrets(
+      payload=payload, network=network, request_timeout=request_timeout,
+    )
+
   def get_node_license_info(
     self, 
     node_address: str = None, 

naeural_core-7.7.332/naeural_core/utils/plugins_base/test_bc_wrapper_dauth.py

@@ -0,0 +1,48 @@
+"""Cover plugin-facing dAuth calls through the real blockchain wrapper."""
+import unittest
+from types import SimpleNamespace
+from unittest.mock import Mock
+
+from naeural_core.utils.plugins_base.bc_wrapper import BCWrapper
+
+
+class BCWrapperDauthTests(unittest.TestCase):
+  def test_verify_preserves_defaults_and_forwards_sensitive_logging_option(self):
+    """Keep legacy callers working and allow suppressing secret-bearing logs."""
+    engine = SimpleNamespace(verify=Mock(return_value=SimpleNamespace(valid=True)))
+    wrapper = BCWrapper(engine, owner=None)
+    body = {"EE_SENDER": "requester"}
+    for log_hash_sign_fails in (True, False):
+      with self.subTest(log_hash_sign_fails=log_hash_sign_fails):
+        kwargs = {} if log_hash_sign_fails else {"log_hash_sign_fails": False}
+        result = wrapper.verify(body, str_signature="signature", sender_address="sender", **kwargs)
+        self.assertTrue(result.valid)
+        engine.verify.assert_called_with(
+          dct_data=body, signature="signature", sender_address="sender",
+          return_full_info=True, log_hash_sign_fails=log_hash_sign_fails,
+        )
+
+    wrapper.verify(body, "signature", "sender", False)
+    engine.verify.assert_called_with(
+      dct_data=body, signature="signature", sender_address="sender",
+      return_full_info=False, log_hash_sign_fails=True,
+    )
+
+  def test_reconciliation_forwards_payload_network_and_timeout_to_sdk(self):
+    """Expose the SDK client through the object plugins actually receive."""
+    result = {"status": "success", "pipeline_cid": "cid"}
+    engine = SimpleNamespace(reconcile_dauth_job_secrets=Mock(return_value=result))
+    wrapper = BCWrapper(engine, owner=None)
+    payload = {"job_id": "74", "pipeline_configs": {"worker": {}}}
+    self.assertIs(wrapper.reconcile_dauth_job_secrets(payload), result)
+    engine.reconcile_dauth_job_secrets.assert_called_once_with(
+      payload=payload, network=None, request_timeout=(10, 60),
+    )
+    self.assertIs(wrapper.reconcile_dauth_job_secrets(payload, network="testnet", request_timeout=(2, 5)), result)
+    engine.reconcile_dauth_job_secrets.assert_called_with(
+      payload=payload, network="testnet", request_timeout=(2, 5),
+    )
+
+
+if __name__ == "__main__":
+  unittest.main()