naeural-core 7.7.332__tar.gz → 7.7.334__tar.gz

{naeural_core-7.7.332 → naeural_core-7.7.334}/AGENTS.md

@@ -399,3 +399,8 @@ Latest:
 - `2026-10-13`: Covered aliases copy `COMM`, `SERVER` and `PID`, not `SHM`; their payloads can use pipe serialization while direct requests use the physical worker's segment. Configuration diagnostics must resolve the physical registry entry to avoid relogging on each alias switch. `business/test_framework/shm_testbed/run_e2e.py` exercises this behavior with a real isolated Edge node, generated frames and a deterministic CPU Torch model; broker traffic stays in a loopback-only namespace and cleanup checks preserve pre-existing service identities.
 
 - `2026-10-19`: Plugins receive `BCWrapper`, not the raw SDK engine. New plugin-facing blockchain methods and verification options must be forwarded explicitly in `naeural_core/utils/plugins_base/bc_wrapper.py`; dAuth reconciliation requires SDK 3.5.58 and must be verified through this wrapper. Run `python -m unittest naeural_core.utils.plugins_base.test_bc_wrapper_dauth` for these contracts.
+- `2026-10-14`: Deeploy node lifecycle is registered by ConfigManager before plugin construction and published independently by `business/deployment_lifecycle.py`. Authenticated scoped stop/retire commands persist abort tombstones; external runners require verified cleanup or `cleanup_deployment_instance` before reporting stopped. CStore deployment namespaces verify signed sequence/field ownership, keep eight recent slots, and protect dAuth secret namespaces/irreversible revocation across repair. Strict cache fsync failures prevent replica acknowledgments. See `docs/DEEPLOY_NODE_LIFECYCLE.md`, `docs/CSTORE.md`; run lifecycle, HSync, plugin-loop, runtime-secret and response tests for changes. This is local attempt protection, not distributed coordinator consensus or failover.
+- `2026-10-15`: Deeploy journals the resolved runner class's workload ownership and interpreter identity before construction. Known native workloads can finish an aborted stop after the owning process exits; unknown, external, and same-process cleanup failures remain unresolved without cleanup evidence. External workload ownership must be declared on the class, including constructor side effects. Retirement preserves this private journal evidence.
+- `2026-10-15`: Existing unversioned pipelines retain dAuth secret resolution after restart. Core rejects missing secret revisions on tracked/nonzero-revision pipelines; SDK omission is only for existing unversioned metadata, and dAuth enforces current CID/runner authorization. The first update captures immutable target and restoration secret revisions. Cross-stack restart coverage lives in Edge's `test_unversioned_secret_resolution.py`.
+
+- `2026-10-20`: Lifecycle reconciliation through SDK 3.5.59 leaves bundles on dAuth. CStore accepts registered dAuth signatures on current job-secret bundles for a registered coordinator, preserves coordinator-only public pointer signatures, and ignores node/EVM signature metadata only for identical-generation semantic retries. Run protected CStore lifecycle tests and Edge cross-stack custody tests.

{naeural_core-7.7.332 → naeural_core-7.7.334}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: naeural_core
-Version: 7.7.332
+Version: 7.7.334
 Summary: Ratio1 Core is the backbone of the Ratio1 Edge Protocol.
 Project-URL: Homepage, https://github.com/Ratio1/naeural_core
 Project-URL: Bug Tracker, https://github.com/Ratio1/naeural_core/issues
@@ -30,7 +30,7 @@ Requires-Dist: pydantic
 Requires-Dist: pymssql
 Requires-Dist: pynvml>=11.4
 Requires-Dist: pyyaml>=6.0
-Requires-Dist: ratio1>=3.5.58
+Requires-Dist: ratio1>=3.5.59
 Requires-Dist: scikit-image
 Requires-Dist: scikit-learn
 Requires-Dist: seaborn

{naeural_core-7.7.332 → naeural_core-7.7.334}/docs/CSTORE.md

@@ -186,3 +186,61 @@ Larger follow-ups:
 - delete/tombstone reconciliation when exact mirror behavior is required
 - automatic app bootstrap wrappers if the platform later decides to own startup
   hydration instead of leaving it to apps
+
+## Deployment lifecycle persistence and protected namespaces
+
+CStore writes now require the strict cache path: atomic pickle/metadata replace,
+file and directory fsync, and propagated failures before acknowledgment. A
+same-value retransmission rechecks persistence/replication. Confirmation counts
+use distinct authenticated transport senders from the actual target set. These
+are durable replica acknowledgments, not consensus, CAS, or distributed locking.
+
+The writer's local copy is persisted before broadcast. Self is excluded from the
+remote target set. A positive configured `MIN_CONFIRMATIONS` takes precedence;
+otherwise the threshold is `floor(remote target count / 2) + 1`. With no remote
+targets a local-origin write returns False before persistence; there is no
+local-only successful acceptance mode. A configured minimum above the reachable
+target count times out. Explicit targets are exclusive only when
+`include_default_peers=False`. The same rules apply after a same-value retry.
+These acknowledgments preserve CStore records across process restarts with the
+same data volume; they do not replicate R1FS content or elect a replacement
+coordinator. Generic `value=None` deletes retain the historical asynchronous
+confirmation behavior; lifecycle secret erasure uses a retained non-null
+revocation marker and redacted tombstone, which receive normal confirmations.
+
+`DEEPLOY_OPERATIONS` and `DEEPLOY_NODE_OPERATIONS` contain signed sequence records.
+Writes and HSync reject invalid signatures, mismatched field identities, and
+lower per-field sequences. Record ownership is derived from the signed principal,
+not an untrusted snapshot wrapper. Operation signers must be registered oracles
+or dAuth registry members. Current fields and eight sequence-modulo slots provide
+bounded recent history; generic HSync namespaces retain their existing semantics.
+
+Both `DAUTH_JOB_SECRETS` and `DAUTH_JOB_SECRET_REVISIONS` are protected. Remote
+writes require an authenticated registered oracle/dAuth sender and a dAuth
+recipient. Local secret writes may originate on an oracle, but network targets
+must be dAuth registry participants. Protected HSync request, response, export,
+and merge require dAuth registry membership, failing closed on registry lookup
+failure. HSync never exports token-bearing records because that API supplies no
+token capability.
+
+Revision fields ending in `|revoked` are immutable once populated. Identical
+retransmission is allowed; deletion/change is rejected. Revocation wins over
+stale payload snapshots. Only deletion or the exact redacted
+`{job_id, revision, revoked: true}` tombstone matching the revocation marker is
+accepted for a revoked payload field. No plaintext secret can be restored by
+late direct writes or merge repair.
+
+Canonical `DEEPLOY_DEPLOYED_JOBS` pointers are signed envelopes containing
+`job_id`, `revision`, `operation_id`, `coordinator`, `pipeline_cid`, and `deleted`.
+Current `DAUTH_JOB_SECRETS` bundles carry equivalent signed writer metadata and
+`deployment_revision`. A registered dAuth custodian may sign these bundles for
+the registered protocol-oracle coordinator after verifying its lifecycle request;
+public pointers must still be signed by that coordinator. Signature-only retry
+differences include both node and EVM signature metadata. These generations increase across successful deployment,
+restoration and deletion commits; restored runtime configuration may retain an
+older content revision. Receivers verify registry signer and job-field binding,
+reject lower generations and divergent equal-generation payloads, and ignore
+signature-only differences for idempotent retries. Once versioned, a field cannot
+be replaced by an unversioned write or deletion. A signed higher generation
+allows explicit reuse after deletion. Both direct writes and HSync enforce the
+same rules, preventing stale cached pipeline discovery from reviving deletion.

naeural_core-7.7.334/docs/DEEPLOY_NODE_LIFECYCLE.md

@@ -0,0 +1,112 @@
+# Node deployment lifecycle
+
+The full cross-repository reference is
+[the Edge deployment lifecycle specification](../../edge_node/docs/deeploy-deployment-lifecycle-spec.md).
+This note describes the Core wire/runtime contract.
+
+## Commands
+
+Use authenticated `UPDATE_CONFIG` with stable pipeline `NAME` and top-level
+`DEEPLOY_OPERATION`: `operation_id`, `attempt`, `revision`, `coordinator`,
+`peers`, `action`, and `job_id`. The transport sender must equal `coordinator`.
+A start carries the complete frozen canonical pipeline configuration. Attempts
+are `deploy`, `retire`, or `restore`. Restore has its own attempt identity even
+when its configuration revision is older.
+
+`action: stop` persists an abort before removing only a matching operation and
+attempt configuration. A delayed start of that attempt is rejected after restart.
+It never removes another operation or restoration attempt by pipeline name alone.
+
+Before an update, `action: retire`, `attempt: retire` binds the old placement to
+`revision` and `expected_previous_operation_id` (null for an untracked revision
+zero). Current canonical `DEEPLOY_SPECS.job_id`, `revision` (default zero), and
+operation identity must match. Mismatch reports `CONFIGURATION_REJECTED` and
+leaves the current pipeline untouched. A successful retirement records the prior
+attempt's abort, adopts existing runtime objects for cleanup evidence, and
+removes the canonical configuration. No new revision may start until the
+controller observes retirement completion on every previous target. If the
+canonical configuration is already absent, any active matching journal record
+still supplies workload identities for cleanup; absence of a configuration alone
+cannot acknowledge a journaled workload as stopped.
+
+## Durable observations and cleanup
+
+`DeploymentLifecycle` is owned by ConfigManager and publishes on an independent
+node worker. It registers accepted instance identities before BusinessManager
+loads code or constructs plugins. The journal is
+`<data folder>/deeploy_node_operations.json`, atomically replaced with file and
+directory fsync. Corrupt journals fail closed. No plaintext configuration or
+secret bundle is stored there.
+
+BusinessManager observes load/constructor failures. The plugin loop records
+initialization/execution/cleanup failures without copying exception messages
+into the public channel. `report_deployment_status(status, error_code=None)`
+accepts the fixed vocabulary in `deployment_lifecycle.py`; readiness still uses
+`is_plugin_ready()` and semaphore semantics are unchanged.
+
+A dead plugin thread is insufficient for external workload cleanup. Runners
+owning external workloads must declare `_deployment_external_workload = True` on the class
+and call `set_deployment_stopped()` only after verifying actual shutdown. A
+runner can implement static `cleanup_deployment_instance(app_id, instance_id)`
+to reap its deterministic scoped workload after restart; it must use a bounded
+timeout, preserve persistent volumes, and return True only after confirming
+absence. Core invokes it without constructing or starting the plugin, while
+holding the local admission lock. Superseded attempts cannot reap a newer
+workload with the same name. Without such evidence cleanup remains unresolved.
+
+Before construction, the journal records the resolved class's native/external
+classification and the owning interpreter's process identity. A native instance
+with no runtime object can be confirmed stopped after that process exits. A new
+journal reader in the same process is insufficient. Unknown classifications,
+same-process constructor/cleanup failures, and external workloads still require
+cleanup evidence. Retirement preserves this classification and process identity;
+both fields remain private to the node journal. Custom code that owns external
+workloads must honor the runner declaration and cleanup contract above.
+
+The signing node writes HSET `DEEPLOY_NODE_OPERATIONS`, fields:
+
+- `<operation_id>|<attempt>|<node_addr>`: current snapshot.
+- `<operation_id>|<attempt>|<node_addr>|<sequence modulo 8>`: bounded recent slots.
+
+Snapshots contain operation/attempt/revision/coordinator, node/app identity,
+status, per-instance signature/ID/status/safe error, monotonically increasing
+sequence, and Unix `updated_at`. They are signed with
+`sign(record, add_data=True, use_digest=True)`. Ready observations continue at
+approximately ten-second intervals; stopped/failed unchanged observations stop
+after acknowledged publication. Sequence is persisted before publication.
+Consumers must verify signatures, expected node/operation identity, and freshness.
+
+These are bounded recent snapshots, not a permanent event log. Tombstones and
+operation journal records are retained; operator retention must not discard aborts
+while delayed commands can still arrive.
+
+## Revision secrets
+
+`DEEPLOY_SPECS.secret_revision` participates in runtime secret cache identity and
+is forwarded as `revision` to the SDK's signed dAuth request. An in-flight request
+for another revision cannot satisfy a restored configuration. Definitive secret
+validation/rejection errors publish `SECRET_RESOLUTION_FAILED`; transport,
+registry, and replica synchronization failures remain retryable.
+
+Existing persisted pipelines without lifecycle metadata may omit
+`secret_revision`, including after restart with an empty runtime cache. The SDK
+sends an unversioned request; dAuth authorizes it only against the existing raw
+CID, current placement, and matching current secret bundle. A tracked pipeline,
+nonzero revision, or explicit null secret revision cannot use this path. New
+deployments and both target/restoration plans of the first update always carry
+immutable secret revisions; they never fall back to unversioned secrets.
+
+## Limits
+
+The receiving oracle owns coordination. There is no distributed election,
+failover, or cross-oracle admission lock. Node tombstones and local active-attempt
+tracking prevent scoped replay and cleanup of superseded attempts; they do not
+provide global concurrent mutation serialization.
+
+Tracked plugin construction is serialized with lifecycle admission and reporting;
+the lock is acquired after prior-instance shutdown and is never held across that
+thread join. Stop acknowledgment cannot race a constructor creating an external
+workload. Runner `get_deployment_runtime_artifacts()` must return cached values
+without I/O. The node journals and reports only `image_id` (full sha256 Docker
+identity) and optional `git_commit` (40 hexadecimal characters), including after
+retirement/restart. Missing bindings remain absent so restoration can fail closed.

{naeural_core-7.7.332 → naeural_core-7.7.334}/naeural_core/business/base/base_plugin_biz_loop.py

@@ -441,6 +441,7 @@ class _BasePluginLoopMixin(object):
       plugin_loop_initialized = True
     except Exception as exc:
       info = traceback.format_exc()
+      self._deployment_error_code = "PLUGIN_INIT_FAILED"
       self.P(f"CRITICAL ERROR in plugin loop initialization: {exc}. Resuming to plugin cleanup:\n{info}", color='r')
     
     if plugin_loop_initialized:
@@ -526,6 +527,7 @@ class _BasePluginLoopMixin(object):
             sleep(sleep_time)
           self.end_timer('loop_sleep')
         except:
+          self._deployment_error_code = "PLUGIN_EXECUTION_FAILED"
           self._plugin_loop_in_exec = False
           SLEEP_ON_LOOP_ERROR = 10
           msg = "CRITICAL ERROR in business plugin loop (sleep {}) on: {}".format(
@@ -544,7 +546,9 @@ class _BasePluginLoopMixin(object):
 
     try:
       self.__cleanup()
+      self._deployment_cleanup_complete = True
     except Exception as exc:
+      self._deployment_error_code = "PLUGIN_STOP_FAILED"
       info = traceback.format_exc()
       self.P(f"CRITICAL ERROR in cleanup: {exc}:\n{info}", color='r')
 

{naeural_core-7.7.332 → naeural_core-7.7.334}/naeural_core/business/business_manager.py

@@ -14,6 +14,7 @@ from copy import deepcopy
 from naeural_core import constants as ct
 from naeural_core import Logger
 from naeural_core.manager import Manager
+from naeural_core.business.deployment_lifecycle import OPERATION_KEY, SERVICE_KEY
 
 from collections import deque, defaultdict
 
@@ -112,6 +113,14 @@ class BusinessManager(Manager):
     super().startup()
     self._dct_current_instances = self._dct_subalterns # this allows usage of `self.get_subaltern(instance_hash)`
     self._initialize_admin_async_dispatch()
+    lifecycle = self.shmem.get(SERVICE_KEY)
+    if lifecycle is not None:
+      lifecycle.start(
+        lambda: self.plugins_shmem.get("__chain_storage_set"),
+        self.owner.blockchain_manager.sign,
+        self.P,
+        cleanup=self._cleanup_deployment_instance,
+      )
     if self.config_data.get('PLUGINS_DEBUG_LOAD_TIMINGS', True):
       self.P(
         "Plugin timing env: python={} dont_write_bytecode={} env_PYTHONDONTWRITEBYTECODE={}".format(
@@ -123,6 +132,32 @@ class BusinessManager(Manager):
       )
     return
 
+  def _cleanup_deployment_instance(self, app_id, signature, instance_id):
+    """Verify external workload cleanup after restart without starting a plugin.
+
+    Parameters
+    ----------
+    app_id : str
+        Pipeline whose aborted attempt still has a cleanup obligation.
+    signature : str
+        Plugin class owning the workload-specific cleanup implementation.
+    instance_id : str
+        Instance identity used to derive a scoped external workload name.
+
+    Returns
+    -------
+    bool
+        True only when the runner positively verifies workload absence.
+    """
+    _, _, plugin_class, _ = self._get_module_name_and_class(
+      locations=ct.PLUGIN_SEARCH.LOC_BIZ_PLUGINS, name=signature,
+      suffix=ct.PLUGIN_SEARCH.SUFFIX_BIZ_PLUGINS, verbose=0,
+      safety_check=True, safe_locations=ct.PLUGIN_SEARCH.SAFE_BIZ_PLUGINS,
+      safe_imports=ct.PLUGIN_SEARCH.SAFE_BIZ_IMPORTS,
+    )
+    cleanup = getattr(plugin_class, "cleanup_deployment_instance", None)
+    return cleanup is not None and cleanup(app_id, instance_id) is True
+
   @property
   def current_instances(self):
     """Return currently active business plugin instances.
@@ -503,7 +538,11 @@ class BusinessManager(Manager):
     """
     self._ensure_admin_async_dispatch_health()
     self.owner.set_loop_stage('2.bm.refresh.entry_update_streams')
-    self._dct_config_streams = dct_config_streams
+    lifecycle = self.shmem.get(SERVICE_KEY)
+    self._dct_config_streams = {
+      name: config for name, config in dct_config_streams.items()
+      if lifecycle is None or lifecycle.allowed(config)
+    }
     self.owner.set_loop_stage('2.bm.refresh._check_instances')
     current_instances = self._check_instances()
     self._refresh_admin_instance_hashes(current_instances)
@@ -734,12 +773,31 @@ class BusinessManager(Manager):
         start_thread_s = 0.0
         update_config_s = 0.0
       is_new_instance = False
+      construction_lock = None
+      operation = None
       try:
         obj_identification = (stream_name, signature, instance_id)
         instance_hash = self.log.hash_object(obj_identification, size=5)
         self._dct_hash_mappings[instance_hash] = obj_identification
         current_instances.append(instance_hash)
+        lifecycle = self.shmem.get(SERVICE_KEY)
+        pipeline = self._dct_config_streams[stream_name]
+        operation = pipeline.get(OPERATION_KEY)
+        existing = self._dct_current_instances.get(instance_hash)
+        if operation and existing is not None and getattr(existing, "_deployment_operation_key", None) != lifecycle.key(operation):
+          # A revision change gets a fresh plugin lifecycle; old readiness must
+          # not confirm the new revision while config update is still pending.
+          if not self._stop_plugin(instance_hash, require_confirmation=True):
+            continue
         if instance_hash not in self._dct_current_instances:
+          if operation:
+            # Keep stop evidence from racing constructor side effects. This
+            # lock starts after replacement shutdown, never across thread join.
+            construction_lock = lifecycle.lock
+            construction_lock.acquire()
+            if not lifecycle.allowed(pipeline):
+              continue
+            lifecycle.attach(pipeline, signature, instance_id, None)
           is_new_instance = True
           self._dct_instance_hash_log[instance_hash] = {
             ct.PAYLOAD_DATA.INITIATOR_ID  : initiator_id,
@@ -776,6 +834,8 @@ class BusinessManager(Manager):
           
           if _cls_def is None:
             self._dct_current_instances[instance_hash] = None
+            if operation:
+              lifecycle.fail(pipeline, signature, instance_id, "PLUGIN_LOAD_FAILED")
             msg = "Error loading business plugin <{}:{}> - No code/script defined.".format(signature, instance_id)
             self.P(msg + " on stream {}".format(stream_name), color='r')
             self._create_notification(
@@ -789,6 +849,11 @@ class BusinessManager(Manager):
             continue
           #endif
 
+          if operation:
+            # Persist the resolved class's workload ownership before constructor
+            # side effects. Only a later process may prove native threads gone.
+            lifecycle.attach(pipeline, signature, instance_id, None, plugin_class=_cls_def)
+
           self.comm_shared_memory['payloads'][instance_hash] = deque(maxlen=1000)
           self.comm_shared_memory['commands'][instance_hash] = deque(maxlen=1000)
 
@@ -844,6 +909,8 @@ class BusinessManager(Manager):
             self.set_loop_stage('2.bm.refresh.new_instance_done: {}:{}:{}'.format(stream_name, signature, instance_id))
           except Exception as exc:
             plugin = None
+            if operation:
+              lifecycle.fail(pipeline, signature, instance_id, "PLUGIN_INIT_FAILED")
             trace = traceback.format_exc()
             msg = "Plugin init FAILED for business plugin {} instance {}".format(signature, instance_id)
             info = str(exc)
@@ -861,6 +928,9 @@ class BusinessManager(Manager):
             )
           #end try-except
 
+          if operation and plugin is not None:
+            plugin._deployment_operation_key = lifecycle.key(operation)
+            lifecycle.attach(pipeline, signature, instance_id, plugin)
           self._dct_current_instances[instance_hash] = plugin
           self.__maybe_register_special_plugin_instance_hash(instance_hash=instance_hash, signature=signature)
 
@@ -894,7 +964,13 @@ class BusinessManager(Manager):
             self.set_loop_stage('2.bm.refresh.maybe_update_instance_config.DONE: {}:{}:{}'.format(stream_name, signature, instance_id))
           #endif
         #endif
+      except Exception:
+        if operation:
+          lifecycle.fail(pipeline, signature, instance_id, "PLUGIN_INIT_FAILED")
+        raise
       finally:
+        if construction_lock is not None:
+          construction_lock.release()
         if is_new_instance and debug_load_timings:
           iter_total_s = perf_counter() - iter_start
           total_elapsed_s = perf_counter() - total_start
@@ -921,14 +997,35 @@ class BusinessManager(Manager):
 
     return current_instances
 
-  def _stop_plugin(self, instance_hash):
+  def _stop_plugin(self, instance_hash, require_confirmation=False):
+    """Stop a plugin, retaining it when deployment shutdown is unconfirmed.
+
+    Parameters
+    ----------
+    instance_hash : str
+        Runtime instance identity.
+    require_confirmation : bool
+        Also require shutdown evidence for the previous untracked revision.
+
+    Returns
+    -------
+    bool
+        True only when this instance can be removed from runtime tracking.
+    """
     plugin = self._dct_current_instances[instance_hash]
 
     if plugin is None:
-      return
+      self._dct_current_instances.pop(instance_hash, None)
+      return True
 
     if self._run_on_threads:
       plugin.stop_thread()
+      if require_confirmation or getattr(plugin, "_deployment_operation_key", None):
+        # A join timeout or failed workload close is not stopped evidence.
+        if plugin.thread.is_alive():
+          return False
+        if getattr(plugin, "_deployment_external_workload", False) and not getattr(plugin, "_deployment_stopped", False):
+          return False
 
     plugin = self._dct_current_instances.pop(instance_hash)
     self._dct_stop_timings.pop(instance_hash, None)
@@ -936,7 +1033,7 @@ class BusinessManager(Manager):
   
     sleep(0.1)
     gc.collect()
-    return
+    return True
 
   def maybe_stop_finished_plugins(self):
     """
@@ -970,7 +1067,7 @@ class BusinessManager(Manager):
         del self._graceful_stop_instances[instance_hash]
       return
 
-    if plugin.done_loop:
+    if plugin.done_loop and not getattr(plugin, "_deployment_operation_key", None):
       self.P("Received STOP for already stopped {}:{}".format(instance_hash, plugin), color='r')
       # TODO: maybe add a sleep so that we do not spam this thing
       self._graceful_stop_instances.pop(instance_hash, 0)
@@ -1008,6 +1105,9 @@ class BusinessManager(Manager):
     for _hash in keys:
       self._send_stop_signal_to_plugin(instance_hash=_hash, forced=True)
     self.P("Done stopping all business plugins.", color='y')
+    lifecycle = self.shmem.get(SERVICE_KEY)
+    if lifecycle is not None:
+      lifecycle.close()
     return
 
   def _deallocate_unused_instances(self, current_instances):