koi-net 1.0.0b6__tar.gz → 1.0.0b7__tar.gz

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: koi-net
-Version: 1.0.0b6
+Version: 1.0.0b7
 Summary: Implementation of KOI-net protocol in Python
 Project-URL: Homepage, https://github.com/BlockScience/koi-net/
 Author-email: Luke Miller <luke@block.science>
@@ -150,7 +150,7 @@ node = NodeInterface(
 
 ## Knowledge Processing
 
-Next we'll set up the knowledge processing flow for our node. This is where most of the node's logic and behavior will come into play. For partial nodes this will be an event loop, and for full nodes we will use webhooks. Make sure to call `node.initialize()` and `node.finalize()` at the beginning and end of your node's life cycle.
+Next we'll set up the knowledge processing flow for our node. This is where most of the node's logic and behavior will come into play. For partial nodes this will be an event loop, and for full nodes we will use webhooks. Make sure to call `node.start()` and `node.stop()` at the beginning and end of your node's life cycle.
 
 ### Partial Node
 Make sure to set `source=KnowledgeSource.External`, this indicates to the knowledge processing pipeline that the incoming knowledge was received from an external source. Where the knowledge is sourced from will impact decisions in the node's knowledge handlers.
@@ -159,7 +159,7 @@ import time
 from koi_net.processor.knowledge_object import KnowledgeSource
 
 if __name__ == "__main__":
-    node.initialize()
+    node.start()
 
     try:
         while True:
@@ -170,20 +170,20 @@ if __name__ == "__main__":
             time.sleep(5)
             
     finally:
-        node.finalize()
+        node.stop()
 ```
 
 ### Full Node
-Setting up a full node is slightly more complex as we'll need a webserver. For this example, we'll use FastAPI and uvicorn. First we need to setup the "lifespan" of the server, to initialize and finalize the node before and after execution, as well as the FastAPI app which will be our web server.
+Setting up a full node is slightly more complex as we'll need a webserver. For this example, we'll use FastAPI and uvicorn. First we need to setup the "lifespan" of the server, to start and stop the node before and after execution, as well as the FastAPI app which will be our web server.
 ```python
 from contextlib import asynccontextmanager
 from fastapi import FastAPI
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
-    node.initialize()
+    node.start()
     yield
-    node.finalize()
+    node.stop()
 
 
 app = FastAPI(lifespan=lifespan, root_path="/koi-net")
@@ -265,12 +265,16 @@ class NodeInterface:
     network: NetworkInterface
     processor: ProcessorInterface
     first_contact: str
+    use_kobj_processor_thread: bool
 
     def __init__(
         self, 
         name: str,
         profile: NodeProfile,
         identity_file_path: str = "identity.json",
+        event_queues_file_path: str = "event_queues.json",
+        cache_directory_path: str = "rid_cache",
+        use_kobj_processor_thread: bool = False,
         first_contact: str | None = None,
         handlers: list[KnowledgeHandler] | None = None,
         cache: Cache | None = None,
@@ -278,8 +282,8 @@ class NodeInterface:
         processor: ProcessorInterface | None = None
     ): ...
 
-    def initialize(self): ...
-    def finalize(self): ...
+    def start(self): ...
+    def stop(self): ...
 ```
 As you can see, only a name and profile are required. The other fields allow for additional customization if needed.
 
@@ -313,7 +317,6 @@ class NetworkInterface:
     
     def flush_poll_queue(self, node: KoiNetNode) -> list[Event]: ...
     def flush_webhook_queue(self, node: RID): ...
-    def flush_all_webhook_queues(self): ...
 
     def fetch_remote_bundle(self, rid: RID): ...
     def fetch_remote_manifest(self, rid: RID): ...
@@ -432,11 +435,14 @@ def poll_events(req: PollEvents) -> EventsPayload:
 The `ProcessorInterface` class provides access to a node's internal knowledge processing pipeline.
 ```python
 class ProcessorInterface:
+    worker_thread: threading.Thread | None = None
+
     def __init__(
         self, 
         cache: Cache, 
         network: NetworkInterface,
         identity: NodeIdentity,
+        use_kobj_processor_thread: bool,
         default_handlers: list[KnowledgeHandler] = []
     ): ...
 

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/README.md

@@ -108,7 +108,7 @@ node = NodeInterface(
 
 ## Knowledge Processing
 
-Next we'll set up the knowledge processing flow for our node. This is where most of the node's logic and behavior will come into play. For partial nodes this will be an event loop, and for full nodes we will use webhooks. Make sure to call `node.initialize()` and `node.finalize()` at the beginning and end of your node's life cycle.
+Next we'll set up the knowledge processing flow for our node. This is where most of the node's logic and behavior will come into play. For partial nodes this will be an event loop, and for full nodes we will use webhooks. Make sure to call `node.start()` and `node.stop()` at the beginning and end of your node's life cycle.
 
 ### Partial Node
 Make sure to set `source=KnowledgeSource.External`, this indicates to the knowledge processing pipeline that the incoming knowledge was received from an external source. Where the knowledge is sourced from will impact decisions in the node's knowledge handlers.
@@ -117,7 +117,7 @@ import time
 from koi_net.processor.knowledge_object import KnowledgeSource
 
 if __name__ == "__main__":
-    node.initialize()
+    node.start()
 
     try:
         while True:
@@ -128,20 +128,20 @@ if __name__ == "__main__":
             time.sleep(5)
             
     finally:
-        node.finalize()
+        node.stop()
 ```
 
 ### Full Node
-Setting up a full node is slightly more complex as we'll need a webserver. For this example, we'll use FastAPI and uvicorn. First we need to setup the "lifespan" of the server, to initialize and finalize the node before and after execution, as well as the FastAPI app which will be our web server.
+Setting up a full node is slightly more complex as we'll need a webserver. For this example, we'll use FastAPI and uvicorn. First we need to setup the "lifespan" of the server, to start and stop the node before and after execution, as well as the FastAPI app which will be our web server.
 ```python
 from contextlib import asynccontextmanager
 from fastapi import FastAPI
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
-    node.initialize()
+    node.start()
     yield
-    node.finalize()
+    node.stop()
 
 
 app = FastAPI(lifespan=lifespan, root_path="/koi-net")
@@ -223,12 +223,16 @@ class NodeInterface:
     network: NetworkInterface
     processor: ProcessorInterface
     first_contact: str
+    use_kobj_processor_thread: bool
 
     def __init__(
         self, 
         name: str,
         profile: NodeProfile,
         identity_file_path: str = "identity.json",
+        event_queues_file_path: str = "event_queues.json",
+        cache_directory_path: str = "rid_cache",
+        use_kobj_processor_thread: bool = False,
         first_contact: str | None = None,
         handlers: list[KnowledgeHandler] | None = None,
         cache: Cache | None = None,
@@ -236,8 +240,8 @@ class NodeInterface:
         processor: ProcessorInterface | None = None
     ): ...
 
-    def initialize(self): ...
-    def finalize(self): ...
+    def start(self): ...
+    def stop(self): ...
 ```
 As you can see, only a name and profile are required. The other fields allow for additional customization if needed.
 
@@ -271,7 +275,6 @@ class NetworkInterface:
     
     def flush_poll_queue(self, node: KoiNetNode) -> list[Event]: ...
     def flush_webhook_queue(self, node: RID): ...
-    def flush_all_webhook_queues(self): ...
 
     def fetch_remote_bundle(self, rid: RID): ...
     def fetch_remote_manifest(self, rid: RID): ...
@@ -390,11 +393,14 @@ def poll_events(req: PollEvents) -> EventsPayload:
 The `ProcessorInterface` class provides access to a node's internal knowledge processing pipeline.
 ```python
 class ProcessorInterface:
+    worker_thread: threading.Thread | None = None
+
     def __init__(
         self, 
         cache: Cache, 
         network: NetworkInterface,
         identity: NodeIdentity,
+        use_kobj_processor_thread: bool,
         default_handlers: list[KnowledgeHandler] = []
     ): ...
 

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/examples/basic_coordinator_node.py

@@ -53,7 +53,10 @@ node = NodeInterface(
             state=[KoiNetNode, KoiNetEdge]
         )
     ),
-    identity_file_path="coordinator_identity.json"
+    use_kobj_processor_thread=True,
+    cache_directory_path="coordinator_node_rid_cache",
+    event_queues_file_path="coordinator_node_event_queus.json",
+    identity_file_path="coordinator_node_identity.json",
 )
 
 
@@ -88,9 +91,9 @@ def handshake_handler(proc: ProcessorInterface, kobj: KnowledgeObject):
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
-    node.initialize()
+    node.start()
     yield
-    node.finalize()
+    node.stop()
 
 app = FastAPI(
     lifespan=lifespan, 
@@ -105,8 +108,6 @@ def broadcast_events(req: EventsPayload, background: BackgroundTasks):
     for event in req.events:
         node.processor.handle(event=event, source=KnowledgeSource.External)
     
-    background.add_task(node.processor.flush_kobj_queue)
-
 @app.post(POLL_EVENTS_PATH)
 def poll_events(req: PollEvents) -> EventsPayload:
     logger.info(f"Request to {POLL_EVENTS_PATH}")

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/examples/basic_partial_node.py

@@ -27,7 +27,9 @@ node = NodeInterface(
     profile=NodeProfile(
         node_type=NodeType.PARTIAL
     ),
-    identity_file_path="partial_identity.json",
+    cache_directory_path="partial_node_rid_cache",
+    event_queues_file_path="parital_node_event_queus.json",
+    identity_file_path="partial_node_identity.json",
     first_contact="http://127.0.0.1:8000/koi-net"
 )
 
@@ -72,11 +74,11 @@ def coordinator_contact(processor: ProcessorInterface, kobj: KnowledgeObject):
     
 
 
-node.initialize()
+node.start()
 
 while True:
     for event in node.network.poll_neighbors():
         node.processor.handle(event=event, source=KnowledgeSource.External)
     node.processor.flush_kobj_queue()
     
-    time.sleep(1)
+    time.sleep(5)

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/examples/full_node_template.py

@@ -50,15 +50,16 @@ node = NodeInterface(
             state=[KoiNetNode, KoiNetEdge]
         )
     ),
+    use_kobj_processor_thread=True,
     first_contact=coordinator_url
 )
 
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
-    node.initialize()
+    node.start()
     yield
-    node.finalize()
+    node.stop()
 
 
 app = FastAPI(lifespan=lifespan, root_path="/koi-net")
@@ -69,8 +70,6 @@ def broadcast_events(req: EventsPayload, background: BackgroundTasks):
     for event in req.events:
         node.processor.handle(event=event, source=KnowledgeSource.External)
     
-    background.add_task(node.processor.flush_kobj_queue)
-
 @app.post(POLL_EVENTS_PATH)
 def poll_events(req: PollEvents) -> EventsPayload:
     logger.info(f"Request to {POLL_EVENTS_PATH}")

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/examples/partial_node_template.py

@@ -27,7 +27,7 @@ node = NodeInterface(
 )
 
 if __name__ == "__main__":
-    node.initialize()
+    node.start()
 
     try:
         while True:
@@ -38,4 +38,4 @@ if __name__ == "__main__":
             time.sleep(5)
             
     finally:
-        node.finalize()
+        node.stop()

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "koi-net"
-version = "1.0.0-beta.6"
+version = "1.0.0-beta.7"
 description = "Implementation of KOI-net protocol in Python"
 authors = [
     {name = "Luke Miller", email = "luke@block.science"}

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/src/koi_net/core.py

@@ -11,25 +11,30 @@ from .protocol.event import Event, EventType
 
 logger = logging.getLogger(__name__)
 
+
 class NodeInterface:
     cache: Cache
     identity: NodeIdentity
     network: NetworkInterface
     processor: ProcessorInterface
     first_contact: str
+    use_kobj_processor_thread: bool
     
     def __init__(
         self, 
         name: str,
         profile: NodeProfile,
         identity_file_path: str = "identity.json",
+        event_queues_file_path: str = "event_queues.json",
+        cache_directory_path: str = "rid_cache",
+        use_kobj_processor_thread: bool = False,
         first_contact: str | None = None,
         handlers: list[KnowledgeHandler] | None = None,
         cache: Cache | None = None,
         network: NetworkInterface | None = None,
         processor: ProcessorInterface | None = None
     ):
-        self.cache = cache or Cache(directory_path=f".cache")
+        self.cache = cache or Cache(cache_directory_path)
         self.identity = NodeIdentity(
             name=name,
             profile=profile,
@@ -38,7 +43,7 @@ class NodeInterface:
         )
         self.first_contact = first_contact
         self.network = network or NetworkInterface(
-            file_path="event_queues.json", 
+            file_path=event_queues_file_path, 
             first_contact=self.first_contact,
             cache=self.cache, 
             identity=self.identity
@@ -50,30 +55,41 @@ class NodeInterface:
                 obj for obj in vars(default_handlers).values() 
                 if isinstance(obj, KnowledgeHandler)
             ]
-        
+
+        self.use_kobj_processor_thread = use_kobj_processor_thread
         self.processor = processor or ProcessorInterface(
             cache=self.cache, 
             network=self.network, 
             identity=self.identity, 
+            use_kobj_processor_thread=self.use_kobj_processor_thread,
             default_handlers=handlers
         )
     
-    def initialize(self) -> None:
-        """Initializes node, call on startup.
+    def start(self) -> None:
+        """Starts a node, call this method first.
         
-        Loads event queues into memory. Generates network graph from nodes and edges in cache. Processes any state changes of node bundle. Initiates handshake with first contact (if provided) if node doesn't have any neighbors.
+        Starts the processor thread (if enabled). Loads event queues into memory. Generates network graph from nodes and edges in cache. Processes any state changes of node bundle. Initiates handshake with first contact (if provided) if node doesn't have any neighbors.
         """
-        self.network._load_event_queues()
+        if self.use_kobj_processor_thread:
+            logger.info("Starting processor worker thread")
+            self.processor.worker_thread.start()
         
+        self.network._load_event_queues()
         self.network.graph.generate()
         
         self.processor.handle(
             bundle=Bundle.generate(
                 rid=self.identity.rid, 
                 contents=self.identity.profile.model_dump()
-            ),
-            flush=True
+            )
         )
+        
+        logger.info("Waiting for kobj queue to empty")
+        if self.use_kobj_processor_thread:
+            self.processor.kobj_queue.join()
+        else:
+            self.processor.flush_kobj_queue()
+        logger.info("Done")
     
         if not self.network.graph.get_neighbors() and self.first_contact:
             logger.info(f"I don't have any neighbors, reaching out to first contact {self.first_contact}")
@@ -94,9 +110,17 @@ class NodeInterface:
                 return
             
                         
-    def finalize(self):
-        """Finalizes node, call on shutdown.
+    def stop(self):
+        """Stops a node, call this method last.
         
-        Saves event queues to storage.
+        Finishes processing knowledge object queue. Saves event queues to storage.
         """
+        logger.info("Stopping node...")
+        
+        if self.use_kobj_processor_thread:
+            logger.info("Waiting for kobj queue to empty")
+            self.processor.kobj_queue.join()
+        else:
+            self.processor.flush_kobj_queue()
+        
         self.network._save_event_queues()

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/src/koi_net/network/graph.py

@@ -74,13 +74,13 @@ class NetworkGraph:
         """Returns edges this node belongs to.
         
         All edges returned by default, specify `direction` to restrict to incoming or outgoing edges only."""
-        
+                
         edges = []
-        if direction != "in":
+        if direction != "in" and self.dg.out_edges:
             out_edges = self.dg.out_edges(self.identity.rid)
             edges.extend([e for e in out_edges])
                 
-        if direction != "out":
+        if direction != "out" and self.dg.in_edges:
             in_edges = self.dg.in_edges(self.identity.rid)
             edges.extend([e for e in in_edges])
                     

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/src/koi_net/network/interface.py

@@ -165,6 +165,8 @@ class NetworkInterface:
             return
         
         events = self._flush_queue(self.webhook_event_queue, node)
+        if not events: return
+        
         logger.info(f"Broadcasting {len(events)} events")
         
         try:  
@@ -173,11 +175,6 @@ class NetworkInterface:
             logger.warning("Broadcast failed, requeuing events")
             for event in events:
                 self.push_event_to(event, node)
-    
-    def flush_all_webhook_queues(self):
-        """Flushes all nodes' webhook queues and broadcasts events."""
-        for node in self.webhook_event_queue.keys():
-            self.flush_webhook_queue(node)
             
     def get_state_providers(self, rid_type: RIDType) -> list[KoiNetNode]:
         """Returns list of node RIDs which provide state for the specified RID type."""

{koi_net-1.0.0b6 → koi_net-1.0.0b7}/src/koi_net/processor/interface.py

@@ -1,5 +1,6 @@
 import logging
-from queue import Queue
+import queue
+import threading
 from typing import Callable
 from rid_lib.core import RID, RIDType
 from rid_lib.ext import Bundle, Cache, Manifest
@@ -30,20 +31,30 @@ class ProcessorInterface:
     network: NetworkInterface
     identity: NodeIdentity
     handlers: list[KnowledgeHandler]
-    kobj_queue: Queue[KnowledgeObject]
+    kobj_queue: queue.Queue[KnowledgeObject]
+    use_kobj_processor_thread: bool
+    worker_thread: threading.Thread | None = None
     
     def __init__(
         self, 
         cache: Cache, 
         network: NetworkInterface,
         identity: NodeIdentity,
+        use_kobj_processor_thread: bool,
         default_handlers: list[KnowledgeHandler] = []
     ):
         self.cache = cache
         self.network = network
         self.identity = identity
+        self.use_kobj_processor_thread = use_kobj_processor_thread
         self.handlers: list[KnowledgeHandler] = default_handlers
-        self.kobj_queue = Queue()
+        self.kobj_queue = queue.Queue()
+        
+        if self.use_kobj_processor_thread:
+            self.worker_thread = threading.Thread(
+                target=self.kobj_processor_worker,
+                daemon=True
+            )
         
     def add_handler(self, handler: KnowledgeHandler):
         self.handlers.append(handler)
@@ -195,18 +206,38 @@ class ProcessorInterface:
             logger.info("No network targets set")
         
         for node in kobj.network_targets:
-            self.network.push_event_to(kobj.normalized_event, node)
-        self.network.flush_all_webhook_queues()
+            self.network.push_event_to(kobj.normalized_event, node, flush=True)
         
         kobj = self.call_handler_chain(HandlerType.Final, kobj)
-                
+
     def flush_kobj_queue(self):
-        """Flushes all knowledge objects from queue and processes them."""
+        """Flushes all knowledge objects from queue and processes them.
+        
+        NOTE: ONLY CALL THIS METHOD IN SINGLE THREADED NODES, OTHERWISE THIS WILL CAUSE RACE CONDITIONS.
+        """
+        if self.use_kobj_processor_thread:
+            logger.warning("You are using a worker thread, calling this method can cause race conditions!")
+        
         while not self.kobj_queue.empty():
             kobj = self.kobj_queue.get()
             logger.info(f"Dequeued {kobj!r}")
             self.process_kobj(kobj)
+            self.kobj_queue.task_done()
             logger.info("Done handling")
+    
+    def kobj_processor_worker(self, timeout=0.1):
+        while True:
+            try:
+                kobj = self.kobj_queue.get(timeout=timeout)
+                logger.info(f"Dequeued {kobj!r}")
+                self.process_kobj(kobj)
+                self.kobj_queue.task_done()
+            
+            except queue.Empty:
+                pass
+            
+            except Exception as e:
+                logger.warning(f"Error processing kobj: {e}")
         
     def handle(
         self,
@@ -216,8 +247,7 @@ class ProcessorInterface:
         event: Event | None = None,
         kobj: KnowledgeObject | None = None,
         event_type: KnowledgeEventType = None,
-        source: KnowledgeSource = KnowledgeSource.Internal,
-        flush: bool = False
+        source: KnowledgeSource = KnowledgeSource.Internal
     ):
         """Queues provided knowledge to be handled by processing pipeline.
         
@@ -238,6 +268,3 @@ class ProcessorInterface:
         
         self.kobj_queue.put(_kobj)
         logger.info(f"Queued {_kobj!r}")
-        
-        if flush:
-            self.flush_kobj_queue()