crowddrop-pubsub-sdk 0.2.0__py3-none-any.whl

crowddrop_pubsub_sdk-0.2.0.dist-info/METADATA

@@ -0,0 +1,181 @@
+Metadata-Version: 2.4
+Name: crowddrop-pubsub-sdk
+Version: 0.2.0
+Summary: Framework-agnostic Google Cloud Pub/Sub step-event publisher/subscriber for embodied AI agents.
+Project-URL: Repository, https://github.com/crowddrop-ai/crowddrop_ai_agents
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: google-cloud-pubsub<3.0.0,>=2.21.0
+Requires-Dist: google-api-core<3.0.0,>=2.17.1
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=0.3.0; extra == "langchain"
+
+# pubsub_sdk
+
+A small, framework-agnostic wrapper around Google Cloud Pub/Sub for publishing and
+consuming structured step/progress events, and for direct agent-to-agent
+instruction/reply exchanges (chat). It has no dependency on LangChain, on any
+specific agent framework, or on any particular storage backend — any Python project
+(an agent, a backend service, a future embodied-AI-agent codebase) can import it
+directly.
+
+## Install
+
+```bash
+pip install ./pubsub_sdk          # core publisher/subscriber only
+pip install ./pubsub_sdk[langchain]  # + the optional LangChain callback adapter
+```
+
+Published to public PyPI as `crowddrop-pubsub-sdk` (the import name stays
+`pubsub_sdk` either way):
+
+```bash
+pip install crowddrop-pubsub-sdk
+pip install crowddrop-pubsub-sdk[langchain]
+```
+
+## Configuration (environment variables)
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `GCP_PROJECT_ID` | `test` | GCP project the client talks to. |
+| `PUBSUB_SDK_TOPIC_ID` | `agent-step-events` | Topic events are published to. |
+| `PUBSUB_SDK_SUBSCRIPTION_ID` | `agent-step-events-firestore-persister` | Subscription a consumer pulls from. |
+| `PUBSUB_SDK_CHAT_TOPIC_ID` | `agent-chat-messages` | Fixed topic for agent-to-agent chat (see below). |
+| `PUBSUB_SDK_ENABLED` | `true` | Kill switch — set to `false` to disable publishing/subscribing entirely. |
+| `PUBSUB_EMULATOR_HOST` | unset | Standard Google client env var; set to point at a local Pub/Sub emulator instead of real GCP. |
+
+Credentials are picked up automatically via `GOOGLE_APPLICATION_CREDENTIALS`, exactly
+like every other `google-cloud-*` client — no SDK-specific credential handling.
+
+## Publishing (any consumer)
+
+```python
+from pubsub_sdk import StepEventPublisher
+
+publisher = StepEventPublisher()
+publisher.publish_step(
+    activity_id="task-123",
+    agent_id="MissionCommander",
+    event_type="tool_start",
+    message="Invoking tool: delegate_to_agent",
+    status="in_progress",
+    metadata={"tool_name": "delegate_to_agent"},
+)
+```
+
+`publish_step()` never raises — if Pub/Sub is unavailable it logs the failure and
+returns `False`. Check `publisher.is_available()` if you need to know the state
+up front.
+
+## Subscribing (any consumer)
+
+```python
+from pubsub_sdk import StepEventSubscriber, StepEvent
+
+def handle(event: StepEvent) -> None:
+    ...  # e.g. persist it somewhere
+
+subscriber = StepEventSubscriber()
+subscriber.pull_forever(handle)  # blocks; acks on success, nacks (redelivers) on exception
+```
+
+## Agent-to-agent chat (ChatMessage)
+
+A direct instruction/reply exchange between two agent instances (e.g. a
+coordinator persona delegating to a field agent), as an alternative transport to
+a live MCP/SSE connection. Both directions of one exchange ride the same fixed
+topic (`agent-chat-messages`) and are distinguished by `role`
+(`ChatMessageRole.INSTRUCTION` / `.REPLY`), correlated by `message_id`.
+
+```python
+from pubsub_sdk import ChatMessage, ChatMessageRole, get_default_chat_publisher, ChatMessageSubscriber
+
+# Sending side: publish an instruction addressed to another agent instance's
+# routing identity (its session_id - never a shared persona name; two ephemeral
+# instances of the same persona would otherwise collide on the same subscription).
+publisher = get_default_chat_publisher()
+publisher.publish(ChatMessage(
+    message_id="m1", from_agent="MissionCommander__task-42", to_agent="PreciseHumanoid",
+    role=ChatMessageRole.INSTRUCTION, content="what is your current battery level?",
+    activity_id="task-42",
+))
+
+# Receiving side: each participating instance self-provisions its own filtered
+# subscription (against real GCP, not just the emulator - a per-instance
+# subscription's identity only exists at runtime and can't be pre-provisioned).
+subscriber = ChatMessageSubscriber(routing_identity="PreciseHumanoid")
+subscriber.pull_forever(lambda msg: ...)  # blocks; ack on success, nack (redeliver) on exception
+```
+
+`GenericSubscriber.pull_forever` also accepts a `stop()` call from another thread
+to cancel an in-progress streaming pull cleanly.
+
+**Security note:** `agent-chat-messages` is one shared topic with no
+per-agent access control beyond GCP IAM on the topic/subscription resources
+themselves - any credential with publish/subscribe rights can spoof
+`from_agent` or read another agent's traffic via an unfiltered subscription.
+Fine within one trust domain (this app's own fleet today); see
+`docs/pubsub_chat_access_control.md` before granting a genuinely independent/
+external party a credential onto this topic.
+
+## LangChain integration (optional extra)
+
+```python
+from pubsub_sdk.langchain_callback import StepEventCallbackHandler
+
+callback = StepEventCallbackHandler(activity_id="task-123", agent_id="MissionCommander")
+callback.log_mission_start()
+# pass `callback` into a LangChain AgentExecutor's `config={"callbacks": [callback]}` —
+# every LLM/tool/chain/agent lifecycle event then publishes automatically.
+```
+
+`langchain_callback` is the only module in this package that imports `langchain` —
+a consumer who only needs `StepEventPublisher`/`StepEventSubscriber` never pulls
+that dependency in.
+
+## Releasing (publishing a new version to PyPI)
+
+Releases are tag-triggered via `.github/workflows/publish-pubsub-sdk.yml`,
+using PyPI's **Trusted Publishing** (OIDC) — no API token is stored as a
+GitHub secret.
+
+1. Bump `version` in `pubsub_sdk/pyproject.toml`.
+2. Commit that change (on a branch, via the normal PR flow).
+3. Once merged, tag the merge commit and push the tag:
+   ```bash
+   git tag pubsub-sdk-v<version>   # e.g. pubsub-sdk-v0.2.1
+   git push origin pubsub-sdk-v<version>
+   ```
+   The tag push is what fires the workflow — it builds `pubsub_sdk/` and
+   uploads it to [pypi.org/project/crowddrop-pubsub-sdk](https://pypi.org/project/crowddrop-pubsub-sdk/).
+   No other trigger publishes this package.
+
+**One-time setup, not yet done as of this writing — needed before the first
+tag push, and again only if this ever moves to a different PyPI
+account/org:**
+- Register a **pending publisher** for `crowddrop-pubsub-sdk` at
+  https://pypi.org/manage/account/publishing/ — this can be done before the
+  PyPI project exists, so it covers the *first-ever* release too, not just
+  subsequent ones. Fill in: PyPI project name `crowddrop-pubsub-sdk`, repo
+  owner `crowddrop-ai`, repo name `crowddrop_ai_agents`, workflow filename
+  `publish-pubsub-sdk.yml`, environment name `pypi`. Requires a PyPI account
+  with 2FA enabled — no API token to generate or store.
+- The `pypi` GitHub Environment referenced by the workflow is created
+  automatically the first time the workflow runs against it; create it
+  manually in this repo's Settings → Environments beforehand only if you
+  want a required-reviewer protection rule (so a tag push pauses for human
+  approval before it actually publishes).
+- `../scripts/publish_python_packages.sh` (manual `build` + `twine upload`)
+  is kept as a fallback/local-dry-run tool only — with a pending publisher
+  registered, it's no longer needed even for the first release.
+
+Versioning is manual — nothing cross-checks the tag against
+`pyproject.toml`'s `version`. Bump the file first, commit, *then* tag that
+exact commit; tagging a commit whose `pyproject.toml` still has an
+already-published version will fail the upload (PyPI rejects re-uploading an
+existing version).
+
+Note: `crowddrop-sdk`'s `cloud-brain` extra depends on `crowddrop-pubsub-sdk`
+(see `../crowddrop_sdk/pyproject.toml`) — release this package first if both
+need a coordinated update.

crowddrop_pubsub_sdk-0.2.0.dist-info/RECORD

@@ -0,0 +1,13 @@
+pubsub_sdk/__init__.py,sha256=-_qVzXE2Y8_s0-U64P4oZoOYo2bpP5C9YWPA7XzlTlQ,959
+pubsub_sdk/chat.py,sha256=n4lsSeYSr1Vp10Yq-pQ2NLCFgo9XIVFBfmjf7RNX2bk,4748
+pubsub_sdk/client.py,sha256=FrmSH3xVLepsUVgXFR3lnygj1bSCVHOd5dwOXPLHeY0,18502
+pubsub_sdk/config.py,sha256=iCatyWtrkeW5tEx2nQ4mbMIrYVt2OCh9zHOE9S8atpA,1464
+pubsub_sdk/events.py,sha256=GGo9SglVNwY1qxY79FzuAyFIlaJtLV7X3GbOGdE5nXM,1740
+pubsub_sdk/langchain_callback.py,sha256=CyyEbIJeaune5MCes8ZNcTxGRNkfb_KdgZyXA-p6hG0,9887
+pubsub_sdk/publisher.py,sha256=xOjXiFMOV8LkptL2aKZvvQvgnrLlMEBeKy5pTig0iNg,2338
+pubsub_sdk/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+pubsub_sdk/subscriber.py,sha256=cuERyg4u1k7dFbZrZuMpTYoO-TGAhyfpfm1sX_l7VxM,1346
+crowddrop_pubsub_sdk-0.2.0.dist-info/METADATA,sha256=aONfOS8RBZa81gkgaIxl6Vm-xHY-afD5HhkeEDVZbf4,8055
+crowddrop_pubsub_sdk-0.2.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+crowddrop_pubsub_sdk-0.2.0.dist-info/top_level.txt,sha256=gzpn2GhMuyKPgkd2n25810spt0QEjFBFiClNvH5d60I,11
+crowddrop_pubsub_sdk-0.2.0.dist-info/RECORD,,

crowddrop_pubsub_sdk-0.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

crowddrop_pubsub_sdk-0.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+pubsub_sdk

pubsub_sdk/__init__.py

@@ -0,0 +1,31 @@
+"""Framework-agnostic Google Cloud Pub/Sub step-event publisher/subscriber.
+
+The LangChain adapter (StepEventCallbackHandler) is intentionally NOT re-exported
+here — importing it requires the optional `langchain` extra, and this top-level
+package must stay importable without that dependency. Import it explicitly from
+`pubsub_sdk.langchain_callback` when you need it.
+"""
+from .chat import (
+    ChatMessage,
+    ChatMessagePublisher,
+    ChatMessageRole,
+    ChatMessageSubscriber,
+    get_default_chat_publisher,
+)
+from .config import PubSubConfig
+from .events import StepEvent
+from .publisher import StepEventPublisher, get_default_publisher
+from .subscriber import StepEventSubscriber
+
+__all__ = [
+    "PubSubConfig",
+    "StepEvent",
+    "StepEventPublisher",
+    "StepEventSubscriber",
+    "get_default_publisher",
+    "ChatMessage",
+    "ChatMessageRole",
+    "ChatMessagePublisher",
+    "ChatMessageSubscriber",
+    "get_default_chat_publisher",
+]

pubsub_sdk/chat.py

@@ -0,0 +1,118 @@
+"""ChatMessage: the wire schema for a direct instruction/reply exchange between
+two agent instances (e.g. MissionCommander delegating to a field agent) over
+Pub/Sub, as an alternative transport to a live MCP/SSE connection.
+
+Both directions of one exchange ride the same fixed topic
+(PubSubConfig.chat_topic_id, default "agent-chat-messages") and are
+distinguished by `role` (ChatMessageRole.INSTRUCTION / .REPLY), correlated by
+`message_id`. Routing to the right recipient is a Pub/Sub subscription filter
+on the `to_agent` attribute - each participating agent INSTANCE self-provisions
+its own filtered subscription, keyed by its session_id (never its persona-wide
+agent_id - see ChatMessageSubscriber).
+"""
+import json
+import re
+from dataclasses import asdict, dataclass
+from typing import Any, Dict, Optional, Tuple
+
+from .client import GenericPublisher, GenericSubscriber
+from .config import PubSubConfig
+
+# Two ephemeral instances of the same persona (e.g. two in-flight
+# MissionCommander missions) can run concurrently, each spawned with its own
+# unique session_id - subscriptions live for as long as any single instance
+# does, so this is a generous safety-net TTL, not a tuning knob for latency.
+_CHAT_SUBSCRIPTION_TTL_SECONDS = 172800  # 48h
+
+
+class ChatMessageRole:
+    INSTRUCTION = "instruction"
+    REPLY = "reply"
+
+
+@dataclass
+class ChatMessage:
+    message_id: str
+    from_agent: str
+    to_agent: str
+    role: str
+    content: str
+    activity_id: Optional[str] = None
+
+    def to_pubsub_message(self) -> Tuple[bytes, Dict[str, str]]:
+        body = asdict(self)
+        data = json.dumps(body).encode("utf-8")
+        attributes = {
+            "message_id": self.message_id,
+            "from_agent": self.from_agent,
+            "to_agent": self.to_agent,
+            "role": self.role,
+        }
+        return data, attributes
+
+    @classmethod
+    def from_pubsub_message(cls, data: bytes, attributes: Optional[Dict[str, str]] = None) -> "ChatMessage":
+        body = json.loads(data.decode("utf-8"))
+        return cls(
+            message_id=body["message_id"],
+            from_agent=body["from_agent"],
+            to_agent=body["to_agent"],
+            role=body["role"],
+            content=body["content"],
+            activity_id=body.get("activity_id"),
+        )
+
+
+class ChatMessagePublisher(GenericPublisher[ChatMessage]):
+    def __init__(self, config: Optional[PubSubConfig] = None):
+        cfg = config or PubSubConfig.from_env()
+        super().__init__(config=cfg, topic_id=cfg.chat_topic_id, label="ChatMessagePublisher")
+
+
+_default_chat_publisher: Optional[ChatMessagePublisher] = None
+
+
+def get_default_chat_publisher(config: Optional[PubSubConfig] = None) -> ChatMessagePublisher:
+    """Process-wide singleton, mirroring get_default_publisher(): one long-lived
+    Pub/Sub client (and its gRPC channel) per process, shared by every attached
+    chat channel, instead of one per agent instance with nothing to close it."""
+    global _default_chat_publisher
+    if _default_chat_publisher is None:
+        _default_chat_publisher = ChatMessagePublisher(config=config)
+    return _default_chat_publisher
+
+
+def _sanitize_subscription_id(routing_identity: str) -> str:
+    """Pub/Sub subscription IDs allow only [A-Za-z0-9-_.~+%]; session_id can
+    contain a platform task_id of unknown format, so sanitize rather than
+    assume it's already safe."""
+    return re.sub(r"[^A-Za-z0-9\-_.~+%]", "-", routing_identity)
+
+
+class ChatMessageSubscriber(GenericSubscriber[ChatMessage]):
+    """One per attached agent INSTANCE (never shared across instances, unlike
+    ChatMessagePublisher) - filtered on `to_agent = routing_identity`, where
+    routing_identity is that instance's session_id. Self-provisions against
+    real GCP (not just the emulator), since a per-instance subscription's
+    identity only exists at runtime and can't be pre-provisioned by infra the
+    way agent-step-events' fixed subscription is."""
+
+    def __init__(
+        self,
+        routing_identity: str,
+        config: Optional[PubSubConfig] = None,
+        expiration_ttl_seconds: int = _CHAT_SUBSCRIPTION_TTL_SECONDS,
+    ):
+        cfg = config or PubSubConfig.from_env()
+        self.routing_identity = routing_identity
+        subscription_id = f"agent-chat-{_sanitize_subscription_id(routing_identity)}"
+        super().__init__(
+            config=cfg,
+            topic_id=cfg.chat_topic_id,
+            subscription_id=subscription_id,
+            message_cls=ChatMessage,
+            label=f"ChatMessageSubscriber[{routing_identity}]",
+            filter_expr=f'attributes.to_agent = "{routing_identity}"',
+            expiration_ttl_seconds=expiration_ttl_seconds,
+            always_self_provision=True,
+        )

pubsub_sdk/client.py

@@ -0,0 +1,381 @@
+"""Generic Pub/Sub publish/subscribe core, extracted from the original
+StepEventPublisher/StepEventSubscriber so a second message type (chat, see
+chat.py) doesn't duplicate the same emulator-retry/error-logging/ack-nack
+machinery a second time. StepEventPublisher/StepEventSubscriber become thin
+subclasses with zero API change - this refactor is behavior-preserving by
+construction (see test_publisher.py/test_subscriber.py, unmodified).
+
+Follows this org's existing degrade-gracefully-and-log-loudly philosophy (see
+ActivityLoggingService): never raises past its own boundary, always logs
+failures with context server-side, and reports its own availability via
+is_available().
+
+The real google-cloud-pubsub client's publish()/subscribe() calls never raise
+NotFound synchronously - they always return a future immediately and defer
+the RPC's own errors to it (confirmed against a real Pub/Sub emulator, not
+just mocks). Every NotFound-driven retry below reacts to the future
+resolving, not to publish()/subscribe() itself raising.
+"""
+import logging
+import threading
+from typing import Any, Callable, Dict, Generic, Optional, Protocol, Tuple, Type, TypeVar
+
+from google.api_core.exceptions import AlreadyExists, NotFound
+from google.cloud import pubsub_v1
+
+from .config import PubSubConfig
+
+logger = logging.getLogger(__name__)
+
+
+class PubsubMessage(Protocol):
+    """Structural contract any message type (StepEvent, ChatMessage, ...) must
+    satisfy to ride on GenericPublisher/GenericSubscriber."""
+
+    def to_pubsub_message(self) -> Tuple[bytes, Dict[str, str]]: ...
+
+    @classmethod
+    def from_pubsub_message(cls, data: bytes, attributes: Optional[Dict[str, str]] = None) -> "PubsubMessage": ...
+
+
+T = TypeVar("T", bound=PubsubMessage)
+
+
+class GenericPublisher(Generic[T]):
+    """Framework-agnostic publish core. A subclass fixes the topic_id and
+    message type; everything else (client construction, degrade-gracefully
+    behavior, emulator-only topic auto-create, async error logging) lives
+    here once."""
+
+    def __init__(self, config: Optional[PubSubConfig], topic_id: str, label: str):
+        self.config = config or PubSubConfig.from_env()
+        self._topic_id = topic_id
+        self._label = label
+        self._client: Optional[pubsub_v1.PublisherClient] = None
+        self._topic_path: Optional[str] = None
+        self._topic_create_attempted = False
+        # Guards _topic_create_attempted and _in_flight - publish()'s
+        # done-callbacks fire on a background gRPC thread, concurrently with
+        # the caller's own thread and with each other. _in_flight counts
+        # logical publish attempts still in progress (an original publish
+        # plus its one possible auto-create retry count as a single unit,
+        # decremented only once we're truly done with it) - _drained is set
+        # exactly when that count reaches zero, so close() has one thing to
+        # wait on without racing a snapshot of individual futures (an
+        # earlier version of this fix took a one-time snapshot of pending
+        # futures, which missed a retry future created moments later and
+        # tore the client down while the retry was still using it).
+        self._lock = threading.Lock()
+        self._in_flight = 0
+        self._drained = threading.Event()
+        self._drained.set()
+
+        if not self.config.enabled:
+            logger.info(f"{label} disabled via PUBSUB_SDK_ENABLED=false.")
+            return
+
+        try:
+            self._client = pubsub_v1.PublisherClient()
+            self._topic_path = self._client.topic_path(self.config.project_id, topic_id)
+            logger.info(
+                f"{label} initialized (project={self.config.project_id}, "
+                f"topic={topic_id}) targeting: {self.config.target_label}"
+            )
+        except Exception as e:
+            self._client = None
+            self._topic_path = None
+            logger.warning(f"{label} initialization failed: {e}. Publishing will be disabled.")
+
+    def is_available(self) -> bool:
+        return self._client is not None and self._topic_path is not None
+
+    def publish(self, message: T) -> bool:
+        """Fire-and-forget: returns True as soon as the message is handed to
+        the client for sending, not once delivery is actually confirmed - a
+        short-lived process (a CLI script, a one-shot Cloud Function) that
+        exits right after calling this can lose the message before it's
+        actually sent over the wire. Call close() before such a process
+        exits; it blocks until every in-flight publish from this instance is
+        confirmed sent (or fails)."""
+        if not self.is_available():
+            return False
+
+        data, attributes = message.to_pubsub_message()
+        with self._lock:
+            self._in_flight += 1
+            self._drained.clear()
+
+        future = self._publish_once(data, attributes)
+        if future is None:
+            self._mark_publish_done()
+            return False
+
+        future.add_done_callback(lambda fut: self._on_publish_done(fut, data, attributes, is_retry=False))
+        return True
+
+    def _publish_once(self, data: bytes, attributes: Dict[str, str]):
+        try:
+            return self._client.publish(self._topic_path, data, **attributes)
+        except Exception as e:
+            logger.error(f"{self._label}.publish: publish call failed: {e}")
+            return None
+
+    def _mark_publish_done(self) -> None:
+        """Decrements _in_flight for one logical publish attempt (an
+        original publish plus its one possible auto-create retry share a
+        single count, so this fires exactly once per publish() call, however
+        many underlying Pub/Sub RPCs it took)."""
+        with self._lock:
+            self._in_flight = max(self._in_flight - 1, 0)
+            if self._in_flight == 0:
+                self._drained.set()
+
+    def _on_publish_done(self, fut: Any, data: bytes, attributes: Dict[str, str], is_retry: bool) -> None:
+        exc = fut.exception()
+        if exc is None:
+            self._mark_publish_done()
+            return
+
+        if not is_retry and isinstance(exc, NotFound) and self._should_attempt_topic_autocreate():
+            logger.info(f"{self._label}.publish: topic not found, auto-creating and retrying (emulator only).")
+            if self._ensure_topic_exists():
+                retry_future = self._publish_once(data, attributes)
+                if retry_future is not None:
+                    # Still in flight under the same count this original
+                    # future held - do not mark done until the retry itself
+                    # resolves, so close() never sees "nothing pending"
+                    # while a retry is actually in progress using self._client.
+                    retry_future.add_done_callback(
+                        lambda f: self._on_publish_done(f, data, attributes, is_retry=True)
+                    )
+                    return
+            logger.error(f"{self._label}.publish: retry after topic auto-create failed: {exc}")
+            self._mark_publish_done()
+            return
+
+        logger.error(f"{self._label}.publish: async publish failed: {exc}")
+        self._mark_publish_done()
+
+    def _should_attempt_topic_autocreate(self) -> bool:
+        """Memoized: only ever attempt auto-create once per process, not once
+        per publish() call - otherwise a misconfigured emulator target pays
+        two RPCs (failed publish + failed create_topic) on every single
+        message forever. Never true against real GCP - production topics are
+        provisioned by infra ahead of time."""
+        if not self.config.emulator_host:
+            return False
+        with self._lock:
+            if self._topic_create_attempted:
+                return False
+            self._topic_create_attempted = True
+        return True
+
+    def _ensure_topic_exists(self) -> bool:
+        """Emulator-only convenience: auto-create the topic if it's missing.
+        Only ever called after _should_attempt_topic_autocreate() has already
+        confirmed we're targeting the emulator. Tolerates AlreadyExists so a
+        second concurrent publish racing this same auto-create doesn't log a
+        false failure for a topic that in fact now exists."""
+        try:
+            self._client.create_topic(name=self._topic_path)
+            logger.info(f"Auto-created emulator topic: {self._topic_path}")
+            return True
+        except AlreadyExists:
+            logger.info(f"Emulator topic already exists (reused): {self._topic_path}")
+            return True
+        except Exception as e:
+            logger.error(f"Failed to auto-create emulator topic {self._topic_path}: {e}")
+            return False
+
+    def close(self, timeout: float = 10.0) -> None:
+        """Blocks until every in-flight publish from this instance is
+        confirmed sent or fails (up to timeout), then shuts the client down.
+        Call this before a short-lived process exits - see publish()'s own
+        docstring for why. If timeout elapses with publishes still
+        in-flight, logs a warning and tears the client down anyway rather
+        than blocking forever - a since-abandoned in-flight retry attempting
+        to use self._client afterward fails cleanly (caught and logged by
+        _publish_once/_ensure_topic_exists) rather than raising out of a
+        background thread."""
+        if self._client is not None:
+            if not self._drained.wait(timeout=timeout):
+                with self._lock:
+                    remaining = self._in_flight
+                logger.warning(f"{self._label}.close: timed out with {remaining} publish(es) still in flight.")
+            try:
+                # PublisherClient has no close() - stop() is its equivalent
+                # (asynchronously flushes outstanding messages and prevents
+                # further publish() calls; the wait above is what actually
+                # guarantees delivery, stop() alone does not block for it).
+                self._client.stop()
+            except Exception as e:
+                logger.warning(f"{self._label}.close: error stopping client: {e}")
+        self._client = None
+        self._topic_path = None
+
+
+class GenericSubscriber(Generic[T]):
+    """Framework-agnostic pull-consumer core. A subclass fixes the topic/
+    subscription ids and message type; `always_self_provision` controls
+    whether the subscription is created eagerly (and against real GCP, not
+    just the emulator) - StepEventSubscriber keeps today's emulator-only,
+    lazy-on-NotFound behavior (always_self_provision=False); a per-instance
+    subscriber whose identity only exists at runtime (e.g. chat) opts in."""
+
+    def __init__(
+        self,
+        config: Optional[PubSubConfig],
+        topic_id: str,
+        subscription_id: str,
+        message_cls: Type[T],
+        label: str,
+        filter_expr: Optional[str] = None,
+        expiration_ttl_seconds: Optional[int] = None,
+        always_self_provision: bool = False,
+    ):
+        self.config = config or PubSubConfig.from_env()
+        self._topic_id = topic_id
+        self._subscription_id = subscription_id
+        self._message_cls = message_cls
+        self._label = label
+        self._filter_expr = filter_expr
+        self._expiration_ttl_seconds = expiration_ttl_seconds
+        self._always_self_provision = always_self_provision
+        self._client: Optional[pubsub_v1.SubscriberClient] = None
+        self._subscription_path: Optional[str] = None
+        self._topic_path: Optional[str] = None
+        self._subscription_confirmed = False
+        self._active_future = None  # the StreamingPullFuture from subscribe(), for stop()
+
+        if not self.config.enabled:
+            logger.info(f"{label} disabled via PUBSUB_SDK_ENABLED=false.")
+            return
+
+        try:
+            self._client = pubsub_v1.SubscriberClient()
+            self._subscription_path = self._client.subscription_path(self.config.project_id, subscription_id)
+            self._topic_path = self._client.topic_path(self.config.project_id, topic_id)
+            logger.info(
+                f"{label} initialized (project={self.config.project_id}, "
+                f"subscription={subscription_id}) targeting: {self.config.target_label}"
+            )
+        except Exception as e:
+            self._client = None
+            self._subscription_path = None
+            logger.warning(f"{label} initialization failed: {e}. Subscribing will be disabled.")
+            return
+
+        if always_self_provision:
+            self._subscription_confirmed = self._ensure_subscription_exists()
+
+    def is_available(self) -> bool:
+        base_available = self._client is not None and self._subscription_path is not None
+        if self._always_self_provision:
+            return base_available and self._subscription_confirmed
+        return base_available
+
+    def _ensure_subscription_exists(self) -> bool:
+        """For StepEventSubscriber (always_self_provision=False): emulator-only
+        convenience, never fires against real GCP - production subscriptions are
+        provisioned by infra ahead of time. For a subscriber that opted into
+        always_self_provision (e.g. chat): fires against real GCP too, since a
+        per-instance subscription's identity only exists at runtime and can't be
+        pre-provisioned; tolerates AlreadyExists so a restarted process finds its
+        own prior subscription instead of erroring."""
+        try:
+            # create_subscription()'s real signature only accepts name/topic/
+            # push_config/ack_deadline_seconds as direct keyword arguments - filter
+            # and expiration_policy are Subscription fields only reachable via the
+            # request= dict. A bare filter=/expiration_policy= kwarg raises
+            # TypeError against the real client; a MagicMock in a unit test
+            # wouldn't catch that mismatch, which is exactly how this went
+            # unnoticed until a real-GCP run surfaced it.
+            request: Dict[str, Any] = {"name": self._subscription_path, "topic": self._topic_path}
+            if self._filter_expr:
+                request["filter"] = self._filter_expr
+            if self._expiration_ttl_seconds:
+                request["expiration_policy"] = {"ttl": {"seconds": self._expiration_ttl_seconds}}
+            self._client.create_subscription(request=request)
+            logger.info(f"Auto-created subscription: {self._subscription_path}")
+            return True
+        except AlreadyExists:
+            logger.info(f"Subscription already exists (reused): {self._subscription_path}")
+            return True
+        except Exception as e:
+            logger.error(f"Failed to auto-create subscription {self._subscription_path}: {e}")
+            return False
+
+    def pull_forever(self, on_event: Callable[[T], None]) -> None:
+        """Blocking pull loop. Acks each message only after on_event() returns without
+        raising; an exception from on_event() nacks the message so Pub/Sub redelivers
+        it instead of silently dropping a message the consumer failed to process.
+
+        A missing subscription surfaces only once the StreamingPullFuture
+        subscribe() returns actually resolves/raises (via future.result()),
+        never synchronously from subscribe() itself - the real client always
+        starts the stream and defers the RPC's own errors to that future."""
+        if not self.is_available():
+            logger.error(f"{self._label}.pull_forever: subscriber unavailable, not starting.")
+            return
+
+        def _callback(message) -> None:
+            try:
+                event = self._message_cls.from_pubsub_message(message.data, dict(message.attributes))
+                on_event(event)
+                message.ack()
+            except Exception as e:
+                logger.error(f"{self._label}.pull_forever: handler failed for message, nacking for redelivery: {e}")
+                message.nack()
+
+        subscription_create_attempted = False
+
+        while True:
+            try:
+                future = self._client.subscribe(self._subscription_path, callback=_callback)
+            except Exception as e:
+                logger.error(f"{self._label}.pull_forever: failed to start streaming pull: {e}")
+                return
+
+            self._active_future = future
+            try:
+                future.result()
+                return  # normal completion (e.g. a clean stop() cancelled it)
+            except NotFound as e:
+                if subscription_create_attempted or not self._should_retry_subscribe():
+                    logger.error(f"{self._label}.pull_forever: subscription {self._subscription_path} not found: {e}")
+                    return
+                subscription_create_attempted = True
+                if not self._ensure_subscription_exists():
+                    return
+                continue  # subscribe() again now that the subscription (should) exist
+            except Exception as e:
+                logger.error(f"{self._label}.pull_forever: streaming pull ended: {e}")
+                return
+
+    def _should_retry_subscribe(self) -> bool:
+        """True if a missing subscription is worth one auto-create-and-retry
+        attempt: always for a self-provisioning subscriber (eagerly
+        provisioned at construction already - a NotFound here means that
+        genuinely failed, or the subscription was deleted out from under a
+        long-lived process mid-run), otherwise only against the emulator -
+        production subscriptions are provisioned by infra ahead of time."""
+        return self._always_self_provision or bool(self.config.emulator_host)
+
+    def stop(self) -> None:
+        """Cancels an in-progress pull_forever() streaming pull, if any, so its
+        background thread's blocking future.result() call returns. Idempotent."""
+        if self._active_future is not None:
+            try:
+                self._active_future.cancel()
+            except Exception as e:
+                logger.warning(f"{self._label}.stop: error cancelling streaming pull: {e}")
+
+    def close(self) -> None:
+        if self._client is not None:
+            try:
+                self._client.close()
+            except Exception as e:
+                logger.warning(f"{self._label}.close: error closing client: {e}")
+        self._client = None
+        self._subscription_path = None
+        self._topic_path = None

pubsub_sdk/config.py

@@ -0,0 +1,43 @@
+"""Environment-driven configuration for pubsub_sdk, following this org's existing
+GCP_PROJECT_ID / *_EMULATOR_HOST conventions (see ActivityLoggingService)."""
+import os
+from dataclasses import dataclass
+from typing import Optional
+
+
+def _env_bool(name: str, default: bool) -> bool:
+    value = os.environ.get(name)
+    if value is None:
+        return default
+    return value.strip().lower() in ("1", "true", "yes", "on")
+
+
+@dataclass
+class PubSubConfig:
+    project_id: str
+    topic_id: str
+    subscription_id: str
+    enabled: bool
+    chat_topic_id: str = "agent-chat-messages"
+
+    @classmethod
+    def from_env(cls) -> "PubSubConfig":
+        return cls(
+            project_id=os.environ.get("GCP_PROJECT_ID", "test"),
+            topic_id=os.environ.get("PUBSUB_SDK_TOPIC_ID", "agent-step-events"),
+            subscription_id=os.environ.get(
+                "PUBSUB_SDK_SUBSCRIPTION_ID", "agent-step-events-firestore-persister"
+            ),
+            enabled=_env_bool("PUBSUB_SDK_ENABLED", True),
+            chat_topic_id=os.environ.get("PUBSUB_SDK_CHAT_TOPIC_ID", "agent-chat-messages"),
+        )
+
+    @property
+    def emulator_host(self) -> Optional[str]:
+        return os.environ.get("PUBSUB_EMULATOR_HOST")
+
+    @property
+    def target_label(self) -> str:
+        """Human-readable label for log lines, mirroring ActivityLoggingService's
+        EMULATOR/CLOUD distinction."""
+        return "EMULATOR" if self.emulator_host else "CLOUD"

pubsub_sdk/events.py

@@ -0,0 +1,47 @@
+"""StepEvent: the wire schema published to / consumed from Pub/Sub.
+
+Field names (activity_id, agent_id, event_type, message, status, metadata) are
+deliberately identical to ActivityLoggingService.log_event()'s signature in the
+crowddrop_ai_agents fastapi app, so a consumer reading both sinks isn't reconciling
+two different vocabularies for the same concept.
+"""
+import json
+from dataclasses import asdict, dataclass, field
+from typing import Any, Dict, Optional, Tuple
+
+
+@dataclass
+class StepEvent:
+    activity_id: str
+    agent_id: str
+    event_type: str
+    message: str
+    status: str = "in_progress"
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    sequence: Optional[int] = None
+
+    def to_pubsub_message(self) -> Tuple[bytes, Dict[str, str]]:
+        """JSON body + string attributes, so a subscriber can filter by
+        activity_id/agent_id/event_type/status without deserializing the body."""
+        body = asdict(self)
+        data = json.dumps(body).encode("utf-8")
+        attributes = {
+            "activity_id": self.activity_id,
+            "agent_id": self.agent_id,
+            "event_type": self.event_type,
+            "status": self.status,
+        }
+        return data, attributes
+
+    @classmethod
+    def from_pubsub_message(cls, data: bytes, attributes: Optional[Dict[str, str]] = None) -> "StepEvent":
+        body = json.loads(data.decode("utf-8"))
+        return cls(
+            activity_id=body["activity_id"],
+            agent_id=body["agent_id"],
+            event_type=body["event_type"],
+            message=body["message"],
+            status=body.get("status", "in_progress"),
+            metadata=body.get("metadata") or {},
+            sequence=body.get("sequence"),
+        )

pubsub_sdk/langchain_callback.py

@@ -0,0 +1,227 @@
+"""LangChain callback adapter. The ONLY module in pubsub_sdk that imports langchain —
+a consumer who only wants StepEventPublisher/StepEventSubscriber never pulls this
+dependency in (see the `langchain` optional extra in pyproject.toml).
+
+Method names deliberately mirror FirestoreCallbackHandler's own surface
+(log_mission_start/complete/error, log_directive, _log_event) so a caller holding
+either kind of handler can call the same method name on it without isinstance
+branching.
+"""
+import logging
+from typing import Any, Dict, List, Optional
+
+from langchain_core.callbacks import BaseCallbackHandler
+
+from .config import PubSubConfig
+from .publisher import StepEventPublisher, get_default_publisher
+
+logger = logging.getLogger(__name__)
+
+
+def _preview(value: Any, limit: int = 500) -> str:
+    """Truncate a value for a log/metadata preview without paying the cost of fully
+    materializing a large non-str object first. A plain str is sliced directly (it's
+    already in memory, so this is cheap); a container (dict/list/tuple) - which is
+    where a large embedded payload such as a base64 image data URI would live - gets
+    only a type+length summary instead of a fully-serialized copy, since calling
+    str()/repr() on it would render the entire structure just to discard most of it."""
+    if isinstance(value, str):
+        return value[:limit]
+    if isinstance(value, (dict, list, tuple)):
+        return f"<{type(value).__name__} len={len(value)}>"
+    return str(value)[:limit]
+
+
+class StepEventCallbackHandler(BaseCallbackHandler):
+    """Publishes every LLM/tool/chain/agent lifecycle event to Pub/Sub automatically —
+    no LLM cooperation required. Each event carries a code-level `sequence` ordinal
+    (incremented here, never LLM-supplied): MissionCommander's own tool calls don't
+    map 1:1 onto its persona's numbered directives (several directives share one tool
+    name, others make no tool call at all), so there is no real directive number to
+    report automatically. This ordinal is the honest, zero-prose ordering signal
+    instead of a fabricated directive number.
+    """
+
+    def __init__(
+        self,
+        activity_id: str,
+        agent_id: str,
+        config: Optional[PubSubConfig] = None,
+        publisher: Optional[StepEventPublisher] = None,
+    ):
+        super().__init__()
+        self.activity_id = activity_id
+        self.agent_id = agent_id
+        # Reuses a process-wide singleton client (get_default_publisher) rather than
+        # constructing a fresh PublisherClient/gRPC channel per mission, which would
+        # otherwise leak one per mission over the life of a long-running process.
+        self.publisher = publisher or get_default_publisher(config=config)
+        self._sequence = 0
+
+        if self.publisher.is_available():
+            logger.info(f"StepEventCallbackHandler ready for activity: {activity_id} (agent: {agent_id})")
+        else:
+            logger.info(
+                f"StepEventCallbackHandler initialized but Pub/Sub unavailable for "
+                f"{agent_id} (activity: {activity_id}). Publishing will be disabled."
+            )
+
+    def _next_sequence(self) -> int:
+        self._sequence += 1
+        return self._sequence
+
+    def _log_event(
+        self,
+        event_type: str,
+        message: str,
+        metadata: Optional[Dict[str, Any]] = None,
+        status: str = "in_progress",
+    ) -> None:
+        if not self.publisher.is_available():
+            return
+        try:
+            self.publisher.publish_step(
+                activity_id=self.activity_id,
+                agent_id=self.agent_id,
+                event_type=event_type,
+                message=message,
+                status=status,
+                metadata=metadata,
+                sequence=self._next_sequence(),
+            )
+        except Exception as e:
+            # Never let a logging failure break the agent's own execution.
+            logger.error(f"StepEventCallbackHandler: failed to publish {event_type} for {self.activity_id}: {e}")
+
+    # ------------------------------------------------------------------
+    # LangChain lifecycle hooks — fire automatically, zero LLM cooperation
+    # ------------------------------------------------------------------
+
+    def on_llm_start(self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) -> None:
+        self._log_event(
+            event_type="llm_start",
+            message=f"LLM invocation started with {len(prompts)} prompt(s)",
+            metadata={"model": serialized.get("name", "unknown"), "prompts_count": len(prompts)},
+            status="in_progress",
+        )
+
+    def on_llm_end(self, response, **kwargs: Any) -> None:
+        self._log_event(
+            event_type="llm_end",
+            message="LLM invocation completed",
+            metadata={"generations": len(response.generations) if response.generations else 0},
+            status="success",
+        )
+
+    def on_llm_error(self, error: BaseException, **kwargs: Any) -> None:
+        self._log_event(
+            event_type="llm_error",
+            message=f"LLM error: {str(error)}",
+            metadata={"exception_type": type(error).__name__, "exception_details": str(error)},
+            status="error",
+        )
+
+    def on_tool_start(self, serialized: Dict[str, Any], input_str: str, **kwargs: Any) -> None:
+        tool_name = serialized.get("name", "unknown_tool")
+        self._log_event(
+            event_type="tool_start",
+            message=f"Invoking tool: {tool_name}",
+            metadata={"tool_name": tool_name, "tool_input": _preview(input_str)},
+            status="in_progress",
+        )
+
+    def on_tool_end(self, output: Any, **kwargs: Any) -> None:
+        self._log_event(
+            event_type="tool_end",
+            message="Tool execution completed",
+            metadata={"output_preview": _preview(output)},
+            status="success",
+        )
+
+    def on_tool_error(self, error: BaseException, **kwargs: Any) -> None:
+        self._log_event(
+            event_type="tool_error",
+            message=f"Tool error: {str(error)}",
+            metadata={"exception_type": type(error).__name__, "exception_details": str(error)},
+            status="error",
+        )
+
+    def on_agent_action(self, action, **kwargs: Any) -> None:
+        action_log = action.log[:500] if hasattr(action, "log") else None
+        self._log_event(
+            event_type="agent_action",
+            message=f"Agent decision: {action.tool}",
+            metadata={
+                "tool": action.tool,
+                "tool_input": _preview(action.tool_input),
+                "log": action_log,
+            },
+            status="in_progress",
+        )
+
+    def on_agent_finish(self, finish, **kwargs: Any) -> None:
+        output = finish.return_values.get("output", "") if hasattr(finish, "return_values") else ""
+        self._log_event(
+            event_type="agent_finish",
+            message="Agent execution completed successfully",
+            metadata={"output_preview": _preview(output)},
+            status="success",
+        )
+
+    def on_chain_start(self, serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) -> None:
+        chain_name = serialized.get("name", "unknown_chain") if serialized else "unknown_chain"
+        self._log_event(
+            event_type="chain_start",
+            message=f"Chain started: {chain_name}",
+            metadata={"chain_name": chain_name, "inputs": _preview(inputs) if inputs else "no_inputs"},
+            status="in_progress",
+        )
+
+    def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None:
+        self._log_event(
+            event_type="chain_end",
+            message="Chain execution completed",
+            metadata={"outputs": _preview(outputs)},
+            status="success",
+        )
+
+    def on_chain_error(self, error: BaseException, **kwargs: Any) -> None:
+        self._log_event(
+            event_type="chain_error",
+            message=f"Chain error: {str(error)}",
+            metadata={"exception_type": type(error).__name__, "exception_details": str(error)},
+            status="error",
+        )
+
+    # ------------------------------------------------------------------
+    # Convenience methods — mirror FirestoreCallbackHandler's surface exactly
+    # ------------------------------------------------------------------
+
+    def log_mission_start(self) -> None:
+        self._log_event(
+            event_type="mission_start",
+            message=f"Mission {self.activity_id} started by {self.agent_id}",
+            status="in_progress",
+        )
+
+    def log_mission_complete(self, summary: str = "Mission completed successfully") -> None:
+        self._log_event(event_type="mission_complete", message=summary, status="success")
+
+    def log_mission_error(self, summary: str = "Mission failed") -> None:
+        self._log_event(event_type="mission_error", message=summary, status="error")
+
+    def log_directive(self, directive_number: int, directive_text: str, status: str = "started") -> None:
+        """Kept for API symmetry with FirestoreCallbackHandler; unused by the automatic
+        path today (MissionCommander's directives don't map 1:1 onto discrete tool
+        calls, so there is no real directive number for the automatic hooks above to
+        report). Available for a future caller that does have a real directive
+        number to report."""
+        event_type = f"directive_{status}"
+        message = f"Directive {directive_number}: {directive_text}"
+        log_status = "success" if status == "completed" else "in_progress" if status == "started" else "error"
+        self._log_event(
+            event_type=event_type,
+            message=message,
+            metadata={"directive_number": directive_number, "directive_status": status},
+            status=log_status,
+        )

pubsub_sdk/publisher.py

@@ -0,0 +1,61 @@
+"""StepEventPublisher: the framework-agnostic core client. Zero LangChain dependency —
+this is the part any future embodied-AI-agent developer imports directly, with no
+tie to any particular agent framework or storage backend.
+
+A thin GenericPublisher[StepEvent] subclass (see client.py) - construction,
+degrade-gracefully behavior, and emulator-only topic auto-create all live there
+now; this module keeps the `pubsub_v1` import so existing tests patching
+`pubsub_sdk.publisher.pubsub_v1.PublisherClient` keep resolving correctly even
+though client construction happens in the parent class.
+"""
+import logging
+from typing import Any, Dict, Optional
+
+from google.cloud import pubsub_v1  # noqa: F401 - kept for existing test patch targets
+
+from .client import GenericPublisher
+from .config import PubSubConfig
+from .events import StepEvent
+
+logger = logging.getLogger(__name__)
+
+
+class StepEventPublisher(GenericPublisher[StepEvent]):
+    def __init__(self, config: Optional[PubSubConfig] = None):
+        cfg = config or PubSubConfig.from_env()
+        super().__init__(config=cfg, topic_id=cfg.topic_id, label="StepEventPublisher")
+
+    def publish_step(
+        self,
+        activity_id: str,
+        agent_id: str,
+        event_type: str,
+        message: str,
+        status: str = "in_progress",
+        metadata: Optional[Dict[str, Any]] = None,
+        sequence: Optional[int] = None,
+    ) -> bool:
+        event = StepEvent(
+            activity_id=activity_id,
+            agent_id=agent_id,
+            event_type=event_type,
+            message=message,
+            status=status,
+            metadata=metadata or {},
+            sequence=sequence,
+        )
+        return self.publish(event)
+
+
+_default_publisher: Optional[StepEventPublisher] = None
+
+
+def get_default_publisher(config: Optional[PubSubConfig] = None) -> StepEventPublisher:
+    """Process-wide singleton, mirroring get_activity_logging_service()'s existing
+    pattern in this org's codebase: one long-lived Pub/Sub client (and its gRPC
+    channel) per process instead of one per mission/callback instance, which would
+    otherwise leak a channel on every mission with nothing to close it."""
+    global _default_publisher
+    if _default_publisher is None:
+        _default_publisher = StepEventPublisher(config=config)
+    return _default_publisher

pubsub_sdk/subscriber.py

@@ -0,0 +1,33 @@
+"""StepEventSubscriber: the generic pull-consumer primitive. Zero Firestore, zero
+LangChain, zero "logging" concept — it just turns Pub/Sub messages back into
+StepEvents and hands them to a callback. This is what keeps pubsub_sdk reusable
+for purposes beyond logging (agent-to-agent chat, see chat.py).
+
+A thin GenericSubscriber[StepEvent] subclass (see client.py) - construction,
+degrade-gracefully behavior, and emulator-only subscription auto-create all
+live there now; this module keeps the `pubsub_v1` import so existing tests
+patching `pubsub_sdk.subscriber.pubsub_v1.SubscriberClient` keep resolving
+correctly even though client construction happens in the parent class.
+"""
+import logging
+from typing import Optional
+
+from google.cloud import pubsub_v1  # noqa: F401 - kept for existing test patch targets
+
+from .client import GenericSubscriber
+from .config import PubSubConfig
+from .events import StepEvent
+
+logger = logging.getLogger(__name__)
+
+
+class StepEventSubscriber(GenericSubscriber[StepEvent]):
+    def __init__(self, config: Optional[PubSubConfig] = None):
+        cfg = config or PubSubConfig.from_env()
+        super().__init__(
+            config=cfg,
+            topic_id=cfg.topic_id,
+            subscription_id=cfg.subscription_id,
+            message_cls=StepEvent,
+            label="StepEventSubscriber",
+        )