@gokiteam/goki-dev 0.2.0

package/src/emulation/pubsub/README.md

@@ -0,0 +1,309 @@
+# GCP Pub/Sub Emulation API
+
+This module provides a Google Cloud Pub/Sub emulator that's compatible with the official `@google-cloud/pubsub` SDK.
+
+## Overview
+
+The Pub/Sub emulator runs on port **8085** and implements the GCP Cloud Pub/Sub REST API v1, allowing you to develop and test applications locally without connecting to actual GCP services.
+
+## Features
+
+- **GCP-Compatible REST API**: Implements the official Pub/Sub v1 REST API
+- **Full Topic Management**: Create, get, list, and delete topics
+- **Subscription Support**: Pull subscriptions with acknowledgment
+- **Message Publishing**: Publish single or batch messages
+- **Persistent Storage**: Messages stored in memory with JSON file persistence
+- **SDK Compatible**: Works with `@google-cloud/pubsub` Node.js SDK
+
+## Quick Start
+
+### 1. Start the Server
+
+```bash
+npm run dev
+```
+
+The Pub/Sub emulator will start on port 8085.
+
+### 2. Configure Your Application
+
+Set the emulator host environment variable:
+
+```bash
+export PUBSUB_EMULATOR_HOST=localhost:8085
+```
+
+### 3. Use the Official SDK
+
+```javascript
+import { PubSub } from '@google-cloud/pubsub'
+
+const pubsub = new PubSub({
+  projectId: 'my-project'
+})
+
+// Create a topic
+const [topic] = await pubsub.createTopic('my-topic')
+
+// Publish a message
+await topic.publishMessage({ data: Buffer.from('Hello World') })
+
+// Create a subscription
+const [subscription] = await topic.createSubscription('my-subscription')
+
+// Pull messages
+const [messages] = await subscription.pull({ maxMessages: 10 })
+
+// Acknowledge messages
+messages.forEach(message => {
+  console.log(message.data.toString())
+  message.ack()
+})
+```
+
+## REST API Endpoints
+
+### Topics
+
+#### Create Topic
+```
+PUT /v1/projects/{project}/topics/{topic}
+```
+
+**Response:**
+```json
+{
+  "name": "projects/my-project/topics/my-topic",
+  "labels": {},
+  "messageStoragePolicy": {},
+  "messageRetentionDuration": "604800s"
+}
+```
+
+#### Get Topic
+```
+GET /v1/projects/{project}/topics/{topic}
+```
+
+#### List Topics
+```
+GET /v1/projects/{project}/topics
+```
+
+**Query Parameters:**
+- `pageSize` (optional): Maximum number of topics to return
+- `pageToken` (optional): Token for pagination
+
+#### Delete Topic
+```
+DELETE /v1/projects/{project}/topics/{topic}
+```
+
+#### Publish Messages
+```
+POST /v1/projects/{project}/topics/{topic}:publish
+```
+
+**Request Body:**
+```json
+{
+  "messages": [
+    {
+      "data": "SGVsbG8gV29ybGQ=",
+      "attributes": {
+        "key": "value"
+      },
+      "orderingKey": ""
+    }
+  ]
+}
+```
+
+**Response:**
+```json
+{
+  "messageIds": [
+    "550e8400-e29b-41d4-a716-446655440000"
+  ]
+}
+```
+
+### Subscriptions
+
+#### Create Subscription
+```
+PUT /v1/projects/{project}/subscriptions/{subscription}
+```
+
+**Request Body:**
+```json
+{
+  "topic": "projects/my-project/topics/my-topic",
+  "ackDeadlineSeconds": 10,
+  "pushConfig": {}
+}
+```
+
+#### Get Subscription
+```
+GET /v1/projects/{project}/subscriptions/{subscription}
+```
+
+#### List Subscriptions
+```
+GET /v1/projects/{project}/subscriptions
+```
+
+#### Delete Subscription
+```
+DELETE /v1/projects/{project}/subscriptions/{subscription}
+```
+
+#### Pull Messages
+```
+POST /v1/projects/{project}/subscriptions/{subscription}:pull
+```
+
+**Request Body:**
+```json
+{
+  "maxMessages": 100,
+  "returnImmediately": false
+}
+```
+
+**Response:**
+```json
+{
+  "receivedMessages": [
+    {
+      "ackId": "550e8400-e29b-41d4-a716-446655440001",
+      "message": {
+        "data": "SGVsbG8gV29ybGQ=",
+        "attributes": {
+          "key": "value"
+        },
+        "messageId": "550e8400-e29b-41d4-a716-446655440000",
+        "publishTime": "2026-02-06T10:30:00.000Z",
+        "orderingKey": ""
+      }
+    }
+  ]
+}
+```
+
+#### Acknowledge Messages
+```
+POST /v1/projects/{project}/subscriptions/{subscription}:acknowledge
+```
+
+**Request Body:**
+```json
+{
+  "ackIds": [
+    "550e8400-e29b-41d4-a716-446655440001"
+  ]
+}
+```
+
+## Storage
+
+The emulator uses a hybrid memory + JSON storage approach:
+
+- **In-Memory**: All operations are performed in memory for fast access
+- **Persistent**: Data is automatically saved to JSON files every 5 seconds
+- **Collections**:
+  - `data/pubsub/topics.json` - Topic metadata
+  - `data/pubsub/subscriptions.json` - Subscription configurations
+  - `data/pubsub/messages.json` - Published messages
+
+## Error Responses
+
+All errors follow the GCP error format:
+
+```json
+{
+  "error": {
+    "code": 404,
+    "message": "Topic not found",
+    "status": "NOT_FOUND"
+  }
+}
+```
+
+### Error Codes
+
+- `400 INVALID_ARGUMENT`: Invalid request parameters
+- `404 NOT_FOUND`: Resource not found
+- `409 ALREADY_EXISTS`: Resource already exists
+- `500 INTERNAL`: Internal server error
+
+## Testing
+
+### Unit Tests
+
+Run the unit tests (no server required):
+
+```bash
+npm run test:emulation
+```
+
+All business logic tests will pass without the server running.
+
+### Integration Tests
+
+To run the HTTP integration tests:
+
+1. Start the server in one terminal:
+```bash
+npm run dev
+```
+
+2. Run the tests in another terminal:
+```bash
+npm run test:emulation
+```
+
+The integration tests verify the HTTP endpoints are working correctly.
+
+## Architecture
+
+```
+src/emulation/pubsub/
+├── Server.js         # Koa server on port 8085
+├── Router.js         # REST API routes
+├── Controllers.js    # Request handlers
+└── Logic.js          # Business logic
+```
+
+## Differences from Production GCP Pub/Sub
+
+This emulator implements core Pub/Sub functionality for local development. Some advanced features are not implemented:
+
+- Push subscriptions (only pull supported)
+- Dead letter queues
+- Message ordering (accepted but not enforced)
+- Schemas
+- Snapshots
+- Seeking
+
+For most development and testing scenarios, the implemented features are sufficient.
+
+## Integration with Goki Services
+
+### device-simulator
+
+```bash
+# config.standalone
+PUBSUB_EMULATOR_HOST=localhost:8085
+```
+
+### core-key
+
+The `@gokiteam/pubsub` package automatically detects the `PUBSUB_EMULATOR_HOST` environment variable and connects to the emulator.
+
+## Related Documentation
+
+- [GCP Pub/Sub REST API Reference](https://cloud.google.com/pubsub/docs/reference/rest)
+- [Platform Specification](../../../plans/platform-specification.md)
+- [MemoryJsonStore](../../singletons/MemoryJsonStore.js)

package/src/emulation/pubsub/Router.js

@@ -0,0 +1,33 @@
+import KoaRouter from 'koa-router'
+import {
+  createTopicController,
+  getTopicController,
+  listTopicsController,
+  deleteTopicController,
+  publishMessagesController,
+  createSubscriptionController,
+  getSubscriptionController,
+  listSubscriptionsController,
+  deleteSubscriptionController,
+  pullMessagesController,
+  acknowledgeMessagesController
+} from './Controllers.js'
+
+const router = new KoaRouter()
+
+// Topic routes
+router.put('/v1/projects/:project/topics/:topic', createTopicController)
+router.get('/v1/projects/:project/topics/:topic', getTopicController)
+router.get('/v1/projects/:project/topics', listTopicsController)
+router.delete('/v1/projects/:project/topics/:topic', deleteTopicController)
+router.post('/v1/projects/:project/topics/:topic\\:publish', publishMessagesController)
+
+// Subscription routes
+router.put('/v1/projects/:project/subscriptions/:subscription', createSubscriptionController)
+router.get('/v1/projects/:project/subscriptions/:subscription', getSubscriptionController)
+router.get('/v1/projects/:project/subscriptions', listSubscriptionsController)
+router.delete('/v1/projects/:project/subscriptions/:subscription', deleteSubscriptionController)
+router.post('/v1/projects/:project/subscriptions/:subscription\\:pull', pullMessagesController)
+router.post('/v1/projects/:project/subscriptions/:subscription\\:acknowledge', acknowledgeMessagesController)
+
+export const Router = router

package/src/emulation/pubsub/Server.js

@@ -0,0 +1,104 @@
+import Koa from 'koa'
+import bodyParser from 'koa-bodyparser'
+import { Application } from '../../configs/Application.js'
+import { Logger } from '../../singletons/Logger.js'
+import { SqliteStore } from '../../singletons/SqliteStore.js'
+import { Router } from './Router.js'
+import { shadowSubscriptionManager } from './ShadowSubscriptionManager.js'
+import { shadowPoller } from './ShadowPoller.js'
+
+const { environment, runId, ports } = Application
+
+/**
+ * Initialize Pub/Sub collections (no-op for SQLite - tables created at DB initialization)
+ */
+const initializeCollections = () => {
+  Logger.log({
+    level: 'info',
+    message: 'Pub/Sub tables ready (SQLite)',
+    data: { tables: ['pubsub_topics', 'pubsub_subscriptions', 'pubsub_messages'] }
+  })
+}
+
+/**
+ * Start Pub/Sub emulation server
+ */
+export const startPubSubServer = async () => {
+  try {
+    // Initialize collections
+    initializeCollections()
+
+    const app = new Koa()
+
+    // Basic error handler
+    app.use(async (ctx, next) => {
+      try {
+        await next()
+      } catch (error) {
+        Logger.log({
+          level: 'error',
+          message: 'Pub/Sub server error',
+          data: { error: error.message, stack: error.stack, path: ctx.path }
+        })
+        ctx.status = error.status || 500
+        ctx.body = {
+          error: {
+            code: ctx.status,
+            message: error.message || 'Internal server error',
+            status: 'INTERNAL'
+          }
+        }
+      }
+    })
+
+    // Body parser
+    app.use(bodyParser())
+
+    // Logger middleware
+    app.use(async (ctx, next) => {
+      const start = Date.now()
+      await next()
+      const ms = Date.now() - start
+      Logger.log({
+        level: 'debug',
+        message: 'Pub/Sub request',
+        data: {
+          method: ctx.method,
+          path: ctx.path,
+          status: ctx.status,
+          duration: `${ms}ms`
+        }
+      })
+    })
+
+    // Routes
+    app.use(Router.routes())
+    app.use(Router.allowedMethods())
+
+    const server = app.listen(ports.pubsub, () => {
+      Logger.log({
+        level: 'info',
+        message: 'Pub/Sub emulation server started',
+        data: {
+          environment,
+          runId,
+          port: ports.pubsub,
+          endpoint: `http://localhost:${ports.pubsub}`
+        }
+      })
+    })
+
+    // Start shadow subscription manager and poller for message capture
+    await shadowSubscriptionManager.start()
+    await shadowPoller.start()
+
+    return server
+  } catch (error) {
+    Logger.log({
+      level: 'error',
+      message: 'Failed to start Pub/Sub emulation server',
+      data: { error: error.message, stack: error.stack }
+    })
+    throw error
+  }
+}

package/src/emulation/pubsub/ShadowPoller.js

@@ -0,0 +1,276 @@
+import { v4 as uuid } from 'uuid'
+import { Application } from '../../configs/Application.js'
+import { Logger } from '../../singletons/Logger.js'
+import { SqliteStore } from '../../singletons/SqliteStore.js'
+import { MessageBroadcaster } from '../../singletons/MessageBroadcaster.js'
+import { FunctionTriggerDispatcher } from '../../singletons/FunctionTriggerDispatcher.js'
+import { shadowSubscriptionManager } from './ShadowSubscriptionManager.js'
+
+const { pubsub } = Application
+
+/**
+ * Extract sender from message attributes
+ */
+const extractSender = attributes => {
+  if (!attributes) return null
+  return attributes.sender || attributes.source || attributes.userId || null
+}
+
+/**
+ * Check if a message has meaningful data worth storing in history.
+ * Filters out empty payloads like {} or empty strings.
+ */
+const hasMeaningfulData = data => {
+  if (!data) return false
+  try {
+    const decoded = Buffer.from(data, 'base64').toString('utf-8')
+    const trimmed = decoded.trim()
+    if (!trimmed || trimmed === '{}' || trimmed === '""' || trimmed === 'null') return false
+    return true
+  } catch {
+    return !!data
+  }
+}
+
+/**
+ * Polls shadow subscriptions to capture messages for logging and UI display.
+ */
+export class ShadowPoller {
+  constructor () {
+    this.baseUrl = `http://${pubsub.emulatorHost}:${pubsub.emulatorPort}`
+    this.projectId = pubsub.projectId
+    this.pollIntervalMs = pubsub.shadowPollIntervalMs
+    this.timeoutId = null
+    this.isRunning = false
+    this.consecutiveEmpty = 0
+    this.processedMessageIds = new Set()
+    this.maxProcessedIds = 10000
+  }
+
+  async request (method, path, body = null) {
+    const url = `${this.baseUrl}${path}`
+    const options = {
+      method,
+      headers: { 'Content-Type': 'application/json' }
+    }
+    if (body) {
+      options.body = JSON.stringify(body)
+    }
+    const response = await fetch(url, options)
+    const text = await response.text()
+    if (!text) return { success: response.ok, status: response.status }
+    try {
+      return JSON.parse(text)
+    } catch {
+      return { success: response.ok, status: response.status, text }
+    }
+  }
+
+  async pullMessages (subscriptionName, maxMessages = 100) {
+    try {
+      const result = await this.request(
+        'POST',
+        `/v1/${subscriptionName}:pull`,
+        { maxMessages, returnImmediately: true }
+      )
+      return result.receivedMessages || []
+    } catch (error) {
+      return []
+    }
+  }
+
+  async acknowledgeMessages (subscriptionName, ackIds) {
+    if (!ackIds || ackIds.length === 0) return
+    try {
+      await this.request(
+        'POST',
+        `/v1/${subscriptionName}:acknowledge`,
+        { ackIds }
+      )
+    } catch (error) {
+      Logger.log({
+        level: 'warn',
+        message: 'Failed to acknowledge messages',
+        data: { subscriptionName, error: error.message }
+      })
+    }
+  }
+
+  ensureTopicExists (topicName) {
+    const existingTopic = SqliteStore.get('pubsub_topics', topicName, 'name')
+    if (!existingTopic) {
+      SqliteStore.create('pubsub_topics', {
+        name: topicName,
+        labels: '{}',
+        message_storage_policy: '{}',
+        kms_key_name: '',
+        schema_settings: null,
+        satisfies_pzs: 0,
+        message_retention_duration: '604800s'
+      })
+    }
+  }
+
+  storeMessage (message, topic) {
+    const { messageId, data, attributes, publishTime } = message
+    if (this.processedMessageIds.has(messageId)) {
+      return false
+    }
+    this.processedMessageIds.add(messageId)
+    // Skip messages with empty/meaningless data
+    if (!hasMeaningfulData(data)) {
+      return false
+    }
+    if (this.processedMessageIds.size > this.maxProcessedIds) {
+      const idsArray = Array.from(this.processedMessageIds)
+      this.processedMessageIds = new Set(idsArray.slice(-this.maxProcessedIds / 2))
+    }
+    try {
+      this.ensureTopicExists(topic)
+    } catch (error) {
+      // Topic may already exist, ignore
+    }
+    // Validate and normalize publishTime
+    const validPublishTime = publishTime && !isNaN(new Date(publishTime).getTime())
+      ? publishTime
+      : new Date().toISOString()
+    const historyRecord = {
+      id: uuid(),
+      message_id: messageId,
+      data,
+      attributes: attributes || {},
+      ordering_key: '',
+      publish_time: validPublishTime,
+      topic,
+      sender: extractSender(attributes),
+      view_count: 0,
+      last_viewed_at: null
+    }
+    try {
+      SqliteStore.create('pubsub_message_history', historyRecord)
+    } catch (error) {
+      if (!error.message?.includes('UNIQUE constraint failed')) {
+        Logger.log({
+          level: 'warn',
+          message: 'Failed to store message in history',
+          data: { messageId, error: error.message }
+        })
+      }
+      return false
+    }
+    MessageBroadcaster.broadcast({
+      messageId,
+      topic,
+      data,
+      attributes: attributes || {},
+      publishTime: validPublishTime,
+      sender: extractSender(attributes)
+    })
+    FunctionTriggerDispatcher.onPubSubMessage(
+      { messageId, data, attributes, publishTime: validPublishTime },
+      topic
+    ).catch(err => {
+      Logger.log({ level: 'error', message: 'Cloud Function dispatch error', data: { error: err.message } })
+    })
+    return true
+  }
+
+  getTopicForSubscription (subscriptionName) {
+    // First try to get from the shadow subscription manager's mapping
+    const mappedTopic = shadowSubscriptionManager.getTopicForShadow(subscriptionName)
+    if (mappedTopic) {
+      return mappedTopic
+    }
+    // Fallback: extract from subscription name (less reliable)
+    const subId = subscriptionName.split('/').pop()
+    const originalSubId = subId.replace('-devtools-shadow', '')
+    const topicId = originalSubId.replace(/-[^-]+-sub$/, '')
+    return `projects/${this.projectId}/topics/${topicId}`
+  }
+
+  async pollAllShadowSubscriptions () {
+    let totalMessages = 0
+    const shadowSubscriptions = shadowSubscriptionManager.getShadowSubscriptions()
+    for (const subName of shadowSubscriptions) {
+      try {
+        const messages = await this.pullMessages(subName)
+        if (messages.length === 0) continue
+        totalMessages += messages.length
+        const ackIds = []
+        for (const receivedMessage of messages) {
+          const { ackId, message } = receivedMessage
+          const topic = this.getTopicForSubscription(subName)
+          this.storeMessage(message, topic)
+          ackIds.push(ackId)
+        }
+        if (ackIds.length > 0) {
+          await this.acknowledgeMessages(subName, ackIds)
+        }
+      } catch (error) {
+        Logger.log({
+          level: 'debug',
+          message: 'Error polling shadow subscription',
+          data: { subscriptionName: subName, error: error.message }
+        })
+      }
+    }
+    return totalMessages
+  }
+
+  getNextPollDelay () {
+    const idleThreshold = 5
+    if (this.consecutiveEmpty < idleThreshold) {
+      return this.pollIntervalMs
+    }
+    return Math.min(this.pollIntervalMs * 10, 2000)
+  }
+
+  async schedulePoll () {
+    if (!this.isRunning) return
+    try {
+      const found = await this.pollAllShadowSubscriptions()
+      if (found > 0) {
+        this.consecutiveEmpty = 0
+      } else {
+        this.consecutiveEmpty++
+      }
+    } catch (error) {
+      Logger.log({
+        level: 'error',
+        message: 'Error in shadow poll cycle',
+        data: { error: error.message }
+      })
+    }
+    if (this.isRunning) {
+      this.timeoutId = setTimeout(() => this.schedulePoll(), this.getNextPollDelay())
+    }
+  }
+
+  async start () {
+    if (this.isRunning) return
+    this.isRunning = true
+    this.consecutiveEmpty = 0
+    Logger.log({
+      level: 'info',
+      message: 'Starting Shadow Poller',
+      data: {
+        pollIntervalMs: this.pollIntervalMs
+      }
+    })
+    this.schedulePoll()
+  }
+
+  stop () {
+    if (this.timeoutId) {
+      clearTimeout(this.timeoutId)
+      this.timeoutId = null
+    }
+    this.isRunning = false
+    Logger.log({
+      level: 'info',
+      message: 'Shadow Poller stopped'
+    })
+  }
+}
+
+export const shadowPoller = new ShadowPoller()