@gokiteam/goki-dev 0.2.0

package/README.md

@@ -0,0 +1,478 @@
+# @gokiteam/goki-dev
+
+**"LocalStack for Goki Services"**
+
+A unified local development platform providing mock implementations of external services (GCP Pub/Sub, AWS IoT MQTT, Redis, PostgreSQL, Google Cloud Logging, Firestore) with a centralized Web UI.
+
+## Install
+
+```bash
+# Global install (CLI available everywhere)
+npm install -g @gokiteam/goki-dev
+
+# Or as a project dev dependency
+npm install --save-dev @gokiteam/goki-dev
+```
+
+## Usage
+
+### CLI — Manage local infrastructure
+
+```bash
+# Interactive menu
+goki-dev
+
+# Start all services (Docker)
+goki-dev start
+
+# Stop services
+goki-dev stop
+
+# Check status
+goki-dev status
+
+# MCP server for Claude Code integration
+goki-dev-mcp
+```
+
+### Programmatic — DevToolsClient for tests
+
+```javascript
+import { DevToolsClient } from '@gokiteam/goki-dev'
+
+const devTools = new DevToolsClient()
+const traceId = devTools.generateTraceId('user-reg')
+
+// Verify Pub/Sub messages
+await devTools.pubsub.waitForMessage({
+  filter: { topic: 'user-events' },
+  timeout: 5000,
+  traceId
+})
+
+// Query PostgreSQL
+await devTools.postgres.query({
+  database: 'auth_dev',
+  query: 'SELECT * FROM users',
+  traceId
+})
+
+// Check Redis
+await devTools.redis.get({ key: 'session:user-001', traceId })
+
+// Assert no errors in logs
+const errors = await devTools.logging.assertNoErrors({ traceId })
+```
+
+Or use the pre-configured singleton:
+
+```javascript
+import { devTools } from '@gokiteam/goki-dev'
+
+const traceId = devTools.generateTraceId('test')
+const message = await devTools.pubsub.waitForMessage({ ... })
+```
+
+### Docker (Direct)
+
+```bash
+docker compose up -d
+docker compose logs -f
+docker compose down
+```
+
+## Access
+
+### Production (Docker)
+- **Web UI:** http://localhost:9001 (nginx + React)
+- **Backend API:** http://localhost:9000/v1/*
+- **Pub/Sub Emulator:** http://localhost:8085
+- **AWS IoT Core HTTPS API:** http://localhost:8086 (POST /topics/:topicName)
+- **Firestore Emulator:** http://localhost:8081
+- **Cloud Logging:** http://localhost:8087
+- **PostgreSQL:** localhost:5432 (user: postgres, password: postgres)
+- **Redis:** localhost:6379
+- **MQTT Broker:** localhost:8883
+
+### Development
+- **Web UI (dev):** http://localhost:9001 (React dev server)
+- **Backend API (dev):** http://localhost:9000/v1/*
+
+## Architecture
+
+The platform uses a **microservices architecture** with separate frontend and backend containers:
+
+### Production Architecture (Docker)
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     Docker Network (goki-network)            │
+├─────────────────────────────────────────────────────────────┤
+│                                                               │
+│  ┌──────────────────┐         ┌───────────────────┐         │
+│  │  Frontend (9001) │         │  Backend (9000)   │         │
+│  │  nginx + React   │◄────────│  Node.js API      │         │
+│  │                  │   API   │  + Emulators      │         │
+│  └──────────────────┘  calls  └─────────┬─────────┘         │
+│                                          │                    │
+│                                          │                    │
+│         ┌────────────────────────────────┼────────┐          │
+│         │                                │        │          │
+│         ▼                                ▼        ▼          │
+│  ┌─────────────┐  ┌───────────────┐  ┌──────┐  ┌──────┐   │
+│  │ Pub/Sub     │  │ Firestore     │  │Redis │  │Postgres│  │
+│  │ (8085)      │  │ (8081)        │  │(6379)│  │(5432)│    │
+│  └─────────────┘  └───────────────┘  └──────┘  └──────┘    │
+│                                                               │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Services
+
+**Frontend Container (`dev-tools-frontend`):**
+- **Base:** nginx:1.25-alpine
+- **Port:** 9001 → 80 (nginx)
+- **Purpose:** Serve React UI with optimized caching and compression
+- **Features:**
+  - Gzip compression
+  - 1-year cache for static assets
+  - No-cache for index.html
+  - Security headers
+  - SPA routing support
+
+**Backend Container (`dev-tools-backend`):**
+- **Base:** node:20-alpine + Java 21
+- **Ports:** 9000 (API), 8085 (Pub/Sub), 8086 (AWS IoT), 8087 (Logging), 8883 (MQTT)
+- **Purpose:** API server and emulators
+- **Features:**
+  - RESTful API endpoints
+  - GCP Pub/Sub emulator
+  - AWS IoT Core HTTPS API emulation
+  - Google Cloud Logging emulator
+  - MQTT broker (Aedes) with Pub/Sub bridge
+  - SQLite for Pub/Sub/Logging persistence
+
+**Dependencies:**
+- PostgreSQL 15 (port 5432)
+- Redis 7 (port 6379)
+- Pub/Sub Emulator (port 8085)
+- Firestore Emulator (port 8081)
+
+### Development Architecture
+
+- **Backend:** Hot-reload with nodemon
+- **Frontend:** React dev server with fast refresh
+- Separate containers for each service
+- Volume mounts for live code updates
+
+## API Endpoints
+
+### Health
+- `GET /v1/health/liveness` - Service is alive
+- `GET /v1/health/readiness` - Service is ready to handle requests
+
+### Dashboard
+- `POST /v1/dashboard/stats` - Get platform statistics
+
+### Services
+- `POST /v1/services/list` - List all services
+- `POST /v1/services/status` - Get service status
+- `POST /v1/services/restart` - Restart a service
+
+### Pub/Sub
+- `POST /v1/pubsub/topics/list` - List topics
+- `POST /v1/pubsub/topics/create` - Create topic
+- `POST /v1/pubsub/messages/publish` - Publish message
+- `POST /v1/pubsub/messages/pull` - Pull messages
+- `GET /v1/pubsub/messages/stream` - Stream messages (SSE)
+
+### PostgreSQL
+- `POST /v1/postgres/connection/test` - Test connection
+- `POST /v1/postgres/databases/list` - List databases
+- `POST /v1/postgres/tables/list` - List tables
+- `POST /v1/postgres/rows/list` - List rows
+- `POST /v1/postgres/query/execute` - Execute query
+
+### Redis
+- `POST /v1/redis/connection/test` - Test connection
+- `POST /v1/redis/info` - Get info
+- `POST /v1/redis/keys/scan` - Scan keys
+- `POST /v1/redis/keys/get` - Get key value
+- `POST /v1/redis/keys/update` - Update key
+- `POST /v1/redis/keys/delete` - Delete key
+
+### Firestore
+- `POST /v1/firestore/collections/list` - List collections
+- `POST /v1/firestore/documents/list` - List documents
+- `POST /v1/firestore/documents/get` - Get document
+- `POST /v1/firestore/query/execute` - Execute query
+
+### Export/Import
+- `POST /v1/data/export` - Export all data
+- `POST /v1/data/import` - Import data
+- `POST /v1/data/clear` - Clear all data
+
+## MQTT and AWS IoT Core Emulation
+
+The platform provides full MQTT broker and AWS IoT Core HTTPS API emulation for local development and testing with backend services like device-simulator.
+
+### Architecture (Docker)
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     Docker Network (goki-network)            │
+├─────────────────────────────────────────────────────────────┤
+│                                                               │
+│  device-simulator container                                  │
+│    ↓ [MQTT to dev-tools-backend:8883]                       │
+│  dev-tools-backend: Aedes MQTT Broker                       │
+│    ↓ [Pub/Sub Bridge]                                        │
+│  pubsub-emulator (port 8085)                                 │
+│    ↓ [Topic: mqttMessageReceived]                           │
+│  device-simulator (subscribes via pubsub-emulator:8085)     │
+│                                                               │
+│  device-simulator                                            │
+│    ↓ [POST to dev-tools-backend:8086/topics/...?qos=1]     │
+│  dev-tools-backend: AWS IoT Core HTTPS API                   │
+│    ↓ [Publishes to Aedes broker]                            │
+│  device-simulator (subscribed to response topic via MQTT)   │
+│                                                               │
+└─────────────────────────────────────────────────────────────┘
+```
+
+**Key Service Names (Docker networking):**
+- `dev-tools-backend` - MQTT broker (8883) + AWS IoT API (8086)
+- `pubsub-emulator` - Pub/Sub emulator (8085)
+- `redis` - Redis (6379)
+- `postgres` - PostgreSQL (5432)
+
+**From Host Machine:** Use `localhost` instead of service names
+
+### MQTT Broker (Port 8883)
+
+- **Implementation:** Aedes MQTT broker (pure Node.js)
+- **Protocol:** MQTT 3.1.1
+- **Authentication:** Relaxed mode (accept all connections for dev)
+- **Features:**
+  - QoS levels 0, 1
+  - Last Will Testament (LWT)
+  - Retained messages
+  - Persistent connections
+
+**Connect with MQTT client:**
+
+From Docker container (using service name):
+```javascript
+import mqtt from 'mqtt'
+
+const client = mqtt.connect('mqtt://dev-tools-backend:8883', {
+  clientId: 'test-device-123'
+})
+
+client.on('connect', () => {
+  client.subscribe('rs/+/test-device-123')
+  client.publish('rq/test/test-device-123', Buffer.from('packet'), { qos: 1 })
+})
+```
+
+From host machine (using localhost):
+```javascript
+import mqtt from 'mqtt'
+
+const client = mqtt.connect('mqtt://localhost:8883', {
+  clientId: 'test-device-123'
+})
+```
+
+### AWS IoT Core HTTPS API (Port 8086)
+
+Mimics AWS IoT Core's HTTPS publishing API for backend services.
+
+**Endpoint:** `POST http://localhost:8086/topics/:topicName?qos=1`
+- **Body:** Binary payload (Buffer)
+- **Query:** `qos` (optional, defaults to 1)
+- **Response:** Empty 200 (matching AWS IoT Core)
+- **Auth:** SigV4 headers accepted but not validated (dev mode)
+
+**Example usage (device-simulator):**
+
+When device-simulator runs in Docker (on goki-network):
+```bash
+# device-simulator Docker environment
+AWS_IOT_CORE_MQTT_ENDPOINT=dev-tools-backend:8086  # Use service name
+PUBSUB_EMULATOR_HOST=pubsub-emulator:8085          # Use service name
+```
+
+When device-simulator runs on host machine:
+```bash
+# device-simulator local development
+AWS_IOT_CORE_MQTT_ENDPOINT=localhost:8086  # Use localhost
+PUBSUB_EMULATOR_HOST=localhost:8085        # Use localhost
+```
+
+Publishing (no code changes needed):
+```javascript
+const response = await axios.post(
+  `http://${AWS_IOT_CORE_MQTT_ENDPOINT}/topics/rs/test/device-123?qos=1`,
+  binaryPacket,  // Buffer
+  { headers: { 'Content-Type': 'application/octet-stream' } }
+)
+```
+
+### MQTT → Pub/Sub Bridge
+
+Automatically routes MQTT messages to Pub/Sub (mimics AWS IoT Core behavior).
+
+**Topic:** `mqttMessageReceived`
+**Message Format:**
+```json
+{
+  "clientId": "test-device-123",
+  "topicName": "rq/test/test-device-123",
+  "packet": "base64-encoded-binary",
+  "receivedAtMilliseconds": 1234567890
+}
+```
+
+**Lifecycle Events:**
+- **Topic:** `simulator/connected` - Client connected
+- **Topic:** `simulator/disconnected` - Client disconnected
+
+### Topic Patterns (Goki Conventions)
+
+- **Request:** `rq/{action}/{clientId}`
+- **Response:** `rs/{action}/{clientId}`
+- **Notification:** `nf/{action}/{clientId}`
+- **Will:** `w/{action}/{clientId}`
+- **Lifecycle:** `simulator/connected`, `simulator/disconnected`
+
+## Testing
+
+### End-to-End Tests
+
+Comprehensive E2E test suite using Playwright:
+
+```bash
+# Run all E2E tests
+npm run test:e2e
+
+# Run UI tests only
+npm run test:e2e:ui
+
+# Run workflow tests only
+npm run test:e2e:workflows
+
+# Run with browser UI
+npm run test:e2e:headed
+
+# Debug mode
+npm run test:e2e:debug
+```
+
+See [E2E_TEST_GUIDE.md](./tests/E2E_TEST_GUIDE.md) for complete testing documentation.
+
+### Integration Tests
+
+```bash
+# Run all integration tests
+npm run test:integration
+
+# Run specific test categories
+npm run test:api
+npm run test:emulation
+```
+
+## Docker Images
+
+### Production Images
+- **Backend:** `dev-tools-dev-tools-backend` (~1.19GB)
+  - Includes Java 21 for Firebase emulator
+  - Python3, make, g++ for native dependencies
+  - Firebase CLI
+
+- **Frontend:** `dev-tools-dev-tools-frontend` (~464MB)
+  - Multi-stage build (React → nginx)
+  - Optimized static assets
+
+### Development Images
+- **Backend:** Uses `Dockerfile.dev` with hot-reload
+- **Frontend:** Uses `ui/Dockerfile.dev` with React dev server
+
+## Environment Variables
+
+### Backend
+- `NODE_ENV` - Environment (development/production)
+- `WEB_UI_PORT` - Backend port (default: 9000)
+- `PUBSUB_EMULATOR_HOST` - Pub/Sub emulator host
+- `FIRESTORE_EMULATOR_HOST` - Firestore emulator host
+- `POSTGRES_HOST` - PostgreSQL host
+- `POSTGRES_PORT` - PostgreSQL port
+- `POSTGRES_USER` - PostgreSQL user
+- `POSTGRES_PASSWORD` - PostgreSQL password
+- `REDIS_HOST` - Redis host
+- `REDIS_PORT` - Redis port
+- `DATA_DIR` - Data directory for SQLite
+
+### Frontend
+- `REACT_APP_API_URL` - Backend API URL (default: http://localhost:9000)
+
+## Troubleshooting
+
+### Backend won't start
+```bash
+# Check logs
+docker logs goki-dev-tools-backend
+
+# Common issue: Corrupted SQLite database
+rm -f data/dev-tools.db*
+docker compose restart dev-tools-backend
+```
+
+### Frontend won't load
+```bash
+# Check logs
+docker logs goki-dev-tools-frontend
+
+# Verify backend is running
+curl http://localhost:9000/v1/health/readiness
+
+# Rebuild frontend
+docker compose build dev-tools-frontend
+docker compose up -d dev-tools-frontend
+```
+
+### PostgreSQL/Redis not connecting
+```bash
+# Check if services are running
+docker compose ps
+
+# Verify ports are accessible
+nc -zv localhost 5432  # PostgreSQL
+nc -zv localhost 6379  # Redis
+
+# Restart services
+docker compose restart postgres redis
+```
+
+## Project Status
+
+✅ **Production Ready** - Complete with comprehensive E2E tests
+
+### Features
+- ✅ Pub/Sub message management with real-time streaming
+- ✅ PostgreSQL database viewer with query execution
+- ✅ Redis key-value browser with type support
+- ✅ Firestore document viewer with queries
+- ✅ Google Cloud Logging viewer with filters
+- ✅ MQTT broker monitoring
+- ✅ Service control dashboard
+- ✅ Export/Import functionality
+- ✅ Docker auto-start for CLI
+- ✅ Split frontend/backend architecture
+
+See `plans/platform-specification.md` for complete specification.
+
+## Documentation
+
+- [Docker Setup](./DOCKER.md)
+- [E2E Testing Guide](./E2E-TESTING-GUIDE.md)