@kangwifi-pro/waliwa 1.0.0

package/docs/ARCHITECTURE.md

@@ -0,0 +1,307 @@
+# Architecture
+
+This document describes the internal architecture of Waliwa library.
+
+## Design Principles
+
+### 1. Hybrid Protocol
+Waliwa implements **two distinct WhatsApp protocols** with seamless switching:
+
+- **WhatsApp Web Multi-Device (MD)** - The primary protocol used by WhatsApp Web. Uses WebSocket + Noise Protocol Framework (XX pattern) + custom binary format.
+- **Mobile API Pairing** - Companion device registration via phone number + pairing code. Used by official WhatsApp Business API clients.
+
+The `HybridSocketManager` automatically falls back between these protocols when one fails.
+
+### 2. RAM-Efficient Design
+Three key strategies to minimize memory usage:
+
+- **Buffer Pooling** - Reuse pre-allocated Uint8Array buffers for WebSocket frames instead of allocating new ones per message
+- **Lazy Loading** - Signal keys/sessions loaded on-demand, not all at startup
+- **Debounced Disk IO** - Batch credential file writes every 50-100ms instead of per-update
+
+### 3. Clean-Room Implementation
+All protocol-specific code is written from scratch based on publicly documented specs:
+- Noise Protocol Framework (noiseprotocol.org)
+- WhatsApp binary format (publicly documented through reverse engineering)
+- Signal protocol (open source spec)
+
+No code is copied from Baileys, ZapoJS, or any other WhatsApp library.
+
+## Layer Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                      Application Layer                      │
+│                  (User code using WASocket)                  │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                    Public API Layer                          │
+│  makeWASocket(), sendMessage(), groupCreate(), ev.on()       │
+└─────────────────────────────────────────────────────────────┘
+                              │
+        ┌─────────────────────┼─────────────────────┐
+        ▼                     ▼                     ▼
+┌──────────────┐     ┌──────────────┐     ┌──────────────┐
+│  Messages    │     │   Groups     │     │    Calls     │
+│  Sender      │     │  Manager     │     │   Handler    │
+│  Receiver    │     │              │     │              │
+│  Media       │     │              │     │              │
+└──────────────┘     └──────────────┘     └──────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                    Socket Layer                              │
+│   WAWebSocket (Noise-encrypted WebSocket frames)             │
+│   MobileAPISocket (Pairing code flow)                        │
+│   HybridSocketManager (Auto-fallback)                         │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                    Core Layer                                │
+│   NoiseHandshake (XX pattern implementation)                 │
+│   TrafficCipher (AES-256-GCM post-handshake)                 │
+│   BinaryEncoder/Decoder (WA binary format)                   │
+│   Crypto utilities (Curve25519, HKDF, AES, HMAC)            │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                    Auth Layer                                 │
+│   FileAuthState (JSON file storage)                          │
+│   QR Code generator (WA Web pairing)                         │
+│   Pairing Code generator (Mobile API)                        │
+│   Signal Key Store (pre-keys, sessions, identity)            │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                    Utilities Layer                           │
+│   Logger (lightweight alternative to pino)                   │
+│   JID utilities (parse, normalize, validate)                 │
+│   Retry/Throttle (exponential backoff, rate limiter)         │
+│   Event Emitter (typed, Baileys-compatible)                  │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Module Dependencies
+
+```
+src/index.ts
+├── socket/wa-socket.ts (main entry point)
+│   ├── socket/ws-socket.ts (WebSocket layer)
+│   │   ├── core/noise.ts (Noise protocol)
+│   │   │   └── core/crypto.ts (crypto primitives)
+│   │   └── core/binary.ts (binary encoding)
+│   ├── socket/hybrid.ts (Mobile API + fallback)
+│   ├── auth/auth-state.ts (file storage)
+│   ├── auth/qr-code.ts (QR pairing)
+│   ├── auth/pairing-code.ts (Mobile pairing)
+│   ├── messages/send.ts (message operations)
+│   │   └── messages/message-encoder.ts (encoding)
+│   ├── messages/media.ts (upload/download)
+│   ├── groups/management.ts (group operations)
+│   ├── calls/handler.ts (call handling)
+│   ├── advanced/features.ts (polls, newsletter, etc)
+│   ├── events/emitter.ts (typed events)
+│   └── utils/{jid,logger,retry}.ts (utilities)
+└── types/index.ts (all type definitions)
+```
+
+## Connection Lifecycle
+
+```
+1. User calls makeWASocket(config)
+   ↓
+2. WASocketImpl created with config
+   ↓
+3. connect() called automatically
+   ↓
+4. WAWebSocket established to wss://web.whatsapp.com/ws/chat
+   ↓
+5. Noise Protocol XX handshake performed
+   - Client sends: ephemeral + encrypted static + payload
+   - Server responds: ephemeral + encrypted static + ES + payload
+   - Both derive traffic keys via HKDF
+   ↓
+6. Traffic cipher initialized (AES-256-GCM)
+   ↓
+7. Auth flow starts:
+   - If creds.registered = true: send login IQ
+   - If mobile mode: request pairing code
+   - Else: generate QR code, start refresh cycle
+   ↓
+8. Server sends success/failure node
+   - On success: emit 'connection.update' with connection='open'
+   - On failure: emit error, retry with exponential backoff
+   ↓
+9. Keep-alive ping every 20s
+   ↓
+10. Incoming messages processed by MessageReceiver
+    - Decode binary node
+    - Parse message type (text, image, video, etc)
+    - Emit 'message.upsert' event
+   ↓
+11. On disconnect: 
+    - Reject all pending messages
+    - Auto-reconnect with exponential backoff
+    - Switch protocol mode if max retries reached
+```
+
+## Message Encoding Flow
+
+```
+User: sock.sendMessage(jid, { text: 'Hello' })
+                          ↓
+1. MessageSender.sendMessage() called
+                          ↓
+2. generateMessageId() returns "3EB0..."
+                          ↓
+3. encodeMessageNode() builds BinaryNode:
+   {
+     tag: 'message',
+     attrs: { to: jid, type: 'text', id: msgId },
+     content: [{ tag: 'conversation', attrs: {}, content: 'Hello' }]
+   }
+                          ↓
+4. encodeBinaryNode() serializes to Uint8Array:
+   - Tag header byte
+   - Tag string (looked up in dictionary, packed nibble, or raw bytes)
+   - Attributes (key-value pairs)
+   - Content (recursive children)
+                          ↓
+5. WAWebSocket.send():
+   - TrafficCipher.encrypt() with AES-256-GCM (counter nonce)
+   - Prepend WA frame header: [0x57 0x41] [version] [3-byte length]
+   - ws.send(Buffer.from(frame))
+                          ↓
+6. Server ACKs message with 'ack' node
+   - MessageTracker resolves pending Promise
+   - Returns SendMessageResult to user
+```
+
+## File Layout
+
+```
+waliwa/
+├── src/
+│   ├── index.ts                    # Public exports
+│   ├── types/index.ts              # All TypeScript types
+│   ├── core/
+│   │   ├── crypto.ts                # Curve25519, AES, HKDF, HMAC
+│   │   ├── binary.ts                # WA binary encoder/decoder
+│   │   └── noise.ts                 # Noise Protocol XX handshake
+│   ├── socket/
+│   │   ├── ws-socket.ts             # WA WebSocket layer
+│   │   ├── wa-socket.ts              # Main WASocket interface
+│   │   └── hybrid.ts                 # Mobile API + Hybrid manager
+│   ├── auth/
+│   │   ├── auth-state.ts             # File-based auth state
+│   │   ├── qr-code.ts                # QR code generation
+│   │   └── pairing-code.ts          # Mobile pairing code
+│   ├── messages/
+│   │   ├── message-encoder.ts        # Message encoding/decoding
+│   │   ├── send.ts                   # Send/receive operations
+│   │   └── media.ts                  # Media upload/download
+│   ├── groups/management.ts          # Group operations
+│   ├── calls/handler.ts              # Call handling
+│   ├── advanced/features.ts          # Polls, newsletter, etc
+│   ├── events/emitter.ts             # Typed event emitter
+│   └── utils/
+│       ├── jid.ts                    # JID utilities
+│       ├── logger.ts                 # Logger
+│       └── retry.ts                  # Retry/throttle
+├── examples/
+│   ├── echo-bot/                     # Basic echo bot
+│   ├── group-bot/                    # Group management bot
+│   ├── ai-bot/                       # AI chatbot with GPT
+│   └── rest-gateway/                 # REST API gateway
+├── docs/                             # Documentation
+├── package.json
+├── tsconfig.json
+├── LICENSE
+└── README.md
+```
+
+## Performance Considerations
+
+### Memory Usage
+- BufferPool: Pre-allocate 16 buffers of 8KB each (configurable)
+- Lazy session loading: Only load Signal sessions when first message exchanged
+- Conversation history: Optional, max 10 messages per user (configurable)
+- Debounced writes: Coalesce 100ms of state changes into single disk write
+
+### Network Usage
+- WebSocket compression disabled (perMessageDeflate: false) - avoids CPU overhead
+- Keep-alive ping every 20s (configurable)
+- Max payload: 16MB (matches WhatsApp server limit)
+- HTTP/2 used for media upload/download (via axios)
+
+### CPU Usage
+- AES-GCM via @noble/ciphers (no native bindings, pure JS)
+- Curve25519 via @noble/curves (constant-time, audited)
+- Binary encoding with dictionary lookup (O(1) for common tags)
+- No protobuf runtime reflection - all encoding done manually
+
+## Error Handling
+
+### Reconnection Strategy
+```
+Disconnect detected
+       ↓
+Increment retry counter
+       ↓
+Delay = min(baseInterval * 2^retries, 60000)
+       ↓
+Reconnect attempt
+       ↓
+┌──→ Success → Reset counter
+│
+└──→ Failure → Loop back if counter < maxRetries
+                ↓
+                If counter >= maxRetries:
+                  - Switch protocol mode (Web ↔ Mobile)
+                  - Reset counter
+                  - Try again
+```
+
+### Message Retry Strategy
+- Max 5 retries per message (configurable)
+- Exponential backoff: 500ms → 1s → 2s → 4s → 8s
+- On final failure: emit 'message.update' with status='error'
+- Pending messages cleared on disconnect (rejected with 'Connection closed' error)
+
+## Extension Points
+
+### Custom Auth State
+```typescript
+import { AuthState, AuthenticationCreds, SignalKeyStore } from 'waliwa';
+
+class RedisAuthState implements AuthState {
+  creds: AuthenticationCreds;
+  keys: SignalKeyStore;
+  // Implement with Redis backend
+}
+
+const sock = makeWASocket({
+  authState: new RedisAuthState(redisClient)
+});
+```
+
+### Custom Logger
+```typescript
+import { WALogger } from 'waliwa';
+
+class MyLogger implements WALogger {
+  // Implement with your logging system (Winston, Bunyan, etc.)
+}
+
+const sock = makeWASocket({
+  logger: new MyLogger()
+});
+```
+
+### Custom Media Storage
+The MediaUploader/MediaDownloader classes can be extended to use S3, Google Cloud Storage, or other backends instead of WhatsApp CDN.

package/docs/BAILEYS_FIXES.md

@@ -0,0 +1,360 @@
+# Baileys Issues Fixes
+
+Waliwa mengatasi 9+ issues yang sering dikeluhkan di Baileys. Setiap fix dijelaskan dengan masalah, solusi, dan cara pakai.
+
+## Daftar Fix
+
+### 1. Pre-key Exhaustion (Baileys Issue)
+
+**Masalah:** Pre-keys habis tanpa warning, bot tiba-tiba tidak bisa menerima pesan baru.
+
+**Solusi:** `PreKeyManager` di SKDM dengan auto-regeneration.
+
+```typescript
+// Otomatis di SKDM - pre-keys di-regenerate saat < minPreKeys (default: 5)
+const skdm = sock.getSKDM();
+console.log(skdm.preKeyManager.getStats());
+// { total: 95, needsRegen: false, ... }
+
+// Manual check
+if (skdm.preKeyManager.needsRegeneration()) {
+  await skdm.preKeyManager.regeneratePreKeys();
+}
+```
+
+**Lihat:** [SKDM documentation](./SKDM.md) untuk detail lengkap.
+
+---
+
+### 2. Race Condition di Message Sending
+
+**Masalah:** Concurrent sends ke same chat cause duplicate messages, wrong order, atau message ID collision.
+
+**Solusi:** `MessageQueue` dengan per-recipient serialization.
+
+```typescript
+// Otomatis di WASocket.sendMessage() - tiap recipient punya queue sendiri
+// Concurrent across different recipients (parallelism)
+// Sequential per recipient (order guarantee)
+
+// Manual usage untuk custom operations:
+import { MessageQueue } from 'waliwa';
+
+const queue = new MessageQueue(10); // Max 10 concurrent recipients
+await queue.enqueue('user1@s.whatsapp.net', async () => {
+  return await sock.sendMessage('user1@s.whatsapp.net', { text: 'Hi' });
+});
+
+// Stats
+console.log(queue.getStats());
+// { totalQueues: 1, totalPending: 0, activeCount: 0, maxConcurrency: 10 }
+```
+
+---
+
+### 3. Memory Leak dari Event Listeners
+
+**Masalah:** Listeners tidak di-cleanup properly saat objects di-destroy, menyebabkan memory leak overtime.
+
+**Solusi:** `WeakEventEmitter` dengan FinalizationRegistry untuk auto-cleanup.
+
+```typescript
+import { WeakEventEmitter } from 'waliwa';
+
+const emitter = new WeakEventEmitter();
+
+class MyComponent {
+  private data: any[] = [];
+
+  constructor() {
+    // Auto-cleanup saat MyComponent di-GC
+    emitter.subscribeWeak(this, 'message', (msg) => {
+      this.data.push(msg);
+    });
+  }
+}
+
+// Atau manual cleanup
+emitter.cleanupTarget(myComponentInstance);
+```
+
+---
+
+### 4. Connection Hang
+
+**Masalah:** WebSocket connection hang, tidak detect disconnect, bot "appear online" tapi tidak receive/send messages.
+
+**Solusi:** `ConnectionHeartbeat` dengan proper dead detection.
+
+```typescript
+import { ConnectionHeartbeat } from 'waliwa';
+
+const heartbeat = new ConnectionHeartbeat({
+  sendPing: async () => {
+    // Send WA ping
+    await sock.send(encodeBinaryNode({
+      tag: 'iq',
+      attrs: { type: 'get', id: `ping-${Date.now()}` },
+      content: [{ tag: 'ping', attrs: {} }]
+    }));
+  },
+  onDead: () => {
+    console.error('Connection dead, triggering reconnect');
+    // Custom recovery logic
+  },
+  pingIntervalMs: 20000,  // Ping every 20s
+  pongTimeoutMs: 10000,   // Wait 10s for pong
+  maxMissedPongs: 3       // Mark dead after 3 misses
+});
+
+heartbeat.start();
+
+// When pong received (di WA message handler):
+heartbeat.receivedPong();
+
+// Check health
+console.log(heartbeat.isHealthy()); // true/false
+```
+
+**Otomatis di WASocket:** Heartbeat aktif secara default, mengirim ping setiap 20 detik.
+
+---
+
+### 5. Decryption Failure untuk Old Messages
+
+**Masalah:** Messages dari long-time-no-chat contacts fail to decrypt karena session keys expired/missing.
+
+**Solusi:** `SessionManager.prefetchSessions()` + `DecryptionRetryQueue`.
+
+```typescript
+const skdm = sock.getSKDM();
+
+// Prefetch sessions sebelum bulk decrypt group messages
+skdm.sessionManager.prefetchSessions([
+  'user1@s.whatsapp.net',
+  'user2@s.whatsapp.net'
+]);
+
+// Decryption retry queue - retry failed decryptions after key sync
+import { DecryptionRetryQueue } from 'waliwa';
+
+const retryQueue = new DecryptionRetryQueue({
+  maxAttempts: 3,
+  maxQueueSize: 1000,
+  retryHandler: async (item) => {
+    // Custom retry logic - request key sync, then retry
+    await requestKeySync(item.senderJid);
+    return await retryDecryption(item.encryptedData);
+  }
+});
+
+// Enqueue failed decryption
+retryQueue.enqueue('msg-123', '6281@s.whatsapp.net', encryptedData);
+
+// Trigger immediate retry (e.g., after key sync complete)
+await retryQueue.retryNow();
+```
+
+---
+
+### 6. Duplicate Messages
+
+**Masalah:** Messages diterima multiple times, bot reply 2x, confusing user.
+
+**Solusi:** `IdempotencyCache` dengan TTL.
+
+```typescript
+// Otomatis di WASocket - incoming message IDs di-check
+// Jika ID sudah pernah dilihat dalam 24 jam, skip
+
+import { IdempotencyCache } from 'waliwa';
+
+const cache = new IdempotencyCache(10000, 24 * 60 * 60 * 1000);
+
+// Check duplicate
+if (cache.isDuplicate(messageId)) {
+  console.log('Skipping duplicate message');
+  return;
+}
+
+// Filter batch
+const uniqueMessages = cache.filterDuplicates(messages);
+```
+
+---
+
+### 7. Group Events Missing
+
+**Masalah:** Group participant events (join/leave) kadang tidak fired, atau fired multiple times rapidly.
+
+**Solusi:** `GroupEventCoalescer` - merge rapid events dalam window 100ms.
+
+```typescript
+import { GroupEventCoalescer } from 'waliwa';
+
+const coalescer = new GroupEventCoalescer(
+  (events) => {
+    // Receive coalesced events
+    for (const event of events) {
+      console.log(`Group ${event.id}: ${event.action} ${event.participants.length} users`);
+    }
+  },
+  100 // Window 100ms
+);
+
+// Add events
+coalescer.addEvent({
+  id: 'group@g.us',
+  participants: ['user1@s.whatsapp.net'],
+  action: 'add',
+  timestamp: Date.now()
+});
+
+// Multiple rapid events akan di-merge
+```
+
+---
+
+### 8. Media Upload Timeout
+
+**Masalah:** Large files (>5MB) sering timeout saat upload, no resume support.
+
+**Solusi:** `ChunkedMediaUploader` dengan chunking + resume.
+
+```typescript
+import { ChunkedMediaUploader } from 'waliwa';
+
+const uploader = new ChunkedMediaUploader({
+  chunkSize: 1024 * 1024, // 1MB chunks
+  maxRetries: 3
+});
+
+// Upload dengan resume support
+const uploadId = 'upload-123';
+const fileBuffer = await readFile('large-video.mp4'); // 50MB
+
+await uploader.uploadChunked(fileBuffer, uploadId, async (chunk, chunkIndex, totalChunks) => {
+  console.log(`Uploading chunk ${chunkIndex + 1}/${totalChunks}`);
+  // Upload chunk ke WhatsApp CDN
+  await uploadChunkToCDN(chunk, chunkIndex);
+});
+
+// Jika upload interrupted, resume dari last successful chunk
+const resumePoint = uploader.getResumePoint(uploadId);
+console.log(`Resume from chunk ${resumePoint}`);
+```
+
+---
+
+### 9. History Sync Incomplete
+
+**Masalah:** Initial history sync tidak complete, messages lost saat bot start.
+
+**Solusi:** `HistorySyncManager` dengan resume support.
+
+```typescript
+import { HistorySyncManager } from 'waliwa';
+
+const syncManager = new HistorySyncManager();
+
+// Start sync
+syncManager.startSync();
+
+// Track progress per chat
+syncManager.trackProgress('user1@s.whatsapp.net', 'msg-123', Date.now());
+
+// Get resume state untuk specific chat
+const resume = syncManager.getResumeState('user1@s.whatsapp.net');
+if (resume) {
+  // Request sync from lastMessageId onwards
+  await requestHistorySync('user1@s.whatsapp.net', resume.lastMessageId);
+}
+
+// Complete sync
+syncManager.completeSync();
+
+// Persist untuk survive restart
+const state = syncManager.serialize();
+await fs.writeFile('./auth/sync-state.json', state);
+
+// Restore on next start
+const saved = await fs.readFile('./auth/sync-state.json', 'utf-8');
+syncManager.deserialize(saved);
+```
+
+---
+
+### 10. Smarter Reconnection Strategy
+
+**Masalah:** Baileys pakai exponential backoff yang sama untuk semua scenarios. Flood server, dapat banned.
+
+**Solusi:** `SmarterReconnector` dengan phased backoff.
+
+```typescript
+import { SmarterReconnector } from 'waliwa';
+
+const reconnector = new SmarterReconnector(
+  async () => {
+    // Reconnect logic
+    await sock.connect();
+  },
+  { maxAttempts: 20 }
+);
+
+// Schedule next reconnect
+reconnector.scheduleNext();
+
+// Phases:
+// Phase 1 (attempts 1-3): Fast retry - 1s, 2s, 4s (network blip)
+// Phase 2 (attempts 4-6): Medium - 10s, 20s, 30s (server issues)
+// Phase 3 (attempts 7+): Slow - 60s+ (long-term issues, with jitter)
+
+console.log(reconnector.getCurrentPhase()); // 1, 2, or 3
+```
+
+---
+
+## All-in-One Usage
+
+Semua fix aktif secara default di WASocket:
+
+```typescript
+const sock = makeWASocket({
+  authState: state,
+  printQRInTerminal: true,
+  // Semua fix aktif otomatis:
+  // - Pre-key auto-regen
+  // - Message queue (race condition fix)
+  // - Idempotency cache (duplicate fix)
+  // - Heartbeat (connection hang fix)
+  // - Health monitor
+  // - Circuit breaker
+  // - Rate limit tracker
+  // - Message persistence
+  // - Webhook system
+});
+
+// Check system stats
+console.log(sock.getSystemStats());
+
+// Check health
+console.log(sock.getHealth());
+
+// Get SKDM stats
+console.log(sock.getSKDMStats());
+```
+
+## Comparison: Baileys vs Waliwa
+
+| Issue | Baileys | Waliwa |
+|-------|---------|--------|
+| Pre-key exhaustion | Manual regen required | ✅ Auto-regen |
+| Race condition | No built-in queue | ✅ Per-recipient queue |
+| Memory leak | Listeners accumulate | ✅ WeakEventEmitter |
+| Connection hang | No detection | ✅ Heartbeat + dead detection |
+| Decryption failure | Lost forever | ✅ Retry queue + prefetch |
+| Duplicate messages | Process 2x | ✅ Idempotency cache |
+| Group events | Missing/duplicate | ✅ Event coalescing |
+| Media upload timeout | No resume | ✅ Chunked upload |
+| History sync | Incomplete | ✅ Resume support |
+| Reconnect strategy | Same backoff | ✅ Phased backoff |