nduka 1.0.0

package/docs/production-bot-guide.md

@@ -0,0 +1,499 @@
+# Production Bot Guide
+
+This guide documents the current production behavior for the WhatsApp Baileys automation bot on `feature/support-inbox-profiles-labels`.
+
+## Current status
+
+The bot is designed to run as one WhatsApp gateway process that owns the Baileys session, receives messages, applies routing and safety gates, and then sends only intentional final replies back to WhatsApp.
+
+Recent hardening focused on these production risks:
+
+- Baileys reconnect replay causing duplicate replies.
+- Direct search/news/fetch behavior being brittle with slow or empty APIs.
+- Tool diagnostics leaking into user-facing answers.
+- Ordinary chat accidentally executing model-generated tool calls.
+- Group messages causing unwanted AI/tool calls when the bot was not mentioned.
+- Voice transcription being accidentally moved away from Groq Whisper.
+
+## Runtime startup
+
+Use the worker entrypoint:
+
+```bash
+npm run build
+npm run start:worker
+```
+
+A healthy startup should show:
+
+```text
+[Gateway] Starting WhatsApp Gateway...
+[Gateway] Session ownership acquired
+[Gateway] API Keys loaded
+[Gateway] Connected to WhatsApp
+[Gateway] READY — processing messages now
+```
+
+Only one process should own the WhatsApp session. If another process owns the session, stop the duplicate process before restarting.
+
+## Required local environment
+
+Keep `.env` local. Do not commit `.env`.
+
+Minimum recommended provider keys:
+
+```env
+OPENROUTER_API_KEY=your_local_key
+KILOCODE_API_KEY=your_local_key
+CRUNCHTROVE_API=your_local_key
+GROQ_API_KEY=your_local_key
+GEMINI_API_KEY=your_local_key
+```
+
+Accepted aliases:
+
+```env
+CRUNCHTROVA_API=your_local_key
+GOOGLE_API_KEY=your_local_gemini_key
+```
+
+Recommended model routing:
+
+```env
+OPENROUTER_MODEL=nvidia/nemotron-3-nano-30b-a3b:free
+KILOCODE_MODEL=kilo-auto/free
+
+GEMINI_MODEL=gemini-3.5-flash
+GEMINI_VISION_MODEL=gemini-3.5-flash
+
+TASK_CHAT_REPLY_PROVIDER=openrouter
+TASK_CHAT_REPLY_MODEL=nvidia/nemotron-3-nano-30b-a3b:free
+TASK_CHAT_REPLY_FALLBACK_PROVIDER=custom
+TASK_CHAT_REPLY_FALLBACK_MODEL=kilo-auto/free
+
+TASK_TOOL_SYNTHESIS_PROVIDER=openrouter
+TASK_TOOL_SYNTHESIS_MODEL=nvidia/nemotron-3-nano-30b-a3b:free
+TASK_TOOL_SYNTHESIS_FALLBACK_PROVIDER=custom
+TASK_TOOL_SYNTHESIS_FALLBACK_MODEL=kilo-auto/free
+
+TASK_VISION_OCR_PROVIDER=gemini
+TASK_VISION_OCR_MODEL=gemini-3.5-flash
+TASK_VISION_OCR_FALLBACK_PROVIDER=openrouter
+TASK_VISION_OCR_FALLBACK_MODEL=nvidia/nemotron-3-nano-30b-a3b:free
+
+TASK_PDF_SUMMARY_PROVIDER=gemini
+TASK_PDF_SUMMARY_MODEL=gemini-3.5-flash
+TASK_PDF_SUMMARY_FALLBACK_PROVIDER=openrouter
+TASK_PDF_SUMMARY_FALLBACK_MODEL=nvidia/nemotron-3-nano-30b-a3b:free
+```
+
+Voice notes should stay on Groq Whisper. Do not add `TASK_AUDIO_TRANSCRIPTION_PROVIDER=gemini` unless the voice pipeline is deliberately redesigned.
+
+Recommended search timeouts and limits:
+
+```env
+TOOL_TIMEOUT_MS=300000
+TOOL_ANSWER_TIMEOUT_MS=180000
+CRUNCHTROVE_TIMEOUT_MS=180000
+SEARCH_RESULT_LIMIT=5
+WEB_CONTENT_LIMIT=2000
+NEWS_CONTENT_LIMIT=1500
+FETCH_CONTENT_LIMIT=1500
+BOT_TIMEZONE=Asia/Dubai
+```
+
+Recommended Crunchtrove endpoint overrides, only if needed:
+
+```env
+CRUNCHTROVE_BASE_URL=https://api.mrcnpin.com
+CRUNCHTROVE_WEB_PATH=/search/web
+CRUNCHTROVE_NEWS_PATH=/search/news
+CRUNCHTROVE_FETCH_PATH=/fetch/url
+```
+
+## Message routing rules
+
+### Direct messages
+
+Customer DMs are stored, routed through the conversation runtime, and answered normally unless a command or direct search route applies.
+
+### Owner self-chat
+
+The owner can use admin commands in DM/self-chat. Admin commands are not exposed to ordinary customers.
+
+### Groups
+
+Group messages should not call AI or tools unless the bot is explicitly mentioned or the group mode rules allow it. Unmentioned group messages can be stored as context without triggering a reply.
+
+### Broadcasts, newsletters, status, and protocol messages
+
+These are ignored or silently skipped. They should not reach the AI runtime.
+
+## Duplicate replay protection
+
+Baileys can replay old messages after reconnect. The bot now deduplicates in `EventHandler` before media download and before runtime dispatch.
+
+Dedupe keys include:
+
+- chat JID
+- message ID
+- participant JID, when present
+- device ID, when present
+
+JIDs are normalized so Baileys device suffix variants such as `111:7@s.whatsapp.net` and `111@s.whatsapp.net` are treated as the same sender identity for replay protection.
+
+Expected behavior:
+
+- First delivery of a message is processed.
+- Replayed delivery of the same message ID is skipped.
+- Replayed media is not downloaded again.
+- Replayed text does not call AI again.
+- Replayed group commands do not generate repeated replies.
+
+## Search, news, and fetch behavior
+
+There are two paths:
+
+1. Explicit commands handled by `ToolRegistry`.
+2. Natural language direct-tool routing handled by `AIResponseEngine` and `SearchOrchestrator`.
+
+### Web search
+
+Use web search for current factual lookup, schedules, official sources, role/office questions, prices, weather, and similar current information.
+
+Examples:
+
+```text
+who is the present president of Nigeria and when did he get into office?
+what time is the next Knicks game tonight?
+what is the weather in Lagos today?
+```
+
+The bot adds freshness and official-source hints where useful.
+
+### News search
+
+Use news search for article-like updates, comments, statements, reactions, controversies, reports, headlines, and breaking/recent news.
+
+Examples:
+
+```text
+search Tinubu wife latest comment
+latest Tinubu statement
+breaking news about oil prices
+```
+
+News queries are simplified before hitting fragile news APIs. For example:
+
+```text
+search Tinubu wife latest comment 2026-06-28
+```
+
+becomes:
+
+```text
+Tinubu wife comment
+```
+
+### Web-to-news fallback
+
+Web search can fall back to news if configured and web search returns nothing useful.
+
+News search should not fall back to web during natural-language direct-tool routing because that can create loops and make article-specific queries noisy.
+
+### Fetch URL
+
+URL fetch only accepts normal `http` and `https` URLs. Localhost, private-network, metadata, unsupported schemes, and embedded credential URLs are blocked before provider calls.
+
+## Tool execution policy
+
+Ordinary chat keeps model tools disabled by default.
+
+The model may talk normally, but it should not execute function tools or XML-style tool requests unless this is explicitly enabled:
+
+```env
+ALLOW_LLM_AUTO_TOOLS=true
+```
+
+Recommended production setting:
+
+```env
+ALLOW_LLM_AUTO_TOOLS=false
+```
+
+Direct tool routing still works when a user clearly asks for current information, search, news, or URL fetch. That route is deterministic and gateway-controlled.
+
+## Provider routing
+
+Recommended split:
+
+- OpenRouter: normal chat and tool synthesis.
+- Kilocode/custom: fallback for normal chat and tool synthesis.
+- Gemini: image/document/PDF vision tasks.
+- Groq Whisper: voice note transcription.
+- Crunchtrove/Crunchtrova: web search, news search, and URL fetch.
+
+Task routing should include fallbacks so one provider failure does not stop the bot.
+
+## Voice notes
+
+Voice notes should follow this path:
+
+```text
+WhatsApp audio message → media download → Groq Whisper transcription → AI response
+```
+
+Voice is intentionally not part of the task-model router. Keep Groq Whisper as the default voice path unless the voice architecture is intentionally changed.
+
+## Visual media and documents
+
+Images and documents should be downloaded early, before links expire. Visual/PDF work should use Gemini where configured, with OpenRouter/Kilocode fallback.
+
+Text-like media should be extracted and stored as context where possible. If extraction fails, it should be labeled for review instead of crashing the bot.
+
+## Owner and user commands
+
+User-safe commands:
+
+```text
+/search <query>
+/news <query>
+/fetch <url>
+/toolstatus
+/help
+/status
+```
+
+Short aliases:
+
+```text
+/s <query>
+/n <query>
+/f <url>
+/h
+```
+
+Owner/admin commands in owner DM/self-chat:
+
+```text
+/model
+/models
+/model <name>
+/config
+/config toggle_group
+/config toggle_tools
+/config thinking on
+/config thinking off
+/config memory <number>
+/config temp <0.0-1.0>
+/memory
+/memory clear <jid>
+/stats
+/context <customer name or ID>
+/audit
+/debug
+/status
+/list
+/help
+```
+
+Support handoff commands may also be available in the wider gateway flow depending on the support inbox modules:
+
+```text
+/handoff status
+/continue <customer>
+/resume <customer>
+/reopen <customer>
+/status <customer> ai
+```
+
+## Production smoke test checklist
+
+After every hardening pass, run:
+
+```bash
+npm run build
+npm test
+```
+
+Then start the worker:
+
+```bash
+npm run start:worker
+```
+
+Live smoke tests:
+
+1. Basic DM chat
+
+```text
+hello, what can you do?
+```
+
+Expected: normal concise answer, no tool call.
+
+2. Web current lookup
+
+```text
+who is the present president of Nigeria and when did he get into office?
+```
+
+Expected log:
+
+```text
+[AIEngine] Direct tool route: web_search
+```
+
+3. News article lookup
+
+```text
+hey search Tinubu wife latest comment
+```
+
+Expected log:
+
+```text
+[AIEngine] Direct tool route: news_search query="Tinubu wife comment"
+```
+
+4. Tool status
+
+```text
+/toolstatus
+```
+
+Expected: configured status, masked key only, no full API key.
+
+5. URL fetch guard
+
+```text
+/fetch http://localhost:3000
+```
+
+Expected: blocked before provider call.
+
+6. Voice note
+
+Send a short WhatsApp voice note.
+
+Expected: Groq Whisper transcription log, then AI answer.
+
+7. Image caption
+
+Send an image with a caption.
+
+Expected: image/download path runs once, caption is preserved, visual handling works.
+
+8. Group unmentioned message
+
+Send a normal group message without mentioning the bot.
+
+Expected: no AI/tool reply.
+
+9. Group mentioned message
+
+Mention the bot in a group with a clear question.
+
+Expected: one reply only.
+
+10. Reconnect replay check
+
+Restart the worker and watch for old-message replays.
+
+Expected: duplicate message IDs are skipped, no repeated replies.
+
+## Test suite coverage added in the hardening pass
+
+The hardening pass added or strengthened tests for:
+
+- Baileys reconnect replay dedupe.
+- JID device-suffix normalization.
+- Dedupe persistence across engine instances.
+- EventHandler duplicate suppression before media download.
+- Mention parsing isolation from persistent dedupe state.
+- Search input guard behavior before provider calls.
+- Configurable web-to-news fallback.
+- One-way fallback behavior so news does not loop back into web in direct routing.
+- Ordinary chat keeping model tools disabled by default.
+- Source guards against hardcoded current-answer shortcuts.
+
+## Troubleshooting
+
+### One test fails in `direct-tool-guard.test.ts` around Crunchtrove timeout
+
+This means the source guard is checking the wrong file or the timeout env hook was removed. The Crunchtrove timeout contract belongs in `src/tools/crunchtrova.ts`.
+
+Expected source:
+
+```ts
+const REQUEST_TIMEOUT_MS = Number(loadEnvVar('CRUNCHTROVE_TIMEOUT_MS', 'CRUNCHTROVA_TIMEOUT_MS') || 180_000);
+```
+
+### News search returns empty for a query that works manually
+
+Check whether the bot is sending a noisy query. The expected normalized query should remove words like `search`, `latest`, `news`, `recent`, and the forced date.
+
+Expected:
+
+```text
+Tinubu wife comment
+```
+
+Not expected:
+
+```text
+search Tinubu wife latest comment 2026-06-28
+```
+
+### Tool synthesis uses the wrong model
+
+Check `.env` for overrides:
+
+```bash
+grep -n "TOOL_PROVIDER\|TOOL_MODEL\|TASK_TOOL_SYNTHESIS_PROVIDER\|TASK_TOOL_SYNTHESIS_MODEL" .env
+```
+
+`TOOL_PROVIDER` and `TOOL_MODEL` override task routing. Remove them unless intentionally forcing a specific tool synthesis model.
+
+### Gemini key not showing at startup
+
+Use either:
+
+```env
+GEMINI_API_KEY=your_local_key
+```
+
+or:
+
+```env
+GOOGLE_API_KEY=your_local_key
+```
+
+Startup logging may not print every provider unless the gateway logging is expanded, but `checkApiKeys()` supports Gemini.
+
+### Bot replies repeatedly to the same message
+
+Check logs for the same message ID. Duplicate replay should now be skipped inside `EventHandler`. If repeats still happen, inspect whether the replay has a genuinely different message ID or whether the reply itself is being treated as a new self-chat instruction.
+
+### Search/fetch fails with local or private URLs
+
+That is expected. The fetch guard blocks localhost, metadata, private-network, link-local, unsupported schemes, and embedded credentials.
+
+## Release readiness gate
+
+A change is ready to smoke test live only when all of this passes:
+
+```bash
+npm run build
+npm test
+```
+
+A change is production-ready only when the live smoke checklist passes without:
+
+- repeated replies,
+- unmentioned group replies,
+- full API key exposure,
+- raw tool diagnostics sent as the final user answer,
+- voice routing away from Groq Whisper,
+- news queries being bloated with command words or dates,
+- unsafe fetch targets reaching the provider.