nduka 1.0.0

package/docs/admin-cli-security.md

@@ -0,0 +1,71 @@
+# Admin CLI Security
+
+Branch: `feature/runtime-admin-milestone`
+
+This document covers the new local admin CLI commands for auth/session management, model choices, and API keys.
+
+## Auth/session commands
+
+```bash
+npm run cli -- auth status
+npm run cli -- auth backup
+npm run cli -- auth reset --yes
+npm run cli -- auth logout --yes
+npm run cli -- auth login --print
+npm run cli -- auth login --reset --yes --print
+```
+
+Behavior:
+
+- `auth status` checks whether the local auth state file exists. It never prints credential contents.
+- `auth backup` copies the auth state to a timestamped backup beside the auth file.
+- `auth reset --yes` and `auth logout --yes` back up the auth state and remove the local file.
+- `auth login --print` prints the worker command that will start the gateway and show QR/pairing output when no auth state exists.
+- `auth login --reset --yes` resets local auth first, then starts the login flow unless `--print` is also provided.
+
+This is local session management. It does not call WhatsApp logout while the gateway socket is offline. The next gateway start will require pairing again when the auth file is missing.
+
+## Local keys and models
+
+```bash
+npm run cli -- secrets status
+npm run cli -- secrets list
+npm run cli -- secrets set openrouter --from-env OPENROUTER_API_KEY_INPUT
+npm run cli -- secrets unset openrouter --yes
+npm run cli -- models choose openrouter openai/gpt-4o-mini
+npm run cli -- tools key crunchtrova --from-env CRUNCHTROVA_API_INPUT
+```
+
+Rules:
+
+- Raw secret values are not accepted directly as CLI arguments.
+- Secrets are read from an environment variable using `--from-env`.
+- Secret values are written to the local `.env` file only.
+- Secret values are masked in status output.
+- Only allowlisted keys can be written.
+- Model choices are non-secret but still validated and written to `.env`.
+- Restart the gateway after changing `.env` values.
+
+## Why this design
+
+Credentials should not be stored in tracked config files. This project already loads `.env` for model providers and tool keys, so the CLI edits `.env` rather than `config/*.json`.
+
+The CLI is intentionally non-interactive in tests and automation. For human use, export a temporary environment variable, run the CLI command, then clear the variable from your shell.
+
+Example:
+
+```bash
+export OPENROUTER_API_KEY_INPUT='your-key-here'
+npm run cli -- secrets set openrouter --from-env OPENROUTER_API_KEY_INPUT
+unset OPENROUTER_API_KEY_INPUT
+```
+
+## Validation
+
+```bash
+npm run build
+npm test -- src/auth/__tests__/session-admin.test.ts
+npm test -- src/config/__tests__/local-env-editor.test.ts
+npm test -- src/cli/__tests__/cli-auth-secrets.test.ts
+npm test
+```

package/docs/async-job-implementation-status.md

@@ -0,0 +1,88 @@
+# Async Job Implementation Status
+
+Status: implemented on `work/v2-async-job-store`.
+
+Last verified local suite after the runtime hook:
+
+```text
+Test Files  131 passed (131)
+Tests       941 passed (941)
+```
+
+## Implemented flow
+
+The runtime can now create a durable async job before sending the first WhatsApp acknowledgement.
+
+1. Customer sends a normal message.
+2. `ConversationRuntime` runs owner commands, slash guidance, deterministic built-ins, secure private tools, and private-domain safety gates first.
+3. `AsyncJobRequestController` checks direct-tool intent and the configured tool catalog.
+4. A job is created only when the selected catalog tool is enabled, public, not owner-only, and `asyncEligible: true`.
+5. `AsyncJobStore.enqueue` persists the job before the acknowledgement is returned.
+6. The first acknowledgement is:
+
+```text
+I’m checking that now.
+Reference: job <id>
+```
+
+7. `Gateway` starts the async worker only when `ASYNC_JOB_WORKER_ENABLED=true`.
+8. The worker claims queued jobs, executes configured tools through `AsyncToolProcessor`, and sends the second WhatsApp message through gateway delivery.
+
+## Main files
+
+- `src/async-jobs/job-store.ts`
+- `src/async-jobs/job-runner.ts`
+- `src/async-jobs/job-worker.ts`
+- `src/async-jobs/job-delivery.ts`
+- `src/async-jobs/tool-processor.ts`
+- `src/async-jobs/gateway-worker.ts`
+- `src/async-jobs/request-controller.ts`
+- `src/conversation/runtime-production.ts`
+- `src/gateway/server.ts`
+- `src/tools/tool-catalog-config.ts`
+
+## Owner audit commands
+
+Implemented and owner-only:
+
+- `/jobs`
+- `/jobs <status>`
+- `/jobs <status> <limit>`
+- `/job <id>`
+- `/cancel_job <id>`
+
+## Config
+
+Start worker:
+
+```env
+ASYNC_JOB_WORKER_ENABLED=true
+```
+
+Optional:
+
+```env
+ASYNC_JOB_WORKER_INTERVAL_MS=15000
+ASYNC_JOB_WORKER_LIMIT=5
+```
+
+Tool eligibility:
+
+```json
+{
+  "public": true,
+  "ownerOnly": false,
+  "asyncEligible": true
+}
+```
+
+## Remaining manual checks before merging
+
+- Run a clean full suite from `work/v2-async-job-store`.
+- Run Docker compose with persisted `data` mounts.
+- Enable `ASYNC_JOB_WORKER_ENABLED=true`.
+- Send a live WhatsApp customer latest-news/search request.
+- Confirm Message 1 appears only after the job row exists.
+- Confirm Message 2 is sent by the worker.
+- Confirm customer slash `/news`, `/search`, and `/fetch` remain blocked or guided.
+- Confirm `/jobs`, `/job <id>`, and `/cancel_job <id>` are owner-only.

package/docs/async-job-workflow.md

@@ -0,0 +1,175 @@
+# Durable Async Job Workflow
+
+Status: design guardrail and implementation plan. Do not treat this as complete runtime behavior until code, tests, Docker checks, and live WhatsApp checks pass.
+
+## Purpose
+
+The bot must never create a two-message flow just because the model wrote a promise. A delayed result is allowed only when the runtime has persisted a job first.
+
+Required shape:
+
+1. Inbound WhatsApp message enters the gateway.
+2. Runtime routes the request through deterministic gates.
+3. A planner may propose a job, but cannot execute it directly.
+4. Runtime validates the proposed job against configured tools, access policy, schema, and async eligibility.
+5. Runtime persists the job record.
+6. Gateway sends the immediate acknowledgement only after persistence succeeds.
+7. Worker claims the job, executes the configured tool, records success or failure, and sends the second WhatsApp message.
+8. Logs and audit retain full technical details while WhatsApp receives safe wording.
+
+## Non-negotiables
+
+- No hardcoded tool routing in TypeScript.
+- No arbitrary customer slash tools.
+- No model-only promise of a future reply.
+- No private account data guessed by the model.
+- No environment-specific paths in code or tests.
+- No in-memory-only async jobs.
+- No customer access to owner-only tools.
+- No unstructured tool failure treated as a valid result.
+
+## Recommended gate order
+
+1. Owner/admin commands.
+2. Customer slash-command guidance.
+3. Deterministic built-ins, such as date/time and capability summary.
+4. Secure private business tools with identity, approval, and verification policy.
+5. Configured public tools from tool JSON.
+6. Async job proposal gate for tools that explicitly allow async mode.
+7. Transactional support cases.
+8. Ordinary complaint handling.
+9. Normal AI chat with no live/current claims unless backed by a tool result.
+
+## Job record
+
+A durable job should store at least:
+
+- job_id
+- chat_jid
+- sender_jid
+- original_message_id
+- original_text
+- tool_name
+- tool_args
+- status
+- attempts
+- max_attempts
+- created_at
+- updated_at
+- scheduled_for
+- started_at
+- completed_at
+- last_error
+- result_preview
+- correlation_id
+
+Allowed statuses:
+
+- queued
+- running
+- succeeded
+- failed
+- cancelled
+- expired
+
+## Persistence recommendation
+
+Use the existing runtime data directory and database layer where possible. This project already depends on SQLite through `better-sqlite3`, so the first implementation should prefer SQLite-backed persistence over Redis or external infrastructure unless a future scale requirement proves otherwise.
+
+The worker must use leases or status transitions so the same job is not processed twice after restart. Duplicate inbound WhatsApp events should map to the same job through an idempotency key based on channel, chat JID, sender JID, and original message ID.
+
+## Tool contract
+
+Async jobs may only run configured tools. The selected tool must be:
+
+- present in the configured catalog or manifest
+- enabled
+- allowed for the actor and chat type
+- allowed for the requested execution mode
+- valid against its input schema
+- within timeout, retry, and response-size limits
+
+A tool result must distinguish success from error. A failed upstream call, schema failure, timeout, or unsafe output must become a structured job failure, not a normal answer.
+
+## Worker behavior
+
+The worker should:
+
+1. Claim queued jobs whose scheduled time has arrived.
+2. Mark the job running with a start timestamp.
+3. Execute the configured tool with validated args.
+4. Record a structured result or structured error.
+5. Send exactly one completion message to the originating chat.
+6. Mark the job terminal.
+7. Preserve technical error details in logs and audit.
+
+If the process restarts, startup recovery should move stale running jobs back to queued or failed based on lease age and attempts.
+
+## User-facing wording
+
+Immediate acknowledgement, only after persistence:
+
+```text
+I am checking this now and will send the result here shortly. Reference: job #<id>.
+```
+
+Safe failure:
+
+```text
+I could not complete that check. The issue has been logged for review. Reference: job #<id>.
+```
+
+Normal chat fallback when no durable job exists:
+
+```text
+I cannot send a follow-up later from this chat unless a background job is created. I can only answer in this current WhatsApp reply using verified context or a configured tool result.
+```
+
+## Owner audit commands for a later phase
+
+- `/jobs`
+- `/job <id>`
+- `/cancel_job <id>`
+
+These must remain owner-only.
+
+## Test checklist
+
+Add tests for:
+
+- no future-reply wording from normal AI chat
+- job is persisted before acknowledgement
+- duplicate inbound message does not create duplicate jobs
+- running job recovery after restart
+- success path sends second message once
+- failure path sends safe second message and logs technical error
+- timeout moves job to failed or expired
+- cancelled jobs do not execute
+- disabled tools cannot create jobs
+- tools without async eligibility cannot create jobs
+- customers cannot invoke owner-only tools through async routing
+- group chats remain quiet unless explicitly allowed
+
+## Implementation phases
+
+Phase 1: Safety guard and docs.
+
+- Add or keep a response sanitizer that blocks future-reply promises unless a durable job reference is present.
+- Add focused tests for that guard.
+- Keep public live/current requests either same-turn through configured tools or honest about being unable to verify.
+
+Phase 2: Configured public-tool routing.
+
+- Extend tool JSON with natural-language metadata.
+- Runtime asks the tool catalog which tools are eligible.
+- Synchronous eligible tools answer in the same reply.
+- Async-eligible tools can proceed to the job gate.
+
+Phase 3: Durable jobs.
+
+- Add job store.
+- Add job worker.
+- Add idempotency.
+- Add restart recovery.
+- Add owner audit commands.
+- Add live WhatsApp verification.

package/docs/auth-persistence.md

@@ -0,0 +1,18 @@
+# Baileys Auth Persistence
+
+Keep this behavior unchanged.
+
+- Auth path in Docker: `/app/data/auth_state.json`.
+- Host volume: `${CHATBOT_DATA_DIR:-./data}:/app/data`.
+- Never bake auth into Docker.
+- Save immediately on `creds.update`.
+- Save again after `connection.open`.
+- Never persist completely empty `initAuthCreds()` as a real session.
+- Never overwrite a paired identity with a non-identity auth state.
+- Safe stop command: `docker compose stop -t 30 wa-chatbot`.
+
+Healthy auth state should show:
+
+- registered: true
+- hasMe: true
+- keyBuckets: 8 or more

package/docs/baileys-resilience.md

@@ -0,0 +1,103 @@
+# Baileys Resilience
+
+Branch: `feature/v2.7-baileys-resilience`
+
+This milestone strengthens reconnect behavior without changing customer, owner, group, or tool-command policy.
+
+## Scope
+
+This branch focuses on WhatsApp/Baileys runtime resilience only:
+
+- Terminal disconnect handling.
+- Bad-session loop prevention.
+- Rapid reconnect loop circuit breaking.
+- Pending reconnect timer cleanup.
+- Tests for known reconnect edge cases.
+
+It does not add custom HTTP tools, prompt changes, customer command access, owner policy changes, or tool catalog changes.
+
+## Reconnect behavior
+
+The reconnect manager still has one job: decide whether a closed Baileys connection should reconnect.
+
+Reconnects are allowed for transient failures such as timeout or closed connection.
+
+Reconnects are blocked for terminal session failures:
+
+- Bad session.
+- Logged out.
+- Forbidden.
+- Connection replaced.
+- Multi-device mismatch.
+- Device removed strings mapped to logged out.
+
+A terminal disconnect now latches until a clean `open` event or process/session reset. This prevents a later close event from accidentally restarting a known-invalid session.
+
+## Restart-required behavior
+
+`restartRequired` remains special. It schedules an immediate reconnect because this is part of the normal Baileys pairing/restart lifecycle.
+
+If restart-required events repeat too quickly, the rapid-loop circuit breaker can still pause reconnect attempts. This prevents a tight 515 loop from burning CPU or spamming logs.
+
+## Rapid-loop circuit breaker
+
+Defaults:
+
+```text
+rapidWindowMs=60000
+maxRapidAttempts=5
+rapidPauseMs=300000
+```
+
+Meaning: after more than 5 reconnect schedules inside 60 seconds, the next reconnect is delayed for at least 5 minutes.
+
+The circuit breaker can be tuned in tests through `ReconnectConfig`, but production defaults are intentionally conservative.
+
+## Tests added
+
+The reconnect manager tests now cover:
+
+- Logout/replaced/device-removed terminal cases.
+- Bad session terminal behavior.
+- Immediate restart-required reconnect.
+- Transient backoff reconnect.
+- Rapid-loop pause after repeated reconnects.
+- Terminal disconnect clearing a pending reconnect timer.
+- Reset after successful open connection.
+
+Run:
+
+```bash
+npm test -- src/gateway/reconnect/__tests__/manager.test.ts
+npm test
+```
+
+## Validation checklist
+
+Before merging:
+
+```bash
+npm run build
+npm test
+npm run cli -- health
+npm run cli -- config validate
+docker compose config
+docker compose build --no-cache
+docker compose run --rm wa-chatbot node dist/cli/index.js health
+docker compose run --rm wa-chatbot node dist/cli/index.js config validate
+```
+
+Then run the normal WhatsApp smoke path and confirm Docker still reaches:
+
+```text
+READY — processing messages now
+```
+
+## Guardrails
+
+- Do not reconnect forever after bad session or logout.
+- Do not fight another active WhatsApp session after connection replacement.
+- Do not let a terminal disconnect leave an old reconnect timer alive.
+- Do not let repeated disconnects create a tight reconnect loop.
+- Do not change customer slash tool blocking.
+- Do not start v2.3 custom tools from this branch.

package/docs/chatbot-architecture-research-plan.md

@@ -0,0 +1,80 @@
+# Chatbot Architecture Research Plan
+
+Date: 2026-06-28
+Branch: feature/support-inbox-profiles-labels
+
+This document records the next implementation direction for the WhatsApp chatbot.
+
+Decision: keep the current chatbot architecture. Do not port the full OpenClaw or Hermes Agent runtime. This repo is scoped to customer chat, owner controls, support inbox, media handling, search/news/fetch, and diagnostics.
+
+## Status
+
+Completed in this implementation pass:
+
+- [x] Add a chatbot config service with safe defaults.
+- [x] Add config/chatbot.config.json and config/chatbot.config.schema.json.
+- [x] Keep private keys in local environment only; use config for non-private behavior.
+- [x] Parse native WhatsApp mentions from message context info.
+- [x] Add group activation gate: direct chats process normally; group chats reply only when mentioned by default.
+- [x] Store unmentioned group messages as quiet context only.
+- [x] Prevent unmentioned group messages from calling AI or tools.
+- [x] Add group status command for the owner.
+- [x] Move search/news/fetch orchestration out of AIEngine into a tool layer.
+- [x] Add structured tool diagnostics with provider, endpoint, latency, attempts, and result count.
+- [x] Make web search fallback and news search fallback configurable.
+- [x] Add search status and tool status commands.
+- [x] Add doctor command to verify config, model keys, media, search, group mode, and runtime readiness.
+- [x] Add tests for config loading, mention parsing, group gating, search fallback, and diagnostics.
+
+Still pending:
+
+- [ ] Finish and push visual media caption handling if local changes are not already pushed.
+- [ ] Run build and full local test suite, then fix any compile or regression failures.
+- [ ] Smoke test direct chat, owner commands, group unmentioned, group mentioned, image caption, voice note, web fallback, and news behavior.
+
+Research references:
+
+- OpenClaw configuration: https://docs.openclaw.ai/gateway/configuration
+- OpenClaw WhatsApp: https://docs.openclaw.ai/channels/whatsapp
+- OpenClaw group messages: https://docs.openclaw.ai/channels/group-messages
+- OpenClaw tools config: https://docs.openclaw.ai/gateway/config-tools
+- Hermes Agent repo: https://github.com/NousResearch/hermes-agent
+- Hermes configuration: https://hermes-agent.nousresearch.com/docs/user-guide/configuration
+- Hermes architecture: https://hermes-agent.nousresearch.com/docs/developer-guide/architecture
+- Hermes messaging: https://hermes-agent.nousresearch.com/docs/user-guide/messaging
+- Hermes tools: https://hermes-agent.nousresearch.com/docs/user-guide/features/tools
+- Hermes issue 38647: https://github.com/NousResearch/hermes-agent/issues/38647
+- Hermes issue 20527: https://github.com/NousResearch/hermes-agent/issues/20527
+
+Implementation milestones:
+
+1. Finish and push visual media caption handling.
+2. Add a chatbot config service with safe defaults. DONE.
+3. Add config/chatbot.config.json and config/chatbot.config.schema.json. DONE.
+4. Keep private keys in local environment only; use config for non-private behavior. DONE.
+5. Parse native WhatsApp mentions from message context info. DONE.
+6. Add group activation gate: direct chats process normally; group chats reply only when mentioned by default. DONE.
+7. Store unmentioned group messages as quiet context only. DONE.
+8. Prevent unmentioned group messages from calling AI or tools. DONE.
+9. Add group status command for the owner. DONE.
+10. Move search/news/fetch orchestration out of AIEngine into a tool layer. DONE.
+11. Add structured tool diagnostics with provider, endpoint, latency, attempts, and result count. DONE.
+12. Make web search fallback and news search fallback configurable. DONE.
+13. Add search status and tool status commands. DONE.
+14. Add doctor command to verify config, model keys, media, search, group mode, and runtime readiness. DONE.
+15. Add tests for config loading, mention parsing, group gating, tool policy, search fallback, and diagnostics. DONE for config, mention parsing, group gating, search fallback, and diagnostics.
+
+Reasoning:
+
+OpenClaw provides the best pattern for configuration and group mention gating. Hermes provides the best pattern for config precedence, toolsets, and diagnostics. The current repo already has the right chatbot runtime shape, so we should borrow those patterns without adopting a full autonomous agent architecture.
+
+Definition of done:
+
+- Direct messages continue working.
+- Owner commands remain owner-only.
+- Group messages do not trigger replies unless mentioned.
+- Mentioned group messages can reply with recent group context.
+- Image captions route to media analysis.
+- Voice remains Groq Whisper.
+- Search and news are diagnosable and configurable.
+- All tests pass.

package/docs/config-write-commands.md

@@ -0,0 +1,78 @@
+# Config Write Commands
+
+Branch: `feature/v2.2-config-write-commands`
+
+This milestone adds safe CLI writes for `config/chatbot.config.json` only. It does not edit `config/tools.chatbot.json` or `config/prompts.chatbot.json`.
+
+## Commands
+
+```bash
+npm run cli -- config set <path> <json-value>
+npm run cli -- config unset <path>
+npm run cli -- config backup
+npm run cli -- config restore <backup-file>
+npm run cli -- config diff
+npm run cli -- config validate
+```
+
+Examples:
+
+```bash
+npm run cli -- config set debug.traceTools false
+npm run cli -- config set channels.whatsapp.groups.*.activation '"always"'
+npm run cli -- config set channels.whatsapp.groups.*.historyLimit 25
+npm run cli -- config set channels.whatsapp.groups.*.mentionPatterns '["@?bot","@?assistant"]'
+npm run cli -- config unset debug.traceTools
+npm run cli -- config backup
+npm run cli -- config diff
+```
+
+String values must be valid JSON strings, so quote them as JSON. For example, use `'"always"'`, not `always`.
+
+## Safety behavior
+
+- A backup is created before every successful mutation.
+- Candidate config is validated before it is written.
+- Writes use a temporary file followed by replacement of the target file.
+- Written files use stable pretty JSON.
+- Unknown paths are rejected.
+- Unsafe internal paths such as command-policy fields are rejected.
+- Prototype-pollution style path segments are rejected.
+- Secret-like values are rejected. Keep secrets in `.env`.
+- `config/tools.chatbot.json` remains read-only in this milestone.
+
+## Editable scope
+
+The first version intentionally supports operational chatbot settings only:
+
+- WhatsApp channel policy and group activation settings.
+- Web/news/fetch operational limits such as enabled flags, result counts, timeouts, cache TTL, and fetch content limit.
+- Media enabled flags and media timeouts.
+- Debug trace/redaction flags.
+
+It does not allow changing owner command policy, user command restrictions, tool allowlists, high-risk tool denial, model provider names, model IDs, custom tool definitions, prompt content, or secrets.
+
+## Validation checklist
+
+Run this before considering the milestone complete:
+
+```bash
+npm run build
+npm test
+npm run cli -- config validate
+npm run cli -- config backup
+npm run cli -- config diff
+```
+
+Manual mutation smoke:
+
+```bash
+npm run cli -- config backup
+npm run cli -- config set debug.traceTools false
+npm run cli -- config get debug.traceTools
+npm run cli -- config unset debug.traceTools
+npm run cli -- config get debug.traceTools
+npm run cli -- config validate
+```
+
+Expected result: `debug.traceTools` becomes `false` after set, returns to the default after unset, and validation still passes.