@researai/deepscientist 1.5.14 → 1.5.16

package/docs/en/16_TELEGRAM_CONNECTOR_GUIDE.md

@@ -0,0 +1,134 @@
+# 16 Telegram Connector Guide
+
+Use this guide when you want DeepScientist to continue a quest through Telegram.
+
+Telegram in the current open-source runtime uses the built-in polling path:
+
+- no public webhook is required
+- the main credential is the BotFather token
+- direct messages can auto-bind to the latest active quest when enabled
+
+## 1. What Telegram support includes
+
+DeepScientist currently supports Telegram through:
+
+- `TelegramPollingService` for inbound polling
+- `GenericRelayChannel` for bindings, inbox/outbox, targets, and runtime status
+- `TelegramConnectorBridge` for direct outbound sends through the Bot API
+
+This means Telegram already fits the same quest-binding model as the other connector surfaces.
+
+## 2. Recommended setup path
+
+1. Open BotFather.
+2. Run `/newbot`.
+3. Save the generated bot token.
+4. Open `Settings > Connectors > Telegram`.
+5. Enable Telegram.
+6. Keep `transport: polling`.
+7. Fill `bot_token`.
+8. Save the connector.
+9. Send one real private message such as `/start` or `/help` to the bot.
+10. Return to DeepScientist and verify that the runtime has discovered the target conversation.
+
+## 3. Important config fields
+
+Main fields:
+
+- `enabled`
+- `transport`
+- `bot_name`
+- `bot_token`
+- `command_prefix`
+- `require_mention_in_groups`
+- `dm_policy`
+- `allow_from`
+- `group_policy`
+- `group_allow_from`
+- `groups`
+- `auto_bind_dm_to_active_quest`
+
+For the full field reference, see [01 Settings Reference](./01_SETTINGS_REFERENCE.md).
+
+## 4. Binding model
+
+Telegram conversations are normalized into quest-aware connector ids like:
+
+- `telegram:direct:<chat_id>`
+- `telegram:group:<chat_id>`
+
+DeepScientist binds quests to those normalized conversation ids, not to transient webhook state.
+
+Important rules:
+
+- one quest keeps local access plus at most one external connector target
+- direct messages can auto-follow the latest active quest when auto-bind is enabled
+- bindings can be changed later from the project settings page
+
+## 5. Group chat behavior
+
+By default:
+
+- Telegram direct messages are allowed
+- group behavior depends on `group_policy`
+- if `require_mention_in_groups` is `true`, the bot only reacts when explicitly mentioned or when a command is used
+
+This is the recommended default for larger shared groups.
+
+## 6. Outbound delivery
+
+Telegram outbound delivery currently focuses on text-first quest updates:
+
+- progress
+- milestone summaries
+- binding notices
+- structured quest replies
+
+The current bridge uses `sendMessage` through the Bot API.
+
+## 7. Troubleshooting
+
+### Telegram does not appear in Settings
+
+Telegram may be hidden by the system connector gate. Confirm that:
+
+- `config.connectors.system_enabled.telegram` is `true`
+
+### Validation says credentials are missing
+
+Check that:
+
+- `bot_token` is filled
+- or `bot_token_env` points at a real environment variable
+
+### The bot receives no messages
+
+Check that:
+
+- the bot token is correct
+- the bot was started from Telegram at least once
+- `transport` is still `polling`
+- no stale public webhook is intercepting updates
+
+### Group messages do not trigger the bot
+
+Check:
+
+- `group_policy`
+- `groups`
+- `group_allow_from`
+- `require_mention_in_groups`
+
+### The quest does not continue from Telegram
+
+Check that:
+
+- the conversation is bound to the intended quest
+- or `auto_bind_dm_to_active_quest` is enabled for direct-message pairing
+
+## 8. Related docs
+
+- [01 Settings Reference](./01_SETTINGS_REFERENCE.md)
+- [02 Start Research Guide](./02_START_RESEARCH_GUIDE.md)
+- [09 Doctor](./09_DOCTOR.md)
+- [13 Core Architecture Guide](./13_CORE_ARCHITECTURE_GUIDE.md)

package/docs/en/17_WHATSAPP_CONNECTOR_GUIDE.md

@@ -0,0 +1,126 @@
+# 17 WhatsApp Connector Guide
+
+Use this guide when you want DeepScientist to continue a quest through WhatsApp.
+
+The current open-source runtime prefers the local-session path for WhatsApp:
+
+- no public webhook is required for the recommended path
+- the local auth/session state stays on your machine
+- direct messages can auto-bind to the latest active quest when enabled
+
+## 1. What WhatsApp support includes
+
+DeepScientist currently supports WhatsApp through:
+
+- `WhatsAppLocalSessionService` for local session sync and inbound ingestion
+- `GenericRelayChannel` for bindings, inbox/outbox, targets, and runtime status
+- `WhatsAppConnectorBridge` for outbound delivery
+
+For the recommended path, outbound delivery is queued into the local-session outbox and handled by the local sidecar/session runtime.
+
+## 2. Recommended setup path
+
+1. Open `Settings > Connectors > WhatsApp`.
+2. Enable WhatsApp.
+3. Keep `transport: local_session`.
+4. Keep or choose a writable `session_dir`.
+5. Save the connector.
+6. Complete the local login flow for the WhatsApp session.
+7. Send one real message from WhatsApp.
+8. Return to DeepScientist and verify that the target conversation has been discovered.
+
+## 3. Important config fields
+
+Main fields:
+
+- `enabled`
+- `transport`
+- `bot_name`
+- `auth_method`
+- `session_dir`
+- `command_prefix`
+- `dm_policy`
+- `allow_from`
+- `group_policy`
+- `group_allow_from`
+- `groups`
+- `auto_bind_dm_to_active_quest`
+
+For the full field reference, see [01 Settings Reference](./01_SETTINGS_REFERENCE.md).
+
+## 4. Binding model
+
+WhatsApp conversations are normalized into quest-aware connector ids like:
+
+- `whatsapp:direct:<jid>`
+- `whatsapp:group:<jid>`
+
+DeepScientist binds quests to those normalized conversation ids instead of transient browser/session state.
+
+Important rules:
+
+- one quest keeps local access plus at most one external connector target
+- direct messages can auto-follow the latest active quest when auto-bind is enabled
+- bindings can be changed later from the project settings page
+
+## 5. Local-session runtime behavior
+
+The current open-source path is local-session oriented:
+
+- runtime status is mirrored into DeepScientist under connector logs
+- inbound messages are drained from the local session inbox
+- outbound messages are queued into the local session outbox
+
+This keeps the recommended WhatsApp path local-first.
+
+## 6. Group behavior
+
+By default:
+
+- direct messages can pair with the active quest
+- group behavior depends on `group_policy`
+- group allowlists can be enforced through `groups` and `group_allow_from`
+
+## 7. Troubleshooting
+
+### WhatsApp does not appear in Settings
+
+WhatsApp may be hidden by the system connector gate. Confirm that:
+
+- `config.connectors.system_enabled.whatsapp` is `true`
+
+### Validation says the connector is not ready
+
+Check that:
+
+- `transport` is `local_session`
+- `session_dir` points to a writable path
+
+### No discovered targets appear
+
+Check that:
+
+- the local login/session flow has completed
+- at least one real inbound message has reached the local session inbox
+
+### The quest does not continue from WhatsApp
+
+Check that:
+
+- the conversation is bound to the intended quest
+- or `auto_bind_dm_to_active_quest` is enabled for direct-message pairing
+
+### Outbound messages do not arrive
+
+Check that:
+
+- the local-session sidecar or local session processor is running
+- the local session outbox is being drained
+- the target JID is correct
+
+## 8. Related docs
+
+- [01 Settings Reference](./01_SETTINGS_REFERENCE.md)
+- [02 Start Research Guide](./02_START_RESEARCH_GUIDE.md)
+- [09 Doctor](./09_DOCTOR.md)
+- [13 Core Architecture Guide](./13_CORE_ARCHITECTURE_GUIDE.md)

package/docs/en/18_FEISHU_CONNECTOR_GUIDE.md

@@ -0,0 +1,136 @@
+# 18 Feishu Connector Guide
+
+Use this guide when you want DeepScientist to continue a quest through Feishu / Lark.
+
+The current open-source runtime prefers the built-in long-connection path:
+
+- no public event callback is required for the recommended setup
+- the main credentials are `app_id` and `app_secret`
+- direct messages can auto-bind to the latest active quest when enabled
+
+## 1. What Feishu support includes
+
+DeepScientist currently supports Feishu through:
+
+- `FeishuLongConnectionService` for inbound long-connection delivery
+- `GenericRelayChannel` for bindings, inbox/outbox, targets, and runtime status
+- `FeishuConnectorBridge` for direct outbound sends
+
+This means Feishu already fits the same quest-binding model as the other connector surfaces.
+
+## 2. Recommended setup path
+
+1. Open the Feishu / Lark developer platform.
+2. Create an app.
+3. Save `app_id` and `app_secret`.
+4. Open `Settings > Connectors > Feishu`.
+5. Enable Feishu.
+6. Keep `transport: long_connection`.
+7. Fill `app_id` and `app_secret`.
+8. Save the connector.
+9. Send one real message to the bot.
+10. Return to DeepScientist and confirm that the target conversation has been discovered.
+
+## 3. Important config fields
+
+Main fields:
+
+- `enabled`
+- `transport`
+- `bot_name`
+- `app_id`
+- `app_secret`
+- `api_base_url`
+- `command_prefix`
+- `dm_policy`
+- `allow_from`
+- `group_policy`
+- `group_allow_from`
+- `groups`
+- `require_mention_in_groups`
+- `auto_bind_dm_to_active_quest`
+
+For the full field reference, see [01 Settings Reference](./01_SETTINGS_REFERENCE.md).
+
+## 4. Binding model
+
+Feishu conversations are normalized into quest-aware connector ids like:
+
+- `feishu:direct:<chat_id>`
+- `feishu:group:<chat_id>`
+
+DeepScientist binds quests to those normalized conversation ids, not to transient callback payloads.
+
+Important rules:
+
+- one quest keeps local access plus at most one external connector target
+- direct messages can auto-follow the latest active quest when auto-bind is enabled
+- bindings can be changed later from the project settings page
+
+## 5. Group behavior
+
+By default:
+
+- direct messages are allowed
+- group behavior depends on `group_policy`
+- if `require_mention_in_groups` is `true`, the bot only reacts when explicitly mentioned or when a command is used
+
+This is the recommended default for larger shared workspaces.
+
+## 6. Outbound delivery
+
+Feishu outbound delivery currently focuses on text-first quest updates:
+
+- progress
+- milestone summaries
+- binding notices
+- structured quest replies
+
+The current bridge delivers through the Feishu Open Platform messaging API.
+
+## 7. Troubleshooting
+
+### Feishu does not appear in Settings
+
+Feishu may be hidden by the system connector gate. Confirm that:
+
+- `config.connectors.system_enabled.feishu` is `true`
+
+### Validation says credentials are missing
+
+Check that:
+
+- `app_id` is filled
+- `app_secret` is filled
+- or `app_secret_env` points at a real environment variable
+
+### No discovered targets appear
+
+Check that:
+
+- the app credentials are correct
+- the bot has received at least one real inbound message
+- `transport` is still `long_connection`
+
+### Group messages do not trigger the bot
+
+Check:
+
+- `group_policy`
+- `groups`
+- `group_allow_from`
+- `require_mention_in_groups`
+
+### The quest does not continue from Feishu
+
+Check that:
+
+- the conversation is bound to the intended quest
+- or `auto_bind_dm_to_active_quest` is enabled for direct-message pairing
+
+## 8. Related docs
+
+- [01 Settings Reference](./01_SETTINGS_REFERENCE.md)
+- [02 Start Research Guide](./02_START_RESEARCH_GUIDE.md)
+- [09 Doctor](./09_DOCTOR.md)
+- [13 Core Architecture Guide](./13_CORE_ARCHITECTURE_GUIDE.md)

package/docs/en/19_EXTERNAL_CONTROLLER_GUIDE.md

@@ -0,0 +1,226 @@
+# 19 External Controller Guide
+
+DeepScientist already exposes enough durable state to support an outer orchestration layer without patching core runtime code.
+
+This guide explains the minimal public pattern for external controllers that:
+
+- inspect recent quest state
+- decide whether the current run should continue
+- enqueue a routed follow-up message through the quest mailbox
+- optionally stop the current run through `quest_control`
+- record a durable report explaining why the guard fired
+
+This is intentionally lighter than a plugin framework.
+
+## When to use an external controller
+
+Use an external controller when the rule is:
+
+- project-specific
+- expensive to hard-code into global prompts or skills
+- better treated as outer governance than as core runtime behavior
+
+Examples:
+
+- publishability admission rules before paper-facing writing
+- repeated figure-polish loops that monopolize the frontier
+- lab-specific stop / branch policies
+
+## Public contracts you can rely on
+
+The safest extension surface is the existing durable runtime contract:
+
+- quest mailbox
+  - queued user-facing messages are stored under `.ds/user_message_queue.json`
+- recent quest state
+  - runtime state, artifact state, and connector-visible outputs are already durable files
+- daemon quest control
+  - `POST /api/quests/<quest_id>/control`
+- connector-visible durable reports
+  - write your own report under the quest tree so the next turn can cite it
+
+Prefer these contracts over prompt patching, private monkey-patching, or editing installed package files.
+
+## Minimal controller loop
+
+An external controller usually follows this sequence:
+
+1. Read the latest durable quest state.
+2. Decide whether a guard condition is active.
+3. Write a durable report describing:
+   - what was observed
+   - why it matters
+   - the recommended next route
+4. If intervention is needed:
+   - optionally stop the current run through `quest_control`
+   - enqueue one clear routed mailbox message for the next turn
+
+The mailbox message should explain the conclusion, not dump raw logs.
+
+## Example control flow
+
+```text
+read quest state
+-> detect low-yield loop or route violation
+-> write durable report
+-> stop current run if needed
+-> enqueue one mailbox message with the required next route
+```
+
+## Example `quest_control` request
+
+```bash
+curl -X POST http://127.0.0.1:20999/api/quests/<quest_id>/control \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "action": "stop",
+    "source": "external-controller"
+  }'
+```
+
+## Example mailbox intervention shape
+
+The exact queue file is runtime-owned, so your controller should preserve the existing schema and only append a normal user-style message payload.
+
+The message content should be short and actionable, for example:
+
+```text
+Hard control message from external orchestration layer: stop the current figure loop.
+Return to the main line and do one bounded route next:
+1. literature scout
+2. reference expansion
+3. manuscript body revision
+```
+
+## Example controllers you can adapt
+
+### Example 1: publishability admission guard
+
+This controller is useful when a quest is drifting into paper-facing writing before the evidence line is ready.
+
+Typical inputs:
+
+- the latest verification note
+- the current draft / summary state
+- recent baseline or utility results
+
+Typical stop condition:
+
+- the claimed paper direction still has weak support
+- one or more mandatory evidence items are still missing
+
+Typical intervention:
+
+1. Write `reports/publishability_guard.md` explaining the missing support.
+2. Stop the current write-heavy run through `quest_control`.
+3. Enqueue one mailbox message that routes the next turn back to `idea`, `analysis`, or a bounded evidence-repair step.
+
+Example mailbox text:
+
+```text
+External controller: do not continue manuscript-facing writing yet.
+Reason: the current evidence line does not pass the publishability admission gate.
+Next route: return to one bounded evidence-building step before write resumes.
+```
+
+### Example 2: figure-loop guard
+
+This controller is useful when the frontier is monopolized by repeated reopen / polish cycles on the same figure.
+
+Typical inputs:
+
+- recent figure artifact history
+- repeated reopen events
+- the latest review or summary note
+
+Typical stop condition:
+
+- the same figure has been reopened multiple times without improving the main claim
+- the next useful action is no longer figure polish, but evidence repair or manuscript revision
+
+Typical intervention:
+
+1. Write `reports/figure_loop_guard.md` summarizing the loop.
+2. Stop the current figure branch if needed.
+3. Enqueue one mailbox message that names exactly one next route.
+
+Example mailbox text:
+
+```text
+External controller: stop the current figure-polish loop.
+Reason: repeated reopen cycles are no longer improving the main evidence line.
+Next route: return to one bounded manuscript or analysis task.
+```
+
+## Example connector customization surface
+
+The control logic should stay connector-agnostic. In practice, adapting the same controller to a different connector usually means changing only the message profile, not the stop logic or durable report contract.
+
+For example:
+
+```yaml
+connector_profiles:
+  weixin:
+    summary_style: concise
+    max_route_options: 2
+    include_report_path: true
+  telegram:
+    summary_style: concise
+    max_route_options: 3
+    include_report_path: true
+  studio:
+    summary_style: detailed
+    max_route_options: 4
+    include_report_excerpt: true
+```
+
+The controller decision stays the same:
+
+- read durable state
+- decide whether to stop
+- write a durable report
+- enqueue one routed mailbox message
+
+What changes per connector is only how much detail you surface to the human operator.
+
+## Durable report shape
+
+Keep reports simple and auditable.
+
+A good report usually includes:
+
+- `generated_at`
+- `quest_id`
+- `status`
+- `recommended_action`
+- `blockers`
+- `evidence_summary`
+
+Markdown plus a machine-readable JSON companion is a practical pattern.
+
+## What not to rely on
+
+Avoid building controllers that depend on:
+
+- private prompt text offsets
+- internal temporary logs that are not documented durable state
+- patching installed package files inside `site-packages`
+- undocumented frontend-only state
+
+If a controller needs one of those, the contract is not stable enough yet.
+
+## Design recommendations
+
+- keep each controller focused on one question
+- prefer additive reports over hidden side effects
+- stop only when the next routed action is clear
+- keep domain- or lab-specific policy outside core defaults
+- treat external controllers as optional governance, not as required runtime plumbing
+
+## Good first controllers
+
+If you want to start small, begin with one of these:
+
+- a publishability admission guard for paper-mode quests
+- a figure-loop guard that stops repeated reopen cycles
+- a route-drift guard that blocks accidental `write` transitions before evidence is ready

package/docs/en/19_LOCAL_BROWSER_AUTH.md

@@ -0,0 +1,70 @@
+# Local Browser Auth
+
+DeepScientist can optionally protect the local web workspace with a generated 16-character password.
+
+This protection is local-first:
+
+- it applies to the browser UI
+- it also applies to `/api/*` requests from the browser
+- it is disabled by default unless you launch with `ds --auth true` or set `ui.auth_enabled: true`
+
+## What Happens On Startup
+
+When you run `ds --auth true`, DeepScientist starts the daemon and prints two important things in the terminal:
+
+- the local browser URL, such as `http://127.0.0.1:20999`
+- the generated local access password for that launch
+
+The browser URL is no longer expected to carry `?token=...`.
+
+If the browser is not already authenticated, DeepScientist shows a password modal on top of the landing page before it loads quests, docs, settings, or workspace data.
+
+## How To View The Password
+
+Use either of these:
+
+```bash
+ds --status
+```
+
+Read the `auth_token` field from the JSON output, or look at the terminal where `ds` was started.
+
+## How Login Persistence Works
+
+After the first successful login:
+
+- the browser stores the local auth state
+- later visits from the same browser usually open directly
+- restarting or reusing the managed daemon can rotate the password again
+
+If the password rotates, the browser will be asked to log in again.
+
+## How To Disable It
+
+Enable the browser password gate for one launch:
+
+```bash
+ds --auth true
+```
+
+In that mode:
+
+- the password modal is enabled
+- browser requests to `/api/*` require the local auth token
+
+To disable it again explicitly for one launch:
+
+```bash
+ds --auth false
+```
+
+In that mode:
+
+- the password modal is disabled
+- browser requests to `/api/*` do not require the local auth token
+
+## Practical Notes
+
+- This is not intended as internet-grade authentication. It is a local browser gate for machines you control.
+- If you share the machine or expose the port remotely, enable auth explicitly.
+- If you script against the local daemon from a browser client, authenticate first or disable auth explicitly for that launch.