@researai/deepscientist 1.5.15 → 1.5.17

package/docs/en/21_LOCAL_MODEL_BACKENDS_GUIDE.md

@@ -0,0 +1,283 @@
+# 21 Local Model Backends Guide: vLLM, Ollama, and SGLang
+
+This guide explains how to run DeepScientist against a local OpenAI-compatible model backend through Codex.
+
+The key point is simple:
+
+- current Codex CLI requires `wire_api = "responses"`
+- a backend that only works through `/v1/chat/completions` is not enough
+- you must verify `/v1/responses` before expecting `ds` or `ds doctor` to succeed
+
+There is one practical fallback:
+
+- if your backend is chat-only, you may still be able to use it through **Codex CLI `0.57.0`**
+- that older path can still work with `wire_api = "chat"` when the provider is configured at the top level
+- DeepScientist now checks this automatically during the Codex startup probe; if it sees `wire_api = "chat"` on any active provider config, it requires `codex-cli 0.57.0` before continuing
+
+## 1. What DeepScientist actually depends on
+
+DeepScientist does not talk to vLLM, Ollama, or SGLang directly.
+
+It talks to:
+
+- `codex`
+- and `codex` talks to your configured provider profile in `~/.codex/config.toml`
+
+So the compatibility chain is:
+
+1. your local backend
+2. Codex profile
+3. Codex startup probe
+4. DeepScientist runner
+
+If step 2 or step 3 fails, DeepScientist cannot start the Codex runner successfully.
+
+## 2. The current Codex rule you must know
+
+On the current Codex CLI:
+
+- `wire_api = "responses"` is supported
+- `wire_api = "chat"` is rejected
+
+In practice that means:
+
+- `vLLM`: recommended if its OpenAI-compatible server exposes `/v1/responses`
+- `Ollama`: only use it if your installed version exposes `/v1/responses`
+- `SGLang`: if your deployment only supports `/v1/chat/completions`, it is not compatible with the latest Codex runner
+
+## 2.1 Support summary
+
+| Backend | `/v1/chat/completions` | `/v1/responses` | Latest Codex | Codex `0.57.0` fallback |
+|---|---|---|---|---|
+| vLLM | yes | yes | supported | usually unnecessary |
+| Ollama | yes | depends on version | supported only when `/v1/responses` works | possible if it is chat-only |
+| SGLang | yes | often missing or incomplete | not supported when it is chat-only | possible fallback path |
+
+## 3. Test the backend first
+
+Before touching DeepScientist, verify the backend directly.
+
+### Step 1: list models
+
+```bash
+curl http://127.0.0.1:8004/v1/models \
+  -H "Authorization: Bearer 1234"
+```
+
+You need one real model id from this output, for example:
+
+```text
+/model/gpt-oss-120b
+```
+
+### Step 2: test chat completions
+
+```bash
+curl http://127.0.0.1:8004/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer 1234" \
+  -d '{
+    "model": "/model/gpt-oss-120b",
+    "messages": [
+      { "role": "user", "content": "Reply with exactly HELLO." }
+    ]
+  }'
+```
+
+If this works, the backend is at least OpenAI-chat-compatible.
+
+### Step 3: test Responses API
+
+```bash
+curl http://127.0.0.1:8004/v1/responses \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer 1234" \
+  -d '{
+    "model": "/model/gpt-oss-120b",
+    "input": "Reply with exactly HELLO."
+  }'
+```
+
+This is the decisive test.
+
+If `/v1/responses` fails, the latest Codex CLI will not work with this backend profile.
+
+## 4. What we actually observed on this server
+
+We tested the local backend at `http://127.0.0.1:8004/v1`.
+
+Observed behavior:
+
+- `GET /v1/models` succeeded
+- `POST /v1/chat/completions` succeeded
+- `POST /v1/responses` returned `500 Internal Server Error`
+- the `/v1/models` payload reported `owned_by: "sglang"`
+
+So this specific `8004` deployment behaves like a chat-compatible SGLang-style server, not a Codex-compatible Responses backend.
+
+That means:
+
+- it can answer raw chat requests
+- but it cannot currently be used by the latest Codex runner
+- and therefore DeepScientist cannot use it through the normal Codex path
+
+We also tested an older Codex path:
+
+- latest Codex + `wire_api = "responses"` failed against this backend
+- Codex `0.57.0` + top-level `model_provider` / `model` + `wire_api = "chat"` succeeded
+
+So for this server specifically:
+
+- **latest Codex path**: no
+- **Codex `0.57.0` fallback**: yes
+
+## 5. Codex profile example for a local Responses-compatible backend
+
+If your backend really supports `/v1/responses`, create a profile like this:
+
+```toml
+[model_providers.local_vllm]
+name = "local_vllm"
+base_url = "http://127.0.0.1:8004/v1"
+env_key = "LOCAL_API_KEY"
+wire_api = "responses"
+requires_openai_auth = false
+
+[profiles.local_vllm]
+model = "/model/gpt-oss-120b"
+model_provider = "local_vllm"
+```
+
+Then test Codex directly first:
+
+```bash
+export LOCAL_API_KEY=1234
+codex exec --profile local_vllm --json --cd /tmp --skip-git-repo-check - <<'EOF'
+Reply with exactly HELLO.
+EOF
+```
+
+If this fails, do not continue to DeepScientist yet.
+
+## 5.1 Chat-only fallback for Codex `0.57.0`
+
+If your backend only supports `/v1/chat/completions`, you can try this fallback path:
+
+1. install Codex `0.57.0`
+2. use `wire_api = "chat"`
+3. put `model_provider` and `model` at the top level
+
+Example:
+
+```toml
+model = "/model/gpt-oss-120b"
+model_provider = "localchat"
+approval_policy = "never"
+sandbox_mode = "workspace-write"
+
+[model_providers.localchat]
+name = "localchat"
+base_url = "http://127.0.0.1:8004/v1"
+env_key = "LOCAL_API_KEY"
+wire_api = "chat"
+requires_openai_auth = false
+```
+
+Then test:
+
+```bash
+export LOCAL_API_KEY=1234
+codex exec --json --cd /tmp --skip-git-repo-check - <<'EOF'
+Reply with exactly HELLO.
+EOF
+```
+
+If this older Codex path works, DeepScientist can usually reuse it with the same runner binary and profile strategy.
+
+## 6. DeepScientist commands after Codex works
+
+Once the direct Codex check works, run:
+
+```bash
+ds doctor --codex-profile local_vllm
+ds --codex-profile local_vllm
+```
+
+`ds doctor` is the canonical command.
+
+`ds docker` is only a legacy alias for `ds doctor`; it is not a Docker deployment command.
+
+If you want to persist it in DeepScientist:
+
+```yaml
+codex:
+  enabled: true
+  binary: codex
+  config_dir: ~/.codex
+  profile: local_vllm
+  model: inherit
+  model_reasoning_effort: high
+  approval_policy: never
+  sandbox_mode: danger-full-access
+```
+
+## 7. Backend compatibility summary
+
+### vLLM
+
+Recommended.
+
+Use it when:
+
+- `/v1/models` works
+- `/v1/responses` works
+- the model id is visible and stable
+
+If those are true, vLLM is the cleanest current local path for Codex + DeepScientist.
+
+### Ollama
+
+Conditionally supported.
+
+Use it only when:
+
+- your Ollama version exposes `/v1/responses`
+- your target model works through that endpoint
+
+If Ollama only gives you chat-completions semantics, it is not enough for the latest Codex CLI, but it may still be usable through Codex `0.57.0`.
+
+### SGLang
+
+Be careful.
+
+If your SGLang deployment behaves like this:
+
+- `/v1/chat/completions` works
+- `/v1/responses` fails
+
+then it is not currently compatible with the latest Codex runner.
+
+If you must use that backend anyway, the realistic fallback is Codex `0.57.0` with `wire_api = "chat"`.
+
+## 8. What to do if you only have chat-completions
+
+If your backend only supports `/v1/chat/completions`, you currently have three practical options:
+
+1. switch to a Responses-compatible backend such as vLLM
+2. upgrade to an Ollama release that really exposes `/v1/responses`
+3. downgrade the Codex CLI path to `0.57.0` and use `wire_api = "chat"`
+4. place a Responses-compatible proxy in front of the backend
+
+Right now, this is a Codex CLI limitation, not a DeepScientist-only setting mistake.
+
+## 9. Recommended workflow
+
+Use this order every time:
+
+1. test `/v1/models`
+2. test `/v1/responses`
+3. test `codex exec --profile <name>`
+4. test `ds doctor --codex-profile <name>`
+5. only then launch `ds --codex-profile <name>`
+
+If step 2 fails, stop there. Do not expect DeepScientist to succeed through the latest Codex path.

package/docs/en/91_DEVELOPMENT.md

@@ -179,6 +179,243 @@ match = service.resolve_binary("example-binary", preferred_tools=("example",))
 - keep clear source reporting such as `tinytex` versus `path`
 - add tests for registration, status, and binary resolution
 
+## Extending Core Runtime
+
+This section is the maintainer checklist for adding one new built-in MCP tool, one new skill, or one new connector.
+
+Keep the extension shape close to the repository's existing registries and contracts. Do not invent a parallel path when the current registry or prompt system already covers the need.
+
+### Add a built-in MCP tool
+
+Public MCP surface must stay limited to:
+
+- `memory`
+- `artifact`
+- `bash_exec`
+
+Do not add a new public namespace such as `git`, `connector`, or `runtime_tool`.
+
+If you need new Git behavior, add it under `artifact`.
+If you need new durable shell behavior, add it under `bash_exec`.
+
+#### Files to change
+
+- `src/deepscientist/mcp/server.py`
+  - add the new `@server.tool(...)` handler under `build_memory_server(...)`, `build_artifact_server(...)`, or `build_bash_exec_server(...)`
+- `src/deepscientist/mcp/context.py`
+  - only if the new tool needs extra quest/runtime context wiring
+- `src/deepscientist/runners/codex.py`
+  - if the tool should appear in built-in MCP approval policy defaults, add it under `_BUILTIN_MCP_TOOL_APPROVALS`
+- `docs/en/07_MEMORY_AND_MCP.md`
+  - update user-visible semantics if the tool changes how the namespace should be used
+- `docs/en/14_PROMPT_SKILLS_AND_MCP_GUIDE.md`
+  - update the built-in MCP description if the tool meaning materially changes
+
+#### Implementation rules
+
+1. Keep the handler thin. Put durable state changes in the underlying service layer such as `ArtifactService`, `MemoryService`, or `BashExecService`.
+2. Use `McpContext` for quest-local paths instead of reconstructing runtime state ad hoc.
+3. Use read-only annotations for non-mutating tools.
+4. If the tool changes durable quest state but does not itself send a user-visible message, follow the existing artifact watchdog pattern in `mcp/server.py`.
+5. Do not bypass the current namespace meaning. Example: do not hide shell execution inside `artifact`.
+
+#### Minimum test checklist
+
+- `tests/test_mcp_servers.py`
+  - namespace wiring and tool call behavior
+- `tests/test_memory_and_artifact.py`
+  - if the tool changes artifact or memory semantics
+- `tests/test_daemon_api.py`
+  - if API payloads or quest projections depend on the new tool's outputs
+
+### Add a skill
+
+Skills are discovered from disk. The canonical location is:
+
+- `src/skills/<skill_id>/SKILL.md`
+
+The runtime discovers skills through:
+
+- `src/deepscientist/skills/registry.py`
+
+The prompt builder projects them through:
+
+- `src/deepscientist/prompts/builder.py`
+- `src/deepscientist/skills/installer.py`
+
+#### Minimal skill shape
+
+Create a directory:
+
+```text
+src/skills/<skill_id>/
+```
+
+and add:
+
+```md
+---
+name: my-skill
+description: One-line purpose statement.
+skill_role: stage
+skill_order: 60
+---
+
+# My Skill
+...
+```
+
+Supported `skill_role` values are:
+
+- `stage`
+- `companion`
+- `custom`
+
+Optional projected agent files:
+
+- `src/skills/<skill_id>/agents/openai.yaml`
+- `src/skills/<skill_id>/agents/claude.md`
+
+#### Files to change
+
+- `src/skills/<skill_id>/SKILL.md`
+  - required
+- `src/deepscientist/skills/registry.py`
+  - update `_DEFAULT_STAGE_SKILLS` or `_DEFAULT_COMPANION_SKILLS` if this is a canonical built-in stage or companion skill rather than an ad hoc custom one
+- `src/deepscientist/prompts/builder.py`
+  - update `STAGE_MEMORY_PLAN` if the skill is a real stage that needs a first-class memory retrieval plan
+- `docs/en/14_PROMPT_SKILLS_AND_MCP_GUIDE.md`
+  - update the skill guide if the public workflow shape changed
+
+#### Skill rules
+
+1. Put execution discipline inside `SKILL.md`, not inside a central Python stage scheduler.
+2. Reuse the existing prompt contract language:
+   - interaction discipline
+   - tool discipline
+   - stage purpose
+   - completion or handoff rules
+3. If the skill is a canonical stage, give it a stable place in the stage order and memory plan.
+4. If the skill is only auxiliary, prefer `skill_role: companion` or `custom` rather than bloating the canonical stage chain.
+5. Remember that `SkillInstaller` mirrors skills into the active Codex/Claude projection trees. Keep paths and file names stable.
+
+#### Minimum test checklist
+
+- `tests/test_stage_skills.py`
+  - stage ordering and canonical skill availability
+- `tests/test_skill_contracts.py`
+  - frontmatter and skill contract expectations
+- `tests/test_prompt_builder.py`
+  - prompt builder output and visible skill path blocks
+
+### Add a connector
+
+Connectors have three distinct layers:
+
+1. config and validation
+2. inbound / outbound transport adaptation
+3. optional background runtime lifecycle
+
+For a simple connector, changing only one layer is usually not enough.
+
+#### Core files to inspect first
+
+- config defaults:
+  - `src/deepscientist/config/models.py`
+- config validation and live test behavior:
+  - `src/deepscientist/config/service.py`
+- connector profile support:
+  - `src/deepscientist/connector/connector_profiles.py`
+- inbound/outbound adaptation:
+  - `src/deepscientist/bridges/base.py`
+  - `src/deepscientist/bridges/connectors.py`
+  - `src/deepscientist/bridges/builtins.py`
+- channel delivery:
+  - `src/deepscientist/channels/base.py`
+  - `src/deepscientist/channels/builtins.py`
+- daemon lifecycle:
+  - `src/deepscientist/daemon/app.py`
+- API endpoints:
+  - `src/deepscientist/daemon/api/router.py`
+  - `src/deepscientist/daemon/api/handlers.py`
+- optional prompt fragment:
+  - `src/prompts/connectors/<connector>.md`
+
+#### Step-by-step connector checklist
+
+1. Add default config in `src/deepscientist/config/models.py`.
+   - If it is a system connector, add it to `SYSTEM_CONNECTOR_NAMES`.
+   - Add its default payload under `default_connectors()`.
+2. Add validation and config test behavior in `src/deepscientist/config/service.py`.
+   - validate required tokens, ids, transport, and live probe behavior
+3. Decide whether it is profileable.
+   - If yes, add a spec in `src/deepscientist/connector/connector_profiles.py`
+4. Add or extend a bridge in `src/deepscientist/bridges/connectors.py`.
+   - subclass `BaseConnectorBridge`
+   - implement inbound parsing and outbound formatting / delivery as needed
+5. Register the bridge in `src/deepscientist/bridges/builtins.py`.
+6. Register a channel in `src/deepscientist/channels/builtins.py`.
+   - use `GenericRelayChannel` when the standard relay flow is enough
+   - add a dedicated channel class only when the connector needs special outbound behavior
+7. If the connector needs a long-running runtime such as polling, gateway, QR session, or long connection:
+   - add the service class under `src/deepscientist/channels/`
+   - add daemon state fields in `DaemonApp.__init__`
+   - wire startup in `DaemonApp._start_background_connectors()`
+   - wire shutdown in `DaemonApp._stop_background_connectors()`
+8. If the connector needs custom web/API flows beyond the generic inbound route:
+   - update `src/deepscientist/daemon/api/router.py`
+   - update `src/deepscientist/daemon/api/handlers.py`
+9. If the connector needs connector-specific prompt behavior:
+   - add `src/prompts/connectors/<connector>.md`
+   - the prompt builder will load it automatically when the connector is active or bound
+10. Add user docs if it is a public connector.
+
+#### Connector rules
+
+1. Prefer the generic bridge/channel model. Do not special-case a connector inside unrelated quest logic.
+2. Keep transport adaptation in bridges and delivery/runtime behavior in channels.
+3. Do not add a connector-specific public MCP namespace.
+4. If the connector is public, keep route and status behavior consistent with the existing `/api/connectors/...` surface.
+5. If it is only a relay connector, try `GenericRelayChannel` first before writing a bespoke channel class.
+
+#### Minimum test checklist
+
+- `tests/test_connector_config_validation.py`
+  - config shape and required fields
+- `tests/test_connector_bridges.py`
+  - inbound parse and outbound formatting
+- connector-specific tests such as:
+  - `tests/test_telegram_connector.py`
+  - `tests/test_weixin_connector.py`
+  - `tests/test_qq_connector.py`
+  - `tests/test_whatsapp_local_session.py`
+  - or a new connector-specific test file
+- `tests/test_daemon_api.py`
+  - if new routes, status payloads, or connector availability behavior changed
+
+### Quick extension map
+
+Use this when you only need the shortest file-level reminder:
+
+- new MCP tool:
+  - `src/deepscientist/mcp/server.py`
+  - maybe `src/deepscientist/runners/codex.py`
+  - tests: `test_mcp_servers.py`
+- new skill:
+  - `src/skills/<skill_id>/SKILL.md`
+  - maybe `src/deepscientist/skills/registry.py`
+  - maybe `src/deepscientist/prompts/builder.py`
+  - tests: `test_stage_skills.py`, `test_skill_contracts.py`, `test_prompt_builder.py`
+- new connector:
+  - `src/deepscientist/config/models.py`
+  - `src/deepscientist/config/service.py`
+  - maybe `src/deepscientist/connector/connector_profiles.py`
+  - `src/deepscientist/bridges/*`
+  - `src/deepscientist/channels/*`
+  - maybe `src/deepscientist/daemon/app.py`
+  - maybe `src/prompts/connectors/<connector>.md`
+  - tests: connector validation + bridge + daemon/API coverage
+
 ## Documentation Rules
 
 When behavior changes:

package/docs/en/README.md

@@ -4,6 +4,8 @@ DeepScientist is not just a long-running autonomous scientific discovery system.
 
 2 minutes to install. 2 minutes to bind Weixin. 2 minutes to launch. Extremely fast and easy to use.
 
+Local Web access now starts without a password gate by default. If you want a generated 16-character browser password for one launch, run `ds --auth true`; DeepScientist then prints the password in the terminal and the browser can reuse the stored login after the first successful entry.
+
 It is also a workshop-style collaboration environment: let it keep moving autonomously, or step in anytime to collaborate, edit code, run the terminal yourself, or keep notes and plans in a Notion-style workspace.
 
 Use DeepScientist anywhere: on the server through TUI, in the browser through Web, on the phone through Weixin or QQ, and even on glasses through Rokid Glasses.
@@ -30,10 +32,16 @@ This page is the shortest path to the right document.
 
 - [00 Quick Start](./00_QUICK_START.md)
   Start here if you want to install DeepScientist, launch it locally, and create your first project.
+- [20 Workspace Modes Guide](./20_WORKSPACE_MODES_GUIDE.md)
+  Read this if you want to choose correctly between Copilot and Autonomous before creating a project.
+- [19 Local Browser Auth](./19_LOCAL_BROWSER_AUTH.md)
+  Read this if you want to understand the local password prompt, where to find the password, and how to disable it.
 - [05 TUI Guide](./05_TUI_GUIDE.md)
   Read this if your main surface is the terminal and you want one end-to-end path through `ds --tui`, quests, connectors, and cross-surface work.
 - [15 Codex Provider Setup](./15_CODEX_PROVIDER_SETUP.md)
-  Read this when you want to run DeepScientist through MiniMax, GLM, Volcengine Ark, Alibaba Bailian, or another Codex profile.
+  Read this when you want to run DeepScientist through MiniMax, GLM, Volcengine Ark, Alibaba Bailian Coding Plan, or another Codex profile.
+- [21 Local Model Backends Guide](./21_LOCAL_MODEL_BACKENDS_GUIDE.md)
+  Read this if you want to run DeepScientist through local OpenAI-compatible backends such as vLLM, Ollama, or SGLang.
 - [12 Guided Workflow Tour](./12_GUIDED_WORKFLOW_TOUR.md)
   Follow the real product flow from landing page to workspace, step by step.
 - [02 Start Research Guide](./02_START_RESEARCH_GUIDE.md)
@@ -43,6 +51,8 @@ This page is the shortest path to the right document.
 
 - [02 Start Research Guide](./02_START_RESEARCH_GUIDE.md)
   Explains the current frontend fields, derived contract fields, and practical examples.
+- [20 Workspace Modes Guide](./20_WORKSPACE_MODES_GUIDE.md)
+  Use this when the main question is not “how do I fill the form?” but “should this project start as Copilot or Autonomous?”.
 - [01 Settings Reference](./01_SETTINGS_REFERENCE.md)
   Use this when you need to configure runners, connectors, runtime defaults, or home paths.
 - [11 License And Risk Notice](./11_LICENSE_AND_RISK.md)
@@ -73,6 +83,8 @@ This page is the shortest path to the right document.
   Explains how the daemon, workspace, canvas, and connector views fit together.
 - [07 Memory and MCP](./07_MEMORY_AND_MCP.md)
   Explains memory, artifacts, and built-in MCP behavior.
+- [19 External Controller Guide](./19_EXTERNAL_CONTROLLER_GUIDE.md)
+  Shows how to build optional outer-orchestration guards on top of mailbox and `quest_control` without patching core runtime code.
 
 ## If something is broken
 
@@ -80,6 +92,8 @@ This page is the shortest path to the right document.
   Start here for diagnostics and common runtime problems.
 - [15 Codex Provider Setup](./15_CODEX_PROVIDER_SETUP.md)
   Check this if the problem is likely in your Codex profile, provider endpoint, API key, or model configuration.
+- [21 Local Model Backends Guide](./21_LOCAL_MODEL_BACKENDS_GUIDE.md)
+  Check this if the problem is specifically about local OpenAI-compatible backends and whether they support `/v1/responses`.
 - [01 Settings Reference](./01_SETTINGS_REFERENCE.md)
   Check this if the problem is likely caused by config, credentials, or connector setup.
 
@@ -88,4 +102,12 @@ This page is the shortest path to the right document.
 - [90 Architecture](./90_ARCHITECTURE.md)
   High-level system contracts and repository structure.
 - [91 Development](./91_DEVELOPMENT.md)
-  Maintainer-facing workflow and implementation notes.
+  Maintainer-facing workflow, implementation notes, and the concrete checklists for adding MCP tools, skills, and connectors.
+
+## Community
+
+Welcome to join the WeChat group for discussion.
+
+<p align="center">
+  <img src="../../assets/readme/wechat4.jpg" alt="DeepScientist WeChat group" width="360" />
+</p>