@lazyingart/agintiflow 0.20.200 → 0.20.201

package/.env.example

@@ -22,6 +22,8 @@ DEEPSEEK_API_KEY=
 OPENAI_API_KEY=
 AGINTI_ALLOW_HOSTED_IMAGE_PERCEPTION=false
 AGINTI_ALLOW_HOSTED_WEB_RESEARCH=false
+AGINTI_WEB_SEARCH_PROVIDER=auto
+# BRAVE_SEARCH_API_KEY=only_when_AGINTI_WEB_SEARCH_PROVIDER_is_brave
 AGINTI_ALLOW_HOSTED_JSON_SPECIALIST=false
 # A cross-provider writer needs both an explicit target and the permission below.
 AGINTI_WRITING_PROVIDER=

package/README.md

@@ -54,7 +54,7 @@ Most agent tools are either a chat box with hidden state or an expensive one-mod
 | Disciplined by default | `AGINTI.md` starts with a behavior contract: surface ambiguity, keep edits surgical, avoid speculative complexity, verify outcomes, and respect permission blockers. |
 | Role-based models | Route, main, spare, wrapper, and auxiliary image roles are separate. LocalLLM supplies the default fast/deep lanes; DeepSeek, OpenAI, OpenRouter, Qwen, and Venice are optional explicit routes. |
 | Writing without agent noise | `writing_specialist` drafts novels, books, scripts, essays, and paper prose in an isolated writing-only context on the active provider. Cross-provider writing requires explicit permission; ambient keys and model arguments cannot silently switch a LocalLLM session to a hosted model. The main agent then handles files, formatting, citations, checks, and artifacts. |
-| Visual and web evidence | `read_image` uses the loopback LocalLLM vision model in local sessions and never falls through to a hosted provider. `web_research` saves sourced snippet artifacts for local synthesis. OpenAI perception/research and wrapper second opinions require explicit provider/tool permission. |
+| Visual and web evidence | `read_image` keeps local pixels local. Fast search has provider fallback; exact page reads preserve hashes and metadata; `deep_research` adds resumable planning, parallel retrieval, exact-quote evidence, gap filling, synthesis, and citation auditing on the active provider. |
 | Scouts before big work | Parallel scouts can cheaply map architecture, tests, risks, symbols, and integration points before the main executor edits anything. |
 | SCS by default | Student-Committee-Supervisor mode adds a typed gate: committee drafts, student approves/monitors, supervisor executes. Use `/scs off` or `--no-scs` only when speed matters more than validation. |
 | AAPS for large workflows | AAPS describes top-down agentic pipeline scripts; AgInTiFlow can act as the interactive backend that validates, compiles, and executes those workflows. |
@@ -174,6 +174,7 @@ aginti --language de
 | Generate images | `/auxiliary image`, then ask for an image |
 | Read screenshots/images | `/image-read path/to/screenshot.png "what changed?"` |
 | Research with sources | `/web-research latest Android Gradle plugin official docs` |
+| Run deep research | `/deep-research standard compare current research-agent citation methods` |
 | Enable research wrapper | `/research-wrapper gpt-5.4-mini medium` |
 | Resume current project | `aginti resume` (`1` is newest/latest; Space shows more) |
 | Browse all sessions | `aginti resume --all-sessions` |
@@ -256,7 +257,7 @@ The website keeps the visual walkthrough in a carousel so this README can stay f
 | Writing specialist | A dedicated writing-only LLM call for prose, chapters, scripts, books, essays, research-paper sections, and revisions, with formatter handoff notes for Markdown/LaTeX/Final Draft. |
 | Patch workflow | Codex-style patch envelopes, unified diffs, exact replacements, hashes, compact diffs, and path guardrails. |
 | Parallel scouts | Optional scout calls for architecture, implementation, review, tests, git flow, research, symbol tracing, and dependency risk. |
-| Image reading and web research | LocalLLM sessions send pixels only to the loopback vision endpoint. `web_research` preserves source lists for the active model to synthesize. Hosted OpenAI and wrapper paths are explicit opt-ins, never credential-driven fallbacks. |
+| Image reading and web research | LocalLLM sessions keep pixels on the loopback vision endpoint. `web_search`, `read_web_page`, and `deep_research` provide fast discovery, exact source extraction, resumable evidence gathering, and claim-level citation audits. Hosted paths remain explicit opt-ins. |
 | SCS mode | Default Student-Committee-Supervisor quality gate with independent planning, execution, and validation roles. |
 | AAPS adapter | Optional `@lazyingart/aaps` integration for `.aaps` workflow init, validate, parse, compile, dry-run, and run commands. |
 | AgentLink | Local-first collaboration between AgInTi sessions through boards, typed messages, action contracts, safe summaries, and evidence bundles. |
@@ -404,6 +405,7 @@ More detail:
 - [docs/model-selection.md](docs/model-selection.md)
 - [docs/auxiliary-image-generation.md](docs/auxiliary-image-generation.md)
 - [docs/perception-and-web-research.md](docs/perception-and-web-research.md)
+- [docs/deep-research-engine.md](docs/deep-research-engine.md)
 - [docs/cli-i18n.md](docs/cli-i18n.md)
 - [docs/skillmesh.md](docs/skillmesh.md)
 
@@ -420,6 +422,7 @@ More detail:
 | Runtime modes and autonomy | [docs/runtime-modes-and-autonomy.md](docs/runtime-modes-and-autonomy.md) |
 | Skills and tools | [docs/skills-and-tools.md](docs/skills-and-tools.md) |
 | Image reading and web research | [docs/perception-and-web-research.md](docs/perception-and-web-research.md) |
+| Deep research engine | [docs/deep-research-engine.md](docs/deep-research-engine.md) |
 | Server-owned text-only fallback | [docs/safe-chat.md](docs/safe-chat.md) |
 | Skill Mesh | [docs/skillmesh.md](docs/skillmesh.md) |
 | Housekeeping logs | [docs/housekeeping.md](docs/housekeeping.md) |

package/docs/deep-research-engine.md

@@ -0,0 +1,172 @@
+# Deep Research Engine
+
+AgInTiFlow has two deliberately different research paths:
+
+- `web_search` is a fast lookup tool.
+- `deep_research` is a bounded, resumable evidence workflow for questions that
+  need multiple sources, disagreement analysis, or a durable report.
+
+Simple questions should stay on the fast path. A deep-research run spends more
+queries and model calls only when breadth, verification, and traceability add
+real value.
+
+## Architecture
+
+The implementation follows the strongest production patterns without making
+every query an unbounded agent swarm:
+
+1. **Plan**: the active provider decomposes the question into non-overlapping
+   subquestions, search queries, preferred source types, and exclusions.
+2. **Search**: queries run with bounded concurrency. The no-key default falls
+   back from DuckDuckGo HTML to Bing RSS. An explicitly configured Brave Search
+   route is optional. Multi-domain corpora receive separate bounded `site:`
+   queries matched to entity-specific subquestions instead of one fragile OR
+   expression.
+3. **Rank, diversify, and deduplicate**: canonical URLs remove tracking state;
+   primary, official, scholarly, and high-relevance results rank ahead of
+   summaries, while a bounded diversity penalty prevents one domain from
+   crowding every selected source.
+4. **Read exact sources**: `read_web_page` validates every redirect before the
+   next request, rejects private DNS resolutions, streams bounded bytes, strips
+   scripts/navigation, extracts article/main text and metadata, records hashes,
+   and marks all retrieved text as untrusted evidence.
+5. **Extract evidence**: isolated structured-output calls identify relevant
+   subquestions, claims, exact quotations, confidence, and limitations.
+6. **Verify**: deterministic code checks that quoted passages occur in the
+   exact retrieved source. Unverified quotations do not enter synthesis.
+7. **Fill gaps**: standard/deep runs may issue one bounded follow-up pass for
+   uncovered subquestions when query and source budgets remain.
+8. **Synthesize**: the active provider receives verified evidence rather than
+   arbitrary page text. Every substantive paragraph and finding carries source
+   IDs.
+9. **Audit**: deterministic code removes invented source IDs and unsupported
+   synthesis statements whose cited sources produced no verified evidence,
+   then reports claim, quotation, citation, question, domain, and
+   primary-source coverage.
+10. **Persist**: every stage checkpoints one JSON state file. The final cited
+    Markdown report is saved beside it and sent to the canvas.
+
+This combines the orchestrator/worker and separate citation-pass lessons
+described by [Anthropic's production research
+system](https://www.anthropic.com/engineering/multi-agent-research-system)
+with the iterative plan/search/read/gap/synthesize and background-resume model
+documented for [Gemini Deep
+Research](https://ai.google.dev/gemini-api/docs/deep-research). AgInTiFlow keeps
+the fan-out explicitly bounded and stays on the active DeepSeek, LocalLLM, or
+other selected provider.
+
+## Research Budgets
+
+| Depth | Queries | Sources | Gap passes | Intended use |
+| --- | ---: | ---: | ---: | --- |
+| `quick` | 3 | 6 | 0 | Small comparison or source check |
+| `standard` | 6 | 12 | 1 | Normal multi-source research |
+| `deep` | 10 | 20 | 1 | Broad review, contested topic, or decision report |
+
+The caller may reduce these budgets, but hard caps remain 12 queries and 24
+sources. Search/page concurrency is bounded to six and defaults to three or
+four.
+
+## Usage
+
+Interactive:
+
+```text
+/deep-research standard Compare current primary-source approaches to citation verification in research agents
+/deep-research deep Review recent optical biosensing methods, emphasizing original papers and unresolved limitations
+```
+
+An agent can call:
+
+```json
+{
+  "query": "What evidence supports the current design choices?",
+  "depth": "standard",
+  "sourcePolicy": "primary",
+  "domains": ["docs.example.org", "arxiv.org"]
+}
+```
+
+To resume a partial or completed same-query run, pass the returned
+`researchId`. A completed same-day run is returned from its checkpoint unless
+`refresh=true` is explicit. A transient run that retrieved zero allowed sources
+is marked failed, preserves its attempts, and retries retrieval on resume
+instead of caching an empty report as success. Checkpoint schema changes
+invalidate old cached runs automatically.
+
+## Artifacts
+
+Each session stores:
+
+```text
+artifacts/deep-research-RESEARCH_ID.json
+artifacts/deep-research-RESEARCH_ID.md
+```
+
+The JSON includes:
+
+- objective, plan, query/source budgets, and source policy;
+- every search attempt and provider failure/fallback;
+- canonical source URLs, metadata, retrieval timestamps, and SHA-256 hashes;
+- readable source text and ranked passages;
+- extracted claims, exact quotations, limitations, and question mappings;
+- missing questions, source diversity, and quote-verification rates;
+- final synthesis and citation audit;
+- stage/status fields used for crash-safe resume.
+
+## Search Providers
+
+The default `auto` policy uses public no-key providers:
+
+```text
+DuckDuckGo HTML -> Bing RSS
+```
+
+For an explicitly configured Brave Search account:
+
+```bash
+export AGINTI_WEB_SEARCH_PROVIDER=brave
+export BRAVE_SEARCH_API_KEY=...
+```
+
+An ambient key does not select Brave. The provider must be selected explicitly
+through configuration or a tool argument.
+
+## Safety And Evidence Rules
+
+- Only public HTTP/HTTPS URLs are accepted. Embedded credentials, single-label
+  internal hosts, loopback, RFC1918, link-local, multicast, `.local`, private
+  DNS resolutions, and redirected private targets are rejected before fetch.
+- Domain allowlists and blocklists apply to search results, direct reads, and
+  redirect targets.
+- Tracking parameters and fragments are removed before deduplication.
+- Page bytes are bounded while streaming; extracted characters are bounded too.
+- HTML scripts, forms, navigation, footers, and similar noise are removed.
+- Retrieved text is always labeled untrusted and never treated as tool or agent
+  instructions.
+- PDFs are hash-verified but not misrepresented as parsed by the dependency-free
+  page reader. A document/PDF tool remains responsible for full PDF extraction.
+- Synthesis sees only verified evidence records. Unknown citation IDs and
+  citations to sources without verified evidence are removed and reported.
+- Provider selection remains explicit. Deep research uses the active provider;
+  it does not silently escape LocalLLM or DeepSeek because another API key is
+  present.
+
+## Quality Signals
+
+The report audit exposes evidence quality rather than hiding it behind fluent
+prose:
+
+- question coverage;
+- readable source count;
+- primary/scholarly source count;
+- independent-domain count;
+- exact-quote verification rate;
+- statement-level citation coverage;
+- unknown/invented citation IDs;
+- unsupported synthesis statements removed before report rendering;
+- contradictions, uncertainties, and unresolved questions.
+
+These metrics are diagnostic gates, not a guarantee that a source is correct.
+High-stakes work still requires domain review and, where appropriate, direct
+inspection of the underlying paper, standard, dataset, or official record.

package/docs/perception-and-web-research.md

@@ -8,7 +8,9 @@ AgInTiFlow separates visual understanding, web search, and wrapper advice so eac
 | --- | --- | --- |
 | `read_image` | Read workspace screenshots, plots, scanned text, diagrams, or allowed remote image URLs. | Saves JSON and Markdown reports under `artifacts/perception/`, records image hashes, and sends the Markdown report to the canvas when used by an agent run. |
 | `web_search` | Cheap raw search snippets. | Returns compact titles, URLs, snippets, and fallback search URL. |
+| `read_web_page` | Read one exact public source instead of relying on a snippet. | Returns bounded article text, metadata, canonical URL, relevant passages, retrieval time, and content hash. |
 | `web_research` | Sourced research unit for current or external information. | Saves `artifacts/research/*-web-research.json` with query, mode, source list, and answer. |
+| `deep_research` | Plan, retrieve, verify, synthesize, and audit a genuinely multi-source question. | Saves resumable JSON state and a cited Markdown report with coverage metrics. |
 | `research_wrapper` | Read-only strict-JSON second opinion from the selected wrapper. | Saves `artifacts/wrappers/*-research-wrapper.json` with wrapper, model, reasoning, metadata, result, and raw fallback output. |
 
 ## Defaults
@@ -18,6 +20,8 @@ AgInTiFlow separates visual understanding, web search, and wrapper advice so eac
 - Codex image reading requires both `provider=codex` and explicitly enabled wrapper tools. It is not an automatic fallback.
 - OpenAI image reading defaults to `AGINTI_PERCEPTION_MODEL=gpt-5.4-mini` and `AGINTI_PERCEPTION_REASONING=medium`, with fallback models remaining inside the same explicitly selected OpenAI provider.
 - `web_research` defaults to lightweight snippet mode. The active LocalLLM can synthesize those returned snippets and sources during the next agent step without a second provider call.
+- `web_search` automatically falls back from DuckDuckGo HTML to Bing RSS. Brave Search is available only when explicitly selected and configured.
+- `deep_research` uses the active provider for planning, evidence extraction, and synthesis. It never treats another provider's ambient credentials as permission.
 - `web_research mode=openai` is accepted only for an active OpenAI session or when `allowHostedWebResearch=true`. A denied or failed hosted synthesis is returned with `ok=false`; any preserved snippets are labeled fallback evidence rather than hosted success.
 - `json_specialist` stays on the active provider. A different provider requires `allowHostedJsonSpecialist=true`; ambient hosted keys and model-generated provider arguments cannot escalate a LocalLLM session.
 - `research_wrapper` defaults to `AGINTI_RESEARCH_WRAPPER_MODEL=gpt-5.4-mini` and `AGINTI_RESEARCH_WRAPPER_REASONING=medium`.
@@ -30,6 +34,7 @@ AgInTiFlow separates visual understanding, web search, and wrapper advice so eac
 /image-read artifacts/screenshots/app.png what looks wrong?
 /image-read --codex artifacts/screenshots/app.png what looks wrong?
 /web-research latest Android Gradle plugin official docs
+/deep-research standard compare current citation-verification methods using primary sources
 /research-wrapper gpt-5.4-mini medium
 /research-wrapper off
 ```
@@ -40,6 +45,8 @@ Use `read_image` when pixels matter. Do not guess from filenames or surrounding
 
 Use `web_research` when freshness, sources, package docs, standards, or current external information matter. Use `domains` to restrict research to official or primary sources.
 
+Use `deep_research` when one lookup is insufficient: literature reviews, disputed claims, broad comparisons, technical decisions, or reports that require explicit coverage and citation auditing. See [Deep Research Engine](deep-research-engine.md).
+
 Use `research_wrapper` when a second model should cross-check image/web/research conclusions. The wrapper receives a strict JSON contract and should preserve uncertainty when it cannot directly inspect a source.
 
 ## Safety
@@ -47,6 +54,7 @@ Use `research_wrapper` when a second model should cross-check image/web/research
 - Local image paths must stay inside the workspace unless the run is explicitly trusted host mode.
 - `.env`, credential files, private keys, and secret-looking paths are blocked.
 - Remote images require web access and respect domain allowlists.
+- Web page reads reject private/local network targets and private redirect destinations.
 - Images are capped at 10 MB and four inputs per call.
 - Artifacts preserve hashes and metadata but never store API keys.
 - Provider selection is credential-neutral: an ambient key authenticates only a backend that the run already selected and permitted.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lazyingart/agintiflow",
-  "version": "0.20.200",
+  "version": "0.20.201",
   "type": "module",
   "description": "AgInTiFlow is a project-aware agent workspace for hybrid wet-dry R&D, hardware-aware intelligence, software automation, and industrial workflows.",
   "license": "Apache-2.0",
@@ -92,6 +92,7 @@
     "scripts/smoke-model-roles.js",
     "scripts/smoke-platform.js",
     "scripts/smoke-perception-research.js",
+    "scripts/smoke-deep-research.js",
     "scripts/smoke-permission-modes.js",
     "scripts/smoke-public-research-wrapper.js",
     "scripts/smoke-safe-chat.js",
@@ -154,6 +155,7 @@
     "smoke:model-roles": "node scripts/smoke-model-roles.js",
     "smoke:platform": "node scripts/smoke-platform.js",
     "smoke:perception-research": "node scripts/smoke-perception-research.js",
+    "smoke:deep-research": "node scripts/smoke-deep-research.js",
     "smoke:permission-modes": "node scripts/smoke-permission-modes.js",
     "smoke:public-research": "node scripts/smoke-public-research-wrapper.js",
     "smoke:safe-chat": "node scripts/smoke-safe-chat.js",
@@ -175,7 +177,7 @@
     "storage:migrate": "node bin/aginti-cli.js storage migrate",
     "publish:env": "node scripts/npm-publish-from-env.js publish --access public",
     "publish:env:whoami": "node scripts/npm-publish-from-env.js whoami",
-    "test": "npm run check && npm run smoke:localllm-provider && npm run smoke:localllm-model-tiers && npm run smoke:localllm-auto-max && npm run smoke:local-resource-policy && npm run smoke:context-budget-recovery && npm run smoke:session-runtime && npm run smoke:runtime-core && npm run smoke:progressive-tools && npm run smoke:truthful-completion && npm run smoke:writing-specialist-routing && npm run eval:local-first-agent && npm run eval:provider-attribution && npm run smoke:runtime-compat && npm run smoke:autoupdate && npm run smoke:web-api && npm run smoke:math-rendering && npm run smoke:web-ui && npm run smoke:web-autostart && npm run smoke:webapp-command && npm run smoke:web-port-fallback && npm run smoke:docker-command && npm run smoke:coding-tools && npm run smoke:dynamic-step-budget && npm run smoke:execution-policy && npm run smoke:aaps-adapter && npm run smoke:auxiliary-tools && npm run smoke:perception-research && npm run smoke:public-research && npm run smoke:safe-chat && npm run smoke:auth && npm run smoke:agentlink && npm run smoke:canvas-artifacts && npm run smoke:capabilities && npm run smoke:mcp && npm run smoke:model-roles && npm run smoke:platform && npm run smoke:permission-modes && npm run smoke:skills && npm run smoke:skillmesh && npm run smoke:tmux-tools && npm run smoke:long-jobs && npm run smoke:run-stdin && npm run smoke:cli-chat && npm run smoke:inbox",
+    "test": "npm run check && npm run smoke:localllm-provider && npm run smoke:localllm-model-tiers && npm run smoke:localllm-auto-max && npm run smoke:local-resource-policy && npm run smoke:context-budget-recovery && npm run smoke:session-runtime && npm run smoke:runtime-core && npm run smoke:progressive-tools && npm run smoke:truthful-completion && npm run smoke:writing-specialist-routing && npm run eval:local-first-agent && npm run eval:provider-attribution && npm run smoke:runtime-compat && npm run smoke:autoupdate && npm run smoke:web-api && npm run smoke:math-rendering && npm run smoke:web-ui && npm run smoke:web-autostart && npm run smoke:webapp-command && npm run smoke:web-port-fallback && npm run smoke:docker-command && npm run smoke:coding-tools && npm run smoke:dynamic-step-budget && npm run smoke:execution-policy && npm run smoke:aaps-adapter && npm run smoke:auxiliary-tools && npm run smoke:perception-research && npm run smoke:deep-research && npm run smoke:public-research && npm run smoke:safe-chat && npm run smoke:auth && npm run smoke:agentlink && npm run smoke:canvas-artifacts && npm run smoke:capabilities && npm run smoke:mcp && npm run smoke:model-roles && npm run smoke:platform && npm run smoke:permission-modes && npm run smoke:skills && npm run smoke:skillmesh && npm run smoke:tmux-tools && npm run smoke:long-jobs && npm run smoke:run-stdin && npm run smoke:cli-chat && npm run smoke:inbox",
     "pack:dry-run": "npm pack --dry-run",
     "smoke:capabilities": "node scripts/smoke-capabilities.js"
   },