alexa-ai 2.1.1 → 2.1.2

package/CHANGELOG.md

@@ -4,6 +4,43 @@ All notable changes to `alexa-ai` are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
 [Semantic Versioning](https://semver.org/).
 
+## [2.1.2] — 2026-09-06
+
+### Changed
+- **`searchWeb()` now searches the web itself** instead of trusting DeepAI's
+  server-side web access. Observed live with `gpt-4o-mini`: the same query
+  produced one report without any sources and one with four invented links,
+  because the model skipped the search and wrote from memory. The engine now
+  queries Bing, Bing News, Wikipedia, Google News and DuckDuckGo in parallel
+  (no API key needed; the four feeds were verified live on 2026-09-06 —
+  DuckDuckGo's html endpoint serves a bot challenge from data-centre IPs, so
+  its lite endpoint is tried first and a challenge page yields no results), hands the results to the model as numbered material, and asks
+  it to cite result numbers and never write URLs. The `*Sources:*` block is
+  built from the search results only — cited first — so a link the model made
+  up can never reach the chat. Citation markers are removed from the prose;
+  URLs the model still types are turned into citations when they match a
+  result and dropped otherwise.
+- Result gained `grounded` (did the engine's search answer?), `providers`,
+  `via` (`'model'` | `'digest'`), and each source carries `date`, `provider`
+  and `cited`.
+- New options: `results` (bring your own search API's results), `search:false`,
+  `providers`, `maxResults`; constructor options `webSearch`,
+  `webSearchProviders`, `webSearchTimeout`, `webSearchResults`,
+  `webSearchProvider(query, { maxResults, signal })`.
+- When every provider fails the request falls back to DeepAI's web access as
+  before (`grounded:false`); when the search worked but the model call failed,
+  a plain digest of the results is returned (`via:'digest'`, still `ok:true`).
+
+### Added
+- `WebSearch` service (exported) with parsers for the DuckDuckGo html/lite
+  pages, RSS 2.0 news feeds and the MediaWiki search API, redirect unwrapping
+  (`duckduckgo.com/l/?uddg=`, `bing.com/news/apiclick.aspx?url=`), ad and
+  duplicate filtering, and provider round-robin.
+- `WebAnswer.formatResults()`, `extractCitations()`, `stripUrls()`, `digest()`.
+- The npm package now ships the test suite (`test/`), so the published
+  tarball is a complete snapshot of the repository at the tagged version:
+  `npm explore alexa-ai -- npm test`.
+
 ## [2.1.1] — 2026-09-06
 
 ### Fixed

package/README.md

@@ -439,11 +439,16 @@ which works on free chat keys. The result reports the route that answered
 **`summarizeText()`** follows the same pattern — `/api/summarization` first, a
 stateless chat request as the fallback.
 
-**`searchWeb()`** is a one-off research request with DeepAI's web access
-enabled. It applies the persona and formatting rules but touches no user's
-memory or history, so it can be called with no jid at all. The default answer
-is long-form and ready to send to WhatsApp — an intro, three to five
-`*Heading:*` sections with numbered `*Title*: detail` points, and one
+**`searchWeb()`** is a one-off research request that touches no user's
+memory or history, so it can be called with no jid at all. The engine does the
+searching itself — Bing, Bing News, Wikipedia, Google News and DuckDuckGo, in
+parallel, no API key needed — and hands the results to the model as numbered
+material. The model writes the report from them and cites result numbers; it
+is told never to write a URL. The `*Sources:*` block is then built from the
+search results only (cited ones first), so every link the bot shows is a page
+that actually came back from a search, never one the model made up. The
+default answer is long-form and ready to send to WhatsApp — an intro, three to
+five `*Heading:*` sections with numbered `*Title*: detail` points, and one
 `*Sources:*` block at the end:
 
 ```
@@ -471,15 +476,25 @@ answer still comes back under `minWords` (default 150), one follow-up turn asks
 the model to rewrite it in full, and the longer reply wins — `attempts` and
 `words` in the result show what happened.
 
-DeepAI reports the pages it used in two different ways depending on the model
-that answered: some send a structured web-results packet, others (observed
-with `gpt-4o-mini`) just write a `Sources:` list in prose. Both end up in the
-`sources` array as `{ title, url, description }`; the list is rendered exactly
-once in `text`, and `answer` holds the prose without it. Sentences in which
-the model talks about itself — *"I'm a language model"*, *"I can't browse the
-web"*, *"based on my training data"* — are removed, and leftover template
-placeholders are dropped. Third-party names in the research itself (news
-about OpenAI or Google) are kept verbatim.
+The result tells you which path answered:
+
+```js
+const r = await ai.searchWeb('coffee');
+// r.grounded   true  → the engine searched; sources are real search results
+//              false → no search results (host offline / providers blocked);
+//                      DeepAI's own web access answered, and its sources are
+//                      whatever the model reported — treat them with care
+// r.providers  ['bing', 'bing-news', 'wikipedia', 'google-news', 'duckduckgo'] — who answered
+// r.sources    [{ title, url, description, date, provider, cited }]
+// r.via        'model' | 'digest' (the model failed but the search worked:
+//                                  a plain list of the results is returned)
+// r.words, r.attempts, r.model
+```
+
+Sentences in which the model talks about itself — *"I'm a language model"*,
+*"I can't browse the web"*, *"based on my training data"* — are removed, and
+leftover template placeholders are dropped. Third-party names in the research
+itself (news about OpenAI or Google) are kept verbatim.
 
 ```js
 await ai.searchWeb('coffee', {
@@ -489,9 +504,37 @@ await ai.searchWeb('coffee', {
     maxSources: 3,            // how many to list in `text` (the array is not capped)
     language: 'Sinhala',      // answer language (default: the language of the query)
     instructions: 'focus on Sri Lanka',
+    providers: ['bing-news', 'google-news'],   // only news for this call
+    maxResults: 5,            // results handed to the model (default 8)
+    search: false,            // skip the built-in search, use DeepAI's web access
+});
+
+// Bring your own search API (Brave, Serper, Tavily, …): pass its results and
+// the built-in search is skipped — the sources block is built from them.
+await ai.searchWeb('coffee', {
+    results: [{ title, url, description, date }],
 });
 ```
 
+Engine-wide settings in the constructor:
+
+```js
+new AlexaAI({
+    webSearch: true,                       // false = never search; always use DeepAI's
+    webSearchProviders: ['bing', 'bing-news', 'wikipedia', 'google-news', 'duckduckgo'],
+    webSearchTimeout: 8000,                // per provider, ms
+    webSearchResults: 8,                   // results handed to the model
+    webSearchProvider: async (query, { maxResults, signal }) => [...],  // replace the built-ins
+});
+```
+
+The built-in providers are public pages and feeds, so they can change or be
+rate-limited; each one is best-effort and a failure just means fewer results.
+When none return anything the request falls back to DeepAI's server-side web
+access — which on the free models is exactly the behaviour that motivated this
+design: it often skips the search and writes a plausible report with invented
+links. Check `grounded` if that matters to you.
+
 ### Moderation and administration
 
 ```js
@@ -601,7 +644,8 @@ AlexaAI                     orchestrator; the only class the bot touches
 │                           no "Alexa Mini", no self-denial)
 ├── AmnesiaGuard            never lets her deny a memory she actually has
 ├── ImageDescriber          vision chain: documents -> DeepAI -> OCR -> fallback
-├── WebAnswer               searchWeb prompt; lifts "Sources:" lists out of prose
+├── WebSearch               the engine's own web search (Bing, Bing/Google News, Wikipedia, DuckDuckGo)
+├── WebAnswer               searchWeb prompt, citation handling, sources block
 ├── Media                   normalises every media input shape
 └── JidParser               normalises @lid / @s.whatsapp.net / @g.us
 ```
@@ -770,7 +814,7 @@ refresh it automatically. Reinstall and verify:
 ```bash
 npm uninstall alexa-ai
 npm install github:AlexaInc/deepai
-node -e "console.log(require('alexa-ai').version)"   # must print 2.1.1 or newer
+node -e "console.log(require('alexa-ai').version)"   # must print 2.1.2 or newer
 ```
 
 `AlexaAI.version` and `AlexaAI.methods()` let the bot assert this at startup;
@@ -780,10 +824,22 @@ node -e "console.log(require('alexa-ai').version)"   # must print 2.1.1 or newer
 `text`**
 Fixed in 2.1.1. Earlier builds asked the model for *"a short, direct answer"*
 and only read sources from DeepAI's structured packet, which `gpt-4o-mini`
-does not send. Reinstall as above and confirm the version is 2.1.1 or newer.
-If a particular model still answers briefly, check `attempts` / `words` in the
-result: the engine retries once below `minWords`, and a persistently short
-model is best swapped with the `model` option.
+does not send. If a particular model still answers briefly, check `attempts` /
+`words` in the result: the engine retries once below `minWords`, and a
+persistently short model is best swapped with the `model` option.
+
+**`searchWeb()` sometimes has no sources, or the sources are links that do
+not exist**
+Fixed in 2.1.2. Before, the model was both researcher and writer: DeepAI's
+server-side search ran unreliably on the free models, so the model often
+wrote from memory — one run had no sources, the next had four invented ones.
+The engine now searches the web itself and the sources block is built only
+from those results. If `sources` is still empty, look at `grounded` in the
+result: `false` means none of the search providers answered from your host
+(firewall, DNS, rate limit — enable `debug:true` to see each provider's
+error), and the reply came from DeepAI's own web access. Point
+`webSearchProvider` at a search API you control if the public endpoints are
+blocked where the bot runs.
 
 **`generateImage()` returns `{ ok: false, error: 'DEEPAI_QUOTA_EXCEEDED' }`**
 Both routes were refused: `/api/text2img` needs credits and the in-chat image

package/index.js

@@ -33,6 +33,7 @@ const ResponseFormatter = require('./src/services/ResponseFormatter');
 const TriggerDetector = require('./src/services/TriggerDetector');
 const ImageDescriber = require('./src/services/ImageDescriber');
 const WebAnswer = require('./src/services/WebAnswer');
+const WebSearch = require('./src/services/WebSearch');
 const JidParser = require('./src/utils/JidParser');
 const Media = require('./src/utils/Media');
 const StreamParser = require('./src/core/StreamParser');
@@ -63,6 +64,7 @@ module.exports.ResponseFormatter = ResponseFormatter;
 module.exports.TriggerDetector = TriggerDetector;
 module.exports.ImageDescriber = ImageDescriber;
 module.exports.WebAnswer = WebAnswer;
+module.exports.WebSearch = WebSearch;
 module.exports.JidParser = JidParser;
 module.exports.Media = Media;
 module.exports.StreamParser = StreamParser;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "alexa-ai",
-  "version": "2.1.1",
+  "version": "2.1.2",
   "description": "AI engine for the Alexa WhatsApp bot: DeepAI-powered chat with PostgreSQL-backed long-term memory, cross-chat identity (@lid <-> phone), vision/OCR, image generation and web search.",
   "main": "index.js",
   "scripts": {
@@ -40,6 +40,7 @@
     "index.js",
     "src/",
     "examples/",
+    "test/",
     "README.md",
     "CHANGELOG.md",
     "LICENSE"

package/src/AlexaAI.js

@@ -17,6 +17,7 @@ const IdentityResolver = require('./services/IdentityResolver');
 const TriggerDetector = require('./services/TriggerDetector');
 const ImageDescriber = require('./services/ImageDescriber');
 const WebAnswer = require('./services/WebAnswer');
+const WebSearch = require('./services/WebSearch');
 const StreamParser = require('./core/StreamParser');
 const JidParser = require('./utils/JidParser');
 const Media = require('./utils/Media');
@@ -69,6 +70,7 @@ class AlexaAI {
 
         this.prompts = new PromptBuilder(this.config);
         this.vision = new ImageDescriber(this.client, this.config);
+        this.webSearch = new WebSearch(this.config);
 
         // Persona-aware guards (renaming the assistant renames these too).
         this.identityGuard = new IdentityGuard({
@@ -785,25 +787,35 @@ class AlexaAI {
      * One-off, stateless web research request — for "search the web for X"
      * commands that must not touch anyone's memory.
      *
-     * By default the answer is long-form: an intro, three to five `*Heading:*`
-     * sections with numbered `*Headline*: detail` points (about 300–450
-     * words), followed by one `*Sources:*` block. Sources come from DeepAI's
-     * web-result packet when the model sends one, and are otherwise lifted
-     * out of the "Sources:" list the model writes — either way they are
-     * returned in `sources` and rendered exactly once in `text`.
+     * HOW IT WORKS
+     * ------------
+     * 1. The engine searches the web itself (`WebSearch`: DuckDuckGo, Bing
+     *    News, Google News, Wikipedia — no API key needed) and collects real
+     *    pages with titles, snippets and dates.
+     * 2. Those results go to the model as numbered material. The model writes
+     *    the report from them and cites result numbers; it is told never to
+     *    write a URL.
+     * 3. The `*Sources:*` block is built from the search results only —
+     *    cited ones first. A URL the model wrote itself is never shown.
      *
-     * Small models sometimes ignore the layout and answer in a sentence. When
-     * a long-form answer comes back under `minWords`, one follow-up turn asks
-     * the model to rewrite it in full; the longer of the two replies wins.
+     * When the search returns nothing (offline host, all providers blocked)
+     * the request falls back to DeepAI's server-side web access, which is
+     * unreliable on free models: it may skip the search and invent pages.
+     * `grounded` in the result tells you which path answered.
      *
-     * The persona still applies (WhatsApp formatting, no renaming of the
-     * assistant), but third-party vendor names in the research itself are
-     * kept, and the identity/memory guards run without a user context, so
-     * this is safe to call with no jid at all.
+     * Default output is long-form: an intro, three to five `*Heading:*`
+     * sections with numbered `*Headline*: detail` points (about 300–450
+     * words), then one `*Sources:*` block. A long-form reply under `minWords`
+     * is retried once; the longer reply wins.
      *
      * @param {string} query
      * @param {object} [opts]
      * @param {'long'|'short'} [opts.detail='long']   `short` = 2–4 sentences
+     * @param {Array<{title?:string,url:string,description?:string,date?:string}>} [opts.results]
+     *        results from the host application's own search API (skips the built-in search)
+     * @param {boolean} [opts.search=true]            `false` = skip the built-in search, use DeepAI's
+     * @param {string[]} [opts.providers]             subset of WebSearch.PROVIDERS for this call
+     * @param {number}  [opts.maxResults]             results handed to the model (default config.webSearchResults)
      * @param {number}  [opts.minWords=150]           long form only: retry once below this (0 disables)
      * @param {boolean} [opts.includeSources=true]    append the *Sources:* block to `text`
      * @param {number}  [opts.maxSources=5]           sources listed in `text` (the array is not capped)
@@ -812,12 +824,12 @@ class AlexaAI {
      * @param {string}  [opts.model]
      * @param {string}  [opts.userName]
      * @param {AbortSignal} [opts.signal]
-     * @returns {Promise<{ok:boolean, text:string, answer:string, sources:Array<{title:string|null,url:string|null,description:string|null}>, words:number, attempts:number, model?:string|null, error?:string, message?:string}>}
+     * @returns {Promise<{ok:boolean, text:string, answer:string, sources:Array<{title:string|null,url:string|null,description:string|null,date?:string|null,provider?:string,cited?:boolean}>, grounded:boolean, providers:string[], words:number, attempts:number, via:'model'|'digest', model?:string|null, error?:string, message?:string}>}
      */
     async searchWeb(query, opts = {}) {
         const question = String(query ?? '').trim();
         const failure = (error, message, extra = {}) => ({
-            ok: false, text: '', answer: '', sources: [], words: 0, attempts: 0, error, message, ...extra,
+            ok: false, text: '', answer: '', sources: [], grounded: false, providers: [], words: 0, attempts: 0, via: 'model', error, message, ...extra,
         });
         if (!question) return failure('VALIDATION_ERROR', 'searchWeb(): query is required');
 
@@ -826,18 +838,48 @@ class AlexaAI {
         const maxSources = Number.isFinite(opts.maxSources) ? opts.maxSources : 5;
         const minWords = detail === 'long' && Number.isFinite(opts.minWords) ? Math.max(0, opts.minWords) : detail === 'long' ? 150 : 0;
 
-        const request = { search: true, webAccess: true, model: opts.model, signal: opts.signal };
+        // ---- 1. search ------------------------------------------------------
+        let results = WebSearch.normalise(opts.results, 'caller');
+        let providers = results.length ? ['caller'] : [];
+        if (!results.length && opts.search !== false) {
+            const found = await this.webSearch.search(question, {
+                providers: opts.providers,
+                maxResults: opts.maxResults,
+                signal: opts.signal,
+            });
+            results = found.results;
+            providers = found.providers;
+            if (!results.length && this.config.debug) {
+                this.log.debug?.(
+                    `[AlexaAI] searchWeb: no web results for "${question}" ` +
+                        (found.errors.length ? `(${found.errors.map((e) => `${e.provider}: ${e.message}`).join('; ')})` : '') +
+                        ' — falling back to DeepAI web access'
+                );
+            }
+        }
+        const grounded = results.length > 0;
+
+        // ---- 2. ask the model ---------------------------------------------
+        const request = { search: !grounded, webAccess: !grounded, model: opts.model, signal: opts.signal };
         const messages = this.prompts.build({
-            message: WebAnswer.prompt(question, { detail, language: opts.language, instructions: opts.instructions }),
+            message: WebAnswer.prompt(question, {
+                detail,
+                results: grounded ? results : null,
+                language: opts.language,
+                instructions: opts.instructions,
+            }),
             memories: {},
             history: [],
             userName: opts.userName || null,
         });
 
+        let reply;
+        let result;
+        let attempts = 0;
         try {
-            let reply = await this.client.chatDetailed(messages, request);
-            let result = this._webAnswerFrom(reply);
-            let attempts = 1;
+            reply = await this.client.chatDetailed(messages, request);
+            result = this._webAnswerFrom(reply, results);
+            attempts = 1;
 
             if (minWords && result.words < minWords && result.answer) {
                 // Second chance: keep the first reply in the transcript so the
@@ -845,12 +887,12 @@ class AlexaAI {
                 const retryMessages = [
                     ...messages,
                     { role: 'assistant', content: reply.text },
-                    { role: 'user', content: WebAnswer.expandPrompt(question, result.words) },
+                    { role: 'user', content: WebAnswer.expandPrompt(question, result.words, { grounded }) },
                 ];
                 attempts = 2;
                 try {
                     const second = await this.client.chatDetailed(retryMessages, request);
-                    const candidate = this._webAnswerFrom(second, result.sources);
+                    const candidate = this._webAnswerFrom(second, results, result.sources);
                     if (candidate.words > result.words) {
                         reply = second;
                         result = candidate;
@@ -859,34 +901,75 @@ class AlexaAI {
                     this.log.debug?.(`[AlexaAI] searchWeb expansion failed, keeping first reply: ${err.message}`);
                 }
             }
-
-            const { answer, sources, words } = result;
-            if (!answer && !sources.length) return failure('DEEPAI_EMPTY', 'DeepAI returned no answer', { attempts });
-
-            const text = WebAnswer.render(answer, sources, { includeSources: includeSources || !answer, maxSources });
-            return { ok: true, text, answer, sources, words, attempts, model: reply.model || null };
         } catch (err) {
             this.log.warn?.(`[AlexaAI] searchWeb failed: ${err.message}`);
-            return failure(err.code || 'SEARCH_FAILED', err.message);
+            if (!grounded) return failure(err.code || 'SEARCH_FAILED', err.message, { attempts });
+            // The search worked even though the model did not: show the results.
+            const sources = AlexaAI._groundedSources(results, []);
+            return {
+                ok: true,
+                text: WebAnswer.render(WebAnswer.digest(results), sources, { includeSources, maxSources }),
+                answer: WebAnswer.digest(results),
+                sources,
+                grounded,
+                providers,
+                words: 0,
+                attempts,
+                via: 'digest',
+                model: null,
+                error: err.code || 'SEARCH_FAILED',
+                message: err.message,
+            };
         }
+
+        // ---- 3. assemble ----------------------------------------------------
+        const { answer, sources, words } = result;
+        if (!answer && !sources.length) return failure('DEEPAI_EMPTY', 'DeepAI returned no answer', { attempts, grounded, providers });
+
+        const text = WebAnswer.render(answer, sources, { includeSources: includeSources || !answer, maxSources });
+        return { ok: true, text, answer, sources, grounded, providers, words, attempts, via: 'model', model: reply.model || null };
     }
 
     /**
      * @private Turn one chat reply into `{ answer, sources, words }`.
-     * Structured packet sources come first (they carry descriptions), then
-     * whatever the model listed itself; duplicates collapse by URL.
+     *
+     * Grounded (we searched): citation markers `[n]` are removed from the
+     * prose and decide the order of the sources; any URL the model wrote is
+     * discarded. Ungrounded: DeepAI's web-results packet comes first (it
+     * carries descriptions), then whatever the model listed; duplicates
+     * collapse by URL.
      */
-    _webAnswerFrom(reply, carried = []) {
+    _webAnswerFrom(reply, results = [], carried = []) {
         let formatted = ResponseFormatter.format(MemoryExtractor.strip(reply.text));
         // Keep third-party vendor names: this is research output, not the
         // assistant introducing herself. WebAnswer removes the sentences in
         // which the model talks about *itself*.
         formatted = this.identityGuard.sanitise(formatted, false, { vendors: false });
         const parsed = WebAnswer.parse(formatted);
+
+        if (results.length) {
+            const { text, cited } = WebAnswer.extractCitations(WebAnswer.stripUrls(parsed.text, results), results.length);
+            return { answer: text, sources: AlexaAI._groundedSources(results, cited), words: WebAnswer.wordCount(text) };
+        }
         const sources = WebAnswer.mergeSources(AlexaAI._sources(reply.webResults), parsed.sources, carried);
         return { answer: parsed.text, sources, words: WebAnswer.wordCount(parsed.text) };
     }
 
+    /** @private search results as sources: cited ones first, in citation order. */
+    static _groundedSources(results, cited) {
+        const order = [...cited.map((n) => n - 1), ...results.map((_, i) => i).filter((i) => !cited.includes(i + 1))];
+        return order
+            .filter((i) => results[i])
+            .map((i) => ({
+                title: results[i].title || null,
+                url: results[i].url,
+                description: results[i].description || null,
+                date: results[i].date || null,
+                provider: results[i].provider || null,
+                cited: cited.includes(i + 1),
+            }));
+    }
+
     /** Is DeepAI reachable and is the key still good? */
     async deepaiHealth() {
         const started = Date.now();

package/src/core/Config.js

@@ -107,6 +107,15 @@ class Config {
         ]);
         this.imageModel = opts.imageModel || 'text2img';
 
+        // ---- Engine web search (searchWeb) ----------------------------------
+        // The engine searches first and hands real results to the model, so
+        // every URL the bot shows comes from a search, never from the model.
+        this.webSearch = opts.webSearch !== false;
+        this.webSearchProviders = Config._list(opts.webSearchProviders, ['bing', 'bing-news', 'wikipedia', 'google-news', 'duckduckgo']);
+        this.webSearchTimeout = Config._int(opts.webSearchTimeout, 8000, 500, 60000);
+        this.webSearchResults = Config._int(opts.webSearchResults, 8, 1, 30);
+        this.webSearchProvider = typeof opts.webSearchProvider === 'function' ? opts.webSearchProvider : null;
+
         // ---- Chat request feature flags (mirrors the deepai.org client) -----
         this.enabledTools = Config._list(opts.enabledTools, ['image_generator', 'image_editor']);
         this.toolActivitySupport = opts.toolActivitySupport !== false;