@eeacms/volto-eea-chatbot 3.0.1 → 4.0.0

package/CHANGELOG.md

@@ -4,7 +4,32 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
-### [3.0.1](https://github.com/eea/volto-eea-chatbot/compare/3.0.0...3.0.1) - 5 August 2026
+### [4.0.0](https://github.com/eea/volto-eea-chatbot/compare/3.0.1...4.0.0) - 13 August 2026
+
+#### :bug: Bug Fixes
+
+- fix: correct 100.00% → High in ClaimModal snapshot [Tiberiu Ichim - [`2f3e620`](https://github.com/eea/volto-eea-chatbot/commit/2f3e620627faae3b050f0cb6ba1e3db4015dc5db)]
+- fix: avoid Betterleaks false positive on LLM_API_KEY example [Tiberiu Ichim - [`44e05bc`](https://github.com/eea/volto-eea-chatbot/commit/44e05bc74a042e7f08cc0e6878930f9acc1964b9)]
+- fix: filter skipped claims in span processor to prevent text duplication [Tiberiu Ichim - [`9477a42`](https://github.com/eea/volto-eea-chatbot/commit/9477a42d363d6961f4def628ed567a2409b316ea)]
+
+#### :house: Internal changes
+
+- chore: bump version to 4.0.0 (breaking change) [Tiberiu Ichim - [`e1038b6`](https://github.com/eea/volto-eea-chatbot/commit/e1038b6db64612f7c772a898b99ff802b3c133e9)]
+- style: Automated code fix [eea-jenkins - [`03891e2`](https://github.com/eea/volto-eea-chatbot/commit/03891e24e48ccd0eccb55d4ed42aabbcdf34251f)]
+
+#### :house: Documentation changes
+
+- docs: document rag-facts-check backend dependency for quality checks [Tiberiu Ichim - [`123c7ea`](https://github.com/eea/volto-eea-chatbot/commit/123c7ea5d56ee48e6d5c787049a010b6bebd5aef)]
+- docs: update README for rag-fact-checker integration [Tiberiu Ichim - [`4ae32b4`](https://github.com/eea/volto-eea-chatbot/commit/4ae32b41e838176d4a3ba1ea55367fc21e1a8cae)]
+
+#### :hammer_and_wrench: Others
+
+- test: update ClaimModal snapshots for categorical score labels [Tiberiu Ichim - [`3794be1`](https://github.com/eea/volto-eea-chatbot/commit/3794be1fc112b9f209a58c6577f99649350b1d59)]
+- Deduplicate sources by text before sending to fact-checker [Tiberiu Ichim - [`67dc793`](https://github.com/eea/volto-eea-chatbot/commit/67dc793f955058d9ba5dbc92ea23a652e4586172)]
+- Fix: use doc.content from final_documents when tool packets lack content [Tiberiu Ichim - [`4174212`](https://github.com/eea/volto-eea-chatbot/commit/4174212a90742c139ad2399f7a07feb99d619fbc)]
+- Replace numeric claim scores with categorical labels [Tiberiu Ichim - [`d0aaf77`](https://github.com/eea/volto-eea-chatbot/commit/d0aaf779a69340386af81e9af4aa5147d93be388)]
+- Send structured sources with document titles to fact-checker [Tiberiu Ichim - [`190c79b`](https://github.com/eea/volto-eea-chatbot/commit/190c79b2a888a7d655157e683633fd9901b8871c)]
+### [3.0.1](https://github.com/eea/volto-eea-chatbot/compare/3.0.0...3.0.1) - 7 August 2026
 
 #### :bug: Bug Fixes
 
@@ -35,7 +60,6 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 #### :house: Internal changes
 
 - chore: bump version to 3.0.0 (EEA design system dependency) [skip ci] [Tiberiu Ichim - [`bc5a8c1`](https://github.com/eea/volto-eea-chatbot/commit/bc5a8c1f72d4e706b096cb96d0d24c2c3f5ac0f8)]
-- chore: [JENKINSFILE] sanitize sonarqube command Refs #305084 [valentinab25 - [`83bf08b`](https://github.com/eea/volto-eea-chatbot/commit/83bf08b7188548a50801b8e464a4d113f0cbbac4)]
 
 #### :house: Documentation changes
 
@@ -113,7 +137,6 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 #### :house: Internal changes
 
 - style: Automated code fix [eea-jenkins - [`8a8c3c4`](https://github.com/eea/volto-eea-chatbot/commit/8a8c3c4172a4f669661378cf1b5a3569d85609e6)]
-- chore: [JENKINSFILE] add package version in sonarqube [valentinab25 - [`535d986`](https://github.com/eea/volto-eea-chatbot/commit/535d986b7adc77743a668bc4ac63f835eef58df3)]
 
 #### :hammer_and_wrench: Others
 
@@ -124,7 +147,6 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 #### :house: Internal changes
 
 - style: Automated code fix [eea-jenkins - [`fdcd884`](https://github.com/eea/volto-eea-chatbot/commit/fdcd8848fd4c3f990ca5ea021f407487aebd6010)]
-- chore: [JENKINSFILE] use sonarqube branches [EEA Jenkins - [`3d428d7`](https://github.com/eea/volto-eea-chatbot/commit/3d428d72f32c3d05452b0961c76f5db1c416e05c)]
 
 #### :hammer_and_wrench: Others
 

package/README.md

@@ -40,7 +40,7 @@ The **Volto Chatbot** block allows the integration of an AI-powered chatbot into
 | `enableShowTotalFailMessage` | Show total failure message.                                                     | Boolean  | `false`                          |
 | `showAssistantTitle`         | Display or hide the assistant's title in the chat interface.                    | Boolean  | `true`                           |
 | `showAssistantDescription`   | Display or hide the assistant's description in the chat interface.              | Boolean  | `true`                           |
-| `qualityCheck`               | Show Halloumi-based automated quality check.                                    | Dropdown | `Disabled`                       |
+| `qualityCheck`               | Show automated fact-checking of AI answers against source documents.            | Dropdown | `Disabled`                       |
 | `onDemandInputToggle`        | Sets the default state of the fact-check AI toggle.                             | Boolean  | `true`                           |
 | `showTools`                  | Show or hide tools in the chat interface.                                       | Array    | `["internal_search_tool_start"]` |
 | `scrollToInput`              | Automatically scroll the page to focus on the chat input when interacting.      | Boolean  | `false`                          |
@@ -116,6 +116,44 @@ Go to http://localhost:3000
 
    For a legacy Volto 17 project, install the package with `yarn` and restart the frontend as usual.
 
+## Quality Checks (Fact-Checking)
+
+When `qualityCheck` is enabled, the chatbot sends AI answers and their source
+documents to a fact-checking backend that extracts claims, verifies each
+against the sources, and returns per-claim verdicts with evidence.
+
+### Backend dependency
+
+The fact-checking feature requires the
+[eea/rag-facts-check](https://github.com/eea/rag-facts-check) service running
+and reachable at the URL configured in `RAG_FACT_CHECKER_URL`.
+
+**Without this backend, quality checks will fail with a connection error.**
+
+The backend exposes a halloumi-compatible endpoint (`POST /halloumi/generate`)
+so the frontend can call it without code changes. It also provides a native
+`POST /check` endpoint with a richer response schema.
+
+### Deploying the backend
+
+```bash
+# Clone and build
+ git clone https://github.com/eea/rag-facts-check.git
+cd rag-facts-check
+docker build -t rag-fact-check .
+
+# Run (requires an LLM endpoint)
+# Set LLM_API_KEY to your actual API key before running
+docker run -p 8000:8000 \
+  -e LLM_API_BASE=http://your-llm:4002/v1 \
+  -e LLM_API_KEY \
+  -e LLM_MODEL=gemma \
+  rag-fact-check
+```
+
+See the [backend README](https://github.com/eea/rag-facts-check#rag-facts-check)
+for full configuration options.
+
 ## Environment Configuration
 
 To properly configure the middleware and authenticate with the Onyx service, ensure that the following environment variables are set:
@@ -131,20 +169,13 @@ This document lists the environment variables used in the Volto Chatbot project.
 - `JEST_USE_SETUP`
   Used in Jest configuration. When set to 'ON', it enables a specific Jest setup.
 
-- `LLMGW_URL`
-  The URL for the LLM Gateway service.
-
-- `LLMGW_TOKEN`
-  The token for authenticating with the LLM Gateway service.
+- `RAG_FACT_CHECKER_URL`
+  The base URL for the [rag-facts-check](https://github.com/eea/rag-facts-check)
+  fact-checking backend. Required when `qualityCheck` is enabled.
+  Default: `http://localhost:8000`.
 
 ### Development-specific environment variables
 
-- `MOCK_HALLOUMI_FILE_PATH`
-  When set, this specifies the absolute path to the JSON file containing the mocked Halloumi response. Setting this variable enables mocking of Halloumi API calls.
-
-- `DUMP_HALLOUMI_FILE_PATH`
-  When set, the Halloumi response will be dumped to the specified absolute file path for debugging or to create new mock files.
-
 - `MOCK_LLM_FILE_PATH`
   When set, this specifies the absolute path to the JSONL file containing the mocked Onyx stream response. Setting this variable enables mocking of Onyx LLM calls.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eeacms/volto-eea-chatbot",
-  "version": "3.0.1",
+  "version": "4.0.0",
   "description": "@eeacms/volto-eea-chatbot: Volto add-on",
   "main": "src/index.js",
   "author": "European Environment Agency: IDM2 A-Team",

package/src/ChatBlock/chat/AIMessage.tsx

@@ -81,31 +81,21 @@ function addQualityMarkersPlugin() {
   };
 }
 
-export function addHalloumiContext(doc: any, text: string) {
-  // TODO: CLEAN UP
-  // const updatedDate = doc.updated_at
-  //   ? new Date(doc.updated_at).toLocaleString('en-GB', {
-  //       year: 'numeric',
-  //       month: 'long',
-  //       day: '2-digit',
-  //       hour: '2-digit',
-  //       minute: '2-digit',
-  //     })
-  //   : '';
-
-  // const docIndex = doc.index ? `DOCUMENT ${doc.index}: ` : '';
-  // const sources: any = { web: 'Website', file: 'File' };
-
-  // const sourceType = doc.source_type
-  //   ? sources[doc.source_type] || capitalize(doc.source_type)
-  //   : '';
-
-  // const header = `${docIndex}${doc.semantic_identifier}${
-  //   sourceType ? `\nSource: ${sourceType}` : ''
-  // }${updatedDate ? `\nUpdated: ${updatedDate}` : ''}`;
-
-  // return `${header}\n${text}`;
-  return text.replace(/\u00A0/g, ' ');
+/**
+ * Build a structured source object for the fact-checker backend.
+ *
+ * The backend accepts {text, title, source_type} dicts so it can
+ * include document titles in LLM prompts without polluting the raw
+ * text (which would break span-offset matching).
+ */
+export function buildHalloumiSource(doc: any, text: string) {
+  const cleanedText = text.replace(/\u00A0/g, ' ');
+  return {
+    text: cleanedText,
+    title: doc.semantic_identifier || null,
+    source_type: doc.source_type || null,
+    link: doc.link || null,
+  };
 }
 
 function mapToolDocumentsToText(message: any) {
@@ -135,24 +125,36 @@ function getContextSources(
   const documentIdToText = mapToolDocumentsToText(message);
 
   return qualityCheckContext === 'citations'
-    ? sources.map((doc: any) => ({
-        ...doc,
-        id: doc.document_id,
-        text: documentIdToText[doc.document_id] || '',
-        halloumiContext: addHalloumiContext(
-          doc,
-          documentIdToText[doc.document_id] || '',
-        ),
-      }))
+    ? sources.map((doc: any) => {
+        // Prefer content from tool packets (may have enriched text),
+        // but fall back to doc.content from final_documents.
+        // Without this fallback, sources sent to the fact-checker are
+        // empty strings when tool packets don't carry full content.
+        const text = documentIdToText[doc.document_id] || doc.content || '';
+        const cleanedText = text.replace(/\u00A0/g, ' ');
+        return {
+          ...doc,
+          id: doc.document_id,
+          text,
+          // Keep for ClaimSegments span highlighting (plain text)
+          halloumiContext: cleanedText,
+          // Structured source for the fact-checker backend
+          halloumiSource: buildHalloumiSource(doc, text),
+        };
+      })
     : (message.toolCalls || []).reduce(
         (acc: any, cur: any) => [
           ...acc,
-          ...(cur.tool_result || []).map((doc: any) => ({
-            ...doc,
-            id: doc.document_id,
-            text: doc.content,
-            halloumiContext: addHalloumiContext(doc, doc.content),
-          })),
+          ...(cur.tool_result || []).map((doc: any) => {
+            const cleanedText = (doc.content || '').replace(/\u00A0/g, ' ');
+            return {
+              ...doc,
+              id: doc.document_id,
+              text: doc.content,
+              halloumiContext: cleanedText,
+              halloumiSource: buildHalloumiSource(doc, doc.content || ''),
+            };
+          }),
         ], // TODO: make sure we don't add multiple times the same doc
         // TODO: this doesn't have the index for source
         [],
@@ -281,7 +283,24 @@ export function AIMessage({
     qualityCheckContext,
   );
 
-  const stableContextSources = useDeepCompareMemoize(contextSources);
+  // Deduplicate sources by text content. The streaming backend often sends
+  // the same document in multiple packets (MESSAGE_START, SEARCH_TOOL_DELTA,
+  // etc.), resulting in 40+ copies of the same text. Dedup keeps the first
+  // occurrence, preserving order. Critical: texts must remain identical
+  // between frontend (span highlighting) and backend (evidence spans).
+  const dedupedSources = useMemo(() => {
+    const seen = new Set<string>();
+    return contextSources.filter((src: any) => {
+      const key = src.halloumiContext || src.halloumiSource?.text || '';
+      if (key && !seen.has(key)) {
+        seen.add(key);
+        return true;
+      }
+      return false;
+    });
+  }, [contextSources]);
+
+  const stableContextSources = useDeepCompareMemoize(dedupedSources);
 
   const doQualityControl =
     messageDisplayed &&

package/src/ChatBlock/components/markdown/ClaimModal.jsx

@@ -1,6 +1,6 @@
 import { Modal, ModalContent, ModalHeader } from 'semantic-ui-react';
 import cx from 'classnames';
-import { convertToPercentage } from '@eeacms/volto-eea-chatbot/ChatBlock/utils';
+import { scoreToLabel } from '@eeacms/volto-eea-chatbot/ChatBlock/utils';
 import SVGIcon from '@eeacms/volto-eea-chatbot/ChatBlock/components/Icon';
 import { getSupportedBgColor } from './colors';
 import { ClaimSegments } from './ClaimSegments';
@@ -70,9 +70,9 @@ export function ClaimModal({ claim, markers, text, citedSources }) {
           <div className="score-badge-section">
             <div className="score-badge">
               <span className="score-percentage">
-                {convertToPercentage(claim.score)}
+                {scoreToLabel(claim.score)}
               </span>
-              <span className="score-label">Citation Support</span>
+              <span className="score-label">Verification</span>
             </div>
             <div className="score-progress-bar">
               <div

package/src/ChatBlock/components/markdown/index.jsx

@@ -43,6 +43,7 @@ export function components(message, markers, citedSources) {
         const end = child.position.end.offset;
         const claims = markers.claims?.filter(
           (claim) =>
+            !claim.skipped &&
             claim.score !== null &&
             ((start >= claim.startOffset && end <= claim.endOffset) ||
               (start <= claim.endOffset && end >= claim.endOffset) ||

package/src/ChatBlock/hooks/useQualityMarkers.js

@@ -48,7 +48,16 @@ export function useQualityMarkers(
 
   React.useEffect(() => {
     async function handler() {
-      const textSources = sources.map(({ halloumiContext }) => halloumiContext);
+      // Prefer structured sources (with title metadata) over plain text.
+      // Fall back to halloumiContext strings for backward compatibility.
+      const firstSource = sources[0];
+      const hasStructuredSources =
+        firstSource && typeof firstSource.halloumiSource === 'object';
+
+      const halloumiSources = hasStructuredSources
+        ? sources.map(({ halloumiSource }) => halloumiSource)
+        : sources.map(({ halloumiContext }) => halloumiContext);
+
       if (sources.length === 0) {
         setHalloumiResponse(empty(message, FAILURE_RATIONALE));
         return;
@@ -59,7 +68,7 @@ export function useQualityMarkers(
       try {
         const feedback = await fetchHalloumi(
           message,
-          textSources,
+          halloumiSources,
           maxContextSegments,
         );
         const body = await feedback.json();

package/src/ChatBlock/services/messageProcessor.ts

@@ -211,8 +211,9 @@ export class MessageProcessor {
       this._documents = Array.from(this.documentMap.values());
     }
 
-    // If we have final_documents and no citations yet, create a fallback mapping
-    // This ensures the Sources tab shows up in v3 when citation_info is missing
+    // If we have final_documents and no citations yet, create a fallback mapping.
+    // This ensures the Sources tab shows up in v3 when citation_info is missing.
+    // The fallback is cleared in processCitations() when real citation_info arrives.
     if (
       packet.obj.type === PacketType.MESSAGE_START &&
       data.final_documents &&
@@ -240,9 +241,12 @@ export class MessageProcessor {
    */
   private processCitations(packet: Packet) {
     if (packet.obj.type === PacketType.CITATION_INFO) {
+      // First real citation_info arrived — discard the fallback mapping
+      // (which mapped ALL final_documents) and start fresh with actual citations.
+      this.clearFallbackCitations();
+
       const citationInfo = packet.obj as any;
       if (citationInfo.citation_number && citationInfo.document_id) {
-        this.clearFallbackCitations();
         this._citations.set(
           citationInfo.citation_number,
           citationInfo.document_id,

package/src/ChatBlock/utils/index.tsx

@@ -58,6 +58,16 @@ export const useCopyToClipboard = (text: string): [boolean, () => void] => {
   return [copied, copy];
 };
 
+/**
+ * Map a 0-1 claim score to a categorical label.
+ * Scores: 1.0 = supported, 0.4 = not_enough_info, 0.0 = contradicted.
+ */
+export function scoreToLabel(score: number): string {
+  if (score >= 0.8) return 'High';
+  if (score >= 0.1) return 'Low';
+  return 'Failed';
+}
+
 export function convertToPercentage(
   floatValue: number,
   digits: number = 2,

package/src/halloumi/README.md

@@ -1 +1,27 @@
-code based on Apache2 licensed https://github.com/oumi-ai/halloumi-demo/tree/d088e1f25e7785326a53bc120113e226ee2f54b7
+# Halloumi Fact-Checking Middleware
+
+This middleware proxies fact-checking requests to the **rag-fact-checker** service (https://github.com/eea/rag-fact-check), which replaces the original Halloumi LLM-based integration.
+
+## Architecture
+
+```
+Frontend (useQualityMarkers.js)
+  → POST /_ha/generate
+    → Express middleware (middleware.js)
+      → POST {RAG_FACT_CHECKER_URL}/halloumi/generate
+        → rag-fact-checker FastAPI service
+```
+
+The rag-fact-checker's `/halloumi/generate` endpoint is a drop-in replacement for the original Halloumi API — it accepts the same request format (`{answer, sources}`) and returns a compatible response (`{claims, segments}`), so all existing frontend components (`HalloumiFeedback`, `useQualityMarkers`, `ClaimSegments`) work without changes.
+
+## Configuration
+
+| Environment Variable   | Default                 | Description                              |
+| ---------------------- | ----------------------- | ---------------------------------------- |
+| `RAG_FACT_CHECKER_URL` | `http://localhost:8000` | Base URL of the rag-fact-checker service |
+
+## Original Halloumi
+
+The original implementation was based on Apache2 licensed https://github.com/oumi-ai/halloumi-demo/tree/d088e1f25e7785326a53bc120113e226ee2f54b7
+
+It used a custom LLM model (HallOumi-8B) accessed via an LLM gateway, with complex prompt preprocessing, chunking, logprob-based postprocessing, and claim filtering. This has been replaced by the rag-fact-checker service which uses a claim extraction + per-claim verification pipeline.

package/src/halloumi/middleware.js

@@ -1,41 +1,18 @@
-// import debug from 'debug';
-import { getVerifyClaimResponse } from './generative';
+import fetch from 'node-fetch';
+import debug from 'debug';
 import { isPathAllowed } from '../middleware';
 
-// const log = debug('halloumi');
+const log = debug('halloumi');
 
 const MSG_INVALID_CONFIGURATION =
-  'Invalid configuration: missing LLMGW_TOKEN or LLMGW_URL';
+  'Invalid configuration: missing RAG_FACT_CHECKER_URL';
 
 // Allowed paths for _ha (Halloumi) proxy.
 // When adding new endpoints, update this list.
-const ALLOWED_HALLOUMI_PATHS = [
-  { path: '/generate', methods: ['POST'] },
-  { path: '/classify', methods: ['POST'] },
-];
+const ALLOWED_HALLOUMI_PATHS = [{ path: '/generate', methods: ['POST'] }];
 
-const LLMGW_URL = process.env.LLMGW_URL;
-const LLMGW_TOKEN = process.env.LLMGW_TOKEN; //betterleaks:allow
-
-const generativeModel = {
-  name: 'Inhouse-LLM/HallOumi-8B',
-  apiUrl: `${LLMGW_URL}/chat/completions`,
-  plattScaling: {
-    a: -0.5764390035379638,
-    b: 0.16648741572432335,
-  },
-  isEmbeddingModel: false,
-};
-
-const classifyModel = {
-  name: 'Inhouse-LLM/HallOumi-8B-classifier',
-  apiUrl: `${LLMGW_URL}/v1/embeddings`,
-  isEmbeddingModel: true,
-  plattScaling: {
-    a: -0.9468640744087437,
-    b: -0.07379217647931409,
-  },
-};
+const RAG_FACT_CHECKER_URL =
+  process.env.RAG_FACT_CHECKER_URL || 'http://localhost:8000';
 
 export default async function middleware(req, res, next) {
   const path = req.url.replace('/_ha/', '/');
@@ -48,39 +25,52 @@ export default async function middleware(req, res, next) {
     return;
   }
 
-  if (!(LLMGW_TOKEN && LLMGW_URL)) {
+  if (!RAG_FACT_CHECKER_URL) {
     res.send({
       error: MSG_INVALID_CONFIGURATION,
     });
     return;
   }
 
-  const model = {
-    ...(path === '/classify' ? classifyModel : generativeModel),
-    apiKey: LLMGW_TOKEN, //betterleaks:allow
-  };
   const body = req.body;
-
-  // log('Halloumi body', body);
   const { sources, answer } = body;
 
+  log('Halloumi request body', {
+    answer,
+    sources: sources?.length,
+  });
+
   res.set('Content-Type', 'application/json');
 
   try {
-    const resp = await getVerifyClaimResponse(
-      model,
-      // TODO: map with citation id
-      sources,
-      answer,
-      { ip: req.headers['x-forwarded-for'] || req.ip },
-    );
-    // log('Halloumi response', resp);
-    res.send(resp);
+    const response = await fetch(`${RAG_FACT_CHECKER_URL}/halloumi/generate`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+      },
+      body: JSON.stringify({
+        answer,
+        sources,
+        max_context_segments: 0,
+      }),
+    });
+
+    const result = await response.json();
+
+    if (!response.ok) {
+      log('Halloumi error response', result);
+      res.status(response.status).send({
+        error: `Fact-checker error: ${result.detail || result.error || 'unknown error'}`,
+      });
+      return;
+    }
+
+    log('Halloumi response', result);
+    res.send(result);
   } catch (error) {
-    res.status(500).send({
-      error: `Halloumi error: ${error}`,
-      traceback: error.stack || null,
+    log('Halloumi fetch error', error);
+    res.status(502).send({
+      error: `Fact-checker unavailable: ${error.message || 'connection failed'}`,
     });
-    // throw error;
   }
 }