unique-search-proxy 2026.44.0.dev13__tar.gz → 2026.44.0.dev15__tar.gz

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: unique-search-proxy
-Version: 2026.44.0.dev13
+Version: 2026.44.0.dev15
 Summary: Web Search Proxy implementation
 Author: ThePhilAz
 Author-email: ThePhilAz <rami.azouz@philico.com>
@@ -21,7 +21,7 @@ Requires-Dist: google-genai>=1.73.0,<2
 Requires-Dist: google-auth>=2.43.0,<3
 Requires-Dist: jinja2>=3.1.6,<4
 Requires-Dist: unique-toolkit[monitoring]>=2026.44.0.dev10,<2026.44.0rc0
-Requires-Dist: unique-search-proxy-core>=2026.44.0.dev9,<2026.44.0rc0
+Requires-Dist: unique-search-proxy-core>=2026.44.0.dev11,<2026.44.0rc0
 Requires-Dist: openai>=1.105.0,<3
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -116,7 +116,7 @@ Built-in providers register in `register_builtin_providers()` during `create_app
 | `GET /ready` | Readiness (httpx pool + registered providers) |
 | `GET /v1/configuration/providers` | Registered search engine, agent engine, and crawler ids |
 | `POST /v1/search` | Standard search (flat body: `engine`, `query`, provider params, `timeout`) |
-| `POST /v1/agent-search` | Grounded agent search — opaque `answer` + `raw` |
+| `POST /v1/agent-search` | Grounded agent search — opaque `answer` + provider `citations` + `raw` |
 | `POST /v1/agent-search/stream` | Same, streamed as SSE (`delta` + `done`) |
 | `POST /v1/crawl` | Crawl URLs (flat body: `crawler`, `urls`, `timeout`, …) |
 | `GET /metrics` | Prometheus (when enabled) |
@@ -124,6 +124,25 @@ Built-in providers register in `register_builtin_providers()` during `create_app
 
 OpenAPI spec: `openapi.json` (input for [SDK codegen](../unique_search_proxy_sdk/README.md)).
 
+### 4.1 Request context and usage
+
+`/v1/*` routes read these headers into a request context that prefixes every log line:
+
+| Header | Required | Use |
+|--------|----------|-----|
+| `x-unique-company-id`, `x-unique-user-id`, `x-unique-chat-id` | Yes (`REQUIRE_CONTEXT_HEADERS`, default `true`) | Tenant identity |
+| `x-unique-entry-point` | No, defaults to `unknown` | Where the request entered the platform (`chat_tool`, `public_api`, `conduct`, ...) |
+| `x-unique-message-id` | No | Reserved for analytics; recorded in the usage line only |
+| `x-service-id` | No, defaults to `unknown` | Calling service; logged as `caller=`, never enforced |
+
+A request whose company or user id is not numeric (`local`, blank, junk) is unattributed: it logs a warning and increments `unique_search_proxy_unattributed_requests_total{entry_point}`. Set `REJECT_UNATTRIBUTED_REQUESTS=true` to return 400 instead. `/v1/configuration/*` skips this check.
+
+`/v1/search`, `/v1/agent-search` (sync and stream) and `/v1/crawl` write one JSON usage line per request (`event="web_search_usage"`) and increment `unique_search_proxy_requests_total{company_id, entry_point, endpoint, provider, status}`. `company_id` is `unattributed` unless both ids are numeric. The line holds no query text or URLs.
+
+`units` is 1 per search or agent search. For a crawl it is the number of URLs that passed URL safety, so 0 when nothing reached the provider. Failed and timed-out requests keep their units, so filter on `status` when counting billable work. Search records also carry `fetch_size`, the number of results requested: Google and Brave page through results (10 and 20 per call), so one search unit can be several provider calls.
+
+The usage line goes through its own logger (`unique_search_proxy_core.usage`) with a bare message format, fixed at INFO, so Loki `| json` can parse it regardless of `LOG_LEVEL`. Nothing verifies these headers: any in-cluster caller can set them.
+
 ---
 
 ## 5. Providers
@@ -191,7 +210,7 @@ Each result requires `url`, `title`, and `snippet`; `content` is optional.
 
 ### 5.3 Agent search (`POST /v1/agent-search`)
 
-Thin egress — proxy returns opaque agent text; callers own parsing and citation extraction:
+Thin egress — proxy returns opaque agent text and the citations the provider attached to it (Bing: Foundry `url_citation` annotations, with offsets into the answer); callers own parsing:
 
 ```json
 {
@@ -254,6 +273,7 @@ Settings use pydantic-settings with per-provider env vars. Copy `.env.example` t
 | Bing agent | `BING_AGENT_` | `BING_AGENT_ENDPOINT`, `BING_AGENT_BING_RESOURCE_CONNECTION_STRING`, optional `BING_AGENT_CLEANUP_ON_START`, `BING_AGENT_DEFAULT_MARKET` |
 | VertexAI agent | `VERTEXAI_AGENT_` | `VERTEXAI_AGENT_SERVICE_ACCOUNT_CREDENTIALS` (optional) |
 | HTTP client | `HTTP_CLIENT_` | `HTTP_CLIENT_PROXY_HOST`, `HTTP_CLIENT_POOL_TIMEOUT_SECONDS` |
+| Request context | (none) | `REQUIRE_CONTEXT_HEADERS`, `REJECT_UNATTRIBUTED_REQUESTS` |
 | Prometheus | `PROMETHEUS_` | `PROMETHEUS_ENABLED` |
 | URL safety | `URL_SAFETY_` | `URL_SAFETY_ENABLED` |
 | Container | (shell) | `HOST`, `PORT`, `WORKERS`, `LOG_LEVEL` |

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/README.md

@@ -88,7 +88,7 @@ Built-in providers register in `register_builtin_providers()` during `create_app
 | `GET /ready` | Readiness (httpx pool + registered providers) |
 | `GET /v1/configuration/providers` | Registered search engine, agent engine, and crawler ids |
 | `POST /v1/search` | Standard search (flat body: `engine`, `query`, provider params, `timeout`) |
-| `POST /v1/agent-search` | Grounded agent search — opaque `answer` + `raw` |
+| `POST /v1/agent-search` | Grounded agent search — opaque `answer` + provider `citations` + `raw` |
 | `POST /v1/agent-search/stream` | Same, streamed as SSE (`delta` + `done`) |
 | `POST /v1/crawl` | Crawl URLs (flat body: `crawler`, `urls`, `timeout`, …) |
 | `GET /metrics` | Prometheus (when enabled) |
@@ -96,6 +96,25 @@ Built-in providers register in `register_builtin_providers()` during `create_app
 
 OpenAPI spec: `openapi.json` (input for [SDK codegen](../unique_search_proxy_sdk/README.md)).
 
+### 4.1 Request context and usage
+
+`/v1/*` routes read these headers into a request context that prefixes every log line:
+
+| Header | Required | Use |
+|--------|----------|-----|
+| `x-unique-company-id`, `x-unique-user-id`, `x-unique-chat-id` | Yes (`REQUIRE_CONTEXT_HEADERS`, default `true`) | Tenant identity |
+| `x-unique-entry-point` | No, defaults to `unknown` | Where the request entered the platform (`chat_tool`, `public_api`, `conduct`, ...) |
+| `x-unique-message-id` | No | Reserved for analytics; recorded in the usage line only |
+| `x-service-id` | No, defaults to `unknown` | Calling service; logged as `caller=`, never enforced |
+
+A request whose company or user id is not numeric (`local`, blank, junk) is unattributed: it logs a warning and increments `unique_search_proxy_unattributed_requests_total{entry_point}`. Set `REJECT_UNATTRIBUTED_REQUESTS=true` to return 400 instead. `/v1/configuration/*` skips this check.
+
+`/v1/search`, `/v1/agent-search` (sync and stream) and `/v1/crawl` write one JSON usage line per request (`event="web_search_usage"`) and increment `unique_search_proxy_requests_total{company_id, entry_point, endpoint, provider, status}`. `company_id` is `unattributed` unless both ids are numeric. The line holds no query text or URLs.
+
+`units` is 1 per search or agent search. For a crawl it is the number of URLs that passed URL safety, so 0 when nothing reached the provider. Failed and timed-out requests keep their units, so filter on `status` when counting billable work. Search records also carry `fetch_size`, the number of results requested: Google and Brave page through results (10 and 20 per call), so one search unit can be several provider calls.
+
+The usage line goes through its own logger (`unique_search_proxy_core.usage`) with a bare message format, fixed at INFO, so Loki `| json` can parse it regardless of `LOG_LEVEL`. Nothing verifies these headers: any in-cluster caller can set them.
+
 ---
 
 ## 5. Providers
@@ -163,7 +182,7 @@ Each result requires `url`, `title`, and `snippet`; `content` is optional.
 
 ### 5.3 Agent search (`POST /v1/agent-search`)
 
-Thin egress — proxy returns opaque agent text; callers own parsing and citation extraction:
+Thin egress — proxy returns opaque agent text and the citations the provider attached to it (Bing: Foundry `url_citation` annotations, with offsets into the answer); callers own parsing:
 
 ```json
 {
@@ -226,6 +245,7 @@ Settings use pydantic-settings with per-provider env vars. Copy `.env.example` t
 | Bing agent | `BING_AGENT_` | `BING_AGENT_ENDPOINT`, `BING_AGENT_BING_RESOURCE_CONNECTION_STRING`, optional `BING_AGENT_CLEANUP_ON_START`, `BING_AGENT_DEFAULT_MARKET` |
 | VertexAI agent | `VERTEXAI_AGENT_` | `VERTEXAI_AGENT_SERVICE_ACCOUNT_CREDENTIALS` (optional) |
 | HTTP client | `HTTP_CLIENT_` | `HTTP_CLIENT_PROXY_HOST`, `HTTP_CLIENT_POOL_TIMEOUT_SECONDS` |
+| Request context | (none) | `REQUIRE_CONTEXT_HEADERS`, `REJECT_UNATTRIBUTED_REQUESTS` |
 | Prometheus | `PROMETHEUS_` | `PROMETHEUS_ENABLED` |
 | URL safety | `URL_SAFETY_` | `URL_SAFETY_ENABLED` |
 | Container | (shell) | `HOST`, `PORT`, `WORKERS`, `LOG_LEVEL` |

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "unique-search-proxy"
-version = "2026.44.0.dev13"
+version = "2026.44.0.dev15"
 description = "Web Search Proxy implementation"
 authors = [{ name = "ThePhilAz", email = "rami.azouz@philico.com" }]
 readme = "README.md"
@@ -23,7 +23,7 @@ dependencies = [
     "google-auth>=2.43.0,<3",
     "jinja2>=3.1.6,<4",
     "unique-toolkit[monitoring]>=2026.44.0.dev10,<2026.44.0rc0",
-    "unique-search-proxy-core>=2026.44.0.dev9,<2026.44.0rc0",
+    "unique-search-proxy-core>=2026.44.0.dev11,<2026.44.0rc0",
     "openai>=1.105.0,<3",
 ]
 

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/unique_search_proxy_client/web/api/v1/agent_search.py

@@ -32,6 +32,7 @@ from unique_search_proxy_client.web.monitoring.metrics import (
     record_agent_search_error,
     record_agent_search_success,
 )
+from unique_search_proxy_client.web.usage import record_request_usage
 
 router = APIRouter(tags=["agent-search"])
 _LOGGER = logging.getLogger(__name__)
@@ -79,6 +80,7 @@ async def agent_search(
     engine_id = engine.value if hasattr(engine, "value") else str(engine)
     timeout = body.timeout
     started = time.perf_counter()
+    succeeded = False
     _LOGGER.info(
         "agent-search start mode=sync engine=%s timeout=%ss",
         engine_id,
@@ -95,6 +97,7 @@ async def agent_search(
         )
         async with asyncio.timeout(timeout):
             result = await engine_service.search(body)
+        succeeded = True
     except TimeoutError as exc:
         record_agent_search_error(
             engine_id,
@@ -138,6 +141,14 @@ async def agent_search(
             (time.perf_counter() - started) * 1000,
         )
         raise
+    finally:
+        record_request_usage(
+            "agent_search",
+            engine_id,
+            units=1,
+            succeeded=succeeded,
+            started=started,
+        )
 
     duration = time.perf_counter() - started
     record_agent_search_success(engine_id, duration)
@@ -161,6 +172,7 @@ async def agent_search_stream(
     engine_id = engine.value if hasattr(engine, "value") else str(engine)
     timeout = body.timeout
     started = time.perf_counter()
+    succeeded = False
     _LOGGER.info(
         "agent-search start mode=stream engine=%s timeout=%ss",
         engine_id,
@@ -181,9 +193,23 @@ async def agent_search_stream(
             exc.code.value if hasattr(exc.code, "value") else str(exc.code),
             time.perf_counter() - started,
         )
+        record_request_usage(
+            "agent_search",
+            engine_id,
+            units=1,
+            succeeded=succeeded,
+            started=started,
+        )
         raise _agent_search_request_context(exc, engine_id=engine_id) from exc
     except Exception as exc:
         record_agent_search_error(engine_id, "INTERNAL_ERROR", 0.0)
+        record_request_usage(
+            "agent_search",
+            engine_id,
+            units=1,
+            succeeded=succeeded,
+            started=started,
+        )
         _LOGGER.exception(
             "agent-search setup error mode=stream engine=%s",
             engine_id,
@@ -194,11 +220,12 @@ async def agent_search_stream(
         ) from exc
 
     async def event_generator() -> AsyncIterator[str]:
-        nonlocal started
+        nonlocal started, succeeded
         try:
             async with asyncio.timeout(timeout):
                 async for event in engine_service.stream(body):
                     yield _format_sse_event(event)
+            succeeded = True
             record_agent_search_success(engine_id, time.perf_counter() - started)
             _LOGGER.info(
                 "agent-search success mode=stream engine=%s duration=%.0fms",
@@ -256,6 +283,14 @@ async def agent_search_stream(
                     engine_id=engine_id,
                 ),
             )
+        finally:
+            record_request_usage(
+                "agent_search",
+                engine_id,
+                units=1,
+                succeeded=succeeded,
+                started=started,
+            )
 
     return StreamingResponse(
         event_generator(),

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/unique_search_proxy_client/web/api/v1/context_headers.py

@@ -8,7 +8,9 @@ from fastapi import Header
 from unique_search_proxy_core.context import (
     CHAT_ID_HEADER,
     COMPANY_ID_HEADER,
+    ENTRY_POINT_HEADER,
     LOCAL_REQUEST_CONTEXT,
+    MESSAGE_ID_HEADER,
     USER_ID_HEADER,
 )
 
@@ -35,6 +37,20 @@ async def document_request_context_headers(
             description="Tenant chat or session identifier.",
         ),
     ] = LOCAL_REQUEST_CONTEXT.chat_id,
+    x_unique_entry_point: Annotated[
+        str,
+        Header(
+            alias=ENTRY_POINT_HEADER,
+            description="Service where the search entered the platform.",
+        ),
+    ] = LOCAL_REQUEST_CONTEXT.entry_point.value,
+    x_unique_message_id: Annotated[
+        str | None,
+        Header(
+            alias=MESSAGE_ID_HEADER,
+            description="Message identifier, reserved for analytics.",
+        ),
+    ] = None,
 ) -> None:
     """Declare context headers on /v1 routes for Swagger; enforcement is in middleware."""
     return None

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/unique_search_proxy_client/web/api/v1/crawl.py

@@ -40,6 +40,7 @@ from unique_search_proxy_client.web.monitoring.metrics import (
 from unique_search_proxy_client.web.settings.request_limits import (
     request_limits_settings,
 )
+from unique_search_proxy_client.web.usage import record_request_usage
 
 router = APIRouter(tags=["crawl"])
 _LOGGER = logging.getLogger(__name__)
@@ -96,6 +97,8 @@ async def crawl(
     crawler_id = body.crawler
     timeout = min(body.timeout, request_limits_settings.max_timeout_seconds)
     started = time.perf_counter()
+    succeeded = False
+    units = 0
     _LOGGER.info(
         "crawl start crawler=%s urls=%d timeout=%ss",
         crawler_id,
@@ -112,7 +115,9 @@ async def crawl(
                 body.urls,
                 redirect_http_client=client,
             )
+            units = len(gate.allowed_targets)
             if not gate.allowed_targets:
+                succeeded = True
                 duration = time.perf_counter() - started
                 record_crawl_success(
                     crawler_id,
@@ -155,6 +160,7 @@ async def crawl(
                 )
             else:
                 crawler_results = await crawler.crawl(crawl_body)
+        succeeded = True
     except TimeoutError as exc:
         record_crawl_error(
             crawler_id,
@@ -198,6 +204,14 @@ async def crawl(
             (time.perf_counter() - started) * 1000,
         )
         raise
+    finally:
+        record_request_usage(
+            "crawl",
+            crawler_id,
+            units=units,
+            succeeded=succeeded,
+            started=started,
+        )
 
     duration = time.perf_counter() - started
     record_crawl_success(crawler_id, len(body.urls), duration)

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/unique_search_proxy_client/web/api/v1/search.py

@@ -34,6 +34,7 @@ from unique_search_proxy_client.web.monitoring.metrics import (
 from unique_search_proxy_client.web.settings.request_limits import (
     request_limits_settings,
 )
+from unique_search_proxy_client.web.usage import record_request_usage
 
 router = APIRouter(tags=["search"])
 _LOGGER = logging.getLogger(__name__)
@@ -68,6 +69,7 @@ async def search(
     engine_id = engine.value if hasattr(engine, "value") else str(engine)
     timeout = min(body.timeout, request_limits_settings.max_timeout_seconds)
     started = time.perf_counter()
+    succeeded = False
     _LOGGER.info("search start engine=%s timeout=%ss", engine_id, timeout)
 
     try:
@@ -79,6 +81,7 @@ async def search(
         )
         async with asyncio.timeout(timeout):
             raw, curated = await engine.search(body)
+        succeeded = True
     except TimeoutError as exc:
         record_search_error(
             engine_id,
@@ -122,6 +125,16 @@ async def search(
             (time.perf_counter() - started) * 1000,
         )
         raise
+    finally:
+        record_request_usage(
+            "search",
+            engine_id,
+            # TODO(UN-24394): count provider pages fetched if billing needs exact call counts.
+            units=1,
+            fetch_size=body.fetch_size,
+            succeeded=succeeded,
+            started=started,
+        )
 
     duration = time.perf_counter() - started
     record_search_success(engine_id, duration)

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/unique_search_proxy_client/web/context.py

@@ -13,7 +13,13 @@ _REQUEST_CONTEXT: ContextVar[RequestContext] = ContextVar(
     default=LOCAL_REQUEST_CONTEXT,
 )
 
-_CONTEXT_FIELDS: Final[tuple[str, ...]] = ("company_id", "user_id", "chat_id")
+_CONTEXT_FIELDS: Final[tuple[str, ...]] = (
+    "company_id",
+    "user_id",
+    "chat_id",
+    "entry_point",
+    "caller",
+)
 
 
 def bind_request_context(context: RequestContext) -> Token[RequestContext]:

unique_search_proxy-2026.44.0.dev15/unique_search_proxy_client/web/core/agent_engines/bing/citation_instructions.j2

@@ -0,0 +1,18 @@
+{{ generation_instructions }}
+
+## Source-isolated evidence list
+Use Bing Grounding to select distinct useful sources.
+The complete response must consist only of lines in this form:
+- <a paragraph of facts supported only by one source> <that source's citation>
+
+Each line is a paragraph, not a sentence. Write several sentences per line that report every relevant fact, figure, date, name and quote from that source, with as much detail as the source offers, often 100 to 200 words. Lines can be long; do not shorten them.
+
+Hard requirements:
+1. Exactly one line per distinct source and exactly one distinct source per line.
+2. Every line begins with `- ` and contains exactly one citation marker as its final content.
+3. Never synthesize, compare, or combine facts across sources within a line.
+4. Put all useful facts from a source into that source's one line; never cite it again.
+5. No headings, preface, conclusion, line breaks inside a paragraph, nested bullets, JSON, or uncited claims.
+6. Never invent, reconstruct, type, or guess citation markers or source URLs.
+
+Before responding, silently verify that the number of lines equals the number of distinct cited sources.

unique_search_proxy-2026.44.0.dev15/unique_search_proxy_client/web/core/agent_engines/bing/instructions.py

@@ -0,0 +1,26 @@
+from __future__ import annotations
+
+from importlib.resources import files
+
+from jinja2 import StrictUndefined, Template
+
+# Describes the output format we want from the agent. Models do not always
+# follow it, so downstream parsing should tolerate deviations and keep what it
+# can rather than reject the whole answer.
+_CITATION_INSTRUCTIONS = Template(
+    files(__package__)
+    .joinpath("citation_instructions.j2")
+    .read_text(encoding="utf-8")
+    .strip(),
+    undefined=StrictUndefined,
+)
+
+
+def build_bing_citation_instructions(generation_instructions: str) -> str:
+    """Add citation-safe output rules without suppressing Foundry annotations."""
+    return _CITATION_INSTRUCTIONS.render(
+        generation_instructions=generation_instructions.rstrip()
+    )
+
+
+__all__ = ["build_bing_citation_instructions"]

{unique_search_proxy-2026.44.0.dev13 → unique_search_proxy-2026.44.0.dev15}/unique_search_proxy_client/web/core/agent_engines/bing/runner.py

@@ -2,7 +2,8 @@ from __future__ import annotations
 
 import logging
 import time
-from collections.abc import AsyncIterator
+from collections.abc import AsyncIterator, Sequence
+from typing import NamedTuple
 
 import openai
 from azure.ai.projects.aio import AIProjectClient
@@ -14,10 +15,8 @@ from azure.ai.projects.models import (
 )
 from httpx import AsyncClient
 from openai.types.responses import ResponseStreamEvent
+from openai.types.responses.response import Response
 from openai.types.responses.response_completed_event import ResponseCompletedEvent
-from openai.types.responses.response_output_item_done_event import (
-    ResponseOutputItemDoneEvent,
-)
 from openai.types.responses.response_output_message import ResponseOutputMessage
 from openai.types.responses.response_output_text import (
     AnnotationURLCitation,
@@ -28,6 +27,7 @@ from unique_search_proxy_core.agent_engines.bing.grounding import (
     BingGroundingConfiguration,
     bing_agent_name,
 )
+from unique_search_proxy_core.schema import AnswerSpanCitation
 
 from unique_search_proxy_client.web.core.agent_engines.bing.client import (
     get_openai_client,
@@ -38,6 +38,7 @@ from unique_search_proxy_client.web.settings.secret_str import NOT_PROVIDED, rea
 _LOGGER = logging.getLogger(__name__)
 
 __all__ = [
+    "BingStreamChunk",
     "create_bing_agent",
     "get_bing_grounding_tool",
     "resolve_bing_agent_name",
@@ -45,6 +46,19 @@ __all__ = [
 ]
 
 
+class BingStreamChunk(NamedTuple):
+    """One step of a Bing Responses stream.
+
+    ``answer`` and ``citations`` are set only for the completed response, and
+    the citation offsets point into ``answer``.
+    """
+
+    delta: str
+    raw_event: dict
+    answer: str | None = None
+    citations: Sequence[AnswerSpanCitation] = ()
+
+
 def resolve_bing_agent_name(
     *,
     model: str,
@@ -147,8 +161,8 @@ async def stream_bing_grounding_agent(
     instructions: str,
     grounding: BingGroundingConfiguration,
     timeout: float,
-) -> AsyncIterator[tuple[str, dict]]:
-    """Stream Bing-grounded Responses events as ``(delta_text, raw_event)`` pairs.
+) -> AsyncIterator[BingStreamChunk]:
+    """Stream Bing-grounded Responses events as :class:`BingStreamChunk` items.
 
     Optimistically calls Responses with a hash-derived agent name. If the agent
     is missing, creates it once and retries.
@@ -181,13 +195,23 @@ async def stream_bing_grounding_agent(
             resolved_name,
             exc,
         )
-        await create_bing_agent(
-            project_client,
-            agent_name=resolved_name,
-            model=model,
-            instructions=instructions,
-            grounding=grounding,
-        )
+        try:
+            await create_bing_agent(
+                project_client,
+                agent_name=resolved_name,
+                model=model,
+                instructions=instructions,
+                grounding=grounding,
+            )
+        except UnicodeDecodeError:
+            # Foundry can persist the agent but return a compressed create response
+            # that azure-core fails to decode. The Responses retry below verifies
+            # whether creation actually succeeded.
+            _LOGGER.warning(
+                "Could not decode create response for Bing agent %s; "
+                "verifying creation through Responses",
+                resolved_name,
+            )
         stream = await _create_responses_stream(
             openai_client,
             agent_name=resolved_name,
@@ -201,21 +225,13 @@ async def stream_bing_grounding_agent(
             if isinstance(event, ResponseTextDeltaEvent):
                 if event.delta:
                     emitted_text = True
-                    yield event.delta, event.model_dump(mode="json", warnings="none")
-            elif isinstance(event, ResponseOutputItemDoneEvent):
-                citations = _extract_url_citations(event)
-                if citations:
-                    yield (
-                        "",
-                        {
-                            "type": event.type,
-                            "citations": citations,
-                            "event": event.model_dump(mode="json", warnings="none"),
-                        },
+                    yield BingStreamChunk(
+                        event.delta, event.model_dump(mode="json", warnings="none")
                     )
             elif isinstance(event, ResponseCompletedEvent):
                 response = event.response
                 output_text = response.output_text
+                citations = _extract_response_url_citations(response)
                 raw_completed = {
                     "type": event.type,
                     "response": response.model_dump(mode="json", warnings="none"),
@@ -223,9 +239,16 @@ async def stream_bing_grounding_agent(
                 # Foundry sometimes delivers the full answer only on completion.
                 if output_text and not emitted_text:
                     emitted_text = True
-                    yield output_text, raw_completed
+                    yield BingStreamChunk(
+                        output_text,
+                        raw_completed,
+                        answer=output_text,
+                        citations=citations,
+                    )
                 else:
-                    yield "", raw_completed
+                    yield BingStreamChunk(
+                        "", raw_completed, answer=output_text, citations=citations
+                    )
 
 
 async def _create_responses_stream(
@@ -258,15 +281,76 @@ async def _create_responses_stream(
     return stream
 
 
-def _extract_url_citations(event: ResponseOutputItemDoneEvent) -> list[str]:
-    item = event.item
-    if not isinstance(item, ResponseOutputMessage) or not item.content:
-        return []
-    text_content = item.content[-1]
-    if not isinstance(text_content, ResponseOutputText):
+def _extract_message_url_citations(
+    item: ResponseOutputMessage,
+    *,
+    start_index_offset: int = 0,
+) -> list[AnswerSpanCitation]:
+    """Turn the URL annotations on one output message into citations.
+
+    Args:
+        item: A completed assistant message. Its text can be split across
+            several ``output_text`` parts, and each annotation's offsets are
+            relative to the part it is attached to.
+        start_index_offset: Length of the answer text before this message, so
+            the returned offsets point into the full answer.
+
+    Returns:
+        One citation per URL annotation, carrying the marker text the
+        annotation spans and its offsets into the full answer. Annotations
+        without a URL or with an empty span are skipped.
+    """
+    if not item.content:
         return []
-    return [
-        annotation.url
-        for annotation in text_content.annotations
-        if isinstance(annotation, AnnotationURLCitation) and annotation.url
-    ]
+
+    citations: list[AnswerSpanCitation] = []
+    text_offset = start_index_offset
+    for text_content in item.content:
+        if not isinstance(text_content, ResponseOutputText):
+            continue
+        for annotation in text_content.annotations or []:
+            if not isinstance(annotation, AnnotationURLCitation):
+                continue
+            marker = text_content.text[annotation.start_index : annotation.end_index]
+            if annotation.url and marker:
+                citations.append(
+                    AnswerSpanCitation(
+                        url=annotation.url,
+                        title=annotation.title,
+                        marker=marker,
+                        start_index=text_offset + annotation.start_index,
+                        end_index=text_offset + annotation.end_index,
+                        output_item_id=item.id,
+                    )
+                )
+        text_offset += len(text_content.text)
+    return citations
+
+
+def _extract_response_url_citations(
+    response: Response,
+) -> list[AnswerSpanCitation]:
+    """Extract citations with offsets into ``response.output_text``."""
+    citations: list[AnswerSpanCitation] = []
+    text_offset = 0
+    for item in response.output or []:
+        if not isinstance(item, ResponseOutputMessage):
+            continue
+        citations.extend(
+            _extract_message_url_citations(
+                item,
+                start_index_offset=text_offset,
+            )
+        )
+        text_offset += sum(
+            len(content.text)
+            for content in item.content or []
+            if isinstance(content, ResponseOutputText)
+        )
+
+    _LOGGER.info(
+        "Foundry Bing citation annotations response_id=%s citation_count=%d",
+        response.id,
+        len(citations),
+    )
+    return citations