nlp2sql 0.2.0rc16__tar.gz → 0.2.0rc17__tar.gz

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/.github/workflows/ci.yml

@@ -90,8 +90,9 @@ jobs:
         NLP2SQL_TEST_POSTGRES_DB: testdb
       run: pytest tests/test_pgvector_example_repository.py -m integration -q
 
-    # README Quick Start per provider and the false-invalid gold set, with the
-    # OpenAI SDK pointed at tests/fakes/fake_openai.py (no keys, no cost).
+    # README Quick Start per provider, the false-invalid gold set and the cost of
+    # one question (LLM calls, tokens, database connections), with the OpenAI SDK
+    # pointed at tests/fakes/fake_openai.py (no keys, no cost).
     - name: First-run regression tests
       env:
         PYTHONPATH: src
@@ -101,7 +102,7 @@ jobs:
         NLP2SQL_TEST_POSTGRES_USER: testuser
         NLP2SQL_TEST_POSTGRES_PASS: testpass
         NLP2SQL_TEST_POSTGRES_DB: testdb
-      run: pytest tests/test_first_run.py tests/test_false_invalid_regression.py -m integration -q
+      run: pytest tests/test_first_run.py tests/test_false_invalid_regression.py tests/test_cost_per_question.py -m integration -q
 
   # Installs the built wheel the way a user does: fresh environment, no lock
   # file, latest dependency versions. Catches undeclared dependencies (rc13-rc15

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/CHANGELOG.md

@@ -22,6 +22,59 @@ pre-1.0.
 
 ## [Unreleased]
 
+## [0.2.0rc17] - 2026-10-20
+
+### Changed
+- **User-visible: the LLM no longer reviews its own SQL by default.** Every `ask()`
+  made a second LLM call that sent the whole schema context back to the model; it
+  doubled the latency of each question and its tokens were not reported. Column,
+  dialect, semantic and read-only checks still run, and execution modes still
+  validate against the database. Turn the review back on with
+  `connect(..., llm_validation=True)`, `NLP2SQL_LLM_VALIDATION=true` or
+  `nlp2sql query --llm-validation`. `NLP2SQL.validate(sql)` still asks the LLM.
+- `QueryResult.tokens_used` now counts every LLM call made for the question
+  (retries, repairs and LLM validation); it used to report only the last
+  generation call. `metadata["token_usage"]` has the breakdown.
+- `refresh_schema()` (PostgreSQL and Redshift) now drops the cached table list so
+  the next `get_tables()` reads the database again. PostgreSQL no longer runs
+  `ANALYZE` on the whole database on the first `connect()` and every 24 hours.
+- Provider calls are retried only on rate limits, timeouts, connection errors and
+  provider-side errors. Authentication and request errors fail on the first
+  attempt, instead of after about 8 seconds and 3 requests.
+- An answer from the model that is not the expected JSON (not JSON, not an object,
+  or without `sql`) is asked for again once, without waiting; it used to be retried
+  up to 3 times with 4-10 second waits. Both answers count in `tokens_used` and in
+  `metadata["token_usage"]["llm_calls"]`.
+- An answer cut off at `max_tokens` (OpenAI `finish_reason="length"`, Anthropic
+  `stop_reason="max_tokens"`, Gemini `MAX_TOKENS`) fails on the first attempt with a
+  `ProviderException` that says to raise `max_tokens`, instead of "Invalid JSON
+  response" after 3 attempts. Anthropic refusals and Gemini safety blocks raise
+  `ProviderRequestError`.
+
+### Added
+- `llm_validation` option on `connect()`, `create_query_service()` and
+  `create_and_initialize_service()`, `NLP2SQL_LLM_VALIDATION`, and
+  `nlp2sql query --llm-validation`.
+- `ProviderAuthenticationError`, `ProviderRequestError`, `ProviderRateLimitError`
+  and `ProviderUnavailableError`, subclasses of `ProviderException`, raised by the
+  OpenAI, Anthropic and Gemini adapters and available as the `__cause__` of the
+  `QueryGenerationException` raised by `ask()`.
+- `ProviderResponseFormatError` (subclass of `ProviderException`): the model answered
+  twice without the expected JSON. `details["tokens_used"]` has the tokens of both
+  answers.
+
+### Fixed
+- Each `ask()` opened 5 new database connections per table in the schema context
+  (41 for a multi-table question). The context is now built from the table list
+  loaded at `connect()`, with no extra connections.
+- `RedshiftRepository.refresh_schema()` was a no-op.
+- The Anthropic and Gemini adapters raised `TypeError` instead of
+  `TokenLimitException` when the context was too large, and dropped the SDK error
+  as the cause of generation failures.
+- The Anthropic adapter reported its own errors as "Failed to parse response: ...".
+- `docs/API.md` showed catching `TokenLimitException` and `ProviderException`
+  directly from `ask()`; they arrive as the cause of `QueryGenerationException`.
+
 ## [0.2.0rc16] - 2026-10-19
 
 ### Changed

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: nlp2sql
-Version: 0.2.0rc16
+Version: 0.2.0rc17
 Summary: Enterprise-ready Natural Language to SQL converter with multi-provider support. Built for production scale (1000+ tables) with Clean Architecture.
 Project-URL: Homepage, https://github.com/luiscarbonel1991/nlp2sql
 Project-URL: Bug Tracker, https://github.com/luiscarbonel1991/nlp2sql/issues

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/docs/API.md

@@ -57,6 +57,7 @@ nlp = await nlp2sql.connect(
 | `hooks`                   | `ExecutionHooks | None`                     | Execution, classification, and repair hooks                         |
 | `semantic_hooks`          | `SemanticHooks | None`                      | Semantic resolver, validator, and optional default semantic context |
 | `semantic_context`        | `SemanticContext | dict | None`             | Default semantic context applied to future `ask()` calls            |
+| `llm_validation`          | `bool | None`                               | Also ask the LLM to review each query (a second call). Default: `NLP2SQL_LLM_VALIDATION`, off |
 
 
 ### `hooks`
@@ -157,7 +158,7 @@ If you do not pass `execution_mode`, the library infers it from `validate` and `
 | `explanation`        | `str | None`        | Optional natural-language explanation          |
 | `provider`           | `str`               | Provider that generated the SQL                |
 | `database_type`      | `str`               | Postgres or Redshift                           |
-| `tokens_used`        | `int`               | Tokens consumed                                |
+| `tokens_used`        | `int`               | Tokens of every LLM call made for the question |
 | `generation_time_ms` | `float`             | End-to-end generation time                     |
 | `examples_used`      | `int`               | Count of selected examples                     |
 | `metadata`           | `dict[str, object]` | Runtime metadata for debugging and integration |
@@ -175,6 +176,7 @@ The metadata payload is intentionally rich. Common fields include:
 | `selected_examples`    | Few-shot examples selected for prompting      |
 | `repair_attempts`      | Repair loop details when repair is enabled    |
 | `execution_validation` | Execution-time validation outcome             |
+| `token_usage`          | `total`, `generation`, `validation` and `llm_calls` for the question |
 | `analysis`             | Query analysis hints                          |
 
 
@@ -272,29 +274,47 @@ print(result["validation"]["is_valid"])
 
 ## Error Handling
 
+`ask()` raises `QueryGenerationException` when SQL cannot be generated. The underlying error is
+its `__cause__`, typed so callers can react to each case:
+
 ```python
 from nlp2sql.exceptions import (
-    ProviderException,
+    ProviderAuthenticationError,
+    ProviderRateLimitError,
+    ProviderRequestError,
+    ProviderResponseFormatError,
+    ProviderUnavailableError,
     QueryGenerationException,
     SchemaException,
-    SecurityException,
     TokenLimitException,
 )
 
 try:
     result = await nlp.ask("Show revenue by month")
-except TokenLimitException:
-    print("The schema context is too large. Add schema filters.")
+except QueryGenerationException as error:
+    cause = error.__cause__
+    if isinstance(cause, ProviderAuthenticationError):
+        print("Invalid or unauthorized API key (HTTP 401/403).")
+    elif isinstance(cause, TokenLimitException):
+        print("The schema context is too large. Add schema filters.")
+    elif isinstance(cause, (ProviderRateLimitError, ProviderUnavailableError)):
+        print("Transient provider error; already retried three times.")
+    elif isinstance(cause, ProviderRequestError):
+        print("The provider rejected the request, e.g. an unknown model.")
+    elif isinstance(cause, ProviderResponseFormatError):
+        print("The model answered twice without the expected JSON.")
+    else:
+        print("The provider could not generate SQL.")
 except SchemaException:
     print("Database connection or schema discovery failed.")
-except QueryGenerationException:
-    print("The provider could not generate SQL.")
-except ProviderException:
-    print("Provider auth, network, or rate-limit error.")
-except SecurityException:
-    print("Generated SQL failed safety checks.")
 ```
 
+Only rate-limit and unavailable errors are retried with backoff; authentication and request
+errors fail on the first attempt. An answer that is not the expected JSON is asked for again
+once, without waiting, and `tokens_used` counts both answers. An answer cut off at `max_tokens`
+is not asked for again: it fails with a `ProviderException` that says to raise `max_tokens`.
+Queries that are not read-only come back with `is_valid=False` and are never executed.
+
 ## CLI Reference
 
 ### `nlp2sql query`

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/docs/CONFIGURATION.md

@@ -73,6 +73,7 @@ export NLP2SQL_EMBEDDING_PROVIDER=local
 
 | Variable | Default | Description |
 |----------|---------|-------------|
+| `NLP2SQL_LLM_VALIDATION` | `false` | Also ask the LLM to review each generated query (a second call per question) |
 | `NLP2SQL_LOG_LEVEL` | `INFO` | Logging verbosity |
 | `NLP2SQL_CACHE_ENABLED` | `true` | Enable or disable query-result caching |
 | `NLP2SQL_ENV` | unset | Optional environment label |
@@ -184,6 +185,12 @@ The runtime has three practical execution modes.
 | `generate_and_validate` | safe validation of generated SQL against a readonly execution target |
 | `generate_validate_repair` | production-like usage where repair is worth the extra latency |
 
+Every mode checks column names, dialect rules, semantic rules and that the query is read-only,
+without extra LLM calls. An additional LLM review of the generated SQL is off by default: it
+doubles the latency of each question, and executing the query against the database gives a
+stronger signal. Turn it on with `connect(..., llm_validation=True)`,
+`NLP2SQL_LLM_VALIDATION=true` or `nlp2sql query --llm-validation`.
+
 ### Python
 
 ```python

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/mcp_server/__init__.py

@@ -1,3 +1,3 @@
 """nlp2sql MCP Server - Expose nlp2sql as MCP tools."""
 
-__version__ = "0.2.0rc16"
+__version__ = "0.2.0rc17"

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/mcp_server/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "nlp2sql-mcp"
-version = "0.2.0rc16"
+version = "0.2.0rc17"
 description = "MCP server for nlp2sql - Natural Language to SQL conversion"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -21,4 +21,4 @@ build-backend = "hatchling.build"
 [tool.mcp]
 name = "nlp2sql"
 description = "Convert natural language queries to SQL using multiple AI providers"
-version = "0.2.0rc16"
+version = "0.2.0rc17"

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "nlp2sql"
-version = "0.2.0rc16"
+version = "0.2.0rc17"
 description = "Enterprise-ready Natural Language to SQL converter with multi-provider support. Built for production scale (1000+ tables) with Clean Architecture."
 readme = "README.md"
 license = {text = "MIT"}

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/src/nlp2sql/__init__.py

@@ -46,7 +46,7 @@ from .ports.semantic_validator import SemanticValidatorPort
 from .schema.example_store import ExampleStore
 from .services.query_service import QueryGenerationService
 
-__version__ = "0.2.0rc16"
+__version__ = "0.2.0rc17"
 __author__ = "Luis Carbonel"
 __email__ = "devhighlevel@gmail.com"
 
@@ -112,6 +112,11 @@ __all__ = [
     "NLP2SQLException",
     "SchemaException",
     "ProviderException",
+    "ProviderAuthenticationError",
+    "ProviderRequestError",
+    "ProviderRateLimitError",
+    "ProviderUnavailableError",
+    "ProviderResponseFormatError",
     "TokenLimitException",
     "QueryGenerationException",
     "OptimizationException",
@@ -219,6 +224,7 @@ def create_query_service(
     semantic_validator: SemanticValidatorPort | None = None,
     *,
     provider_config: Optional[ProviderConfig] = None,
+    llm_validation: Optional[bool] = None,
 ) -> QueryGenerationService:
     """
     Create a configured query service instance.
@@ -239,6 +245,8 @@ def create_query_service(
         provider_config: Optional ProviderConfig for full control over provider settings
             (model, temperature, max_tokens, extra_kwargs). Individual kwargs like api_key
             and model take precedence over values in provider_config.
+        llm_validation: Also ask the LLM to review each generated query (a second call per
+            question). If None, uses NLP2SQL_LLM_VALIDATION (default: off).
 
     Returns:
         Configured QueryGenerationService instance
@@ -352,6 +360,7 @@ def create_query_service(
         repair_policy=repair_policy,
         semantic_resolver=semantic_resolver,
         semantic_validator=semantic_validator,
+        llm_validation=llm_validation,
     )
 
 
@@ -373,6 +382,7 @@ async def create_and_initialize_service(
     semantic_validator: SemanticValidatorPort | None = None,
     *,
     provider_config: Optional[ProviderConfig] = None,
+    llm_validation: Optional[bool] = None,
 ) -> QueryGenerationService:
     """
     Create and initialize a query service with automatic schema loading.
@@ -393,6 +403,8 @@ async def create_and_initialize_service(
         schema_name: Database schema name (default: 'public')
         example_store: Optional pre-loaded few-shot examples for better SQL generation
         provider_config: Optional ProviderConfig for full control over provider settings
+        llm_validation: Also ask the LLM to review each generated query. If None, uses
+            NLP2SQL_LLM_VALIDATION (default: off).
 
     Returns:
         Initialized QueryGenerationService ready for queries
@@ -423,6 +435,7 @@ async def create_and_initialize_service(
         semantic_resolver,
         semantic_validator,
         provider_config=provider_config,
+        llm_validation=llm_validation,
     )
     await service.initialize(database_type)
     return service

nlp2sql-0.2.0rc17/src/nlp2sql/adapters/_provider_errors.py

@@ -0,0 +1,99 @@
+"""Translate provider SDK errors into nlp2sql exceptions, and decide what is retried.
+
+The AI adapters retry only transient failures. Before, every exception was retried
+three times with backoff, so an invalid API key took about 8 seconds to surface.
+Works without importing any SDK: OpenAI and Anthropic errors carry ``status_code``,
+google.api_core errors carry an integer ``code``.
+
+Output that is not the expected JSON is a different failure: the request worked, the
+model answered badly. ``call_with_format_retry`` asks again once, without waiting,
+and counts the tokens of both answers.
+"""
+
+from typing import Any, Awaitable, Callable, Dict, NamedTuple, Optional, TypeVar
+
+import structlog
+
+from ..exceptions import (
+    ProviderAuthenticationError,
+    ProviderException,
+    ProviderRateLimitError,
+    ProviderRequestError,
+    ProviderResponseFormatError,
+    ProviderUnavailableError,
+)
+
+logger = structlog.get_logger()
+
+TRANSIENT_PROVIDER_ERRORS = (ProviderRateLimitError, ProviderUnavailableError)
+FORMAT_RETRY_ATTEMPTS = 2
+
+ResponseT = TypeVar("ResponseT")
+
+
+def _http_status(exc: BaseException) -> Optional[int]:
+    for attribute in ("status_code", "code"):
+        value = getattr(exc, attribute, None)
+        if isinstance(value, int) and 100 <= value < 600:
+            return int(value)
+    return None
+
+
+def _is_transport_error(exc: BaseException) -> bool:
+    if isinstance(exc, (TimeoutError, ConnectionError)):
+        return True
+    name = type(exc).__name__
+    return "Timeout" in name or "Connection" in name
+
+
+def classify_provider_error(exc: BaseException, message: str) -> ProviderException:
+    """Return the nlp2sql exception for ``exc``; nlp2sql exceptions pass through unchanged."""
+    if isinstance(exc, ProviderException):
+        return exc
+
+    status = _http_status(exc)
+    if status in (401, 403):
+        return ProviderAuthenticationError(message, {"status_code": status})
+    if status == 429:
+        return ProviderRateLimitError(message, {"status_code": status})
+    if status is not None and (status >= 500 or status in (408, 409)):
+        return ProviderUnavailableError(message, {"status_code": status})
+    if status is not None and 400 <= status < 500:
+        return ProviderRequestError(message, {"status_code": status})
+    if _is_transport_error(exc):
+        return ProviderUnavailableError(message)
+    return ProviderException(message)
+
+
+class ParsedGeneration(NamedTuple):
+    response: Any
+    result: Dict[str, Any]
+    tokens_used: int
+    format_retries: int
+
+
+async def call_with_format_retry(
+    send: Callable[[], Awaitable[ResponseT]],
+    parse: Callable[[ResponseT], Dict[str, Any]],
+    tokens_of: Callable[[ResponseT], int],
+    *,
+    provider: str,
+    attempts: int = FORMAT_RETRY_ATTEMPTS,
+) -> ParsedGeneration:
+    """Send a generation request and parse it, asking again when the output is malformed.
+
+    Tokens are read before parsing, so a discarded answer still counts. Every other
+    error propagates on the first attempt; transient ones are retried by the caller.
+    """
+    tokens_used = 0
+    for attempt in range(1, attempts + 1):
+        response = await send()
+        tokens_used += tokens_of(response)
+        try:
+            return ParsedGeneration(response, parse(response), tokens_used, attempt - 1)
+        except ProviderResponseFormatError as error:
+            error.details["tokens_used"] = tokens_used
+            if attempt == attempts:
+                raise
+            logger.warning("Malformed model output, asking again", provider=provider, error=str(error))
+    raise ValueError("attempts must be at least 1")

{nlp2sql-0.2.0rc16 → nlp2sql-0.2.0rc17}/src/nlp2sql/adapters/anthropic_adapter.py

@@ -5,14 +5,15 @@ from typing import Any, Dict, Optional
 
 import anthropic
 import structlog
-from tenacity import retry, stop_after_attempt, wait_exponential
+from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential
 
 from ..config.settings import settings
 from ..core.provider_config import ProviderConfig
-from ..exceptions import ProviderException, TokenLimitException
+from ..exceptions import ProviderException, ProviderRequestError, ProviderResponseFormatError, TokenLimitException
 from ..ports.ai_provider import AIProviderPort, AIProviderType, QueryContext, QueryResponse
 from ..utils.helpers import first_not_none
 from ..utils.semantic_prompt import format_semantic_context_lines, format_sql_intent_plan_lines
+from ._provider_errors import TRANSIENT_PROVIDER_ERRORS, call_with_format_retry, classify_provider_error
 
 logger = structlog.get_logger()
 
@@ -119,7 +120,12 @@ class AnthropicAdapter(AIProviderPort):
         """Get maximum input context size for the model."""
         return self._max_context_size
 
-    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10), reraise=True)
+    @retry(
+        stop=stop_after_attempt(3),
+        wait=wait_exponential(multiplier=1, min=4, max=10),
+        retry=retry_if_exception_type(TRANSIENT_PROVIDER_ERRORS),
+        reraise=True,
+    )
     async def generate_query(self, context: QueryContext) -> QueryResponse:
         """Generate SQL query using Claude."""
         try:
@@ -133,22 +139,27 @@ class AnthropicAdapter(AIProviderPort):
             # Create messages
             messages = [{"role": "user", "content": prompt}]
 
-            # Call Claude API (no sampling parameters: see class docstring)
-            response = await self.client.messages.create(
-                model=self.model,
-                max_tokens=context.max_tokens,
-                system=system_prompt,
-                messages=messages,
+            # Call Claude API (no sampling parameters: see class docstring); malformed
+            # output is asked for again once
+            generation = await call_with_format_retry(
+                lambda: self.client.messages.create(
+                    model=self.model,
+                    max_tokens=context.max_tokens,
+                    system=system_prompt,
+                    messages=messages,
+                ),
+                self._parse_response,
+                self._total_tokens,
+                provider=self.provider_type.value,
             )
-
-            # Parse response
-            result = self._parse_response(response)
+            response, result = generation.response, generation.result
 
             # Create QueryResponse
             metadata = {
                 "model": self.model,
                 "input_tokens": response.usage.input_tokens,
                 "output_tokens": response.usage.output_tokens,
+                "format_retries": generation.format_retries,
             }
 
             # Include raw response if available
@@ -159,14 +170,17 @@ class AnthropicAdapter(AIProviderPort):
                 sql=result["sql"],
                 explanation=result.get("explanation", ""),
                 confidence=result.get("confidence", 0.8),
-                tokens_used=response.usage.input_tokens + response.usage.output_tokens,
+                tokens_used=generation.tokens_used,
                 provider=self.provider_type.value,
                 metadata=metadata,
             )
 
         except Exception as e:
-            logger.error("Anthropic query generation failed", error=str(e))
-            raise ProviderException(f"Anthropic query generation failed: {e!s}")
+            logger.error("Anthropic query generation failed", error=str(e), error_type=type(e).__name__)
+            error = classify_provider_error(e, f"Anthropic query generation failed: {e!s}")
+            if error is e:
+                raise
+            raise error from e
 
     def _build_prompt(self, context: QueryContext) -> str:
         """Build prompt for query generation."""
@@ -256,6 +270,15 @@ Ensure the JSON is properly formatted with no syntax errors. Escape any quotes i
     def _parse_response(self, response) -> Dict[str, Any]:
         """Parse Anthropic response."""
         try:
+            stop_reason = getattr(response, "stop_reason", None)
+            if stop_reason == "max_tokens":
+                # An answer cut at max_tokens would be cut again, so it is not retried.
+                raise ProviderException("Response truncated at max_tokens (stop_reason=max_tokens); raise max_tokens")
+            if stop_reason == "refusal":
+                raise ProviderRequestError("Claude declined to answer (stop_reason=refusal)")
+            if not response.content:
+                raise ProviderResponseFormatError(f"Empty response (stop_reason={stop_reason})")
+
             raw_content = response.content[0].text.strip()
             content = raw_content
 
@@ -279,8 +302,8 @@ Ensure the JSON is properly formatted with no syntax errors. Escape any quotes i
             result = json.loads(content)
 
             # Validate required fields
-            if "sql" not in result:
-                raise ProviderException("Response missing required 'sql' field")
+            if not isinstance(result, dict) or "sql" not in result:
+                raise ProviderResponseFormatError("Response missing required 'sql' field")
 
             # Set defaults
             result.setdefault("explanation", "")
@@ -291,14 +314,21 @@ Ensure the JSON is properly formatted with no syntax errors. Escape any quotes i
 
             return result
 
+        except ProviderException:
+            raise
         except json.JSONDecodeError as e:
             # Log the problematic content
             logger.error("JSON parsing failed", content=content, error=str(e))
-            raise ProviderException(f"Invalid JSON response: {e!s}")
+            raise ProviderResponseFormatError(f"Invalid JSON response: {e!s}") from e
         except Exception as e:
             logger.error("Response parsing failed", error=str(e))
             raise ProviderException(f"Failed to parse response: {e!s}")
 
+    @staticmethod
+    def _total_tokens(response) -> int:
+        usage = response.usage
+        return (getattr(usage, "input_tokens", 0) or 0) + (getattr(usage, "output_tokens", 0) or 0)
+
     async def _validate_token_count(self, context: QueryContext) -> None:
         """Validate that context doesn't exceed token limits."""
         # Build prompt to count tokens
@@ -315,7 +345,11 @@ Ensure the JSON is properly formatted with no syntax errors. Escape any quotes i
         available_tokens = max_context - context.max_tokens  # Reserve space for response
 
         if total_tokens > available_tokens:
-            raise TokenLimitException(f"Context too large: {total_tokens} tokens exceeds limit of {available_tokens}")
+            raise TokenLimitException(
+                f"Context too large: {total_tokens} tokens exceeds limit of {available_tokens}",
+                tokens_used=total_tokens,
+                max_tokens=available_tokens,
+            )
 
         logger.debug(
             "Token validation passed",
@@ -379,6 +413,11 @@ Check for:
             result.setdefault("issues", [])
             result.setdefault("suggestions", [])
             result.setdefault("complexity", "moderate")
+            # Reported so the service can account for every LLM call of a question
+            usage = getattr(message, "usage", None)
+            input_tokens = getattr(usage, "input_tokens", 0) or 0
+            output_tokens = getattr(usage, "output_tokens", 0) or 0
+            result["_tokens_used"] = input_tokens + output_tokens
 
             return result