tempest-fastapi-sdk 0.96.0__tar.gz → 0.97.0__tar.gz

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/CHANGELOG.md

@@ -5,6 +5,28 @@ All notable changes to **tempest-fastapi-sdk** are listed below.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.97.0] — 2026-07-05
+
+### Added
+
+- **Self-hosted GenAI — RAG context (`tempest_fastapi_sdk.genai.rag`)**,
+  slice 2: feed a local LLM with web + PDF knowledge, without shipping
+  data to a third party.
+  - **Web search** — `WebSearchBackend` Protocol + `SearxngBackend` (the
+    leviathan pattern: SearXNG JSON API over an injected `httpx` client)
+    + `WebSearch` facade. Returns `SearchResult`s.
+  - **Content extraction** — `ContentExtractor` fetches a URL and pulls
+    the clean article body via `trafilatura`; failures surface as
+    `ExtractionResult(failed=True)`, never raised.
+  - **PDF reading** — `PdfReader` (PyMuPDF, detailed reading-order
+    extraction) → `Document` (text + per-page + metadata) and overlapping
+    `Chunk`s (`read` / `chunks`).
+  - **Context assembly** — `build_context(question, sources)` renders
+    `SearchResult`s and/or `Chunk`s into one prompt-ready, source-labeled
+    block (mix web + PDF, optional per-source truncation).
+  - New `[genai-rag]` extra (httpx + trafilatura + pymupdf); everything
+    imports lazily without it. Import from `tempest_fastapi_sdk.genai.rag`.
+
 ## [0.96.0] — 2026-07-05
 
 ### Added

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/CLAUDE.md

@@ -139,9 +139,15 @@ The SDK currently covers (Sep 2025+, post-v0.31.x):
   `ModelDtype`. Capacity fns import WITHOUT the extra. **Planned slices:**
   `TextGenerator` (LLM + int4/int8 quant, lazy load + idle-unload),
   `Embedder`, model/result cache, `BatchScheduler` (coalesce concurrent
-  calls), RAG context (pluggable web search "leviathan-style" + PDF
-  reader → chunks) to inject into LLMs. Classes-only (no bundled router).
-  Submodule import like queue/tasks/vision.
+  calls). Classes-only (no bundled router). Submodule import like
+  queue/tasks/vision. **Shipped (v0.97) — RAG context**
+  (`tempest_fastapi_sdk.genai.rag`, `[genai-rag]` extra = httpx +
+  trafilatura + pymupdf): `WebSearchBackend` Protocol + `SearxngBackend`
+  (SearXNG JSON API, leviathan pattern) + `WebSearch`; `ContentExtractor`
+  (trafilatura, failures never raise); `PdfReader` (PyMuPDF detailed
+  extraction → `Document`/`Chunk`, `read`/`chunks` with overlap);
+  `build_context(question, sources)` → prompt block mixing web +
+  PDF. All import lazily.
 - **SSR** (`[ssr]` extra) — `tempest_fastapi_sdk.ssr`: typed Python
   pages rendered to HTML via `tempestweb`'s `render_to_html` /
   `render_document`. `Page` (typed `Component` base — `body()` +

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tempest-fastapi-sdk
-Version: 0.96.0
+Version: 0.97.0
 Summary: Shared FastAPI building blocks: base schemas, ORM model, async repository, exceptions, pagination and settings — the conventions used across Tempest projects.
 Project-URL: Homepage, https://github.com/mauriciobenjamin700/tempest-fastapi-sdk
 Project-URL: Repository, https://github.com/mauriciobenjamin700/tempest-fastapi-sdk
@@ -77,6 +77,10 @@ Requires-Dist: torch>=2.2.0; extra == 'genai'
 Requires-Dist: transformers>=4.44.0; extra == 'genai'
 Provides-Extra: genai-quant
 Requires-Dist: bitsandbytes>=0.43.0; extra == 'genai-quant'
+Provides-Extra: genai-rag
+Requires-Dist: httpx>=0.28.0; extra == 'genai-rag'
+Requires-Dist: pymupdf>=1.24.0; extra == 'genai-rag'
+Requires-Dist: trafilatura>=1.12.0; extra == 'genai-rag'
 Provides-Extra: http
 Requires-Dist: httpx>=0.28.0; extra == 'http'
 Provides-Extra: metrics
@@ -235,6 +239,7 @@ Feature-rich helpers pull in third-party dependencies that you only need when yo
 | `[mfa]` | `pyotp` | `TOTPHelper` + MFA/2FA endpoints on the bundled auth flow |
 | `[vision]` | `ort-vision-sdk` | `Detector` / `Classifier` / `Segmenter` (ONNX) + prediction schemas |
 | `[genai]` (+ `[genai-quant]`) | `transformers`, `torch`, `accelerate`, `safetensors`, `huggingface-hub` (+ `bitsandbytes`) | Self-hosted GenAI — hardware capacity check (`probe_hardware` / `can_run` / `recommend`); model runners upcoming |
+| `[genai-rag]` | `httpx`, `trafilatura`, `pymupdf` | RAG context for local LLMs — SearXNG web search, page extraction, PDF reading, `build_context` |
 | `[ssr]` | `tempestweb` | `Page`, `html_response`, `make_htmx_router` — typed Python pages rendered to HTML |
 | `[otel]` | `opentelemetry-sdk` + OTLP/gRPC exporter + FastAPI/SQLAlchemy/httpx instrumentors | `setup_tracing` — distributed tracing |
 | `[sqlite]` | `aiosqlite` | SQLite async driver for `sqlite+aiosqlite://` URLs (dev default) |
@@ -277,7 +282,8 @@ Since `0.7.1` every optional dependency is imported lazily at first instantiatio
 | `tempest_fastapi_sdk.utils.storage_backends` *(extra: `[upload]`)* | `UploadStorage` protocol, `LocalUploadStorage`, `MinIOUploadStorage`, `UploadResult`, `ContentValidator` |
 | `tempest_fastapi_sdk.tasks` *(extra: `[tasks]`)* | `TaskQueue` (typed TaskIQ facade — `.rabbitmq`/`.redis`/`.memory`, `@tq.task` → `Task.enqueue`/`.run`, `@tq.cron`/`@tq.interval`, `start_scheduler`), `Task`; class-based tasks (`TaskDef` + `@task_method` + `TaskQueue.register`); cron helpers (`Cron` / `CronOffset` / `Weekday` + `daily`/`weekdays`/`every_n_minutes`/… builders); `AsyncTaskBrokerManager` / `AsyncTaskScheduler` (legacy wrappers) |
 | `tempest_fastapi_sdk.vision` *(extra: `[vision]`)* | `Detector`, `Classifier`, `Segmenter` (ONNX, lazy), `DetectionSchema`/`ClassificationSchema`/`SegmentationSchema`/`BoundingBoxSchema`/`ClassProbabilitySchema`, `to_detection_schemas`/`to_classification_schema`/`to_segmentation_schemas` |
-| `tempest_fastapi_sdk.genai` *(extra: `[genai]`; capacity check imports without it)* | Self-hosted GenAI capacity check: `probe_hardware`/`HardwareInfo`, `can_run`/`recommend`/`CapacityReport`, `estimate_model_bytes`/`bytes_per_param`/`fetch_num_params`, `ModelDtype` — model runners (`TextGenerator`, `Embedder`), caching, batching and RAG context in upcoming slices |
+| `tempest_fastapi_sdk.genai` *(extra: `[genai]`; capacity check imports without it)* | Self-hosted GenAI capacity check: `probe_hardware`/`HardwareInfo`, `can_run`/`recommend`/`CapacityReport`, `estimate_model_bytes`/`bytes_per_param`/`fetch_num_params`, `ModelDtype` — model runners (`TextGenerator`, `Embedder`), caching and batching in upcoming slices |
+| `tempest_fastapi_sdk.genai.rag` *(extra: `[genai-rag]`)* | RAG context for local LLMs: `WebSearch`/`SearxngBackend`/`WebSearchBackend` + `SearchResult`, `ContentExtractor` (trafilatura), `PdfReader` (PyMuPDF) + `Document`/`Chunk`/`PdfPage`, `build_context` |
 | `tempest_fastapi_sdk.ssr` *(extra: `[ssr]`)* | `Page` (typed component base), `html_response` (widget tree → `HTMLResponse`, full document or HTMX fragment), `make_htmx_router` (serves bundled HTMX locally, no CDN) |
 | `tempest_fastapi_sdk.utils` | `to_utc`, `utcnow`, `modify_dict`, `LogUtils`, `AttemptThrottle`/`ThrottleBackend`/`ThrottleStatus`, `generate_opaque_token`/`hash_opaque_token`/`verify_opaque_token`, `get_client_ip`/`get_client_ip_from_scope`, `PasswordUtils` *(extra: `[auth]`)*, `JWTUtils` *(extra: `[auth]`)*, `TOTPHelper` *(extra: `[mfa]`)*, `EmailUtils` *(extra: `[email]`)*, `UploadUtils`/`sniff_mime` *(extra: `[upload]`)*, `DownloadUtils`/`build_content_disposition` *(no extra)*, `FileStoreUtils` — unified upload+download+presign facade *(extra: `[upload]` local / `[minio]` MinIO)*, `MetricsUtils`/`CPUMetrics`/`MemoryMetrics`/`DiskMetrics`/`GPUMetrics`/`SystemMetrics` *(extra: `[metrics]`)*, validated field types (`PositiveIntField`, `NonNegativeIntField`, `CentsField`, `PortField`, `PositiveFloatField`, `NonNegativeFloatField`, `PercentField`, `RatioField`, `LatitudeField`, `LongitudeField`, `PriceField`, `NonEmptyStrField`, `SlugField`, `HexColorField`), BR regex helpers (`CPFField`, `CNPJField`, `CPFOrCNPJField`, `PhoneBRField`, `CEPField`, `PixKeyField` — old names without the suffix kept as deprecated aliases — `PixKeyType`/`detect_pix_key_type`/`is_valid_pix_key`, `is_valid_*`, `normalize_*`, `only_digits`, `*_PATTERN`), BR states/cities (`UF`, `Region`, `StateBR`, `CityBR`, `ChoiceBR`, `UFField`, `CityNameField`, `list_states`, `get_state`, `cities_by_uf`, `states_by_region`, `uf_choices`/`region_choices`/`city_choices`, `is_valid_uf`/`normalize_uf`, `is_valid_city`/`normalize_city`) |
 | `tempest_fastapi_sdk.cli` | `tempest` console script — `new <name>` (scaffold layered service), `lint` / `format` / `fmt-check` / `type` / `test` / `check` (run preferred quality gates), `version` / `--version` |

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/README.md

@@ -118,6 +118,7 @@ Feature-rich helpers pull in third-party dependencies that you only need when yo
 | `[mfa]` | `pyotp` | `TOTPHelper` + MFA/2FA endpoints on the bundled auth flow |
 | `[vision]` | `ort-vision-sdk` | `Detector` / `Classifier` / `Segmenter` (ONNX) + prediction schemas |
 | `[genai]` (+ `[genai-quant]`) | `transformers`, `torch`, `accelerate`, `safetensors`, `huggingface-hub` (+ `bitsandbytes`) | Self-hosted GenAI — hardware capacity check (`probe_hardware` / `can_run` / `recommend`); model runners upcoming |
+| `[genai-rag]` | `httpx`, `trafilatura`, `pymupdf` | RAG context for local LLMs — SearXNG web search, page extraction, PDF reading, `build_context` |
 | `[ssr]` | `tempestweb` | `Page`, `html_response`, `make_htmx_router` — typed Python pages rendered to HTML |
 | `[otel]` | `opentelemetry-sdk` + OTLP/gRPC exporter + FastAPI/SQLAlchemy/httpx instrumentors | `setup_tracing` — distributed tracing |
 | `[sqlite]` | `aiosqlite` | SQLite async driver for `sqlite+aiosqlite://` URLs (dev default) |
@@ -160,7 +161,8 @@ Since `0.7.1` every optional dependency is imported lazily at first instantiatio
 | `tempest_fastapi_sdk.utils.storage_backends` *(extra: `[upload]`)* | `UploadStorage` protocol, `LocalUploadStorage`, `MinIOUploadStorage`, `UploadResult`, `ContentValidator` |
 | `tempest_fastapi_sdk.tasks` *(extra: `[tasks]`)* | `TaskQueue` (typed TaskIQ facade — `.rabbitmq`/`.redis`/`.memory`, `@tq.task` → `Task.enqueue`/`.run`, `@tq.cron`/`@tq.interval`, `start_scheduler`), `Task`; class-based tasks (`TaskDef` + `@task_method` + `TaskQueue.register`); cron helpers (`Cron` / `CronOffset` / `Weekday` + `daily`/`weekdays`/`every_n_minutes`/… builders); `AsyncTaskBrokerManager` / `AsyncTaskScheduler` (legacy wrappers) |
 | `tempest_fastapi_sdk.vision` *(extra: `[vision]`)* | `Detector`, `Classifier`, `Segmenter` (ONNX, lazy), `DetectionSchema`/`ClassificationSchema`/`SegmentationSchema`/`BoundingBoxSchema`/`ClassProbabilitySchema`, `to_detection_schemas`/`to_classification_schema`/`to_segmentation_schemas` |
-| `tempest_fastapi_sdk.genai` *(extra: `[genai]`; capacity check imports without it)* | Self-hosted GenAI capacity check: `probe_hardware`/`HardwareInfo`, `can_run`/`recommend`/`CapacityReport`, `estimate_model_bytes`/`bytes_per_param`/`fetch_num_params`, `ModelDtype` — model runners (`TextGenerator`, `Embedder`), caching, batching and RAG context in upcoming slices |
+| `tempest_fastapi_sdk.genai` *(extra: `[genai]`; capacity check imports without it)* | Self-hosted GenAI capacity check: `probe_hardware`/`HardwareInfo`, `can_run`/`recommend`/`CapacityReport`, `estimate_model_bytes`/`bytes_per_param`/`fetch_num_params`, `ModelDtype` — model runners (`TextGenerator`, `Embedder`), caching and batching in upcoming slices |
+| `tempest_fastapi_sdk.genai.rag` *(extra: `[genai-rag]`)* | RAG context for local LLMs: `WebSearch`/`SearxngBackend`/`WebSearchBackend` + `SearchResult`, `ContentExtractor` (trafilatura), `PdfReader` (PyMuPDF) + `Document`/`Chunk`/`PdfPage`, `build_context` |
 | `tempest_fastapi_sdk.ssr` *(extra: `[ssr]`)* | `Page` (typed component base), `html_response` (widget tree → `HTMLResponse`, full document or HTMX fragment), `make_htmx_router` (serves bundled HTMX locally, no CDN) |
 | `tempest_fastapi_sdk.utils` | `to_utc`, `utcnow`, `modify_dict`, `LogUtils`, `AttemptThrottle`/`ThrottleBackend`/`ThrottleStatus`, `generate_opaque_token`/`hash_opaque_token`/`verify_opaque_token`, `get_client_ip`/`get_client_ip_from_scope`, `PasswordUtils` *(extra: `[auth]`)*, `JWTUtils` *(extra: `[auth]`)*, `TOTPHelper` *(extra: `[mfa]`)*, `EmailUtils` *(extra: `[email]`)*, `UploadUtils`/`sniff_mime` *(extra: `[upload]`)*, `DownloadUtils`/`build_content_disposition` *(no extra)*, `FileStoreUtils` — unified upload+download+presign facade *(extra: `[upload]` local / `[minio]` MinIO)*, `MetricsUtils`/`CPUMetrics`/`MemoryMetrics`/`DiskMetrics`/`GPUMetrics`/`SystemMetrics` *(extra: `[metrics]`)*, validated field types (`PositiveIntField`, `NonNegativeIntField`, `CentsField`, `PortField`, `PositiveFloatField`, `NonNegativeFloatField`, `PercentField`, `RatioField`, `LatitudeField`, `LongitudeField`, `PriceField`, `NonEmptyStrField`, `SlugField`, `HexColorField`), BR regex helpers (`CPFField`, `CNPJField`, `CPFOrCNPJField`, `PhoneBRField`, `CEPField`, `PixKeyField` — old names without the suffix kept as deprecated aliases — `PixKeyType`/`detect_pix_key_type`/`is_valid_pix_key`, `is_valid_*`, `normalize_*`, `only_digits`, `*_PATTERN`), BR states/cities (`UF`, `Region`, `StateBR`, `CityBR`, `ChoiceBR`, `UFField`, `CityNameField`, `list_states`, `get_state`, `cities_by_uf`, `states_by_region`, `uf_choices`/`region_choices`/`city_choices`, `is_valid_uf`/`normalize_uf`, `is_valid_city`/`normalize_city`) |
 | `tempest_fastapi_sdk.cli` | `tempest` console script — `new <name>` (scaffold layered service), `lint` / `format` / `fmt-check` / `type` / `test` / `check` (run preferred quality gates), `version` / `--version` |

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/docs/recipes/genai.en.md

@@ -6,10 +6,11 @@ slices; this page covers the **first**: knowing, *before* you download
 gigabytes of weights, whether the machine can handle the model.
 
 !!! info "Module roadmap"
-    - **Now (v0.96):** `genai.hardware` — probing + `can_run` / `recommend`.
+    - **v0.96:** `genai.hardware` — probing + `can_run` / `recommend`.
+    - **v0.97:** `genai.rag` — RAG context (SearXNG web search + PDF
+      reading) to inject into LLMs (this page, [RAG context](#rag-context)).
     - **Coming:** `TextGenerator` (LLM + int4/int8 quantization),
-      `Embedder`, model/result caching, `BatchScheduler`, and RAG context
-      (web search + PDF reading) to inject into LLMs.
+      `Embedder`, model/result caching, `BatchScheduler`.
 
 The `[genai]` extra (transformers + torch + accelerate) is only needed to
 **run** models. The capacity functions **import without the extra** —
@@ -81,10 +82,83 @@ The parameter count can be passed explicitly (`num_params=`) or read from
 the Hub by `model_id` (via `huggingface_hub`, without downloading the
 weights — safetensors metadata).
 
+## RAG context
+
+A local LLM only knows what it trained on. For current, grounded answers,
+inject context: `tempest_fastapi_sdk.genai.rag` searches the web
+(self-hosted SearXNG), extracts page bodies, reads PDFs, and assembles a
+prompt-ready block — without sending data outside. Needs the `[genai-rag]`
+extra (httpx + trafilatura + pymupdf); the pieces import lazily.
+
+### Web search (SearXNG)
+
+```python
+import httpx
+from tempest_fastapi_sdk.genai.rag import SearxngBackend, WebSearch, build_context
+
+client = httpx.AsyncClient()
+search = WebSearch(SearxngBackend("http://localhost:8080", http_client=client))
+
+results = await search.search("what is PIX?", max_results=5)   # list[SearchResult]
+context = build_context("what is PIX?", results, long_text=False, max_chars=2000)
+# -> a string ready to inject into your TextGenerator prompt
+```
+
+The backend is a `Protocol` (`WebSearchBackend`) — swap SearXNG for another
+provider without touching call sites. The `httpx.AsyncClient` is injected
+(pool reuse; wire it in the FastAPI lifespan).
+
+### Extract page bodies
+
+Search snippets are thin. To give the LLM ground truth, fetch each page
+and extract the clean text (via `trafilatura`):
+
+```python
+from tempest_fastapi_sdk.genai.rag import ContentExtractor
+
+extractor = ContentExtractor(http_client=client)
+for result in results:
+    outcome = await extractor.extract(result.url)
+    result.content = outcome.text          # "" on failure; outcome.failed marks it
+context = build_context("what is PIX?", results)   # now with full bodies
+```
+
+Failures (timeout, 4xx/5xx, empty page) **never** raise — they come back
+as `ExtractionResult(text="", failed=True)`, so no source is silently
+dropped.
+
+### Read PDFs (knowledge base)
+
+`PdfReader` (PyMuPDF — detailed, reading-order extraction) turns PDF paths
+into text and prompt/embedding-ready chunks:
+
+```python
+from tempest_fastapi_sdk.genai.rag import PdfReader, build_context
+
+reader = PdfReader()
+doc = reader.read("/kb/manual.pdf")             # Document: text + pages + metadata
+chunks = reader.chunks("/kb/manual.pdf", max_chars=2000, overlap=200)
+
+context = build_context("how to refund?", chunks)   # cites "file (page N)"
+```
+
+`chunks(..., overlap=200)` shares characters between neighbors so a fact on
+a boundary isn't cut in half; `per_page=True` (default) keeps each chunk on
+a single page and records its page number.
+
+!!! tip "Mix web + PDF in one context"
+    `build_context` accepts `SearchResult` and `Chunk` in the same list —
+    it delimits each source with `---` and labels the origin (URL or `file
+    (page N)`) so the LLM can cite. Pass `long_text=False` to truncate each
+    source to `max_chars`.
+
 ## Recap
 
 - **`can_run` / `recommend`** — answer whether the host runs the model
   and what to do if not, **before** the download.
+- **RAG** — `WebSearch`/`SearxngBackend` (search), `ContentExtractor`
+  (page bodies), `PdfReader` (PDF → text/chunks) and `build_context`
+  (prompt block), all under the `[genai-rag]` extra.
 - **`probe_hardware`** — a CPU/RAM/GPU/disk snapshot; degrades without the
   extras.
 - **`estimate_model_bytes` / `bytes_per_param`** — the estimation math,

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/docs/recipes/genai.md

@@ -6,10 +6,11 @@ está sendo entregue em fatias; esta página cobre a **primeira**: saber,
 *antes* de baixar gigabytes de pesos, se a máquina aguenta o modelo.
 
 !!! info "Roadmap do módulo"
-    - **Agora (v0.96):** `genai.hardware` — sondagem + `can_run` / `recommend`.
+    - **v0.96:** `genai.hardware` — sondagem + `can_run` / `recommend`.
+    - **v0.97:** `genai.rag` — contexto RAG (busca web SearXNG + leitura de
+      PDF) pra injetar nas LLMs (esta página, seção [Contexto RAG](#contexto-rag)).
     - **Em breve:** `TextGenerator` (LLM + quantização int4/int8),
-      `Embedder`, cache de modelo/resultado, `BatchScheduler`, e
-      contexto RAG (busca web + leitura de PDF) pra injetar nas LLMs.
+      `Embedder`, cache de modelo/resultado, `BatchScheduler`.
 
 O extra `[genai]` (transformers + torch + accelerate) só é necessário pra
 **rodar** modelos. As funções de capacidade **importam sem o extra** — o
@@ -81,10 +82,84 @@ O número de parâmetros pode vir explícito (`num_params=`) ou ser lido do
 Hub por `model_id` (via `huggingface_hub`, sem baixar os pesos —
 metadados safetensors).
 
+## Contexto RAG
+
+Uma LLM local só sabe o que treinou. Pra respostas atuais e fundamentadas,
+injete contexto: `tempest_fastapi_sdk.genai.rag` busca na web (SearXNG
+self-hosted), extrai o corpo das páginas, lê PDFs e monta um bloco pronto
+pro prompt — tudo sem enviar dados pra fora. Requer o extra `[genai-rag]`
+(httpx + trafilatura + pymupdf); as peças importam preguiçosamente.
+
+### Busca web (SearXNG)
+
+```python
+import httpx
+from tempest_fastapi_sdk.genai.rag import SearxngBackend, WebSearch, build_context
+
+client = httpx.AsyncClient()
+search = WebSearch(SearxngBackend("http://localhost:8080", http_client=client))
+
+results = await search.search("o que é PIX?", max_results=5)   # list[SearchResult]
+context = build_context("o que é PIX?", results, long_text=False, max_chars=2000)
+# -> string pronta pra injetar no prompt do seu TextGenerator
+```
+
+O backend é um `Protocol` (`WebSearchBackend`) — troque o SearXNG por
+outro provedor sem mexer no call site. O `httpx.AsyncClient` é injetado
+(reaproveita o pool; ligue no lifespan do FastAPI).
+
+### Extrair o corpo das páginas
+
+O snippet do buscador é raso. Pra dar "verdade" pra LLM, busque cada
+página e extraia o texto limpo (via `trafilatura`):
+
+```python
+from tempest_fastapi_sdk.genai.rag import ContentExtractor
+
+extractor = ContentExtractor(http_client=client)
+for result in results:
+    outcome = await extractor.extract(result.url)
+    result.content = outcome.text          # "" quando falha; outcome.failed marca
+context = build_context("o que é PIX?", results)   # agora com corpo completo
+```
+
+Falhas (timeout, 4xx/5xx, página sem corpo) **nunca** levantam — voltam
+como `ExtractionResult(text="", failed=True)`, então nenhuma fonte some
+silenciosamente.
+
+### Ler PDFs (base de conhecimento)
+
+`PdfReader` (PyMuPDF — extração detalhada, ordem de leitura) transforma
+caminhos de PDF em texto e em chunks prontos pra prompt ou índice de
+embeddings:
+
+```python
+from tempest_fastapi_sdk.genai.rag import PdfReader, build_context
+
+reader = PdfReader()
+doc = reader.read("/base/manual.pdf")            # Document: text + pages + metadata
+chunks = reader.chunks("/base/manual.pdf", max_chars=2000, overlap=200)
+
+context = build_context("como estornar?", chunks)   # cita "arquivo (page N)"
+```
+
+`chunks(..., overlap=200)` compartilha caracteres entre pedaços vizinhos,
+pra um fato na fronteira não ser cortado ao meio; `per_page=True` (padrão)
+mantém cada chunk numa página só, carregando o número dela.
+
+!!! tip "Misture web + PDF no mesmo contexto"
+    `build_context` aceita `SearchResult` e `Chunk` na mesma lista —
+    delimita cada fonte com `---` e rotula a origem (URL ou `arquivo
+    (page N)`), pra LLM citar. Passe `long_text=False` pra truncar cada
+    fonte a `max_chars`.
+
 ## Recap
 
 - **`can_run` / `recommend`** — respondem se o host roda o modelo e o que
   fazer se não rodar, **antes** do download.
+- **RAG** — `WebSearch`/`SearxngBackend` (busca), `ContentExtractor`
+  (corpo das páginas), `PdfReader` (PDF → texto/chunks) e `build_context`
+  (bloco pro prompt), todos sob o extra `[genai-rag]`.
 - **`probe_hardware`** — snapshot de CPU/RAM/GPU/disco; degrada sem os
   extras.
 - **`estimate_model_bytes` / `bytes_per_param`** — a matemática da

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tempest-fastapi-sdk"
-version = "0.96.0"
+version = "0.97.0"
 description = "Shared FastAPI building blocks: base schemas, ORM model, async repository, exceptions, pagination and settings — the conventions used across Tempest projects."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -101,6 +101,11 @@ genai = [
 genai-quant = [
     "bitsandbytes>=0.43.0",
 ]
+genai-rag = [
+    "httpx>=0.28.0",
+    "trafilatura>=1.12.0",
+    "pymupdf>=1.24.0",
+]
 ssr = [
     "tempestweb>=0.9.0",
 ]
@@ -205,6 +210,8 @@ dev = [
     "nvidia-ml-py>=12.560.30",
     "redis>=5.0.0",
     "ruff>=0.8.0",
+    "trafilatura>=1.12.0",
+    "pymupdf>=1.24.0",
     "faststream[rabbit]>=0.7.1",
     "taskiq>=0.11.0",
     "taskiq-aio-pika>=0.4.0",
@@ -320,6 +327,10 @@ module = [
     "huggingface_hub.*",
     "accelerate",
     "accelerate.*",
+    "trafilatura",
+    "trafilatura.*",
+    "pymupdf",
+    "fitz",
     "pyotp",
     "pyotp.*",
     "ort_vision_sdk",

{tempest_fastapi_sdk-0.96.0 → tempest_fastapi_sdk-0.97.0}/tempest_fastapi_sdk/__init__.py

@@ -372,7 +372,7 @@ from tempest_fastapi_sdk.websockets import (
     make_websocket_router,
 )
 
-__version__: str = "0.96.0"
+__version__: str = "0.97.0"
 
 __all__: list[str] = [
     "BASE_COLUMN_ORDER",

tempest_fastapi_sdk-0.97.0/tempest_fastapi_sdk/genai/rag/__init__.py

@@ -0,0 +1,39 @@
+"""RAG context — feed LLMs with web search + PDF knowledge.
+
+Retrieve grounding for a local LLM without shipping data to a third
+party: search a self-hosted SearXNG, extract page bodies, read PDFs, and
+assemble it all into a prompt-ready context block. The heavy bits
+(``httpx`` / ``trafilatura`` / ``pymupdf``) live behind the
+``[genai-rag]`` extra and import lazily, so this package imports without
+it — you only need the extra to actually fetch and read.
+"""
+
+from tempest_fastapi_sdk.genai.rag.context import build_context as build_context
+from tempest_fastapi_sdk.genai.rag.extract import (
+    ContentExtractor as ContentExtractor,
+)
+from tempest_fastapi_sdk.genai.rag.extract import (
+    ExtractionResult as ExtractionResult,
+)
+from tempest_fastapi_sdk.genai.rag.pdf import PdfReader as PdfReader
+from tempest_fastapi_sdk.genai.rag.schemas import Chunk as Chunk
+from tempest_fastapi_sdk.genai.rag.schemas import Document as Document
+from tempest_fastapi_sdk.genai.rag.schemas import PdfPage as PdfPage
+from tempest_fastapi_sdk.genai.rag.schemas import SearchResult as SearchResult
+from tempest_fastapi_sdk.genai.rag.search import SearxngBackend as SearxngBackend
+from tempest_fastapi_sdk.genai.rag.search import WebSearch as WebSearch
+from tempest_fastapi_sdk.genai.rag.search import WebSearchBackend as WebSearchBackend
+
+__all__: list[str] = [
+    "Chunk",
+    "ContentExtractor",
+    "Document",
+    "ExtractionResult",
+    "PdfPage",
+    "PdfReader",
+    "SearchResult",
+    "SearxngBackend",
+    "WebSearch",
+    "WebSearchBackend",
+    "build_context",
+]

tempest_fastapi_sdk-0.97.0/tempest_fastapi_sdk/genai/rag/context.py

@@ -0,0 +1,85 @@
+"""Assemble retrieved sources into a single prompt-ready context block.
+
+Turns web :class:`SearchResult`s and PDF :class:`Chunk`s into one plain-text
+block you drop straight into a system/user prompt. Each source is
+delimited and labeled with its origin so the model can cite it. Mirrors
+the leviathan ContextBuilder: plain text, clear structure, optional
+truncation per source.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from tempest_fastapi_sdk.genai.rag.schemas import Chunk, SearchResult
+
+
+def _body(source: SearchResult | Chunk, *, long_text: bool, max_chars: int) -> str:
+    """Return the text body of a source, truncated when not ``long_text``."""
+    if isinstance(source, SearchResult):
+        text = source.content or source.snippet
+    else:
+        text = source.text
+    text = text.strip()
+    if not long_text and len(text) > max_chars:
+        return text[:max_chars].rstrip() + " …"
+    return text
+
+
+def _label(source: SearchResult | Chunk) -> str:
+    """Return a citation label for a source."""
+    if isinstance(source, SearchResult):
+        title = source.title or source.url
+        return f"{title} ({source.url})"
+    if source.page is not None:
+        return f"{source.source} (page {source.page})"
+    return source.source
+
+
+def build_context(
+    question: str,
+    sources: Sequence[SearchResult | Chunk],
+    *,
+    long_text: bool = True,
+    max_chars: int = 2000,
+) -> str:
+    """Render retrieved sources into a single context string for an LLM.
+
+    Args:
+        question (str): The user's question, echoed at the top so the model
+            has the task in view.
+        sources (Sequence[SearchResult | Chunk]): Web results and/or PDF
+            chunks, in the order they should appear.
+        long_text (bool): When ``True`` (default), include each source's
+            full body; when ``False``, truncate each to ``max_chars``.
+        max_chars (int): Per-source truncation cap when ``long_text`` is
+            ``False``.
+
+    Returns:
+        str: A prompt-ready block. Sources are delimited by ``---`` and
+        labeled with their origin; returns a short notice when ``sources``
+        is empty.
+    """
+    if not sources:
+        return f"Question: {question}\n\nNo sources were retrieved for this question."
+
+    parts: list[str] = [
+        f"Question: {question}",
+        "",
+        f"Below are {len(sources)} sources retrieved to answer this question. "
+        "Each is delimited by `---`. Use them as ground truth; cite the "
+        "source when relevant.",
+        "",
+    ]
+    for index, source in enumerate(sources, start=1):
+        parts.append("---")
+        parts.append(f"[{index}] {_label(source)}")
+        parts.append(_body(source, long_text=long_text, max_chars=max_chars))
+        parts.append("")
+    parts.append("---")
+    return "\n".join(parts)
+
+
+__all__: list[str] = [
+    "build_context",
+]

tempest_fastapi_sdk-0.97.0/tempest_fastapi_sdk/genai/rag/extract.py

@@ -0,0 +1,111 @@
+"""Fetch a page and extract its main text (HTML → clean text).
+
+Search snippets are thin; to give an LLM real ground truth you fetch each
+result and pull the article body out of the HTML. Uses ``trafilatura``
+(the leviathan choice) for cleaning. Failures never raise — a page that
+times out or yields nothing comes back as ``failed=True`` with empty
+text, so no source is silently dropped.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    import httpx
+
+
+@dataclass(slots=True)
+class ExtractionResult:
+    """Outcome of fetching + extracting one URL.
+
+    Attributes:
+        text (str): The cleaned page body, or ``""`` on failure.
+        failed (bool): ``True`` when the page couldn't be fetched or no
+            text could be extracted.
+    """
+
+    text: str
+    failed: bool
+
+
+def _require_trafilatura() -> object:
+    """Import ``trafilatura`` or raise a helpful error.
+
+    Returns:
+        object: The ``trafilatura`` module.
+
+    Raises:
+        ImportError: When the ``[genai-rag]`` extra is not installed.
+    """
+    try:
+        import trafilatura
+    except ImportError as exc:
+        raise ImportError(
+            "Content extraction requires the optional [genai-rag] extra. "
+            "Install with: pip install tempest-fastapi-sdk[genai-rag]",
+        ) from exc
+    return trafilatura
+
+
+class ContentExtractor:
+    """Fetch URLs and extract their main text via ``trafilatura``.
+
+    The ``httpx.AsyncClient`` is injected so the connection pool is shared.
+
+    Attributes:
+        user_agent (str): ``User-Agent`` header sent with each fetch.
+        timeout (float): Per-request timeout in seconds.
+    """
+
+    def __init__(
+        self,
+        *,
+        http_client: httpx.AsyncClient,
+        user_agent: str = "tempest-fastapi-sdk/genai",
+        timeout: float = 10.0,
+    ) -> None:
+        """Initialize the extractor.
+
+        Args:
+            http_client (httpx.AsyncClient): Injected async client.
+            user_agent (str): ``User-Agent`` for fetches.
+            timeout (float): Per-request timeout in seconds.
+        """
+        self._http = http_client
+        self.user_agent = user_agent
+        self.timeout = timeout
+
+    async def extract(self, url: str) -> ExtractionResult:
+        """Fetch ``url`` and return its extracted main text.
+
+        Never raises: fetch/extraction failures come back as
+        ``ExtractionResult(text="", failed=True)``.
+
+        Args:
+            url (str): The page to fetch.
+
+        Returns:
+            ExtractionResult: The extracted text or a failure marker.
+        """
+        trafilatura = _require_trafilatura()
+        try:
+            response = await self._http.get(
+                url,
+                headers={"User-Agent": self.user_agent},
+                timeout=self.timeout,
+                follow_redirects=True,
+            )
+            response.raise_for_status()
+        except Exception:
+            return ExtractionResult(text="", failed=True)
+
+        text = trafilatura.extract(response.text) or ""  # type: ignore[attr-defined]
+        return ExtractionResult(text=text, failed=not text)
+
+
+__all__: list[str] = [
+    "ContentExtractor",
+    "ExtractionResult",
+]