quantilica-cli 0.11.0__tar.gz → 0.13.0__tar.gz

quantilica_cli-0.13.0/.github/workflows/public-hygiene.yml

@@ -0,0 +1,104 @@
+name: Public hygiene
+
+# Veta termos internos (planos, decisões, repositórios privados, infraestrutura)
+# em CHANGELOG.md, README e nas mensagens de commit de PRs. O repositório é
+# público: o que entra aqui fica no histórico para sempre.
+# Regra e lista de termos: docs/normas/changelog.md, seção 5.1.
+# Falso positivo legítimo: termine a linha com `<!-- hygiene-ok -->`.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  hygiene:
+    name: Termos internos
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Escrever lista de termos
+        run: |
+          cat > "$RUNNER_TEMP/termos-comuns.txt" <<'EOF'
+          \bOnda\b
+          \bADR\b
+          \bplanning/
+          \bplano (`|[a-z0-9-]+-[a-z0-9-]+|§)
+          \bPlano:
+          uploader
+          \bCAS\b
+          \bVPS\b
+          crontab
+          \bcrons?\b
+          systemd
+          runs\.csv
+          quantilica-(workers|cloud|web|events|portal|manifests-db)
+          bcb-sgs-app
+          rtn-app
+          crons-dashboard
+          datasus-analytics
+          [a-z-]+-metadata-db
+          tddata-db
+          dev-plugins
+          \bOKF\b
+          Flamingo
+          Alpaca
+          Hedgehog
+          Sloth
+          \bFish\b
+          Telegram
+          Umami
+          Cloudflare
+          (^|[^[:alnum:].])/home/[a-z]+
+          \.agents/
+          CLAUDE\.md
+          AGENTS\.md
+          EOF
+          cat > "$RUNNER_TEMP/termos-arquivos.txt" <<'EOF'
+          \bworkspace\b
+          ingestão efêmera
+          retenção efêmera
+          pipeline de ingestão
+          EOF
+          cat "$RUNNER_TEMP/termos-comuns.txt" "$RUNNER_TEMP/termos-arquivos.txt" \
+            > "$RUNNER_TEMP/termos-todos.txt"
+
+      - name: CHANGELOG e README
+        run: |
+          files=$(ls CHANGELOG.md README*.md docs/changelog.md 2>/dev/null || true)
+          if [ -z "$files" ]; then
+            echo "Nenhum CHANGELOG/README para verificar."
+            exit 0
+          fi
+          # shellcheck disable=SC2086
+          hits=$(grep -nIiE -f "$RUNNER_TEMP/termos-todos.txt" $files /dev/null \
+            | grep -v 'hygiene-ok' || true)
+          if [ -n "$hits" ]; then
+            echo "$hits"
+            echo "::error::Termos internos em changelog/README. Reescreva em termos públicos (docs/normas/changelog.md, seção 5.1)."
+            exit 1
+          fi
+          echo "Limpo."
+
+      - name: Mensagens de commit do PR
+        if: github.event_name == 'pull_request'
+        run: |
+          range="${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }}"
+          msgs=$(git log --format='%h %s%n%b' "$range")
+          hits=$(printf '%s\n' "$msgs" \
+            | grep -iE -f "$RUNNER_TEMP/termos-comuns.txt" \
+            | grep -v 'hygiene-ok' || true)
+          if [ -n "$hits" ]; then
+            echo "$hits"
+            echo "::error::Termos internos em mensagem de commit. Reescreva o commit (docs/normas/changelog.md, seção 5.1)."
+            exit 1
+          fi
+          echo "Limpo."

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/CHANGELOG.md

@@ -5,6 +5,30 @@ Todas as mudanças notáveis deste projeto serão documentadas neste arquivo.
 O formato segue [Keep a Changelog](https://keepachangelog.com/pt-BR/1.1.0/),
 e este projeto adere ao [Semantic Versioning](https://semver.org/lang/pt-BR/).
 
+## [0.13.0] - 2026-10-27
+
+### Adicionado
+
+- `convert`: novas opções `--lake` (raiz do lake de destino; padrão `lake_root()` de `quantilica.core.lake`, ou `default_output` se o core não tiver o módulo), `--dataset` (repetível) e `--force`. `-o/--output` continua aceito como alias de `--lake`. O conversor é chamado como `func(input, lake)` e recebe `dataset=` e `force=` por keyword só quando a assinatura os aceita (ou tem `**kwargs`); pedir `--dataset`/`--force` a um conversor que não os aceita sai com código 2.
+- `convert` e `pipeline`: se o conversor devolver um objeto (ou dicionário) com `converted`, `skipped` e `errors`, o sumário é exibido, e `errors > 0` sai com código 1.
+- `pipeline --lake` (com `--parquet-dir` mantido como alias), repassado ao conversor.
+- `quantilica doctor`: informa, por fonte, se há conversor do lake registrado no entry point `quantilica.lake`.
+
+### Alterado
+
+- `pipeline`: o destino da conversão passa a ser `lake_root()` por padrão, em vez de igual a `--output`.
+
+## [0.12.0] - 2026-10-19
+
+### Adicionado
+
+- `FetcherApp(source_id=...)`: `source_id` explícito, gravado nos manifests de download. Sem ele, continua derivado de `name` sem o sufixo `-fetcher`, e o `default_output` padrão passa a ser `/data/{source_id}`.
+
+### Corrigido
+
+- `check`: o probe `GET Range` tenta de novo (com backoff) em 429/5xx e em erros de transporte (`DEFAULT_RETRY_EXCEPTIONS`: timeout, reset de conexão) antes de desistir. Antes, uma falha transitória virava 404 e a URL saía como indisponível. Quando o probe desiste, a causa é logada em `WARNING` (`[check] probe Range falhou para <url> após N tentativa(s): ...`), e não mais em `DEBUG`.
+- `check`: uma exceção inesperada em uma entrada (por exemplo, `OSError` ao montar o caminho local) não aborta mais o `check` inteiro com código 1. A entrada é verificada de novo uma vez e, se falhar outra vez, vira `download` com `reason=check-error: ...`, e o plano segue cobrindo as demais.
+
 ## [0.11.0] - 2026-10-14
 
 ### Adicionado
@@ -35,7 +59,7 @@ e este projeto adere ao [Semantic Versioning](https://semver.org/lang/pt-BR/).
 ### Corrigido
 
 - `sdk`: `check_entry` considera `local.exists() or sidecar.exists()`, preservando
-  o status de `manifest-fresh` em ambientes de ingestão efêmera sem forçar
+  o status de `manifest-fresh` em ambientes sem retenção local dos arquivos sem forçar
   re-download redundante quando o dado bruto já foi espelhado.
 
 ## [0.10.0] - 2026-10-07
@@ -60,7 +84,7 @@ e este projeto adere ao [Semantic Versioning](https://semver.org/lang/pt-BR/).
 
 ## [0.9.1] - 2026-10-04
 
-Correção de escape inválido introduzido na v0.8.0 (Onda A.2 do SDK).
+Correção de escape inválido introduzido na v0.8.0 (extensões do SDK).
 
 ### Corrigido
 - `SyntaxWarning: invalid escape sequence '\['` nas mensagens de `convert`,
@@ -84,7 +108,7 @@ fetcher (wheel no GitHub Releases, entrada no índice PEP 503 do ecossistema).
 
 ## [0.8.0] - 2026-10-03
 
-Onda A.2 do plano `2026-10-03-padronizacao-core-e-consolidacao-fetchers` —
+Padronização do SDK e consolidação dos fetchers —
 decorators de ciclo de vida no SDK (`quantilica.cli.sdk`) e abstração de
 pré-visualização tabular de sincronização.
 
@@ -115,7 +139,7 @@ pré-visualização tabular de sincronização.
 
 ## [0.7.0] - 2026-10-02
 
-Onda 2 — extensões do SDK (`quantilica.cli.sdk`) para eliminação de
+Extensões do SDK (`quantilica.cli.sdk`) para eliminação de
 boilerplate nos fetchers (decisão
 `2026-10-02-padronizacao-e-deduplicacao-fetchers`).
 
@@ -149,7 +173,7 @@ boilerplate nos fetchers (decisão
 ## [0.3.1] - 2026-08-22
 
 ### Corrigido
-- `DEFAULT_INDEX_URL` apontado para `https://index.quantilica.com/simple/` — a URL anterior (`quantilica.com/quantilica-index/`) parou de ser servida quando o domínio passou ao portal, quebrando o `quantilica install` para fetchers fora do PyPI legado (detalhes no ADR de distribuição de 2026-08-22).
+- `DEFAULT_INDEX_URL` apontado para `https://index.quantilica.com/simple/` — a URL anterior (`quantilica.com/quantilica-index/`) parou de ser servida quando o domínio passou ao portal, quebrando o `quantilica install` para fetchers fora do PyPI legado (detalhes na decisão de distribuição de 2026-08-22).
 - `install`/`uninstall` agora mesclam o registro remoto (`sources.json`) com o registro local, resolvendo também nomes canônicos do índice (ex.: `tesouro-direto`, além de `td`).
 
 ## [0.3.0] - 2026-08-10

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: quantilica-cli
-Version: 0.11.0
+Version: 0.13.0
 Summary: Unified CLI for Quantilica open data fetchers
 Author-email: "Komesu, D.K." <daniel@dkko.me>
 License-Expression: MIT

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "quantilica-cli"
-version = "0.11.0"
+version = "0.13.0"
 description = "Unified CLI for Quantilica open data fetchers"
 readme = "README.md"
 authors = [{ name = "Komesu, D.K.", email = "daniel@dkko.me" }]

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/src/quantilica/cli/sdk.py

@@ -9,9 +9,11 @@ import concurrent.futures
 import contextlib
 import datetime as dt
 import email.utils
+import inspect
 import json
 import os
 import threading
+import time
 from collections.abc import Callable, Iterable
 from dataclasses import dataclass, field
 from importlib.metadata import entry_points
@@ -22,6 +24,7 @@ import typer
 from quantilica.core.exceptions import FetchError
 from quantilica.core.ftp import FtpClient
 from quantilica.core.http import (
+    DEFAULT_RETRY_EXCEPTIONS,
     HttpClient,
     HttpStatusError,
     ProgressCallback,
@@ -37,6 +40,7 @@ from quantilica.core.storage import (
 )
 from rich.console import Console, Group
 from rich.live import Live
+from rich.markup import escape
 from rich.rule import Rule
 from rich.table import Table
 
@@ -165,6 +169,13 @@ def _short_remote(item: CheckPlanItem) -> str:
     return remote
 
 
+# Probe Range: novas tentativas em status ou erro de transporte transitório
+# (``DEFAULT_RETRY_EXCEPTIONS``) antes de desistir.
+_PROBE_RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
+_PROBE_ATTEMPTS = 3
+_PROBE_BACKOFF_S = 1.0
+
+
 class _ProbeNotFoundError(Exception):
     """O GET de probe Range respondeu 404 (URL removida no servidor)."""
 
@@ -513,6 +524,9 @@ class FetcherApp:
             :func:`default_path_builder` (canonical ``DataRepository`` layout).
         default_output: Default output directory path.
         client: HTTP or FTP client instance. Defaults to default_client().
+        source_id: Source ID recorded in download manifests. Defaults to
+            ``name`` without the ``-fetcher`` suffix. When ``default_output``
+            is omitted it is derived as ``/data/{source_id}``.
         build_default_commands: If True (default), registers the built-in
             ``sync`` and ``list`` commands. Set to False to register only
             custom commands via :meth:`attach_command`.
@@ -530,6 +544,7 @@ class FetcherApp:
         path_builder: Callable[[Path, dict[str, Any], dt.date | None], Path]
         | None = None,
         default_output: Path | None = None,
+        source_id: str | None = None,
         client: HttpClient | FtpClient | None = None,
         build_default_commands: bool = True,
     ):
@@ -539,9 +554,10 @@ class FetcherApp:
         self.aliases = aliases_dict
         self.list_datasets = list_datasets
         self.path_builder = path_builder or default_path_builder
-        self.default_output = default_output or Path(
-            f"/data/{name.replace('-fetcher', '')}"
-        )
+        # source_id explícito vence; senão deriva de `name` sem o sufixo
+        # "-fetcher" (comportamento histórico).
+        self.source_id = source_id or name.replace("-fetcher", "")
+        self.default_output = default_output or Path(f"/data/{self.source_id}")
         self.client = client or default_client()
 
         self.all_group_keys = list(self.groups.keys())
@@ -594,24 +610,132 @@ class FetcherApp:
             pass
         return f"quantilica install {key} --extra analysis"
 
-    def command_convert(
-        self, func: Callable[[Path, Path], Any]
-    ) -> Callable[[Path, Path], Any]:
-        """Register the standard ``convert`` subcommand.
+    def _default_lake(self) -> Path:
+        """Raiz padrão do lake: ``lake_root()`` do core, se disponível.
+
+        ``quantilica.core.lake`` só existe a partir de versões recentes do core;
+        sem ele, cai no ``default_output`` (comportamento anterior).
+        """
+        try:
+            from quantilica.core.lake import lake_root
+        except ImportError:  # pragma: no cover - core sem o módulo lake
+            return self.default_output
+        return lake_root()
+
+    def _run_convert(
+        self,
+        func: Callable[..., Any],
+        input: Path,
+        lake: Path,
+        *,
+        dataset: list[str] | None = None,
+        force: bool = False,
+    ) -> Any:
+        """Chama ``func`` com os argumentos que ela aceita.
+
+        Conversores legados têm a forma ``func(input, lake)``. ``dataset`` e
+        ``force`` só são repassados (como keyword) quando a assinatura os aceita
+        ou tem ``**kwargs``. Se o usuário pediu um deles e ``func`` não o
+        aceita, sai com código 2 em vez de ignorar a opção em silêncio.
+        """
+        try:
+            params = inspect.signature(func).parameters
+        except (TypeError, ValueError):  # pragma: no cover - builtins/C callables
+            return func(input, lake)
+        var_kw = any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values())
+        wanted = {"dataset": dataset or None, "force": force or None}
+        kwargs: dict[str, Any] = {}
+        for name, value in wanted.items():
+            if name in params or var_kw:
+                kwargs[name] = dataset if name == "dataset" else force
+            elif value is not None:
+                get_console().print(
+                    f"[red]Erro:[/red] este conversor não suporta --{name}."
+                )
+                raise typer.Exit(2)
+        return func(input, lake, **kwargs)
+
+    @staticmethod
+    def _convert_counts(result: Any) -> dict[str, Any] | None:
+        """Extrai ``converted``/``skipped``/``errors`` do retorno de um conversor.
 
-        The generated command exposes the canonical flags (``-i/--input``,
-        ``-o/--output``, ``--verbose``) and delegates to ``func(input, output)``.
-        An ``ImportError`` propagating from ``func`` is treated as missing
-        analytical extras and exits gracefully with code 1.
+        Aceita um objeto com esses atributos ou um dicionário de contagens.
+        Valores podem ser inteiros ou coleções (usa-se o tamanho nas contagens).
+
+        Returns:
+            ``{"converted": int, "skipped": int, "errors": int,
+            "error_items": list}`` ou ``None`` se o retorno não tiver o formato.
+        """
+        keys = ("converted", "skipped", "errors")
+        if isinstance(result, dict):
+            raw = {k: result.get(k) for k in keys}
+            present = any(k in result for k in keys)
+        else:
+            raw = {k: getattr(result, k, None) for k in keys}
+            present = any(hasattr(result, k) for k in keys)
+        if not present:
+            return None
+
+        def count(value: Any) -> int:
+            if value is None:
+                return 0
+            if isinstance(value, int | float):
+                return int(value)
+            return len(value)
+
+        errors = raw["errors"]
+        items = (
+            list(errors)
+            if errors is not None and not isinstance(errors, int | float)
+            else []
+        )
+        return {
+            "converted": count(raw["converted"]),
+            "skipped": count(raw["skipped"]),
+            "errors": count(errors),
+            "error_items": items,
+        }
+
+    def _report_convert(self, result: Any, console: Console) -> bool:
+        """Imprime o sumário da conversão. Retorna ``True`` se houve erros."""
+        counts = self._convert_counts(result)
+        if counts is None:
+            return False
+        table = Table(show_header=False, box=None, pad_edge=False)
+        table.add_row("[green]convertidos[/green]", str(counts["converted"]))
+        table.add_row("[dim]pulados[/dim]", str(counts["skipped"]))
+        table.add_row(
+            "[red]erros[/red]" if counts["errors"] else "[dim]erros[/dim]",
+            str(counts["errors"]),
+        )
+        console.print(table)
+        for item in counts["error_items"]:
+            console.print(f"  [red]✗[/red] {escape(str(item))}")
+        return counts["errors"] > 0
+
+    def command_convert(self, func: Callable[..., Any]) -> Callable[..., Any]:
+        """Register the standard ``convert`` subcommand (raw → lake).
+
+        The generated command exposes ``-i/--input``, ``--lake`` (aliases
+        ``-o``/``--output``; default ``lake_root()``), ``--dataset`` (repeatable),
+        ``--force`` and ``--verbose``. ``func`` is called as
+        ``func(input, lake, dataset=..., force=...)``; the keywords are only
+        passed when its signature accepts them, so legacy ``func(input, output)``
+        converters keep working.
+
+        If ``func`` returns an object (or dict) with ``converted``, ``skipped``
+        and ``errors``, a summary is rendered and the command exits with code 1
+        when there are errors. An ``ImportError`` propagating from ``func`` is
+        treated as missing analytical extras and exits with code 1.
 
         Args:
-            func: Callable receiving ``(input, output)`` paths and performing
-                the conversion.
+            func: Callable performing the conversion.
 
         Returns:
             The original ``func`` (the method can be used as a decorator).
         """
         console = get_console()
+        default_lake = self._default_lake()
 
         @self.app.command("convert")
         def convert(
@@ -623,26 +747,47 @@ class FetcherApp:
                     help="Diretório de origem com arquivos brutos",
                 ),
             ] = self.default_output,
-            output: Annotated[
+            lake: Annotated[
                 Path,
-                typer.Option("-o", "--output", help="Diretório de destino"),
-            ] = self.default_output,
+                typer.Option(
+                    "--lake",
+                    "-o",
+                    "--output",
+                    help="Raiz do lake de destino (padrão: QUANTILICA_LAKE_DIR)",
+                ),
+            ] = default_lake,
+            dataset: Annotated[
+                list[str] | None,
+                typer.Option(
+                    "--dataset", help="Filtrar dataset(s) específico(s) a converter"
+                ),
+            ] = None,
+            force: Annotated[
+                bool,
+                typer.Option(
+                    "--force", help="Forçar reconversão mesmo se sha256 coincidir"
+                ),
+            ] = False,
             verbose: Annotated[
                 bool, typer.Option("--verbose", help="Logs detalhados")
             ] = False,
         ) -> None:
             setup_rich_logging(verbose, console=console)
             try:
-                func(input, output)
+                result = self._run_convert(
+                    func, input, lake, dataset=dataset, force=force
+                )
             except ImportError:
                 console.print(
                     f"[red]Erro:[/red] convert requer extras de análise: "
                     f"{self._analysis_hint()}"
                 )
                 raise typer.Exit(1) from None
-            console.print(
-                f"[green]✓[/green] Conversão concluída em [dim]{output}[/dim]."
-            )
+            failed = self._report_convert(result, console)
+            if failed:
+                console.print("[red]✗[/red] Conversão concluída com erros.")
+                raise typer.Exit(1)
+            console.print(f"[green]✓[/green] Conversão concluída em [dim]{lake}[/dim].")
 
         return func
 
@@ -697,26 +842,27 @@ class FetcherApp:
 
         return func
 
-    def command_pipeline(
-        self, convert_func: Callable[[Path, Path], Any]
-    ) -> Callable[[Path, Path], Any]:
+    def command_pipeline(self, convert_func: Callable[..., Any]) -> Callable[..., Any]:
         """Register the standard ``pipeline`` subcommand (sync → convert).
 
         Step 1/2 invokes the built-in ``sync`` command (same flags: groups,
-        ``-o/--output``, ``--parquet-dir``, ``--workers``, ``--dry-run``,
-        ``--verbose``). Step 2/2 invokes ``convert_func(output, parquet_dir)``.
+        ``-o/--output``, ``--workers``, ``--dry-run``, ``--verbose``) and step
+        2/2 invokes ``convert_func(output, lake)`` as in :meth:`command_convert`
+        (``--lake`` defaults to ``lake_root()``; ``--parquet-dir`` is kept as an
+        alias for backwards compatibility).
         With ``--dry-run`` the plan is only listed and the conversion step is
         skipped. An ``ImportError`` from ``convert_func`` is treated as
         missing analytical extras.
 
         Args:
-            convert_func: Callable receiving ``(output, parquet_dir)`` paths
-                and performing the conversion.
+            convert_func: Callable receiving ``(output, lake)`` paths and
+                performing the conversion.
 
         Returns:
             The original ``convert_func`` (usable as a decorator).
         """
         console = get_console()
+        default_lake = self._default_lake()
 
         @self.app.command("pipeline")
         def pipeline(
@@ -732,11 +878,12 @@ class FetcherApp:
                 Path | None,
                 typer.Option("-o", "--output", help="Diretório de saída"),
             ] = None,
-            parquet_dir: Annotated[
+            lake: Annotated[
                 Path | None,
                 typer.Option(
+                    "--lake",
                     "--parquet-dir",
-                    help="Diretório para os Parquet (padrão: igual a --output)",
+                    help="Raiz do lake de destino (padrão: QUANTILICA_LAKE_DIR)",
                 ),
             ] = None,
             workers: Annotated[
@@ -751,7 +898,7 @@ class FetcherApp:
         ) -> None:
             setup_rich_logging(verbose, console=console)
             actual_output = output or self.default_output
-            parquet_out = parquet_dir or actual_output
+            lake_out = lake or default_lake
 
             console.print(Rule("[bold]Passo 1/2: Download[/bold]"))
             sync_cmd = next(
@@ -785,16 +932,18 @@ class FetcherApp:
 
             console.print(Rule("[bold]Passo 2/2: Conversão[/bold]"))
             try:
-                convert_func(actual_output, parquet_out)
+                result = self._run_convert(convert_func, actual_output, lake_out)
             except ImportError:
                 console.print(
                     f"[red]Erro:[/red] pipeline (conversão) requer extras de "
                     f"análise: {self._analysis_hint()}"
                 )
                 raise typer.Exit(1) from None
+            if self._report_convert(result, console):
+                console.print("[red]✗[/red] Pipeline concluído com erros na conversão.")
+                raise typer.Exit(1)
             console.print(
-                f"[green]✓[/green] Pipeline concluído: Parquet em "
-                f"[dim]{parquet_out}[/dim]."
+                f"[green]✓[/green] Pipeline concluído: lake em [dim]{lake_out}[/dim]."
             )
 
         return convert_func
@@ -819,7 +968,7 @@ class FetcherApp:
         return self.client.download_with_manifest(
             url,
             output,
-            source_id=self.name.replace("-fetcher", ""),
+            source_id=self.source_id,
             dataset_id=dataset_id,
             producer=self.name,
             progress=progress,
@@ -1146,24 +1295,41 @@ class FetcherApp:
         stream = getattr(self.client, "stream", None)
         if not callable(stream):
             return None
-        try:
-            with stream(
-                "GET",
+        for attempt in range(1, _PROBE_ATTEMPTS + 1):
+            try:
+                with stream(
+                    "GET",
+                    url,
+                    # identity: Content-Length/Content-Range devem refletir o
+                    # tamanho real, não o comprimido.
+                    headers={"Range": "bytes=0-0", "Accept-Encoding": "identity"},
+                ) as response:
+                    headers = response.headers
+                    status = response.status_code
+                    final_url = str(response.url)
+                break
+            except HttpStatusError as exc:
+                if exc.status_code == 404:
+                    raise _ProbeNotFoundError(url) from exc
+                failure: Exception = exc
+                retryable = exc.status_code in _PROBE_RETRY_STATUSES
+            except DEFAULT_RETRY_EXCEPTIONS as exc:
+                failure, retryable = exc, True
+            except Exception as exc:  # noqa: BLE001 - probe é best effort
+                failure, retryable = exc, False
+            # 429/5xx ou erro de transporte transitório (ex.: 503 intermitente
+            # do gov.br): sem nova tentativa, uma URL removida viraria
+            # "download" e falharia com 404 no sync.
+            if retryable and attempt < _PROBE_ATTEMPTS:
+                time.sleep(_PROBE_BACKOFF_S * attempt)
+                continue
+            # WARNING: o cron não roda com --verbose e precisa registrar a causa.
+            logger.warning(
+                "[check] probe Range falhou para %s após %d tentativa(s): %r",
                 url,
-                # identity: Content-Length/Content-Range devem refletir o
-                # tamanho real, não o comprimido.
-                headers={"Range": "bytes=0-0", "Accept-Encoding": "identity"},
-            ) as response:
-                headers = response.headers
-                status = response.status_code
-                final_url = str(response.url)
-        except HttpStatusError as exc:
-            if exc.status_code == 404:
-                raise _ProbeNotFoundError(url) from exc
-            logger.debug("[check] probe Range falhou para %s: %r", url, exc)
-            return None
-        except Exception as exc:  # noqa: BLE001 - probe é best effort
-            logger.debug("[check] probe Range falhou para %s: %r", url, exc)
+                attempt,
+                failure,
+            )
             return None
 
         total: int | None = None
@@ -1208,6 +1374,45 @@ class FetcherApp:
             entry=dict(entry),
         )
 
+    def _check_entry_isolated(
+        self, entry: dict[str, Any], output_dir: Path
+    ) -> CheckPlanItem:
+        """Run :meth:`check_entry` without letting one entry abort the check.
+
+        An unexpected error (e.g. ``OSError`` while building the local path)
+        is retried once; if it persists the entry becomes a conservative
+        ``download`` verdict, so the plan still covers every other entry
+        instead of the whole ``check`` exiting 1.
+        """
+        for attempt in (1, 2):
+            try:
+                return self.check_entry(entry, output_dir)
+            except Exception as exc:  # noqa: BLE001 - isolate per entry
+                failure = exc
+                if attempt == 1:
+                    time.sleep(1.0)
+        eid = str(entry.get("id", "unknown"))
+        logger.warning(
+            "[check] falha ao verificar %s (%s: %s); marcada para download",
+            eid,
+            type(failure).__name__,
+            failure,
+        )
+        return CheckPlanItem(
+            dataset=str(entry.get("group") or entry.get("id") or "datasets"),
+            id=eid,
+            url=str(entry.get("url", "")),
+            partition=_partition_label(entry),
+            local_path="",
+            remote_etag=None,
+            remote_last_modified=None,
+            remote_size=None,
+            local_exists=False,
+            action="download",
+            reason=f"check-error: {type(failure).__name__}: {failure}",
+            entry=dict(entry),
+        )
+
     def check_datasets(
         self,
         entries: list[dict[str, Any]],
@@ -1227,7 +1432,7 @@ class FetcherApp:
         results: list[CheckPlanItem | None] = [None] * len(entries)
 
         def _worker(index: int, entry: dict[str, Any]) -> None:
-            results[index] = self.check_entry(entry, output_dir)
+            results[index] = self._check_entry_isolated(entry, output_dir)
             if (index + 1) % 100 == 0 or index + 1 == len(entries):
                 logger.info("[check] %d/%d verificados", index + 1, len(entries))
 

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/src/quantilica/cli/sources.py

@@ -24,6 +24,7 @@ app = typer.Typer(
 console = Console()
 
 FETCHER_GROUP = "quantilica.fetchers"
+LAKE_GROUP = "quantilica.lake"
 DEFAULT_INDEX_URL = "https://index.quantilica.com/simple/"
 
 # Mapeamento estático padrão de comandos CLI para nome das distribuições
@@ -31,15 +32,22 @@ SOURCES_REGISTRY: dict[str, str] = {
     "anac": "anac-fetcher",
     "anp": "anp-fetcher",
     "b3": "b3-fetcher",
+    "bcb": "bcb-sgs-fetcher",
     "bcb-sgs": "bcb-sgs-fetcher",
     "comex": "comex-fetcher",
     "cvm": "cvm-fetcher",
     "datasus": "datasus-fetcher",
+    "inep": "inep-fetcher",
     "inmet": "inmet-fetcher",
     "pdet": "pdet-fetcher",
+    "rfb": "rfb-cnpj-fetcher",
+    "rfb-cnpj": "rfb-cnpj-fetcher",
     "rtn": "rtn-fetcher",
     "sidra": "sidra-fetcher",
+    "sql": "quantilica-sql",
     "td": "tesouro-direto-fetcher",
+    "tesouro-direto": "tesouro-direto-fetcher",
+    "tesouro_direto": "tesouro-direto-fetcher",
     "tse": "tse-fetcher",
 }
 
@@ -115,6 +123,34 @@ def get_installed_entry_points() -> set[str]:
     return {ep.name for ep in eps}
 
 
+def get_lake_entry_points() -> list[tuple[str, str | None]]:
+    """Lista os conversores do lake (entry points ``quantilica.lake``) instalados.
+
+    Não carrega os entry points: importar o conversor exige os extras de
+    análise, e o ``doctor`` só precisa saber se ele está registrado.
+
+    Returns:
+        Pares ``(nome do entry point, nome da distribuição ou None)``.
+    """
+    return [
+        (ep.name, ep.dist.name if ep.dist is not None else None)
+        for ep in entry_points(group=LAKE_GROUP)
+    ]
+
+
+def has_lake_converter(
+    source: str, lake_eps: list[tuple[str, str | None]], dist: str | None = None
+) -> bool:
+    """Indica se a fonte tem conversor do lake registrado.
+
+    Casa pelo nome do entry point (``source_id``) ou pela distribuição da fonte.
+    """
+    dists = {d for _, d in lake_eps if d}
+    return any(name == source for name, _ in lake_eps) or (
+        dist is not None and dist in dists
+    )
+
+
 def get_index_url() -> str:
     """Retorna a URL do índice pip customizado ou default.
 
@@ -361,7 +397,10 @@ def cmd_uninstall(
 
 @app.command("doctor")
 def cmd_doctor() -> None:
-    """Verifica inconsistências entre o estado registrado e os plugins instalados."""
+    """Verifica inconsistências entre o estado registrado e os plugins instalados.
+
+    Também informa, por fonte, se há conversor do lake (``quantilica.lake``).
+    """
     state = load_state()
     registered = state.get("installed", {})
     installed_eps = get_installed_entry_points()
@@ -375,18 +414,33 @@ def cmd_doctor() -> None:
         console.print(
             "[green]✓ Todas as fontes registradas estão instaladas e ativas.[/green]"
         )
-        return
+    else:
+        console.print(
+            f"[yellow]Encontradas {len(missing)} fonte(s) registrada(s) mas ausente(s) no ambiente:[/yellow]"
+        )
+        for name, dist in missing.items():
+            console.print(f" - {name} ({dist})")
 
-    console.print(
-        f"[yellow]Encontradas {len(missing)} fonte(s) registrada(s) mas ausente(s) no ambiente:[/yellow]"
-    )
-    for name, dist in missing.items():
-        console.print(f" - {name} ({dist})")
-
-    for name, dist in missing.items():
-        console.print(f"[cyan]Reinstalando {name} ({dist})...[/cyan]")
-        try:
-            install_package(dist)
-            console.print(f"[green]✓ {name} recuperado.[/green]")
-        except Exception as exc:
-            console.print(f"[red]Falha ao recuperar {name}:[/red] {exc}")
+        for name, dist in missing.items():
+            console.print(f"[cyan]Reinstalando {name} ({dist})...[/cyan]")
+            try:
+                install_package(dist)
+                console.print(f"[green]✓ {name} recuperado.[/green]")
+            except Exception as exc:
+                console.print(f"[red]Falha ao recuperar {name}:[/red] {exc}")
+
+    _print_lake_status(installed_eps | set(registered), registered)
+
+
+def _print_lake_status(sources: set[str], registered: dict[str, str]) -> None:
+    """Imprime, por fonte instalada/registrada, se há conversor do lake."""
+    if not sources:
+        return
+    lake_eps = get_lake_entry_points()
+    console.print("\n[bold]Lake (conversor quantilica.lake):[/bold]")
+    for name in sorted(sources):
+        dist = registered.get(name) or SOURCES_REGISTRY.get(name)
+        if has_lake_converter(name, lake_eps, dist):
+            console.print(f" - {name}: [green]✓[/green] lake: sim")
+        else:
+            console.print(f" - {name}: [dim]lake: não[/dim]")

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/tests/test_sdk.py

@@ -497,3 +497,225 @@ def test_check_entry_ephemeral_stale_downloads(tmp_path: Path):
     item = app.check_entry(entry, tmp_path)
     assert item.action == "download"
     assert item.reason == "remote-newer"
+
+
+def test_source_id_defaults_to_name_without_suffix():
+    app = make_app()
+    assert app.source_id == "comex"
+    assert app.default_output == Path("/data/comex")
+
+
+def test_explicit_source_id_derives_default_output():
+    app = make_app(source_id="custom")
+    assert app.source_id == "custom"
+    assert app.default_output == Path("/data/custom")
+
+
+def test_explicit_default_output_wins_over_source_id(tmp_path: Path):
+    app = make_app(source_id="custom", default_output=tmp_path)
+    assert app.default_output == tmp_path
+
+
+def test_check_datasets_isolates_entry_failure(tmp_path: Path, monkeypatch):
+    """Uma entrada que levanta (ex.: OSError) não derruba o check inteiro."""
+    monkeypatch.setattr("quantilica.cli.sdk.time.sleep", lambda _s: None)
+    app = make_app(
+        client=HttpClient(
+            attempts=1,
+            transport=httpx2.MockTransport(lambda r: httpx2.Response(200)),
+        )
+    )
+    entries = sample_list_datasets("exp")
+    calls: dict[str, int] = {}
+    real = app.check_entry
+
+    def flaky(entry, output_dir):
+        key = str(entry["id"])
+        calls[key] = calls.get(key, 0) + 1
+        if key == str(entries[0]["id"]):
+            raise FileExistsError(17, "File exists", "/data")
+        return real(entry, output_dir)
+
+    monkeypatch.setattr(app, "check_entry", flaky)
+    plan = app.check_datasets(entries, tmp_path)
+    assert len(plan.items) == len(entries)
+    bad = next(i for i in plan.items if i.id == str(entries[0]["id"]))
+    assert bad.action == "download"
+    assert bad.reason.startswith("check-error: FileExistsError")
+    assert calls[str(entries[0]["id"])] == 2  # uma nova tentativa
+
+
+def test_check_datasets_retry_recovers_transient_failure(tmp_path: Path, monkeypatch):
+    monkeypatch.setattr("quantilica.cli.sdk.time.sleep", lambda _s: None)
+    app = make_app(
+        client=HttpClient(
+            attempts=1,
+            transport=httpx2.MockTransport(lambda r: httpx2.Response(200)),
+        )
+    )
+    entries = sample_list_datasets("exp")[:1]
+    real = app.check_entry
+    state = {"n": 0}
+
+    def once(entry, output_dir):
+        state["n"] += 1
+        if state["n"] == 1:
+            raise OSError("transiente")
+        return real(entry, output_dir)
+
+    monkeypatch.setattr(app, "check_entry", once)
+    plan = app.check_datasets(entries, tmp_path)
+    assert plan.items[0].reason != "" and not plan.items[0].reason.startswith(
+        "check-error"
+    )
+
+
+def test_check_entry_403_head_without_probe_support_downloads(tmp_path: Path):
+    """HEAD 403 e probe Range 403 → download (nunca exceção/exit 1)."""
+    app = make_app(
+        client=HttpClient(
+            attempts=1,
+            transport=httpx2.MockTransport(lambda r: httpx2.Response(403)),
+        )
+    )
+    item = app.check_entry(sample_list_datasets("exp")[0], tmp_path)
+    assert item.action == "download"
+
+
+# ---------------------------------------------------------------
+# command_convert / command_pipeline: Lake Hive
+# ---------------------------------------------------------------
+
+
+def test_convert_lake_flag_and_output_alias(tmp_path: Path):
+    app = make_app()
+    calls: list[tuple[Path, Path]] = []
+
+    @app.command_convert
+    def conv(input: Path, lake: Path) -> None:
+        calls.append((input, lake))
+
+    for flag in ("--lake", "-o", "--output"):
+        result = runner.invoke(
+            _grouped(app.app),
+            ["fd", "convert", "-i", str(tmp_path), flag, str(tmp_path / "l")],
+        )
+        assert result.exit_code == 0, result.output
+    assert calls == [(tmp_path, tmp_path / "l")] * 3
+
+
+def test_convert_passes_dataset_and_force_when_accepted(tmp_path: Path):
+    app = make_app()
+    seen: dict = {}
+
+    @app.command_convert
+    def conv(input: Path, lake: Path, dataset=None, force=False) -> None:
+        seen.update(dataset=dataset, force=force)
+
+    result = runner.invoke(
+        _grouped(app.app),
+        ["fd", "convert", "--dataset", "a", "--dataset", "b", "--force"],
+    )
+    assert result.exit_code == 0, result.output
+    assert seen == {"dataset": ["a", "b"], "force": True}
+
+
+def test_convert_kwargs_func_receives_dataset_and_force():
+    app = make_app()
+    seen: dict = {}
+
+    @app.command_convert
+    def conv(input: Path, lake: Path, **kwargs) -> None:
+        seen.update(kwargs)
+
+    result = runner.invoke(_grouped(app.app), ["fd", "convert"])
+    assert result.exit_code == 0, result.output
+    assert seen == {"dataset": None, "force": False}
+
+
+def test_convert_legacy_two_arg_func_ignores_unused_defaults():
+    app = make_app()
+    calls: list[tuple[Path, Path]] = []
+
+    @app.command_convert
+    def conv(input: Path, lake: Path) -> None:
+        calls.append((input, lake))
+
+    result = runner.invoke(_grouped(app.app), ["fd", "convert"])
+    assert result.exit_code == 0, result.output
+    assert len(calls) == 1
+
+
+@pytest.mark.parametrize("flags", [["--dataset", "x"], ["--force"]])
+def test_convert_legacy_func_rejects_requested_unsupported_flag(flags: list[str]):
+    app = make_app()
+
+    @app.command_convert
+    def conv(input: Path, lake: Path) -> None: ...
+
+    result = runner.invoke(_grouped(app.app), ["fd", "convert", *flags])
+    assert result.exit_code == 2
+    assert "não suporta" in result.output
+
+
+def test_convert_summary_from_object_and_exit_0_without_errors():
+    from types import SimpleNamespace
+
+    app = make_app()
+
+    @app.command_convert
+    def conv(input: Path, lake: Path):
+        return SimpleNamespace(converted=3, skipped=2, errors=0)
+
+    result = runner.invoke(_grouped(app.app), ["fd", "convert"])
+    assert result.exit_code == 0, result.output
+    assert "convertidos" in result.output
+    assert "pulados" in result.output
+    assert "Conversão concluída" in result.output
+
+
+def test_convert_summary_from_dict_with_errors_exits_1():
+    app = make_app()
+
+    @app.command_convert
+    def conv(input: Path, lake: Path):
+        return {"converted": ["a"], "skipped": [], "errors": ["boom [x]"]}
+
+    result = runner.invoke(_grouped(app.app), ["fd", "convert"])
+    assert result.exit_code == 1
+    assert "boom [x]" in result.output
+    assert "com erros" in result.output
+
+
+def test_convert_unrelated_return_value_is_ignored():
+    app = make_app()
+
+    @app.command_convert
+    def conv(input: Path, lake: Path):
+        return 42
+
+    result = runner.invoke(_grouped(app.app), ["fd", "convert"])
+    assert result.exit_code == 0, result.output
+    assert "convertidos" not in result.output
+
+
+def test_pipeline_lake_flag_and_parquet_dir_alias(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+):
+    app = make_app()
+    monkeypatch.setattr(
+        app, "download_datasets", lambda entries, output_dir, workers=4: (0, 0, [])
+    )
+    calls: list[tuple[Path, Path]] = []
+
+    @app.command_pipeline
+    def conv(input: Path, lake: Path) -> None:
+        calls.append((input, lake))
+
+    for flag in ("--lake", "--parquet-dir"):
+        result = runner.invoke(
+            _grouped(app.app),
+            ["fd", "pipeline", "exp", "-o", str(tmp_path), flag, str(tmp_path / "l")],
+        )
+        assert result.exit_code == 0, result.output
+    assert calls == [(tmp_path, tmp_path / "l")] * 2

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/tests/test_sdk_force_probe.py

@@ -485,3 +485,92 @@ def test_head_403_probe_500_stays_download(tmp_path: Path):
     item = _probe_app(handler).check_entry(ENTRY, tmp_path)
     assert item.action == "download"
     assert item.reason.startswith("metadata-unavailable")
+
+
+@pytest.mark.parametrize(
+    ("statuses", "action", "reason"),
+    [
+        # 503 transitório e depois 404 → removida de fato
+        ([503, 404], "skip-unavailable", "remote-404"),
+        # 429 e depois 206 → metadados pelo Range
+        ([429, 206], "download", "range-probe"),
+        # 503 persistente → desiste após as tentativas
+        ([503, 503, 503], "download", "metadata-unavailable"),
+        # 403 no GET não é transitório → sem nova tentativa
+        ([403], "download", "metadata-unavailable"),
+    ],
+)
+def test_probe_retries_transient_status(
+    tmp_path: Path, monkeypatch, statuses, action, reason
+):
+    from quantilica.cli import sdk
+
+    monkeypatch.setattr(sdk, "_PROBE_BACKOFF_S", 0.0)
+    gets: list[int] = []
+
+    def handler(request):
+        if request.method == "HEAD":
+            return httpx2.Response(403)
+        status = statuses[min(len(gets), len(statuses) - 1)]
+        gets.append(status)
+        headers = {"Content-Range": "bytes 0-0/10"} if status == 206 else {}
+        return httpx2.Response(status, headers=headers)
+
+    item = _probe_app(handler).check_entry(ENTRY, tmp_path)
+    assert item.action == action
+    assert item.reason.startswith(reason)
+    assert gets == statuses
+
+
+@pytest.mark.parametrize(
+    ("failures", "action", "reason"),
+    [
+        # timeout transitório e depois 404 → removida de fato
+        ([httpx2.ReadTimeout], "skip-unavailable", "remote-404"),
+        # conexão derrubada e depois 206 → metadados pelo Range
+        ([httpx2.RemoteProtocolError], "download", "range-probe"),
+        # erro de transporte persistente → desiste após as tentativas
+        ([httpx2.ConnectError] * 3, "download", "metadata-unavailable"),
+    ],
+)
+def test_probe_retries_transport_errors(
+    tmp_path: Path, monkeypatch, failures, action, reason
+):
+    from quantilica.cli import sdk
+
+    monkeypatch.setattr(sdk, "_PROBE_BACKOFF_S", 0.0)
+    gets: list[int] = []
+
+    def handler(request):
+        if request.method == "HEAD":
+            return httpx2.Response(403)
+        gets.append(1)
+        if len(gets) <= len(failures):
+            raise failures[len(gets) - 1]("transitório", request=request)
+        if reason == "remote-404":
+            return httpx2.Response(404)
+        return httpx2.Response(206, headers={"Content-Range": "bytes 0-0/10"})
+
+    item = _probe_app(handler).check_entry(ENTRY, tmp_path)
+    assert item.action == action
+    assert item.reason.startswith(reason)
+    assert len(gets) == min(len(failures) + 1, 3)
+
+
+def test_probe_failure_logs_warning_with_cause(tmp_path: Path, monkeypatch):
+    from quantilica.cli import sdk
+
+    monkeypatch.setattr(sdk, "_PROBE_BACKOFF_S", 0.0)
+    warnings: list[str] = []
+    monkeypatch.setattr(
+        sdk.logger, "warning", lambda msg, *args: warnings.append(msg % args)
+    )
+
+    def handler(request):
+        return httpx2.Response(403 if request.method == "HEAD" else 503)
+
+    item = _probe_app(handler).check_entry(ENTRY, tmp_path)
+    assert item.reason.startswith("metadata-unavailable")
+    assert len(warnings) == 1
+    assert "após 3 tentativa(s)" in warnings[0]
+    assert "503" in warnings[0]

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/tests/test_sdk_onda_a2.py

@@ -104,7 +104,10 @@ def test_command_convert_success(tmp_path: Path):
     assert "✓" in result.output
 
 
-def test_command_convert_defaults_to_default_output(tmp_path: Path):
+def test_command_convert_defaults_to_lake_root(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+):
+    monkeypatch.setenv("QUANTILICA_LAKE_DIR", str(tmp_path / "lake"))
     app = make_app()
     calls: list[tuple[Path, Path]] = []
 
@@ -112,10 +115,9 @@ def test_command_convert_defaults_to_default_output(tmp_path: Path):
     def conv(input: Path, output: Path) -> None:
         calls.append((input, output))
 
-    default = app.default_output
     result = runner.invoke(_grouped(app.app), ["fd", "convert"])
     assert result.exit_code == 0, result.output
-    assert calls == [(default, default)]
+    assert calls == [(app.default_output, tmp_path / "lake")]
 
 
 def test_command_convert_graceful_import_error(tmp_path: Path):
@@ -235,9 +237,10 @@ def test_command_pipeline_graceful_import_error(
     assert "--extra analysis" in saida
 
 
-def test_command_pipeline_parquet_dir_defaults_to_output(
+def test_command_pipeline_lake_defaults_to_lake_root(
     tmp_path: Path, monkeypatch: pytest.MonkeyPatch
 ):
+    monkeypatch.setenv("QUANTILICA_LAKE_DIR", str(tmp_path / "lake"))
     app = make_app()
     monkeypatch.setattr(
         app, "download_datasets", lambda entries, output_dir, workers=4: (0, 0, [])
@@ -251,7 +254,7 @@ def test_command_pipeline_parquet_dir_defaults_to_output(
     raw = tmp_path / "raw"
     result = runner.invoke(_grouped(app.app), ["fd", "pipeline", "exp", "-o", str(raw)])
     assert result.exit_code == 0, result.output
-    assert calls == [(raw, raw)]
+    assert calls == [(raw, tmp_path / "lake")]
 
 
 def test_command_pipeline_requires_sync_command():

{quantilica_cli-0.11.0 → quantilica_cli-0.13.0}/tests/test_sources.py

@@ -122,6 +122,37 @@ def test_cmd_doctor(temp_config_dir: Path):
         mock_install.assert_called_once_with("rtn-fetcher")
 
 
+def test_cmd_doctor_lists_lake_converters(temp_config_dir: Path):
+    from quantilica.cli.cli import app
+
+    save_state({"installed": {"comex": "comex-fetcher", "rtn": "rtn-fetcher"}})
+    with (
+        patch(
+            "quantilica.cli.sources.get_installed_entry_points",
+            return_value={"comex", "rtn"},
+        ),
+        patch(
+            "quantilica.cli.sources.get_lake_entry_points",
+            return_value=[("comex", "comex-fetcher")],
+        ),
+    ):
+        result = CliRunner().invoke(app, ["doctor"])
+    assert result.exit_code == 0
+    assert "quantilica.lake" in result.output
+    assert "comex: ✓ lake: sim" in result.output
+    assert "rtn: lake: não" in result.output
+
+
+def test_has_lake_converter_matches_by_name_or_dist():
+    from quantilica.cli.sources import has_lake_converter
+
+    eps = [("pdet", "pdet-fetcher")]
+    assert has_lake_converter("pdet", eps)
+    assert has_lake_converter("outro", eps, "pdet-fetcher")
+    assert not has_lake_converter("outro", eps, "x-fetcher")
+    assert not has_lake_converter("outro", [])
+
+
 def test_dist_spec_monta_extras():
     from quantilica.cli.sources import _dist_spec