sci-etl-cli 0.2.0__tar.gz → 0.4.0__tar.gz

sci_etl_cli-0.4.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: sci-etl-cli
+Version: 0.4.0
+Summary: Command-line tool for running sci-etl-core extraction pipelines from a YAML file.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/xueromll/sci-etl-cli
+Project-URL: Documentation, https://xueromll.github.io/sci-etl-core/latest/cli/
+Project-URL: Repository, https://github.com/xueromll/sci-etl-cli
+Project-URL: Issues, https://github.com/xueromll/sci-etl-cli/issues
+Keywords: etl,cli,arxiv,llm,text-mining,scientific-literature
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Text Processing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sci-etl-core[arxiv,async,config,html,llm,pdf,xml]<0.8,>=0.7.0
+Requires-Dist: click>=8.2
+Requires-Dist: rich>=13.7
+Requires-Dist: pydantic>=2
+Requires-Dist: python-dotenv
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"
+Requires-Dist: pytest-mock; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Provides-Extra: lint
+Requires-Dist: ruff; extra == "lint"
+Requires-Dist: mypy; extra == "lint"
+Requires-Dist: types-PyYAML; extra == "lint"
+Provides-Extra: docs
+Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
+Requires-Dist: mkdocs-material<10,>=9.7; extra == "docs"
+Requires-Dist: mkdocs-click<1,>=0.9; extra == "docs"
+Dynamic: license-file
+
+# sci-etl-cli
+
+`sci-etl` runs [sci-etl-core](https://github.com/xueromll/sci-etl-core)
+extraction pipelines from a single YAML file, with no wiring code. It searches
+arXiv, PubMed, Semantic Scholar, or OpenAlex, asks an LLM which papers are
+relevant, extracts structured entities from their full text, and writes them
+to a CSV, one row per entity tagged with the paper it came from. Runs resume where the last one stopped, and `forget` hands
+chosen papers back for another pass.
+
+**Documentation: https://xueromll.github.io/sci-etl-core/latest/cli/**
+
+```text
+$ sci-etl init hot-jupiters
+$ sci-etl validate hot-jupiters/config.yaml
+$ sci-etl search hot-jupiters/config.yaml --limit 5
+$ sci-etl run hot-jupiters/config.yaml --limit 20
+$ sci-etl status hot-jupiters/config.yaml
+```
+
+## Installation
+
+Python 3.11 or newer is required.
+
+```bash
+pip install sci-etl-cli
+```
+
+This installs the `sci-etl` command along with
+`sci-etl-core[config,async,arxiv,html,llm,pdf,xml]`, `click`, and `rich`, so
+every source works out of the box. `pipx install sci-etl-cli` keeps it in an environment of
+its own.
+
+## Quick start
+
+1. `sci-etl init hot-jupiters` writes a working example project.
+2. Copy `.env.example` to `.env` beside the config and set `LLM_API_KEY`.
+3. `sci-etl validate hot-jupiters/config.yaml` checks the config, prompts, key,
+   and packages.
+4. `sci-etl search hot-jupiters/config.yaml` previews matching papers without
+   calling the LLM.
+5. `sci-etl run hot-jupiters/config.yaml` exports entities to `out/planets.csv`.
+   Run it again to continue where it stopped.
+
+## Documentation
+
+| Topic | Where |
+|-------|-------|
+| A first project, step by step | [Quick start](https://xueromll.github.io/sci-etl-core/latest/cli/quick-start/) |
+| Every command and option, stopping, resuming, token usage | [Commands](https://xueromll.github.io/sci-etl-core/latest/cli/commands/) |
+| Every config key and its default | [Config file](https://xueromll.github.io/sci-etl-core/latest/cli/config-file/) |
+| Domain rules written in Python | [Plug-ins](https://xueromll.github.io/sci-etl-core/latest/cli/plugins/) |
+| What each exit code means | [Exit codes](https://xueromll.github.io/sci-etl-core/latest/cli/exit-codes/) |
+
+## Development
+
+```bash
+git clone https://github.com/xueromll/sci-etl-cli.git
+cd sci-etl-cli
+python -m venv .venv
+source .venv/bin/activate        # Windows: .venv\Scripts\activate
+pip install -e ".[dev,lint]"
+pytest --cov=sci_etl_cli --cov-report=term-missing
+ruff check .
+mypy
+```
+
+To work against a local sci-etl-core checkout, install it first with
+`pip install -e "../sci-etl-core[config,async,arxiv,html,llm,pdf,xml]"`.
+
+The suite runs offline: every source is served by an `httpx.MockTransport` and
+the LLM by a scripted client, so `run` is exercised end to end against real CSV
+and state files. Coverage must stay at 100%.
+
+### Documentation
+
+The pages under `docs/` are published as the CLI section of the
+[sci-etl-core site](https://xueromll.github.io/sci-etl-core/), which reads this
+repository's `docs/` folder and the `nav` in `mkdocs.yml`. Preview them on
+their own with:
+
+```bash
+pip install -e ".[docs]"
+mkdocs serve
+```
+
+Link to sci-etl-core pages with absolute URLs, since the pages are also built
+on their own. `mkdocs build --strict` runs in CI. Publishing a release asks
+sci-etl-core to rebuild the site when the `DOCS_DISPATCH_TOKEN` secret is set;
+otherwise run sci-etl-core's Docs workflow by hand.
+
+## License
+
+Released under the MIT License. See [LICENSE](LICENSE).

sci_etl_cli-0.4.0/README.md

@@ -0,0 +1,93 @@
+# sci-etl-cli
+
+`sci-etl` runs [sci-etl-core](https://github.com/xueromll/sci-etl-core)
+extraction pipelines from a single YAML file, with no wiring code. It searches
+arXiv, PubMed, Semantic Scholar, or OpenAlex, asks an LLM which papers are
+relevant, extracts structured entities from their full text, and writes them
+to a CSV, one row per entity tagged with the paper it came from. Runs resume where the last one stopped, and `forget` hands
+chosen papers back for another pass.
+
+**Documentation: https://xueromll.github.io/sci-etl-core/latest/cli/**
+
+```text
+$ sci-etl init hot-jupiters
+$ sci-etl validate hot-jupiters/config.yaml
+$ sci-etl search hot-jupiters/config.yaml --limit 5
+$ sci-etl run hot-jupiters/config.yaml --limit 20
+$ sci-etl status hot-jupiters/config.yaml
+```
+
+## Installation
+
+Python 3.11 or newer is required.
+
+```bash
+pip install sci-etl-cli
+```
+
+This installs the `sci-etl` command along with
+`sci-etl-core[config,async,arxiv,html,llm,pdf,xml]`, `click`, and `rich`, so
+every source works out of the box. `pipx install sci-etl-cli` keeps it in an environment of
+its own.
+
+## Quick start
+
+1. `sci-etl init hot-jupiters` writes a working example project.
+2. Copy `.env.example` to `.env` beside the config and set `LLM_API_KEY`.
+3. `sci-etl validate hot-jupiters/config.yaml` checks the config, prompts, key,
+   and packages.
+4. `sci-etl search hot-jupiters/config.yaml` previews matching papers without
+   calling the LLM.
+5. `sci-etl run hot-jupiters/config.yaml` exports entities to `out/planets.csv`.
+   Run it again to continue where it stopped.
+
+## Documentation
+
+| Topic | Where |
+|-------|-------|
+| A first project, step by step | [Quick start](https://xueromll.github.io/sci-etl-core/latest/cli/quick-start/) |
+| Every command and option, stopping, resuming, token usage | [Commands](https://xueromll.github.io/sci-etl-core/latest/cli/commands/) |
+| Every config key and its default | [Config file](https://xueromll.github.io/sci-etl-core/latest/cli/config-file/) |
+| Domain rules written in Python | [Plug-ins](https://xueromll.github.io/sci-etl-core/latest/cli/plugins/) |
+| What each exit code means | [Exit codes](https://xueromll.github.io/sci-etl-core/latest/cli/exit-codes/) |
+
+## Development
+
+```bash
+git clone https://github.com/xueromll/sci-etl-cli.git
+cd sci-etl-cli
+python -m venv .venv
+source .venv/bin/activate        # Windows: .venv\Scripts\activate
+pip install -e ".[dev,lint]"
+pytest --cov=sci_etl_cli --cov-report=term-missing
+ruff check .
+mypy
+```
+
+To work against a local sci-etl-core checkout, install it first with
+`pip install -e "../sci-etl-core[config,async,arxiv,html,llm,pdf,xml]"`.
+
+The suite runs offline: every source is served by an `httpx.MockTransport` and
+the LLM by a scripted client, so `run` is exercised end to end against real CSV
+and state files. Coverage must stay at 100%.
+
+### Documentation
+
+The pages under `docs/` are published as the CLI section of the
+[sci-etl-core site](https://xueromll.github.io/sci-etl-core/), which reads this
+repository's `docs/` folder and the `nav` in `mkdocs.yml`. Preview them on
+their own with:
+
+```bash
+pip install -e ".[docs]"
+mkdocs serve
+```
+
+Link to sci-etl-core pages with absolute URLs, since the pages are also built
+on their own. `mkdocs build --strict` runs in CI. Publishing a release asks
+sci-etl-core to rebuild the site when the `DOCS_DISPATCH_TOKEN` secret is set;
+otherwise run sci-etl-core's Docs workflow by hand.
+
+## License
+
+Released under the MIT License. See [LICENSE](LICENSE).

{sci_etl_cli-0.2.0 → sci_etl_cli-0.4.0}/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sci-etl-cli"
-version = "0.2.0"
+version = "0.4.0"
 description = "Command-line tool for running sci-etl-core extraction pipelines from a YAML file."
 readme = "README.md"
 license = "MIT"
 license-files = ["LICENSE"]
-requires-python = ">=3.10"
+requires-python = ">=3.11"
 keywords = ["etl", "cli", "arxiv", "llm", "text-mining", "scientific-literature"]
 classifiers = [
     "Development Status :: 3 - Alpha",
@@ -18,7 +18,6 @@ classifiers = [
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3 :: Only",
-    "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
@@ -29,7 +28,7 @@ classifiers = [
 ]
 
 dependencies = [
-    "sci-etl-core[async,llm,pdf]>=0.2.0,<0.3",
+    "sci-etl-core[config,async,arxiv,html,llm,pdf,xml]>=0.7.0,<0.8",
     "click>=8.2",
     "rich>=13.7",
     "pydantic>=2",
@@ -39,12 +38,18 @@ dependencies = [
 [project.optional-dependencies]
 dev = ["pytest", "pytest-asyncio", "pytest-mock", "pytest-cov"]
 lint = ["ruff", "mypy", "types-PyYAML"]
+docs = [
+    "mkdocs>=1.6,<2",
+    "mkdocs-material>=9.7,<10",
+    "mkdocs-click>=0.9,<1",
+]
 
 [project.scripts]
 sci-etl = "sci_etl_cli.app:main"
 
 [project.urls]
 Homepage = "https://github.com/xueromll/sci-etl-cli"
+Documentation = "https://xueromll.github.io/sci-etl-core/latest/cli/"
 Repository = "https://github.com/xueromll/sci-etl-cli"
 Issues = "https://github.com/xueromll/sci-etl-cli/issues"
 
@@ -68,7 +73,7 @@ exclude_also = ["if TYPE_CHECKING:"]
 
 [tool.ruff]
 line-length = 120
-target-version = "py310"
+target-version = "py311"
 src = ["src", "tests"]
 
 [tool.ruff.lint]

{sci_etl_cli-0.2.0 → sci_etl_cli-0.4.0}/src/sci_etl_cli/app.py

@@ -6,6 +6,7 @@ from typing import Any
 import click
 from sci_etl_core.exceptions import SciEtlError
 
+from sci_etl_cli.commands.forget import forget_command
 from sci_etl_cli.commands.init import init_command
 from sci_etl_cli.commands.parse import parse_command
 from sci_etl_cli.commands.run import run_command
@@ -63,6 +64,7 @@ cli.add_command(validate_command)
 cli.add_command(search_command)
 cli.add_command(run_command)
 cli.add_command(status_command)
+cli.add_command(forget_command)
 cli.add_command(parse_command)
 
 

sci_etl_cli-0.4.0/src/sci_etl_cli/assembly.py

@@ -0,0 +1,311 @@
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Callable
+
+from sci_etl_core.exceptions import ConfigurationError
+
+from sci_etl_cli.errors import CliFailure
+
+if TYPE_CHECKING:
+    import httpx
+    from sci_etl_core import (
+        AsyncCsvExporter,
+        AsyncETLPipeline,
+        AsyncOpenAICompatibleClient,
+        AsyncSqliteLLMResponseCache,
+        AsyncSqliteStateManager,
+        AsyncStateManager,
+        PipelineMetadata,
+    )
+    from sci_etl_core.extractors.async_base import AsyncExtractor
+    from sci_etl_core.processors import RecordValidator
+    from sci_etl_core.signals import ShutdownSignal
+
+    from sci_etl_cli.settings import CliConfig
+
+
+@dataclass(frozen=True, slots=True)
+class ProjectParts:
+    """What a run loads from the project folder before touching the network."""
+
+    relevance_prompt: str
+    extraction_prompt: str
+    validators: tuple[RecordValidator, ...]
+
+
+def load_project_parts(config: CliConfig) -> ProjectParts:
+    """Read both prompts and build the plug-ins.
+
+    Raises:
+        ConfigurationError: A prompt file or a plug-in cannot be loaded.
+    """
+    return ProjectParts(
+        relevance_prompt=read_prompt(config.prompts.relevance),
+        extraction_prompt=read_prompt(config.prompts.extraction),
+        validators=tuple(build_validators(config)),
+    )
+
+
+def read_prompt(path: Path) -> str:
+    """Return a prompt file's text.
+
+    Raises:
+        ConfigurationError: The file is missing, unreadable, not UTF-8, or blank.
+    """
+    try:
+        text = path.read_text(encoding="utf-8")
+    except FileNotFoundError as exc:
+        raise ConfigurationError(f"Prompt file not found: {path}") from exc
+    except (OSError, UnicodeDecodeError) as exc:
+        raise ConfigurationError(f"Prompt file could not be read: {path}: {exc}") from exc
+    if not text.strip():
+        raise ConfigurationError(f"Prompt file is empty: {path}")
+    return text
+
+
+def build_validators(config: CliConfig) -> list[RecordValidator]:
+    """Return the ``export.validators`` plug-ins in the order they are listed."""
+    from sci_etl_core.processors import RecordValidator
+
+    from sci_etl_cli.plugins import load_plugin, wrong_type
+
+    validators: list[RecordValidator] = []
+    for reference in config.export.validators:
+        plugin = load_plugin(reference, config.project_root)
+        if not isinstance(plugin, RecordValidator):
+            raise wrong_type(reference, plugin, "RecordValidator")
+        validators.append(plugin)
+    return validators
+
+
+def build_http_client(config: CliConfig) -> httpx.AsyncClient:
+    return config.http.build_client()
+
+
+def build_llm_client(config: CliConfig) -> AsyncOpenAICompatibleClient:
+    from sci_etl_core import AsyncOpenAICompatibleClient
+
+    return AsyncOpenAICompatibleClient.from_config(config.llm)
+
+
+def build_llm_cache(config: CliConfig) -> AsyncSqliteLLMResponseCache | None:
+    """Return the SQLite response cache at ``llm.cache``, or ``None`` when caching is off."""
+    if config.llm.cache is None:
+        return None
+    from sci_etl_core import AsyncSqliteLLMResponseCache
+
+    return AsyncSqliteLLMResponseCache(config.llm.cache)
+
+
+def build_extractor(config: CliConfig, http_client: httpx.AsyncClient) -> AsyncExtractor:
+    """Build the extractor ``source.kind`` names, with the parsers its full text needs.
+
+    Every source takes ``max_retries`` and ``backoff_factor`` from ``http`` and
+    its rate limiter from ``full_text``. arXiv also waits ``pipeline.search_delay``
+    before each listing request and parses LaTeX e-prints and PDFs; PubMed
+    parses PubMed Central's JATS XML; Semantic Scholar and OpenAlex parse the
+    open-access PDF, and every source falls back to the abstract.
+    """
+    source = config.source
+    if source.kind == "arxiv":
+        from sci_etl_core import AsyncArxivExtractor
+        from sci_etl_core.parsers import LatexTarballParser, PdfPlumberParser
+
+        return AsyncArxivExtractor.from_config(
+            config.http,
+            config.pipeline,
+            client=http_client,
+            pdf_parser=PdfPlumberParser(),
+            latex_parser=LatexTarballParser(),
+            full_text=config.full_text,
+        )
+    settings: dict[str, Any] = {
+        "max_retries": config.http.max_retries,
+        "backoff_factor": config.http.backoff_factor,
+        "rate_limiter": config.full_text.build_limiter(),
+        "api_key": source.api_key.get_secret_value() or None,
+    }
+    if source.kind == "pubmed":
+        from sci_etl_core import AsyncPubMedExtractor
+        from sci_etl_core.parsers import JatsXmlParser
+
+        return AsyncPubMedExtractor(
+            http_client,
+            tool=source.tool,
+            email=source.email,
+            sort=source.sort,
+            full_text_parser=JatsXmlParser(),
+            **settings,
+        )
+    from sci_etl_core.parsers import PdfPlumberParser
+
+    if source.kind == "semantic_scholar":
+        from sci_etl_core import AsyncSemanticScholarExtractor
+
+        return AsyncSemanticScholarExtractor(
+            http_client, PdfPlumberParser(), year=source.year, fields_of_study=source.fields_of_study, **settings
+        )
+    from sci_etl_core import AsyncOpenAlexExtractor
+
+    return AsyncOpenAlexExtractor(
+        http_client, PdfPlumberParser(), filter=source.filter, sort=source.sort, mailto=source.mailto, **settings
+    )
+
+
+def build_state_manager(config: CliConfig) -> AsyncStateManager:
+    if config.state.backend == "sqlite":
+        from sci_etl_core import AsyncSqliteStateManager
+
+        return AsyncSqliteStateManager(config.state.database)
+    from sci_etl_core import AsyncFileStateManager
+
+    return AsyncFileStateManager(config.state.processed_ids, config.state.metadata)
+
+
+def build_exporter(config: CliConfig) -> AsyncCsvExporter:
+    """Write one CSV row per entity: ``record_id``, the key and value columns, and ``extra``."""
+    from sci_etl_core import AsyncCsvExporter
+
+    export = config.export
+    return AsyncCsvExporter(
+        export.destination,
+        [export.key_column, *export.value_columns],
+        escape_formulas=export.escape_formulas,
+    )
+
+
+def build_pipeline(
+    config: CliConfig,
+    log: Callable[[str], None],
+    http_client: httpx.AsyncClient,
+    llm_client: AsyncOpenAICompatibleClient,
+    llm_cache: AsyncSqliteLLMResponseCache | None,
+    state_manager: AsyncStateManager,
+    parts: ProjectParts,
+    shutdown: ShutdownSignal,
+) -> AsyncETLPipeline:
+    """Assemble the pipeline, answering LLM requests from ``llm_cache`` when there is one.
+
+    ``log`` receives one line per listing page. The validators run inside the
+    entity extractor, which logs each rejected entity with its reasons.
+    ``shutdown`` stops a run cleanly: records in flight finish and are marked
+    processed, and ``run()`` raises ``PipelineInterrupted``.
+    """
+    from sci_etl_core import (
+        AsyncETLPipeline,
+        AsyncLLMEntityExtractor,
+        AsyncLLMRelevanceFilter,
+        AsyncSqliteStateManager,
+        CachingLLMClient,
+    )
+
+    from sci_etl_cli.reporting import listing_reporter
+
+    answering = llm_client if llm_cache is None else CachingLLMClient(llm_client, llm_cache, model=config.llm.model)
+    entity_extractor: Any = AsyncLLMEntityExtractor(
+        answering,
+        parts.extraction_prompt,
+        result_key=config.prompts.result_key,
+        timeout=config.llm.timeout,
+        validator=build_validator(parts.validators),
+        label_field=config.export.key_column,
+    )
+    return AsyncETLPipeline.from_config(
+        config.pipeline,
+        extractor=listing_reporter(build_extractor(config, http_client), log),
+        relevance_filter=AsyncLLMRelevanceFilter(llm_client=answering, system_prompt=parts.relevance_prompt),
+        entity_extractor=entity_extractor,
+        exporter=build_exporter(config),
+        state_manager=state_manager,
+        closeables=[
+            http_client,
+            llm_client,
+            *([] if llm_cache is None else [llm_cache]),
+            *_closeable_state(state_manager, AsyncSqliteStateManager),
+        ],
+        shutdown=shutdown,
+    )
+
+
+def build_validator(validators: tuple[RecordValidator, ...]) -> RecordValidator | None:
+    """Combine the ``export.validators`` plug-ins into one validator, or ``None`` when there are none."""
+    if not validators:
+        return None
+    from sci_etl_core.processors import CompositeValidator
+
+    return CompositeValidator(list(validators))
+
+
+def _closeable_state(
+    state_manager: AsyncStateManager, closeable: type[AsyncSqliteStateManager]
+) -> list[AsyncSqliteStateManager]:
+    return [state_manager] if isinstance(state_manager, closeable) else []
+
+
+async def read_state(config: CliConfig) -> tuple[set[str], PipelineMetadata]:
+    """Load processed ids and metadata without creating state that does not exist yet."""
+    from sci_etl_core.models import PipelineMetadata
+
+    if _state_is_absent(config):
+        return set(), PipelineMetadata()
+    manager = build_state_manager(config)
+    try:
+        return await manager.load_processed_ids(), await manager.load_metadata()
+    finally:
+        await _release_state(manager)
+
+
+@dataclass(frozen=True, slots=True)
+class ForgetReport:
+    """Which requested ids the state tracked and so forgot, and which it did not, each in request order."""
+
+    forgotten: tuple[str, ...]
+    untracked: tuple[str, ...]
+
+    @property
+    def requested(self) -> int:
+        return len(self.forgotten) + len(self.untracked)
+
+
+async def forget_records(config: CliConfig, record_ids: Sequence[str]) -> ForgetReport:
+    """Clear the processed mark and failed attempts of every tracked id in ``record_ids``.
+
+    Calls the state manager's ``forget``, which the ``Forgetting`` protocol
+    describes, so the next run that lists the records again processes them
+    again. An id the state neither processed nor failed is reported as
+    untracked, a repeated id counts once, and state that does not exist yet
+    is not created.
+
+    Raises:
+        CliFailure: The configured state manager cannot forget records.
+    """
+    from sci_etl_core import Forgetting
+
+    requested = tuple(dict.fromkeys(record_ids))
+    if _state_is_absent(config):
+        return ForgetReport(forgotten=(), untracked=requested)
+    manager = build_state_manager(config)
+    try:
+        if not isinstance(manager, Forgetting):
+            raise CliFailure(f"The {config.state.backend} state backend cannot forget records")
+        tracked = await manager.load_processed_ids() | (await manager.failure_counts()).keys()
+        forgotten = tuple(record_id for record_id in requested if record_id in tracked)
+        await manager.forget(forgotten)
+    finally:
+        await _release_state(manager)
+    return ForgetReport(
+        forgotten=forgotten, untracked=tuple(record_id for record_id in requested if record_id not in tracked)
+    )
+
+
+def _state_is_absent(config: CliConfig) -> bool:
+    return config.state.backend == "sqlite" and not config.state.database.exists()
+
+
+async def _release_state(manager: AsyncStateManager) -> None:
+    aclose = getattr(manager, "aclose", None)
+    if aclose is not None:
+        await aclose()

sci_etl_cli-0.4.0/src/sci_etl_cli/commands/forget.py

@@ -0,0 +1,62 @@
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+
+import click
+from rich.markup import escape
+from rich.table import Table
+
+from sci_etl_cli import assembly
+from sci_etl_cli.options import config_argument
+from sci_etl_cli.output import print_json, stdout_console
+from sci_etl_cli.settings import CliConfig, load_cli_config
+
+
+@click.command("forget", short_help="Hand processed or failed records back to the next rescan.")
+@config_argument
+@click.argument("record_ids", metavar="RECORD_ID...", nargs=-1, required=True)
+@click.option("--json", "as_json", is_flag=True, help="Print JSON instead of a table.")
+def forget_command(config_path: Path, record_ids: tuple[str, ...], as_json: bool) -> None:
+    """Forget RECORD_IDs in CONFIG's state so the next rescan processes them again.
+
+    Clears each record's processed mark and failed attempts, for example after
+    a prompt or model change. The saved offset is kept, so follow it with
+    `run --rescan`, which starts at offset 0 and skips every other processed
+    record by id. A record's rows in the export CSV stay until it is processed
+    again, which replaces them. Use it between runs: a run reads the processed
+    ids when it starts. An id the state does not track is reported, not an
+    error.
+    """
+    config = load_cli_config(config_path)
+    report = asyncio.run(assembly.forget_records(config, record_ids))
+    if as_json:
+        print_json({"forgotten": list(report.forgotten), "untracked": list(report.untracked)})
+        return
+
+    table = Table(show_header=False, box=None, pad_edge=False)
+    table.add_column(no_wrap=True)
+    table.add_column(no_wrap=True)
+    for record_id in report.forgotten:
+        table.add_row(escape(record_id), "[green]forgotten[/]")
+    for record_id in report.untracked:
+        table.add_row(escape(record_id), "[yellow]not tracked[/]")
+    console = stdout_console()
+    console.print(table)
+    console.print(f"Forgot {len(report.forgotten)} of {describe_count(report.requested)}.")
+    console.print(escape(next_step(config, config_path)), soft_wrap=True)
+
+
+def describe_count(count: int) -> str:
+    return f"{count} record{'' if count == 1 else 's'}"
+
+
+def next_step(config: CliConfig, config_path: Path) -> str:
+    """Name the run that processes the forgotten records again, which depends on ``pipeline.newest_first``."""
+    command = f"sci-etl run {config_path} --rescan"
+    if config.pipeline.newest_first:
+        return (
+            f"Next: pipeline.newest_first is set and a rescan cannot be combined with it, "
+            f"so turn it off for one run and use: {command}"
+        )
+    return f"Next: {command}"