batchor 0.0.1__tar.gz → 0.0.3__tar.gz

{batchor-0.0.1 → batchor-0.0.3}/.github/workflows/ci.yml

@@ -46,7 +46,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.12", "3.13"]
+        python-version: ["3.12", "3.13", "3.14"]
 
     steps:
       - uses: actions/checkout@v4

{batchor-0.0.1 → batchor-0.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: batchor
-Version: 0.0.1
+Version: 0.0.3
 Summary: Structured-first OpenAI Batch runner with typed Pydantic results.
 Project-URL: Documentation, https://github.com/AnsonDev42/batchor#readme
 Project-URL: Repository, https://github.com/AnsonDev42/batchor
@@ -88,6 +88,7 @@ Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: <4,>=3.12
 Requires-Dist: openai>=2.7.1
@@ -102,12 +103,15 @@ Description-Content-Type: text/markdown
 
 # batchor
 
+`batchor` grew out of a recurring problem in academic research: running large datasets through LLMs in batch — reliably, reproducibly, and without reinventing the same glue code across every project. The patterns that kept emerging (durable state, typed results, safe resume after failure) were extracted into this library so they do not have to be rebuilt each time.
+
 `batchor` is a durable OpenAI Batch runner for Python teams that want:
 
 - typed Pydantic results
 - resumable durable runs
 - replayable request artifacts
 - deterministic source checkpoints
+- provider-side enqueue limit controls to stay within token budgets
 - library-first run controls
 - a small operator CLI for CSV and JSONL jobs
 
@@ -154,6 +158,7 @@ Important constraints:
 - structured-output rehydration requires an importable module-level Pydantic model
 - raw output artifacts are retained by default and must be exported before raw pruning
 - pause/resume/cancel and incremental terminal-result APIs are library-first today
+- OpenAI control-plane or batch-level 429 quota/billing exhaustion auto-pauses the run instead of burning attempts across the remaining backlog
 
 ## Mental model
 
@@ -173,6 +178,7 @@ Durability is split on purpose:
 - the artifact store keeps replayable request JSONL and downloaded raw batch payloads
 
 That split is what allows retries and fresh-process resume without keeping every request inline in the control-plane store.
+On resume, existing active provider batches are polled before `batchor` materializes or submits new local work, so completed batches and retry backoff are reconciled first. Deterministic sources also use stored checkpoint completion metadata to avoid opening a fully materialized Parquet or composite source just to discover there are no rows left.
 
 ## Architecture
 
@@ -256,6 +262,7 @@ Supported Python versions:
 
 - `3.12`
 - `3.13`
+- `3.14`
 
 ## Authentication
 
@@ -334,6 +341,8 @@ run.wait()
 print(run.results()[0].output)
 ```
 
+While waiting, `batchor` polls active batches before submitting more work. If a refresh consumes a completed batch, records terminal item state, or submits another provider batch, `Run.wait()` immediately continues draining instead of sleeping for the next poll interval.
+
 Structured-output models are validated up front against the OpenAI strict-schema subset used by `batchor`.
 
 - root schema must be an object
@@ -479,6 +488,8 @@ run.cancel()
 
 Run control and incremental terminal-result APIs are Python-first in this release. The CLI does not yet expose `pause`, `resume`, `cancel`, or incremental terminal-result export commands.
 
+OpenAI insufficient-quota failures during upload/create/polling, or as a batch-level terminal error, are treated as an automatic pause with `summary().control_reason == "openai_insufficient_quota"`. Row-level insufficient-quota records inside a completed batch output stay item-scoped: those rows become retryable without consuming attempts, emit `openai_insufficient_quota` failure telemetry, and use retry backoff before resubmission. Auto-pause does not replace an in-progress cancellation.
+
 ## CLI quickstart
 
 The CLI is intentionally narrower than the Python API:
@@ -539,12 +550,13 @@ def observer(event: RunEvent) -> None:
 runner = BatchRunner(observer=observer)
 ```
 
-Current events include run creation/resume, item ingestion, batch submission/polling/completion, item completion/failure, and artifact export/prune.
+Current events include run creation/resume, automatic quota pause, item ingestion, batch submission/polling/completion, item completion/failure, and artifact export/prune.
 
 ## Storage notes
 
 - SQLite remains the default durable backend.
-- `PostgresStorage` is available for shared control-plane state, but the CLI remains SQLite-only today.
+- `PostgresStorage` is available for shared control-plane state, but the CLI remains SQLite-only today. Plain `postgresql://` and `postgres://` DSNs are normalized to the package's psycopg v3 driver URL.
+- Item `attempt_count` means consumed provider attempts: successful submitted completions increment it, counted item-level failures increment it, and local or batch-level retry/reset paths that did not consume an item attempt leave it unchanged.
 - Durable artifacts now go through an `ArtifactStore` seam. The built-in implementation is `LocalArtifactStore`, intended for local disk or a shared mounted volume.
 - Fresh-process resume requeues any locally claimed but not yet submitted items before continuing, so a process crash after request-artifact persistence does not strand work in `queued_local`.
 
@@ -591,51 +603,6 @@ The default pytest configuration enforces an `85%` coverage floor.
 
 GitHub Actions pull requests run:
 
-- `test (3.12)` and `test (3.13)`: `uv run ty check src` plus `uv run pytest -q`
-- `docs`: `uv run mkdocs build --strict`
-- `build (3.13)`: `uv build`
-- `postgres-contract`: `uv run pytest tests/unit/test_batchor_storage_contracts.py --no-cov -q` with an ephemeral PostgreSQL service
-
-The live OpenAI smoke remains manual-only and is not part of required CI.
-
-Manual live OpenAI smoke:
-
-```bash
-export OPENAI_API_KEY=sk-...
-export BATCHOR_RUN_LIVE_TESTS=1
-uv run pytest tests/integration/test_batchor_live_openai.py --no-cov -q
-```
-
-If `BATCHOR_LIVE_OPENAI_MODEL` is unset, the harness defaults to `gpt-5-nano`.
-If `BATCHOR_LIVE_OPENAI_REASONING_EFFORT` is unset, no reasoning field is sent.
-The live smoke also requires an OpenAI account with Batch API access and available billing quota.
-
-## Documentation guide
-
-- `docs/design_docs/ARCHITECTURE.md`: canonical runtime diagrams and module boundaries
-- `docs/design_docs/STORAGE_AND_RUNS.md`: durable run, resume, and artifact lifecycle semantics
-- `docs/getting-started/use-cases.md`: practical single-file, multi-file, and pipeline examples
-- `docs/getting-started/python-api.md`: Python-first workflows
-- `docs/getting-started/cli.md`: operator CLI workflows
-- `docs/reference/api.md`: generated API surface
-- `docs/design_docs/ARCHITECTURE.md`: package boundaries
-- `docs/design_docs/OPENAI_BATCHING.md`: OpenAI-specific behavior
-- `docs/design_docs/STORAGE_AND_RUNS.md`: durability, rehydration, and artifacts
-- `docs/smoke-test.md`: minimum validation bar
-
-## Scope
-
-Today `batchor` is intentionally narrow:
-
-- one built-in provider: OpenAI Batch
-- one default durable backend: SQLite
-- one opt-in durable backend: Postgres
-- one ephemeral backend: in-memory storage
-- structured output is Python API-first
-- CLI job creation is file-backed and supports both text and structured-output jobs
-
-Anything outside that scope should be treated as out of scope or `TBD`, not implied support.
-
 ## License
 
 This project is licensed under PolyForm Noncommercial 1.0.0. See [LICENSE](LICENSE).

{batchor-0.0.1 → batchor-0.0.3}/README.md

@@ -1,11 +1,14 @@
 # batchor
 
+`batchor` grew out of a recurring problem in academic research: running large datasets through LLMs in batch — reliably, reproducibly, and without reinventing the same glue code across every project. The patterns that kept emerging (durable state, typed results, safe resume after failure) were extracted into this library so they do not have to be rebuilt each time.
+
 `batchor` is a durable OpenAI Batch runner for Python teams that want:
 
 - typed Pydantic results
 - resumable durable runs
 - replayable request artifacts
 - deterministic source checkpoints
+- provider-side enqueue limit controls to stay within token budgets
 - library-first run controls
 - a small operator CLI for CSV and JSONL jobs
 
@@ -52,6 +55,7 @@ Important constraints:
 - structured-output rehydration requires an importable module-level Pydantic model
 - raw output artifacts are retained by default and must be exported before raw pruning
 - pause/resume/cancel and incremental terminal-result APIs are library-first today
+- OpenAI control-plane or batch-level 429 quota/billing exhaustion auto-pauses the run instead of burning attempts across the remaining backlog
 
 ## Mental model
 
@@ -71,6 +75,7 @@ Durability is split on purpose:
 - the artifact store keeps replayable request JSONL and downloaded raw batch payloads
 
 That split is what allows retries and fresh-process resume without keeping every request inline in the control-plane store.
+On resume, existing active provider batches are polled before `batchor` materializes or submits new local work, so completed batches and retry backoff are reconciled first. Deterministic sources also use stored checkpoint completion metadata to avoid opening a fully materialized Parquet or composite source just to discover there are no rows left.
 
 ## Architecture
 
@@ -154,6 +159,7 @@ Supported Python versions:
 
 - `3.12`
 - `3.13`
+- `3.14`
 
 ## Authentication
 
@@ -232,6 +238,8 @@ run.wait()
 print(run.results()[0].output)
 ```
 
+While waiting, `batchor` polls active batches before submitting more work. If a refresh consumes a completed batch, records terminal item state, or submits another provider batch, `Run.wait()` immediately continues draining instead of sleeping for the next poll interval.
+
 Structured-output models are validated up front against the OpenAI strict-schema subset used by `batchor`.
 
 - root schema must be an object
@@ -377,6 +385,8 @@ run.cancel()
 
 Run control and incremental terminal-result APIs are Python-first in this release. The CLI does not yet expose `pause`, `resume`, `cancel`, or incremental terminal-result export commands.
 
+OpenAI insufficient-quota failures during upload/create/polling, or as a batch-level terminal error, are treated as an automatic pause with `summary().control_reason == "openai_insufficient_quota"`. Row-level insufficient-quota records inside a completed batch output stay item-scoped: those rows become retryable without consuming attempts, emit `openai_insufficient_quota` failure telemetry, and use retry backoff before resubmission. Auto-pause does not replace an in-progress cancellation.
+
 ## CLI quickstart
 
 The CLI is intentionally narrower than the Python API:
@@ -437,12 +447,13 @@ def observer(event: RunEvent) -> None:
 runner = BatchRunner(observer=observer)
 ```
 
-Current events include run creation/resume, item ingestion, batch submission/polling/completion, item completion/failure, and artifact export/prune.
+Current events include run creation/resume, automatic quota pause, item ingestion, batch submission/polling/completion, item completion/failure, and artifact export/prune.
 
 ## Storage notes
 
 - SQLite remains the default durable backend.
-- `PostgresStorage` is available for shared control-plane state, but the CLI remains SQLite-only today.
+- `PostgresStorage` is available for shared control-plane state, but the CLI remains SQLite-only today. Plain `postgresql://` and `postgres://` DSNs are normalized to the package's psycopg v3 driver URL.
+- Item `attempt_count` means consumed provider attempts: successful submitted completions increment it, counted item-level failures increment it, and local or batch-level retry/reset paths that did not consume an item attempt leave it unchanged.
 - Durable artifacts now go through an `ArtifactStore` seam. The built-in implementation is `LocalArtifactStore`, intended for local disk or a shared mounted volume.
 - Fresh-process resume requeues any locally claimed but not yet submitted items before continuing, so a process crash after request-artifact persistence does not strand work in `queued_local`.
 
@@ -489,51 +500,6 @@ The default pytest configuration enforces an `85%` coverage floor.
 
 GitHub Actions pull requests run:
 
-- `test (3.12)` and `test (3.13)`: `uv run ty check src` plus `uv run pytest -q`
-- `docs`: `uv run mkdocs build --strict`
-- `build (3.13)`: `uv build`
-- `postgres-contract`: `uv run pytest tests/unit/test_batchor_storage_contracts.py --no-cov -q` with an ephemeral PostgreSQL service
-
-The live OpenAI smoke remains manual-only and is not part of required CI.
-
-Manual live OpenAI smoke:
-
-```bash
-export OPENAI_API_KEY=sk-...
-export BATCHOR_RUN_LIVE_TESTS=1
-uv run pytest tests/integration/test_batchor_live_openai.py --no-cov -q
-```
-
-If `BATCHOR_LIVE_OPENAI_MODEL` is unset, the harness defaults to `gpt-5-nano`.
-If `BATCHOR_LIVE_OPENAI_REASONING_EFFORT` is unset, no reasoning field is sent.
-The live smoke also requires an OpenAI account with Batch API access and available billing quota.
-
-## Documentation guide
-
-- `docs/design_docs/ARCHITECTURE.md`: canonical runtime diagrams and module boundaries
-- `docs/design_docs/STORAGE_AND_RUNS.md`: durable run, resume, and artifact lifecycle semantics
-- `docs/getting-started/use-cases.md`: practical single-file, multi-file, and pipeline examples
-- `docs/getting-started/python-api.md`: Python-first workflows
-- `docs/getting-started/cli.md`: operator CLI workflows
-- `docs/reference/api.md`: generated API surface
-- `docs/design_docs/ARCHITECTURE.md`: package boundaries
-- `docs/design_docs/OPENAI_BATCHING.md`: OpenAI-specific behavior
-- `docs/design_docs/STORAGE_AND_RUNS.md`: durability, rehydration, and artifacts
-- `docs/smoke-test.md`: minimum validation bar
-
-## Scope
-
-Today `batchor` is intentionally narrow:
-
-- one built-in provider: OpenAI Batch
-- one default durable backend: SQLite
-- one opt-in durable backend: Postgres
-- one ephemeral backend: in-memory storage
-- structured output is Python API-first
-- CLI job creation is file-backed and supports both text and structured-output jobs
-
-Anything outside that scope should be treated as out of scope or `TBD`, not implied support.
-
 ## License
 
 This project is licensed under PolyForm Noncommercial 1.0.0. See [LICENSE](LICENSE).

{batchor-0.0.1 → batchor-0.0.3}/docs/design_docs/ARCHITECTURE.md

@@ -133,13 +133,16 @@ Internally that expands to:
 
 1. Resolve provider and storage implementations.
 2. Persist run config and ingest items into durable state.
-3. Claim a bounded submission window from pending items.
-4. Build or replay request JSONL rows.
-5. Persist request artifacts before upload.
-6. Submit one or more OpenAI batch files.
-7. Poll active batches.
-8. Download output/error files.
-9. Parse terminal item results back into the state store.
+3. On resume, poll any already-active provider batches before ingesting or submitting new work.
+4. Claim a bounded submission window from pending items.
+5. Build or replay request JSONL rows.
+6. Persist request artifacts before upload.
+7. Submit one or more OpenAI batch files.
+8. Poll active batches.
+9. Download output/error files.
+10. Parse terminal item results back into the state store.
+
+When a caller uses `Run.wait()`, the runtime repeats that poll-and-submit pass until the run is terminal. A pass that changes durable work state, such as consuming completed batches or submitting more items, immediately triggers the next pass rather than sleeping for the configured poll interval.
 
 ## Execution Sequence
 
@@ -175,6 +178,9 @@ sequenceDiagram
                 Provider-->>BatchRunner: successes + errors
                 BatchRunner->>StateStore: mark_items_completed(successes)
                 BatchRunner->>StateStore: mark_items_failed(errors)
+                opt row-level insufficient quota
+                    BatchRunner->>StateStore: record_batch_retry_failure(row_insufficient_quota)
+                end
             else status in {failed, cancelled, expired}
                 BatchRunner->>Provider: download_file_content(output_file_id, error_file_id)
                 Provider-->>BatchRunner: output/error JSONL
@@ -182,6 +188,11 @@ sequenceDiagram
                 BatchRunner->>StateStore: record_batch_retry_failure()
                 BatchRunner->>StateStore: reset_batch_items_to_pending()
             end
+
+            opt control-plane or batch-level OpenAI insufficient quota detected
+                BatchRunner->>StateStore: set_run_control_state(paused, control_reason)
+                BatchRunner-->>User: RunPausedError / run_auto_paused event
+            end
         end
 
         BatchRunner->>StateStore: claim_items_for_submission(limit)
@@ -217,7 +228,7 @@ stateDiagram-v2
     QUEUED_LOCAL --> FAILED_PERMANENT : rejected pre-submission (e.g. token budget exceeded, max attempts)
     QUEUED_LOCAL --> SUBMITTED : batch created and registered with provider
 
-    SUBMITTED --> COMPLETED : batch completed, result parsed OK
+    SUBMITTED --> COMPLETED : batch completed, result parsed OK — attempt consumed
     SUBMITTED --> FAILED_RETRYABLE : batch error / item error — attempt count below max
     SUBMITTED --> FAILED_PERMANENT : batch error / item error — attempt count reached max
 
@@ -243,7 +254,7 @@ stateDiagram-v2
 
     state "Control State" as cs {
         [*] --> RUNNING2 : run created
-        RUNNING2 --> PAUSED : pause_run() called
+        RUNNING2 --> PAUSED : pause_run() or quota auto-pause
         PAUSED --> RUNNING2 : resume_run() called
         RUNNING2 --> CANCEL_REQUESTED : cancel_run() called
 
@@ -258,6 +269,15 @@ stateDiagram-v2
 Detailed storage, resume, and artifact-retention semantics live in
 [`STORAGE_AND_RUNS.md`](STORAGE_AND_RUNS.md).
 
+Paused runs may include a durable `control_reason`. Manual pauses use
+`"manual"`; control-plane or batch-level OpenAI quota/billing exhaustion uses
+`"openai_insufficient_quota"` and preserves retryable work for later resume.
+Row-level quota records inside completed batch output remain item-level
+retryable failures and use retry backoff without pausing the run.
+Quota auto-pause never replaces `cancel_requested`, and non-checkpointed
+finite iterables continue materializing after auto-pause so unpersisted input
+rows are not lost.
+
 ## Module boundaries
 
 ### `core/`
@@ -302,6 +322,8 @@ Owns execution behavior:
 - `Run`
 - Typer CLI entrypoint for operator workflows
 - persisted run control state with `pause`, `resume`, and drain-style `cancel`
+- automatic OpenAI control-plane/batch-level insufficient-quota pause with durable `control_reason`
+- item-level insufficient-quota retries with backoff and no attempt consumption
 - optional observer callback for provider lifecycle events
 - token estimation and request chunking
 - bounded pending-item claim windows before submission
@@ -319,6 +341,14 @@ Owns execution behavior:
 
 This is where the durable lifecycle lives. It bridges the domain models, providers, storage, and artifact store.
 
+The public facade is still `BatchRunner` plus `Run`, but the current internal layering under `runtime/` is intentionally split so execution concerns do not collapse back into one file:
+
+- `context.py`: persisted config building, resume config comparison, output-model resolution, and provider-backed `RunContext` creation
+- `ingestion.py`: checkpoint-aware item materialization, ingest checkpoint updates, and resume-aware ingestion flow
+- `submission.py`: pending-item claiming, request replay/build, token-budget gating, request artifact persistence, and batch submission
+- `polling.py`: refresh orchestration, active-batch polling, terminal batch consumption, and batch-failure reset/backoff handling
+- `artifacts.py` and `results.py`: request replay helpers, raw artifact writes/deletes, public result mapping, and JSONL result serialization
+
 ### `sources/`
 
 Owns streaming input adapters:
@@ -347,6 +377,7 @@ The storage layer persists:
 
 - run config
 - run control state
+- run control reason
 - item state and attempts
 - active batch metadata
 - ingest checkpoints

{batchor-0.0.1 → batchor-0.0.3}/docs/design_docs/OPENAI_BATCHING.md

@@ -55,6 +55,7 @@ Before request artifacts exist, built-in deterministic sources can resume ingest
 That currently includes CSV, JSONL, Parquet, and `CompositeItemSource` for ordered composition of checkpointed sources.
 When `CompositeItemSource` is used, child-source row IDs are auto-namespaced into run-unique `item_id` values, while the original row ID is preserved in lineage metadata.
 Custom non-file sources must implement a durable checkpoint contract explicitly; arbitrary iterables and live DB cursors are still `TBD`.
+Before resumed ingestion continues, active OpenAI batches are polled once. If that poll consumes terminal batches or records enqueue/backoff failures, those state changes are applied before any new prompt rendering, request replay, or submission happens.
 
 ## Raw output retention
 
@@ -99,6 +100,7 @@ For structured outputs:
 - the result is validated through the declared Pydantic model
 
 Validation failures consume item attempts because they represent item-level response failure, not a transient provider control-plane problem.
+Successful submitted responses also consume one item attempt. A successful first provider response is exported with `attempt_count == 1`; a counted retryable item failure followed by success is exported with the total consumed attempts.
 
 ## Token estimation
 
@@ -125,6 +127,7 @@ From those settings, `batchor` derives:
 - an effective per-batch token limit
 
 Submission is constrained by both token budget and generic chunking rules such as max request count and max request file size.
+During `Run.wait()`, each drain cycle polls active batches before submitting new work, so capacity freed by completed batches is visible before the next submit attempt. If that cycle makes durable progress, the wait loop immediately runs another cycle instead of sleeping.
 
 ## Batch splitting
 
@@ -137,6 +140,7 @@ Splitting considers:
 - estimated request tokens
 
 Submission claims only a bounded pending-item window per refresh. This is intentional: large backlogs should not pay full prompt-build and token-estimation cost long before provider capacity exists to send them.
+The default claim window is still large enough to prepare multiple provider batches per cycle, so runs with high remote enqueue headroom can keep filling capacity without waiting for one local batch at a time.
 
 If a single request exceeds the allowed OpenAI token limit by itself, that item is marked as a permanent item failure instead of aborting the whole run.
 
@@ -150,16 +154,23 @@ Control-plane failures:
 - retryable control-plane failures do not consume item attempts
 - batch submit failures can trigger batch-level backoff
 - transient poll failures do not stall unrelated submissions when other capacity remains
+- resume honors existing batch backoff before materializing more source rows
+- OpenAI 429 insufficient-quota or billing exhaustion during upload/create/polling auto-pauses the run with `control_reason="openai_insufficient_quota"` instead of continuing through the backlog
+- batch-level terminal insufficient-quota failures also auto-pause after submitted rows are reset to pending
 
 Item-level failures:
 
+- successful submitted responses consume attempts
 - structured-output parse failures consume attempts
 - validation failures consume attempts
 - oversized requests become permanent item failures
+- item-level insufficient-quota records inside completed batch output are retryable without consuming attempts
+- row-level insufficient-quota records emit `openai_insufficient_quota` failure telemetry and record retry backoff before resubmission instead of pausing the whole run
 
 Cleanup behavior:
 
 - if upload succeeds but batch creation fails, `batchor` makes a best-effort attempt to delete the uploaded OpenAI input file
+- if upload or batch creation fails because OpenAI reports insufficient quota, local queued items are released back to pending before the run is paused
 - if a process dies after local artifact persistence but before durable batch registration, fresh-process resume requeues those items and resubmits from persisted request artifacts
 
 ## Run control
@@ -169,6 +180,8 @@ Run control is local control-plane state, not a separate OpenAI provider feature
 - `pause` stops new ingestion, new submission, and provider polling
 - `resume` restarts those local activities
 - `cancel` stops new ingestion/submission, continues polling already-submitted batches, and then marks any remaining local non-terminal items as `run_cancelled`
+- automatic quota pause uses the same `paused` control state as manual pause for control-plane and batch-level quota failures, but records `control_reason="openai_insufficient_quota"` so operators can tell why `wait()` exited
+- automatic quota pause is ignored once `cancel_requested` is set, because cancellation is not reversible
 
 Provider-side remote batch cancellation is not implemented in v1.
 

{batchor-0.0.1 → batchor-0.0.3}/docs/design_docs/STORAGE_AND_RUNS.md

@@ -18,6 +18,7 @@ The public handle exposes:
 - `run_id`
 - cached `status`
 - cached `control_state`
+- cached `control_reason`
 - `is_finished`
 - `refresh()`
 - `wait()`
@@ -36,7 +37,7 @@ Important semantics:
 
 - `status` is cached from the last summary read or refresh
 - `refresh()` performs one poll-and-submit pass
-- `wait()` repeatedly refreshes until the run is terminal
+- `wait()` repeatedly refreshes until the run is terminal, and skips the poll-interval sleep after refreshes that complete items, fail items, or submit additional batches
 - `results()` and artifact lifecycle operations are terminal-only
 - terminal currently means either `completed` or `completed_with_failures`
 - `read_terminal_results()` and `export_terminal_results()` are incremental APIs for already-terminal items and are safe to call before the whole run finishes
@@ -49,6 +50,7 @@ The control plane is the state store:
 
 - run config
 - run control state
+- run control reason
 - item rows and attempts
 - active batch rows
 - parsed outputs and failure records
@@ -80,6 +82,7 @@ Current storage responsibilities include:
 
 - persisting public run config
 - persisting run control state
+- persisting run control reason
 - persisting deterministic-source ingest checkpoints when available
 - persisting item state and attempts
 - persisting terminal result sequence metadata for incremental reads
@@ -111,6 +114,7 @@ For SQLite-backed runs, the default artifact root is a sibling `*_artifacts/` di
 ## Postgres behavior
 
 Postgres exists as an opt-in control-plane backend for cases where SQLite is not enough, such as shared state across processes or hosts.
+`PostgresStorage` uses psycopg v3. Plain `postgresql://` and `postgres://` DSNs are normalized to `postgresql+psycopg://` before engine creation; explicit SQLAlchemy driver schemes are preserved.
 
 Important operational rule:
 
@@ -118,6 +122,16 @@ Important operational rule:
 
 Postgres stores the control plane, not the large request/output files themselves.
 
+## Attempt accounting
+
+Item `attempt_count` records consumed provider attempts.
+
+- Successful submitted completions increment the counter once.
+- Counted item-level provider or parse failures increment the counter before retry/permanent-failure decisions.
+- Local pre-submission rejections, control-plane quota pauses, batch-level reset-to-pending paths, and missing output rows marked with `count_attempt=False` do not increment it.
+
+This keeps terminal exports aligned with audit expectations: a first-attempt success reports `attempt_count == 1`, and a retryable counted failure followed by success reports `attempt_count == 2`.
+
 ## File-source checkpoints
 
 For deterministic built-in sources, storage persists a source checkpoint with:
@@ -163,6 +177,7 @@ Successful rehydration depends on:
 - credentials being available when a refresh needs to talk to the provider
 
 Fresh-process resume also requeues any `queued_local` items back to `pending` before submission resumes.
+When `start(job, run_id=...)` resumes a run with active provider batches, it first performs a poll-only reconciliation pass. Terminal provider batches are consumed, failed batches update retry backoff, and any active backoff prevents new source materialization or submission until the backoff expires.
 
 Resume compatibility intentionally ignores non-persisted secret fields such as provider API keys.
 
@@ -176,6 +191,8 @@ Built-in deterministic sources currently include:
 - `JsonlItemSource`
 - `ParquetItemSource`
 
+For checkpointed ingestion, durable stores append materialized item chunks and advance the ingest checkpoint in one storage operation. Parquet and composite sources can also identify an end-of-source checkpoint from cheap metadata, so a resume after all rows were materialized but before the final completion flag was written can mark ingestion complete without re-reading the source data or re-rendering prompts.
+
 Once an item has a durable request artifact pointer, `batchor` prunes large inline request-building fields from the control-plane store and relies on the artifact for later retries.
 
 ## Artifact lifecycle
@@ -222,7 +239,15 @@ Semantics:
 - `resume` restarts those local activities from persisted state
 - `cancel` stops new ingestion/submission, keeps polling active provider batches, and then permanently fails remaining local non-terminal items with `error_class="run_cancelled"`
 
-`wait()` fails fast on paused runs instead of sleeping indefinitely.
+Manual pauses record `control_reason="manual"`. OpenAI upload/create/polling and batch-level insufficient-quota pauses record `control_reason="openai_insufficient_quota"`, clear batch-control-plane backoff, and preserve affected items for later retry without consuming attempts.
+
+Row-level insufficient-quota records inside completed batch outputs do not pause the run. Those items are stored as `failed_retryable` with `error_class="openai_insufficient_quota"`, do not consume attempts, and rely on retry backoff before they are submitted again.
+
+Auto-pause must not overwrite `cancel_requested`. Cancellation is non-reversible, so quota errors observed while draining already-submitted batches leave the run in cancel flow.
+
+For non-checkpointed finite iterables, ingestion continues materializing remaining items after an auto-pause so `resume()` cannot silently lose input rows that were not yet persisted. Checkpointed sources may stop on pause because their stored checkpoint can resume materialization later.
+
+`wait()` fails fast on paused runs instead of sleeping indefinitely. The raised `RunPausedError` carries the same `control_reason` as the summary.
 Provider-side remote cancellation is still `TBD`.
 
 ## Python API versus CLI

{batchor-0.0.1 → batchor-0.0.3}/docs/design_docs/STORAGE_MIGRATIONS.md

@@ -7,7 +7,8 @@ This document describes the current SQLite schema-versioning story for `batchor`
 - SQLite remains the default durable backend
 - the SQLite schema is additive and self-healing for supported columns/tables
 - storage metadata now persists a `schema_version`
-- the current published schema version is `1`
+- the current published schema version is `4`
+- schema version `4` adds the nullable `runs.control_reason` column for durable pause reasons
 
 ## Compatibility Rules
 

{batchor-0.0.1 → batchor-0.0.3}/docs/getting-started/python-api.md

@@ -303,6 +303,8 @@ runner = BatchRunner(
 )
 ```
 
+`PostgresStorage` uses psycopg v3. Plain `postgresql://` and `postgres://` DSNs are accepted and normalized to `postgresql+psycopg://`; explicit SQLAlchemy driver schemes are left unchanged.
+
 ## Artifacts, export, prune, and retention
 
 `batchor` stores request artifacts for replay and raw output artifacts for audit/export.

{batchor-0.0.1 → batchor-0.0.3}/docs/smoke-test.md

@@ -31,6 +31,7 @@ uv run pytest -q
 This path uses the default pytest configuration, which:
 
 - runs in parallel
+- collects this package's `tests/` directory by default, so local reference checkouts are not treated as part of the smoke suite
 - enforces strict pytest config/marker handling
 - enforces the `85%` coverage gate
 
@@ -77,15 +78,21 @@ Expected:
 - composite deterministic sources can namespace duplicate row IDs across explicit inputs and resume across source boundaries
 - Parquet source adapters can resume from opaque checkpoints and project only required columns
 - retry/resume from persisted request artifacts still works for SQLite-backed runs
+- resumed runs reconcile existing active batches and persisted backoff before materializing more source rows
 - transient batch-poll failures do not block unrelated pending submissions from being sent when capacity remains
 - paused runs stop polling/submission until resumed
+- OpenAI control-plane and batch-level insufficient-quota provider failures auto-pause with a durable `control_reason` and do not consume item attempts
+- OpenAI row-level insufficient-quota records in completed batch output remain retryable, do not consume attempts, and back off without pausing the run
+- quota auto-pause preserves `cancel_requested` and does not strand non-checkpointed input rows during initial ingestion
 - cancelled runs drain already-submitted work and mark remaining local items as `run_cancelled`
 - incremental terminal-result reads/exports remain sequence-based and idempotent across repeated calls
 - raw output/error artifacts can be exported and require export before raw pruning
 - raw output/error artifact persistence can be disabled without breaking parsed terminal results or request-artifact replay
 - terminal runs, including `completed_with_failures`, can prune request artifacts without losing persisted results
 - shared storage-contract behavior remains aligned across SQLite and opt-in Postgres
+- completed submitted items report consumed attempts consistently across storage backends
 - OpenAI request splitting and enqueue-limit logic still behave as expected
+- wait-mode refresh cycles keep draining immediately after poll/submission progress, without introducing idle poll sleeps while more local work can be sent
 - structured-output parsing remains stable
 
 Notes:
@@ -127,19 +134,25 @@ This is not part of default CI. Prefer fake-provider tests for automated coverag
 Manual only. Recommended local flow:
 
 ```bash
-export OPENAI_API_KEY=sk-...
 export BATCHOR_RUN_LIVE_TESTS=1
 export BATCHOR_LIVE_OPENAI_MODEL=gpt-5-nano
 uv run pytest tests/integration/test_batchor_live_openai.py --no-cov -q
 ```
 
+Minimum single-item live smoke:
+
+```bash
+export BATCHOR_RUN_LIVE_TESTS=1
+uv run pytest tests/integration/test_batchor_live_openai.py -k text_job_smoke --no-cov -q
+```
+
 Behavior:
 
 - runs a single-item text smoke and a two-CSV composition smoke against the real OpenAI Batch API
 - uses SQLite durability and the normal `BatchRunner` flow
 - defaults to `gpt-5-nano` unless `BATCHOR_LIVE_OPENAI_MODEL` is set
 - sends no reasoning field unless `BATCHOR_LIVE_OPENAI_REASONING_EFFORT` is set
-- loads `.env` when present through the test harness for local use
+- loads `.env` when present through the test harness for local use, so `OPENAI_API_KEY` may come from the shell or `.env`
 - is skipped unless `BATCHOR_RUN_LIVE_TESTS=1`
 - requires an OpenAI account with Batch API access and available billing quota
 

{batchor-0.0.1 → batchor-0.0.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "batchor"
-version = "0.0.1"
+version = "0.0.3"
 description = "Structured-first OpenAI Batch runner with typed Pydantic results."
 readme = "README.md"
 license = { file = "LICENSE" }
@@ -29,6 +29,7 @@ classifiers = [
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "Topic :: Software Development :: Libraries :: Python Modules",
 ]
 
@@ -60,6 +61,7 @@ packages = ["src/batchor"]
 [tool.pytest.ini_options]
 addopts = "-q -n auto --strict-config --strict-markers --cov=src --cov-report=term-missing --cov-fail-under=85"
 pythonpath = ["src"]
+testpaths = ["tests"]
 markers = [
     "unit: fast isolated tests",
     "integration: fake-provider and multi-component tests",