streamgate 2.1.0__tar.gz → 3.1.0__tar.gz

{streamgate-2.1.0 → streamgate-3.1.0}/.github/workflows/ci.yml

@@ -48,7 +48,7 @@ jobs:
 
   bare-install:
     # Regression gate for the pure-core contract: a bare install
-    # (`pip install .`) must pull exactly aiokafka/loguru/pydantic (+transitive),
+    # (`pip install .`) must pull exactly aiokafka/pydantic (+transitive),
     # import cleanly, and run the pure_pipeline example end to end
     # (ingest → Kafka → consume → sqlite sink) against a real broker.
     runs-on: ubuntu-latest

{streamgate-2.1.0 → streamgate-3.1.0}/CHANGELOG.md

@@ -5,6 +5,150 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.1.0] - 2026-09-14
+
+### Added
+
+- **`JsonFormatter`** — a built-in, out-of-the-box stdlib `logging.Formatter`
+  that renders every record as one JSON line, restoring the structured-field
+  visibility 2.x hosts got for free from loguru's auto-configured JSON sink.
+  3.0.0 kept emitting all structured fields (as flat `LogRecord` extras) but
+  stdlib's default formatter never renders them — a batch consumer logged just
+  the bare `batch_handle_success` event name, hiding `batch_size` /
+  `duration_ms`. Non-breaking; import from `streamgate` or `streamgate.obs`
+  and wire it in three lines:
+
+  ```python
+  import logging
+  from streamgate import JsonFormatter
+
+  handler = logging.StreamHandler()        # stderr, same destination as 2.x
+  handler.setFormatter(JsonFormatter())
+  logging.basicConfig(level=logging.INFO, handlers=[handler])
+  ```
+
+  Output layout per record: `timestamp` (UTC ISO-8601 with `Z` suffix, taken
+  from the `LogRecord` clock, not format time), `level`, `logger` (hierarchical
+  namespace — new versus 2.x), `event` (the stable event-name contract; host
+  `%s`-style messages render identically), then every flat extra field
+  (`batch_size`, `duration_ms`, `partition`, `offset`, ...), plus `error`
+  (formatted traceback) when a record carries `exc_info`. Serialization: one
+  `json.dumps` per record, `ensure_ascii=False`, `default=str` fallback.
+  The formatter is a generic component — any logger's records render correctly
+  through it (no extras ⇒ base fields only), and it never raises on
+  `LogRecord` reserved-attribute collisions (host-side extras are skipped;
+  rendering-side tolerance, the emission-side `emit()` fail-fast is unchanged).
+  Attach the handler to the root logger for unified host + streamgate JSON
+  logs, or to `logging.getLogger("streamgate")` for streamgate-only.
+- `streamgate.obs.logging.RESERVED_RECORD_ATTRS` (also re-exported from
+  `streamgate.obs`) — the reserved-attribute set as a single shared constant,
+  used by `emit()` for fail-fast and by `JsonFormatter` for skip-tolerance
+  (previously privately defined per side).
+- All `examples/` now configure logging through the built-in `JsonFormatter`,
+  so copy-pasting an example yields field-visible JSON logs out of the box
+  (previously `basicConfig(format=...)` rendered no extras).
+
+## [3.0.0] - 2026-09-14
+
+### BREAKING — loguru removed, logging switched to stdlib `logging`
+
+The library no longer depends on loguru. All internal loggers now live under
+the standard-library `streamgate.*` logger tree
+(`streamgate.ingest.producer`, `streamgate.consumer.loop`, ...), and the
+library **only emits records — it never configures logging**: `import
+streamgate` adds no handlers, changes no levels, touches no logging global
+state. Wheel dependencies drop from three to two (`aiokafka`, `pydantic`).
+
+**Log event names and structured field names are unchanged** — only the
+transport moved. What breaks and how to migrate:
+
+| 2.x behavior | 3.0.0 replacement |
+|--------------|-------------------|
+| `from streamgate import logger` | `logging.getLogger("streamgate")` (module-level loggers are `logging.getLogger(__name__)`) |
+| `configure_logger(level=..., fmt=...)` | removed — configure stdlib logging in your app (`logging.basicConfig` / dictConfig) |
+| auto-configured stderr JSON sink on `import streamgate` | gone — unconfigured hosts see WARNING+ only via stdlib `lastResort`; add `logging.basicConfig(level=logging.INFO, ...)` to restore visibility |
+| JSON parsing of the library's stderr output (`{"timestamp", "level", "event", ...extra}`) | attach an equivalent stdlib JSON formatter yourself (recipe below) |
+| per-level control of library logs | `logging.getLogger("streamgate").setLevel(logging.WARNING)` (or any subtree) |
+
+Equivalent JSON formatter recipe (fields `timestamp` / `level` / `event` plus
+flat extras, UTC ISO-8601):
+
+```python
+import json
+import logging
+
+
+class JsonFormatter(logging.Formatter):
+    def format(self, record: logging.LogRecord) -> str:
+        entry = {
+            "timestamp": self.formatTime(record, datefmt="%Y-%m-%dT%H:%M:%S%z"),
+            "level": record.levelname,
+            "event": record.getMessage(),
+        }
+        reserved = set(logging.LogRecord("", 0, "", 0, "", (), None).__dict__) | {
+            "asctime", "message", "taskName",
+        }
+        for key, value in record.__dict__.items():
+            if key not in reserved and key not in entry:
+                entry[key] = value
+        return json.dumps(entry, ensure_ascii=False, default=str)
+
+
+handler = logging.StreamHandler()
+handler.setFormatter(JsonFormatter())
+logging.getLogger("streamgate").addHandler(handler)
+logging.getLogger("streamgate").setLevel(logging.INFO)
+```
+
+Notes:
+
+- `streamgate.obs.logging.emit` is the new internal emission helper; passing a
+  structured field whose name collides with a `LogRecord` reserved attribute
+  (`message`, `exc_info`, `asctime`, ...) raises `ValueError` (fail-fast) —
+  rename such fields. The one internal `exc_info=True` site now uses stdlib's
+  native `exc_info` parameter.
+- `LoggingMetricsSink` keeps its level-string dispatch (`debug` / `info` /
+  `warning` / `error`, default `info`) and now writes through stdlib logging.
+
+### BREAKING — environment-variable fallbacks removed
+
+The library no longer reads environment variables. Configuration has exactly
+two sources of truth: **required settings are true required constructor
+arguments** — missing one raises a native Python `TypeError` (missing
+argument), never a silently guessed default such as `kafka:9092` — and
+**optional settings hold their built-in defaults directly** on the options
+objects (`RuntimeTuning` fields are plain `int` / `float` / `str`, not
+`int | None` placeholders). Every `KAFKA__*` / `CONSUMER__*` / `METRICS__*` /
+`BACKPRESSURE__*` key is gone. Hosts that injected configuration via the
+environment (K8s ConfigMap, docker-compose) must resolve it themselves and
+pass the values explicitly after upgrading: a missing required argument now
+fails loudly at construction instead of silently connecting to the wrong
+cluster, and optional knobs fall back to the built-in defaults instead of the
+env value. Migrate key by key:
+
+| Old behavior (≤2.x and earlier in this release) | 3.0.0 |
+|---|---|
+| `Producer(bootstrap_servers=None → KAFKA__BOOTSTRAP_SERVERS → "kafka:9092")` | `Producer(bootstrap_servers, topic, key=None, options=...)` — both truly required, no default |
+| `Consumer(topic=None → KAFKA__TOPIC fail-fast ValueError)` | `Consumer(bootstrap_servers, topic, group_id, handler)` — all truly required; missing ⇒ native `TypeError` |
+| `DlqOptions.enabled=None → CONSUMER__DLQ_ENABLED → True` | `enabled: bool = True` |
+| `DlqOptions.send_retries=None → CONSUMER__DLQ_SEND_RETRIES → 3` | `send_retries: int = 3` |
+| `DlqOptions.topic=None → KAFKA__DLQ_TOPIC` | `topic: str \| None = None` — still required when DLQ is enabled (runtime validation preserved) |
+| `RuntimeTuning` 9 fields `None → CONSUMER__* → default` | fields hold defaults directly (`max_retries=3`, `retry_backoff_base=1.0`, `reconnect_base=1.0`, `reconnect_max=30.0`, `max_poll_records=500`, `session_timeout_ms=30000`, `max_poll_interval_ms=300000`, `auto_offset_reset="earliest"`, `backlog_check_interval=30.0`) |
+| `METRICS__WINDOW_SECONDS` | `metrics_window_seconds=60` passed explicitly on `ProducerOptions` / `ConsumerOptions` |
+| `CONSUMER__BATCH_SIZE` / `CONSUMER__FLUSH_TIMEOUT_SECONDS` | `batch_size` / `flush_timeout` explicit arguments (defaults unchanged: `500` / `5.0`) |
+| `SqliteConsumer(...) / MssqlConsumer(...)` identity arguments optional | `bootstrap_servers` / `topic` / `group_id` are required keyword arguments, mirroring the core `Consumer` |
+| `streamgate.consumer.options.resolve_tuning` / `ResolvedRuntimeTuning` | removed (internal resolution layer); `RuntimeTuning` holds concrete values and its own `backoff_seconds()` |
+
+Notes:
+
+- All runtime validations are preserved (not fallbacks): DLQ topic required
+  when DLQ is enabled, metrics window range 1–600, `batch_size >= 1`,
+  `flush_timeout > 0`.
+- `KafkaConfig` / `BackpressureConfig` field defaults are unchanged — they
+  are pure config objects and never read the environment; the `Producer`
+  constructor values always override `KafkaConfig.bootstrap_servers` /
+  `.topic`.
+
 ## [2.1.0] - 2026-09-11
 
 ### Added

streamgate-3.1.0/CONFIGURATION.md

@@ -0,0 +1,217 @@
+# Configuration reference
+
+streamgate components are configured with typed parameters and Pydantic
+objects. There are exactly **two sources of truth**:
+
+1. **Required settings are true required constructor arguments** —
+   `Producer(bootstrap_servers, topic, ...)` and
+   `Consumer(bootstrap_servers, topic, group_id, handler)`. Missing one fails
+   with a native Python `TypeError` (missing argument) at construction. The
+   library never reads environment variables and never silently defaults a
+   required setting.
+2. **Optional settings directly hold their built-in defaults** — advanced
+   knobs fold into `ProducerOptions` / `ConsumerOptions` (dedup, backpressure,
+   DLQ, tuning, metrics window); omit them and the documented defaults apply.
+
+If your host wants env-driven configuration, resolve the environment yourself
+and pass the values explicitly (the examples do exactly that with
+`os.environ.get(..., default)`).
+
+Database and Redis connection settings are **not part of the core framework**:
+storage carriers are injected and their configuration belongs to your
+application. The official strategy implementations under `streamgate.contrib`
+ship typed config objects (below) that you construct and pass to the factories.
+
+## Logging (not a configuration surface)
+
+streamgate emits logs through the standard-library `logging` module and ships
+**no logging configuration** — the library only emits records under the
+`streamgate.*` logger tree; level, handlers and formats are configured by the
+host application (e.g. `logging.getLogger("streamgate")`).
+
+For structured output, attach the built-in `JsonFormatter`
+(`from streamgate import JsonFormatter`, also exported from
+`streamgate.obs`): one JSON line per record with `timestamp` (UTC
+ISO-8601, `Z` suffix) / `level` / `logger` / `event` base fields, every flat
+extra field, and `error` when the record carries `exc_info`:
+
+```python
+import logging
+from streamgate import JsonFormatter
+
+handler = logging.StreamHandler()      # stderr
+handler.setFormatter(JsonFormatter())
+logging.basicConfig(level=logging.INFO, handlers=[handler])
+```
+
+Rendering-side notes: the formatter is generic (any logger's records render
+correctly — records without extras produce just the base fields) and never
+raises on `LogRecord` reserved-attribute collisions (host-side extras are
+skipped; only the library's own `emit()` fails fast on them). The reserved
+attribute set is shared as `streamgate.obs.logging.RESERVED_RECORD_ATTRS`.
+
+---
+
+## Consumer (flat arguments)
+
+| Argument | Default | Description |
+|----------|---------|-------------|
+| `bootstrap_servers` | **required** | Comma-separated broker list. Missing ⇒ native `TypeError`. |
+| `topic` | **required** | Topic to consume from. Missing ⇒ native `TypeError`. |
+| `group_id` | **required** | Kafka consumer group id. Missing ⇒ native `TypeError`. |
+| `handler` | **required** | Async callable `handler(batch, context)` — the only outlet. |
+| `batch_size` | `500` | Records buffered before a flush. `1` = single-record real-time. |
+| `flush_timeout` | `5.0` | Max wait before a partially filled batch is flushed. |
+| `options` | `None` | `ConsumerOptions` — everything below. |
+
+## ConsumerOptions
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `expected_type` | `None` | Envelope `type` validation (`None` = accept any; mismatched records are quarantined to the DLQ). |
+| `probe` | `None` | Single-record callable with handler-equivalent, idempotent semantics. Provided ⇒ POISON batches are located record-by-record and bad records quarantined precisely; omitted ⇒ the whole batch is quarantined. |
+| `classifier` | `DefaultErrorClassifier` | RETRY/POISON/FATAL mapping for handler exceptions. Connecting a DB? Inject a DB-aware classifier (`contrib.sql_upsert.SQLAlchemyErrorClassifier`). |
+| `persist_hook` | `None` | `DedupCarrier` hooked into the consume loop: `on_persisted` runs after each successful batch (dedup linkage with the producer side). Lifecycle is framework-managed. |
+| `collapse_key` | `None` | In-batch dedup key (keeps the last record) applied to hook notifications. |
+| `log_context` | `None` | Per-record extra fields for quarantine logs. |
+| `codec` | `JsonEnvelopeCodec` | Message envelope codec. |
+| `health_probe` | `None` | Side-channel health component; implements `check_health()` ⇒ observed in snapshots; `start()`/`close()` ⇒ lifecycle managed. |
+| `backlog_ttl_seconds` | `18000` | Backlog alert budget: WARN > TTL/2, ERROR > TTL×0.8. |
+| `metrics_window_seconds` | `60` | Sliding-window length (1–600) for rate/latency snapshot fields; out-of-range values fail at construction. |
+| `dlq` | `DlqOptions()` | DLQ options (below). |
+| `tuning` | `RuntimeTuning()` | Runtime tuning (below). |
+
+## DlqOptions
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `enabled` | `true` | Master switch; `false` falls back to the paused behavior (emergency escape hatch). |
+| `topic` | `None` | Dead-letter topic; required when DLQ is enabled (validated at DLQ-producer construction). |
+| `message_type` | `streamgate_dlq` | Envelope `type` of DLQ messages. |
+| `send_retries` | `3` | Total attempts per DLQ send (exhausting them pauses the batch). |
+
+## RuntimeTuning
+
+Every field directly holds its built-in default:
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `max_retries` | `3` | Handler retries per batch. |
+| `retry_backoff_base` | `1.0` | Retry backoff base (seconds, exponential). |
+| `reconnect_base` | `1.0` | Reconnect/paused-recovery backoff base (seconds). |
+| `reconnect_max` | `30.0` | Reconnect backoff cap (seconds). |
+| `max_poll_records` | `500` | Max records per poll. |
+| `session_timeout_ms` | `30000` | Kafka session timeout. |
+| `max_poll_interval_ms` | `300000` | Max time between polls before rebalance. |
+| `auto_offset_reset` | `earliest` | Offset reset policy. |
+| `backlog_check_interval` | `30.0` | Backlog age check period (seconds). |
+
+---
+
+## Contrib strategy configs
+
+These are plain constructor arguments. All belong to `streamgate.contrib`
+packages — install the matching extra first (`[redis]` / `[sql]` / `[http]`).
+
+### `contrib.redis_dedup.RedisConfig`
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `url` | `redis://localhost:6379/0` | Redis connection URL. |
+| `key_prefix` | `streamgate:` | Key prefix for dedup keys (`<prefix>dedup:<identity>`). |
+| `identity_ttl_seconds` | `18000` | Single-key placeholder/summary TTL (idle-GC heartbeat: every write renews it). |
+| `group_key_prefix` | `dedup_group:` | Group-carrier key prefix (`<prefix>dedup_group:<group>` HASH; type-isolated from the single-key STRING). |
+| `group_ttl_seconds` | `18000` | Group HASH TTL (idle-GC heartbeat: any group write renews it). |
+| `socket_timeout_ms` | `1000` | Read/query path timeout. |
+| `recv_timeout_ms` | `500` | Push path (reserve/summary write) timeout. |
+
+Strategy-level knobs live on `RedisDedupCarrierConfig`
+(`fail_closed_on_unavailable`, `cold_path_max_concurrency`, gate-full /
+unavailable `retry_after` seconds, `log_context` mapping) and on
+`RedisGroupDedupCarrierConfig` (`group_cold_path_max_concurrency`, same
+family of knobs).
+
+### `contrib.sql_upsert.DbConfig`
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `connection_string` | **required** | SQLAlchemy async URL (`sqlite+aiosqlite:///...` or `mssql+aioodbc://...`). Missing ⇒ assembly error with fix instructions. |
+| `echo` | `false` | SQLAlchemy echo logging. |
+| `write_timeout_seconds` | `20` | Driver statement timeout (MSSQL/pyodbc hook; must be < `write_wait_seconds`). |
+| `write_wait_seconds` | `25` | Batch upsert call-level `wait_for` ceiling. |
+| `pool_timeout_seconds` | `3` | Pool checkout timeout (fail fast when exhausted). |
+| `write_pool_size` | `10` | Fixed write-pool connections. |
+| `write_pool_max_overflow` | `20` | Write-pool overflow connections. |
+
+The dialect (SQLite vs MSSQL) is selected by the connection string;
+`contrib.sqlite_upsert` and `contrib.mssql_upsert` are the pre-assembled
+factory entries (`SqliteConsumer` / `MssqlConsumer`).
+
+### Backpressure probing (contrib.http_probe)
+
+`HttpProbeSignal` is constructed with a `BackpressureConfig` instance (see
+below) — build the config object yourself and pass it to the producer via
+`ProducerOptions(signal=HttpProbeSignal(config), backpressure=config)`.
+
+---
+
+## KafkaConfig (`ProducerOptions.kafka`)
+
+The producer takes `bootstrap_servers`/`topic` as **required** flat
+constructor arguments; the remaining connection/self-healing knobs live on
+`KafkaConfig`, injectable via `ProducerOptions.kafka` (defaults below). The
+constructor values always override the `bootstrap_servers` / `topic` fields.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `bootstrap_servers` | `kafka:9092` | Comma-separated broker list. `http://` / `https://` / `kafka://` scheme prefixes are stripped automatically. Always overridden by the `Producer` constructor value. |
+| `topic` | `None` | Topic to push to (constructor value wins; validated when a raw `KafkaConfig` is used directly). |
+| `acks` | `all` | Producer acks level. |
+| `request_timeout_ms` | `10000` | Producer request timeout (ms). |
+| `enable_idempotence` | `true` | Idempotent producer (safe retries). |
+| `health_check_interval_seconds` | `30.0` | Periodic health probe interval; unhealthy instances are rebuilt. |
+| `reconnect_base_backoff_seconds` | `1.0` | Reconnect exponential backoff base. |
+| `reconnect_max_backoff_seconds` | `30.0` | Reconnect backoff cap. |
+| `reconnect_failure_threshold` | `1` | Consecutive send failures before a rebuild is triggered (raise to tolerate broker jitter). |
+| `send_failure_window_seconds` | `30.0` | A send failure within this window marks the instance unhealthy. |
+| `unhealthy_check_interval_seconds` | `5.0` | High-frequency probe interval while unhealthy/rebuilding. |
+
+The dead-letter topic is configured on the consumer side via
+`DlqOptions.topic` (see above) — no longer on `KafkaConfig`.
+
+## BackpressureConfig (`ProducerOptions.backpressure`)
+
+The producer defaults to the built-in `ManualBackpressureSignal` (static
+switch, backpressure off). For dynamic probing pass
+`options=ProducerOptions(backpressure=..., signal=...)` —
+`contrib.http_probe.HttpProbeSignal` reads this config object.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `enabled` | `true` | Master switch; `false` = fully open (emergency rollback). |
+| `consumer_health_url` | `http://localhost:9109/health` | Consumer health endpoint polled by the probe. |
+| `check_interval_seconds` | `30.0` | Poll period while OPEN. |
+| `timeout_seconds` | `2.0` | Per-probe timeout. |
+| `probe_retries` | `3` | Retries per cycle (excluding the first attempt); `0` disables. |
+| `probe_retry_interval_seconds` | `15.0` | Interval between retries. |
+| `trip_seconds` | `9000.0` | Reject when backlog age exceeds this. |
+| `recover_seconds` | `7200.0` | Resume when backlog age falls below this (hysteresis anti-flapping). |
+| `retry_after_seconds` | `60` | `Retry-After` hint (seconds) while rejecting. |
+| `fail_closed_on_unreachable` | `true` | Treat unreachable probe (retries exhausted) as backlog-exceeded. |
+| `reject_on_any_degraded` | `false` | Legacy escape hatch: reject when any component is degraded. |
+| `unhealthy_check_interval_seconds` | `5.0` | Poll period while REJECTING. |
+
+## Metrics window
+
+Health-snapshot rate metrics: both `Producer` and `Consumer` maintain
+in-memory sliding windows and expose computed rates / latencies through their
+health snapshots (`push_rate`, `handle_rate`, `produce_latency_ms_avg`,
+...). Windowed aggregates are computed on read — no background tasks, no extra
+endpoint; rates decay to `0.0` once traffic stops for longer than the window.
+
+Configure via `metrics_window_seconds` on `ProducerOptions` /
+`ConsumerOptions`:
+
+| Value | Description |
+|-------|-------------|
+| `60` (default) | Sliding-window length (seconds) for all rate/latency fields. Valid range 1–600; out-of-range values fail at construction. Short windows react faster but are noisier. |

{streamgate-2.1.0 → streamgate-3.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: streamgate
-Version: 2.1.0
+Version: 3.1.0
 Summary: A pure-core Kafka data pipeline framework: conditional admission, reliable delivery, and one obvious consume-loop outlet via a typed handler.
 Project-URL: Homepage, https://github.com/pwg-code/streamgate
 Project-URL: Repository, https://github.com/pwg-code/streamgate
@@ -21,7 +21,6 @@ Classifier: Topic :: System :: Distributed Computing
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
 Requires-Dist: aiokafka>=0.10
-Requires-Dist: loguru>=0.7
 Requires-Dist: pydantic>=2.5
 Provides-Extra: http
 Requires-Dist: httpx>=0.27; extra == 'http'
@@ -45,7 +44,7 @@ English | [中文](README.zh-CN.md)
 
 **streamgate** is a pure-core Kafka data pipeline framework: a dedup-aware door on the way in, reliable delivery through Kafka, and one obvious outlet on the other side — `Producer(bootstrap_servers, topic, ...)` / `Consumer(bootstrap_servers, topic, group_id, handler)`.
 
-The wheel installs exactly three dependencies (`aiokafka`, `loguru`, `pydantic`) and the core contains **zero database, Redis or HTTP-client code**. Official I/O strategy implementations live in [`streamgate.contrib`](#contrib-official-strategy-implementations) — opt-in via extras — and runnable demos live in [`examples/`](examples/).
+The wheel installs exactly two dependencies (`aiokafka`, `pydantic`) and the core contains **zero database, Redis or HTTP-client code**. Logging goes through the standard-library `logging` module — the library emits records and never configures them. Official I/O strategy implementations live in [`streamgate.contrib`](#contrib-official-strategy-implementations) — opt-in via extras — and runnable demos live in [`examples/`](examples/).
 
 - 中文提示：本仓库对外文档以英文为主；内部代码注释保留中文。
 
@@ -68,6 +67,8 @@ aiokafka gives you **transport**; streamgate gives you the **operating semantics
 
 A complete produce → Kafka → consume → SQLite round trip that runs on a **bare install** (stdlib outlet, zero extra packages) — [`examples/pure_pipeline/`](examples/pure_pipeline/):
 
+> Logging note: streamgate emits via stdlib `logging` and never configures it. Without host configuration only WARNING+ shows (stdlib `lastResort`); wire the built-in [`JsonFormatter`](#logging) at your entry point (3 lines, above) to see INFO logs with every structured field rendered as JSON (the examples all do this — see [Logging](#logging)).
+
 ```python
 import asyncio
 from pydantic import BaseModel
@@ -218,7 +219,7 @@ await consumer.run()
 ## Installation
 
 ```bash
-pip install streamgate             # aiokafka + loguru + pydantic. That's it.
+pip install streamgate             # aiokafka + pydantic. That's it.
 pip install "streamgate[redis]"    # + Redis distributed dedup carrier
 pip install "streamgate[sql]"      # + SQL upsert outlet (SQLite + MSSQL)
 pip install "streamgate[http]"     # + HTTP probe backpressure
@@ -228,7 +229,29 @@ For anything else (PostgreSQL, Elasticsearch, your own destination...), write th
 
 ## Configuration
 
-Both sides take required settings as flat constructor arguments with environment-variable fallbacks (`KAFKA__BOOTSTRAP_SERVERS`, `KAFKA__TOPIC`, `CONSUMER__GROUP_ID`); advanced knobs fold into `ProducerOptions` / `ConsumerOptions` (dedup, backpressure, DLQ, tuning, metrics window) with the same env fallbacks. Required settings fail fast at startup with fix instructions. Database/Redis connection settings belong to your application (see the examples' local configs). See [CONFIGURATION.md](CONFIGURATION.md).
+There are exactly two sources of truth. **Required settings are true required constructor parameters** — `Producer(bootstrap_servers, topic, ...)` and `Consumer(bootstrap_servers, topic, group_id, handler)` fail with a native `TypeError` the moment an argument is missing (they are never silently defaulted or read from the environment). **Optional settings directly hold their built-in defaults** and fold into `ProducerOptions` / `ConsumerOptions` (dedup, backpressure, DLQ, tuning, metrics window) — omit them and the documented defaults apply. The library reads no environment variables; if your host wants env-driven config, resolve it yourself and pass the values explicitly (the examples do exactly that). Database/Redis connection settings belong to your application (see the examples' local configs). See [CONFIGURATION.md](CONFIGURATION.md).
+
+## Logging
+
+streamgate logs through the standard-library `logging` module — **the library emits records and never configures logging** (no handlers added, no levels changed at import). Every logger lives under the `streamgate` namespace (`streamgate.ingest.producer`, `streamgate.consumer.loop`, ...), so the whole tree is controlled by the host. Event names (`batch_handle_start`, `poison_message_skipped`, ...) are a stable public contract; structured fields travel as flat `LogRecord` extras.
+
+### JSON output: built-in `JsonFormatter`
+
+stdlib's default formatter does **not** render `LogRecord` extras — with `basicConfig(format="%(message)s")` a batch consumer logs just `batch_handle_success`, hiding `batch_size` / `duration_ms`. Use the built-in `JsonFormatter` to make every structured field visible (this is what all examples do):
+
+```python
+import logging
+
+from streamgate import JsonFormatter   # equivalently: from streamgate.obs import JsonFormatter
+
+handler = logging.StreamHandler()      # stderr — same destination as 2.x
+handler.setFormatter(JsonFormatter())
+logging.basicConfig(level=logging.INFO, handlers=[handler])
+```
+
+Each record renders as one JSON line: `timestamp` (UTC ISO-8601 with `Z` suffix, taken from the `LogRecord` clock), `level`, `logger` (the hierarchical namespace — a 3.0+ extra versus 2.x), `event` (the stable event name; host `%s`-style messages render the same way), then every flat extra field (`batch_size`, `duration_ms`, `partition`, `offset`, ...), plus `error` (formatted traceback) when a record carries `exc_info`. The formatter is a generic component — any logger renders correctly through it: records without extras produce just the base fields, and reserved-attribute collisions are skipped, never raised.
+
+Attach the handler to the root logger as above and host + streamgate logs share one JSON format; attach the same way to `logging.getLogger("streamgate")` instead and only the streamgate tree is formatted. Level control is orthogonal: `logging.getLogger("streamgate").setLevel(logging.WARNING)` silences INFO noise regardless of formatter. Unconfigured hosts still see WARNING+ only (stdlib `lastResort` on stderr).
 
 ## Examples
 
@@ -236,7 +259,7 @@ See [`examples/`](examples/) for five runnable pipelines plus a `docker-compose.
 
 ## Migrating from 0.x
 
-1.0.0 redesigned **both** sides (breaking): the consumer side replaces `ConsumerWorker` + `ConsumeSpec` + `RecordWriter` with the flat `Consumer` constructor; the producer side replaces `IngestBinding` + `IngestGateway` with the flat `Producer` constructor (`process()` → `push()`, `IngestOutcome` → `PushResult`, `overwrite` → `force`, `admission` → `DedupOptions`, `entity_key`/`slot_key` split into routing `key` + identity `DedupOptions.key`). See the [CHANGELOG](CHANGELOG.md) for the complete old→new mapping (API, result kinds, metrics names, log events, env keys, import paths).
+1.0.0 redesigned **both** sides (breaking): the consumer side replaces `ConsumerWorker` + `ConsumeSpec` + `RecordWriter` with the flat `Consumer` constructor; the producer side replaces `IngestBinding` + `IngestGateway` with the flat `Producer` constructor (`process()` → `push()`, `IngestOutcome` → `PushResult`, `overwrite` → `force`, `admission` → `DedupOptions`, `entity_key`/`slot_key` split into routing `key` + identity `DedupOptions.key`). 3.0.0 removed the bundled loguru logging (stdlib `logging` instead — `from streamgate import logger` is gone) **and all environment-variable fallbacks**: required settings are now true required constructor parameters and every `KAFKA__*` / `CONSUMER__*` / `METRICS__*` key is gone — options hold their built-in defaults directly. See the [CHANGELOG](CHANGELOG.md) for the complete old→new mapping (API, result kinds, metrics names, log events, import paths) and the per-key migration table.
 
 ## Roadmap
 

{streamgate-2.1.0 → streamgate-3.1.0}/README.md

@@ -9,7 +9,7 @@ English | [中文](README.zh-CN.md)
 
 **streamgate** is a pure-core Kafka data pipeline framework: a dedup-aware door on the way in, reliable delivery through Kafka, and one obvious outlet on the other side — `Producer(bootstrap_servers, topic, ...)` / `Consumer(bootstrap_servers, topic, group_id, handler)`.
 
-The wheel installs exactly three dependencies (`aiokafka`, `loguru`, `pydantic`) and the core contains **zero database, Redis or HTTP-client code**. Official I/O strategy implementations live in [`streamgate.contrib`](#contrib-official-strategy-implementations) — opt-in via extras — and runnable demos live in [`examples/`](examples/).
+The wheel installs exactly two dependencies (`aiokafka`, `pydantic`) and the core contains **zero database, Redis or HTTP-client code**. Logging goes through the standard-library `logging` module — the library emits records and never configures them. Official I/O strategy implementations live in [`streamgate.contrib`](#contrib-official-strategy-implementations) — opt-in via extras — and runnable demos live in [`examples/`](examples/).
 
 - 中文提示：本仓库对外文档以英文为主；内部代码注释保留中文。
 
@@ -32,6 +32,8 @@ aiokafka gives you **transport**; streamgate gives you the **operating semantics
 
 A complete produce → Kafka → consume → SQLite round trip that runs on a **bare install** (stdlib outlet, zero extra packages) — [`examples/pure_pipeline/`](examples/pure_pipeline/):
 
+> Logging note: streamgate emits via stdlib `logging` and never configures it. Without host configuration only WARNING+ shows (stdlib `lastResort`); wire the built-in [`JsonFormatter`](#logging) at your entry point (3 lines, above) to see INFO logs with every structured field rendered as JSON (the examples all do this — see [Logging](#logging)).
+
 ```python
 import asyncio
 from pydantic import BaseModel
@@ -182,7 +184,7 @@ await consumer.run()
 ## Installation
 
 ```bash
-pip install streamgate             # aiokafka + loguru + pydantic. That's it.
+pip install streamgate             # aiokafka + pydantic. That's it.
 pip install "streamgate[redis]"    # + Redis distributed dedup carrier
 pip install "streamgate[sql]"      # + SQL upsert outlet (SQLite + MSSQL)
 pip install "streamgate[http]"     # + HTTP probe backpressure
@@ -192,7 +194,29 @@ For anything else (PostgreSQL, Elasticsearch, your own destination...), write th
 
 ## Configuration
 
-Both sides take required settings as flat constructor arguments with environment-variable fallbacks (`KAFKA__BOOTSTRAP_SERVERS`, `KAFKA__TOPIC`, `CONSUMER__GROUP_ID`); advanced knobs fold into `ProducerOptions` / `ConsumerOptions` (dedup, backpressure, DLQ, tuning, metrics window) with the same env fallbacks. Required settings fail fast at startup with fix instructions. Database/Redis connection settings belong to your application (see the examples' local configs). See [CONFIGURATION.md](CONFIGURATION.md).
+There are exactly two sources of truth. **Required settings are true required constructor parameters** — `Producer(bootstrap_servers, topic, ...)` and `Consumer(bootstrap_servers, topic, group_id, handler)` fail with a native `TypeError` the moment an argument is missing (they are never silently defaulted or read from the environment). **Optional settings directly hold their built-in defaults** and fold into `ProducerOptions` / `ConsumerOptions` (dedup, backpressure, DLQ, tuning, metrics window) — omit them and the documented defaults apply. The library reads no environment variables; if your host wants env-driven config, resolve it yourself and pass the values explicitly (the examples do exactly that). Database/Redis connection settings belong to your application (see the examples' local configs). See [CONFIGURATION.md](CONFIGURATION.md).
+
+## Logging
+
+streamgate logs through the standard-library `logging` module — **the library emits records and never configures logging** (no handlers added, no levels changed at import). Every logger lives under the `streamgate` namespace (`streamgate.ingest.producer`, `streamgate.consumer.loop`, ...), so the whole tree is controlled by the host. Event names (`batch_handle_start`, `poison_message_skipped`, ...) are a stable public contract; structured fields travel as flat `LogRecord` extras.
+
+### JSON output: built-in `JsonFormatter`
+
+stdlib's default formatter does **not** render `LogRecord` extras — with `basicConfig(format="%(message)s")` a batch consumer logs just `batch_handle_success`, hiding `batch_size` / `duration_ms`. Use the built-in `JsonFormatter` to make every structured field visible (this is what all examples do):
+
+```python
+import logging
+
+from streamgate import JsonFormatter   # equivalently: from streamgate.obs import JsonFormatter
+
+handler = logging.StreamHandler()      # stderr — same destination as 2.x
+handler.setFormatter(JsonFormatter())
+logging.basicConfig(level=logging.INFO, handlers=[handler])
+```
+
+Each record renders as one JSON line: `timestamp` (UTC ISO-8601 with `Z` suffix, taken from the `LogRecord` clock), `level`, `logger` (the hierarchical namespace — a 3.0+ extra versus 2.x), `event` (the stable event name; host `%s`-style messages render the same way), then every flat extra field (`batch_size`, `duration_ms`, `partition`, `offset`, ...), plus `error` (formatted traceback) when a record carries `exc_info`. The formatter is a generic component — any logger renders correctly through it: records without extras produce just the base fields, and reserved-attribute collisions are skipped, never raised.
+
+Attach the handler to the root logger as above and host + streamgate logs share one JSON format; attach the same way to `logging.getLogger("streamgate")` instead and only the streamgate tree is formatted. Level control is orthogonal: `logging.getLogger("streamgate").setLevel(logging.WARNING)` silences INFO noise regardless of formatter. Unconfigured hosts still see WARNING+ only (stdlib `lastResort` on stderr).
 
 ## Examples
 
@@ -200,7 +224,7 @@ See [`examples/`](examples/) for five runnable pipelines plus a `docker-compose.
 
 ## Migrating from 0.x
 
-1.0.0 redesigned **both** sides (breaking): the consumer side replaces `ConsumerWorker` + `ConsumeSpec` + `RecordWriter` with the flat `Consumer` constructor; the producer side replaces `IngestBinding` + `IngestGateway` with the flat `Producer` constructor (`process()` → `push()`, `IngestOutcome` → `PushResult`, `overwrite` → `force`, `admission` → `DedupOptions`, `entity_key`/`slot_key` split into routing `key` + identity `DedupOptions.key`). See the [CHANGELOG](CHANGELOG.md) for the complete old→new mapping (API, result kinds, metrics names, log events, env keys, import paths).
+1.0.0 redesigned **both** sides (breaking): the consumer side replaces `ConsumerWorker` + `ConsumeSpec` + `RecordWriter` with the flat `Consumer` constructor; the producer side replaces `IngestBinding` + `IngestGateway` with the flat `Producer` constructor (`process()` → `push()`, `IngestOutcome` → `PushResult`, `overwrite` → `force`, `admission` → `DedupOptions`, `entity_key`/`slot_key` split into routing `key` + identity `DedupOptions.key`). 3.0.0 removed the bundled loguru logging (stdlib `logging` instead — `from streamgate import logger` is gone) **and all environment-variable fallbacks**: required settings are now true required constructor parameters and every `KAFKA__*` / `CONSUMER__*` / `METRICS__*` key is gone — options hold their built-in defaults directly. See the [CHANGELOG](CHANGELOG.md) for the complete old→new mapping (API, result kinds, metrics names, log events, import paths) and the per-key migration table.
 
 ## Roadmap
 

{streamgate-2.1.0 → streamgate-3.1.0}/README.zh-CN.md

@@ -14,7 +14,7 @@
               判重、背压、自愈                    批量缓冲、重试、坏数据隔离
 ```
 
-框架核心只装 3 个依赖（`aiokafka` / `loguru` / `pydantic`），核心层**没有一行数据库、Redis、HTTP 客户端代码**。官方 I/O 策略实现收录在 [`streamgate.contrib`](#不想自己写用-contrib)——按 extras 按需安装、装完即可 import；可运行的完整演示在 [`examples/`](examples/)。
+框架核心只装 2 个依赖（`aiokafka` / `pydantic`），核心层**没有一行数据库、Redis、HTTP 客户端代码**。日志走标准库 `logging`——库只发日志记录、永不配置。官方 I/O 策略实现收录在 [`streamgate.contrib`](#不想自己写用-contrib)——按 extras 按需安装、装完即可 import；可运行的完整演示在 [`examples/`](examples/)。
 
 ---
 
@@ -358,7 +358,7 @@ await consumer.run()
 ## 安装
 
 ```bash
-pip install streamgate             # 只装 aiokafka + loguru + pydantic
+pip install streamgate             # 只装 aiokafka + pydantic
 pip install "streamgate[redis]"    # + Redis 分布式判重
 pip install "streamgate[sql]"      # + SQL 幂等出口（SQLite + MSSQL）
 pip install "streamgate[http]"     # + HTTP 探活背压
@@ -368,7 +368,29 @@ pip install "streamgate[http]"     # + HTTP 探活背压
 
 ## 配置
 
-`Producer` / `Consumer` 必填项直接传参，未传时回退同名环境变量（`KAFKA__BOOTSTRAP_SERVERS` / `KAFKA__TOPIC` / `CONSUMER__GROUP_ID`）；高级项收口在 `ProducerOptions` / `ConsumerOptions`（判重、背压、DLQ、调优、指标窗口等），同样跟随 `KAFKA__*` / `CONSUMER__*` 环境变量回退。必填项缺失即启动失败并附修复指引，绝不带猜测默认值上路。数据库/Redis 连接配置归你的应用管（示例各自带本地配置）。详见 [CONFIGURATION.md](CONFIGURATION.md)。
+配置只有两个真相来源：**必填项是真必填构造参数**——`Producer(bootstrap_servers, topic, ...)` 与 `Consumer(bootstrap_servers, topic, group_id, handler)` 缺任一参数即原生 `TypeError`（绝不静默默认、绝不读环境变量）；**可选项直接持有内置默认值**，收口在 `ProducerOptions` / `ConsumerOptions`（判重、背压、DLQ、调优、指标窗口等），不传即文档默认行为。库代码不读取任何环境变量；宿主若要 env 驱动配置，自行解析后显式传参（示例即如此）。数据库/Redis 连接配置归你的应用管（示例各自带本地配置）。详见 [CONFIGURATION.md](CONFIGURATION.md)。
+
+## 日志
+
+streamgate 走标准库 `logging`——**库只发日志记录、永不配置日志**（import 时不增删任何 handler、不改任何级别）。所有 logger 挂在 `streamgate` 命名空间下（`streamgate.ingest.producer`、`streamgate.consumer.loop` 等），宿主整树控制。事件名（`batch_handle_start`、`poison_message_skipped` 等）是稳定的对外契约；结构化字段以 `LogRecord` extra 平铺。
+
+### JSON 输出：内置 `JsonFormatter`
+
+stdlib 默认 Formatter **不渲染** `LogRecord` extra——用 `basicConfig(format="%(message)s")` 时，批量消费只会打出裸事件名 `batch_handle_success`，看不到 `batch_size` / `duration_ms`。接内置 `JsonFormatter` 即可让全部结构化字段可见（所有示例均如此）：
+
+```python
+import logging
+
+from streamgate import JsonFormatter   # 等价：from streamgate.obs import JsonFormatter
+
+handler = logging.StreamHandler()      # stderr，与 2.x 输出目标一致
+handler.setFormatter(JsonFormatter())
+logging.basicConfig(level=logging.INFO, handlers=[handler])
+```
+
+每条日志渲染为一行 JSON：`timestamp`（UTC ISO8601，`Z` 后缀，取自 LogRecord 时钟）、`level`、`logger`（层级命名空间，2.x 没有的新字段）、`event`（稳定事件名；宿主 `%s` 风格日志同样适用），其余键为全部平铺的结构化字段（`batch_size`、`duration_ms`、`partition`、`offset` 等），记录携带 `exc_info` 时附 `error`（格式化堆栈）。Formatter 是通用组件——任何 logger 经它渲染都正确：无 extra 的记录只输出基础字段，与 LogRecord 保留属性撞键时直接跳过、不抛错。
+
+按上面方式挂 root logger，宿主与 streamgate 日志统一为 JSON 格式；同样三行挂 `logging.getLogger("streamgate")` 则仅 streamgate 树生效。级别控制与 Formatter 正交：`logging.getLogger("streamgate").setLevel(logging.WARNING)` 可关掉 INFO 噪音。未配置日志的宿主仍只能看到 WARNING 及以上（stdlib `lastResort` 输出到 stderr）。
 
 ## 架构
 
@@ -396,7 +418,7 @@ pip install "streamgate[http]"     # + HTTP 探活背压
 
 ## 从 0.x 迁移
 
-1.0.0 对**两端**都做了破坏性重设计：消费侧 `ConsumerWorker` + `ConsumeSpec` + `RecordWriter` 由扁平 `Consumer` 构造器替代；生产侧 `IngestBinding` + `IngestGateway` 由扁平 `Producer` 构造器替代（`process()` → `push()`、`IngestOutcome` → `PushResult`、`overwrite` → `force`、`admission` → `DedupOptions`、`entity_key`/`slot_key` 拆分为路由键 `key` + 身份键 `DedupOptions.key`）。完整的新旧对照（API、结果 kind、指标名、日志事件、环境变量、import 路径）见 [CHANGELOG](CHANGELOG.md)。
+1.0.0 对**两端**都做了破坏性重设计：消费侧 `ConsumerWorker` + `ConsumeSpec` + `RecordWriter` 由扁平 `Consumer` 构造器替代；生产侧 `IngestBinding` + `IngestGateway` 由扁平 `Producer` 构造器替代（`process()` → `push()`、`IngestOutcome` → `PushResult`、`overwrite` → `force`、`admission` → `DedupOptions`、`entity_key`/`slot_key` 拆分为路由键 `key` + 身份键 `DedupOptions.key`）。3.0.0 移除了内置 loguru 日志（改走标准库 `logging`，`from streamgate import logger` 不复存在），**并移除了全部环境变量回退**：必填项成为真必填构造参数，所有 `KAFKA__*` / `CONSUMER__*` / `METRICS__*` 键不复存在——可选项字段直接持有内置默认值。完整的新旧对照（API、结果 kind、指标名、日志事件、import 路径）与逐项迁移表见 [CHANGELOG](CHANGELOG.md)。
 
 ## 路线图
 

{streamgate-2.1.0 → streamgate-3.1.0}/examples/http_probe/consume.py

@@ -8,11 +8,12 @@
 """
 
 import asyncio
+import logging
 import os
 
 from health_server import serve_consumer_health
 
-from streamgate import ConsumeContext, Consumer, JsonObject
+from streamgate import ConsumeContext, Consumer, JsonFormatter, JsonObject
 
 
 async def print_orders(
@@ -23,6 +24,9 @@ async def print_orders(
 
 
 async def main() -> None:
+    handler = logging.StreamHandler()
+    handler.setFormatter(JsonFormatter())
+    logging.basicConfig(level=logging.INFO, handlers=[handler])
     consumer = Consumer(
         bootstrap_servers=os.environ.get("KAFKA__BOOTSTRAP_SERVERS", "localhost:29092"),
         topic=os.environ.get("KAFKA__TOPIC", "orders"),