streamgate 0.3.0__tar.gz → 0.4.0__tar.gz

{streamgate-0.3.0 → streamgate-0.4.0}/.github/workflows/ci.yml

@@ -83,8 +83,9 @@ jobs:
           uv venv bare
           uv pip install --python bare/bin/python .
       - name: "Smoke: dependency tree is pure (no DB/redis/httpx carriers)"
+        # uv venv 不内置 pip：用 uv pip list 读取裸环境依赖清单
         run: |
-          bare/bin/python -m pip list --format=freeze > deps.txt
+          uv pip list --python bare/bin/python --format=freeze > deps.txt
           cat deps.txt
           for pkg in sqlalchemy sqlmodel redis aiosqlite aioodbc httpx; do
             if grep -qi "^${pkg}==" deps.txt; then

{streamgate-0.3.0 → streamgate-0.4.0}/AGENTS.md

@@ -10,3 +10,10 @@
 - 禁止手动逐字段映射：用 `model_validate()`（ORM 对象加 `from_attributes=True`）代替 `_to_response()` 和 `Schema(**obj)`；用 `model_dump(exclude=.../exclude_unset=True)` 代替手写字典；字段定义只出现在 Schema 一处。响应体 Schema 有默认值，请求体 Schema 保持必填。
 - 新增代码必须通过 pyright 静态检查。
 
+## 发布流程（PyPI）
+
+- PyPI Trusted Publisher 已配置（owner=pwg-code, repo=streamgate, workflow=release.yml, environment=pypi）。发布一律走 GitHub Actions 自动流程，禁止本地 `uv publish`/`twine upload`。
+- 步骤：更新 `src/streamgate/__init__.py` 的 `__version__` 与 `CHANGELOG.md`（两者必须与 tag 一致，workflow 会校验）→ commit → `git tag vX.Y.Z` → `git push origin main vX.Y.Z` → tag 推送自动触发 `.github/workflows/release.yml` 完成构建与发布。
+- 本地只做 `uv build` 验证，不承担发布职责。
+
+

{streamgate-0.3.0 → streamgate-0.4.0}/CHANGELOG.md

@@ -5,6 +5,61 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2026-09-08
+
+### Added
+
+- **Health snapshots now carry ops rate/latency metrics.** Both
+  `IngestHealthResponse` and `ConsumerHealthResponse` gain sliding-window
+  fields (all defaulting to `0.0`, computed on read over a 1-second bucket
+  sliding window — no background tasks, no extra endpoint):
+  - ingest: `receive_rate`, `admission_conflict_rate`,
+    `backpressure_reject_rate`, `produce_success_rate`,
+    `produce_failure_rate`, `produce_latency_ms_avg`,
+    `produce_latency_ms_max`;
+  - consume: `consume_rate`, `sink_write_rate`, `sink_write_failure_rate`,
+    `retry_rate`, `sink_write_latency_ms_avg`, `sink_write_latency_ms_max`.
+  Rates are records/second, latencies milliseconds; the window length is
+  configurable via the new `MetricsConfig` (`METRICS__WINDOW_SECONDS`,
+  default `60`, range 1–600). `IngestGateway(metrics_config=...)` and
+  `ConsumerWorker(metrics_config=...)` accept it; when omitted everything
+  works with the default window. Purely additive — old consumers that
+  deserialize the snapshot ignore the new fields.
+- **`MetricsConfig`** exported from the package root.
+
+### Changed
+
+- `ConsumeRuntime.__init__`: the structured-metrics sink parameter
+  `metrics` was renamed to `metrics_sink` to free the `metrics` name for
+  the new rate-metrics container (`ConsumeMetrics`). No in-repo caller
+  passed it explicitly; external callers using `metrics=<MetricsSink>`
+  must rename the keyword.
+
+## [0.3.1] - 2026-09-08
+
+### Fixed
+
+- **`ConsumerWorker` now manages the `ConsumeSpec.persist_policy` lifecycle**
+  (the `AdmissionPolicy` protocol promises "the framework guarantees call
+  ordering", but the consumer entry point never honored it — unlike
+  `IngestGateway`, which calls `start()`/`close()`). `prepare()` now calls
+  `await policy.start()` before starting the sink, and `shutdown()` calls
+  `await policy.close()` before closing the writer. Previously the consumer
+  side never started the policy, so `on_persisted` authoritative refresh /
+  TTL heartbeats failed on every batch (e.g. `RedisExistenceAdmission`
+  logging `cache_refresh_failed: RedisExistenceCache not started`) and the
+  failure surfaced only as per-record WARN logs. Lifecycle calls are
+  duck-typed (defensive `getattr`) so legacy policy implementations without
+  `start()`/`close()` keep working.
+- **`ConsumerWorker(cache=...)` lifecycle is now symmetric.** The bypass
+  `cache` parameter (health-probe component) is started in `prepare()` when
+  it implements `start()`, instead of being closed at shutdown without ever
+  having been started. `RedisExistenceCache.start()`/`close()` are
+  idempotent, so sharing one cache instance between the policy and the
+  bypass parameter remains safe (double start/close is a no-op).
+  The pre-fix workaround (`await cache.start()` before `worker.run()`)
+  is no longer required.
+
 ## [0.3.0] - 2026-09-08
 
 **Contrib release.** The production-grade I/O strategies that 0.2.0 demoted to

{streamgate-0.3.0 → streamgate-0.4.0}/CONFIGURATION.md

@@ -9,6 +9,7 @@ underscore):
 | `KAFKA__` | `KafkaConfig` |
 | `CONSUMER__` | `ConsumerConfig` |
 | `BACKPRESSURE__` | `BackpressureConfig` |
+| `METRICS__` | `MetricsConfig` |
 
 **Fail-fast semantics:** required settings are not defaulted. If a required value
 is missing, assembly fails at startup with an error that names the setting and
@@ -123,3 +124,16 @@ switch). For dynamic probing inject your own `BackpressureSignal` —
 | `BACKPRESSURE__FAIL_CLOSED_ON_UNREACHABLE` | `true` | Treat unreachable probe (retries exhausted) as backlog-exceeded. |
 | `BACKPRESSURE__REJECT_ON_ANY_DEGRADED` | `false` | Legacy escape hatch: reject when any component is degraded. |
 | `BACKPRESSURE__UNHEALTHY_CHECK_INTERVAL_SECONDS` | `5.0` | Poll period while REJECTING. |
+
+## MetricsConfig (`METRICS__*`)
+
+Health-snapshot rate metrics: both `IngestGateway` and `ConsumerWorker`
+maintain in-memory sliding windows and expose computed rates / latencies
+through their health snapshots (`receive_rate`, `sink_write_rate`,
+`produce_latency_ms_avg`, ...). Windowed aggregates are computed on read —
+no background tasks, no extra endpoint; rates decay to `0.0` once traffic
+stops for longer than the window.
+
+| Env var | Default | Description |
+|---------|---------|-------------|
+| `METRICS__WINDOW_SECONDS` | `60` | Sliding-window length (seconds) for all rate/latency fields. Valid range 1–600; out-of-range values fail at startup. Short windows react faster but are noisier. |

{streamgate-0.3.0 → streamgate-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: streamgate
-Version: 0.3.0
+Version: 0.4.0
 Summary: A pure-core Kafka data pipeline framework: conditional admission, reliable delivery, pluggable sinks via typed hooks.
 Project-URL: Homepage, https://github.com/pwg-code/streamgate
 Project-URL: Repository, https://github.com/pwg-code/streamgate
@@ -57,6 +57,7 @@ The wheel installs exactly three dependencies (`aiokafka`, `loguru`, `pydantic`)
 - **Error classification as a hook.** Transient vs. poison vs. fatal is storage-specific knowledge. Inject an `ErrorClassifier`; the default only knows generic exceptions (and maps unknowns to poison — probe-protected bisect, never silent drops). A production-hardened SQLAlchemy classifier ships as `streamgate.contrib.sql_sink.SQLAlchemyErrorClassifier`.
 - **Backpressure with hysteresis.** Ingest probes consumer health on a side channel and rejects based on backlog age with separate trip/recover thresholds. The core ships zero-I/O signals (`ManualBackpressureSignal`); the HTTP-probe implementation ships as [`streamgate.contrib.http_probe`](#contrib-official-strategy-implementations).
 - **DLQ bisection instead of head-of-line blocking.** A poison message never wedges a partition: the failing batch is bisected with probe comparison, the bad record is quarantined to a dead-letter topic, the rest commits.
+- **Ops visibility built into the snapshot.** Health snapshots carry rate/latency metrics (receive rate, admission conflicts, produce success/failure, sink write rate, retries, write latency avg/max) computed over a configurable sliding window — your existing `/health` endpoint doubles as a monitoring feed, no Prometheus required.
 
 ## Quick Start
 
@@ -136,7 +137,7 @@ your HTTP app (presentation is yours: auth/routing/OpenAPI)
         ▼                                                  │      (bisect + probe compare)
   ConsumerWorker ──► RecordWriter (your storage) ──────────┘
         │             └─ ErrorClassifier (injected)
-        └─ health snapshot (expose it with your own web framework)
+        └─ health snapshot (state + rates/latencies; expose it with your own web framework)
 ```
 
 The package contains **zero web-framework code** (no fastapi/uvicorn), and the core layers never import `streamgate.contrib` — enforced by the import-linter layered contract in CI.

{streamgate-0.3.0 → streamgate-0.4.0}/README.md

@@ -21,6 +21,7 @@ The wheel installs exactly three dependencies (`aiokafka`, `loguru`, `pydantic`)
 - **Error classification as a hook.** Transient vs. poison vs. fatal is storage-specific knowledge. Inject an `ErrorClassifier`; the default only knows generic exceptions (and maps unknowns to poison — probe-protected bisect, never silent drops). A production-hardened SQLAlchemy classifier ships as `streamgate.contrib.sql_sink.SQLAlchemyErrorClassifier`.
 - **Backpressure with hysteresis.** Ingest probes consumer health on a side channel and rejects based on backlog age with separate trip/recover thresholds. The core ships zero-I/O signals (`ManualBackpressureSignal`); the HTTP-probe implementation ships as [`streamgate.contrib.http_probe`](#contrib-official-strategy-implementations).
 - **DLQ bisection instead of head-of-line blocking.** A poison message never wedges a partition: the failing batch is bisected with probe comparison, the bad record is quarantined to a dead-letter topic, the rest commits.
+- **Ops visibility built into the snapshot.** Health snapshots carry rate/latency metrics (receive rate, admission conflicts, produce success/failure, sink write rate, retries, write latency avg/max) computed over a configurable sliding window — your existing `/health` endpoint doubles as a monitoring feed, no Prometheus required.
 
 ## Quick Start
 
@@ -100,7 +101,7 @@ your HTTP app (presentation is yours: auth/routing/OpenAPI)
         ▼                                                  │      (bisect + probe compare)
   ConsumerWorker ──► RecordWriter (your storage) ──────────┘
         │             └─ ErrorClassifier (injected)
-        └─ health snapshot (expose it with your own web framework)
+        └─ health snapshot (state + rates/latencies; expose it with your own web framework)
 ```
 
 The package contains **zero web-framework code** (no fastapi/uvicorn), and the core layers never import `streamgate.contrib` — enforced by the import-linter layered contract in CI.

{streamgate-0.3.0 → streamgate-0.4.0}/README.zh-CN.md

@@ -26,6 +26,7 @@
 - **错误分类也是钩子。** "连接超时该重试"还是"数据脏了该隔离"，只有懂你存储的人知道。注入一个分类器即可；不注入也有安全的默认行为（不会丢数据）。
 - **背压带磁滞防抖。** 消费端积压超标自动拒绝接收，回落后自动放行，来回抖动有磁滞带压着。
 - **坏数据不堵管道。** 一条毒消息不会卡死整个分区：二分定位、隔离进死信 topic，好数据继续走。
+- **运维观测长在快照里。** 健康快照自带滑动窗口算好的速率/延迟指标（接收速率、准入冲突、生产成败、入库速率、重试速率、写入耗时 avg/max），现有健康端点直接当监控数据源用——不配 Prometheus 也能看趋势、设告警。
 
 ---
 
@@ -294,7 +295,7 @@ pip install "streamgate[http]"     # + HTTP 探活背压
         ▼                                                  │      （二分 + 探针对照）
   ConsumerWorker ──► RecordWriter（你的存储） ──────────────┘
         │             └─ ErrorClassifier（注入）
-        └─ 健康快照（用你自己的 Web 框架暴露）
+        └─ 健康快照（状态 + 速率/延迟指标，用你自己的 Web 框架暴露）
 ```
 
 核心层禁止 import `streamgate.contrib`（contrib 反向依赖核心是合法的）——由 import-linter 分层契约在 CI 强制拦截；contrib 的第三方依赖全部走 extras 可选声明，裸装 `pip install streamgate` 不携带。

{streamgate-0.3.0 → streamgate-0.4.0}/src/streamgate/__init__.py

@@ -22,7 +22,12 @@ Tier 2 逃生口：ConsumeSpec.on_record / ConsumeContext（只读快照）
 不得深入其余内部子模块（import-linter 门禁强制）。
 """
 
-from streamgate.config import BackpressureConfig, ConsumerConfig, KafkaConfig
+from streamgate.config import (
+    BackpressureConfig,
+    ConsumerConfig,
+    KafkaConfig,
+    MetricsConfig,
+)
 from streamgate.consumer.classifier import DefaultErrorClassifier
 from streamgate.consumer.dlq import (
     BisectOutcome,
@@ -74,7 +79,7 @@ from streamgate.specs import ConsumeSpec, IngestBinding, IngestRecordT
 from streamgate.transport.codec import JsonEnvelopeCodec
 from streamgate.transport.kafka import KafkaConsumerService
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 
 __all__ = [
     "AdmissionPolicy",
@@ -113,6 +118,7 @@ __all__ = [
     "LoggingMetricsSink",
     "ManualBackpressureSignal",
     "MessageCodec",
+    "MetricsConfig",
     "MetricsSink",
     "NoAdmission",
     "NoBackfill",

{streamgate-0.3.0 → streamgate-0.4.0}/src/streamgate/config.py

@@ -1,87 +1,103 @@
-"""streamgate 组件配置对象（每组件独立，Spec 内可覆写）。
-
-配置键名与既有环境变量逐字对应（KAFKA__* / CONSUMER__* / BACKPRESSURE__*），
-调用方零配置迁移。DB/Redis 是使用方的世界：连接配置由使用方/示例自带
-（不进框架核心）。
-"""
-
-from pydantic import BaseModel, field_validator
-
-
-class ConsumerConfig(BaseModel):
-    # 必填：消费服务装配时校验（缺失即启动失败，不做项目专属默认）
-    group_id: str | None = None
-    batch_size: int = 100
-    batch_timeout_seconds: float = 5.0
-    max_retries: int = 3
-    retry_backoff_base: float = 1.0
-    # 重连/暂停恢复指数退避（对齐 ingest producer 的自愈策略，替代原硬编码 1.0/30.0）
-    reconnect_base_backoff_seconds: float = 1.0
-    reconnect_max_backoff_seconds: float = 30.0
-    max_poll_records: int = 500
-    session_timeout_ms: int = 30000
-    max_poll_interval_ms: int = 300000
-    auto_offset_reset: str = "earliest"
-    backlog_check_interval_seconds: float = 30.0  # 积压时长检查周期
-    # --- DLQ 隔离：消费端坏数据兜底 ---
-    dlq_enabled: bool = True    # 总开关；false=回退 paused 旧行为（紧急逃生门）
-    dlq_send_retries: int = 3   # DLQ 单条发送总尝试次数（含首次；耗尽即批次转 paused）
-
-
-class KafkaConfig(BaseModel):
-    bootstrap_servers: str = "kafka:9092"
-    # 必填：producer/consumer 装配时校验（缺失即启动失败，不做项目专属默认）
-    topic: str | None = None
-    # 死信 topic（消费端隔离坏数据留档）；启用 DLQ 时必填
-    dlq_topic: str | None = None
-    acks: str = "all"
-    request_timeout_ms: int = 10000
-    enable_idempotence: bool = True
-    # 自愈监控：周期性探活，不健康时销毁旧实例重建连接（覆盖启动期/运行期断线）
-    health_check_interval_seconds: float = 30.0   # 探活轮询周期
-    reconnect_base_backoff_seconds: float = 1.0   # 重连指数退避基数
-    reconnect_max_backoff_seconds: float = 30.0   # 重连退避上限
-    # R1 防抖：连续 send 失败 N 次才触发重建（默认 1=首次失败即重建；
-    # broker 秒级抖动时可调高减少误重建）
-    reconnect_failure_threshold: int = 1
-    # 探活口径：近窗口内有 send 失败即判定不健康（秒）
-    send_failure_window_seconds: float = 30.0
-    # R3 周期分级：异常/重建中高频探活间隔（秒）；健康态仍用 health_check_interval_seconds
-    unhealthy_check_interval_seconds: float = 5.0
-
-    @field_validator("bootstrap_servers", mode="before")
-    @classmethod
-    def strip_url_scheme(cls, v: object) -> object:
-        if not isinstance(v, str):
-            return v
-        cleaned: list[str] = []
-        for part in v.split(","):
-            part = part.strip()
-            for scheme in ("http://", "https://", "kafka://"):
-                if part.lower().startswith(scheme):
-                    part = part[len(scheme):]
-                    break
-            cleaned.append(part)
-        return ",".join(cleaned)
-
-
-class BackpressureConfig(BaseModel):
-    """ingest 背压拒绝配置（consumption-backpressure）。
-
-    默认值与 existence TTL 联动：trip = existence_ttl/2 = 9000s（漏报窗口打开前拦截，
-    与积压告警线 backlog_age_warn 一致）；recover = trip*80% = 7200s（磁滞防抖）。
-    """
-
-    enabled: bool = True                                # 总开关；False=完全放行（紧急回滚）
-    consumer_health_url: str = "http://localhost:9109/health"  # consumer 健康接口地址
-    check_interval_seconds: float = 30.0                # 轮询周期（与积压检查周期一致）
-    timeout_seconds: float = 2.0                        # 单次探活超时
-    probe_retries: int = 3                              # 单周期内探活重试次数（不含首次）；0=禁用
-    probe_retry_interval_seconds: float = 15.0          # 重试间隔（秒）
-    trip_seconds: float = 9000.0                        # 触发阈值：积压超此值开始拒绝
-    recover_seconds: float = 7200.0                     # 恢复阈值：积压回落至此值以下放行（磁滞）
-    retry_after_seconds: int = 60                       # 拒绝时 Retry-After 头（秒）
-    fail_closed_on_unreachable: bool = True             # 探活不可达时是否拒绝（fail-closed）
-    reject_on_any_degraded: bool = False                # 旧行为逃生门：任何 degraded 即拒绝（恢复全量拒绝语义）
-    # R3 背压周期分级：REJECTING 态高频探活间隔（秒）；OPEN 态仍用 check_interval_seconds
-    unhealthy_check_interval_seconds: float = 5.0
+"""streamgate 组件配置对象（每组件独立，Spec 内可覆写）。
+
+配置键名与既有环境变量逐字对应（KAFKA__* / CONSUMER__* / BACKPRESSURE__*），
+调用方零配置迁移。DB/Redis 是使用方的世界：连接配置由使用方/示例自带
+（不进框架核心）。
+"""
+
+from pydantic import BaseModel, field_validator
+
+
+class ConsumerConfig(BaseModel):
+    # 必填：消费服务装配时校验（缺失即启动失败，不做项目专属默认）
+    group_id: str | None = None
+    batch_size: int = 100
+    batch_timeout_seconds: float = 5.0
+    max_retries: int = 3
+    retry_backoff_base: float = 1.0
+    # 重连/暂停恢复指数退避（对齐 ingest producer 的自愈策略，替代原硬编码 1.0/30.0）
+    reconnect_base_backoff_seconds: float = 1.0
+    reconnect_max_backoff_seconds: float = 30.0
+    max_poll_records: int = 500
+    session_timeout_ms: int = 30000
+    max_poll_interval_ms: int = 300000
+    auto_offset_reset: str = "earliest"
+    backlog_check_interval_seconds: float = 30.0  # 积压时长检查周期
+    # --- DLQ 隔离：消费端坏数据兜底 ---
+    dlq_enabled: bool = True    # 总开关；false=回退 paused 旧行为（紧急逃生门）
+    dlq_send_retries: int = 3   # DLQ 单条发送总尝试次数（含首次；耗尽即批次转 paused）
+
+
+class KafkaConfig(BaseModel):
+    bootstrap_servers: str = "kafka:9092"
+    # 必填：producer/consumer 装配时校验（缺失即启动失败，不做项目专属默认）
+    topic: str | None = None
+    # 死信 topic（消费端隔离坏数据留档）；启用 DLQ 时必填
+    dlq_topic: str | None = None
+    acks: str = "all"
+    request_timeout_ms: int = 10000
+    enable_idempotence: bool = True
+    # 自愈监控：周期性探活，不健康时销毁旧实例重建连接（覆盖启动期/运行期断线）
+    health_check_interval_seconds: float = 30.0   # 探活轮询周期
+    reconnect_base_backoff_seconds: float = 1.0   # 重连指数退避基数
+    reconnect_max_backoff_seconds: float = 30.0   # 重连退避上限
+    # R1 防抖：连续 send 失败 N 次才触发重建（默认 1=首次失败即重建；
+    # broker 秒级抖动时可调高减少误重建）
+    reconnect_failure_threshold: int = 1
+    # 探活口径：近窗口内有 send 失败即判定不健康（秒）
+    send_failure_window_seconds: float = 30.0
+    # R3 周期分级：异常/重建中高频探活间隔（秒）；健康态仍用 health_check_interval_seconds
+    unhealthy_check_interval_seconds: float = 5.0
+
+    @field_validator("bootstrap_servers", mode="before")
+    @classmethod
+    def strip_url_scheme(cls, v: object) -> object:
+        if not isinstance(v, str):
+            return v
+        cleaned: list[str] = []
+        for part in v.split(","):
+            part = part.strip()
+            for scheme in ("http://", "https://", "kafka://"):
+                if part.lower().startswith(scheme):
+                    part = part[len(scheme):]
+                    break
+            cleaned.append(part)
+        return ",".join(cleaned)
+
+
+class BackpressureConfig(BaseModel):
+    """ingest 背压拒绝配置（consumption-backpressure）。
+
+    默认值与 existence TTL 联动：trip = existence_ttl/2 = 9000s（漏报窗口打开前拦截，
+    与积压告警线 backlog_age_warn 一致）；recover = trip*80% = 7200s（磁滞防抖）。
+    """
+
+    enabled: bool = True                                # 总开关；False=完全放行（紧急回滚）
+    consumer_health_url: str = "http://localhost:9109/health"  # consumer 健康接口地址
+    check_interval_seconds: float = 30.0                # 轮询周期（与积压检查周期一致）
+    timeout_seconds: float = 2.0                        # 单次探活超时
+    probe_retries: int = 3                              # 单周期内探活重试次数（不含首次）；0=禁用
+    probe_retry_interval_seconds: float = 15.0          # 重试间隔（秒）
+    trip_seconds: float = 9000.0                        # 触发阈值：积压超此值开始拒绝
+    recover_seconds: float = 7200.0                     # 恢复阈值：积压回落至此值以下放行（磁滞）
+    retry_after_seconds: int = 60                       # 拒绝时 Retry-After 头（秒）
+    fail_closed_on_unreachable: bool = True             # 探活不可达时是否拒绝（fail-closed）
+    reject_on_any_degraded: bool = False                # 旧行为逃生门：任何 degraded 即拒绝（恢复全量拒绝语义）
+    # R3 背压周期分级：REJECTING 态高频探活间隔（秒）；OPEN 态仍用 check_interval_seconds
+    unhealthy_check_interval_seconds: float = 5.0
+
+
+class MetricsConfig(BaseModel):
+    """健康快照速率指标配置（滑动窗口长度，ingest/consume 两侧共用）。"""
+
+    window_seconds: int = 60
+
+    @field_validator("window_seconds")
+    @classmethod
+    def check_window_seconds(cls, v: int) -> int:
+        if not 1 <= v <= 600:
+            raise ValueError(
+                "window_seconds out of range; "
+                "set METRICS__WINDOW_SECONDS to a value between 1 and 600"
+            )
+        return v