svc-infra 0.1.621__py3-none-any.whl → 0.1.623__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of svc-infra might be problematic. Click here for more details.

Files changed (33) hide show
  1. svc_infra/cli/cmds/docs/docs_cmds.py +102 -179
  2. svc_infra/docs/acceptance-matrix.md +71 -0
  3. svc_infra/docs/acceptance.md +44 -0
  4. svc_infra/docs/adr/0002-background-jobs-and-scheduling.md +40 -0
  5. svc_infra/docs/adr/0003-webhooks-framework.md +24 -0
  6. svc_infra/docs/adr/0004-tenancy-model.md +42 -0
  7. svc_infra/docs/adr/0005-data-lifecycle.md +86 -0
  8. svc_infra/docs/adr/0006-ops-slos-and-metrics.md +47 -0
  9. svc_infra/docs/adr/0007-docs-and-sdks.md +83 -0
  10. svc_infra/docs/adr/0008-billing-primitives.md +109 -0
  11. svc_infra/docs/adr/0009-acceptance-harness.md +40 -0
  12. svc_infra/docs/api.md +59 -0
  13. svc_infra/docs/auth.md +11 -0
  14. svc_infra/docs/cache.md +18 -0
  15. svc_infra/docs/cli.md +74 -0
  16. svc_infra/docs/contributing.md +34 -0
  17. svc_infra/docs/data-lifecycle.md +52 -0
  18. svc_infra/docs/database.md +14 -0
  19. svc_infra/docs/docs-and-sdks.md +62 -0
  20. svc_infra/docs/environment.md +114 -0
  21. svc_infra/docs/idempotency.md +111 -0
  22. svc_infra/docs/jobs.md +67 -0
  23. svc_infra/docs/observability.md +16 -0
  24. svc_infra/docs/ops.md +33 -0
  25. svc_infra/docs/rate-limiting.md +121 -0
  26. svc_infra/docs/repo-review.md +48 -0
  27. svc_infra/docs/security.md +155 -0
  28. svc_infra/docs/tenancy.md +35 -0
  29. svc_infra/docs/webhooks.md +112 -0
  30. {svc_infra-0.1.621.dist-info → svc_infra-0.1.623.dist-info}/METADATA +16 -16
  31. {svc_infra-0.1.621.dist-info → svc_infra-0.1.623.dist-info}/RECORD +33 -5
  32. {svc_infra-0.1.621.dist-info → svc_infra-0.1.623.dist-info}/WHEEL +0 -0
  33. {svc_infra-0.1.621.dist-info → svc_infra-0.1.623.dist-info}/entry_points.txt +0 -0
@@ -0,0 +1,114 @@
1
+ # Environment Reference
2
+
3
+ This guide consolidates every environment variable consumed by the svc-infra helpers in FastAPI, jobs, observability, security, and webhooks. Defaults shown below reflect the library's fallbacks when a variable is absent. Where a helper relies on `svc_infra.app.pick`, the note column calls out the environment-specific behavior.
4
+
5
+ ## FastAPI helpers
6
+
7
+ ### App bootstrap (`easy_service_app` / `setup_service_api`)
8
+
9
+ | Variable | Default | Consumed by | Notes |
10
+ | --- | --- | --- | --- |
11
+ | `ENABLE_LOGGING` | `true` | `EasyAppOptions.from_env()` | Disables `setup_logging` when set to false. |
12
+ | `LOG_LEVEL` | Auto (`INFO` in prod/test, `DEBUG` in dev/local via `pick()`) | `easy_service_app()` | Overrides the log level chosen by `svc_infra.app.pick`. |
13
+ | `LOG_FORMAT` | Auto (JSON in prod, plain elsewhere) | `easy_service_app()` | Explicit `json` or `plain` format overrides auto-detection. |
14
+ | `ENABLE_OBS` | `true` | `EasyAppOptions.from_env()` / `easy_service_app()` | Turns observability instrumentation on/off. |
15
+ | `METRICS_PATH` | `None` → falls back to Observability settings | `EasyAppOptions.from_env()` | Use to expose metrics at a non-default path. |
16
+ | `OBS_SKIP_PATHS` | `None` → defaults to metrics + health endpoints | `EasyAppOptions.from_env()` | Comma/space-separated list of paths skipped by Prometheus middleware. |
17
+ | `CORS_ALLOW_ORIGINS` | `""` (no origins) | `_setup_cors()` | Adds `CORSMiddleware` allow-list when non-empty. |
18
+
19
+ ### SQL helpers (`add_sql_db`, `setup_sql`)
20
+
21
+ | Variable | Default | Consumed by | Notes |
22
+ | --- | --- | --- | --- |
23
+ | `SQL_URL` (overridable via `dsn_env`) | _required_ | `add_sql_db()` / `setup_sql()` | Missing value raises `RuntimeError`; point at your primary database URL. |
24
+
25
+ ### Mongo helpers (`add_mongo_db`, `init_mongo`)
26
+
27
+ | Variable | Default | Consumed by | Notes |
28
+ | --- | --- | --- | --- |
29
+ | `MONGO_URL` / `MONGODB_URL` | `mongodb://localhost:27017` | `MongoSettings`, `add_mongo_db()` | Primary Mongo connection string; `_FILE` suffix or `MONGO_URL_FILE` allow secret mounts. |
30
+ | `MONGO_DB` / `MONGODB_DB` / `MONGO_DATABASE` | unset (optional) | `get_mongo_dbname_from_env()` | When set, verified against the connected database name. |
31
+ | `MONGO_APPNAME` | `svc-infra` | `MongoSettings` | Sets the Mongo client `appname`. |
32
+ | `MONGO_MIN_POOL` | `0` | `MongoSettings` | Minimum Motor/Mongo client pool size. |
33
+ | `MONGO_MAX_POOL` | `100` | `MongoSettings` | Maximum Motor/Mongo client pool size. |
34
+ | `MONGO_URL_FILE` | unset | `get_mongo_url_from_env()` | Alternate secret file path when not using `_FILE` suffix envs. |
35
+ | `/run/secrets/mongo_url` | unset | `get_mongo_url_from_env()` | Auto-mounted Docker/K8s secret fallback for the URL. |
36
+
37
+ ### Auth settings (`get_auth_settings` → `AuthSettings`)
38
+
39
+ Pydantic loads these with the `AUTH_` prefix and `__` as the nested delimiter.
40
+
41
+ | Variable | Default | Consumed by | Notes |
42
+ | --- | --- | --- | --- |
43
+ | `AUTH_JWT__SECRET` | _required when JWT auth enabled_ | `AuthSettings.jwt.secret` | Primary HS256 signing secret. |
44
+ | `AUTH_JWT__LIFETIME_SECONDS` | `604800` (7 days) | `AuthSettings.jwt.lifetime_seconds` | Adjusts refresh token lifetime. |
45
+ | `AUTH_JWT__OLD_SECRETS__*` | `[]` | `AuthSettings.jwt.old_secrets` | Accepted legacy secrets during rotation. |
46
+ | `AUTH_PASSWORD_CLIENTS__{n}__CLIENT_ID` | `[]` | `AuthSettings.password_clients[*].client_id` | Register password clients (list entries indexed by `{n}`). |
47
+ | `AUTH_PASSWORD_CLIENTS__{n}__CLIENT_SECRET` | `[]` | `AuthSettings.password_clients[*].client_secret` | Secret per password client. |
48
+ | `AUTH_REQUIRE_CLIENT_SECRET_ON_PASSWORD_LOGIN` | `false` | `AuthSettings.require_client_secret_on_password_login` | Enforces client secret on password grant. |
49
+ | `AUTH_MFA_DEFAULT_ENABLED_FOR_NEW_USERS` | `false` | `AuthSettings.mfa_default_enabled_for_new_users` | Enable TOTP by default on signup. |
50
+ | `AUTH_MFA_ENFORCE_FOR_ALL_USERS` | `false` | `AuthSettings.mfa_enforce_for_all_users` | Force MFA globally. |
51
+ | `AUTH_MFA_ENFORCE_FOR_TENANTS` | `[]` | `AuthSettings.mfa_enforce_for_tenants` | Tenant allow-list requiring MFA. |
52
+ | `AUTH_MFA_ISSUER` | `"svc-infra"` | `AuthSettings.mfa_issuer` | Label for TOTP apps. |
53
+ | `AUTH_MFA_PRE_TOKEN_LIFETIME_SECONDS` | `300` | `AuthSettings.mfa_pre_token_lifetime_seconds` | Lifespan of MFA pre-token. |
54
+ | `AUTH_MFA_RECOVERY_CODES` | `8` | `AuthSettings.mfa_recovery_codes` | Number of recovery codes issued. |
55
+ | `AUTH_MFA_RECOVERY_CODE_LENGTH` | `10` | `AuthSettings.mfa_recovery_code_length` | Digits per recovery code. |
56
+ | `AUTH_EMAIL_OTP_TTL_SECONDS` | `300` | `AuthSettings.email_otp_ttl_seconds` | Email OTP validity window. |
57
+ | `AUTH_EMAIL_OTP_COOLDOWN_SECONDS` | `60` | `AuthSettings.email_otp_cooldown_seconds` | Cooldown between OTP sends. |
58
+ | `AUTH_EMAIL_OTP_ATTEMPTS` | `5` | `AuthSettings.email_otp_attempts` | Maximum OTP attempts before lock. |
59
+ | `AUTH_SMTP_HOST` | `None` | `AuthSettings.smtp_host` | SMTP hostname (required for prod email). |
60
+ | `AUTH_SMTP_PORT` | `587` | `AuthSettings.smtp_port` | SMTP port. |
61
+ | `AUTH_SMTP_USERNAME` | `None` | `AuthSettings.smtp_username` | SMTP username. |
62
+ | `AUTH_SMTP_PASSWORD` | `None` | `AuthSettings.smtp_password` | SMTP password/secret. |
63
+ | `AUTH_SMTP_FROM` | `None` | `AuthSettings.smtp_from` | Default From address. |
64
+ | `AUTH_AUTO_VERIFY_IN_DEV` | `true` | `AuthSettings.auto_verify_in_dev` | Auto-confirms accounts outside prod. |
65
+ | `AUTH_GOOGLE_CLIENT_ID` | `None` | `AuthSettings.google_client_id` | Built-in Google OAuth client ID. |
66
+ | `AUTH_GOOGLE_CLIENT_SECRET` | `None` | `AuthSettings.google_client_secret` | Built-in Google OAuth secret. |
67
+ | `AUTH_GITHUB_CLIENT_ID` | `None` | `AuthSettings.github_client_id` | GitHub OAuth client ID. |
68
+ | `AUTH_GITHUB_CLIENT_SECRET` | `None` | `AuthSettings.github_client_secret` | GitHub OAuth secret. |
69
+ | `AUTH_MS_CLIENT_ID` | `None` | `AuthSettings.ms_client_id` | Microsoft OAuth client ID. |
70
+ | `AUTH_MS_CLIENT_SECRET` | `None` | `AuthSettings.ms_client_secret` | Microsoft OAuth secret. |
71
+ | `AUTH_MS_TENANT` | `None` | `AuthSettings.ms_tenant` | Microsoft tenant ID. |
72
+ | `AUTH_LI_CLIENT_ID` | `None` | `AuthSettings.li_client_id` | LinkedIn OAuth client ID. |
73
+ | `AUTH_LI_CLIENT_SECRET` | `None` | `AuthSettings.li_client_secret` | LinkedIn OAuth secret. |
74
+ | `AUTH_OIDC_PROVIDERS__{n}__NAME` | `[]` | `AuthSettings.oidc_providers[*].name` | Custom OIDC providers (list entries indexed by `{n}`). |
75
+ | `AUTH_OIDC_PROVIDERS__{n}__ISSUER` | `[]` | `AuthSettings.oidc_providers[*].issuer` | OIDC issuer URL. |
76
+ | `AUTH_OIDC_PROVIDERS__{n}__CLIENT_ID` | `[]` | `AuthSettings.oidc_providers[*].client_id` | OIDC client ID. |
77
+ | `AUTH_OIDC_PROVIDERS__{n}__CLIENT_SECRET` | `[]` | `AuthSettings.oidc_providers[*].client_secret` | OIDC client secret. |
78
+ | `AUTH_OIDC_PROVIDERS__{n}__SCOPE` | `"openid email profile"` | `AuthSettings.oidc_providers[*].scope` | Additional OIDC scopes. |
79
+ | `AUTH_POST_LOGIN_REDIRECT` | `http://localhost:3000/app` | `AuthSettings.post_login_redirect` | Default redirect after login. |
80
+ | `AUTH_REDIRECT_ALLOW_HOSTS_RAW` | `"localhost,127.0.0.1"` | `AuthSettings.redirect_allow_hosts_raw` | CSV/JSON allow-list for redirects. |
81
+ | `AUTH_SESSION_COOKIE_NAME` | `"svc_session"` | `AuthSettings.session_cookie_name` | Session cookie key. |
82
+ | `AUTH_AUTH_COOKIE_NAME` | `"svc_auth"` | `AuthSettings.auth_cookie_name` | Auth cookie key. |
83
+ | `AUTH_SESSION_COOKIE_SECURE` | `false` | `AuthSettings.session_cookie_secure` | Marks session cookie `Secure`. |
84
+ | `AUTH_SESSION_COOKIE_SAMESITE` | `"lax"` | `AuthSettings.session_cookie_samesite` | SameSite policy. |
85
+ | `AUTH_SESSION_COOKIE_DOMAIN` | `None` | `AuthSettings.session_cookie_domain` | Explicit cookie domain. |
86
+ | `AUTH_SESSION_COOKIE_MAX_AGE_SECONDS` | `14400` (4 hours) | `AuthSettings.session_cookie_max_age_seconds` | Session cookie lifetime. |
87
+
88
+ ## Jobs helpers
89
+
90
+ | Variable | Default | Consumed by | Notes |
91
+ | --- | --- | --- | --- |
92
+ | `JOBS_DRIVER` | `memory` | `JobsConfig`, `easy_jobs()` | Choose `redis` to activate Redis-backed queue. |
93
+ | `REDIS_URL` | `redis://localhost:6379/0` | `easy_jobs()` (Redis driver) | Redis connection string when `JOBS_DRIVER=redis`. |
94
+ | `JOBS_SCHEDULE_JSON` | unset | `schedule_from_env()` | JSON array of scheduler tasks (name, interval_seconds, target). |
95
+
96
+ ## Observability helpers
97
+
98
+ | Variable | Default | Consumed by | Notes |
99
+ | --- | --- | --- | --- |
100
+ | `METRICS_ENABLED` | `true` | `ObservabilitySettings` | Gate for Prometheus middleware registration. |
101
+ | `METRICS_PATH` | `/metrics` | `ObservabilitySettings`, `add_observability()` | Metrics endpoint path. |
102
+ | `METRICS_DEFAULT_BUCKETS` | `0.005,0.01,0.025,0.05,0.1,0.25,0.5,1.0,2.0,5.0,10.0` | `ObservabilitySettings` | Histogram buckets for request latency. |
103
+ | `SVC_INFRA_DISABLE_PROMETHEUS` | unset (`"1"` disables) | `metrics.asgi` | Skip Prometheus setup when toggled. |
104
+ | `SVC_INFRA_RATE_WINDOW` | unset | `cloud_dash.push_dashboards_from_pkg()` | Overrides `$__rate_interval` in dashboards. |
105
+ | `SVC_INFRA_DASHBOARD_REFRESH` | `5s` | `cloud_dash.push_dashboards_from_pkg()` | Grafana dashboard auto-refresh interval. |
106
+ | `SVC_INFRA_DASHBOARD_RANGE` | `now-6h` | `cloud_dash.push_dashboards_from_pkg()` | Default Grafana time range start. |
107
+
108
+ ## Security helpers
109
+
110
+ The primitives under `svc_infra.security` rely on configuration objects passed from application code; they do not read environment variables directly beyond the shared `AuthSettings` listed above.
111
+
112
+ ## Webhook helpers
113
+
114
+ Current webhook helpers (`fastapi.require_signature`, `InMemoryWebhookSubscriptions`, `WebhookService`) rely on dependency injection for secrets and stores and do not read environment variables directly.
@@ -0,0 +1,111 @@
1
+ # Idempotency & Concurrency Controls
2
+
3
+ This guide explains how idempotency works in svc-infra and how to enable it for safe retries.
4
+
5
+ ## What it does
6
+ - Prevents duplicate processing of mutating requests (POST/PATCH/DELETE).
7
+ - Replays the previously successful response when the same `Idempotency-Key` is used.
8
+ - Detects conflicts when the same key is reused with a different request body, returning 409.
9
+
10
+ ## Middleware usage
11
+ ```python
12
+ from svc_infra.api.fastapi.middleware.idempotency import IdempotencyMiddleware
13
+
14
+ app.add_middleware(IdempotencyMiddleware) # default: in-memory store, 24h TTL
15
+ ```
16
+ - Header name: `Idempotency-Key` (configurable via `header_name`)
17
+ - TTL: defaults to 24 hours (`ttl_seconds`)
18
+
19
+ ### Redis store (recommended for multi-instance)
20
+ ```python
21
+ import redis
22
+ from svc_infra.api.fastapi.middleware.idempotency import IdempotencyMiddleware
23
+ from svc_infra.api.fastapi.middleware.idempotency_store import RedisIdempotencyStore
24
+
25
+ r = redis.Redis.from_url("redis://localhost:6379/0")
26
+ store = RedisIdempotencyStore(r, prefix="idmp")
27
+ app.add_middleware(IdempotencyMiddleware, store=store, ttl_seconds=24*3600)
28
+ ```
29
+
30
+ ## Per-route enforcement
31
+ If an endpoint must require idempotency always, add the dependency:
32
+ ```python
33
+ from fastapi import Depends
34
+ from svc_infra.api.fastapi.middleware.idempotency import require_idempotency_key
35
+
36
+ @app.post("/payments/intents", dependencies=[Depends(require_idempotency_key)])
37
+ async def create_intent(...):
38
+ ...
39
+ ```
40
+
41
+ ## Semantics
42
+ - First request with a key:
43
+ - The middleware claims the key and records a hash of the request body.
44
+ - On success (2xx), the response envelope is cached until TTL.
45
+ - Replay with same key and same body:
46
+ - Returns the cached response with the original status and headers.
47
+ - Replay with same key but different body:
48
+ - Returns 409 Conflict (don’t reuse keys for different logical operations).
49
+
50
+ ## Testing
51
+ - Marker: `-m concurrency` selects concurrency tests in this repo.
52
+ - Scenarios covered:
53
+ - Successful first request and replay
54
+ - Conflict on mismatched payload reusing the same key
55
+
56
+ ## Notes and pitfalls
57
+ - Use a unique key per logical operation (e.g., `order-{id}-capture-1`).
58
+ - TTL should exceed your max retry horizon.
59
+ - For stronger guarantees in Redis, consider a Lua script to make the claim + response update atomic (future improvement).
60
+ - If you also use optimistic locking, surface 409 when `If-Match` version mismatches during updates.
61
+
62
+ ---
63
+
64
+ ## Optimistic Locking
65
+
66
+ Use the `If-Match` header and a version field on your models.
67
+
68
+ ```python
69
+ from svc_infra.api.fastapi.middleware.optimistic_lock import require_if_match, check_version_or_409
70
+
71
+ @app.patch("/resource/{rid}")
72
+ async def update_resource(rid: str, v: str = Depends(require_if_match)):
73
+ current = await repo.get(...)
74
+ check_version_or_409(lambda: current.version, v)
75
+ current.version += 1
76
+ await repo.save(current)
77
+ return {...}
78
+ ```
79
+
80
+ Pitfalls:
81
+ - Always bump the version on successful updates.
82
+ - Return 428 when `If-Match` is missing on mutating routes that require optimistic locking.
83
+ - Consider ETag headers for GETs to complement conditional requests.
84
+
85
+ ---
86
+
87
+ ## Outbox / Inbox
88
+
89
+ Outbox: record events/changes that must be delivered to external systems; a relay fetches and delivers reliably.
90
+
91
+ ```python
92
+ from svc_infra.db.outbox import InMemoryOutboxStore
93
+
94
+ ob = InMemoryOutboxStore()
95
+ msg = ob.enqueue("orders.created", {"order_id": 123})
96
+ nxt = ob.fetch_next(topics=["orders.created"]) # process
97
+ ob.mark_processed(nxt.id)
98
+ ```
99
+
100
+ Inbox: deduplicate external deliveries (e.g., webhook replays) with TTL.
101
+
102
+ ```python
103
+ from svc_infra.db.inbox import InMemoryInboxStore
104
+
105
+ ib = InMemoryInboxStore()
106
+ if not ib.mark_if_new("provider-evt-abc", ttl_seconds=86400):
107
+ return 200 # duplicate
108
+ ```
109
+
110
+ Notes:
111
+ - In-memory stores are for tests/local dev; implement SQL/Redis for production with row locks and `SKIP LOCKED` (or Lua) as needed.
svc_infra/docs/jobs.md ADDED
@@ -0,0 +1,67 @@
1
+ # Background Jobs & Scheduling
2
+
3
+ This module provides a lightweight JobQueue abstraction with a Redis backend for production and in-memory implementations for local/tests. A simple interval scheduler and CLI runner are included.
4
+
5
+ > ℹ️ Job-related environment variables are documented in [Environment Reference](environment.md).
6
+
7
+ ## Quickstart
8
+
9
+ - Initialize in app code:
10
+
11
+ ```python
12
+ from svc_infra.jobs.easy import easy_jobs
13
+ queue, scheduler = easy_jobs() # uses JOBS_DRIVER=memory|redis
14
+ ```
15
+
16
+ - Enqueue a job:
17
+
18
+ ```python
19
+ job = queue.enqueue("send_email", {"to": "user@example.com"})
20
+ ```
21
+
22
+ - Process one job (async):
23
+
24
+ ```python
25
+ from svc_infra.jobs.worker import process_one
26
+ await process_one(queue, handler)
27
+ ```
28
+
29
+ - Run the CLI runner:
30
+
31
+ ```bash
32
+ svc-infra jobs run
33
+ ```
34
+
35
+ ## Redis Backend
36
+
37
+ Set environment variables:
38
+ - JOBS_DRIVER=redis
39
+ - REDIS_URL=redis://localhost:6379/0
40
+
41
+ Features:
42
+ - Visibility timeout during processing
43
+ - Exponential backoff on failures (base backoff_seconds * attempts)
44
+ - DLQ after max_attempts
45
+
46
+ ## Scheduler
47
+
48
+ An interval-based scheduler runs async callables. You can define tasks via environment JSON:
49
+
50
+ - JOBS_SCHEDULE_JSON:
51
+
52
+ ```json
53
+ [
54
+ {"name": "ping", "interval_seconds": 60, "target": "myapp.tasks:ping"}
55
+ ]
56
+ ```
57
+
58
+ The `target` must be an import path of the form `module:function`. Sync functions are wrapped.
59
+
60
+ ## Testing
61
+
62
+ - In-memory queue and scheduler enable fast, deterministic tests.
63
+ - Redis tests use `fakeredis` and cover enqueue/reserve/ack/fail and DLQ path.
64
+
65
+ ## Next
66
+
67
+ - Add metrics, Redis Lua for atomic multi-ops, SQL-backed queue, distributed scheduler.
@@ -0,0 +1,16 @@
1
+ # Observability guide
2
+
3
+ `svc_infra.obs` adds Prometheus metrics, OpenTelemetry instrumentation, and a CLI to spin up Grafana dashboards.
4
+
5
+ ```python
6
+ from svc_infra.api.fastapi.ease import easy_service_app
7
+
8
+ app = easy_service_app(name="Billing", release="1.2.3") # ENABLE_OBS toggles middleware
9
+ ```
10
+
11
+ ### Environment
12
+
13
+ - `ENABLE_OBS`, `METRICS_PATH`, `OBS_SKIP_PATHS` – FastAPI bootstrap toggles for metrics exposure. 【F:src/svc_infra/api/fastapi/ease.py†L67-L111】
14
+ - `SVC_INFRA_DISABLE_PROMETHEUS` – disable the ASGI middleware entirely (used by tests/sidecars). 【F:src/svc_infra/obs/metrics/asgi.py†L49-L206】
15
+ - `SVC_INFRA_RATE_WINDOW`, `SVC_INFRA_DASHBOARD_REFRESH`, `SVC_INFRA_DASHBOARD_RANGE` – tune the Grafana dashboards generated by the CLI. 【F:src/svc_infra/obs/cloud_dash.py†L85-L108】
16
+ - Grafana Cloud support looks for `GRAFANA_CLOUD_URL`, `GRAFANA_CLOUD_TOKEN`, `GRAFANA_CLOUD_PROM_URL`, `GRAFANA_CLOUD_PROM_USERNAME`, `GRAFANA_CLOUD_RW_TOKEN`. 【F:src/svc_infra/obs/README.md†L37-L119】
svc_infra/docs/ops.md ADDED
@@ -0,0 +1,33 @@
1
+ # SLOs & Ops
2
+
3
+ This guide explains how to use svc-infra’s probes, circuit breaker, and metrics for SLOs.
4
+
5
+ ## Probes
6
+
7
+ - `add_probes(app, prefix="/_ops")` exposes:
8
+ - `GET /_ops/live` — liveness
9
+ - `GET /_ops/ready` — readiness
10
+ - `GET /_ops/startup` — startup
11
+
12
+ ## Circuit breaker (placeholder)
13
+
14
+ - `circuit_breaker_dependency()` returns a dependency that returns 503 when `CIRCUIT_OPEN` is truthy.
15
+ - Use it per-route: `@app.get("/x", dependencies=[Depends(circuit_breaker_dependency())])`.
16
+
17
+ ## Metrics and route classification
18
+
19
+ - `add_observability(app, ...)` enables Prometheus metrics and optional DB pool metrics.
20
+ - You can pass an optional `route_classifier(base_path, method) -> class` callable. When provided, the metrics middleware encodes the resolved route label as `"{base}|{class}"`. Dashboards can split this label to filter public/internal/admin routes.
21
+
22
+ ## Dashboards
23
+
24
+ - A minimal Grafana dashboard JSON is provided at `src/svc_infra/obs/grafana/dashboards/http-overview.json` (import into Grafana).
25
+ - It shows:
26
+ - Success rate over 5 minutes
27
+ - p99 latency
28
+ - Top routes by 5xx rate
29
+
30
+ ## Defaults & environment
31
+
32
+ - Prometheus middleware is enabled unless `SVC_INFRA_DISABLE_PROMETHEUS=1`.
33
+ - Observability settings: `METRICS_ENABLED`, `METRICS_PATH`, and optional histogram buckets.
@@ -0,0 +1,121 @@
1
+ # Rate Limiting & Abuse Protection
2
+
3
+ This guide shows how to enable and tune the built-in rate limiter and request-size guard, and how to hook simple metrics for abuse detection.
4
+
5
+ ## Features
6
+
7
+ - Global middleware-based rate limiting with standard headers
8
+ - Per-route dependency for fine-grained limits
9
+ - 429 responses include `Retry-After`
10
+ - Pluggable store interface (in-memory provided; Redis store available)
11
+ - Request size limit middleware returning 413
12
+ - Metrics hooks for rate-limiting events and suspect payloads
13
+
14
+ ## Global middleware
15
+
16
+ ```python
17
+ from svc_infra.api.fastapi.middleware.ratelimit import SimpleRateLimitMiddleware
18
+
19
+ app.add_middleware(
20
+ SimpleRateLimitMiddleware,
21
+ limit=120, # requests
22
+ window=60, # seconds
23
+ key_fn=lambda r: r.headers.get("X-API-Key") or r.client.host,
24
+ )
25
+ ```
26
+
27
+ Responses include headers:
28
+ - `X-RateLimit-Limit`
29
+ - `X-RateLimit-Remaining`
30
+ - `X-RateLimit-Reset` (epoch seconds)
31
+
32
+ When exceeded, responses are 429 with `Retry-After` and the same headers.
33
+
34
+ Advanced options:
35
+ - `scope_by_tenant=True` to automatically include tenant id in the key if tenancy is configured.
36
+ - `limit_resolver=request -> int | None` to override the limit dynamically per request (e.g., per-plan quotas).
37
+
38
+ ## Per-route dependency
39
+
40
+ ```python
41
+ from fastapi import Depends
42
+ from svc_infra.api.fastapi.dependencies.ratelimit import rate_limiter
43
+
44
+ limiter = rate_limiter(limit=10, window=60, key_fn=lambda r: r.client.host)
45
+
46
+ @app.get("/resource", dependencies=[Depends(limiter)])
47
+ def get_resource():
48
+ return {"ok": True}
49
+ ```
50
+
51
+ The dependency supports the same options as the middleware: `scope_by_tenant`, `limit_resolver`, and a custom `store`.
52
+
53
+ ## Store interface
54
+
55
+ The limiter uses a store abstraction so you can inject Redis or other backends.
56
+
57
+ - Default: `InMemoryRateLimitStore` (best-effort, single-process)
58
+ - Interface: `RateLimitStore` with `incr(key, window) -> (count, limit, resetEpoch)`
59
+
60
+ ### Redis store
61
+
62
+ Use `RedisRateLimitStore` for multi-instance deployments. It implements a fixed-window counter
63
+ with atomic increments and sets expiry to the end of the window.
64
+
65
+ ```python
66
+ import redis
67
+ from svc_infra.api.fastapi.middleware.ratelimit_store import RedisRateLimitStore
68
+ from svc_infra.api.fastapi.middleware.ratelimit import SimpleRateLimitMiddleware
69
+
70
+ r = redis.Redis.from_url("redis://localhost:6379/0")
71
+ store = RedisRateLimitStore(r, limit=120, prefix="rl")
72
+
73
+ app.add_middleware(SimpleRateLimitMiddleware, limit=120, window=60, store=store)
74
+ ```
75
+
76
+ Notes:
77
+ - Fixed-window counters are simple and usually sufficient. For smoother limits, consider
78
+ sliding window or token bucket in a future iteration.
79
+ - Use a namespace/prefix per environment/tenant if needed.
80
+
81
+ ## Request size guard
82
+
83
+ ```python
84
+ from svc_infra.api.fastapi.middleware.request_size_limit import RequestSizeLimitMiddleware
85
+
86
+ app.add_middleware(RequestSizeLimitMiddleware, max_bytes=1_000_000)
87
+ ```
88
+
89
+ - Returns 413 with a Problem+JSON-like structure when `Content-Length` exceeds `max_bytes`.
90
+
91
+ ## Metrics hooks
92
+
93
+ Hooks live in `svc_infra.obs.metrics` and are no-ops by default. Assign them to log or emit metrics.
94
+
95
+ ```python
96
+ import logging
97
+ import svc_infra.obs.metrics as metrics
98
+
99
+ logger = logging.getLogger(__name__)
100
+
101
+ metrics.on_rate_limit_exceeded = lambda key, limit, retry: logger.warning(
102
+ "rate_limited", extra={"key": key, "limit": limit, "retry_after": retry}
103
+ )
104
+
105
+ metrics.on_suspect_payload = lambda path, size: logger.warning(
106
+ "suspect_payload", extra={"path": path, "size": size}
107
+ )
108
+ ```
109
+
110
+ ## Tuning tips
111
+
112
+ - Prefer API key or user ID for `key_fn`; fall back to IP if unauthenticated.
113
+ - Keep windows small (e.g., 60s); layer multiple limits when needed.
114
+ - For distributed deployments, use `RedisRateLimitStore` for atomic increments.
115
+ - Consider separate limits for read vs write routes.
116
+ - Combine with request size limits and auth lockout for layered defense.
117
+
118
+ ## Testing
119
+
120
+ - Use `-m ratelimit` to select rate-limiting tests.
121
+ - `-m security` also includes these in this repo by default.
@@ -0,0 +1,48 @@
1
+ # svc-infra Integration Review
2
+
3
+ ## Executive Summary
4
+ - `setup_service_api` remains the high-level entry point for API construction. It auto-wires common middleware (request IDs, error handling, idempotency, rate limiting) and OpenAPI customisation so downstream services only supply router packages and metadata. 【F:src/svc_infra/api/fastapi/setup.py†L1-L187】
5
+ - Auth, SQL, jobs, cache, and observability still expose single-call helpers (`add_auth_users`, `add_sql_db`/`setup_sql`, `easy_jobs`, `init_cache`, `add_observability`) that hide wiring details while keeping flags and dependency overrides for advanced scenarios. 【F:src/svc_infra/api/fastapi/auth/add.py†L27-L329】【F:src/svc_infra/api/fastapi/db/sql/add.py†L1-L118】【F:src/svc_infra/jobs/easy.py†L1-L28】【F:src/svc_infra/cache/__init__.py†L1-L29】【F:src/svc_infra/obs/add.py†L1-L68】
6
+ - Security primitives (lockout, session rotation, RBAC/ABAC, audit chain) are packaged as small composable functions and documented patterns, making them easy to reuse outside FastAPI while still integrating with the default auth routers. 【F:src/svc_infra/security/lockout.py†L1-L96】【F:src/svc_infra/security/session.py†L1-L98】【F:src/svc_infra/security/permissions.py†L1-L148】【F:src/svc_infra/security/audit_service.py†L1-L73】【F:docs/security.md†L1-L96】
7
+ - Webhook utilities provide signature verification, a subscription service, and a default router, but they do not yet follow the same add-helper pattern (there is no `add_webhooks(app, ...)` or env-driven persistence wiring). This is the primary deviation from the established DX story. 【F:src/svc_infra/webhooks/fastapi.py†L1-L37】【F:src/svc_infra/webhooks/service.py†L1-L45】【F:src/svc_infra/webhooks/router.py†L1-L55】【F:docs/webhooks.md†L1-L59】
8
+
9
+ ## Detailed Findings
10
+
11
+ ### API bootstrap and middleware
12
+ - `setup_service_api` builds a parent app and any versioned child apps, runs the OpenAPI mutation pipeline, registers default routers, and attaches middleware without extra code in downstream projects. 【F:src/svc_infra/api/fastapi/setup.py†L1-L187】
13
+ - Scoped docs helpers automatically group endpoints when `add_auth_users` or versioned routers are mounted, so projects get organised API docs with minimal configuration. 【F:src/svc_infra/api/fastapi/auth/add.py†L269-L329】
14
+ - Idempotency and optimistic locking remain opt-in via middleware or dependencies with clear docs; defaults are safe for local use and can be swapped for Redis-backed stores. 【F:docs/idempotency.md†L1-L110】
15
+
16
+ ### Authentication and security
17
+ - `add_auth_users` continues to be the main integration surface. It mounts routers based on flags, configures session middleware from env, and exposes API-key plus OAuth support as toggles. Projects can override policy objects or provider models if needed, preserving flexibility. 【F:src/svc_infra/api/fastapi/auth/add.py†L27-L329】
18
+ - Runtime security helpers (principal resolution, `RequireRoles`, `RequireScopes`, `RequirePermission`, ABAC predicates) make it straightforward to guard routes consistently with the dual routers that enforce doc security defaults. 【F:src/svc_infra/api/fastapi/auth/security.py†L1-L146】【F:src/svc_infra/api/fastapi/dual/protected.py†L1-L74】【F:src/svc_infra/security/permissions.py†L1-L148】
19
+ - Low-level primitives (lockout, refresh token rotation, audit chain) remain framework agnostic so other stacks can reuse them. Documentation in `docs/security.md` is aligned with the code and offers wiring recipes. 【F:src/svc_infra/security/lockout.py†L1-L96】【F:src/svc_infra/security/session.py†L1-L98】【F:src/svc_infra/security/audit_service.py†L1-L73】【F:docs/security.md†L1-L96】
20
+ - Recommendation: add a security bundle similar to `setup_sql` that can register `SecurityHeadersMiddleware` and strict CORS when FastAPI is not created via `setup_service_api`. 【F:src/svc_infra/security/headers.py†L1-L35】
21
+
22
+ ### Data and jobs
23
+ - SQL helpers offer both granular (`add_sql_db`, `add_sql_resources`) and bundled (`setup_sql`) flows. They guard against duplicate setup and generate CRUD schemas when not provided, keeping the quick-start feel while allowing custom services. 【F:src/svc_infra/api/fastapi/db/sql/add.py†L1-L118】
24
+ - Jobs expose `easy_jobs()` that switches between memory and Redis based on env, matching the pick-style ergonomics used in logging. Documentation explains scheduler tasks and the CLI runner. 【F:src/svc_infra/jobs/easy.py†L1-L28】【F:docs/jobs.md†L1-L65】
25
+ - Cache decorators export both `cache_read` and the friendlier aliases (`cached`, `mutates`) so consumers can adopt them incrementally. 【F:src/svc_infra/cache/__init__.py†L1-L29】
26
+
27
+ ### Observability
28
+ - `add_observability` lazily imports instrumentation, respects env flags, and returns a shutdown no-op to keep signature compatibility. It tolerates missing dependencies to avoid burdening projects that do not enable metrics. 【F:src/svc_infra/obs/add.py†L1-L68】
29
+ - Grafana and Prometheus templates are packaged separately, but the code path aligns with the simple toggle approach used elsewhere.
30
+
31
+ ### Webhooks and outbox
32
+ - Signature utilities (`require_signature`, `verify_any`) and service abstractions (`WebhookService`, `InMemoryWebhookSubscriptions`) follow the composable style, and docs describe usage alongside the shared job and outbox infrastructure. 【F:src/svc_infra/webhooks/fastapi.py†L1-L37】【F:src/svc_infra/webhooks/service.py†L1-L45】【F:docs/webhooks.md†L1-L59】
33
+ - Gaps relative to the rest of the repo:
34
+ - Router dependencies always create new in-memory stores, so deployments must override dependencies manually. Providing an `add_webhooks(app, store=..., subs=...)` helper that honours env (for example `REDIS_URL`) would mirror `add_sql_db` and `add_observability`. 【F:src/svc_infra/webhooks/router.py†L1-L55】
35
+ - No convenience function registers the router or scheduler tick automatically; consumers must read docs to wire `make_outbox_tick` and queue processors. A top-level helper such as `setup_webhooks(app, outbox=None, inbox=None)` could bundle router inclusion and scheduler guidance.
36
+
37
+ ### Documentation and DX consistency
38
+ - Feature guides (security, jobs, webhooks, idempotency) contain quickstart snippets that mirror the API surface, reinforcing the ease-of-setup story. 【F:docs/security.md†L1-L96】【F:docs/jobs.md†L1-L65】【F:docs/webhooks.md†L1-L59】【F:docs/idempotency.md†L1-L110】
39
+ - README remains empty; adding a high-level index pointing to these guides would help teams discover the single-call helpers faster. 【F:README.md†L1-L4】
40
+ - Consider adding a matrix listing which environment variables drive each helper (`SQL_URL`, `REDIS_URL`, `METRICS_PATH`, `AUTH_*`) to highlight parity across modules.
41
+
42
+ ## Actionable Suggestions
43
+ 1. **Webhook setup helper** — provide `add_webhooks` or `setup_webhooks` to mount the router, accept dependency overrides, and optionally wire scheduler tasks. Default to env-driven outbox or inbox selection to match other modules. 【F:src/svc_infra/webhooks/router.py†L1-L55】
44
+ 2. **Security bundle** — offer a helper that installs `SecurityHeadersMiddleware`, strict CORS, and optional signed-cookie settings for apps that bypass `setup_service_api`. This keeps manual FastAPI apps aligned with the default hardening posture. 【F:src/svc_infra/security/headers.py†L1-L35】
45
+ 3. **Documentation index** — populate `README.md` with links to the feature guides and highlight key add-style helpers so new projects immediately see the plug-and-play building blocks. 【F:README.md†L1-L4】
46
+ 4. **Environment reference** — add a doc or table enumerating env vars consumed by helpers (SQL_URL, REDIS_URL, METRICS_PATH, AUTH_*), emphasising how swapping values maintains flexibility across deployments.
47
+
48
+ Overall, aside from the webhook ergonomics gap and missing README guidance, the codebase maintains the quick-start plus override pattern highlighted in your sample usage while exposing low-level hooks for teams that need deeper control.