@keelcore/standards 0.9.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,158 @@
1
+ # Observability Standards
2
+
3
+ These rules govern metrics, logging, tracing, and alerting across all services. They are non-negotiable.
4
+
5
+ **Maturity:** Required
6
+ **Version:** 1.0.0
7
+ **Last Reviewed:** 2026-03-09
8
+
9
+ ## Metrics
10
+
11
+ ### Vendors and Retention
12
+
13
+ 1. The approved metrics vendor(s) are documented in the internal vendor registry. No additional metrics
14
+ backends are introduced without ARB approval.
15
+ 2. Metrics retention: real-time resolution (≤60s) for 15 days; 5-minute rollup for 90 days;
16
+ 1-hour rollup for 13 months.
17
+ 3. Long-term retention beyond 13 months requires explicit data classification and archival approval.
18
+
19
+ ### Taxonomy and Namespace Schema
20
+
21
+ 4. All metric names follow the schema: `{org}_{service}_{noun}_{unit}` (e.g., `keelcore_auth_requests_total`).
22
+ 5. Metric namespaces are registered centrally; unregistered namespaces are rejected at the ingestion layer.
23
+ 6. Label keys are lowercase snake_case. Label values must not contain high-cardinality free-text (e.g.,
24
+ user IDs, trace IDs, URLs). Cardinality limit per metric: 10,000 distinct label value combinations.
25
+
26
+ ### Required Metric Ontologies
27
+
28
+ 7. Every service exposes RED metrics: **R**ate (requests per second), **E**rrors (error rate),
29
+ **D**uration (latency distribution — p50, p95, p99).
30
+ 8. Every resource-bound component (database, cache, message queue, worker pool) exposes USE metrics:
31
+ **U**tilization, **S**aturation, **E**rrors.
32
+ 9. RED and USE dashboards are provisioned automatically from the service's metric registration.
33
+
34
+ ### SLOs and SLAs
35
+
36
+ 10. Every public-facing service defines at least one SLO expressed as an error budget (e.g., 99.9%
37
+ availability over a 30-day rolling window).
38
+ 11. SLO burn-rate alerts fire at 2× and 10× burn rates and are routed to the on-call rotation.
39
+ 12. SLAs, where contractually defined, are backed by SLOs with sufficient headroom. SLA breach
40
+ thresholds are stricter than internal SLO thresholds.
41
+ 13. Service dashboards are published to the central dashboard registry and linked from the service runbook.
42
+
43
+ ## Logging
44
+
45
+ ### Format
46
+
47
+ 14. All log output is structured JSON. Human-readable plain text logs are prohibited in staging and
48
+ production. Development environments may use formatted output via a local flag.
49
+ 15. Log lines are single-line JSON objects terminated by `\n`. Multi-line log entries are prohibited;
50
+ stack traces are serialized into a single string field.
51
+
52
+ ### Required Fields
53
+
54
+ 16. Every log entry must include:
55
+ - `timestamp` — RFC 3339 with nanosecond precision (e.g., `2026-03-09T12:00:00.000000000Z`)
56
+ - `level` — one of: `debug`, `info`, `warn`, `error`, `fatal`
57
+ - `service` — the service identifier (matches the metric namespace prefix)
58
+ - `version` — the deployed artifact version (SemVer)
59
+ - `trace_id` — W3C TraceContext trace ID (zero value if no active trace)
60
+ - `span_id` — W3C TraceContext span ID (zero value if no active trace)
61
+ - `message` — human-readable description of the event
62
+ 17. Additional fields are encouraged but must not collide with the reserved field list above.
63
+
64
+ ### SDK-Level Enforcement
65
+
66
+ 18. Services must use the organization-approved logging SDK. Calls to raw `fmt.Println`,
67
+ `console.log`, `print()`, or equivalent bypass the structured logger and are prohibited.
68
+ 19. The logging SDK enforces required fields at construction time. A log entry missing a required
69
+ field must fail at build time or panic at startup, not silently emit a malformed log.
70
+
71
+ ### Aggregation and Retention
72
+
73
+ 20. All logs from all environments (staging, production) are shipped to the central log aggregation
74
+ platform. Local-only logging is prohibited in staging and production.
75
+ 21. Log retention: 90 days at full fidelity; 13 months for `warn` and above; 7 years for `audit`
76
+ classified logs.
77
+ 22. Log access is role-gated. Read access to production logs requires an approved role. Queries
78
+ on `audit`-classified logs are themselves logged.
79
+
80
+ ### Redaction and Privacy
81
+
82
+ 23. Personally identifiable information (PII) must not appear in log fields except `audit`-classified
83
+ logs with explicit data handling approval.
84
+ 24. Secrets, tokens, passwords, and private keys must never be logged, even at `debug` level.
85
+ 25. The logging SDK must provide a redaction hook that masks known-sensitive field patterns before
86
+ emission. Services must register PII-bearing fields with the redaction registry.
87
+ 26. Audit logs that require PII (e.g., access logs for compliance) are stored in a segregated
88
+ audit log store with stricter access controls and encryption-at-rest.
89
+
90
+ ## Distributed Tracing
91
+
92
+ ### Framework
93
+
94
+ 27. All services instrument using OpenTelemetry (OTel). Vendor-specific tracing SDKs are prohibited
95
+ as the primary instrumentation layer. OTel exporters may target any approved backend.
96
+ 28. Trace context is propagated using the W3C TraceContext standard (`traceparent`, `tracestate` headers).
97
+ Proprietary propagation formats (B3, X-Cloud-Trace-Context) are acceptable as secondary propagation
98
+ only for legacy integration.
99
+
100
+ ### Propagation Policy
101
+
102
+ 29. Every inbound request must extract trace context from headers if present, or generate a new root
103
+ span if absent.
104
+ 30. Every outbound call must inject the current trace context into request headers.
105
+ 31. Internal background jobs and queue consumers must attach to the trace context embedded in the
106
+ job/message payload.
107
+ 32. Trace context must cross service, database, queue, and cache boundaries. Traces that terminate
108
+ at a service boundary rather than propagating downstream are a defect.
109
+
110
+ ### Sampling
111
+
112
+ 33. Production sampling: head-based sampling at 1% for high-volume steady-state traffic, with tail-based
113
+ sampling capturing 100% of error traces and traces exceeding the p99 latency threshold.
114
+ 34. Sampling configuration is centralized (not per-service). Services do not configure their own sampling
115
+ rates without approval.
116
+ 35. Staging and development environments use 100% sampling.
117
+
118
+ ## Audit Logging
119
+
120
+ ### Immutability and Trail Requirements
121
+
122
+ 36. Sensitive actions (authentication events, authorization decisions, configuration changes, data
123
+ exports, administrative actions) must emit an `audit`-classified log entry.
124
+ 37. Audit logs are written to an append-only, tamper-evident store. Application processes have write-only
125
+ access; no application process may read, modify, or delete audit log entries.
126
+ 38. Every audit log entry includes: `timestamp`, `actor_id`, `actor_type` (human or service),
127
+ `action`, `resource_type`, `resource_id`, `outcome` (success or failure), `client_ip`, `trace_id`.
128
+ 39. Access to resources and modifications to data are logged, including read access to sensitive data.
129
+
130
+ ### Retention
131
+
132
+ 40. Audit logs are retained for a minimum of 7 years unless a longer period is required by applicable
133
+ regulation.
134
+ 41. Audit log exports for compliance review require dual-approval and are themselves logged.
135
+
136
+ ## Alerting
137
+
138
+ ### Thresholds and Routing
139
+
140
+ 42. Every SLO has an associated burn-rate alert. Alerts fire at 2× burn (page candidate) and 10× burn
141
+ (immediate page).
142
+ 43. `error` and `fatal` log events that exceed a per-service threshold trigger an alert.
143
+ 44. Alert routing follows the on-call schedule managed in the central incident management platform.
144
+ 45. Alerts include: service name, environment, metric/log source, threshold breached, current value,
145
+ link to the relevant dashboard, and link to the runbook.
146
+ 46. Runbook links in alerts must resolve. Broken runbook links are treated as a P2 incident.
147
+
148
+ ## Do Not
149
+
150
+ - Emit plain-text or unstructured logs in staging or production.
151
+ - Log PII outside of approved audit log stores.
152
+ - Log secrets, tokens, or credentials at any log level.
153
+ - Define per-service sampling rates without central approval.
154
+ - Omit required log fields.
155
+ - Use vendor-specific tracing SDKs as the primary instrumentation layer.
156
+ - Allow traces to terminate at a service boundary rather than propagating downstream.
157
+ - Deploy a service without RED metrics, SLO definition, and a linked dashboard.
158
+ - Ship alerts without runbook links.
@@ -0,0 +1,167 @@
1
+ # Platform Architecture Standards
2
+
3
+ These rules govern platform-level decisions across networking, DNS, TLS, WAF, and traffic management.
4
+ They are non-negotiable unless explicitly superseded by a signed ADR.
5
+
6
+ **Maturity:** Required
7
+ **Version:** 1.0.0
8
+ **Last Reviewed:** 2026-03-09
9
+
10
+ ## Governance Process
11
+
12
+ ### Decision-Making (DACI)
13
+
14
+ 1. All platform architecture decisions use DACI: one Driver, one Approver (Architecture Review Board
15
+ chair or delegate), Consulted stakeholders, and Informed parties.
16
+ 2. The Driver authors the RFC and owns shepherding it through review.
17
+ 3. The Approver has final authority; no decision is binding without Approver sign-off.
18
+ 4. Decisions that affect more than one service domain require ARB review before implementation.
19
+
20
+ ### RFC Process
21
+
22
+ 5. Any change to platform standards, vendor selection, or network topology requires an RFC.
23
+ 6. RFC template: problem statement, proposed solution, alternatives considered, rollout plan, rollback plan.
24
+ 7. RFC comment period is a minimum of five business days for standard changes; urgent security patches
25
+ may compress to 24 hours with ARB chair notification.
26
+ 8. Accepted RFCs are archived in `docs/rfc/` with status and the date of acceptance.
27
+
28
+ ### Architecture Review Board (ARB)
29
+
30
+ 9. ARB meets on a defined cadence (minimum monthly); decisions and attendance are published to the
31
+ engineering wiki within five business days of each meeting.
32
+ 10. ARB publishes a versioned Architecture Standards document at least once per quarter.
33
+ 11. Previous versions of Architecture Standards are archived and permanently accessible.
34
+
35
+ ## Domain and IP
36
+
37
+ ### Domain Registrar
38
+
39
+ 12. All domains are registered through the approved corporate registrar (documented in the internal
40
+ vendor registry). Personal or team-owned registrar accounts are prohibited for production domains.
41
+ 13. Domain transfers require ARB approval and a change record.
42
+ 14. WHOIS privacy protection is enabled on all externally registered domains.
43
+
44
+ ### IP Address Management
45
+
46
+ 15. IP allocations are managed through the approved IPAM vendor (documented in the internal vendor registry).
47
+ 16. All IP blocks are tagged with owner, environment, and region.
48
+ 17. IPv4 and IPv6 are both required for all new public-facing services. Dual-stack is the baseline; IPv4-only
49
+ services are not accepted for new deployments.
50
+ 18. Private RFC 1918 ranges are used for internal pod and node networks. Overlap with VPN or on-premises
51
+ ranges must be approved through the IPAM change process.
52
+
53
+ ## DNS
54
+
55
+ ### Zone Management
56
+
57
+ 19. DNS zones are managed through the approved DNS vendor (documented in the internal vendor registry).
58
+ Manual zone edits via vendor UI are prohibited except in a documented break-glass scenario.
59
+ 20. All zone changes are driven from infrastructure-as-code (Terraform, Pulumi, or equivalent) in a
60
+ version-controlled repository. Zone state is reconciled on every deployment.
61
+ 21. Internal and external zones are segregated. Internal zone resolvers are not exposed to the public internet.
62
+
63
+ ### Required Record Types
64
+
65
+ 22. Public services must expose: A, AAAA, and the appropriate CNAME or alias records.
66
+ 23. Email domains must have SPF, DKIM, and DMARC records. DMARC policy must be `p=quarantine` or stricter
67
+ in production.
68
+ 24. Wildcard DNS records (`*.example.com`) are permitted only for edge/ingress controllers that validate
69
+ per-host routing internally. Wildcard records must not resolve to application pods directly.
70
+ 25. Internal service discovery uses a dedicated internal TLD (e.g., `.internal` or cluster DNS suffix).
71
+ Internal records are never propagated to public resolvers.
72
+
73
+ ## TLS and Certificate Management
74
+
75
+ ### Certificate Issuance
76
+
77
+ 26. TLS certificates for public endpoints are issued from an approved CA vendor (documented in the internal
78
+ vendor registry). Self-signed certificates are prohibited on any public-facing endpoint.
79
+ 27. Certificate issuance and renewal must use the ACME protocol. Manual certificate downloads and uploads
80
+ are prohibited except in a documented break-glass scenario.
81
+ 28. Certificates are provisioned and renewed automatically. Expiry-based alerts fire at 30 days and again
82
+ at 7 days before expiration.
83
+
84
+ ### Mutual TLS (mTLS)
85
+
86
+ 29. East-west traffic between services must use mTLS. Plaintext service-to-service communication is
87
+ prohibited in staging and production.
88
+ 30. mTLS certificates are issued by the internal PKI (SPIFFE/SPIRE or equivalent). Leaf certificates
89
+ rotate on a maximum 24-hour lifetime. Rotation is automatic and zero-downtime.
90
+ 31. mTLS policy is enforced at the service mesh layer, not individually per application.
91
+
92
+ ### Supported Protocols and Cipher Suites
93
+
94
+ 32. Minimum TLS version: TLS 1.2. TLS 1.3 is required for all new integrations.
95
+ 33. TLS 1.0 and TLS 1.1 are prohibited on all endpoints.
96
+ 34. Cipher suites: prefer ECDHE key exchange with AES-GCM or ChaCha20-Poly1305. RC4, DES, 3DES,
97
+ and export-grade ciphers are prohibited. DHE suites below 2048-bit DH parameters are prohibited.
98
+ 35. HSTS is required on all HTTPS endpoints with `max-age` of at least one year.
99
+
100
+ ## Web Application Firewall (WAF) and DDoS
101
+
102
+ ### WAF
103
+
104
+ 36. All public HTTP/HTTPS ingress passes through the WAF layer before reaching application pods.
105
+ 37. The OWASP Core Rule Set (CRS) is enforced in blocking mode. Detection-only mode is not
106
+ acceptable for production. Tuning exceptions require a documented rule with justification.
107
+ 38. WAF rules are version-controlled. Rule changes follow the RFC process.
108
+ 39. WAF logs are ingested into the central log aggregation platform with a minimum 90-day retention.
109
+
110
+ ### DDoS Protection
111
+
112
+ 40. All public endpoints are protected by volumetric DDoS mitigation at the network edge.
113
+ 41. L7 DDoS (HTTP flood) mitigation is enforced by the WAF layer with rate limiting rules.
114
+ 42. Incident response runbooks for DDoS events are maintained and reviewed quarterly.
115
+
116
+ ## Security Headers
117
+
118
+ 43. All HTTP responses from public endpoints must include:
119
+ - `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
120
+ - `X-Content-Type-Options: nosniff`
121
+ - `X-Frame-Options: DENY`
122
+ - `Content-Security-Policy` (service-specific, reviewed by security team)
123
+ - `Referrer-Policy: strict-origin-when-cross-origin`
124
+ - `Permissions-Policy` (restrictive default; expand per-service with justification)
125
+ 44. `Server` and `X-Powered-By` response headers must be suppressed or overridden to not reveal
126
+ implementation details.
127
+
128
+ ## Traffic Management
129
+
130
+ ### Payload Limits
131
+
132
+ 45. Default maximum upload size: 10 MB. Services requiring larger uploads must document and configure
133
+ an explicit limit; unbounded uploads are prohibited.
134
+ 46. Maximum request body size: 1 MB by default for API endpoints. Streaming endpoints must implement
135
+ backpressure.
136
+ 47. Maximum response size is bounded by the service contract. Unbounded response streaming must
137
+ implement flow control.
138
+
139
+ ### Rate Limiting
140
+
141
+ 48. Global rate limiting is enforced at the edge/ingress layer.
142
+ 49. Per-service rate limits are configured and documented in each service's runbook.
143
+ 50. Rate limit responses use HTTP 429 with a `Retry-After` header.
144
+ 51. Rate limit state is shared across edge replicas (e.g., Redis-backed or edge-native distributed
145
+ counters). Per-replica rate limiting that allows bypass via replica selection is prohibited.
146
+
147
+ ### Session Management
148
+
149
+ 52. Session tokens are cryptographically random with a minimum of 128 bits of entropy.
150
+ 53. Session lifetime: configurable per service, with a maximum idle timeout of 15 minutes and a
151
+ maximum absolute session lifetime of 8 hours for human-facing sessions.
152
+ 54. Sessions are invalidated server-side on logout. Client-side deletion alone is insufficient.
153
+ 55. Session refresh tokens must be rotated on every use.
154
+
155
+ ## Do Not
156
+
157
+ - Register production domains outside the approved registrar.
158
+ - Assign IP blocks without IPAM tracking.
159
+ - Modify DNS zones outside of infrastructure-as-code.
160
+ - Deploy new services without IPv6 support.
161
+ - Use TLS 1.1 or below on any endpoint.
162
+ - Issue certificates manually when ACME automation is available.
163
+ - Allow service-to-service plaintext in staging or production.
164
+ - Run OWASP WAF rules in detection-only mode in production.
165
+ - Deploy public endpoints without DDoS protection.
166
+ - Omit required security headers.
167
+ - Accept unbounded file uploads without an explicit, documented size cap.
@@ -0,0 +1,171 @@
1
+ # Release Policy
2
+
3
+ **Maturity:** Required
4
+ **Version:** 1.0.0
5
+ **Last Reviewed:** 2026-03-11
6
+
7
+ This document covers how Keel versions are tagged and released. The authoritative
8
+ policy lives in the [keelcore/standards `ci.md`](ci.md)
9
+ under **Release Tagging**. This document is the developer-facing how-to that
10
+ complements that policy.
11
+
12
+ ---
13
+
14
+ ## Versioning Specification
15
+
16
+ Keel follows **Semantic Versioning 2.0.0** — [https://semver.org](https://semver.org).
17
+ That document is the canonical reference; the rules below are derived from it and
18
+ must not contradict it.
19
+
20
+ Key semver.org rules in force here:
21
+
22
+ - Version numbers take the form `MAJOR.MINOR.PATCH` where each component is a
23
+ non-negative integer with no leading zeroes.
24
+ - MAJOR is incremented for incompatible API changes; MINOR and PATCH reset to 0.
25
+ - MINOR is incremented for new backward-compatible functionality; PATCH resets to 0.
26
+ - PATCH is incremented for backward-compatible bug fixes only.
27
+ - Major version zero (`0.y.z`) is for initial development. The public API is not
28
+ yet stable and anything may change at any time.
29
+ - `1.0.0` defines the first stable public API.
30
+ - Pre-release identifiers (`-alpha.1`, `-rc.2`, etc.) and build metadata (`+sha`)
31
+ are valid semver extensions but are **not supported by `create-release.sh`** at
32
+ this time. `--force` only accepts bare `MAJOR.MINOR.PATCH` versions.
33
+
34
+ **Scope of automated detection:** The script detects breaking changes by diffing
35
+ the config YAML field surface (`pkg/config/schema.yaml`). Other API-breaking
36
+ changes — Go type signatures, HTTP API shape, security policy, behavior semantics
37
+ — are not automatically detected. Use `--force` with the appropriate bump level
38
+ for those cases.
39
+
40
+ ---
41
+
42
+ ## Core Rule: Tags Are Developer-Created, Never CI-Created
43
+
44
+ CI reacts to pushed tags to run the release pipeline. CI never creates tags.
45
+ A version tag is a human commitment about what a commit represents — that
46
+ decision requires review and cannot be automated.
47
+
48
+ ---
49
+
50
+ ## Tooling
51
+
52
+ | Script | Purpose |
53
+ | --- | --- |
54
+ | `scripts/release/create-release.sh` | Compute, approve, tag, and push a release |
55
+ | `scripts/release/gen-schema.sh` | Regenerate `pkg/config/schema.yaml` after config changes |
56
+ | `cmd/config-schema/main.go` | Go tool: reflection-walk of `config.Config` → dotted YAML paths |
57
+ | `make gen-schema` | Alias for `gen-schema.sh` |
58
+ | `make create-release` | Alias for `create-release.sh` (auto mode) |
59
+ | `make create-release FORCE=v1.0.0` | Alias for `create-release.sh --force v1.0.0` |
60
+
61
+ ---
62
+
63
+ ## How Version Bumps Are Computed
64
+
65
+ The script diffs `pkg/config/schema.yaml` at the previous tag against the file
66
+ at `HEAD`. The schema lists every fully-flattened dotted YAML field path in
67
+ `config.Config` (e.g. `sidecar.circuit_breaker.reset_timeout`).
68
+
69
+ | Schema change | Bump level | semver.org rationale |
70
+ | --- | --- | --- |
71
+ | One or more fields **removed** | **major** — breaking change | Removing a field is an incompatible API change ([item 4](https://semver.org/#spec-item-4)) |
72
+ | One or more fields **added**, none removed | **minor** — new feature | New fields are backward-compatible additions ([item 5](https://semver.org/#spec-item-5)) |
73
+ | No field surface changes | **patch** — internal improvements | No public API surface change ([item 6](https://semver.org/#spec-item-6)) |
74
+
75
+ ---
76
+
77
+ ## Normal Release Flow
78
+
79
+ ```bash
80
+ git checkout main && git pull --ff-only
81
+ # ... merge your work ...
82
+ ./scripts/release/create-release.sh
83
+ ```
84
+
85
+ The script will:
86
+
87
+ 1. Verify you are on `main`, your branch's configured upstream is in sync, and the working tree is clean.
88
+ 2. Compute the field diff and determine the bump level.
89
+ 3. Print the summary (fields added/removed, bump level, proposed tag message).
90
+ 4. Prompt for `y/N` approval.
91
+ 5. On approval: create a signed annotated tag and push it to the upstream remote.
92
+
93
+ CI picks up the pushed tag and runs the release pipeline (build, sign, SBOM,
94
+ publish).
95
+
96
+ ---
97
+
98
+ ## Forcing a Specific Version
99
+
100
+ Use `--force vX.Y.Z` to override the computed version. Only bare `MAJOR.MINOR.PATCH`
101
+ versions are accepted — pre-release extensions (`-alpha.1`) and build metadata
102
+ (`+sha`) are not supported.
103
+
104
+ ```bash
105
+ ./scripts/release/create-release.sh --force v1.0.0
106
+ ```
107
+
108
+ ### Pre-1.0 rules (current major = 0)
109
+
110
+ Per [semver.org item 9](https://semver.org/#spec-item-9), `0.y.z` is for initial
111
+ development where the API is not yet stable. Only the following targets are accepted:
112
+
113
+ - Any `0.x.y` where `0.x.y > current` — normal initial-development progression.
114
+ - Exactly `v1.0.0` — the stable API promotion per [semver.org item 10](https://semver.org/#spec-item-10),
115
+ regardless of detected bump level.
116
+
117
+ Forcing `v1.1.0`, `v2.0.0`, or any other version outside this range is rejected.
118
+
119
+ ### Post-1.0 rules (current major ≥ 1)
120
+
121
+ Once `1.0.0` is released the public API is stable. `--force` is restricted to
122
+ exact single-step increments per semver.org's ordering rules:
123
+
124
+ | `--force` target | Condition | Result |
125
+ | --- | --- | --- |
126
+ | `cur_maj.cur_min.(cur_pat+1)` | any | **Allowed** — single patch step |
127
+ | `cur_maj.(cur_min+1).0` | any | **Allowed** — single minor step (e.g. internal improvements not reflected in schema diff) |
128
+ | `(cur_maj+1).0.0` | breaking changes detected AND matches computed version | **Allowed** |
129
+ | `(cur_maj+1).0.0` | no breaking changes, or doesn't match computed | **Rejected** — major bump requires a detected breaking change |
130
+ | anything else | — | **Rejected** — must be a single-step increment |
131
+
132
+ ---
133
+
134
+ ## Keeping schema.yaml Fresh
135
+
136
+ `pkg/config/schema.yaml` is a committed artifact generated from `config.Config`
137
+ by reflection. The pre-commit hook regenerates and stages it automatically
138
+ whenever any `pkg/config/*.go` file is part of a commit — no manual step required.
139
+
140
+ To regenerate it outside of a commit (e.g. to inspect the current output):
141
+
142
+ ```bash
143
+ ./scripts/release/gen-schema.sh
144
+ # or
145
+ make gen-schema
146
+ ```
147
+
148
+ The consistency test suite enforces that `schema.yaml` is never stale — a
149
+ mismatch between the committed file and the reflection output fails CI.
150
+
151
+ ---
152
+
153
+ ## What Goes in the Tag Message
154
+
155
+ The script auto-generates a signed annotated tag with a structured message:
156
+
157
+ - **Subject line:** `{bump}: {short summary}` (e.g. `minor: added 2 config field(s)`)
158
+ - **Body:** full field list, bump level, previous/new version, commit SHA, and
159
+ provenance trailers (`Generated-by:`, `Schema:`)
160
+
161
+ The GitHub Release body is populated directly from the tag message via
162
+ `gh release create --notes-from-tag`.
163
+
164
+ ---
165
+
166
+ ## Emergency / Break-Glass
167
+
168
+ If a critical patch must ship and the normal flow is blocked, follow the
169
+ break-glass procedure in `docs/break-glass.md`. The bypass still requires two
170
+ senior engineers and an incident record — it does not bypass the tag-creation
171
+ requirement, only the precondition checks.
@@ -0,0 +1,165 @@
1
+ # Runtime and Orchestration Standards
2
+
3
+ These rules govern container design, orchestration, deployment, and policy compliance.
4
+ They are non-negotiable unless explicitly superseded by a signed ADR.
5
+
6
+ **Maturity:** Required
7
+ **Version:** 1.0.0
8
+ **Last Reviewed:** 2026-03-09
9
+
10
+ ## Containers and Artifacts
11
+
12
+ ### Versioning
13
+
14
+ 1. All published artifacts (container images, packages, binaries) use Semantic Versioning (SemVer).
15
+ Unversioned or date-stamped-only artifacts are prohibited.
16
+ 2. The artifact tag is the SemVer string. The `latest` tag must not be used in staging or
17
+ production manifests.
18
+
19
+ ### Container Design
20
+
21
+ 3. One process per container. Process supervisors (supervisord, s6) and multi-process containers
22
+ are prohibited except for sidecar patterns enforced by the service mesh.
23
+ 4. Multiple entry points (different commands/args) for the same image are allowed where the image
24
+ provides genuinely distinct modes (e.g., `serve` and `migrate`). Each entry point must be
25
+ documented in the image's README.
26
+ 5. Containers run as a non-root user. `USER root` in the final image stage is prohibited.
27
+ 6. Container filesystems are read-only where possible. Writable directories are explicitly declared
28
+ and limited to ephemeral paths (`/tmp`, named volumes).
29
+
30
+ ### Artifact Metadata
31
+
32
+ 7. Every artifact must embed: Git commit SHA, repository URL, branch, build timestamp, and SemVer.
33
+ These are queryable via a `version` subcommand or equivalent metadata endpoint.
34
+ 8. Artifacts are tagged with: SemVer, commit SHA short form, and environment target (`dev`, `staging`,
35
+ `prod`). Promotion from one environment to the next reuses the same image digest — it is never rebuilt.
36
+
37
+ ### Registry Lifecycle
38
+
39
+ 9. Image promotion lifecycle: `dev` registry → `staging` registry → `prod` registry.
40
+ Artifacts flow in one direction; no artifact is pulled from `prod` into `dev`.
41
+ 10. Artifacts not promoted to `staging` within 30 days of build are expired from the `dev` registry.
42
+ 11. Artifacts in `prod` are retained for a minimum of 12 months.
43
+
44
+ ### Security Scanning
45
+
46
+ 12. CVE scanning runs on every artifact before it is pushed to any registry. The approved scanner
47
+ is documented in the internal vendor registry.
48
+ 13. Critical CVEs block promotion. High CVEs require a documented exception with an accepted
49
+ remediation timeline (maximum 30 days). Medium and below are tracked but do not block promotion.
50
+ 14. A Software Bill of Materials (SBOM) in SPDX or CycloneDX format is generated for every release
51
+ artifact. The SBOM is attached to the release and stored alongside the artifact digest.
52
+
53
+ ## Orchestration
54
+
55
+ ### Declarative Manifests
56
+
57
+ 15. All workloads are declared via version-controlled orchestration manifests (Kubernetes YAML,
58
+ Helm charts, Kustomize overlays, or equivalent). Manual `kubectl apply` of locally generated
59
+ YAML in production is prohibited.
60
+ 16. Manifests are validated (lint + schema check) as a required CI step before merge.
61
+
62
+ ### Probes
63
+
64
+ 17. Every container that serves traffic must define `livenessProbe`, `readinessProbe`, and
65
+ optionally `startupProbe`.
66
+ 18. Readiness probes must reflect actual service readiness (e.g., database connection established,
67
+ cache warm) not just process startup.
68
+ 19. Liveness probe failures must trigger a container restart, not a silent hang. Liveness probe
69
+ endpoints must not have side effects.
70
+
71
+ ### Resource Quotas
72
+
73
+ 20. Every workload declares `resources.requests` and `resources.limits` for CPU and memory.
74
+ Pods without resource declarations are rejected by admission control.
75
+ 21. Resource requests must reflect measured steady-state usage. Over-requesting by more than 3×
76
+ the p99 measured usage requires justification.
77
+ 22. Namespace-level `ResourceQuota` and `LimitRange` objects are applied to all tenant namespaces.
78
+
79
+ ### High Availability
80
+
81
+ 23. All production workloads run in a minimum of two availability zones. Single-zone deployments
82
+ are prohibited in production.
83
+ 24. `PodAntiAffinity` rules are configured to prevent all replicas of a deployment from scheduling
84
+ onto a single node or zone.
85
+ 25. Minimum replica count for production services is 2. Services with replica count 1 require
86
+ documented justification and ARB approval.
87
+
88
+ ## Deployment
89
+
90
+ ### Deployment Strategy
91
+
92
+ 26. All production deployments use a rolling, blue/green, or canary strategy. Big-bang
93
+ (recreate) deploys are prohibited in production.
94
+ 27. Deployments support rollback to the previous artifact version within 5 minutes.
95
+ 28. Deployment events are logged in the audit log with: artifact version, deploy actor, environment,
96
+ start time, end time, and outcome.
97
+
98
+ ### Zero-Downtime Deployment
99
+
100
+ 29. Zero-downtime deployment capability is required for all production services.
101
+ 30. Traffic shaping during rollout: network load balancer or service mesh gradually shifts
102
+ traffic from the old version to the new version. Hard cutover is prohibited.
103
+ 31. APIs must be backward-compatible across a deployment window. Breaking API changes require
104
+ a versioned endpoint strategy before the old version is retired.
105
+ 32. Schema migrations (database, event schema) are non-breaking and phased:
106
+ - Phase 1: add new column/field (backward-compatible with old code).
107
+ - Phase 2: deploy new code that writes both old and new.
108
+ - Phase 3: backfill and migrate data.
109
+ - Phase 4: remove old column/field only after old code is fully retired.
110
+ 33. Feature flags or service mesh routing rules are used to gate new behavior during deployment.
111
+ A feature can be promoted or rolled back independently of the artifact version.
112
+
113
+ ### Service Mesh
114
+
115
+ 34. All east-west (pod-to-pod) traffic runs through the service mesh. Direct pod-to-pod
116
+ communication bypassing the mesh is prohibited in staging and production.
117
+ 35. The mesh enforces mTLS, authorization policy, and traffic observability on all managed traffic.
118
+ 36. Circuit breakers and retry budgets are configured per service. Default retry-all-failures
119
+ behavior is prohibited; retries must be limited and idempotent-safe.
120
+
121
+ ### Secrets Management
122
+
123
+ 37. Application secrets (database passwords, API keys, private keys) are injected at runtime
124
+ from the approved secrets manager. Secrets must not be embedded in container images,
125
+ manifests, environment variable literals, or version-controlled configuration.
126
+ 38. Secrets are mounted as in-memory volumes or environment variables injected by the secrets
127
+ sidecar/operator. Secrets written to the container filesystem at rest are prohibited.
128
+ 39. Secret access is logged by the secrets manager. Unexpected access patterns alert the
129
+ security team.
130
+
131
+ ## Policy and Compliance
132
+
133
+ ### Change Management
134
+
135
+ 40. All infrastructure changes are managed via GitOps or a declarative change management system.
136
+ Out-of-band changes (manual console, ad-hoc CLI) are prohibited in staging and production and
137
+ will be detected and reverted by reconciliation.
138
+ 41. Production changes require approval from at least one person who is not the change author.
139
+ 42. Every production change record includes a rollback plan. Changes without a tested rollback plan
140
+ are not approved.
141
+ 43. CTO and ARB policy documents are archived and versioned. Historical policy versions are accessible.
142
+
143
+ ### Compliance
144
+
145
+ 44. The platform maintains SOC-2 Type II compliance. Evidence collection is automated where possible.
146
+ 45. GDPR and CCPA requirements apply to all data processing pipelines. Privacy impact assessments
147
+ are required for new data processing features that handle personal data.
148
+ 46. A Secure Development Lifecycle (SDL) is followed: threat modeling, SAST, DAST, dependency
149
+ scanning, and penetration testing are performed on a defined cadence.
150
+ 47. ISO 27001 and NIST SP 800-53 are used as optional benchmarks to inform control selection.
151
+ Controls are documented but full certification is optional unless contractually required.
152
+
153
+ ## Do Not
154
+
155
+ - Use the `latest` tag in staging or production manifests.
156
+ - Run containers as root.
157
+ - Deploy workloads without liveness and readiness probes.
158
+ - Deploy workloads without CPU and memory resource limits.
159
+ - Run a production service with fewer than two availability zones.
160
+ - Perform big-bang (recreate) deploys in production.
161
+ - Embed secrets in images, manifests, or version-controlled configuration.
162
+ - Allow service-to-service traffic to bypass the service mesh in staging or production.
163
+ - Perform breaking schema migrations in a single phase.
164
+ - Make infrastructure changes outside of the approved GitOps workflow.
165
+ - Approve a production change without a rollback plan.