@wazir-dev/cli 1.0.0

package/expertise/architecture/scaling/feature-flags-and-rollouts.md

@@ -0,0 +1,757 @@
+# Feature Flags and Rollouts — Architecture Expertise Module
+
+> Feature flags decouple deployment from release — code ships to production but features activate independently via configuration. Combined with progressive rollouts (canary, percentage-based, ring-based), they dramatically reduce deployment risk. But feature flags accumulate as technical debt and require active lifecycle management. Treat every flag as a short-lived construct with an expiration date, not a permanent branch in your code.
+
+> **Category:** Scaling
+> **Complexity:** Moderate
+> **Applies when:** Teams practicing continuous deployment, releasing features to subsets of users, A/B testing, managing feature access by tenant/plan, or needing kill switches for operational safety
+
+---
+
+## What This Is (and What It Isn't)
+
+A feature flag (also called a feature toggle, feature switch, or feature gate) is a **runtime configuration mechanism that controls whether a code path is active without redeploying the application**. At its simplest, it is a conditional branch:
+
+```python
+if feature_flags.is_enabled("new_checkout_flow", user=current_user):
+    return render_new_checkout(cart)
+else:
+    return render_legacy_checkout(cart)
+```
+
+But a production-grade feature flag system is far more than an `if` statement. It includes:
+
+- **Targeting rules** — flag checks per-user, per-segment, per-tenant, per-geography, per-percentage
+- **Flag lifecycle management** — creation, activation, GA promotion, and removal
+- **Audit trails** — who changed what flag, when, and why
+- **Real-time propagation** — flag changes take effect in seconds across all running instances
+- **SDK-based checking** — server-side and client-side SDKs that check flags locally for performance
+
+The concept originated in the mid-2000s at companies like Flickr (feature flippers, 2009) and Facebook (Gatekeeper, 2008), where deploying to millions of users demanded a way to separate "code is deployed" from "feature is live." Martin Fowler formalized the taxonomy in his seminal 2017 article on Feature Toggles, which remains the definitive reference.
+
+### The Four Categories of Feature Flags
+
+Not all flags are equal. Each category has different lifespans, ownership, and risk profiles:
+
+| Category | Purpose | Lifespan | Owner | Example |
+|---|---|---|---|---|
+| **Release flags** | Decouple deployment from release; hide incomplete features | Days to weeks | Engineering | `enable_new_search_ui` |
+| **Experiment flags** | A/B testing, multivariate testing | Weeks to months | Product/Growth | `checkout_flow_variant` (A/B/C) |
+| **Ops flags** | Kill switches, circuit breakers, load shedding | Permanent (but rarely toggled) | SRE/Operations | `disable_recommendation_engine` |
+| **Permission flags** | Entitlement gating, plan-based access | Permanent | Product/Business | `enable_premium_analytics` |
+
+**Release flags** are the most common and the most dangerous from a debt perspective — they should be removed within days of a feature going to 100%, but in practice they linger for months or years.
+
+**Experiment flags** carry multivariate values (not just boolean) and require statistical rigor — they integrate with analytics pipelines and demand sample size calculations before activation.
+
+**Ops flags** are the safest category because they represent an intentional permanent control plane. A kill switch for a non-critical subsystem (`disable_email_notifications`) is a legitimate operational tool, not debt.
+
+**Permission flags** overlap with entitlement/authorization systems. In multi-tenant SaaS, these often map to plan tiers (free/pro/enterprise) and should ideally live in the authorization layer, not the feature flag system.
+
+### What Feature Flags Are NOT
+
+- **Not a replacement for proper branching strategy.** Flags enable trunk-based development, but the code behind the flag still needs design, review, and testing. A flag does not excuse merging untested code.
+- **Not a permanent architectural construct.** Release and experiment flags must have an expiration date. A flag that has been "on" for all users for six months is not a feature flag — it is dead code and a liability.
+- **Not an authorization system.** While permission flags can gate features by plan/role, complex authorization logic belongs in a dedicated authz layer (RBAC, ABAC, policy engines) — not in feature flag conditionals scattered across the codebase.
+- **Not a configuration management system.** Feature flags control feature activation. Application configuration (database URLs, timeout values, retry counts) belongs in environment variables or configuration services. Conflating the two creates a sprawling, unmanageable flag inventory.
+- **Not free.** Every flag adds a branch to your code. Two flags create 4 possible states. Ten flags create 1,024. Twenty flags create over a million. The combinatorial explosion in testing and reasoning about system behavior is a real cost.
+
+---
+
+## When to Use It
+
+### Trunk-based development with continuous deployment
+
+Feature flags are the essential enabler of trunk-based development (TBD) — the practice where all developers commit to a single main branch, integrating changes multiple times per day. Without flags, incomplete features on trunk would be exposed to users. With flags, developers merge daily, incomplete features ship behind flags in the "off" state, and the codebase stays permanently deployable.
+
+**Google, Facebook, and LinkedIn** all practice trunk-based development at massive scale. Google's entire codebase (billions of lines) lives in a single repository with a single branch, and feature flags (via their internal "Gating" system) control what is visible to users.
+
+The workflow: developer creates a flag, writes code behind the flag, merges to trunk daily, deploys to production (flag off), gradually rolls out via targeting rules, reaches 100%, removes the flag. No long-lived branches, no merge hell, no "big bang" integration.
+
+### Gradual rollouts to reduce blast radius
+
+When a feature impacts millions of users, releasing to 100% simultaneously is reckless. Progressive rollout strategies use feature flags to control exposure:
+
+- **Canary (1%)** — deploy to a tiny fraction, monitor error rates and latency
+- **Early adopters (10%)** — expand to opted-in users or internal employees
+- **Broader rollout (25% to 50%)** — increase while watching metrics
+- **General availability (100%)** — full release, then remove the flag
+
+**Netflix** uses this pattern extensively. A new recommendation algorithm might serve 1% of users for a week while engineers monitor engagement metrics, error rates, and latency percentiles. Only when the data shows positive or neutral impact does the rollout expand.
+
+### A/B testing and experimentation
+
+Multivariate feature flags are the backbone of product experimentation. An experiment flag assigns users to cohorts (control vs. variant A vs. variant B) and tracks conversion metrics per cohort. This requires:
+
+- Consistent assignment (same user always sees same variant for the experiment duration)
+- Statistical significance tracking
+- Integration with analytics/data pipelines
+- Mutual exclusion between overlapping experiments
+
+**Booking.com** runs over 1,000 concurrent A/B tests at any time. Their experimentation platform is built on feature flags with multivariate assignment, and they attribute much of their revenue optimization to disciplined experimentation.
+
+### Kill switches for operational safety
+
+Ops flags let you instantly disable non-critical subsystems when the system is under stress. During a traffic spike that threatens database capacity, an ops flag can disable the recommendation engine, email notifications, or analytics tracking — shedding load without a deployment.
+
+This is one of the few legitimate uses of a permanent feature flag. The flag exists as a documented operational control, not as a temporary release mechanism.
+
+### Tenant-specific and plan-based feature gating
+
+In multi-tenant SaaS systems, feature flags control which tenants or plan tiers have access to specific features. Enterprise customers get advanced analytics; free-tier users see the basic dashboard. This is particularly useful during migrations — you can gradually roll out a new billing system tenant-by-tenant rather than switching everyone simultaneously.
+
+---
+
+## When NOT to Use It
+
+**This section is deliberately long because feature flag debt is one of the most insidious forms of technical debt — it accumulates silently, creates combinatorial complexity, and has caused some of the most expensive incidents in software history.**
+
+### When you have no plan for flag removal
+
+If your team creates flags but has no process for removing them, you will accumulate hundreds of stale flags within a year. Each stale flag is a branch in your code that no one understands, no one tests, and no one wants to remove because "what if we need it again?"
+
+**Real-world evidence:** Industry surveys consistently show that most organizations have hundreds of stale flags in their codebases. The average flag lives 10x longer than intended. A codebase with 200 stale flags is harder to maintain, harder to reason about, and harder to test than a codebase with zero flags.
+
+**The rule:** Never create a flag without an expiration date and an assigned owner responsible for removal. If your team cannot commit to this discipline, do not adopt feature flags — they will hurt you more than they help.
+
+### When testing discipline is insufficient
+
+With N boolean flags, you have 2^N possible system states. Ten flags create 1,024 states. Most teams test exactly two: all flags on, all flags off. The remaining 1,022 states are untested and potentially broken.
+
+**Practical mitigation exists** (pairwise testing, modular isolation, testing only the "current production state" plus the "next flag flip"), but it requires engineering discipline. Teams that do not test flag combinations will eventually ship a broken state to production.
+
+**Martin Fowler's guidance:** "It can be very challenging to do traditional automated integration testing with feature flags. You do not need to test every combination — but you must test the disaster scenario (all flags off) and the current production state."
+
+### When flags substitute for proper architecture
+
+A feature flag should wrap a cohesive feature, not be woven through dozens of files. If enabling a feature requires 47 `if` checks scattered across the codebase, the problem is not the flag — it is the architecture. The feature is not properly modularized.
+
+**Anti-pattern:** Using flags to manage divergent implementations of the same feature across multiple services. If your "new payments flow" requires coordinated flag states across the payments service, the notifications service, the analytics service, and the frontend, you have a distributed feature flag — which is a distributed monolith in disguise.
+
+### When you are using flags as an excuse for sloppy code
+
+Wrapping performance-critical code in feature toggles as kill switches creates a false sense of security. "We can always turn it off" becomes an excuse for shipping code that is too slow, too resource-hungry, or too fragile. The kill switch does not fix the underlying problem — it hides it until someone forgets to test the "on" state.
+
+### The Knight Capital disaster — $460 million in 45 minutes
+
+On August 1, 2012, Knight Capital Group — which controlled 17% of NYSE trading volume — lost $460 million in 45 minutes because of a stale feature flag.
+
+**What happened:** Developers replaced legacy trading code with new logic and repurposed an old feature flag to control activation. The deployment succeeded on 7 of 8 servers, but silently failed on the 8th. When the flag was enabled, 7 servers ran the new code correctly; the 8th ran the legacy code, which executed an obsolete trading strategy that hemorrhaged money.
+
+**The cascade:** When operators noticed the problem, they attempted a rollback — but they did not know the flag was the root cause. They did not turn the flag off. Instead, they redeployed the old code to all 8 servers, which meant all servers now ran the legacy trading strategy. Losses compounded from millions to hundreds of millions.
+
+**The lessons:**
+1. Stale flags are ticking time bombs — the repurposed flag should have been removed years earlier.
+2. Flag states must be observable — operators could not see which flag states were active on which servers.
+3. Rollback procedures must account for flag states — "redeploy old code" is not sufficient when flags control behavior independently of the deployment.
+
+Knight Capital filed for bankruptcy protection within days.
+
+### When flag checking becomes a performance bottleneck
+
+Every flag check has a cost. In hot code paths (inner loops, per-request middleware, real-time data processing), flag lookups can add measurable latency. Server-side SDKs with local caching mitigate this (lookup is a local hash table read, not a network call), but client-side SDKs that phone home for every check can degrade performance significantly.
+
+**Rule of thumb:** If your code path executes more than 10,000 times per second, ensure flag lookup is a local in-memory read with sub-microsecond latency. Never make a network call per lookup in a hot path.
+
+---
+
+## How It Works
+
+### Flag Checking Architecture
+
+There are two fundamentally different models, and choosing the wrong one creates security and performance problems.
+
+**Server-side model (recommended for most use cases):**
+
+```
++--------------+     +------------------+     +--------------+
+|  Flag Mgmt   |---->|  Server-Side SDK |---->| Application  |
+|  Service     |     |  (local cache)   |     | Code         |
+|              |     |                  |     |              |
+| - Rules      |     | - Fetches config |     | - Calls      |
+| - Segments   |     |   on startup +   |     |   sdk.check()|
+| - Targeting  |     |   streams updates|     |   with user  |
+|              |     | - Checks         |     |   context    |
+|              |     |   locally        |     |              |
++--------------+     +------------------+     +--------------+
+```
+
+The server-side SDK downloads the full flag configuration (rules, segments, percentages) on startup and keeps it in memory via a streaming connection (SSE or WebSocket). Flag lookups are a local operation — no network call per request. The application passes user context (user ID, email, plan, country) to the SDK, which applies targeting rules locally and returns the flag value.
+
+**Advantages:** Fast lookups (microseconds), no client-side data exposure, full targeting rule complexity, works for any backend language.
+
+**Client-side model (for frontend/mobile):**
+
+```
++--------------+     +------------------+     +--------------+
+|  Flag Mgmt   |---->|  Checking        |---->|  Client SDK  |
+|  Service     |     |  Service (edge)  |     |  (browser/   |
+|              |     |                  |     |   mobile)    |
+| - Rules      |     | - Receives user  |     |              |
+| - Segments   |     |   context        |     | - Sends user |
+|              |     | - Applies rules  |     |   context    |
+|              |     |   server-side    |     | - Caches     |
+|              |     | - Returns only   |     |   results    |
+|              |     |   flag values    |     |              |
++--------------+     +------------------+     +--------------+
+```
+
+The client SDK sends user context to a service (often at the CDN edge), which applies rules server-side and returns only the computed flag values — never the rules themselves. This prevents exposing targeting logic, segment definitions, and API keys to the client.
+
+**Why not run rules on the client?** Client-side rule processing would require shipping all flag rules, segment definitions, and targeting logic to the browser or mobile app. This exposes sensitive business logic (who is in which experiment, pricing tier rules, internal user segments) and creates a security risk. Additionally, client-side contexts are relatively static (same user for the session), making server-side processing with caching highly efficient.
+
+### Targeting Rules
+
+Modern flag systems support layered targeting rules applied in priority order:
+
+```
+Flag: "new_dashboard"
++-- Rule 1: If user.email ends with "@company.com" -> ON     (internal dogfooding)
++-- Rule 2: If user.segment = "beta_testers" -> ON           (beta program)
++-- Rule 3: If user.plan = "enterprise" -> ON                (plan gating)
++-- Rule 4: If user.country in ["US", "CA"] -> percentage(25%) (geo rollout)
++-- Default: OFF
+```
+
+Rules are applied top-to-bottom. The first matching rule wins. This allows sophisticated rollout strategies: internal employees see the feature immediately, beta testers get it next, enterprise customers get priority access, and a percentage of US/Canadian users see it for geographic validation — all controlled by a single flag.
+
+**Percentage-based targeting** uses deterministic hashing (typically MurmurHash3 of `flagKey + userId`) to assign users to buckets. This ensures:
+- The same user always gets the same flag value (consistency)
+- Different flags assign users to different buckets (independence)
+- Increasing the percentage from 10% to 25% adds users but never removes existing ones (monotonic rollout)
+
+### Flag Lifecycle
+
+Every flag should follow a documented lifecycle with clear state transitions:
+
+```
++----------+     +----------+     +--------------+     +----------+     +---------+
+| CREATED  |---->| TESTING  |---->| ROLLING OUT  |---->|    GA     |---->| REMOVED |
+|          |     |          |     |              |     |          |     |         |
+| Flag     |     | Internal |     | 1% -> 10% ->|     | 100% ON  |     | Code +  |
+| defined, |     | users    |     | 50% -> 100% |     | for all  |     | flag    |
+| code     |     | only     |     |              |     | users    |     | deleted |
+| merged   |     |          |     | Monitoring   |     |          |     |         |
+| (OFF)    |     |          |     | at each step |     | REMOVE   |     |         |
+|          |     |          |     |              |     | THE FLAG |     |         |
++----------+     +----------+     +--------------+     +----------+     +---------+
+                                                              |
+                                                              v
+                                                       Expiration timer
+                                                       starts (14 days)
+```
+
+**The critical transition is GA to REMOVED.** Once a flag reaches 100% and the feature is confirmed stable, a timer starts. If the flag is not removed within the grace period (typically 7-30 days), it should trigger an alert, a Jira ticket, or a linting error. Automated flag cleanup tools (like Piranha by Uber) can even submit pull requests to remove stale flags.
+
+### Progressive Rollout Strategies
+
+**Canary rollout (percentage-based):**
+
+```
+Day 1:   1% of users  -> monitor error rates, latency p99, business metrics
+Day 3:   5% of users  -> watch for edge cases at slightly larger scale
+Day 5:   25% of users -> statistical significance for A/B comparison
+Day 8:   50% of users -> performance under meaningful load
+Day 10:  100%         -> GA, start flag removal timer
+```
+
+**Ring-based rollout (audience-based):**
+
+```
+Ring 0: Internal employees (dogfooding)          -> 1-2 days
+Ring 1: Beta program users (opted-in)            -> 3-5 days
+Ring 2: Low-risk geography (e.g., New Zealand)   -> 3-5 days
+Ring 3: Full production                          -> GA
+```
+
+Ring-based and percentage-based rollouts are complementary. A common pattern is to use rings for early validation (employees, beta users) and then switch to percentage-based rollout for the broader audience.
+
+**Monitoring triggers for rollback:**
+
+At each rollout stage, automated monitoring should watch for:
+- Error rate increase > 1% over baseline
+- Latency p99 increase > 20% over baseline
+- Business metric degradation (conversion rate, revenue per session)
+- Client-side crash rate increase
+- Support ticket volume spike
+
+If any metric crosses its threshold, the flag should automatically roll back to the previous percentage (or to OFF). This is the "automated rollback" capability that makes progressive delivery safe.
+
+---
+
+## Trade-Offs Matrix
+
+| Dimension | Without Feature Flags | With Feature Flags | Notes |
+|---|---|---|---|
+| **Deployment risk** | All-or-nothing releases; rollback requires redeployment | Instant feature disable via flag toggle; no redeployment needed | The primary value proposition — decoupling deploy from release |
+| **Rollout speed** | Gated by release cycles (weekly, bi-weekly) | Continuous — features activate independently of deploys | Enables true continuous delivery |
+| **Code complexity** | Linear code paths | Branched code paths per flag; N flags = 2^N possible states | The primary cost — combinatorial complexity grows exponentially |
+| **Testing burden** | Test one code path | Must test both flag states; combinations multiply effort | Pairwise testing and modular isolation are mitigations, not solutions |
+| **Technical debt** | Standard code debt | Flag-specific debt: stale flags, dead code, orphaned branches | Requires active lifecycle management that most teams neglect |
+| **Operational visibility** | Behavior determined by deployed code version | Behavior determined by code version AND flag states | Must track flag states alongside deployments for incident response |
+| **Time to rollback** | Minutes (redeploy previous version) | Seconds (toggle flag off) | Dramatic improvement for critical incidents |
+| **Experimentation** | Requires separate A/B testing infrastructure | Built-in with multivariate flags | Significant product development advantage |
+| **Infrastructure cost** | None additional | Flag service, SDK overhead, streaming connections | Ranges from free (env vars) to $100K+/year (LaunchDarkly enterprise) |
+| **Cognitive load** | Developers reason about one code path | Developers must reason about all active flag states | Increases linearly with active flag count |
+| **Audit and compliance** | Change tracked via deploy logs | Flag changes tracked via audit trail, separate from deploys | Can be an advantage (fine-grained tracking) or a burden (two audit streams) |
+
+---
+
+## Evolution Path
+
+Feature flag systems typically progress through four stages, each adding capability and complexity:
+
+### Stage 1: Environment Variables and Config Files
+
+```python
+# .env
+ENABLE_NEW_SEARCH=false
+
+# application code
+if os.environ.get("ENABLE_NEW_SEARCH") == "true":
+    return new_search(query)
+```
+
+**Characteristics:** No external service, no targeting rules, no gradual rollout. A flag is either on or off for the entire environment. Change requires redeployment or container restart.
+
+**Appropriate for:** Small teams (fewer than 10 engineers), low deployment frequency, features that are either 100% on or 100% off. Startups before product-market fit should not invest in anything more sophisticated.
+
+### Stage 2: Database-Backed Flags with Admin UI
+
+```python
+# Flag stored in PostgreSQL, editable via admin panel
+flag = FlagModel.objects.get(key="new_search")
+if flag.is_enabled:
+    return new_search(query)
+```
+
+**Characteristics:** Flags can be toggled without redeployment. Simple admin UI for non-engineers to toggle flags. No user targeting or percentage rollouts. Flag changes propagate on next request (no streaming).
+
+**Appropriate for:** Teams of 10-30, products with moderate deployment frequency, need for non-engineer flag control (PM wants to launch a feature on a specific date).
+
+### Stage 3: Purpose-Built Flag Service with Targeting
+
+```python
+# SDK-based flag check with user targeting
+client = FlagClient(sdk_key="...")
+if client.variation("new_search", user={"key": user.id, "plan": "enterprise"}):
+    return new_search(query)
+```
+
+**Characteristics:** Full targeting rules (user, segment, percentage), audit trails, real-time propagation via streaming, server-side and client-side SDKs. This is where most organizations should aim.
+
+**Appropriate for:** Teams of 30+, products with continuous deployment, need for gradual rollouts and A/B testing, multi-tenant SaaS with plan-based gating.
+
+**Technology options:** LaunchDarkly (hosted), Unleash (self-hosted/hosted), Flagsmith (self-hosted/hosted), GrowthBook (open source), PostHog (bundled with analytics), Statsig (bundled with experimentation).
+
+### Stage 4: Experimentation Platform with Statistical Rigor
+
+```python
+# Full experimentation with variants, metrics, and statistical analysis
+variant = experiment_client.get_assignment(
+    experiment="checkout_redesign",
+    user=user,
+    metrics=["conversion_rate", "revenue_per_session", "cart_abandonment"]
+)
+if variant == "control":
+    return render_old_checkout()
+elif variant == "variant_a":
+    return render_new_checkout_a()
+elif variant == "variant_b":
+    return render_new_checkout_b()
+```
+
+**Characteristics:** Multivariate assignment, statistical significance tracking, mutual exclusion between experiments, metric attribution, automatic experiment termination when significance is reached. This is a dedicated experimentation platform that happens to use feature flags as its delivery mechanism.
+
+**Appropriate for:** Large product organizations (100+ engineers), data-driven product teams, high-traffic products where 1% improvement in conversion is worth millions.
+
+**Technology options:** Statsig, Eppo, Optimizely, GrowthBook, LaunchDarkly (Experimentation add-on).
+
+---
+
+## Failure Modes
+
+### Flag debt — the silent killer
+
+**Symptom:** 300+ flags in the codebase, fewer than 50 actively toggled. Engineers are afraid to remove flags because they do not know if the code path is still needed. New hires cannot understand the codebase because every function has 3-5 flag-gated branches.
+
+**Root cause:** No flag lifecycle process. Flags are created eagerly and removed never.
+
+**Impact:** Increased cognitive load, slower onboarding, increased testing surface area, higher risk of Knight Capital-style incidents from stale flag interactions.
+
+**Prevention:**
+- Every flag has an owner and an expiration date at creation time
+- CI/CD pipeline fails if a flag exceeds its expiration date without an explicit extension
+- Automated cleanup tools (Uber's Piranha, DevCycle's code references) scan for flags that have been 100% on for more than 14 days and create removal PRs
+- Flag count per service is a tracked metric — alert if it exceeds a threshold (e.g., 20 active flags per service)
+
+### Flag interaction — unexpected combinations
+
+**Symptom:** Feature A works correctly. Feature B works correctly. Features A and B enabled simultaneously produce a broken user experience or data corruption.
+
+**Root cause:** Flags are treated as independent, but the code paths they control share state, depend on the same data, or produce conflicting UI elements.
+
+**Example:** Flag A enables a new checkout flow. Flag B enables a new payment processor. Each was tested independently. When both are on, the new checkout flow sends data in a format the new payment processor does not expect, and orders silently fail to process.
+
+**Prevention:**
+- Document flag dependencies explicitly (this flag requires/conflicts with these other flags)
+- Test the "current production state + one flag flip" matrix rather than all combinations
+- Isolate flagged code paths using modular architecture — if two features cannot interact, their flags cannot interact
+- For critical flags, use mutually exclusive groups in the flag service
+
+### Inconsistent flag state across services
+
+**Symptom:** The frontend shows the new feature (flag ON), but the backend API returns data in the old format (flag OFF). Users see a broken experience.
+
+**Root cause:** In microservices architectures, each service checks flags independently. If flag propagation is not instant, or if services cache flag values with different TTLs, a flag change creates a window of inconsistency.
+
+**Prevention:**
+- Use a single flag service with streaming updates to all SDKs (not polling)
+- For flags that span services, use a "flag context" passed in request headers so all services in a request chain see the same flag state
+- Design flagged features to be tolerant of mixed states (new frontend gracefully handles old API response format)
+
+### Flag lookup performance degradation
+
+**Symptom:** p99 latency increases 20ms. Investigation reveals that flag checks in a hot code path are making network calls or processing complex targeting rules per request.
+
+**Root cause:** Client-side SDK configured to phone home for every check, or server-side SDK running complex segment rules (involving database lookups) per request.
+
+**Prevention:**
+- Server-side SDKs must cache flag configuration locally and run without network calls
+- Pre-compute segment membership asynchronously, not during flag checks
+- For hot paths (more than 10K checks per second), use a local boolean cache with a short TTL rather than full rule processing per call
+- Monitor flag lookup latency as a system metric
+
+### Configuration drift between environments
+
+**Symptom:** A feature works in staging but fails in production. Investigation reveals that flag targeting rules differ between environments, or a flag exists in staging but not in production.
+
+**Root cause:** Flag configuration is not managed as code. Teams modify flags via the admin UI in each environment independently.
+
+**Prevention:**
+- Treat flag configuration as code — store targeting rules in version-controlled YAML/JSON, apply via CI/CD
+- Use environment-aware flag definitions: a single flag definition with per-environment overrides
+- Audit differences between environment flag states regularly
+
+### The "permanent beta" trap
+
+**Symptom:** A feature has been at 50% rollout for six months. No one has decided whether to go to 100% or roll back. The flag has become a permanent partition in the user experience.
+
+**Root cause:** No decision criteria were defined before the rollout began. No one owns the go/no-go decision. The flag drifts into an indefinite state.
+
+**Prevention:**
+- Before any rollout, define success criteria (metrics, thresholds, timeline)
+- Assign a decision owner who will call GA or rollback by a specific date
+- Flag service should alert on flags that have been in a partial rollout state for more than 30 days
+
+---
+
+## Technology Landscape
+
+### Managed Services (SaaS)
+
+| Tool | Strengths | Weaknesses | Pricing Model | Best For |
+|---|---|---|---|---|
+| **LaunchDarkly** | Most mature, best targeting, excellent SDKs (25+ languages), strong enterprise features, real-time streaming | Most expensive at scale, no self-hosted option, no source code visibility | Per-seat + MAU tiers; enterprise $$$$ | Large enterprises with budget, complex targeting needs |
+| **Statsig** | Integrated experimentation + flags, statistical engine, warehouse-native analytics | Younger platform, less mature enterprise features | Free tier generous; usage-based | Data-driven product teams wanting experimentation built-in |
+| **PostHog** | Bundled with product analytics, session recording, surveys; open-source core | Flag system is secondary to analytics; less sophisticated targeting | Free self-hosted; cloud usage-based | Teams wanting an all-in-one product analytics suite |
+| **Optimizely** | Strong experimentation heritage, visual editor for non-engineers | Expensive, enterprise sales process, slower innovation | Enterprise pricing | Marketing-driven experimentation teams |
+
+### Self-Hosted / Open-Source
+
+| Tool | Strengths | Weaknesses | License | Best For |
+|---|---|---|---|---|
+| **Unleash** | Self-hosted or cloud, strong RBAC, good API, 11 well-documented principles for flag systems | Enterprise features (SAML, audit log, change requests) behind paid tier | Apache 2.0 (core) | Teams needing self-hosted with strong access control |
+| **Flagsmith** | Flexible deployment (cloud/self-hosted/on-prem), multi-platform SDKs, edge proxy | Smaller community than Unleash, fewer integrations | BSD 3-Clause | Organizations with strict data governance requirements |
+| **GrowthBook** | Open-source with built-in experimentation, Bayesian statistics engine, warehouse-native | Smaller team/community, less polished UI | MIT | Data teams wanting open-source experimentation |
+| **OpenFeature** | CNCF incubating project, vendor-agnostic API specification, prevents vendor lock-in | Not a flag service itself — a standard that wraps other providers | Apache 2.0 | Teams wanting to avoid vendor lock-in at the code level |
+
+### Build vs. Buy Decision
+
+**Build your own when:**
+- You need fewer than 20 flags with no targeting rules (database-backed flags with admin UI, approximately 2 weeks to build)
+- You have extreme compliance requirements that prohibit any external service
+- You are using OpenFeature as the API layer and can swap providers later
+
+**Buy when:**
+- You need targeting rules, percentage rollouts, or experimentation
+- You have more than 50 active flags
+- You need audit trails for compliance (SOC2, HIPAA)
+- The cost of the service ($5K-50K/year) is less than the engineering time to build and maintain a custom solution (almost always true)
+
+**Use OpenFeature when:**
+- You want to avoid vendor lock-in at the code level
+- You might switch providers in the future
+- You want a consistent API across server-side and client-side SDKs
+- You are assessing multiple providers and want to defer the decision
+
+OpenFeature is a CNCF incubating project supported by LaunchDarkly, Split, Flagsmith, and end users including eBay, Google, SAP, and Spotify. It provides a vendor-agnostic API: you code against the OpenFeature SDK, and swap the backend provider without changing application code.
+
+---
+
+## Decision Tree
+
+```
+START: Do you need to control feature visibility independent of deployment?
+|
++-- NO -> Do you need kill switches for operational safety?
+|         +-- NO -> You don't need feature flags. Ship features via normal deploys.
+|         +-- YES -> Use simple env var / config flags (Stage 1). No targeting needed.
+|
++-- YES -> How many engineers on the team?
+           |
+           +-- < 10 -> Use database-backed flags with admin UI (Stage 2).
+           |           Keep flag count under 15. Track manually.
+           |
+           +-- 10+ -> Do you need user targeting or percentage rollouts?
+                      |
+                      +-- NO -> Database-backed flags (Stage 2) may still suffice.
+                      |         Consider upgrading if flag count exceeds 30.
+                      |
+                      +-- YES -> Do you need A/B testing with statistical rigor?
+                                |
+                                +-- NO -> Purpose-built flag service (Stage 3).
+                                |         +-- Budget for SaaS? -> LaunchDarkly, Statsig
+                                |         +-- Need self-hosted? -> Unleash, Flagsmith
+                                |         +-- Want vendor-agnostic? -> Use OpenFeature API
+                                |
+                                +-- YES -> Experimentation platform (Stage 4).
+                                           +-- Want integrated analytics? -> Statsig, PostHog
+                                           +-- Want open-source? -> GrowthBook
+                                           +-- Want enterprise? -> Optimizely, LaunchDarkly
+```
+
+**The most common mistake** is jumping to Stage 3 or 4 when Stage 1 or 2 is sufficient. A startup with 5 engineers does not need LaunchDarkly. An `ENABLE_NEW_SEARCH=true` environment variable and a 10-line middleware function will serve them for years.
+
+**The second most common mistake** is staying at Stage 1 too long. When your team grows past 30 engineers and you have 40 environment variables controlling feature visibility with no targeting, no audit trail, and no lifecycle management, you are accumulating dangerous debt.
+
+---
+
+## Implementation Sketch
+
+### Minimal Flag Service (Stage 2 — Database-Backed)
+
+```python
+# models.py -- Flag storage
+class FeatureFlag(Model):
+    key = CharField(max_length=100, unique=True, db_index=True)
+    enabled = BooleanField(default=False)
+    description = TextField(blank=True)
+    owner = CharField(max_length=100)          # who owns this flag
+    created_at = DateTimeField(auto_now_add=True)
+    expires_at = DateTimeField()               # MANDATORY expiration
+    category = CharField(choices=["release", "experiment", "ops", "permission"])
+
+    def is_expired(self):
+        return timezone.now() > self.expires_at and self.category == "release"
+
+
+# service.py -- Flag checking with in-memory cache
+class FlagService:
+    def __init__(self):
+        self._cache = {}
+        self._cache_ttl = 30  # seconds
+        self._last_refresh = 0
+
+    def is_enabled(self, key: str) -> bool:
+        self._refresh_if_stale()
+        flag = self._cache.get(key)
+        if flag is None:
+            return False  # unknown flags default to OFF
+        return flag["enabled"]
+
+    def _refresh_if_stale(self):
+        now = time.time()
+        if now - self._last_refresh > self._cache_ttl:
+            flags = FeatureFlag.objects.values("key", "enabled")
+            self._cache = {f["key"]: f for f in flags}
+            self._last_refresh = now
+
+
+# middleware.py -- Inject flags into request context
+class FeatureFlagMiddleware:
+    def __init__(self, get_response):
+        self.get_response = get_response
+        self.flag_service = FlagService()
+
+    def __call__(self, request):
+        request.flags = self.flag_service
+        return self.get_response(request)
+
+
+# views.py -- Usage
+def dashboard_view(request):
+    if request.flags.is_enabled("new_dashboard"):
+        return render(request, "dashboard_v2.html")
+    return render(request, "dashboard.html")
+```
+
+### Flag Service with Targeting Rules (Stage 3)
+
+```python
+# targeting.py -- Rule application engine
+class TargetingEngine:
+    def check(self, flag_config: dict, user_context: dict) -> bool:
+        # Apply rules in priority order
+        for rule in flag_config.get("rules", []):
+            if self._matches_rule(rule, user_context):
+                return self._resolve_variation(rule, user_context)
+
+        # Fall through to default
+        return flag_config.get("default", False)
+
+    def _matches_rule(self, rule: dict, user: dict) -> bool:
+        for condition in rule["conditions"]:
+            attr = user.get(condition["attribute"])
+            op = condition["operator"]
+
+            if op == "in" and attr not in condition["values"]:
+                return False
+            elif op == "equals" and attr != condition["value"]:
+                return False
+            elif op == "ends_with" and not str(attr).endswith(condition["value"]):
+                return False
+            elif op == "gte" and attr < condition["value"]:
+                return False
+        return True
+
+    def _resolve_variation(self, rule: dict, user: dict) -> bool:
+        if "percentage" in rule:
+            # Deterministic hashing for consistent assignment
+            hash_key = f"{rule['flag_key']}:{user['key']}"
+            bucket = mmh3.hash(hash_key, signed=False) % 100
+            return bucket < rule["percentage"]
+        return rule.get("value", True)
+
+
+# Flag configuration (stored as JSON in DB or config file)
+flag_config = {
+    "key": "new_checkout",
+    "rules": [
+        {
+            "flag_key": "new_checkout",
+            "conditions": [
+                {"attribute": "email", "operator": "ends_with", "value": "@company.com"}
+            ],
+            "value": True  # All internal users see it
+        },
+        {
+            "flag_key": "new_checkout",
+            "conditions": [
+                {"attribute": "plan", "operator": "in", "values": ["enterprise"]}
+            ],
+            "value": True  # All enterprise users see it
+        },
+        {
+            "flag_key": "new_checkout",
+            "conditions": [
+                {"attribute": "country", "operator": "in", "values": ["US", "CA"]}
+            ],
+            "percentage": 25  # 25% of US/CA users
+        }
+    ],
+    "default": False
+}
+```
+
+### Flag Cleanup Automation
+
+```python
+# cleanup.py -- Automated stale flag detection (run as scheduled job)
+class FlagCleanupJob:
+    """
+    Runs daily. Identifies flags that should be removed and creates
+    tickets/PRs for cleanup.
+    """
+    GRACE_PERIOD_DAYS = 14
+
+    def run(self):
+        stale_flags = self._find_stale_flags()
+        for flag in stale_flags:
+            days_overdue = (timezone.now() - flag.expires_at).days
+            if days_overdue > self.GRACE_PERIOD_DAYS * 2:
+                self._create_critical_alert(flag)   # Slack alert to flag owner
+            elif days_overdue > self.GRACE_PERIOD_DAYS:
+                self._create_cleanup_ticket(flag)   # Jira ticket
+            else:
+                self._send_reminder(flag)           # Email to flag owner
+
+    def _find_stale_flags(self):
+        return FeatureFlag.objects.filter(
+            category="release",
+            enabled=True,
+            expires_at__lt=timezone.now()
+        )
+
+
+# ci_check.py -- Fail CI if code references expired flags
+def check_flag_references(codebase_path: str, flag_service: FlagService):
+    """
+    Scan source code for flag references. If any reference an expired
+    flag, fail the build with a clear error message.
+    """
+    expired_flags = flag_service.get_expired_flags()
+    violations = []
+
+    for filepath in glob(f"{codebase_path}/**/*.py", recursive=True):
+        content = open(filepath).read()
+        for flag in expired_flags:
+            if flag.key in content:
+                violations.append(
+                    f"{filepath}: references expired flag '{flag.key}' "
+                    f"(expired {flag.expires_at.date()}, owner: {flag.owner})"
+                )
+
+    if violations:
+        print("EXPIRED FLAG REFERENCES FOUND:")
+        for v in violations:
+            print(f"  - {v}")
+        sys.exit(1)
+```
+
+### OpenFeature Integration (Vendor-Agnostic)
+
+```python
+# Using OpenFeature API -- swap providers without changing application code
+
+from openfeature import api
+from openfeature.contrib.provider.flagd import FlagdProvider
+# Alternatively: from openfeature_launchdarkly import LaunchDarklyProvider
+
+# Configure provider once at startup
+api.set_provider(FlagdProvider())  # or LaunchDarklyProvider(sdk_key="...")
+
+# Application code uses the OpenFeature API -- never the provider directly
+client = api.get_client()
+
+# Boolean flag check
+new_checkout = client.get_boolean_value(
+    flag_key="new_checkout",
+    default_value=False,
+    evaluation_context=EvaluationContext(
+        targeting_key=user.id,
+        attributes={"plan": user.plan, "country": user.country}
+    )
+)
+
+# String flag check (multivariate)
+checkout_variant = client.get_string_value(
+    flag_key="checkout_variant",
+    default_value="control",
+    evaluation_context=EvaluationContext(targeting_key=user.id)
+)
+```
+
+---
+
+## Cross-References
+
+- **[Multi-Tenancy / Microservices](../patterns/microservices.md)** — Permission flags are the feature-gating layer in multi-tenant SaaS. Plan-based feature access (free/pro/enterprise) is often implemented via feature flags tied to tenant attributes. Consider whether permission flags belong in the flag system or in a dedicated authorization layer. In microservices architectures, feature flags require careful attention to flag state consistency across services. A flag change must propagate to all services atomically, or features that span services will experience split-brain states. Pass flag context in request headers to ensure consistent results across service boundaries.
+
+- **[Plugin Architecture](../patterns/plugin-architecture.md)** — Plugin systems and feature flags solve overlapping problems (enabling/disabling functionality at runtime) but at different granularities. Plugins are coarse-grained (entire modules), feature flags are fine-grained (individual code paths). In mature systems, plugin activation may itself be controlled by feature flags.
+
+- **[Monolith](../patterns/monolith.md)** — Monolithic architectures benefit enormously from feature flags because flag checks are in-process (no network consistency issues), deployment is atomic (no split-brain across services), and the blast radius of a flag misconfiguration is limited to a single process. Feature flags in monoliths are simpler, faster, and safer than in distributed systems.
+
+- **[Event-Driven Architecture](../patterns/event-driven.md)** — Feature flags in event-driven systems require special care. If a flag changes while events are in-flight, consumers may process events under different flag states than the producer intended. Consider embedding flag state in event metadata for consistent processing.
+
+- **[CQRS / Event Sourcing](../patterns/cqrs-event-sourcing.md)** — Flag state changes can be modeled as events in an event-sourced system, providing a complete audit trail of flag lifecycle. This is particularly valuable for compliance-heavy domains where regulators require proof of when features were enabled for which users.