@lenardangeloolajay/len-toolkit 1.1.0

package/templates/skills/designing-data-intensive-applications/designing-data-intensive-applications.md

@@ -0,0 +1,393 @@
+# OBEY Designing Data-Intensive Applications by Martin Kleppmann
+
+## Purpose
+
+This repository follows **Designing Data-Intensive Applications** in the sense of Martin Kleppmann:
+design systems around explicit trade-offs in reliability, scalability, maintainability, consistency, and data flow.
+
+All code generation, edits, and reviews must optimize for:
+- explicit data and consistency semantics
+- idempotent and replay-safe processing
+- clear ownership of truth
+- durable boundaries between storage, messaging, and computation
+- schema evolution awareness
+- realistic distributed systems assumptions
+
+This file is a binding engineering policy: `MUST` is binding, `SHOULD` is a strong default, and `MUST NOT` is forbidden.
+
+---
+
+## Primary Directive
+
+Data systems are defined by trade-offs.
+When uncertain, make those trade-offs explicit instead of hiding them behind vague abstractions.
+
+Always ask:
+1. what is the source of truth?
+2. what are the consistency expectations?
+3. what happens on retries, duplicates, reordering, and partial failure?
+4. how does the data evolve over time?
+5. where is state durable, cached, derived, or ephemeral?
+
+Do not design distributed behavior as if everything were local, ordered, and exactly once.
+
+---
+
+## Reliability Rules
+
+1. Treat crashes, partial writes, duplicate work, timeouts, and stale reads as normal design inputs.
+2. Make write acknowledgment semantics explicit.
+3. Avoid hidden assumptions about durable success.
+4. Design for restart, replay, and partial failure recovery.
+
+Anti-patterns (MUST NOT):
+- side effects that cannot be retried safely
+- no distinction between accepted, persisted, and applied
+- assuming one successful response means all downstream effects succeeded
+
+---
+
+## Scalability and Maintainability Rules
+
+1. Describe load with concrete parameters before changing architecture.
+2. Describe performance with latency, throughput, percentiles, and tail behavior where they matter.
+3. Do not claim scalability from node count alone; identify the bottleneck, access pattern, and contention point.
+4. Keep operability, simplicity, and evolvability as first-class design goals.
+5. Prefer designs that make production behavior inspectable and changeable over opaque clever mechanisms.
+6. Avoid accidental complexity from unnecessary distribution, premature heterogeneity, or hidden coupling.
+
+---
+
+## Data Model and Storage Rules
+
+1. Choose storage shape based on access patterns, consistency needs, and update behavior.
+2. Do not force one storage pattern onto all workloads.
+3. Keep the ownership of each dataset explicit.
+4. Distinguish primary data from indexes, caches, projections, and search copies.
+
+### Source of Truth
+For every important piece of data, identify:
+- primary owner
+- derived copies
+- replication path
+- update path
+- consistency expectation
+
+Anti-patterns (MUST NOT):
+- many writable copies with no ownership
+- cache quietly becoming the real source of truth
+- denormalized copies with no repair strategy
+
+---
+
+## Query Model and Data Shape Rules
+
+1. Choose relational, document, graph, key-value, or analytical models according to relationships, query needs, update locality, and evolution pressure.
+2. Do not use a document model when many-to-one or many-to-many relationships require awkward duplication or application-side joins.
+3. Do not force a relational shape when data is naturally self-contained and usually accessed together.
+4. Use declarative query languages where they make intent clearer and leave optimization to the engine.
+5. Use graph models when relationships are first-class and traversal is central.
+6. Treat Cypher, SPARQL, Datalog, SQL, MapReduce, and application code as different expression choices with different maintainability and optimization tradeoffs.
+
+---
+
+## Storage Engine and Indexing Rules
+
+1. Match indexing strategy to write pattern, read pattern, range scans, update cost, and recovery needs.
+2. Use log-structured storage, SSTables, and LSM-tree style approaches when write throughput and sequential writes are the dominant fit.
+3. Use B-tree style indexes when ordered access, point lookups, and mature transactional behavior fit the workload.
+4. Treat secondary indexes as separate data structures with write amplification, partitioning, and consistency costs.
+5. Distinguish OLTP access from analytical workloads; do not force one layout to serve both well.
+6. Use column-oriented storage, compression, sort order, materialized views, or cubes only when analytical access patterns justify them.
+7. Keep in-memory assumptions explicit; memory residency is a performance strategy, not a durability model.
+
+---
+
+## Consistency Rules
+
+1. Be explicit about read-after-write expectations.
+2. Be explicit about staleness tolerance.
+3. Be explicit about conflict handling.
+4. Use strong consistency only where the product truly requires it.
+5. Use eventual consistency intentionally, not accidentally.
+
+### Write Semantics
+Document or encode:
+- when a write is durable
+- when it is visible
+- whether readers may see stale data
+- how conflicts are detected or resolved
+
+Anti-patterns (MUST NOT):
+- “eventual consistency” used as a slogan instead of a contract
+- stale-read bugs blamed on infrastructure with no product decision behind them
+- no conflict model for concurrent updates
+
+---
+
+## Idempotency and Replay Rules
+
+1. Handlers of commands, jobs, and events must tolerate retries where delivery or acknowledgment is uncertain.
+2. Prefer deduplication keys or naturally idempotent state transitions.
+3. Design processing to survive replay after crashes.
+4. Never assume exactly-once delivery unless the system boundary truly provides it and the design proves it.
+
+Anti-patterns (MUST NOT):
+- duplicate billing/order/send on retry
+- handlers with non-repeatable side effects and no guard
+- event processors depending on “it probably won't happen twice”
+
+---
+
+## Ordering Rules
+
+1. Do not assume global order in distributed systems.
+2. Require only the minimum ordering guarantees the business logic actually needs.
+3. When ordering matters, define its scope:
+   - per key
+   - per stream
+   - per partition
+   - per record or entity whose history is being updated
+4. Keep ordering-sensitive logic close to the key or stream that defines the order.
+
+Anti-patterns (MUST NOT):
+- implicit reliance on total ordering
+- out-of-order events corrupting state because no versioning or sequence policy exists
+- parallel consumers updating the same key with no ordering plan
+
+---
+
+## Event, Log, and Stream Rules
+
+1. Distinguish commands, events, and materialized views clearly.
+2. Events describe facts that happened; commands request action.
+3. Logs and streams are durable histories, not merely transport pipes.
+4. Consumers must tolerate lag, duplicates, restart, and replay.
+5. Derived projections must be rebuildable where feasible.
+
+### Event Design
+- use stable identifiers
+- include enough metadata for correlation and replay
+- version payloads carefully
+- keep semantics explicit
+
+Anti-patterns (MUST NOT):
+- event payloads tied to one serializer or internal object layout
+- projections that cannot be rebuilt
+- assuming consumers keep up forever
+
+---
+
+## Schema Evolution Rules
+
+1. Schemas will change; plan for it.
+2. Version contracts intentionally.
+3. Prefer backward- and forward-compatible changes where possible.
+4. Keep old readers and writers in mind during rollout.
+5. Distinguish internal refactors from contract changes.
+
+Anti-patterns (MUST NOT):
+- breaking payloads or DB semantics without migration strategy
+- reusing fields with new meaning
+- silently changing enum or status semantics across services
+
+---
+
+## Encoding and Data Flow Rules
+
+1. Choose encoding formats by compatibility needs, schema guarantees, readability, size, and language independence.
+2. Do not rely on language-specific serialization for long-lived or cross-service data.
+3. Treat JSON, XML, binary encodings, Thrift, Protocol Buffers, and Avro as contract choices with different schema-evolution tradeoffs.
+4. Define reader and writer compatibility during rolling upgrades.
+5. Keep database writes, service calls, and asynchronous messages explicit about who reads old and new formats during migration.
+6. Avoid RPC designs that hide network failure, version skew, latency, or partial failure behind local-call syntax.
+
+---
+
+## Partitioning and Locality Rules
+
+1. Keep data and work colocated by the key that most often drives consistency or aggregation.
+2. Partition by a workload-relevant key, not by convenience alone.
+3. Be explicit about hot-key risk and skew.
+4. Design cross-partition operations carefully.
+
+Anti-patterns (MUST NOT):
+- partitioning that makes every common query cross-node
+- no plan for skew or hotspots
+- requiring cross-partition transactions for ordinary operations
+
+---
+
+## Replication Rules
+
+1. Choose leader-follower, multi-leader, or leaderless replication according to write topology, failure tolerance, latency, and conflict handling.
+2. Be explicit about synchronous and asynchronous replication tradeoffs.
+3. Define behavior during node outages, follower catch-up, failover, and reconfiguration.
+4. Preserve read-your-writes, monotonic reads, and consistent prefix reads only when the product or workflow requires them and the design provides them.
+5. Do not rely on quorum formulas without checking stale reads, sloppy quorums, hinted handoff, and concurrent writes.
+6. Make conflict detection and resolution explicit for concurrent writes.
+
+---
+
+## Transaction Rules
+
+1. Use local transactions where they solve a real consistency problem cleanly.
+2. Avoid distributed transactions as a default coordination strategy.
+3. When cross-boundary coordination is required, define the commit, recovery, reconciliation, and failure semantics explicitly.
+4. Make atomicity scope explicit.
+
+### Isolation and Invariants
+- Know whether read committed, snapshot isolation, serial execution, two-phase locking, or serializable snapshot isolation is required for the invariant.
+- Protect against lost updates, write skew, and phantoms where application correctness depends on them.
+- Do not accept weaker isolation for correctness-critical invariants without a deliberate design that preserves the invariant another way.
+
+Anti-patterns (MUST NOT):
+- multi-system two-phase coordination by default
+- side effects emitted outside transactional boundaries with no repair path
+- pretending asynchronous side effects are atomic because they “usually happen”
+
+---
+
+## Derived Data Rules
+
+1. Treat indexes, search copies, caches, and read models as derived data unless they are explicitly authoritative.
+2. Derived data must be repairable, rebuildable, or re-syncable.
+3. Know how lag affects user-visible behavior.
+4. Keep derivation pipelines observable.
+
+Anti-patterns (MUST NOT):
+- no way to rebuild projections
+- no lag visibility
+- mixing primary writes directly into derived stores with no ownership model
+
+---
+
+## Distributed Fault, Clock, and Consensus Rules
+
+1. Treat network delay, packet loss, partitions, duplicated messages, and arbitrary pauses as normal distributed-system risks.
+2. Do not infer remote failure or success from timeout alone.
+3. Use monotonic clocks for measuring elapsed time; do not use wall clocks for ordering unless clock assumptions are explicit and safe.
+4. Do not rely on synchronized clocks for correctness unless uncertainty bounds and failure behavior are part of the design.
+5. Treat majority decisions, leases, locks, and leadership as assumptions that need a fault model.
+6. Use linearizability only where a single up-to-date value is required and the availability/latency cost is acceptable.
+7. Use total order broadcast, atomic commit, or consensus only when the coordination problem truly requires it.
+8. Make membership and coordination-service dependencies explicit; they are part of the system design, not invisible plumbing.
+
+---
+
+## Batch and Stream Processing Rules
+
+1. Design batch jobs so inputs, outputs, and intermediate state can be recomputed or recovered.
+2. Keep external side effects out of replayable jobs unless idempotency is explicit.
+3. Use MapReduce-style, dataflow, or high-level batch APIs according to join strategy, intermediate materialization, and operational needs.
+4. Distinguish event time, processing time, and ingestion time in stream processing.
+5. Define windowing, late data, joins, state storage, checkpoints, and fault tolerance for streams that affect correctness.
+6. Treat change data capture, event sourcing, and log-based synchronization as ways to derive and propagate data, not as magic consistency.
+7. Define at-most-once, at-least-once, or exactly-once processing guarantees for each source-to-sink path.
+
+---
+
+## API and Service Boundary Rules
+
+1. Service boundaries must reflect data ownership and update semantics.
+2. Do not split one tightly consistent business concept across many services casually.
+3. Avoid chatty cross-service joins on hot paths.
+4. Contracts must encode identifiers, versions, and failure semantics clearly.
+
+---
+
+## Review Rules
+
+When reviewing code, actively look for:
+- hidden assumptions about ordering
+- hidden assumptions about exactly-once delivery
+- lack of idempotency
+- no source-of-truth ownership
+- broken schema evolution practices
+- no versioning or sequencing where concurrency matters
+- side effects that cannot be repaired
+- write paths that update several stores with unclear guarantees
+- projections that cannot be rebuilt
+- partitioning blind to locality or hotspots
+
+---
+
+## Forbidden Patterns
+
+### Exactly-Once Wishful Thinking
+- assuming a broker or queue magically prevents all duplicates
+- writing non-idempotent handlers without safeguards
+
+### Hidden Consistency Contract
+- readers and writers disagreeing on freshness requirements
+- stale or conflicting behavior treated as incidental instead of product design
+
+### Uncoordinated Multi-Writes
+- writing to several authorities in one operation with no atomicity or repair strategy
+- side effects sent before durable state with no recovery path
+
+### Schema Drift by Accident
+- changing payload meaning without versioning
+- reusing fields for new concepts
+- no rollout compatibility strategy
+
+---
+
+## Code Generation Rules
+
+When generating code, default to:
+1. explicit identifiers and ownership
+2. explicit idempotency where retries or duplicates can happen
+3. explicit versioning or conflict strategy where ordering matters
+4. explicit distinction between authoritative and derived data
+5. repairable or rebuildable downstream state
+6. compatibility-aware schema changes
+7. observability for lag, retries, and failures
+
+Avoid by default:
+- assuming strict global order
+- exactly-once promises with no proof
+- writing the same fact into several places as if they were one transaction
+- treating streams and queues as fire-and-forget
+
+---
+
+## Testing Rules
+
+1. Test duplicate delivery handling.
+2. Test out-of-order event or message handling where applicable.
+3. Test replay safety.
+4. Test conflict resolution or optimistic concurrency behavior.
+5. Test schema compatibility when contracts evolve.
+6. Test rebuild or repair of derived views where that capability exists.
+
+---
+
+## Review Checklist
+
+Before finalizing any change, verify:
+- Is the source of truth explicit?
+- Are consistency expectations explicit?
+- Is the code safe under retry or duplicate delivery?
+- Is ordering dependency explicit and scoped?
+- Can derived data be rebuilt or repaired?
+- Is schema evolution considered?
+- Is atomicity scope honest?
+- Did we avoid exactly-once wishful thinking?
+- Are service boundaries aligned with data ownership?
+- Are lag and failure observable?
+
+If any answer is no, revise before shipping.
+
+---
+
+## Final Instruction
+
+When uncertain, prefer the design that:
+1. makes data ownership explicit
+2. makes consistency semantics explicit
+3. survives retries, duplicates, and replay
+4. supports evolution without silent breakage
+5. treats distributed systems trade-offs honestly
+
+Do not hide distributed complexity behind local-looking code.

package/templates/skills/designing-data-intensive-applications/designing-data-intensive-applications.mini.md

@@ -0,0 +1,55 @@
+# OBEY Designing Data-Intensive Applications by Martin Kleppmann
+
+## When to use
+
+Use for systems where correctness depends on data ownership, consistency, durability, replication, partitioning, schema evolution, event flow, replay, or derived-data maintenance.
+
+## Primary bias to correct
+
+Do not design distributed data behavior as if every write, read, queue, cache, replica, clock, and downstream side effect were local, ordered, fresh, and exactly once.
+
+## Decision rules
+
+- Make core trade-offs explicit: source of truth, consistency expectation, retry behavior, duplicate and reordered work, partial failure, data evolution, and whether state is durable, cached, derived, or ephemeral.
+- Treat crashes, partial writes, duplicate work, timeouts, stale reads, and unknown downstream success as normal inputs. Distinguish accepted, persisted, applied, and durable success.
+- Describe load and performance with concrete request rates, data volume, access patterns, latency, throughput, percentiles, bottlenecks, contention, and tail behavior before changing architecture.
+- Choose data models, query models, and ownership boundaries from relationships, access patterns, consistency needs, update locality, evolution pressure, and whether data is primary or derived.
+- Match storage engines, indexes, and analytical layouts to write patterns, read patterns, range scans, recovery needs, write amplification, OLTP-vs-analytics separation, and memory-vs-durability assumptions.
+- Treat indexes, caches, search copies, read models, materialized views, and denormalized copies as derived data with explicit propagation, lag, observability, repair, and rebuild paths.
+- Define write semantics: when a write is durable, when it is visible, whether stale reads are allowed, which conflicts can happen, and how conflicts are detected or resolved.
+- Make commands, jobs, events, batch jobs, and stream processors safe under retry and replay with deduplication keys, naturally idempotent transitions, or an explicit transactional recovery contract.
+- Preserve only the ordering the business logic actually needs. Scope it per key, stream, partition, record, entity history, or stronger contract, and keep ordering-sensitive logic close to that scope.
+- Separate commands, events, durable logs, streams, and materialized views. Events describe facts; consumers must tolerate lag, duplicates, restart, replay, stable identifiers, correlation metadata, and versioned payloads.
+- Design schemas, encodings, APIs, messages, events, and database changes as evolving contracts across old readers, old writers, old data, in-flight messages, rolling upgrades, and cross-service formats.
+- Choose replication topology from write topology, latency, failure tolerance, lag, failover, reconfiguration, conflict handling, read-your-writes, monotonic-read, consistent-prefix, quorum, and convergence needs.
+- Partition by workload-relevant locality and consistency keys, with hot-key, skew, routing, secondary-index, rebalancing, and cross-partition-operation costs explicit.
+- Match transactions and isolation to invariants. Make atomicity scope, commit behavior, recovery, reconciliation, lost-update, write-skew, phantom, and side-effect repair semantics explicit.
+- Treat network delay, packet loss, partitions, duplicate messages, pauses, stale leaders, timeouts, wall-clock uncertainty, leases, locks, majorities, and leadership as assumptions needing a fault model.
+- Use linearizability, total order broadcast, atomic commit, or consensus only where the coordination problem truly requires agreement and the availability or latency cost is acceptable.
+- Make batch and stream processing recomputable and recoverable: define inputs, outputs, intermediate state, checkpoints, external side effects, event time, processing time, ingestion time, windows, late data, joins, and source-to-sink guarantees.
+- Align service boundaries with data ownership and update semantics. Do not casually split one tightly consistent business concept across services or put chatty cross-service joins on hot paths.
+
+## Trigger rules
+
+- When changing a write path, state the source of truth, consistency boundary, durability point, visibility point, downstream effects, rollback or repair path, and behavior after timeout or unknown success.
+- When adding or changing a cache, index, projection, search copy, read model, warehouse, or denormalized field, define ownership, propagation, staleness, write cost, lag visibility, rebuild, and repair.
+- When changing a schema, API, message, event, enum, status, or payload meaning, plan compatibility for old readers, old writers, old stored data, old messages, new writers, rollout, and migration.
+- When adding retries, jobs, consumers, queues, CDC, event sourcing, stream processors, or replayable batch work, prove duplicate, replay, ordering, retention, side-effect, and recovery safety.
+- When routing reads to replicas or using asynchronous replication, identify read-your-writes, monotonic-read, consistent-prefix, staleness, catch-up, failover, and conflict expectations before allowing the read.
+- When partitioning data or work, test the ordinary query path for locality, skew, hot keys, routing metadata, rebalancing cost, secondary-index behavior, and cross-partition coordination.
+- When choosing transaction isolation or weakening consistency, map each anomaly to the invariant it can break and add serializable isolation, locks, compare-and-set, versioning, reconciliation, or another compensating design where needed.
+- When using timestamps, leases, locks, leadership, majority decisions, coordination services, or consensus-like mechanisms, define the clock assumption, quorum/session semantics, stale-authority behavior, and fencing.
+- When reviewing or testing data-intensive code, look specifically for hidden source-of-truth ownership, missing idempotency, accidental exactly-once assumptions, unscoped ordering, schema drift, unrebuildable projections, unclear multi-writes, and unobservable lag or failure.
+
+## Final checklist
+
+- Source of truth and derived representations are explicit.
+- Consistency expectations, durability points, visibility points, staleness, and conflict rules are concrete.
+- Retries, duplicate delivery, replay, reordering, timeouts, crashes, and unknown success are handled.
+- Schemas, encodings, APIs, messages, events, enums, and statuses evolve safely across mixed versions.
+- Storage, indexing, replication, partitioning, routing, and analytical layouts match the actual workload.
+- Transaction isolation and coordination choices protect the named invariants.
+- Events, logs, streams, batch jobs, and projections are replayable or have explicit repair paths.
+- Service boundaries follow data ownership and update semantics.
+- Lag, retries, failures, rebuilds, and repair paths are observable.
+- The design avoids exactly-once wishful thinking and hidden distributed-system contracts.

package/templates/skills/designing-data-intensive-applications/designing-data-intensive-applications.nano.md

@@ -0,0 +1,34 @@
+# OBEY Designing Data-Intensive Applications by Martin Kleppmann
+
+## When to use
+
+Use when data correctness, durability, or distributed write semantics matter more than local code style.
+
+## Primary bias to correct
+
+Hidden data contracts are still contracts.
+
+## Decision rules
+
+- State the source of truth, consistency expectation, durability point, visibility point, retry semantics, and evolution path for every important data change.
+- Choose data models, storage, indexes, replication, partitioning, transactions, queues, streams, and APIs from workload, access pattern, consistency, reliability, maintainability, and operational cost.
+- Treat caches, indexes, projections, search copies, denormalized data, and materialized views as derived data with staleness, lag visibility, repair, and rebuild paths.
+- Make retried, replayed, queued, batch, stream, and event-driven work idempotent or transactional; reject casual exactly-once claims.
+- Treat schemas, encodings, service APIs, messages, logs, and events as versioned contracts that must survive old code, old data, rolling upgrades, and in-flight messages.
+- Assume distributed uncertainty: crashes, partial writes, timeouts, duplicate messages, reordered events, stale replicas, lag, clock error, pauses, stale leaders, and unknown success.
+- Match replication, partitioning, isolation, transactions, and coordination to the invariant; do not rely on follower freshness, quorum formulas, weak isolation, wall-clock order, or ad hoc leadership without proof.
+
+## Trigger rules
+
+- When adding retries, jobs, consumers, queues, CDC, event sourcing, or stream processing, prove duplicate, replay, ordering, side-effect, and recovery safety.
+- When changing schemas, APIs, messages, events, enum values, or status meanings, plan backward and forward compatibility plus migration, bootstrap, or rebuild paths.
+- When reading from replicas or partitioning data, define staleness, routing, hot-key, ordering, rebalancing, and cross-partition behavior.
+- When using locks, leases, timestamps, leadership, majorities, or coordination services, define the fault model, quorum/session semantics, stale-authority behavior, and fencing.
+
+## Final checklist
+
+- Clear owner and source of truth?
+- Explicit consistency, durability, visibility, and staleness semantics?
+- Safe under retry, replay, duplicate delivery, reordering, and unknown success?
+- Compatible across old data, old code, new code, and messages in flight?
+- Isolation, replication, partitioning, transactions, and coordination checked against actual invariants?

package/templates/skills/domain-driven-design/SKILL.md

@@ -0,0 +1,11 @@
+---
+name: domain-driven-design
+description: Apply Eric Evans-inspired DDD rules when modeling a complex domain, defining bounded contexts, protecting ubiquitous language, or designing aggregates.
+license: MIT
+---
+
+# Domain-Driven Design Skill
+
+Use this skill when a task involves strategic domain modeling, ubiquitous language, bounded contexts, context mapping, aggregates, repositories, domain services, or model-driven design.
+
+Before making design or code decisions, read and apply [domain-driven-design.mini.md](domain-driven-design.mini.md). Use [domain-driven-design.md](domain-driven-design.md) only as a deeper reference when the mini rules are not enough for the current domain-modeling tradeoff.