@enki-dev-expert/global-engineering-standards 0.0.0-stage → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/AGENTS.md +303 -0
  2. package/CHANGELOG.md +28 -0
  3. package/CITATION.cff +21 -0
  4. package/CODE_OF_CONDUCT.md +13 -0
  5. package/CONTRIBUTING.md +27 -0
  6. package/LICENSE +21 -0
  7. package/README.md +159 -2
  8. package/SECURITY.md +25 -0
  9. package/dist/cli.js +40 -0
  10. package/dist/content.js +71 -0
  11. package/dist/filesystem.js +83 -0
  12. package/dist/installer.js +243 -0
  13. package/dist/layout.js +42 -0
  14. package/dist/transaction.js +143 -0
  15. package/docs/ai-and-llm.md +1312 -0
  16. package/docs/architecture.md +1007 -0
  17. package/docs/cost-engineering.md +1322 -0
  18. package/docs/data.md +993 -0
  19. package/docs/governance.md +1744 -0
  20. package/docs/mobile-and-web.md +1362 -0
  21. package/docs/observability.md +1345 -0
  22. package/docs/platforms-and-applications.md +1581 -0
  23. package/docs/privacy.md +566 -0
  24. package/docs/quality-and-testing.md +1278 -0
  25. package/docs/scalability.md +1012 -0
  26. package/docs/security.md +373 -0
  27. package/handbook/claude-plugin.md +31 -0
  28. package/handbook/compatibility.md +27 -0
  29. package/handbook/recovery.md +29 -0
  30. package/handbook/releasing.md +50 -0
  31. package/handbook/security-review.md +41 -0
  32. package/handbook/signing.md +18 -0
  33. package/handbook/validation.md +50 -0
  34. package/package.json +65 -4
  35. package/scripts/content-baseline.json +25 -0
  36. package/skills/engineering-architecture-review/SKILL.md +1531 -0
  37. package/skills/engineering-cost-estimate/SKILL.md +1599 -0
  38. package/skills/engineering-database-review/SKILL.md +1364 -0
  39. package/skills/engineering-privacy-check/SKILL.md +1295 -0
  40. package/skills/engineering-safe-implementation/SKILL.md +1502 -0
  41. package/skills/engineering-security-standards-check/SKILL.md +1112 -0
  42. package/src/cli.ts +45 -0
  43. package/src/content.ts +82 -0
  44. package/src/filesystem.ts +94 -0
  45. package/src/installer.ts +280 -0
  46. package/src/layout.ts +70 -0
  47. package/src/transaction.ts +167 -0
  48. package/templates/ADR.md +1302 -0
  49. package/templates/AUDIT_REPORT.md +1693 -0
  50. package/templates/EXCEPTIONS.md +647 -0
  51. package/templates/PROJECT_AGENTS.md +868 -0
package/docs/data.md ADDED
@@ -0,0 +1,993 @@
1
+
2
+ # Global Data Engineering Standards
3
+
4
+ ## 1. Mission
5
+
6
+ Design, maintain and evolve secure, reliable, scalable and cost-efficient data systems.
7
+
8
+ These standards apply to new and existing production applications, including web, iOS, Android, SaaS, AI systems and enterprise platforms.
9
+
10
+ They apply to all relevant database technologies, cloud providers, storage systems, data processing services and deployment environments.
11
+
12
+ These standards are database-agnostic, cloud-agnostic and vendor-neutral.
13
+
14
+ Never assume a specific database, cloud provider or infrastructure architecture before inspecting the project.
15
+
16
+ Prioritize:
17
+
18
+ 1. Security and privacy.
19
+ 2. Data integrity and correctness.
20
+ 3. Availability and recoverability.
21
+ 4. Backward compatibility.
22
+ 5. Performance and scalability.
23
+ 6. Cost efficiency.
24
+
25
+ Never modify production data, schemas or infrastructure without explicit human approval.
26
+
27
+ ## 2. Cloud and Database Provider Independence
28
+
29
+ These standards apply independently of the technology or provider selected.
30
+
31
+ No database engine, cloud provider, storage platform or infrastructure technology is mandatory or preferred by default.
32
+
33
+ Apply universal engineering principles to every system.
34
+
35
+ Apply provider-specific requirements only when that technology is actually used.
36
+
37
+ ### Amazon Web Services (AWS)
38
+
39
+ Support architectures using relevant services, including:
40
+
41
+ - Amazon RDS.
42
+ - Amazon Aurora.
43
+ - Amazon DynamoDB.
44
+ - Amazon S3.
45
+ - Amazon ElastiCache.
46
+ - Amazon Redshift.
47
+ - Amazon EKS.
48
+ - Other relevant AWS data and infrastructure services.
49
+
50
+ ### Microsoft Azure
51
+
52
+ Support architectures using relevant services, including:
53
+
54
+ - Azure SQL Database.
55
+ - Azure Database for PostgreSQL.
56
+ - Azure Cosmos DB.
57
+ - Azure Blob Storage.
58
+ - Azure Managed Redis.
59
+ - Azure Kubernetes Service (AKS).
60
+ - Other relevant Azure data and infrastructure services.
61
+
62
+ ### Google Cloud Platform (GCP)
63
+
64
+ Support architectures using relevant services, including:
65
+
66
+ - Cloud SQL.
67
+ - AlloyDB.
68
+ - Cloud Spanner.
69
+ - Firestore.
70
+ - Bigtable.
71
+ - BigQuery.
72
+ - Cloud Storage.
73
+ - Memorystore.
74
+ - Google Kubernetes Engine (GKE).
75
+ - Other relevant Google Cloud data and infrastructure services.
76
+
77
+ ### Other Providers and Technologies
78
+
79
+ Support technologies including, but not limited to:
80
+
81
+ - Supabase.
82
+ - Firebase.
83
+ - PostgreSQL.
84
+ - MySQL.
85
+ - MongoDB Atlas.
86
+ - Neon.
87
+ - PlanetScale.
88
+ - Redis.
89
+ - Valkey.
90
+ - Oracle Database.
91
+ - ClickHouse.
92
+ - Snowflake.
93
+ - Managed and self-hosted vector databases.
94
+ - Managed Kubernetes providers.
95
+ - Self-hosted databases.
96
+ - Hybrid infrastructure.
97
+ - On-premises environments.
98
+
99
+ Do not assume that services with similar purposes have identical security, consistency, availability or billing models.
100
+
101
+ Verify provider-specific behavior using current official documentation.
102
+
103
+ ### Provider Selection Rules
104
+
105
+ Before recommending a provider:
106
+
107
+ 1. Inspect the existing technology stack.
108
+ 2. Identify business and functional requirements.
109
+ 3. Define data structures and relationships.
110
+ 4. Evaluate consistency and transaction requirements.
111
+ 5. Evaluate security and privacy.
112
+ 6. Evaluate expected workloads and growth.
113
+ 7. Evaluate geographic and latency requirements.
114
+ 8. Review availability and disaster recovery.
115
+ 9. Assess operational complexity.
116
+ 10. Compare total cost of ownership.
117
+ 11. Evaluate vendor lock-in.
118
+ 12. Evaluate migration risks.
119
+
120
+ For existing production applications, preserve the current provider unless a migration is justified by strong evidence and explicitly approved.
121
+
122
+ Never recommend switching database providers merely because another technology is popular.
123
+
124
+ Avoid unnecessary multi-cloud complexity.
125
+
126
+ Design for millions of users while provisioning according to measured demand.
127
+
128
+ ## 3. Kubernetes and Containerized Data Infrastructure
129
+
130
+ These standards support Kubernetes architectures across AWS, Azure, GCP, other cloud providers, hybrid environments and on-premises infrastructure.
131
+
132
+ Kubernetes is an orchestration platform, not a database provider.
133
+
134
+ It must never be assumed to be necessary solely because an application may grow to millions of users.
135
+
136
+ ### Kubernetes Adoption
137
+
138
+ Before recommending Kubernetes, evaluate:
139
+
140
+ - Existing infrastructure.
141
+ - Workload characteristics.
142
+ - Expected concurrency.
143
+ - Independent scaling requirements.
144
+ - Deployment frequency.
145
+ - Reliability requirements.
146
+ - Security and isolation requirements.
147
+ - Operational capabilities.
148
+ - Infrastructure costs.
149
+ - Management complexity.
150
+ - Simpler managed and serverless alternatives.
151
+
152
+ Provide evidence-based justification.
153
+
154
+ ### Supported Kubernetes Environments
155
+
156
+ Consider where relevant:
157
+
158
+ - Amazon Elastic Kubernetes Service (EKS).
159
+ - Azure Kubernetes Service (AKS).
160
+ - Google Kubernetes Engine (GKE).
161
+ - Other managed Kubernetes services.
162
+ - Self-managed Kubernetes.
163
+ - Hybrid Kubernetes deployments.
164
+ - On-premises Kubernetes.
165
+
166
+ Evaluate actual provider capabilities rather than assuming identical functionality.
167
+
168
+ ### Stateful Data Architecture
169
+
170
+ For databases and stateful workloads, evaluate:
171
+
172
+ - Managed databases versus Kubernetes-hosted databases.
173
+ - StatefulSets.
174
+ - PersistentVolumes.
175
+ - PersistentVolumeClaims.
176
+ - StorageClasses.
177
+ - Storage performance.
178
+ - Storage availability zones.
179
+ - Database replication.
180
+ - Backup and restoration.
181
+ - Data consistency.
182
+ - Recovery objectives.
183
+ - Storage expansion.
184
+ - Operational responsibility.
185
+
186
+ Prefer managed databases when they provide better reliability, operational simplicity, security or cost efficiency.
187
+
188
+ Never assume Kubernetes automatically provides database high availability, consistency, backups or disaster recovery.
189
+
190
+ ### Kubernetes Security
191
+
192
+ Apply appropriate controls including:
193
+
194
+ - Kubernetes RBAC.
195
+ - Least-privilege service accounts.
196
+ - Secure workload identities.
197
+ - Pod security controls.
198
+ - NetworkPolicies where supported.
199
+ - Namespace isolation.
200
+ - Tenant isolation.
201
+ - Secrets management.
202
+ - Container image scanning.
203
+ - Trusted image registries.
204
+ - Image provenance verification.
205
+ - Restricted administrative access.
206
+ - Audit logging.
207
+ - Secure persistent storage.
208
+
209
+ Never embed privileged credentials in container images.
210
+
211
+ Never grant unrestricted cluster privileges to application workloads or AI development agents.
212
+
213
+ ### Kubernetes Scalability
214
+
215
+ Evaluate:
216
+
217
+ - Horizontal Pod Autoscaling (HPA).
218
+ - Vertical Pod Autoscaling (VPA).
219
+ - Node autoscaling.
220
+ - CPU and memory requests.
221
+ - CPU and memory limits.
222
+ - Resource quotas.
223
+ - Pod disruption budgets.
224
+ - Load balancing.
225
+ - Health and readiness probes.
226
+ - Rolling deployments.
227
+ - Workload scheduling.
228
+ - Capacity planning.
229
+ - Load testing.
230
+
231
+ Use measurable thresholds.
232
+
233
+ Never claim Kubernetes alone guarantees support for millions of users.
234
+
235
+ ### Kubernetes Cost Engineering
236
+
237
+ Evaluate:
238
+
239
+ - Control plane charges where applicable.
240
+ - Worker nodes.
241
+ - CPU utilization.
242
+ - Memory utilization.
243
+ - Persistent storage.
244
+ - Storage operations.
245
+ - Load balancers.
246
+ - Network transfers.
247
+ - Cross-zone and cross-region traffic.
248
+ - Logging and monitoring.
249
+ - Backup and recovery.
250
+ - Cluster administration.
251
+ - Operational complexity.
252
+
253
+ Avoid permanently overprovisioned clusters.
254
+
255
+ Compare Kubernetes against managed application platforms, serverless services and simpler architectures.
256
+
257
+ ### Production Changes
258
+
259
+ Before modifying production Kubernetes resources:
260
+
261
+ 1. Inspect existing resources and dependencies.
262
+ 2. Identify affected services.
263
+ 3. Identify affected data and storage.
264
+ 4. Evaluate availability and integrity risks.
265
+ 5. Review security implications.
266
+ 6. Review operational costs.
267
+ 7. Validate configuration and deployment manifests.
268
+ 8. Test in a suitable non-production environment.
269
+ 9. Define rollback and recovery procedures.
270
+ 10. Obtain explicit human approval.
271
+
272
+ Never execute production cluster changes automatically.
273
+
274
+ ## 4. Data Architecture First
275
+
276
+ Before implementing significant data functionality:
277
+
278
+ 1. Identify business requirements.
279
+ 2. Identify entities and relationships.
280
+ 3. Identify authoritative data sources.
281
+ 4. Define business invariants.
282
+ 5. Map data ownership and flows.
283
+ 6. Classify sensitive information.
284
+ 7. Establish authorization requirements.
285
+ 8. Identify transaction boundaries.
286
+ 9. Estimate data volumes and growth.
287
+ 10. Evaluate performance and costs.
288
+ 11. Define migration and recovery strategies.
289
+ 12. Present an implementation plan.
290
+ 13. Wait for approval.
291
+
292
+ Document confirmed facts separately from assumptions.
293
+
294
+ ## 5. Data Modeling
295
+
296
+ Design schemas according to actual domain requirements.
297
+
298
+ Use appropriate:
299
+
300
+ - Primary keys.
301
+ - Foreign keys.
302
+ - Unique constraints.
303
+ - NOT NULL constraints.
304
+ - CHECK constraints.
305
+ - Referential actions.
306
+ - Data types.
307
+ - Domain-specific invariants.
308
+
309
+ Apply normalization pragmatically.
310
+
311
+ Use denormalization only when justified by measured or modeled requirements.
312
+
313
+ Avoid unnecessary duplicated data.
314
+
315
+ Document relationships, ownership and cardinality.
316
+
317
+ Never store the same authoritative business fact in multiple independent places without a defined consistency mechanism.
318
+
319
+ ## 6. Single Source of Truth
320
+
321
+ Identify one authoritative source for each business fact.
322
+
323
+ Examples:
324
+
325
+ - User identity.
326
+ - Account ownership.
327
+ - Reservation status.
328
+ - Payment confirmation.
329
+ - Subscription entitlement.
330
+ - Inventory availability.
331
+
332
+ Derived data may be cached or replicated when appropriate.
333
+
334
+ Define synchronization, invalidation and reconciliation rules.
335
+
336
+ Never treat an unverified client-side value as authoritative.
337
+
338
+ ## 7. Data Classification
339
+
340
+ Classify information as:
341
+
342
+ - Public.
343
+ - Internal.
344
+ - Confidential.
345
+ - Sensitive or restricted.
346
+
347
+ Apply stronger safeguards to personal, financial, health, biometric, location, authentication and other high-risk information.
348
+
349
+ Classification must consider actual processing risks and applicable legal requirements.
350
+
351
+ Never expose confidential or sensitive information unnecessarily in logs, telemetry, analytics or AI prompts.
352
+
353
+ Consult docs/privacy.md for detailed requirements.
354
+
355
+ ## 8. Data Access and Authorization
356
+
357
+ Apply least privilege and secure defaults.
358
+
359
+ Enforce authorization at trusted boundaries.
360
+
361
+ Validate ownership and tenant membership.
362
+
363
+ Prevent horizontal and vertical privilege escalation.
364
+
365
+ Restrict administrative and privileged database operations.
366
+
367
+ Use separate credentials and roles when appropriate.
368
+
369
+ Never rely solely on frontend access restrictions.
370
+
371
+ Verify authorization through tests using different users, roles and tenants.
372
+
373
+ ## 9. Multi-Tenant Data Isolation
374
+
375
+ For multi-tenant applications:
376
+
377
+ - Define tenant ownership.
378
+ - Enforce isolation consistently.
379
+ - Prevent cross-tenant queries.
380
+ - Review shared caches.
381
+ - Protect file storage.
382
+ - Validate background jobs.
383
+ - Review administrative access.
384
+ - Test cross-tenant attack scenarios.
385
+
386
+ Choose shared-schema, separate-schema or separate-database isolation according to verified security, scale, compliance and cost requirements.
387
+
388
+ Do not assume every application requires the same tenant architecture.
389
+
390
+ ## 10. Supabase Security
391
+
392
+ For Supabase applications:
393
+
394
+ - Review exposed schemas.
395
+ - Enable RLS for applicable exposed tables.
396
+ - Verify SQL grants and privileges.
397
+ - Review SELECT policies.
398
+ - Review INSERT policies.
399
+ - Review UPDATE policies.
400
+ - Review DELETE policies.
401
+ - Validate user ownership rules.
402
+ - Validate tenant isolation.
403
+ - Audit SECURITY DEFINER functions.
404
+ - Restrict privileged database roles.
405
+ - Protect secret and service-role credentials.
406
+ - Review Storage bucket policies.
407
+ - Review Realtime authorization.
408
+ - Review Edge Function authentication.
409
+ - Verify permission boundaries using actual test identities.
410
+
411
+ RLS policies and SQL privileges are complementary controls.
412
+
413
+ Never assume RLS alone guarantees security.
414
+
415
+ Never expose privileged Supabase credentials in web or mobile clients.
416
+
417
+ Supabase publishable keys may be client-visible when the associated security model is correctly configured.
418
+
419
+ ## 11. Transactions and Atomicity
420
+
421
+ Use database transactions when operations must succeed or fail together.
422
+
423
+ Identify critical business invariants.
424
+
425
+ Protect against partial updates.
426
+
427
+ Examples include:
428
+
429
+ - Inventory reservations.
430
+ - Financial records.
431
+ - Subscription changes.
432
+ - Ownership transfers.
433
+ - Multi-record state transitions.
434
+
435
+ Define isolation and locking requirements according to the operation.
436
+
437
+ Avoid long-running transactions unnecessarily.
438
+
439
+ Do not split atomic business operations across independent writes without a consistency strategy.
440
+
441
+ ## 12. Concurrency Control
442
+
443
+ Analyze concurrent operations and potential race conditions.
444
+
445
+ Consider:
446
+
447
+ - Unique constraints.
448
+ - Transaction isolation.
449
+ - Row locking.
450
+ - Optimistic concurrency.
451
+ - Version numbers.
452
+ - Atomic updates.
453
+ - Conflict detection.
454
+
455
+ Never rely solely on client-side checks to prevent concurrent data inconsistencies.
456
+
457
+ Test critical operations under simultaneous requests.
458
+
459
+ ## 13. Idempotency
460
+
461
+ Operations that may be retried must have defined duplicate-handling behavior.
462
+
463
+ Particularly evaluate:
464
+
465
+ - Payments.
466
+ - Webhooks.
467
+ - Reservations.
468
+ - Background jobs.
469
+ - Notifications.
470
+ - External service events.
471
+
472
+ Use appropriate idempotency keys, uniqueness constraints or equivalent mechanisms.
473
+
474
+ Persist enough information to recognize repeated operations safely.
475
+
476
+ Define expiration and retention of idempotency records.
477
+
478
+ Never assume external systems deliver events exactly once.
479
+
480
+ ## 14. Query Engineering
481
+
482
+ Write efficient, understandable queries.
483
+
484
+ Evaluate:
485
+
486
+ - Query execution plans.
487
+ - Filter selectivity.
488
+ - Join strategies.
489
+ - Index usage.
490
+ - Returned row counts.
491
+ - Payload sizes.
492
+ - Sorting.
493
+ - Pagination.
494
+ - Repeated queries.
495
+ - N+1 query patterns.
496
+
497
+ Use parameterized queries.
498
+
499
+ Avoid unbounded result sets.
500
+
501
+ Prefer keyset pagination where justified by workload and consistency requirements.
502
+
503
+ Do not optimize queries based only on intuition.
504
+
505
+ ## 15. Indexing Strategy
506
+
507
+ Create indexes according to real access patterns.
508
+
509
+ Evaluate:
510
+
511
+ - Query frequency.
512
+ - Selectivity.
513
+ - Table size.
514
+ - Read/write balance.
515
+ - Index maintenance costs.
516
+ - Storage consumption.
517
+ - Composite indexes.
518
+ - Partial indexes.
519
+ - Unique indexes.
520
+
521
+ Use execution plans and measurements where available.
522
+
523
+ Avoid indexing every column.
524
+
525
+ Review index changes for production locking and resource impact.
526
+
527
+ For PostgreSQL, consider concurrent index creation when appropriate, while respecting its limitations and operational requirements.
528
+
529
+ Never execute production indexing operations without approval.
530
+
531
+ ## 16. Database Connections
532
+
533
+ Manage connections carefully.
534
+
535
+ Evaluate:
536
+
537
+ - Connection limits.
538
+ - Pooling.
539
+ - Serverless workloads.
540
+ - Transaction pooling compatibility.
541
+ - Long-running operations.
542
+ - Connection leaks.
543
+ - Timeouts.
544
+ - Peak concurrency.
545
+
546
+ Avoid creating unnecessary database connections per request.
547
+
548
+ Use provider-supported pooling where appropriate.
549
+
550
+ Monitor connection saturation and related costs.
551
+
552
+ ## 17. Scaling for Millions of Users
553
+
554
+ Design data architecture to evolve toward:
555
+
556
+ - 10,000 users.
557
+ - 100,000 users.
558
+ - 1,000,000 users.
559
+ - 5,000,000 users.
560
+ - 10,000,000 users.
561
+
562
+ Distinguish registered users, active users and peak concurrent workloads.
563
+
564
+ Model:
565
+
566
+ - Records per user.
567
+ - Reads per second.
568
+ - Writes per second.
569
+ - Data growth.
570
+ - Index growth.
571
+ - Storage growth.
572
+ - Connection demand.
573
+ - Network transfer.
574
+ - Backup size.
575
+ - Recovery duration.
576
+
577
+ Consider:
578
+
579
+ - Query optimization.
580
+ - Connection pooling.
581
+ - Caching.
582
+ - Read replicas.
583
+ - Background processing.
584
+ - Partitioning.
585
+ - Archival.
586
+ - Horizontal scaling.
587
+ - Sharding only when justified.
588
+
589
+ Use measurable thresholds.
590
+
591
+ Never claim million-user capacity without supporting benchmarks, load testing or validated capacity modeling.
592
+
593
+ ## 18. Caching
594
+
595
+ Use caching where justified.
596
+
597
+ Define:
598
+
599
+ - Cached data ownership.
600
+ - Cache keys.
601
+ - Tenant isolation.
602
+ - TTL.
603
+ - Invalidation.
604
+ - Consistency requirements.
605
+ - Failure behavior.
606
+ - Memory and storage costs.
607
+
608
+ Never share sensitive cached data across unauthorized users.
609
+
610
+ Avoid using cache as the only source of important persistent business information.
611
+
612
+ ## 19. Realtime Data
613
+
614
+ For realtime systems, evaluate:
615
+
616
+ - Active connections.
617
+ - Message frequency.
618
+ - Subscription authorization.
619
+ - Presence tracking.
620
+ - Ordering.
621
+ - Duplicate messages.
622
+ - Delivery guarantees.
623
+ - Reconnection.
624
+ - Mobile background behavior.
625
+ - Network interruptions.
626
+ - Cost per connection and message.
627
+
628
+ For Supabase Realtime, review applicable quotas, authorization rules and billing dimensions.
629
+
630
+ Do not assume realtime subscriptions are free or infinitely scalable.
631
+
632
+ ## 20. Object Storage
633
+
634
+ For images, videos, documents and attachments:
635
+
636
+ - Separate public and private resources.
637
+ - Enforce ownership and authorization.
638
+ - Validate file types and sizes.
639
+ - Protect signed URLs.
640
+ - Define expiration rules.
641
+ - Implement retention and deletion.
642
+ - Consider malware scanning where appropriate.
643
+ - Optimize image and media formats.
644
+ - Avoid unnecessary duplicate storage.
645
+ - Monitor bandwidth and egress.
646
+
647
+ Do not expose confidential objects through public buckets.
648
+
649
+ ## 21. Vector Databases and Embeddings
650
+
651
+ For applications using RAG or vector search:
652
+
653
+ - Define source document ownership.
654
+ - Enforce authorization before retrieval.
655
+ - Preserve tenant isolation.
656
+ - Track document and embedding versions.
657
+ - Avoid unnecessary duplicate embeddings.
658
+ - Define indexing and reindexing workflows.
659
+ - Handle document updates and deletion.
660
+ - Remove or invalidate embeddings when required.
661
+ - Monitor embedding and storage costs.
662
+ - Evaluate retrieval quality.
663
+ - Prevent unauthorized cross-user retrieval.
664
+
665
+ Never assume vector search automatically respects source-system permissions.
666
+
667
+ ## 22. Data Migration Strategy
668
+
669
+ All significant schema changes require planning.
670
+
671
+ Use staged, backward-compatible migrations where possible.
672
+
673
+ Prefer Expand-Migrate-Contract:
674
+
675
+ 1. Expand the schema compatibly.
676
+ 2. Deploy compatible application code.
677
+ 3. Backfill or migrate data safely.
678
+ 4. Validate the new behavior.
679
+ 5. Switch consumers progressively.
680
+ 6. Remove obsolete structures only after approval.
681
+
682
+ Consider older mobile application versions.
683
+
684
+ Identify locking, downtime and rollback risks.
685
+
686
+ Never perform destructive production migrations automatically.
687
+
688
+ ## 23. Migration Safety
689
+
690
+ Before production migration:
691
+
692
+ - Review the exact migration.
693
+ - Assess affected tables.
694
+ - Estimate data volume.
695
+ - Evaluate locking.
696
+ - Evaluate transaction behavior.
697
+ - Establish backup or recovery capability.
698
+ - Validate in staging.
699
+ - Define monitoring.
700
+ - Establish rollback or forward-recovery procedures.
701
+ - Obtain explicit human approval.
702
+
703
+ A schema rollback may not reverse lost data.
704
+
705
+ Never claim a migration is reversible without verifying that property.
706
+
707
+ ## 24. Backup and Recovery
708
+
709
+ Define recovery objectives according to business requirements.
710
+
711
+ Document:
712
+
713
+ - Recovery Point Objective (RPO).
714
+ - Recovery Time Objective (RTO).
715
+ - Backup frequency.
716
+ - Backup retention.
717
+ - Encryption.
718
+ - Access controls.
719
+ - Restore procedures.
720
+ - Recovery responsibilities.
721
+
722
+ Test restoration periodically.
723
+
724
+ A backup that has never been restored is not sufficient evidence of recoverability.
725
+
726
+ Consider point-in-time recovery when justified.
727
+
728
+ ## 25. Data Retention and Deletion
729
+
730
+ Define retention for each data category.
731
+
732
+ Address:
733
+
734
+ - Primary tables.
735
+ - Storage objects.
736
+ - Caches.
737
+ - Search indexes.
738
+ - Vector indexes.
739
+ - Logs.
740
+ - Analytics datasets.
741
+ - Backups.
742
+ - Third-party services.
743
+
744
+ Apply privacy and legal requirements.
745
+
746
+ Implement appropriate deletion workflows.
747
+
748
+ Prevent accidental resurrection of deleted data after backup restoration.
749
+
750
+ Where legal retention or backup lifecycle constraints apply, document and enforce them.
751
+
752
+ Consult docs/privacy.md.
753
+
754
+ ## 26. Data Observability
755
+
756
+ Monitor relevant indicators:
757
+
758
+ - Slow queries.
759
+ - Query latency.
760
+ - Database CPU.
761
+ - Memory usage.
762
+ - Disk usage.
763
+ - Connection utilization.
764
+ - Lock contention.
765
+ - Transaction failures.
766
+ - Deadlocks.
767
+ - Replication lag.
768
+ - Storage growth.
769
+ - Backup failures.
770
+ - Recovery readiness.
771
+ - Database costs.
772
+
773
+ Define actionable alerts.
774
+
775
+ Avoid collecting sensitive data unnecessarily.
776
+
777
+ ## 27. Data Security Testing
778
+
779
+ Test:
780
+
781
+ - User-level access.
782
+ - Tenant isolation.
783
+ - Unauthorized reads.
784
+ - Unauthorized writes.
785
+ - Ownership changes.
786
+ - Privilege escalation.
787
+ - Storage permissions.
788
+ - Realtime access.
789
+ - Administrative operations.
790
+ - Migration compatibility.
791
+
792
+ For Supabase, use dedicated test identities to evaluate RLS policies and database permissions.
793
+
794
+ Do not assume authentication tests also prove authorization correctness.
795
+
796
+ ## 28. Database Cost Engineering
797
+
798
+ Estimate total database-related costs.
799
+
800
+ Include:
801
+
802
+ - Compute.
803
+ - Storage.
804
+ - Index storage.
805
+ - Backup storage.
806
+ - Connections and pooling.
807
+ - Read/write workload.
808
+ - Network transfer.
809
+ - Replication.
810
+ - Realtime usage.
811
+ - Logs and monitoring.
812
+ - Data recovery services.
813
+ - High-availability options.
814
+
815
+ Consider provider-specific billing rules and current prices.
816
+
817
+ Never fabricate pricing data.
818
+
819
+ Use explicit assumptions and measured utilization whenever possible.
820
+
821
+ ## 29. Supabase Cost Engineering
822
+
823
+ For Supabase projects, investigate applicable billing dimensions such as:
824
+
825
+ - Project compute.
826
+ - Database storage.
827
+ - Storage objects.
828
+ - Cached and uncached egress.
829
+ - Monthly active authentication users.
830
+ - Edge Function invocations.
831
+ - Realtime messages.
832
+ - Peak realtime connections.
833
+ - Additional database services.
834
+ - Backups and point-in-time recovery.
835
+ - Optional add-ons.
836
+
837
+ Verify which dimensions are billed under the current plan.
838
+
839
+ Check organizational quotas and project-level resource costs.
840
+
841
+ Recommend:
842
+
843
+ - Usage dashboards.
844
+ - Budget alerts.
845
+ - Resource limits where supported.
846
+ - Efficient queries.
847
+ - Optimized connection usage.
848
+ - Appropriate storage lifecycle.
849
+ - Reduced unnecessary realtime traffic.
850
+ - Safe request and retry limits.
851
+
852
+ Never assume alerts automatically stop charges.
853
+
854
+ ## 30. Cost Forecasting
855
+
856
+ For significant new functionality, model reasonable scenarios.
857
+
858
+ Include:
859
+
860
+ - Current utilization.
861
+ - Expected utilization.
862
+ - Growth assumptions.
863
+ - Peak demand.
864
+ - Cost per active user.
865
+ - Cost per business operation.
866
+ - Cost per stored record where meaningful.
867
+ - Third-party API interactions.
868
+ - Data transfer expenses.
869
+
870
+ Compare reasonable alternatives.
871
+
872
+ Clearly distinguish measured costs, estimates and unknowns.
873
+
874
+ Do not recommend expensive infrastructure without evidence-based justification.
875
+
876
+ ## 31. Provider Portability
877
+
878
+ Evaluate vendor dependence pragmatically.
879
+
880
+ Document:
881
+
882
+ - Provider-specific features.
883
+ - Data export capabilities.
884
+ - Migration complexity.
885
+ - Contractual restrictions.
886
+ - Operational switching costs.
887
+ - Data transfer costs.
888
+ - Required architectural adaptations.
889
+
890
+ Use abstraction when it provides meaningful portability or testability.
891
+
892
+ Do not build complex abstractions for hypothetical migrations without a clear benefit.
893
+
894
+ ## 32. Data Testing
895
+
896
+ Test important data workflows using appropriate:
897
+
898
+ - Unit tests.
899
+ - Integration tests.
900
+ - Transaction tests.
901
+ - Constraint tests.
902
+ - Concurrency tests.
903
+ - Migration tests.
904
+ - Authorization tests.
905
+ - Recovery tests.
906
+ - Load tests.
907
+
908
+ Use synthetic or appropriately anonymized datasets.
909
+
910
+ Avoid using real production data in development without a specifically authorized and lawful process.
911
+
912
+ Never weaken tests merely to make them pass.
913
+
914
+ ## 33. Existing Production Database Audits
915
+
916
+ When auditing an existing production application:
917
+
918
+ 1. Inspect available schema definitions.
919
+ 2. Analyze migrations.
920
+ 3. Review database access patterns.
921
+ 4. Map data ownership and security boundaries.
922
+ 5. Evaluate integrity constraints.
923
+ 6. Identify confirmed authorization risks.
924
+ 7. Review query and indexing strategies.
925
+ 8. Assess scalability and operational costs.
926
+ 9. Identify migration and compatibility risks.
927
+ 10. Recommend incremental improvements.
928
+ 11. Present an evidence-based report.
929
+ 12. Wait for explicit approval.
930
+
931
+ The initial audit must be read-only.
932
+
933
+ Do not alter tables, records, policies, indexes, triggers, functions or production settings.
934
+
935
+ If direct database access is necessary, request authorization and prefer appropriately restricted read-only access.
936
+
937
+ Do not claim to have inspected live database state if only source files were reviewed.
938
+
939
+ ## 34. Data Change Approval
940
+
941
+ Require explicit approval before:
942
+
943
+ - Schema changes.
944
+ - New production indexes.
945
+ - Data backfills.
946
+ - Bulk updates.
947
+ - Destructive migrations.
948
+ - Retention policy changes.
949
+ - New data collection.
950
+ - New external data providers.
951
+ - Database technology changes.
952
+ - Significant storage restructuring.
953
+ - Changes to tenant isolation.
954
+ - Changes to privileged access.
955
+
956
+ Provide expected impact, validation and recovery plans before approval.
957
+
958
+ ## 35. Data Definition of Done
959
+
960
+ For significant data changes:
961
+
962
+ - Requirements documented.
963
+ - Data ownership identified.
964
+ - Schema and constraints reviewed.
965
+ - Authorization verified.
966
+ - Transaction behavior tested.
967
+ - Concurrency risks evaluated.
968
+ - Migration risks assessed.
969
+ - Performance evaluated.
970
+ - Operational costs considered.
971
+ - Privacy obligations reviewed.
972
+ - Backups and recovery considered.
973
+ - Compatibility verified.
974
+ - Tests executed and results reported.
975
+ - Remaining risks documented.
976
+ - Human approval obtained.
977
+
978
+ Never claim that data architecture is correct merely because the application appears to work.
979
+
980
+ ## 36. Final Data Principles
981
+
982
+ Protect data integrity.
983
+ Enforce least privilege.
984
+ Preserve one authoritative source of truth.
985
+ Validate access at trusted boundaries.
986
+ Use transactions where atomicity matters.
987
+ Design for concurrency and retries.
988
+ Measure query performance.
989
+ Scale based on evidence.
990
+ Control database and storage costs.
991
+ Protect backups and recovery.
992
+ Preserve production compatibility.
993
+ Never change production data without approval.