@wazir-dev/cli 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +111 -0
- package/CHANGELOG.md +14 -0
- package/CONTRIBUTING.md +101 -0
- package/LICENSE +21 -0
- package/README.md +314 -0
- package/assets/composition-engine.mmd +34 -0
- package/assets/demo-script.sh +17 -0
- package/assets/logo-dark.svg +14 -0
- package/assets/logo.svg +14 -0
- package/assets/pipeline.mmd +39 -0
- package/assets/record-demo.sh +51 -0
- package/docs/README.md +51 -0
- package/docs/adapters/context-mode.md +60 -0
- package/docs/concepts/architecture.md +87 -0
- package/docs/concepts/artifact-model.md +60 -0
- package/docs/concepts/composition-engine.md +36 -0
- package/docs/concepts/indexing-and-recall.md +160 -0
- package/docs/concepts/observability.md +41 -0
- package/docs/concepts/roles-and-workflows.md +59 -0
- package/docs/concepts/terminology-policy.md +27 -0
- package/docs/getting-started/01-installation.md +78 -0
- package/docs/getting-started/02-first-run.md +102 -0
- package/docs/getting-started/03-adding-to-project.md +15 -0
- package/docs/getting-started/04-host-setup.md +15 -0
- package/docs/guides/ci-integration.md +15 -0
- package/docs/guides/creating-skills.md +15 -0
- package/docs/guides/expertise-module-authoring.md +15 -0
- package/docs/guides/hook-development.md +15 -0
- package/docs/guides/memory-and-learnings.md +34 -0
- package/docs/guides/multi-host-export.md +15 -0
- package/docs/guides/troubleshooting.md +101 -0
- package/docs/guides/writing-custom-roles.md +15 -0
- package/docs/plans/2026-03-15-cli-pipeline-integration-design.md +592 -0
- package/docs/plans/2026-03-15-cli-pipeline-integration-plan.md +598 -0
- package/docs/plans/2026-03-15-docs-enforcement-plan.md +238 -0
- package/docs/readmes/INDEX.md +99 -0
- package/docs/readmes/features/expertise/README.md +171 -0
- package/docs/readmes/features/exports/README.md +222 -0
- package/docs/readmes/features/hooks/README.md +103 -0
- package/docs/readmes/features/hooks/loop-cap-guard.md +133 -0
- package/docs/readmes/features/hooks/post-tool-capture.md +121 -0
- package/docs/readmes/features/hooks/post-tool-lint.md +130 -0
- package/docs/readmes/features/hooks/pre-compact-summary.md +122 -0
- package/docs/readmes/features/hooks/pre-tool-capture-route.md +100 -0
- package/docs/readmes/features/hooks/protected-path-write-guard.md +128 -0
- package/docs/readmes/features/hooks/session-start.md +119 -0
- package/docs/readmes/features/hooks/stop-handoff-harvest.md +125 -0
- package/docs/readmes/features/roles/README.md +157 -0
- package/docs/readmes/features/roles/clarifier.md +152 -0
- package/docs/readmes/features/roles/content-author.md +190 -0
- package/docs/readmes/features/roles/designer.md +193 -0
- package/docs/readmes/features/roles/executor.md +184 -0
- package/docs/readmes/features/roles/learner.md +210 -0
- package/docs/readmes/features/roles/planner.md +182 -0
- package/docs/readmes/features/roles/researcher.md +164 -0
- package/docs/readmes/features/roles/reviewer.md +184 -0
- package/docs/readmes/features/roles/specifier.md +162 -0
- package/docs/readmes/features/roles/verifier.md +215 -0
- package/docs/readmes/features/schemas/README.md +178 -0
- package/docs/readmes/features/skills/README.md +63 -0
- package/docs/readmes/features/skills/brainstorming.md +96 -0
- package/docs/readmes/features/skills/debugging.md +148 -0
- package/docs/readmes/features/skills/design.md +120 -0
- package/docs/readmes/features/skills/prepare-next.md +109 -0
- package/docs/readmes/features/skills/run-audit.md +159 -0
- package/docs/readmes/features/skills/scan-project.md +109 -0
- package/docs/readmes/features/skills/self-audit.md +176 -0
- package/docs/readmes/features/skills/tdd.md +137 -0
- package/docs/readmes/features/skills/using-skills.md +92 -0
- package/docs/readmes/features/skills/verification.md +120 -0
- package/docs/readmes/features/skills/writing-plans.md +104 -0
- package/docs/readmes/features/tooling/README.md +320 -0
- package/docs/readmes/features/workflows/README.md +186 -0
- package/docs/readmes/features/workflows/author.md +181 -0
- package/docs/readmes/features/workflows/clarify.md +154 -0
- package/docs/readmes/features/workflows/design-review.md +171 -0
- package/docs/readmes/features/workflows/design.md +169 -0
- package/docs/readmes/features/workflows/discover.md +162 -0
- package/docs/readmes/features/workflows/execute.md +173 -0
- package/docs/readmes/features/workflows/learn.md +167 -0
- package/docs/readmes/features/workflows/plan-review.md +165 -0
- package/docs/readmes/features/workflows/plan.md +170 -0
- package/docs/readmes/features/workflows/prepare-next.md +167 -0
- package/docs/readmes/features/workflows/review.md +169 -0
- package/docs/readmes/features/workflows/run-audit.md +191 -0
- package/docs/readmes/features/workflows/spec-challenge.md +159 -0
- package/docs/readmes/features/workflows/specify.md +160 -0
- package/docs/readmes/features/workflows/verify.md +177 -0
- package/docs/readmes/packages/README.md +50 -0
- package/docs/readmes/packages/ajv.md +117 -0
- package/docs/readmes/packages/context-mode.md +118 -0
- package/docs/readmes/packages/gray-matter.md +116 -0
- package/docs/readmes/packages/node-test.md +137 -0
- package/docs/readmes/packages/yaml.md +112 -0
- package/docs/reference/configuration-reference.md +159 -0
- package/docs/reference/expertise-index.md +52 -0
- package/docs/reference/git-flow.md +43 -0
- package/docs/reference/hooks.md +87 -0
- package/docs/reference/host-exports.md +50 -0
- package/docs/reference/launch-checklist.md +172 -0
- package/docs/reference/marketplace-listings.md +76 -0
- package/docs/reference/release-process.md +34 -0
- package/docs/reference/roles-reference.md +77 -0
- package/docs/reference/skills.md +33 -0
- package/docs/reference/templates.md +29 -0
- package/docs/reference/tooling-cli.md +94 -0
- package/docs/truth-claims.yaml +222 -0
- package/expertise/PROGRESS.md +63 -0
- package/expertise/README.md +18 -0
- package/expertise/antipatterns/PROGRESS.md +56 -0
- package/expertise/antipatterns/backend/api-design-antipatterns.md +1271 -0
- package/expertise/antipatterns/backend/auth-antipatterns.md +1195 -0
- package/expertise/antipatterns/backend/caching-antipatterns.md +622 -0
- package/expertise/antipatterns/backend/database-antipatterns.md +1038 -0
- package/expertise/antipatterns/backend/index.md +24 -0
- package/expertise/antipatterns/backend/microservices-antipatterns.md +850 -0
- package/expertise/antipatterns/code/architecture-antipatterns.md +919 -0
- package/expertise/antipatterns/code/async-antipatterns.md +622 -0
- package/expertise/antipatterns/code/code-smells.md +1186 -0
- package/expertise/antipatterns/code/dependency-antipatterns.md +1209 -0
- package/expertise/antipatterns/code/error-handling-antipatterns.md +1360 -0
- package/expertise/antipatterns/code/index.md +27 -0
- package/expertise/antipatterns/code/naming-and-abstraction.md +1118 -0
- package/expertise/antipatterns/code/state-management-antipatterns.md +1076 -0
- package/expertise/antipatterns/code/testing-antipatterns.md +1053 -0
- package/expertise/antipatterns/design/accessibility-antipatterns.md +1136 -0
- package/expertise/antipatterns/design/dark-patterns.md +1121 -0
- package/expertise/antipatterns/design/index.md +22 -0
- package/expertise/antipatterns/design/ui-antipatterns.md +1202 -0
- package/expertise/antipatterns/design/ux-antipatterns.md +680 -0
- package/expertise/antipatterns/frontend/css-layout-antipatterns.md +691 -0
- package/expertise/antipatterns/frontend/flutter-antipatterns.md +1827 -0
- package/expertise/antipatterns/frontend/index.md +23 -0
- package/expertise/antipatterns/frontend/mobile-antipatterns.md +573 -0
- package/expertise/antipatterns/frontend/react-antipatterns.md +1128 -0
- package/expertise/antipatterns/frontend/spa-antipatterns.md +1235 -0
- package/expertise/antipatterns/index.md +31 -0
- package/expertise/antipatterns/performance/index.md +20 -0
- package/expertise/antipatterns/performance/performance-antipatterns.md +1013 -0
- package/expertise/antipatterns/performance/premature-optimization.md +623 -0
- package/expertise/antipatterns/performance/scaling-antipatterns.md +785 -0
- package/expertise/antipatterns/process/ai-coding-antipatterns.md +853 -0
- package/expertise/antipatterns/process/code-review-antipatterns.md +656 -0
- package/expertise/antipatterns/process/deployment-antipatterns.md +920 -0
- package/expertise/antipatterns/process/index.md +23 -0
- package/expertise/antipatterns/process/technical-debt-antipatterns.md +647 -0
- package/expertise/antipatterns/security/index.md +20 -0
- package/expertise/antipatterns/security/secrets-antipatterns.md +849 -0
- package/expertise/antipatterns/security/security-theater.md +843 -0
- package/expertise/antipatterns/security/vulnerability-patterns.md +801 -0
- package/expertise/architecture/PROGRESS.md +70 -0
- package/expertise/architecture/data/caching-architecture.md +671 -0
- package/expertise/architecture/data/data-consistency.md +574 -0
- package/expertise/architecture/data/data-modeling.md +536 -0
- package/expertise/architecture/data/event-streams-and-queues.md +634 -0
- package/expertise/architecture/data/index.md +25 -0
- package/expertise/architecture/data/search-architecture.md +663 -0
- package/expertise/architecture/data/sql-vs-nosql.md +708 -0
- package/expertise/architecture/decisions/architecture-decision-records.md +640 -0
- package/expertise/architecture/decisions/build-vs-buy.md +616 -0
- package/expertise/architecture/decisions/index.md +23 -0
- package/expertise/architecture/decisions/monolith-to-microservices.md +790 -0
- package/expertise/architecture/decisions/technology-selection.md +616 -0
- package/expertise/architecture/distributed/cap-theorem-and-tradeoffs.md +800 -0
- package/expertise/architecture/distributed/circuit-breaker-bulkhead.md +741 -0
- package/expertise/architecture/distributed/consensus-and-coordination.md +796 -0
- package/expertise/architecture/distributed/distributed-systems-fundamentals.md +564 -0
- package/expertise/architecture/distributed/idempotency-and-retry.md +796 -0
- package/expertise/architecture/distributed/index.md +25 -0
- package/expertise/architecture/distributed/saga-pattern.md +797 -0
- package/expertise/architecture/foundations/architectural-thinking.md +460 -0
- package/expertise/architecture/foundations/coupling-and-cohesion.md +770 -0
- package/expertise/architecture/foundations/design-principles-solid.md +649 -0
- package/expertise/architecture/foundations/domain-driven-design.md +719 -0
- package/expertise/architecture/foundations/index.md +25 -0
- package/expertise/architecture/foundations/separation-of-concerns.md +472 -0
- package/expertise/architecture/foundations/twelve-factor-app.md +797 -0
- package/expertise/architecture/index.md +34 -0
- package/expertise/architecture/integration/api-design-graphql.md +638 -0
- package/expertise/architecture/integration/api-design-grpc.md +804 -0
- package/expertise/architecture/integration/api-design-rest.md +892 -0
- package/expertise/architecture/integration/index.md +25 -0
- package/expertise/architecture/integration/third-party-integration.md +795 -0
- package/expertise/architecture/integration/webhooks-and-callbacks.md +1152 -0
- package/expertise/architecture/integration/websockets-realtime.md +791 -0
- package/expertise/architecture/mobile-architecture/index.md +22 -0
- package/expertise/architecture/mobile-architecture/mobile-app-architecture.md +780 -0
- package/expertise/architecture/mobile-architecture/mobile-backend-for-frontend.md +670 -0
- package/expertise/architecture/mobile-architecture/offline-first.md +719 -0
- package/expertise/architecture/mobile-architecture/push-and-sync.md +782 -0
- package/expertise/architecture/patterns/cqrs-event-sourcing.md +717 -0
- package/expertise/architecture/patterns/event-driven.md +797 -0
- package/expertise/architecture/patterns/hexagonal-clean-architecture.md +870 -0
- package/expertise/architecture/patterns/index.md +27 -0
- package/expertise/architecture/patterns/layered-architecture.md +736 -0
- package/expertise/architecture/patterns/microservices.md +753 -0
- package/expertise/architecture/patterns/modular-monolith.md +692 -0
- package/expertise/architecture/patterns/monolith.md +626 -0
- package/expertise/architecture/patterns/plugin-architecture.md +735 -0
- package/expertise/architecture/patterns/serverless.md +780 -0
- package/expertise/architecture/scaling/database-scaling.md +615 -0
- package/expertise/architecture/scaling/feature-flags-and-rollouts.md +757 -0
- package/expertise/architecture/scaling/horizontal-vs-vertical.md +606 -0
- package/expertise/architecture/scaling/index.md +24 -0
- package/expertise/architecture/scaling/multi-tenancy.md +800 -0
- package/expertise/architecture/scaling/stateless-design.md +787 -0
- package/expertise/backend/embedded-firmware.md +625 -0
- package/expertise/backend/go.md +853 -0
- package/expertise/backend/index.md +24 -0
- package/expertise/backend/java-spring.md +448 -0
- package/expertise/backend/node-typescript.md +625 -0
- package/expertise/backend/python-fastapi.md +724 -0
- package/expertise/backend/rust.md +458 -0
- package/expertise/backend/solidity.md +711 -0
- package/expertise/composition-map.yaml +443 -0
- package/expertise/content/foundations/content-modeling.md +395 -0
- package/expertise/content/foundations/editorial-standards.md +449 -0
- package/expertise/content/foundations/index.md +24 -0
- package/expertise/content/foundations/microcopy.md +455 -0
- package/expertise/content/foundations/terminology-governance.md +509 -0
- package/expertise/content/index.md +34 -0
- package/expertise/content/patterns/accessibility-copy.md +518 -0
- package/expertise/content/patterns/index.md +24 -0
- package/expertise/content/patterns/notification-content.md +433 -0
- package/expertise/content/patterns/sample-content.md +486 -0
- package/expertise/content/patterns/state-copy.md +439 -0
- package/expertise/design/PROGRESS.md +58 -0
- package/expertise/design/disciplines/dark-mode-theming.md +577 -0
- package/expertise/design/disciplines/design-systems.md +595 -0
- package/expertise/design/disciplines/index.md +25 -0
- package/expertise/design/disciplines/information-architecture.md +800 -0
- package/expertise/design/disciplines/interaction-design.md +788 -0
- package/expertise/design/disciplines/responsive-design.md +552 -0
- package/expertise/design/disciplines/usability-testing.md +516 -0
- package/expertise/design/disciplines/user-research.md +792 -0
- package/expertise/design/foundations/accessibility-design.md +796 -0
- package/expertise/design/foundations/color-theory.md +797 -0
- package/expertise/design/foundations/iconography.md +795 -0
- package/expertise/design/foundations/index.md +26 -0
- package/expertise/design/foundations/motion-and-animation.md +653 -0
- package/expertise/design/foundations/rtl-design.md +585 -0
- package/expertise/design/foundations/spacing-and-layout.md +607 -0
- package/expertise/design/foundations/typography.md +800 -0
- package/expertise/design/foundations/visual-hierarchy.md +761 -0
- package/expertise/design/index.md +32 -0
- package/expertise/design/patterns/authentication-flows.md +474 -0
- package/expertise/design/patterns/content-consumption.md +789 -0
- package/expertise/design/patterns/data-display.md +618 -0
- package/expertise/design/patterns/e-commerce.md +1494 -0
- package/expertise/design/patterns/feedback-and-states.md +642 -0
- package/expertise/design/patterns/forms-and-input.md +819 -0
- package/expertise/design/patterns/gamification.md +801 -0
- package/expertise/design/patterns/index.md +31 -0
- package/expertise/design/patterns/microinteractions.md +449 -0
- package/expertise/design/patterns/navigation.md +800 -0
- package/expertise/design/patterns/notifications.md +705 -0
- package/expertise/design/patterns/onboarding.md +700 -0
- package/expertise/design/patterns/search-and-filter.md +601 -0
- package/expertise/design/patterns/settings-and-preferences.md +768 -0
- package/expertise/design/patterns/social-and-community.md +748 -0
- package/expertise/design/platforms/desktop-native.md +612 -0
- package/expertise/design/platforms/index.md +25 -0
- package/expertise/design/platforms/mobile-android.md +825 -0
- package/expertise/design/platforms/mobile-cross-platform.md +983 -0
- package/expertise/design/platforms/mobile-ios.md +699 -0
- package/expertise/design/platforms/tablet.md +794 -0
- package/expertise/design/platforms/web-dashboard.md +790 -0
- package/expertise/design/platforms/web-responsive.md +550 -0
- package/expertise/design/psychology/behavioral-nudges.md +449 -0
- package/expertise/design/psychology/cognitive-load.md +1191 -0
- package/expertise/design/psychology/error-psychology.md +778 -0
- package/expertise/design/psychology/index.md +22 -0
- package/expertise/design/psychology/persuasive-design.md +736 -0
- package/expertise/design/psychology/user-mental-models.md +623 -0
- package/expertise/design/tooling/open-pencil.md +266 -0
- package/expertise/frontend/angular.md +1073 -0
- package/expertise/frontend/desktop-electron.md +546 -0
- package/expertise/frontend/flutter.md +782 -0
- package/expertise/frontend/index.md +27 -0
- package/expertise/frontend/native-android.md +409 -0
- package/expertise/frontend/native-ios.md +490 -0
- package/expertise/frontend/react-native.md +1160 -0
- package/expertise/frontend/react.md +808 -0
- package/expertise/frontend/vue.md +1089 -0
- package/expertise/humanize/domain-rules-code.md +79 -0
- package/expertise/humanize/domain-rules-content.md +67 -0
- package/expertise/humanize/domain-rules-technical-docs.md +56 -0
- package/expertise/humanize/index.md +35 -0
- package/expertise/humanize/self-audit-checklist.md +87 -0
- package/expertise/humanize/sentence-patterns.md +218 -0
- package/expertise/humanize/vocabulary-blacklist.md +105 -0
- package/expertise/i18n/PROGRESS.md +65 -0
- package/expertise/i18n/advanced/accessibility-and-i18n.md +28 -0
- package/expertise/i18n/advanced/bidirectional-text-algorithm.md +38 -0
- package/expertise/i18n/advanced/complex-scripts.md +30 -0
- package/expertise/i18n/advanced/performance-and-i18n.md +27 -0
- package/expertise/i18n/advanced/testing-i18n.md +28 -0
- package/expertise/i18n/content/content-adaptation.md +23 -0
- package/expertise/i18n/content/locale-specific-formatting.md +23 -0
- package/expertise/i18n/content/machine-translation-integration.md +28 -0
- package/expertise/i18n/content/translation-management.md +29 -0
- package/expertise/i18n/foundations/date-time-calendars.md +67 -0
- package/expertise/i18n/foundations/i18n-architecture.md +272 -0
- package/expertise/i18n/foundations/locale-and-language-tags.md +79 -0
- package/expertise/i18n/foundations/numbers-currency-units.md +61 -0
- package/expertise/i18n/foundations/pluralization-and-gender.md +109 -0
- package/expertise/i18n/foundations/string-externalization.md +236 -0
- package/expertise/i18n/foundations/text-direction-bidi.md +241 -0
- package/expertise/i18n/foundations/unicode-and-encoding.md +86 -0
- package/expertise/i18n/index.md +38 -0
- package/expertise/i18n/platform/backend-i18n.md +31 -0
- package/expertise/i18n/platform/flutter-i18n.md +148 -0
- package/expertise/i18n/platform/native-android-i18n.md +36 -0
- package/expertise/i18n/platform/native-ios-i18n.md +36 -0
- package/expertise/i18n/platform/react-i18n.md +103 -0
- package/expertise/i18n/platform/web-css-i18n.md +81 -0
- package/expertise/i18n/rtl/arabic-specific.md +175 -0
- package/expertise/i18n/rtl/hebrew-specific.md +149 -0
- package/expertise/i18n/rtl/rtl-animations-and-transitions.md +111 -0
- package/expertise/i18n/rtl/rtl-forms-and-input.md +161 -0
- package/expertise/i18n/rtl/rtl-fundamentals.md +211 -0
- package/expertise/i18n/rtl/rtl-icons-and-images.md +181 -0
- package/expertise/i18n/rtl/rtl-layout-mirroring.md +252 -0
- package/expertise/i18n/rtl/rtl-navigation-and-gestures.md +107 -0
- package/expertise/i18n/rtl/rtl-testing-and-qa.md +147 -0
- package/expertise/i18n/rtl/rtl-typography.md +160 -0
- package/expertise/index.md +113 -0
- package/expertise/index.yaml +216 -0
- package/expertise/infrastructure/cloud-aws.md +597 -0
- package/expertise/infrastructure/cloud-gcp.md +599 -0
- package/expertise/infrastructure/cybersecurity.md +816 -0
- package/expertise/infrastructure/database-mongodb.md +447 -0
- package/expertise/infrastructure/database-postgres.md +400 -0
- package/expertise/infrastructure/devops-cicd.md +787 -0
- package/expertise/infrastructure/index.md +27 -0
- package/expertise/performance/PROGRESS.md +50 -0
- package/expertise/performance/backend/api-latency.md +1204 -0
- package/expertise/performance/backend/background-jobs.md +506 -0
- package/expertise/performance/backend/connection-pooling.md +1209 -0
- package/expertise/performance/backend/database-query-optimization.md +515 -0
- package/expertise/performance/backend/index.md +23 -0
- package/expertise/performance/backend/rate-limiting-and-throttling.md +971 -0
- package/expertise/performance/foundations/algorithmic-complexity.md +954 -0
- package/expertise/performance/foundations/caching-strategies.md +489 -0
- package/expertise/performance/foundations/concurrency-and-parallelism.md +847 -0
- package/expertise/performance/foundations/index.md +24 -0
- package/expertise/performance/foundations/measuring-and-profiling.md +440 -0
- package/expertise/performance/foundations/memory-management.md +964 -0
- package/expertise/performance/foundations/performance-budgets.md +1314 -0
- package/expertise/performance/index.md +31 -0
- package/expertise/performance/infrastructure/auto-scaling.md +1059 -0
- package/expertise/performance/infrastructure/cdn-and-edge.md +1081 -0
- package/expertise/performance/infrastructure/index.md +22 -0
- package/expertise/performance/infrastructure/load-balancing.md +1081 -0
- package/expertise/performance/infrastructure/observability.md +1079 -0
- package/expertise/performance/mobile/index.md +23 -0
- package/expertise/performance/mobile/mobile-animations.md +544 -0
- package/expertise/performance/mobile/mobile-memory-battery.md +416 -0
- package/expertise/performance/mobile/mobile-network.md +452 -0
- package/expertise/performance/mobile/mobile-rendering.md +599 -0
- package/expertise/performance/mobile/mobile-startup-time.md +505 -0
- package/expertise/performance/platform-specific/flutter-performance.md +647 -0
- package/expertise/performance/platform-specific/index.md +22 -0
- package/expertise/performance/platform-specific/node-performance.md +1307 -0
- package/expertise/performance/platform-specific/postgres-performance.md +1366 -0
- package/expertise/performance/platform-specific/react-performance.md +1403 -0
- package/expertise/performance/web/bundle-optimization.md +1239 -0
- package/expertise/performance/web/image-and-media.md +636 -0
- package/expertise/performance/web/index.md +24 -0
- package/expertise/performance/web/network-optimization.md +1133 -0
- package/expertise/performance/web/rendering-performance.md +1098 -0
- package/expertise/performance/web/ssr-and-hydration.md +918 -0
- package/expertise/performance/web/web-vitals.md +1374 -0
- package/expertise/quality/accessibility.md +985 -0
- package/expertise/quality/evidence-based-verification.md +499 -0
- package/expertise/quality/index.md +24 -0
- package/expertise/quality/ml-model-audit.md +614 -0
- package/expertise/quality/performance.md +600 -0
- package/expertise/quality/testing-api.md +891 -0
- package/expertise/quality/testing-mobile.md +496 -0
- package/expertise/quality/testing-web.md +849 -0
- package/expertise/security/PROGRESS.md +54 -0
- package/expertise/security/agentic-identity.md +540 -0
- package/expertise/security/compliance-frameworks.md +601 -0
- package/expertise/security/data/data-encryption.md +364 -0
- package/expertise/security/data/data-privacy-gdpr.md +692 -0
- package/expertise/security/data/database-security.md +1171 -0
- package/expertise/security/data/index.md +22 -0
- package/expertise/security/data/pii-handling.md +531 -0
- package/expertise/security/foundations/authentication.md +1041 -0
- package/expertise/security/foundations/authorization.md +603 -0
- package/expertise/security/foundations/cryptography.md +1001 -0
- package/expertise/security/foundations/index.md +25 -0
- package/expertise/security/foundations/owasp-top-10.md +1354 -0
- package/expertise/security/foundations/secrets-management.md +1217 -0
- package/expertise/security/foundations/secure-sdlc.md +700 -0
- package/expertise/security/foundations/supply-chain-security.md +698 -0
- package/expertise/security/index.md +31 -0
- package/expertise/security/infrastructure/cloud-security-aws.md +1296 -0
- package/expertise/security/infrastructure/cloud-security-gcp.md +1376 -0
- package/expertise/security/infrastructure/container-security.md +721 -0
- package/expertise/security/infrastructure/incident-response.md +1295 -0
- package/expertise/security/infrastructure/index.md +24 -0
- package/expertise/security/infrastructure/logging-and-monitoring.md +1618 -0
- package/expertise/security/infrastructure/network-security.md +1337 -0
- package/expertise/security/mobile/index.md +23 -0
- package/expertise/security/mobile/mobile-android-security.md +1218 -0
- package/expertise/security/mobile/mobile-binary-protection.md +1229 -0
- package/expertise/security/mobile/mobile-data-storage.md +1265 -0
- package/expertise/security/mobile/mobile-ios-security.md +1401 -0
- package/expertise/security/mobile/mobile-network-security.md +1520 -0
- package/expertise/security/smart-contract-security.md +594 -0
- package/expertise/security/testing/index.md +22 -0
- package/expertise/security/testing/penetration-testing.md +1258 -0
- package/expertise/security/testing/security-code-review.md +1765 -0
- package/expertise/security/testing/threat-modeling.md +1074 -0
- package/expertise/security/testing/vulnerability-scanning.md +1062 -0
- package/expertise/security/web/api-security.md +586 -0
- package/expertise/security/web/cors-and-headers.md +433 -0
- package/expertise/security/web/csrf.md +562 -0
- package/expertise/security/web/file-upload.md +1477 -0
- package/expertise/security/web/index.md +25 -0
- package/expertise/security/web/injection.md +1375 -0
- package/expertise/security/web/session-management.md +1101 -0
- package/expertise/security/web/xss.md +1158 -0
- package/exports/README.md +17 -0
- package/exports/hosts/claude/.claude/agents/clarifier.md +42 -0
- package/exports/hosts/claude/.claude/agents/content-author.md +63 -0
- package/exports/hosts/claude/.claude/agents/designer.md +55 -0
- package/exports/hosts/claude/.claude/agents/executor.md +55 -0
- package/exports/hosts/claude/.claude/agents/learner.md +51 -0
- package/exports/hosts/claude/.claude/agents/planner.md +53 -0
- package/exports/hosts/claude/.claude/agents/researcher.md +43 -0
- package/exports/hosts/claude/.claude/agents/reviewer.md +54 -0
- package/exports/hosts/claude/.claude/agents/specifier.md +47 -0
- package/exports/hosts/claude/.claude/agents/verifier.md +71 -0
- package/exports/hosts/claude/.claude/commands/author.md +42 -0
- package/exports/hosts/claude/.claude/commands/clarify.md +38 -0
- package/exports/hosts/claude/.claude/commands/design-review.md +46 -0
- package/exports/hosts/claude/.claude/commands/design.md +44 -0
- package/exports/hosts/claude/.claude/commands/discover.md +37 -0
- package/exports/hosts/claude/.claude/commands/execute.md +48 -0
- package/exports/hosts/claude/.claude/commands/learn.md +38 -0
- package/exports/hosts/claude/.claude/commands/plan-review.md +42 -0
- package/exports/hosts/claude/.claude/commands/plan.md +39 -0
- package/exports/hosts/claude/.claude/commands/prepare-next.md +37 -0
- package/exports/hosts/claude/.claude/commands/review.md +40 -0
- package/exports/hosts/claude/.claude/commands/run-audit.md +41 -0
- package/exports/hosts/claude/.claude/commands/spec-challenge.md +41 -0
- package/exports/hosts/claude/.claude/commands/specify.md +38 -0
- package/exports/hosts/claude/.claude/commands/verify.md +37 -0
- package/exports/hosts/claude/.claude/settings.json +34 -0
- package/exports/hosts/claude/CLAUDE.md +19 -0
- package/exports/hosts/claude/export.manifest.json +38 -0
- package/exports/hosts/claude/host-package.json +67 -0
- package/exports/hosts/codex/AGENTS.md +19 -0
- package/exports/hosts/codex/export.manifest.json +38 -0
- package/exports/hosts/codex/host-package.json +41 -0
- package/exports/hosts/cursor/.cursor/hooks.json +16 -0
- package/exports/hosts/cursor/.cursor/rules/wazir-core.mdc +19 -0
- package/exports/hosts/cursor/export.manifest.json +38 -0
- package/exports/hosts/cursor/host-package.json +42 -0
- package/exports/hosts/gemini/GEMINI.md +19 -0
- package/exports/hosts/gemini/export.manifest.json +38 -0
- package/exports/hosts/gemini/host-package.json +41 -0
- package/hooks/README.md +18 -0
- package/hooks/definitions/loop_cap_guard.yaml +21 -0
- package/hooks/definitions/post_tool_capture.yaml +24 -0
- package/hooks/definitions/pre_compact_summary.yaml +19 -0
- package/hooks/definitions/pre_tool_capture_route.yaml +19 -0
- package/hooks/definitions/protected_path_write_guard.yaml +19 -0
- package/hooks/definitions/session_start.yaml +19 -0
- package/hooks/definitions/stop_handoff_harvest.yaml +20 -0
- package/hooks/loop-cap-guard +17 -0
- package/hooks/post-tool-lint +36 -0
- package/hooks/protected-path-write-guard +17 -0
- package/hooks/session-start +41 -0
- package/llms-full.txt +2355 -0
- package/llms.txt +43 -0
- package/package.json +79 -0
- package/roles/README.md +20 -0
- package/roles/clarifier.md +42 -0
- package/roles/content-author.md +63 -0
- package/roles/designer.md +55 -0
- package/roles/executor.md +55 -0
- package/roles/learner.md +51 -0
- package/roles/planner.md +53 -0
- package/roles/researcher.md +43 -0
- package/roles/reviewer.md +54 -0
- package/roles/specifier.md +47 -0
- package/roles/verifier.md +71 -0
- package/schemas/README.md +24 -0
- package/schemas/accepted-learning.schema.json +20 -0
- package/schemas/author-artifact.schema.json +156 -0
- package/schemas/clarification.schema.json +19 -0
- package/schemas/design-artifact.schema.json +80 -0
- package/schemas/docs-claim.schema.json +18 -0
- package/schemas/export-manifest.schema.json +20 -0
- package/schemas/hook.schema.json +67 -0
- package/schemas/host-export-package.schema.json +18 -0
- package/schemas/implementation-plan.schema.json +19 -0
- package/schemas/proposed-learning.schema.json +19 -0
- package/schemas/research.schema.json +18 -0
- package/schemas/review.schema.json +29 -0
- package/schemas/run-manifest.schema.json +18 -0
- package/schemas/spec-challenge.schema.json +18 -0
- package/schemas/spec.schema.json +20 -0
- package/schemas/usage.schema.json +102 -0
- package/schemas/verification-proof.schema.json +29 -0
- package/schemas/wazir-manifest.schema.json +173 -0
- package/skills/README.md +40 -0
- package/skills/brainstorming/SKILL.md +77 -0
- package/skills/debugging/SKILL.md +50 -0
- package/skills/design/SKILL.md +61 -0
- package/skills/dispatching-parallel-agents/SKILL.md +128 -0
- package/skills/executing-plans/SKILL.md +70 -0
- package/skills/finishing-a-development-branch/SKILL.md +169 -0
- package/skills/humanize/SKILL.md +123 -0
- package/skills/init-pipeline/SKILL.md +124 -0
- package/skills/prepare-next/SKILL.md +20 -0
- package/skills/receiving-code-review/SKILL.md +123 -0
- package/skills/requesting-code-review/SKILL.md +105 -0
- package/skills/requesting-code-review/code-reviewer.md +108 -0
- package/skills/run-audit/SKILL.md +197 -0
- package/skills/scan-project/SKILL.md +41 -0
- package/skills/self-audit/SKILL.md +153 -0
- package/skills/subagent-driven-development/SKILL.md +154 -0
- package/skills/subagent-driven-development/code-quality-reviewer-prompt.md +26 -0
- package/skills/subagent-driven-development/implementer-prompt.md +102 -0
- package/skills/subagent-driven-development/spec-reviewer-prompt.md +61 -0
- package/skills/tdd/SKILL.md +23 -0
- package/skills/using-git-worktrees/SKILL.md +163 -0
- package/skills/using-skills/SKILL.md +95 -0
- package/skills/verification/SKILL.md +22 -0
- package/skills/wazir/SKILL.md +463 -0
- package/skills/writing-plans/SKILL.md +30 -0
- package/skills/writing-skills/SKILL.md +157 -0
- package/skills/writing-skills/anthropic-best-practices.md +122 -0
- package/skills/writing-skills/persuasion-principles.md +50 -0
- package/templates/README.md +20 -0
- package/templates/artifacts/README.md +10 -0
- package/templates/artifacts/accepted-learning.md +19 -0
- package/templates/artifacts/accepted-learning.template.json +12 -0
- package/templates/artifacts/author.md +74 -0
- package/templates/artifacts/author.template.json +19 -0
- package/templates/artifacts/clarification.md +21 -0
- package/templates/artifacts/clarification.template.json +12 -0
- package/templates/artifacts/execute-notes.md +19 -0
- package/templates/artifacts/implementation-plan.md +21 -0
- package/templates/artifacts/implementation-plan.template.json +11 -0
- package/templates/artifacts/learning-proposal.md +19 -0
- package/templates/artifacts/next-run-handoff.md +21 -0
- package/templates/artifacts/plan-review.md +19 -0
- package/templates/artifacts/proposed-learning.template.json +12 -0
- package/templates/artifacts/research.md +21 -0
- package/templates/artifacts/research.template.json +12 -0
- package/templates/artifacts/review-findings.md +19 -0
- package/templates/artifacts/review.template.json +11 -0
- package/templates/artifacts/run-manifest.template.json +8 -0
- package/templates/artifacts/spec-challenge.md +19 -0
- package/templates/artifacts/spec-challenge.template.json +11 -0
- package/templates/artifacts/spec.md +21 -0
- package/templates/artifacts/spec.template.json +12 -0
- package/templates/artifacts/verification-proof.md +19 -0
- package/templates/artifacts/verification-proof.template.json +11 -0
- package/templates/examples/accepted-learning.example.json +14 -0
- package/templates/examples/author.example.json +152 -0
- package/templates/examples/clarification.example.json +15 -0
- package/templates/examples/docs-claim.example.json +8 -0
- package/templates/examples/export-manifest.example.json +7 -0
- package/templates/examples/host-export-package.example.json +11 -0
- package/templates/examples/implementation-plan.example.json +17 -0
- package/templates/examples/proposed-learning.example.json +13 -0
- package/templates/examples/research.example.json +15 -0
- package/templates/examples/research.example.md +6 -0
- package/templates/examples/review.example.json +17 -0
- package/templates/examples/run-manifest.example.json +9 -0
- package/templates/examples/spec-challenge.example.json +14 -0
- package/templates/examples/spec.example.json +21 -0
- package/templates/examples/verification-proof.example.json +21 -0
- package/templates/examples/wazir-manifest.example.yaml +65 -0
- package/templates/task-definition-schema.md +99 -0
- package/tooling/README.md +20 -0
- package/tooling/src/adapters/context-mode.js +50 -0
- package/tooling/src/capture/command.js +376 -0
- package/tooling/src/capture/store.js +99 -0
- package/tooling/src/capture/usage.js +270 -0
- package/tooling/src/checks/branches.js +50 -0
- package/tooling/src/checks/brand-truth.js +110 -0
- package/tooling/src/checks/changelog.js +231 -0
- package/tooling/src/checks/command-registry.js +36 -0
- package/tooling/src/checks/commits.js +102 -0
- package/tooling/src/checks/docs-drift.js +103 -0
- package/tooling/src/checks/docs-truth.js +201 -0
- package/tooling/src/checks/runtime-surface.js +156 -0
- package/tooling/src/cli.js +116 -0
- package/tooling/src/command-options.js +56 -0
- package/tooling/src/commands/validate.js +320 -0
- package/tooling/src/doctor/command.js +91 -0
- package/tooling/src/export/command.js +77 -0
- package/tooling/src/export/compiler.js +498 -0
- package/tooling/src/guards/loop-cap-guard.js +52 -0
- package/tooling/src/guards/protected-path-write-guard.js +67 -0
- package/tooling/src/index/command.js +152 -0
- package/tooling/src/index/storage.js +1061 -0
- package/tooling/src/index/summarizers.js +261 -0
- package/tooling/src/loaders.js +18 -0
- package/tooling/src/project-root.js +22 -0
- package/tooling/src/recall/command.js +225 -0
- package/tooling/src/schema-validator.js +30 -0
- package/tooling/src/state-root.js +40 -0
- package/tooling/src/status/command.js +71 -0
- package/wazir.manifest.yaml +135 -0
- package/workflows/README.md +19 -0
- package/workflows/author.md +42 -0
- package/workflows/clarify.md +38 -0
- package/workflows/design-review.md +46 -0
- package/workflows/design.md +44 -0
- package/workflows/discover.md +37 -0
- package/workflows/execute.md +48 -0
- package/workflows/learn.md +38 -0
- package/workflows/plan-review.md +42 -0
- package/workflows/plan.md +39 -0
- package/workflows/prepare-next.md +37 -0
- package/workflows/review.md +40 -0
- package/workflows/run-audit.md +41 -0
- package/workflows/spec-challenge.md +41 -0
- package/workflows/specify.md +38 -0
- package/workflows/verify.md +37 -0
|
@@ -0,0 +1,708 @@
|
|
|
1
|
+
# SQL vs NoSQL — Architecture Expertise Module
|
|
2
|
+
|
|
3
|
+
> The SQL vs NoSQL decision is one of the most consequential and most frequently gotten wrong. PostgreSQL is the right default for 90% of applications. NoSQL databases solve specific problems (key-value caching, document flexibility, time-series, graph) but are often chosen for the wrong reasons — usually "we don't want to define a schema" or "NoSQL scales better," both of which are misleading at best and destructive at worst.
|
|
4
|
+
|
|
5
|
+
> **Category:** Data
|
|
6
|
+
> **Complexity:** Moderate
|
|
7
|
+
> **Applies when:** Choosing a primary data store for a new system or evaluating whether the current database choice is appropriate
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## What This Is
|
|
12
|
+
|
|
13
|
+
### The Database Family Tree
|
|
14
|
+
|
|
15
|
+
Databases are not a binary SQL-or-NoSQL choice. They are a family of specialized tools, each optimized for a specific access pattern. Understanding what each type actually is — not what marketing says it is — is the foundation for making the right choice.
|
|
16
|
+
|
|
17
|
+
**Relational (SQL) Databases** — Store data in tables with rows and columns. Relationships between tables are expressed through foreign keys and enforced by the database engine. Data is accessed through SQL, a declarative query language that has been the industry standard since the 1970s. The schema is defined before data is inserted, and the database enforces that schema on every write. Examples: PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, SQLite, CockroachDB.
|
|
18
|
+
|
|
19
|
+
The defining property of relational databases is not "tables" — it is **relational algebra**. The ability to JOIN data across tables, filter with arbitrary WHERE clauses, aggregate with GROUP BY, and compose these operations into complex queries is what makes relational databases uniquely powerful for ad-hoc analysis, reporting, and any use case where the question being asked of the data is not known at design time.
|
|
20
|
+
|
|
21
|
+
**Document Databases** — Store data as semi-structured documents, typically JSON or BSON. Each document is self-contained — it carries its data and its structure together. Documents in the same collection can have different fields. There is no enforced schema at the database level (though application-level schema validation exists). Examples: MongoDB, CouchDB, Amazon DocumentDB, Firestore.
|
|
22
|
+
|
|
23
|
+
The key insight: document databases trade query flexibility for write flexibility. You gain the ability to store heterogeneous data without migrations, but you lose the ability to efficiently query across documents in ways the original access pattern did not anticipate. Every query pattern must be designed into the data model upfront.
|
|
24
|
+
|
|
25
|
+
**Key-Value Stores** — The simplest data model: a key maps to a value. The database knows nothing about the value's structure — it is an opaque blob. Operations are limited to GET, PUT, DELETE by key. This extreme simplicity enables extreme performance — sub-millisecond latency at millions of operations per second. Examples: Redis, Memcached, Amazon DynamoDB (also supports document model), etcd, Riak.
|
|
26
|
+
|
|
27
|
+
Key-value stores are not general-purpose databases. They solve one problem: fast lookup by a known key. The moment you need to query by anything other than the key — filter by a field, search by a range, join two datasets — a key-value store is the wrong tool.
|
|
28
|
+
|
|
29
|
+
**Column-Family (Wide-Column) Databases** — Store data in rows and column families, where each row can have a different set of columns. Optimized for writing and reading large volumes of data across distributed clusters. Data is partitioned by a partition key and sorted within each partition by a clustering key. Designed for write-heavy workloads at massive scale. Examples: Apache Cassandra, ScyllaDB, HBase, Google Bigtable.
|
|
30
|
+
|
|
31
|
+
Column-family databases require you to model data around your query patterns, not around your entities. You will duplicate data across multiple tables, each designed to answer a specific query. This is the opposite of relational normalization — and it is intentional. The trade-off: write amplification and data duplication in exchange for predictable read performance at any scale.
|
|
32
|
+
|
|
33
|
+
**Graph Databases** — Store data as nodes (entities) and edges (relationships between entities). Optimized for traversing relationships — "find all friends of friends who live in city X and have purchased product Y" — queries that would require multiple self-joins in a relational database. Examples: Neo4j, Amazon Neptune, ArangoDB, JanusGraph, Dgraph.
|
|
34
|
+
|
|
35
|
+
Graph databases solve a specific problem: multi-hop relationship traversal. In a relational database, each additional "hop" requires another JOIN, and query performance degrades exponentially with depth. In a graph database, traversal depth has near-constant cost per hop. If your queries do not traverse relationships beyond 2-3 hops, a relational database with proper indexing handles graph-like queries perfectly well.
|
|
36
|
+
|
|
37
|
+
**Time-Series Databases** — Optimized for append-heavy workloads where data arrives as timestamped observations: metrics, sensor readings, financial ticks, application logs. They provide built-in time-window aggregation, downsampling, retention policies, and compression optimized for sequential time-ordered data. Examples: TimescaleDB, InfluxDB, QuestDB, Prometheus, ClickHouse (also OLAP), Amazon Timestream.
|
|
38
|
+
|
|
39
|
+
Time-series databases are not just "databases with a timestamp column." Their storage engines are physically optimized for time-ordered append patterns. They compress sequential data far more efficiently than general-purpose databases, and their query engines natively support operations like "average over 5-minute windows" that would require complex SQL expressions in a relational database.
|
|
40
|
+
|
|
41
|
+
**Search Engines** — Not databases in the traditional sense, but often used as one. Store data in inverted indices optimized for full-text search, fuzzy matching, faceted search, and relevance scoring. Examples: Elasticsearch, OpenSearch, Apache Solr, Meilisearch, Typesense.
|
|
42
|
+
|
|
43
|
+
Search engines should almost never be your primary data store. They are secondary indices that are populated from a source-of-truth database. Using Elasticsearch as a primary database is a common and expensive mistake — it lacks ACID transactions, has eventual consistency by default, and its documents are immutable (updates require full document re-indexing).
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## When to Use SQL (Relational Databases)
|
|
48
|
+
|
|
49
|
+
### The Default Choice for Most Applications
|
|
50
|
+
|
|
51
|
+
If you are building a new application and do not have a specific, well-articulated reason to choose something else, use PostgreSQL. This is not tribalism — it is engineering pragmatism based on the following properties:
|
|
52
|
+
|
|
53
|
+
**Your data has relationships.** Users have orders. Orders have line items. Line items reference products. Products belong to categories. If you drew your data model on a whiteboard, it would have arrows between entities. This is relational data, and relational databases handle it natively. Trying to model this in a document database means either embedding (which creates update anomalies) or referencing (which means you have reinvented foreign keys without the database enforcing them).
|
|
54
|
+
|
|
55
|
+
**You need ACID transactions.** Transfer $100 from account A to account B. Debit A and credit B must both succeed or both fail. This is not optional in financial systems, e-commerce, healthcare, inventory management, or any system where data inconsistency has real-world consequences. Relational databases have provided ACID guarantees since the 1970s. While MongoDB added multi-document transactions in 4.0 (2018), the performance characteristics and operational complexity of distributed transactions in document databases are substantially worse than in purpose-built relational systems.
|
|
56
|
+
|
|
57
|
+
**You need complex queries.** "Show me all customers who placed an order in the last 30 days, spent more than $500 total, and have not purchased from category X." This is a single SQL query with JOINs, aggregation, and subqueries. In a document database, this query either requires denormalization (storing redundant data to avoid lookups) or multiple application-level queries stitched together in code. SQL handles ad-hoc analytical queries that were never anticipated at design time.
|
|
58
|
+
|
|
59
|
+
**You need data integrity enforcement.** NOT NULL constraints, UNIQUE constraints, CHECK constraints, foreign key constraints — relational databases enforce these at the storage layer. Every write is validated against the schema. In a document database, these constraints exist only in application code, which means every microservice, every migration script, and every ad-hoc data fix must independently enforce them. One bug in one service and your data is corrupt with no database-level guardrail.
|
|
60
|
+
|
|
61
|
+
**You need a mature ecosystem.** PostgreSQL has 35+ years of production hardening, thousands of extensions (PostGIS for geospatial, pg_trgm for fuzzy search, pgvector for embeddings, TimescaleDB for time-series), world-class tooling (pg_dump, pg_restore, pgAdmin, psql), and the largest pool of experienced database administrators of any open-source database. The debugging, monitoring, and optimization toolchain is unmatched.
|
|
62
|
+
|
|
63
|
+
### Specific Use Cases Where SQL Excels
|
|
64
|
+
|
|
65
|
+
| Use Case | Why SQL | Example |
|
|
66
|
+
|---|---|---|
|
|
67
|
+
| E-commerce | Orders, inventory, payments all require ACID | Shopify runs on MySQL/PostgreSQL |
|
|
68
|
+
| Financial systems | Transaction integrity is non-negotiable | Banking cores are universally relational |
|
|
69
|
+
| SaaS applications | Multi-tenant data with complex access control | Most B2B SaaS runs on PostgreSQL |
|
|
70
|
+
| Content management | Structured content with relationships | WordPress runs on MySQL |
|
|
71
|
+
| Healthcare | Regulatory compliance demands data integrity | Epic, Cerner use relational stores |
|
|
72
|
+
| ERP / CRM | Deep entity relationships, complex reporting | SAP, Salesforce are relational |
|
|
73
|
+
| Analytics / Reporting | Ad-hoc queries, aggregation, windowing | Data warehouses are relational (Snowflake, BigQuery) |
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## When to Use NoSQL (By Type)
|
|
78
|
+
|
|
79
|
+
### Key-Value Stores — Caching and Session Data
|
|
80
|
+
|
|
81
|
+
**Use Redis when:**
|
|
82
|
+
- You need sub-millisecond response times for cache lookups
|
|
83
|
+
- You are caching database query results, API responses, or computed values
|
|
84
|
+
- You need session storage for web applications
|
|
85
|
+
- You need pub/sub messaging, rate limiting, or distributed locking
|
|
86
|
+
- You need data structures beyond simple strings (sorted sets for leaderboards, HyperLogLog for cardinality estimation, streams for event logs)
|
|
87
|
+
|
|
88
|
+
**Use Memcached when:**
|
|
89
|
+
- You need pure cache with no persistence requirement
|
|
90
|
+
- Your workload is simple key-value GET/SET with no need for data structures
|
|
91
|
+
- You want multi-threaded performance on a single node (Memcached is multi-threaded; Redis is single-threaded per shard)
|
|
92
|
+
|
|
93
|
+
**Use DynamoDB when:**
|
|
94
|
+
- You are building on AWS and want zero operational overhead (fully managed, serverless)
|
|
95
|
+
- Your access patterns are strictly key-based lookups with known partition keys
|
|
96
|
+
- You need single-digit millisecond latency with automatic scaling
|
|
97
|
+
- You need global tables for multi-region replication
|
|
98
|
+
- Your data model fits the single-table design pattern (advanced — requires expertise)
|
|
99
|
+
|
|
100
|
+
**Do NOT use key-value stores when:**
|
|
101
|
+
- You need to query by arbitrary fields (use a relational database)
|
|
102
|
+
- You need relationships between entities (use a relational database)
|
|
103
|
+
- You need ACID transactions across multiple keys (use a relational database)
|
|
104
|
+
- Your "cache" is actually your only data store (every cache needs a source of truth)
|
|
105
|
+
|
|
106
|
+
### Document Databases — Truly Variable Schemas
|
|
107
|
+
|
|
108
|
+
**Use MongoDB or Firestore when:**
|
|
109
|
+
- Your documents are genuinely heterogeneous — different documents in the same collection have fundamentally different structures, not just nullable fields
|
|
110
|
+
- You are prototyping rapidly and the data model is changing weekly (but plan to migrate to a relational database when the model stabilizes)
|
|
111
|
+
- You are building a content management system where each content type has a different structure and new content types are added frequently
|
|
112
|
+
- Your access pattern is "fetch one document by ID and return the whole thing" — the document is the unit of retrieval
|
|
113
|
+
- You are building a mobile app with Firebase/Firestore and need real-time sync with offline support
|
|
114
|
+
|
|
115
|
+
**The critical nuance:** "We don't want to define a schema" is not a valid reason to choose a document database. You always have a schema — it is either enforced by the database (explicit, safe) or enforced by application code (implicit, fragile). The "schemaless" flexibility of document databases means every consumer of the data must handle every possible shape of every document, forever. This is a maintenance burden, not a feature.
|
|
116
|
+
|
|
117
|
+
### Column-Family Databases — Write-Heavy at Massive Scale
|
|
118
|
+
|
|
119
|
+
**Use Cassandra or ScyllaDB when:**
|
|
120
|
+
- You are writing millions of events per second (IoT telemetry, application logging, messaging at Discord scale)
|
|
121
|
+
- You need linear horizontal scalability — add nodes, get proportionally more throughput
|
|
122
|
+
- You need multi-datacenter replication with tunable consistency
|
|
123
|
+
- Your query patterns are known and fixed at design time (you model tables per query)
|
|
124
|
+
- You can tolerate eventual consistency for most operations
|
|
125
|
+
|
|
126
|
+
**The Discord case study is instructive.** Discord started with MongoDB in 2015 for message storage. By November 2015, with 100 million messages, MongoDB's limitations became apparent — it could not handle the write volume or the data size. They migrated to Cassandra in 2017, which handled the write throughput but introduced its own problems. By 2022, the Cassandra cluster had grown to 177 nodes storing trillions of messages, and performance degradation required increasing effort just to maintain. They then migrated to ScyllaDB (a C++ rewrite of Cassandra), reducing the cluster from 177 nodes to 72 while cutting p99 read latency from 40-125ms to 15ms and p99 write latency from 5-70ms to 5ms. The migration of trillions of messages was accomplished in 9 days using a Rust-based data service with request coalescing.
|
|
127
|
+
|
|
128
|
+
The lesson: column-family databases solve real problems at extreme scale, but even the right tool requires iteration and operational investment.
|
|
129
|
+
|
|
130
|
+
### Graph Databases — Relationship-Heavy Queries
|
|
131
|
+
|
|
132
|
+
**Use Neo4j or Amazon Neptune when:**
|
|
133
|
+
- Your core value proposition depends on traversing relationships: social networks (friends of friends), recommendation engines (users who bought X also bought Y), fraud detection (find transaction rings), knowledge graphs, network topology analysis
|
|
134
|
+
- You need queries that traverse 4+ relationship hops — the point where relational JOINs become impractically slow
|
|
135
|
+
- You need path-finding algorithms (shortest path, all paths, weighted paths) as first-class operations
|
|
136
|
+
|
|
137
|
+
**Do NOT use graph databases when:**
|
|
138
|
+
- Your relationships are simple and shallow (1-2 hops) — SQL JOINs handle this efficiently
|
|
139
|
+
- You are storing graph-shaped data but only querying it by node ID (use a relational or document database)
|
|
140
|
+
- You need ACID transactions across your entire dataset (graph databases vary widely in transaction support)
|
|
141
|
+
|
|
142
|
+
### Time-Series Databases — Metrics, Sensors, and Events
|
|
143
|
+
|
|
144
|
+
**Use TimescaleDB when:**
|
|
145
|
+
- You have time-series data AND need SQL compatibility (it is a PostgreSQL extension)
|
|
146
|
+
- You need to join time-series data with relational data (sensor readings joined with device metadata)
|
|
147
|
+
- You have moderate-to-high cardinality data (thousands to millions of unique series)
|
|
148
|
+
- You want one database for both relational and time-series workloads
|
|
149
|
+
|
|
150
|
+
**Use InfluxDB when:**
|
|
151
|
+
- You have low-cardinality time-series data (hundreds of unique series)
|
|
152
|
+
- Raw ingestion performance and lightweight storage are your top priorities
|
|
153
|
+
- You are building a monitoring/observability stack (Telegraf + InfluxDB + Grafana)
|
|
154
|
+
|
|
155
|
+
**Use ClickHouse when:**
|
|
156
|
+
- You need analytical queries over massive time-series or event data
|
|
157
|
+
- Your workload is OLAP (Online Analytical Processing) — aggregations over billions of rows
|
|
158
|
+
- You can trade write latency for query performance
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## When NOT to Use NoSQL
|
|
163
|
+
|
|
164
|
+
### The Most Common Wrong Reasons
|
|
165
|
+
|
|
166
|
+
This section is the most important in the entire module. The majority of NoSQL adoption mistakes come from a small set of misconceptions. If you recognize any of these as your motivation, stop and reconsider.
|
|
167
|
+
|
|
168
|
+
#### Wrong Reason 1: "We don't want to define a schema"
|
|
169
|
+
|
|
170
|
+
This is the most common and most damaging reason for choosing a document database. The logic goes: "Our requirements are changing fast. We don't want to run migrations. Let's use MongoDB so we can just throw JSON in there."
|
|
171
|
+
|
|
172
|
+
**Why this is wrong:** You always have a schema. The question is where it is enforced:
|
|
173
|
+
|
|
174
|
+
| Schema Location | Enforcement | Failure Mode |
|
|
175
|
+
|---|---|---|
|
|
176
|
+
| Database (SQL) | Every write validated | Bad data rejected immediately |
|
|
177
|
+
| Application code (NoSQL) | Each service validates independently | One service's bug corrupts data for all consumers |
|
|
178
|
+
| Nowhere | No validation | Silent data corruption, discovered weeks later |
|
|
179
|
+
|
|
180
|
+
The "no schema" advantage is real for approximately the first 3 months of a project — when the data model is genuinely unstable. After that, the schema stabilizes, and you are left paying the maintenance tax of application-level validation forever. PostgreSQL's `ALTER TABLE` with transactional DDL makes schema migrations safe, fast, and reversible.
|
|
181
|
+
|
|
182
|
+
**What to do instead:** Use PostgreSQL. When you genuinely need schema flexibility for a specific subset of your data, use a JSONB column. You get the best of both worlds: rigid schema for the 90% of your data that is structured, flexible JSON for the 10% that genuinely varies.
|
|
183
|
+
|
|
184
|
+
#### Wrong Reason 2: "NoSQL scales better"
|
|
185
|
+
|
|
186
|
+
This is a half-truth that has caused enormous damage. The full truth:
|
|
187
|
+
|
|
188
|
+
- **NoSQL databases are designed for horizontal scaling.** Cassandra, DynamoDB, and MongoDB can distribute data across hundreds of nodes. This is architecturally true.
|
|
189
|
+
- **Most applications never need horizontal scaling.** A single PostgreSQL instance on modern hardware (128GB RAM, NVMe SSD) can handle millions of rows, thousands of concurrent connections, and tens of thousands of queries per second. PostgreSQL with read replicas can handle read-heavy workloads well beyond what 95% of applications will ever need.
|
|
190
|
+
- **Horizontal scaling has costs.** Distributed transactions are harder. Cross-partition queries are slower or impossible. Operational complexity increases dramatically. Debugging distributed consistency issues requires specialized expertise.
|
|
191
|
+
- **PostgreSQL can scale horizontally when needed.** Citus (now part of Azure) provides transparent sharding for PostgreSQL. Read replicas handle read scaling. Partitioning handles large tables.
|
|
192
|
+
|
|
193
|
+
The correct question is not "which database scales better?" but "will my application actually need to scale beyond what a single relational database instance can handle?" For most applications, the answer is no — and by the time it is yes, you will have the engineering resources, operational expertise, and data access pattern knowledge to make an informed migration decision.
|
|
194
|
+
|
|
195
|
+
#### Wrong Reason 3: "MongoDB is faster for development"
|
|
196
|
+
|
|
197
|
+
The argument: "Our developers can just dump JSON into MongoDB without writing migrations or defining schemas. This is faster."
|
|
198
|
+
|
|
199
|
+
**Why this is wrong in practice:**
|
|
200
|
+
- You trade upfront schema design (hours) for ongoing data quality debugging (weeks)
|
|
201
|
+
- Every new feature that touches the data must handle every historical document shape
|
|
202
|
+
- Aggregation pipelines in MongoDB are more complex and less readable than equivalent SQL
|
|
203
|
+
- The lack of JOINs means either data duplication (and sync bugs) or multiple round-trips to the database
|
|
204
|
+
- Reporting and analytics require ETL pipelines to move data into a relational warehouse
|
|
205
|
+
|
|
206
|
+
Multiple development teams have publicly documented their regret at choosing MongoDB for relational data. A curated collection of these stories is maintained at [github.com/shekhargulati/nosql-to-rdbms-stories](https://github.com/shekhargulati/nosql-to-rdbms-stories), documenting teams that migrated back from NoSQL to relational databases after discovering the total cost of ownership was higher, not lower.
|
|
207
|
+
|
|
208
|
+
#### Wrong Reason 4: "It's what the tutorial used"
|
|
209
|
+
|
|
210
|
+
Many web development tutorials (especially in the JavaScript ecosystem) use MongoDB because it has a low barrier to entry — `npm install mongoose`, define a schema in JavaScript, and start writing. This has led to a generation of developers who reach for MongoDB by default, not because it is the right tool, but because it is the first tool they learned.
|
|
211
|
+
|
|
212
|
+
### PostgreSQL JSONB: The Document Database Inside Your Relational Database
|
|
213
|
+
|
|
214
|
+
A critical factor in the SQL vs NoSQL decision: PostgreSQL's JSONB support is good enough to eliminate the need for a separate document database in most cases.
|
|
215
|
+
|
|
216
|
+
**What PostgreSQL JSONB provides:**
|
|
217
|
+
- Store arbitrary JSON documents in a column alongside structured relational data
|
|
218
|
+
- Index JSON fields with GIN indices for fast querying
|
|
219
|
+
- Query JSON fields with operators: `->`, `->>`, `@>`, `?`, `?|`, `?&`
|
|
220
|
+
- Full-text search within JSON documents
|
|
221
|
+
- Partial updates of JSON documents (not full-document replacement)
|
|
222
|
+
- JSON schema validation via CHECK constraints
|
|
223
|
+
|
|
224
|
+
**Performance characteristics:** In benchmark comparisons, PostgreSQL performs comparably to MongoDB for document workloads until document sizes exceed approximately 2KB (where PostgreSQL's TOAST mechanism introduces overhead). For documents under 2KB — which covers the vast majority of application data — PostgreSQL JSONB performance is on par with MongoDB. In a comprehensive benchmark study, PostgreSQL won 9 out of 17 test cases against MongoDB (with 7 for MongoDB and 1 draw). MongoDB does have advantages in update-heavy workloads on large documents, where its BSON format allows updates with lower memory and I/O amplification.
|
|
225
|
+
|
|
226
|
+
**The hybrid pattern:**
|
|
227
|
+
```sql
|
|
228
|
+
CREATE TABLE products (
|
|
229
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
230
|
+
name TEXT NOT NULL,
|
|
231
|
+
price DECIMAL(10,2) NOT NULL,
|
|
232
|
+
category_id UUID REFERENCES categories(id),
|
|
233
|
+
created_at TIMESTAMPTZ DEFAULT now(),
|
|
234
|
+
-- Structured data above, flexible data below
|
|
235
|
+
attributes JSONB DEFAULT '{}'::jsonb
|
|
236
|
+
);
|
|
237
|
+
|
|
238
|
+
-- Index for querying JSON attributes
|
|
239
|
+
CREATE INDEX idx_products_attrs ON products USING GIN (attributes);
|
|
240
|
+
|
|
241
|
+
-- Query: find products with color = 'red' and weight > 500
|
|
242
|
+
SELECT * FROM products
|
|
243
|
+
WHERE attributes @> '{"color": "red"}'
|
|
244
|
+
AND (attributes->>'weight')::int > 500;
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
This pattern gives you relational integrity for core data and document flexibility for variable attributes — in one database, one transaction, one operational footprint.
|
|
248
|
+
|
|
249
|
+
---
|
|
250
|
+
|
|
251
|
+
## How It Works — Core Concepts
|
|
252
|
+
|
|
253
|
+
### ACID vs BASE
|
|
254
|
+
|
|
255
|
+
These are the two fundamental consistency models that underpin the SQL vs NoSQL divide.
|
|
256
|
+
|
|
257
|
+
**ACID (Atomicity, Consistency, Isolation, Durability):**
|
|
258
|
+
|
|
259
|
+
| Property | Meaning | Practical Impact |
|
|
260
|
+
|---|---|---|
|
|
261
|
+
| Atomicity | All operations in a transaction succeed or all fail | Transfer $100: debit AND credit, never just one |
|
|
262
|
+
| Consistency | Every transaction moves the database from one valid state to another | Foreign keys, constraints always enforced |
|
|
263
|
+
| Isolation | Concurrent transactions don't interfere with each other | Two users buying the last item — only one succeeds |
|
|
264
|
+
| Durability | Once committed, data survives crashes | Power failure after COMMIT — data is safe |
|
|
265
|
+
|
|
266
|
+
ACID is the default for SQL databases. It prioritizes correctness over performance. The cost: coordination overhead (locks, write-ahead logs, two-phase commit for distributed transactions).
|
|
267
|
+
|
|
268
|
+
**BASE (Basically Available, Soft-state, Eventually consistent):**
|
|
269
|
+
|
|
270
|
+
| Property | Meaning | Practical Impact |
|
|
271
|
+
|---|---|---|
|
|
272
|
+
| Basically Available | System always returns a response, even if stale | Read might return data from 2 seconds ago |
|
|
273
|
+
| Soft-state | State may change without explicit writes (due to propagation) | Replica might show different data than primary |
|
|
274
|
+
| Eventually consistent | Given enough time without new writes, all replicas converge | After a write, all nodes will eventually agree |
|
|
275
|
+
|
|
276
|
+
BASE is the default for most NoSQL databases. It prioritizes availability and partition tolerance over immediate consistency. The benefit: higher throughput and lower latency at scale. The cost: application code must handle stale reads, conflict resolution, and the absence of guarantees that SQL developers take for granted.
|
|
277
|
+
|
|
278
|
+
### CAP Theorem — The Fundamental Trade-Off
|
|
279
|
+
|
|
280
|
+
The CAP theorem (Eric Brewer, 2000) states that a distributed data store can provide at most two of three guarantees:
|
|
281
|
+
|
|
282
|
+
- **Consistency (C):** Every read receives the most recent write or an error
|
|
283
|
+
- **Availability (A):** Every request receives a response (not an error)
|
|
284
|
+
- **Partition Tolerance (P):** The system continues to operate despite network partitions between nodes
|
|
285
|
+
|
|
286
|
+
Since network partitions are inevitable in distributed systems, the real choice is between **CP** (consistent but may be unavailable during partitions) and **AP** (available but may return stale data during partitions).
|
|
287
|
+
|
|
288
|
+
| Database | CAP Classification | Behavior During Partition |
|
|
289
|
+
|---|---|---|
|
|
290
|
+
| PostgreSQL (single node) | CA (no partition) | Not distributed — N/A |
|
|
291
|
+
| PostgreSQL (with replicas) | CP | Replicas may be unavailable during partition |
|
|
292
|
+
| MongoDB (default) | CP | Primary election may cause brief unavailability |
|
|
293
|
+
| Cassandra | AP | Returns data, may be stale |
|
|
294
|
+
| DynamoDB | AP (configurable) | Returns data, eventual consistency by default |
|
|
295
|
+
| Redis (cluster) | AP | Returns cached data, may diverge |
|
|
296
|
+
| CockroachDB | CP | Prioritizes consistency, rejects writes during partition |
|
|
297
|
+
|
|
298
|
+
**The critical nuance:** CAP theorem applies only to distributed systems during network partitions. A single-node PostgreSQL instance is not subject to CAP — it provides full ACID with no partition concerns. Most applications start (and stay) on a single database node. CAP becomes relevant only when you distribute data across multiple nodes, which is precisely when NoSQL databases offer their scaling advantages.
|
|
299
|
+
|
|
300
|
+
### Query Patterns by Database Type
|
|
301
|
+
|
|
302
|
+
Each database type optimizes for a specific query pattern. Using a database against its optimized pattern results in poor performance, complex code, or both.
|
|
303
|
+
|
|
304
|
+
| Database Type | Optimized Query Pattern | Anti-Pattern |
|
|
305
|
+
|---|---|---|
|
|
306
|
+
| Relational (SQL) | Arbitrary JOINs, aggregation, filtering by any column | Billions of simple key lookups per second |
|
|
307
|
+
| Document | Fetch full document by ID, query within single collection | JOINs across collections, complex aggregations |
|
|
308
|
+
| Key-Value | GET/SET by exact key | Range queries, filtering, aggregation |
|
|
309
|
+
| Column-Family | Scan within partition, time-range queries | Ad-hoc queries across partitions, JOINs |
|
|
310
|
+
| Graph | Multi-hop traversals, path-finding | Simple CRUD, aggregation, full-text search |
|
|
311
|
+
| Time-Series | Time-range aggregation, downsampling | Random key lookups, complex JOINs |
|
|
312
|
+
| Search Engine | Full-text search, fuzzy matching, faceting | Primary data storage, transactions |
|
|
313
|
+
|
|
314
|
+
### Indexing Strategies by Type
|
|
315
|
+
|
|
316
|
+
| Database | Primary Index | Secondary Indices | Full-Text Search |
|
|
317
|
+
|---|---|---|---|
|
|
318
|
+
| PostgreSQL | B-tree (default), Hash | B-tree, GIN, GiST, BRIN, partial, expression | tsvector + GIN (built-in) |
|
|
319
|
+
| MongoDB | B-tree | Compound, multikey, text, geospatial, hashed | Atlas Search (Lucene-based) |
|
|
320
|
+
| DynamoDB | Partition key + sort key | Global Secondary Index (GSI), Local Secondary Index (LSI) | Not supported natively |
|
|
321
|
+
| Cassandra | Partition key + clustering key | Secondary indexes (limited, use materialized views) | Not supported natively |
|
|
322
|
+
| Redis | Hash key | Not applicable (key-value) | RediSearch module |
|
|
323
|
+
| Neo4j | Node labels + property | Composite, full-text (Lucene) | Built-in (Lucene) |
|
|
324
|
+
| Elasticsearch | Inverted index (all fields) | All fields indexed by default | Core capability |
|
|
325
|
+
|
|
326
|
+
---
|
|
327
|
+
|
|
328
|
+
## Trade-Offs Matrix
|
|
329
|
+
|
|
330
|
+
### Comprehensive Comparison
|
|
331
|
+
|
|
332
|
+
| Dimension | PostgreSQL | MySQL | MongoDB | DynamoDB | Redis | Cassandra/ScyllaDB | Neo4j | TimescaleDB |
|
|
333
|
+
|---|---|---|---|---|---|---|---|---|
|
|
334
|
+
| **Data Model** | Relational tables | Relational tables | JSON documents | Key-value / document | Key-value + structures | Wide-column | Graph (nodes + edges) | Relational + time-series |
|
|
335
|
+
| **Consistency** | Strong (ACID) | Strong (ACID) | Strong (single-doc), configurable | Configurable (strong/eventual) | Eventual (cluster) | Tunable (per-query) | ACID (single-node) | Strong (ACID) |
|
|
336
|
+
| **Scaling Model** | Vertical + read replicas + Citus | Vertical + read replicas + Vitess | Horizontal (sharding) | Horizontal (automatic) | Horizontal (cluster) | Horizontal (linear) | Vertical (primarily) | Vertical + partitioning |
|
|
337
|
+
| **Query Flexibility** | Excellent (full SQL) | Good (SQL) | Moderate (MQL) | Low (key-based) | Minimal (key-based) | Low (CQL, no JOINs) | Excellent (Cypher for graphs) | Excellent (full SQL) |
|
|
338
|
+
| **Write Throughput** | Good (10K-50K TPS) | Good (10K-50K TPS) | Good (sharded) | Excellent (auto-scales) | Excellent (100K+ TPS) | Excellent (linear scaling) | Moderate | Good (optimized for append) |
|
|
339
|
+
| **Read Latency** | Low (sub-ms with index) | Low (sub-ms with index) | Low (by _id) | Consistent single-digit ms | Sub-millisecond | Low (within partition) | Low (indexed lookups) | Low (time-range queries) |
|
|
340
|
+
| **Operational Cost** | Moderate (well-understood) | Moderate (well-understood) | Moderate-High (sharding complexity) | Low (fully managed) | Low-Moderate | High (tuning required) | Moderate | Moderate (PostgreSQL expertise) |
|
|
341
|
+
| **Schema Flexibility** | Rigid + JSONB for flexibility | Rigid + JSON support | Fully flexible | Flexible (schemaless) | Schemaless | Semi-flexible | Flexible properties | Rigid + JSONB |
|
|
342
|
+
| **JOINs** | Native, optimized | Native, optimized | $lookup (limited, slow) | None | None | None | Traversals (native) | Native, optimized |
|
|
343
|
+
| **ACID Transactions** | Full | Full | Multi-document (since 4.0) | Limited (25 items, single partition) | Limited (MULTI/EXEC) | Lightweight transactions | Full (single-node) | Full |
|
|
344
|
+
| **Ecosystem Maturity** | 35+ years, massive | 30+ years, massive | 15+ years, large | AWS-only, growing | 15+ years, large | 15+ years, moderate | 15+ years, niche | 7+ years, growing |
|
|
345
|
+
| **Best For** | 90% of applications | Web apps, WordPress ecosystem | Truly variable schemas, rapid prototyping | Serverless, AWS-native key lookups | Caching, sessions, real-time | Write-heavy at extreme scale | Relationship traversals | Metrics joined with relational data |
|
|
346
|
+
|
|
347
|
+
---
|
|
348
|
+
|
|
349
|
+
## Evolution Path
|
|
350
|
+
|
|
351
|
+
### How Database Needs Change as Applications Grow
|
|
352
|
+
|
|
353
|
+
**Phase 1: Single Database (0 → 100K users)**
|
|
354
|
+
Start with PostgreSQL. One database. One operational footprint. Handle everything — relational data, JSON for flexible attributes, full-text search with tsvector, even basic time-series with partitioned tables. This phase can last years. Do not prematurely add database infrastructure.
|
|
355
|
+
|
|
356
|
+
**Phase 2: Add Caching Layer (100K → 1M users)**
|
|
357
|
+
Add Redis for caching hot data, session storage, and rate limiting. PostgreSQL remains the source of truth. Cache invalidation is the hard problem — start with TTL-based expiration and add event-driven invalidation only when stale data becomes a user-visible problem.
|
|
358
|
+
|
|
359
|
+
**Phase 3: Add Search (1M → 10M users)**
|
|
360
|
+
If your application has user-facing search (product search, content search, log analysis), add Elasticsearch or Meilisearch as a secondary index. PostgreSQL's full-text search is good for simple use cases but lacks relevance tuning, fuzzy matching, and faceted navigation at scale. Populate the search index from PostgreSQL via change data capture (Debezium) or application-level dual writes.
|
|
361
|
+
|
|
362
|
+
**Phase 4: Specialize (10M+ users)**
|
|
363
|
+
At this scale, specific bottlenecks emerge that justify specialized databases:
|
|
364
|
+
- Time-series data (metrics, events) → TimescaleDB or ClickHouse
|
|
365
|
+
- Message/notification fanout → Cassandra or ScyllaDB
|
|
366
|
+
- Recommendation engine → Neo4j or a feature store
|
|
367
|
+
- Global distribution → CockroachDB or DynamoDB Global Tables
|
|
368
|
+
|
|
369
|
+
**The critical rule:** Add each database to solve a measured bottleneck, not an anticipated one. Every additional database doubles operational complexity — monitoring, backups, failover, data synchronization, team expertise.
|
|
370
|
+
|
|
371
|
+
---
|
|
372
|
+
|
|
373
|
+
## Failure Modes
|
|
374
|
+
|
|
375
|
+
### Wrong Database Choice — The Most Expensive Architectural Mistake
|
|
376
|
+
|
|
377
|
+
Choosing the wrong primary database is one of the costliest mistakes a team can make. Unlike most architectural decisions, which can be refactored incrementally, a database migration requires:
|
|
378
|
+
- Migrating all existing data (potentially terabytes)
|
|
379
|
+
- Rewriting all data access code
|
|
380
|
+
- Updating all integrations and ETL pipelines
|
|
381
|
+
- Running dual-write during migration for zero-downtime cutover
|
|
382
|
+
- Retraining the operations team
|
|
383
|
+
|
|
384
|
+
This typically takes 6-18 months for a production system. The cost is measured in engineering years.
|
|
385
|
+
|
|
386
|
+
### Failure Mode 1: MongoDB for Relational Data
|
|
387
|
+
|
|
388
|
+
**The pattern:** Team chooses MongoDB because "it's easy to get started" and "we don't need schema migrations." Application grows. Data becomes deeply relational. Queries become `$lookup`-heavy aggregation pipelines that are slower and harder to debug than equivalent SQL. Data consistency bugs emerge because foreign key relationships are not enforced. Reporting requires a separate ETL pipeline into a data warehouse.
|
|
389
|
+
|
|
390
|
+
**Real-world occurrence:** This is by far the most common database mistake in the industry. The GitHub repository [nosql-to-rdbms-stories](https://github.com/shekhargulati/nosql-to-rdbms-stories) catalogs numerous companies that migrated back to relational databases after choosing MongoDB. Common complaints: excessive memory consumption over time, lack of native JOIN support forcing data duplication, poor performance for GROUP BY and reporting queries, and no database-level enforcement of data integrity.
|
|
391
|
+
|
|
392
|
+
**How to avoid it:** Default to PostgreSQL. Use JSONB columns for the genuinely flexible parts of your data model. If after 6 months you still have no relational patterns in your data, you might actually have a document database use case.
|
|
393
|
+
|
|
394
|
+
### Failure Mode 2: Premature Horizontal Scaling
|
|
395
|
+
|
|
396
|
+
**The pattern:** Team chooses Cassandra or DynamoDB because "we need to scale." Application has 10,000 users and 50GB of data. A single PostgreSQL instance could handle this for years. But the team now deals with eventual consistency bugs, cross-partition query limitations, and the operational complexity of a distributed database cluster — all for scale they do not need and may never need.
|
|
397
|
+
|
|
398
|
+
**How to avoid it:** Measure first. PostgreSQL on a single modern server handles millions of rows and thousands of concurrent connections. Only move to a distributed database when you have specific, measured bottlenecks that vertical scaling cannot address. By the time you hit true horizontal scaling needs, you will have the engineering resources and data access pattern knowledge to make an informed choice.
|
|
399
|
+
|
|
400
|
+
### Failure Mode 3: Using a Search Engine as a Primary Database
|
|
401
|
+
|
|
402
|
+
**The pattern:** Team uses Elasticsearch to store application data because "we need search." Elasticsearch becomes the source of truth. Then they discover: documents are immutable (updates re-index the entire document), there are no ACID transactions, consistency is eventual, and cluster management is operationally demanding.
|
|
403
|
+
|
|
404
|
+
**How to avoid it:** Always have a relational database as the source of truth. Populate Elasticsearch as a secondary index via change data capture. If Elasticsearch loses data, you can rebuild the index from the primary database.
|
|
405
|
+
|
|
406
|
+
### Failure Mode 4: Discord's Database Journey
|
|
407
|
+
|
|
408
|
+
Discord's database migration story is one of the most well-documented and instructive case studies in the industry.
|
|
409
|
+
|
|
410
|
+
**2015 — MongoDB:** Discord started with a single MongoDB replica set for message storage. By November 2015, with 100 million messages, MongoDB could not handle the write volume or data size. The limitations were fundamental — MongoDB's storage engine and replication model were not designed for Discord's write-heavy, time-series-like access pattern (messages are written once, read by time range, and rarely updated).
|
|
411
|
+
|
|
412
|
+
**2017 — Cassandra:** Discord migrated to Cassandra, which was designed for exactly this workload — write-heavy, distributed, with data modeled for time-range reads. Cassandra handled the write throughput well, and Discord scaled to trillions of messages.
|
|
413
|
+
|
|
414
|
+
**2022 — Cassandra Hits Limits:** By 2022, the Cassandra cluster had grown to 177 nodes. Despite Cassandra being the right architectural choice, operational challenges mounted: garbage collection pauses caused latency spikes, compaction competed for I/O with live reads, and maintaining the cluster required increasing engineering effort just to prevent degradation.
|
|
415
|
+
|
|
416
|
+
**2023 — ScyllaDB:** Discord migrated to ScyllaDB, a C++ rewrite of Cassandra that eliminates JVM garbage collection overhead. Results: 177 nodes reduced to 72. P99 read latency: 40-125ms (Cassandra) to 15ms (ScyllaDB). P99 write latency: 5-70ms (Cassandra) to 5ms (ScyllaDB). They built a Rust-based data service layer with gRPC API and request coalescing, migrating trillions of messages in 9 days.
|
|
417
|
+
|
|
418
|
+
**The lesson:** Even when you choose the right category of database (column-family for write-heavy messaging), the specific implementation matters enormously. And at true scale, operational characteristics (GC pauses, compaction overhead, memory model) dominate over feature sets.
|
|
419
|
+
|
|
420
|
+
---
|
|
421
|
+
|
|
422
|
+
## Technology Landscape
|
|
423
|
+
|
|
424
|
+
### Primary Databases — Detailed Profiles
|
|
425
|
+
|
|
426
|
+
**PostgreSQL** — The Swiss Army knife. Relational core with extensions for almost everything: JSONB for documents, PostGIS for geospatial, pgvector for AI embeddings, TimescaleDB for time-series, pg_trgm for fuzzy text search, Citus for horizontal sharding. The safest default for any new application. Open source, no vendor lock-in, massive community.
|
|
427
|
+
|
|
428
|
+
**MySQL / MariaDB** — The web's workhorse. Powers WordPress, Drupal, and much of the web. Simpler than PostgreSQL, with fewer advanced features but sufficient for most web applications. MySQL (Oracle-owned) vs MariaDB (community fork) — prefer MariaDB for open-source alignment, MySQL for Oracle ecosystem integration.
|
|
429
|
+
|
|
430
|
+
**MongoDB** — The most popular document database. Legitimate use cases: content management systems with truly heterogeneous content types, real-time analytics on semi-structured data, rapid prototyping where the data model is genuinely unstable. Has improved significantly since early versions: multi-document ACID transactions (4.0+), schema validation, Atlas managed service. Still frequently chosen for the wrong reasons.
|
|
431
|
+
|
|
432
|
+
**Amazon DynamoDB** — Fully managed, serverless key-value and document database. Legitimate use cases: AWS-native applications with known access patterns, serverless architectures (pairs with Lambda), global applications needing multi-region replication with DynamoDB Global Tables. The single-table design pattern (modeling all entities in one table) requires significant expertise but enables remarkable performance. The DAX (DynamoDB Accelerator) in-memory cache layer provides microsecond read latency.
|
|
433
|
+
|
|
434
|
+
**Redis** — In-memory data structure store. Not a primary database (despite Redis Inc.'s marketing). Legitimate use cases: caching, session storage, real-time leaderboards (sorted sets), pub/sub messaging, rate limiting, distributed locking. Supports persistence (RDB snapshots, AOF log) but should not be your only copy of data. Redis 7.0+ adds Redis Functions (server-side scripting) and improved cluster management.
|
|
435
|
+
|
|
436
|
+
**Apache Cassandra / ScyllaDB** — Distributed column-family databases for write-heavy workloads at extreme scale. Cassandra (Java, Apache Foundation) was designed by Facebook for inbox search. ScyllaDB (C++) is a drop-in Cassandra replacement that eliminates JVM garbage collection overhead. Use when: millions of writes per second, multi-datacenter replication, data modeled per query pattern. Do not use when: you need JOINs, ad-hoc queries, or your data is under 1TB.
|
|
437
|
+
|
|
438
|
+
**Neo4j** — The most mature graph database. Uses the Cypher query language. Legitimate use cases: social networks, fraud detection, recommendation engines, knowledge graphs, network topology. The AuraDB managed service reduces operational burden. Community edition is open source; Enterprise requires a commercial license.
|
|
439
|
+
|
|
440
|
+
**TimescaleDB** — A PostgreSQL extension for time-series data. The key advantage: it IS PostgreSQL, so you get full SQL, JOINs with relational data, the entire PostgreSQL ecosystem, and time-series optimizations (hypertables, continuous aggregates, compression, retention policies) in one database. Outperforms InfluxDB for moderate-to-high cardinality workloads and complex queries.
|
|
441
|
+
|
|
442
|
+
**InfluxDB** — Purpose-built time-series database. InfluxDB 3.0 (rebuilt on Apache Arrow and DataFusion) supports SQL queries. Best for low-cardinality monitoring data (hundreds of unique metric series). The TICK stack (Telegraf, InfluxDB, Chronograf, Kapacitor) provides a complete monitoring pipeline.
|
|
443
|
+
|
|
444
|
+
**Elasticsearch / OpenSearch** — Distributed search and analytics engines built on Apache Lucene. Not a primary database. Use as a secondary index for full-text search, log analytics (ELK stack), and application search. OpenSearch is the community fork after Elastic changed its license. Meilisearch and Typesense are lighter alternatives for application search.
|
|
445
|
+
|
|
446
|
+
**CockroachDB** — Distributed SQL database that provides PostgreSQL-compatible SQL with automatic horizontal scaling and serializable ACID transactions. Designed for global applications needing strong consistency across regions. The "NewSQL" approach: relational guarantees at distributed scale. Use when you genuinely need globally distributed ACID transactions.
|
|
447
|
+
|
|
448
|
+
---
|
|
449
|
+
|
|
450
|
+
## Decision Tree
|
|
451
|
+
|
|
452
|
+
Use this concrete decision tree when choosing a database. Start at the top and follow the path that matches your requirements.
|
|
453
|
+
|
|
454
|
+
```
|
|
455
|
+
START: What is your primary data access pattern?
|
|
456
|
+
|
|
457
|
+
1. Relational data with JOINs, complex queries, transactions?
|
|
458
|
+
└─→ PostgreSQL (default)
|
|
459
|
+
├─ Need horizontal scaling? → CockroachDB or Citus
|
|
460
|
+
├─ Already in Oracle/Microsoft ecosystem? → Oracle / SQL Server
|
|
461
|
+
└─ Simple web app, WordPress? → MySQL / MariaDB
|
|
462
|
+
|
|
463
|
+
2. Key-value lookups by exact key?
|
|
464
|
+
├─ Need sub-ms latency, in-memory? → Redis
|
|
465
|
+
├─ Need pure cache, no persistence? → Memcached
|
|
466
|
+
└─ Need managed, serverless, AWS-native? → DynamoDB
|
|
467
|
+
|
|
468
|
+
3. Documents with truly variable schema?
|
|
469
|
+
├─ Need mobile offline sync? → Firestore
|
|
470
|
+
├─ Need managed cloud service? → MongoDB Atlas or DynamoDB
|
|
471
|
+
└─ Actually, is the variability limited to a few fields?
|
|
472
|
+
└─ YES → PostgreSQL with JSONB column (don't use a document DB)
|
|
473
|
+
|
|
474
|
+
4. Write-heavy, millions of events/sec, distributed?
|
|
475
|
+
├─ Need linear horizontal scaling? → ScyllaDB (preferred) or Cassandra
|
|
476
|
+
└─ Need AWS-managed? → DynamoDB or Amazon Keyspaces
|
|
477
|
+
|
|
478
|
+
5. Graph traversals, 4+ hop relationship queries?
|
|
479
|
+
└─→ Neo4j (or Amazon Neptune for AWS-managed)
|
|
480
|
+
└─ Only 1-2 hops? → PostgreSQL with recursive CTEs
|
|
481
|
+
|
|
482
|
+
6. Time-series data (metrics, sensors, events)?
|
|
483
|
+
├─ Need SQL + relational JOINs? → TimescaleDB
|
|
484
|
+
├─ Low-cardinality monitoring? → InfluxDB
|
|
485
|
+
└─ Analytical queries over billions of events? → ClickHouse
|
|
486
|
+
|
|
487
|
+
7. Full-text search, fuzzy matching, faceted navigation?
|
|
488
|
+
└─→ Elasticsearch / OpenSearch (as secondary index)
|
|
489
|
+
├─ Simple search needs? → PostgreSQL full-text search
|
|
490
|
+
└─ Lightweight application search? → Meilisearch / Typesense
|
|
491
|
+
|
|
492
|
+
8. Not sure / multiple patterns?
|
|
493
|
+
└─→ PostgreSQL. Start here. Add specialized databases
|
|
494
|
+
only when you measure a specific bottleneck.
|
|
495
|
+
```
|
|
496
|
+
|
|
497
|
+
### The One-Sentence Rule
|
|
498
|
+
|
|
499
|
+
> If you cannot articulate a specific, technical reason why PostgreSQL will not work for your use case, use PostgreSQL.
|
|
500
|
+
|
|
501
|
+
---
|
|
502
|
+
|
|
503
|
+
## Implementation Sketch
|
|
504
|
+
|
|
505
|
+
### Pattern: PostgreSQL as Primary + Specialized Secondaries
|
|
506
|
+
|
|
507
|
+
This is the architecture that works for the vast majority of applications from startup to scale.
|
|
508
|
+
|
|
509
|
+
```
|
|
510
|
+
┌──────────────────────────────────────────────────────────┐
|
|
511
|
+
│ Application Layer │
|
|
512
|
+
├──────────┬──────────┬───────────┬──────────┬─────────────┤
|
|
513
|
+
│ API │ Worker │ Scheduler│ Stream │ Analytics │
|
|
514
|
+
│ Server │ Queue │ │ Proc. │ Service │
|
|
515
|
+
└────┬─────┴────┬─────┴─────┬─────┴────┬─────┴──────┬──────┘
|
|
516
|
+
│ │ │ │ │
|
|
517
|
+
▼ ▼ ▼ ▼ ▼
|
|
518
|
+
┌─────────┐ ┌──────┐ ┌──────────┐ ┌────────┐ ┌──────────┐
|
|
519
|
+
│PostgreSQL│ │Redis │ │PostgreSQL│ │Timescale│ │Elastic- │
|
|
520
|
+
│ (primary │ │(cache│ │ (same │ │ DB │ │ search │
|
|
521
|
+
│ source │ │ + │ │ instance │ │(metrics│ │(search │
|
|
522
|
+
│ of │ │ pub/ │ │ or │ │ + │ │ index, │
|
|
523
|
+
│ truth) │ │ sub) │ │ replica)│ │ events)│ │ fed by │
|
|
524
|
+
│ │ │ │ │ │ │ │ │ CDC) │
|
|
525
|
+
└─────────┘ └──────┘ └──────────┘ └────────┘ └──────────┘
|
|
526
|
+
│ ▲
|
|
527
|
+
│ Change Data Capture (Debezium) │
|
|
528
|
+
└────────────────────────────────────────────┘
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
### Starter Schema: PostgreSQL with JSONB Flexibility
|
|
532
|
+
|
|
533
|
+
```sql
|
|
534
|
+
-- Core relational tables with strict schema
|
|
535
|
+
CREATE TABLE users (
|
|
536
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
537
|
+
email TEXT NOT NULL UNIQUE,
|
|
538
|
+
name TEXT NOT NULL,
|
|
539
|
+
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
540
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
541
|
+
);
|
|
542
|
+
|
|
543
|
+
CREATE TABLE orders (
|
|
544
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
545
|
+
user_id UUID NOT NULL REFERENCES users(id),
|
|
546
|
+
status TEXT NOT NULL CHECK (status IN ('pending','paid','shipped','delivered','cancelled')),
|
|
547
|
+
total DECIMAL(12,2) NOT NULL CHECK (total >= 0),
|
|
548
|
+
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
549
|
+
-- Flexible metadata for order-specific attributes
|
|
550
|
+
metadata JSONB DEFAULT '{}'::jsonb
|
|
551
|
+
);
|
|
552
|
+
|
|
553
|
+
CREATE TABLE order_items (
|
|
554
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
555
|
+
order_id UUID NOT NULL REFERENCES orders(id) ON DELETE CASCADE,
|
|
556
|
+
product_id UUID NOT NULL REFERENCES products(id),
|
|
557
|
+
quantity INT NOT NULL CHECK (quantity > 0),
|
|
558
|
+
unit_price DECIMAL(10,2) NOT NULL CHECK (unit_price >= 0)
|
|
559
|
+
);
|
|
560
|
+
|
|
561
|
+
-- Flexible attributes for products (different categories have different attributes)
|
|
562
|
+
CREATE TABLE products (
|
|
563
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
564
|
+
name TEXT NOT NULL,
|
|
565
|
+
sku TEXT NOT NULL UNIQUE,
|
|
566
|
+
category TEXT NOT NULL,
|
|
567
|
+
base_price DECIMAL(10,2) NOT NULL CHECK (base_price >= 0),
|
|
568
|
+
-- JSONB for category-specific attributes
|
|
569
|
+
-- Electronics: {"weight_kg": 1.5, "battery_mah": 5000, "color": "black"}
|
|
570
|
+
-- Clothing: {"size": "L", "material": "cotton", "color": "blue"}
|
|
571
|
+
-- Books: {"isbn": "978-...", "pages": 350, "language": "en"}
|
|
572
|
+
attributes JSONB DEFAULT '{}'::jsonb
|
|
573
|
+
);
|
|
574
|
+
|
|
575
|
+
-- GIN index for fast JSONB queries
|
|
576
|
+
CREATE INDEX idx_products_attributes ON products USING GIN (attributes);
|
|
577
|
+
|
|
578
|
+
-- Partial index: only index products that have a specific attribute
|
|
579
|
+
CREATE INDEX idx_products_color ON products ((attributes->>'color'))
|
|
580
|
+
WHERE attributes ? 'color';
|
|
581
|
+
|
|
582
|
+
-- Full-text search on product names (no Elasticsearch needed for simple search)
|
|
583
|
+
ALTER TABLE products ADD COLUMN search_vector tsvector
|
|
584
|
+
GENERATED ALWAYS AS (to_tsvector('english', name)) STORED;
|
|
585
|
+
CREATE INDEX idx_products_search ON products USING GIN (search_vector);
|
|
586
|
+
```
|
|
587
|
+
|
|
588
|
+
### Redis Caching Layer
|
|
589
|
+
|
|
590
|
+
```python
|
|
591
|
+
import redis
|
|
592
|
+
import json
|
|
593
|
+
from functools import wraps
|
|
594
|
+
|
|
595
|
+
r = redis.Redis(host='localhost', port=6379, db=0)
|
|
596
|
+
|
|
597
|
+
def cache(ttl_seconds=300, prefix="cache"):
|
|
598
|
+
"""Simple TTL-based cache decorator."""
|
|
599
|
+
def decorator(func):
|
|
600
|
+
@wraps(func)
|
|
601
|
+
def wrapper(*args, **kwargs):
|
|
602
|
+
key = f"{prefix}:{func.__name__}:{hash((args, tuple(sorted(kwargs.items()))))}"
|
|
603
|
+
cached = r.get(key)
|
|
604
|
+
if cached:
|
|
605
|
+
return json.loads(cached)
|
|
606
|
+
result = func(*args, **kwargs)
|
|
607
|
+
r.setex(key, ttl_seconds, json.dumps(result, default=str))
|
|
608
|
+
return result
|
|
609
|
+
return wrapper
|
|
610
|
+
return decorator
|
|
611
|
+
|
|
612
|
+
# Invalidate on write
|
|
613
|
+
def invalidate_product_cache(product_id: str):
|
|
614
|
+
"""Delete all cache entries related to a product."""
|
|
615
|
+
# Pattern-based invalidation — use with care in production
|
|
616
|
+
for key in r.scan_iter(f"cache:get_product:{product_id}*"):
|
|
617
|
+
r.delete(key)
|
|
618
|
+
|
|
619
|
+
# Session storage
|
|
620
|
+
def store_session(session_id: str, user_data: dict, ttl: int = 3600):
|
|
621
|
+
r.setex(f"session:{session_id}", ttl, json.dumps(user_data))
|
|
622
|
+
|
|
623
|
+
def get_session(session_id: str) -> dict | None:
|
|
624
|
+
data = r.get(f"session:{session_id}")
|
|
625
|
+
return json.loads(data) if data else None
|
|
626
|
+
```
|
|
627
|
+
|
|
628
|
+
### When You Actually Need MongoDB (Content CMS Example)
|
|
629
|
+
|
|
630
|
+
```javascript
|
|
631
|
+
// Legitimate MongoDB use case: CMS with truly heterogeneous content types
|
|
632
|
+
// Each content type has a completely different structure
|
|
633
|
+
|
|
634
|
+
// Blog post
|
|
635
|
+
db.content.insertOne({
|
|
636
|
+
type: "blog_post",
|
|
637
|
+
title: "Understanding Databases",
|
|
638
|
+
slug: "understanding-databases",
|
|
639
|
+
author: ObjectId("..."),
|
|
640
|
+
body: "...",
|
|
641
|
+
tags: ["databases", "architecture"],
|
|
642
|
+
published_at: new Date(),
|
|
643
|
+
seo: { meta_description: "...", og_image: "..." }
|
|
644
|
+
});
|
|
645
|
+
|
|
646
|
+
// Product review (completely different structure)
|
|
647
|
+
db.content.insertOne({
|
|
648
|
+
type: "product_review",
|
|
649
|
+
product_name: "PostgreSQL 17",
|
|
650
|
+
rating: 4.8,
|
|
651
|
+
pros: ["Performance", "Extensibility", "JSONB"],
|
|
652
|
+
cons: ["Learning curve for advanced features"],
|
|
653
|
+
verdict: "Best general-purpose database",
|
|
654
|
+
comparison_table: [
|
|
655
|
+
{ feature: "ACID", postgresql: true, mongodb: "limited" }
|
|
656
|
+
]
|
|
657
|
+
});
|
|
658
|
+
|
|
659
|
+
// Event listing (yet another structure)
|
|
660
|
+
db.content.insertOne({
|
|
661
|
+
type: "event",
|
|
662
|
+
name: "PGConf 2026",
|
|
663
|
+
location: { type: "Point", coordinates: [-73.97, 40.77] },
|
|
664
|
+
dates: { start: new Date("2026-06-15"), end: new Date("2026-06-17") },
|
|
665
|
+
speakers: [{ name: "...", topic: "..." }],
|
|
666
|
+
registration: { url: "...", price_usd: 299, early_bird_deadline: new Date() }
|
|
667
|
+
});
|
|
668
|
+
```
|
|
669
|
+
|
|
670
|
+
This is a legitimate document database use case because the content types are genuinely heterogeneous — not just "nullable columns" but fundamentally different structures. Even here, you could use PostgreSQL with JSONB, but MongoDB's document model is a natural fit.
|
|
671
|
+
|
|
672
|
+
---
|
|
673
|
+
|
|
674
|
+
## Cross-References
|
|
675
|
+
|
|
676
|
+
- **[data-modeling](./data-modeling.md)** — How to design schemas for relational and document databases, normalization vs denormalization trade-offs
|
|
677
|
+
- **[data-consistency](./data-consistency.md)** — Deep dive into consistency models, eventual consistency patterns, conflict resolution
|
|
678
|
+
- **[database-scaling](../scaling/database-scaling.md)** — Read replicas, sharding strategies, connection pooling, query optimization at scale
|
|
679
|
+
- **[caching-architecture](../patterns/caching-architecture.md)** — Cache invalidation strategies, cache-aside vs write-through, multi-tier caching
|
|
680
|
+
- **[search-architecture](../integration/search-architecture.md)** — Elasticsearch/OpenSearch architecture, indexing strategies, relevance tuning
|
|
681
|
+
|
|
682
|
+
---
|
|
683
|
+
|
|
684
|
+
## Sources
|
|
685
|
+
|
|
686
|
+
Research and case studies referenced in this module:
|
|
687
|
+
|
|
688
|
+
- [SQL vs NoSQL: 5 Critical Differences — Integrate.io](https://www.integrate.io/blog/the-sql-vs-nosql-difference/)
|
|
689
|
+
- [NoSQL vs SQL in 2026 — Hackr.io](https://hackr.io/blog/nosql-vs-sql)
|
|
690
|
+
- [SQL vs NoSQL in 2025 — Medium / AI & Analytics Diaries](https://medium.com/ai-analytics-diaries/sql-vs-nosql-in-2025-what-you-should-really-be-using-a21a7c2bd73c)
|
|
691
|
+
- [MongoDB vs PostgreSQL — MongoDB](https://www.mongodb.com/resources/compare/mongodb-postgresql)
|
|
692
|
+
- [Postgres vs MongoDB: Complete Comparison 2025 — Bytebase](https://www.bytebase.com/blog/postgres-vs-mongodb/)
|
|
693
|
+
- [NoSQL to RDBMS Stories — GitHub](https://github.com/shekhargulati/nosql-to-rdbms-stories)
|
|
694
|
+
- [NoSQL Was a Mistake: Why SQL Is Making a Comeback — Medium](https://medium.com/@toyezyadav/nosql-was-a-mistake-why-sql-is-making-a-comeback-c7e3d9e800f4)
|
|
695
|
+
- [Goodbye MongoDB — Stuart Spence](https://blog.stuartspence.ca/2023-05-goodbye-mongo.html)
|
|
696
|
+
- [JSON Performance: PostgreSQL vs MongoDB — DocumentDatabase.org](https://documentdatabase.org/blog/json-performance-postgres-vs-mongodb/)
|
|
697
|
+
- [MongoDB vs PostgreSQL JSONB: Deep Dive — DEV Community](https://dev.to/sparsh9/mongodb-vs-postgresql-jsonb-a-deep-dive-into-performance-and-use-cases-5bge)
|
|
698
|
+
- [DynamoDB vs PostgreSQL — Airbyte](https://airbyte.com/data-engineering-resources/dynamodb-vs-postgres)
|
|
699
|
+
- [DynamoDB vs PostgreSQL 2026 — Dynobase](https://dynobase.dev/dynamodb-vs-postgres/)
|
|
700
|
+
- [How Discord Stores Trillions of Messages — Discord Blog](https://discord.com/blog/how-discord-stores-trillions-of-messages)
|
|
701
|
+
- [Discord: MongoDB to Cassandra to ScyllaDB — HackerNoon](https://hackernoon.com/discord-went-from-mongodb-to-cassandra-then-scylladb-why)
|
|
702
|
+
- [Discord Migrates Trillions of Messages from Cassandra to ScyllaDB — InfoQ](https://www.infoq.com/news/2023/06/discord-cassandra-scylladb/)
|
|
703
|
+
- [ACID vs BASE Databases — AWS](https://aws.amazon.com/compare/the-difference-between-acid-and-base-database/)
|
|
704
|
+
- [CAP, PACELC, ACID, BASE — ByteByteGo](https://blog.bytebytego.com/p/cap-pacelc-acid-base-essential-concepts)
|
|
705
|
+
- [TimescaleDB vs InfluxDB — Tiger Data](https://www.tigerdata.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877)
|
|
706
|
+
- [ClickHouse vs TimescaleDB vs InfluxDB: 2025 Benchmarks — sanj.dev](https://sanj.dev/post/clickhouse-timescaledb-influxdb-time-series-comparison)
|
|
707
|
+
- [Redis vs DynamoDB — Severalnines](https://severalnines.com/blog/redis-vs-dynamodb-comparison/)
|
|
708
|
+
- [Redis vs Memcached — AWS](https://aws.amazon.com/elasticache/redis-vs-memcached/)
|