webspresso 0.0.105 → 0.0.106
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/AGENTS.md +15 -2
- package/.agents/CLI.md +219 -7
- package/.agents/FLAGS.md +118 -0
- package/.agents/LOGGING.md +93 -0
- package/.agents/METRICS.md +127 -0
- package/.agents/MIGRATION_JOBS.md +95 -0
- package/.agents/PLUGINS.md +10 -3
- package/.agents/QUEUE.md +42 -0
- package/.agents/RATE_LIMIT.md +147 -0
- package/.agents/SCALE_OUT.md +109 -0
- package/.agents/SECRETS.md +117 -0
- package/AGENTS.md +10 -0
- package/bin/commands/add-aws.js +396 -0
- package/bin/commands/add-ci.js +523 -0
- package/bin/commands/add-cloudflare.js +240 -0
- package/bin/commands/add-cloudrun.js +381 -0
- package/bin/commands/add-migration-job.js +396 -0
- package/bin/commands/assets-sync.js +209 -0
- package/bin/commands/client-generate.js +52 -0
- package/bin/commands/console.js +53 -0
- package/bin/commands/db-migrate.js +47 -2
- package/bin/commands/db-studio.js +84 -0
- package/bin/commands/deploy-check.js +242 -13
- package/bin/commands/export.js +87 -0
- package/bin/commands/secrets-pull.js +208 -0
- package/bin/utils/db.js +7 -1
- package/bin/webspresso.js +22 -0
- package/core/app.js +421 -0
- package/core/auth/authenticate.js +93 -0
- package/core/auth/index.js +34 -4
- package/core/auth/manager.js +50 -5
- package/core/auth/middleware.js +16 -68
- package/core/auth/redis-session.js +1008 -0
- package/core/auth/sessions.js +156 -0
- package/core/cache/cdn.js +296 -0
- package/core/cache/index.js +12 -0
- package/core/cache/tagged-lru.js +9 -1
- package/core/console/commands.js +148 -0
- package/core/console/context.js +260 -0
- package/core/console/eval.js +70 -0
- package/core/console/index.js +22 -0
- package/core/console/repl.js +102 -0
- package/core/errors/domain.js +1 -0
- package/core/events/dispatcher.js +7 -0
- package/core/events/signal.js +7 -0
- package/core/export/discovery.js +191 -0
- package/core/export/exporter.js +277 -0
- package/core/export/index.js +24 -0
- package/core/flags/evaluator.js +252 -0
- package/core/flags/hasher.js +37 -0
- package/core/flags/index.js +21 -0
- package/core/flags/manager.js +263 -0
- package/core/flags/store.js +126 -0
- package/core/health/middleware.js +3 -3
- package/core/http/adapter.js +74 -0
- package/core/http/express-adapter.js +46 -0
- package/core/http/hono-adapter.js +531 -0
- package/core/http/index.js +41 -0
- package/core/http/pipeline.js +485 -0
- package/core/index.js +336 -0
- package/core/limits/middleware.js +9 -1
- package/core/logger/formatters.js +264 -0
- package/core/logger/index.js +260 -0
- package/core/logger/trace.js +137 -0
- package/core/metrics/collectors.js +125 -0
- package/core/metrics/index.js +45 -0
- package/core/metrics/middleware.js +230 -0
- package/core/metrics/registry.js +366 -0
- package/core/orm/d1.js +265 -0
- package/core/orm/index.js +105 -5
- package/core/orm/migrate-cloud.js +194 -0
- package/core/orm/pool.js +97 -0
- package/core/orm/query-builder.js +9 -0
- package/core/queue/adapters/cloudflare.js +131 -0
- package/core/queue/adapters/pubsub.js +400 -0
- package/core/queue/adapters/sqs.js +419 -0
- package/core/queue/adapters/wait-until.js +77 -0
- package/core/queue/index.js +8 -0
- package/core/queue/manager.js +11 -0
- package/core/rate-limit/index.js +22 -0
- package/core/rate-limit/limiter.js +219 -0
- package/core/rate-limit/stores/memory.js +129 -0
- package/core/rate-limit/stores/redis.js +151 -0
- package/core/rate-limit/utils.js +112 -0
- package/core/resilience/circuit-breaker.js +31 -0
- package/core/secrets/index.js +113 -0
- package/core/secrets/manager.js +308 -0
- package/core/secrets/providers/aws.js +288 -0
- package/core/secrets/providers/env.js +44 -0
- package/core/secrets/providers/gcp.js +307 -0
- package/core/secrets/providers/memory.js +76 -0
- package/core/shutdown/shutdown-manager.js +9 -0
- package/core/stream/index.js +19 -0
- package/core/stream/text-stream.js +326 -0
- package/core/studio/index.js +43 -0
- package/core/studio/inspector.js +258 -0
- package/core/studio/server.js +325 -0
- package/core/studio/ui.js +681 -0
- package/index.d.ts +934 -2
- package/index.js +125 -3
- package/package.json +3 -1
- package/plugins/admin-panel/index.js +12 -5
- package/plugins/api-key/index.js +3 -2
- package/plugins/audit-log/index.js +3 -1
- package/plugins/basic-auth/index.js +3 -2
- package/plugins/cors.js +3 -2
- package/plugins/csrf/index.js +4 -3
- package/plugins/csv/index.js +8 -5
- package/plugins/email/index.js +11 -9
- package/plugins/index.js +3 -0
- package/plugins/queue/index.js +18 -2
- package/plugins/redirect/index.js +7 -2
- package/plugins/seo-checker/index.js +2 -2
- package/plugins/site-analytics/index.js +3 -1
- package/plugins/swagger.js +8 -2
- package/plugins/upload/gcs-provider.js +148 -0
- package/plugins/upload/index.js +132 -74
- package/plugins/upload/s3-provider.js +257 -0
- package/plugins/webhooks/index.js +27 -23
- package/plugins/xlsx/index.js +7 -5
- package/src/api/api-loader.js +12 -95
- package/src/api/define-api.js +1 -0
- package/src/client-runtime/constants.js +10 -0
- package/src/client-runtime/mount.js +2 -3
- package/src/dev/client-gen.js +458 -0
- package/src/discovery/scan-routes.js +5 -2
- package/src/file-router.js +2 -0
- package/src/helpers.js +53 -0
- package/src/pages/page-loader.js +12 -205
- package/src/plugin-manager.js +9 -1
- package/src/server.js +231 -140
- package/src/services/executor.js +9 -0
- package/src/services/registry.js +9 -1
- package/src/services/validator.js +38 -20
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# Cloud Migration Jobs & Zero-Downtime Database Deploy Guide
|
|
2
|
+
|
|
3
|
+
Webspresso provides an enterprise, zero-downtime database migration architecture designed specifically for Google Cloud Run, AWS ECS Fargate, AWS App Runner, and Kubernetes.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 🎯 The Cloud Migration Challenge
|
|
8
|
+
|
|
9
|
+
In containerized, autoscaling cloud platforms (such as Google Cloud Run or AWS ECS), running migrations inside the web application container startup command (`CMD ["node", "server.js"]`) causes severe production risks:
|
|
10
|
+
|
|
11
|
+
1. **Concurrent Migration Race Conditions**: If a new revision autoscales to multiple container instances, multiple containers attempt to run migrations simultaneously, causing deadlock or corrupt schema states.
|
|
12
|
+
2. **Crash Loops & Downtime**: If a migration script fails, web containers fail health checks and crash-loop, but HTTP traffic might already hit partially-migrated schemas.
|
|
13
|
+
3. **Stale Lock Hangs**: If a container is forcefully terminated by Cloud Run or ECS during migration, Knex's `knex_migrations_lock` table is left locked (`Migration table is already locked`), breaking all subsequent CI/CD deployments.
|
|
14
|
+
|
|
15
|
+
### The Webspresso Solution: Dedicated Pre-Deploy Cloud Migration Jobs
|
|
16
|
+
|
|
17
|
+
Webspresso solves this by separating database schema migrations into a single, isolated pre-deploy job that runs to completion before new web service revisions receive traffic:
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
[ Git Push / CI/CD ]
|
|
21
|
+
│
|
|
22
|
+
▼
|
|
23
|
+
[ Build Container Image ]
|
|
24
|
+
│
|
|
25
|
+
▼
|
|
26
|
+
[ Run Cloud Migration Job ] ──── (Fails? Stop deployment, keep old web version running)
|
|
27
|
+
│ (Success)
|
|
28
|
+
▼
|
|
29
|
+
[ Deploy Web Service Revision ] ─── (Safely route 100% traffic to new code)
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## 🚀 CLI Commands
|
|
35
|
+
|
|
36
|
+
### 1. Scaffold Migration Jobs (`add:migration-job`)
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
# For Google Cloud Run (cloudrun-job.yaml + deploy-cloudrun-migration.sh)
|
|
40
|
+
npx webspresso add:migration-job --platform cloudrun --app my-app --cloudsql "project:region:instance"
|
|
41
|
+
|
|
42
|
+
# For AWS ECS Fargate & App Runner (ecs-migration-task.json + deploy-aws-migration.sh)
|
|
43
|
+
npx webspresso add:migration-job --platform aws --app my-app --region us-east-1
|
|
44
|
+
|
|
45
|
+
# For Kubernetes / Helm (k8s/migration-job.yaml with pre-upgrade hooks)
|
|
46
|
+
npx webspresso add:migration-job --platform k8s --app my-app
|
|
47
|
+
|
|
48
|
+
# For all platforms
|
|
49
|
+
npx webspresso add:migration-job --platform all
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
## ⚡ Cloud-Safe Migration Options (`db:migrate`)
|
|
55
|
+
|
|
56
|
+
Webspresso's `db:migrate` command features cloud resilience flags powered by `core/orm/migrate-cloud.js`:
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
# Cloud-Safe mode: exponential backoff connection retries + automatic stale lock auto-healing
|
|
60
|
+
npx webspresso db:migrate --safe --env production
|
|
61
|
+
|
|
62
|
+
# Dry-run: preview pending migrations without altering the database
|
|
63
|
+
npx webspresso db:migrate --dry-run --env production
|
|
64
|
+
|
|
65
|
+
# Auto-heal hanging locks left by aborted serverless jobs
|
|
66
|
+
npx webspresso db:migrate --unlock-if-stale
|
|
67
|
+
|
|
68
|
+
# Force unlock migration table immediately
|
|
69
|
+
npx webspresso db:migrate --force-unlock
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## 🛡️ Zero-Downtime Migration Rules for Schema Evolution
|
|
75
|
+
|
|
76
|
+
When modifying production schemas under zero-downtime requirements, follow the **Expand and Contract (Parallel Run)** pattern:
|
|
77
|
+
|
|
78
|
+
| Goal | ❌ Risky (Breaks running old version) | ✅ Zero-Downtime Pattern |
|
|
79
|
+
| :--- | :--- | :--- |
|
|
80
|
+
| **Rename Column** | `table.renameColumn('old', 'new')` | 1. Add `new` column<br>2. Dual-write in application<br>3. Backfill data<br>4. Drop `old` column in subsequent deploy |
|
|
81
|
+
| **Drop Column** | `table.dropColumn('field')` | Remove column references in application code first, deploy code, then drop column in the next deployment. |
|
|
82
|
+
| **Add NOT NULL** | `table.string('col').notNullable()` | 1. Add nullable column or column with default value<br>2. Backfill existing rows<br>3. Alter to `notNullable` |
|
|
83
|
+
| **Add Index** | Raw `CREATE INDEX` (locks table) | In Postgres: use `CREATE INDEX CONCURRENTLY` |
|
|
84
|
+
|
|
85
|
+
---
|
|
86
|
+
|
|
87
|
+
## 🔍 Preflight Verification (`deploy:check`)
|
|
88
|
+
|
|
89
|
+
Webspresso's preflight scanner automatically audits migration strategies:
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
npx webspresso deploy:check --platform cloudrun --strict
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
If migrations exist and no cloud migration job is detected, it emits an actionable warning to prevent production container race conditions.
|
package/.agents/PLUGINS.md
CHANGED
|
@@ -161,12 +161,19 @@ Schema-driven headless CMS & inline content editing.
|
|
|
161
161
|
- **Features**: Dynamic content types & entries, public API (`/api/content/:type/:slug`), visual inline editor widget.
|
|
162
162
|
|
|
163
163
|
### 2.4 `uploadPlugin` (`plugins/upload`)
|
|
164
|
-
Multipart file upload manager with local
|
|
165
|
-
- **Options**: `{ path: '/api/upload', local: { destDir, publicBasePath }, maxBytes: 10000000, mimeAllowlist: ['image/png', 'image/jpeg'] }`
|
|
166
|
-
- **
|
|
164
|
+
Multipart file upload manager with local disk, AWS S3, Google Cloud Storage (GCS), and Cloudflare R2 drivers.
|
|
165
|
+
- **Options**: `{ path: '/api/upload', provider, local: { destDir, publicBasePath }, maxBytes: 10000000, mimeAllowlist: ['image/png', 'image/jpeg'], presigned: true, presignedRoute: '/api/upload/presign' }`
|
|
166
|
+
- **Supported Storage Drivers**:
|
|
167
|
+
- `createLocalFileProvider({ destDir, publicBasePath })`: Local disk storage (monolith / VPS).
|
|
168
|
+
- `createS3FileProvider({ s3Client, bucket, publicBaseUrl, prefix })`: AWS S3, MinIO, or S3-compatible endpoints with direct presigned PUT URLs (`createSignedUploadUrl`).
|
|
169
|
+
- `createGcsFileProvider({ bucket, storageClient, publicBaseUrl, prefix })`: Google Cloud Storage with V4 signed URLs for serverless direct uploads.
|
|
170
|
+
- `createR2FileProvider({ bucket, publicBaseUrl, prefix })`: Cloudflare Workers native R2 binding.
|
|
171
|
+
- **Direct-to-Cloud Presigned Uploads**: Set `presigned: true` to mount `POST /api/upload/presign`, allowing web browsers to upload directly to S3 or GCS without passing large payloads through memory-limited or ephemeral containers.
|
|
172
|
+
- **Static Asset Sync**: `webspresso assets:sync [--bucket <name>] [--provider s3|gcs|r2] [--dry-run]` to upload compiled JS/CSS bundles to cloud storage with `Cache-Control: public, max-age=31536000, immutable`.
|
|
167
173
|
- **Orphan Pruning**: `scanOrphanedUploads({ db, destDir })` and `pruneOrphanedUploads({ db, destDir, dryRun })` to detect and remove disk files no longer referenced by any database model, accompanied by CLI command `webspresso upload:prune [--dry-run]`.
|
|
168
174
|
|
|
169
175
|
|
|
176
|
+
|
|
170
177
|
### 2.5 `fileManagerPlugin` (`plugins/file-manager`)
|
|
171
178
|
Filesystem-first media library, directory explorer, and modal asset picker for Admin Panel.
|
|
172
179
|
- **Options**: `{ baseDir: './public/uploads', publicBasePath: '/uploads', apiPrefix: '/files', pageTitle: 'File Manager', maxUploadBytes: 10485760, allowedExtensions, allowedMimeTypes, includePage: true, lazy: false }`
|
package/.agents/QUEUE.md
CHANGED
|
@@ -92,6 +92,48 @@ queuePlugin({
|
|
|
92
92
|
});
|
|
93
93
|
```
|
|
94
94
|
|
|
95
|
+
### 4. `SqsQueueAdapter` (AWS SQS)
|
|
96
|
+
Zero-dependency AWS SQS adapter powered by native Node.js HTTP and standard AWS SigV4 request signing. Supports long-polling worker loops (`dequeue`), direct enqueue with delay seconds calculation, and AWS Lambda push batch execution (`processBatch`).
|
|
97
|
+
|
|
98
|
+
```js
|
|
99
|
+
const { SqsQueueAdapter } = require('webspresso');
|
|
100
|
+
|
|
101
|
+
// Worker daemon or long-polling runner:
|
|
102
|
+
const adapter = new SqsQueueAdapter({
|
|
103
|
+
queueUrl: process.env.AWS_SQS_QUEUE_URL,
|
|
104
|
+
region: process.env.AWS_REGION || 'us-east-1',
|
|
105
|
+
waitTimeSeconds: 10,
|
|
106
|
+
visibilityTimeout: 30,
|
|
107
|
+
});
|
|
108
|
+
|
|
109
|
+
// AWS Lambda SQS Event Handler:
|
|
110
|
+
exports.handler = async (event) => {
|
|
111
|
+
return adapter.processBatch(event.Records, queueManager);
|
|
112
|
+
};
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### 5. `PubSubQueueAdapter` (Google Cloud Pub/Sub)
|
|
116
|
+
Zero-dependency Google Cloud Pub/Sub adapter using standard REST API (`:publish`, `:pull`, `:acknowledge`, `:modifyAckDeadline`) with base64 payload serialization. Supports both background pull workers and Cloud Run / App Engine Push Subscription HTTP webhooks (`processPushRequest`).
|
|
117
|
+
|
|
118
|
+
```js
|
|
119
|
+
const { PubSubQueueAdapter } = require('webspresso');
|
|
120
|
+
|
|
121
|
+
const adapter = new PubSubQueueAdapter({
|
|
122
|
+
projectId: process.env.GOOGLE_CLOUD_PROJECT,
|
|
123
|
+
topic: 'app-jobs',
|
|
124
|
+
subscription: 'app-jobs-sub',
|
|
125
|
+
});
|
|
126
|
+
|
|
127
|
+
// Cloud Run Push Subscription Webhook Route:
|
|
128
|
+
app.post('/api/queue/push', async (req, res) => {
|
|
129
|
+
await adapter.processPushRequest(req, res, req.queue);
|
|
130
|
+
});
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
### 6. `WaitUntilQueueAdapter` & `CloudflareQueueAdapter`
|
|
134
|
+
- `WaitUntilQueueAdapter`: Serverless & Edge execution (Cloudflare Workers `ctx.waitUntil`, Vercel Edge).
|
|
135
|
+
- `CloudflareQueueAdapter`: Cloudflare Queues integration (`env.MY_QUEUE.send`).
|
|
136
|
+
|
|
95
137
|
---
|
|
96
138
|
|
|
97
139
|
## 4. File-Based Auto-Discovery (`jobs/`)
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# Native Zero-Dependency Rate Limiting Engine (`core/rate-limit`)
|
|
2
|
+
|
|
3
|
+
Webspresso features a high-performance, zero-dependency Rate Limiting Engine located in `core/rate-limit/`. It works seamlessly across both traditional Express middleware pipelines and modern edge/serverless contexts (such as Hono), providing standard RFC 6585 and IETF Draft rate limit headers with memory leak-free in-memory and distributed Redis sliding-window stores.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 🎯 Native Core vs. Plugin Comparison
|
|
8
|
+
|
|
9
|
+
| Feature | Core Rate Limiter (`core/rate-limit`) | Rate Limit Plugin (`plugins/rate-limit`) |
|
|
10
|
+
| :--- | :--- | :--- |
|
|
11
|
+
| **Dependencies** | **Zero external dependencies** (pure Node.js) | Requires `express-rate-limit` peer dependency |
|
|
12
|
+
| **Import** | `const { rateLimit, RateLimiter } = require('webspresso')` | `const { rateLimitPlugin } = require('webspresso')` |
|
|
13
|
+
| **Framework Agnostic** | ✅ Express `(req, res, next)` & Hono `(c, next)` | Express only |
|
|
14
|
+
| **Stores** | Native Sliding Window `MemoryRateLimitStore` & `RedisRateLimitStore` | `express-rate-limit` memory or external store |
|
|
15
|
+
| **Headers** | RFC 6585 `RateLimit-*`, `Retry-After`, legacy `X-RateLimit-*` | `standardHeaders: 'draft-7'`, `legacyHeaders` |
|
|
16
|
+
| **File Route Usage** | Universal middleware in API routes and standalone apps | Named middleware via `ctx.middlewares.rateLimit` |
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## 🚀 Quick Start
|
|
21
|
+
|
|
22
|
+
### 1. Basic Express Route Limiting
|
|
23
|
+
|
|
24
|
+
```javascript
|
|
25
|
+
const { rateLimit } = require('webspresso');
|
|
26
|
+
|
|
27
|
+
// 60 requests per minute
|
|
28
|
+
const apiLimiter = rateLimit({
|
|
29
|
+
window: '1m',
|
|
30
|
+
limit: 60,
|
|
31
|
+
});
|
|
32
|
+
|
|
33
|
+
// Use as standard middleware
|
|
34
|
+
app.use('/api', apiLimiter);
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
### 2. File-Based API Route (`pages/api/...`)
|
|
38
|
+
|
|
39
|
+
```javascript
|
|
40
|
+
const { defineApi, rateLimit } = require('webspresso');
|
|
41
|
+
|
|
42
|
+
const strictLimiter = rateLimit({
|
|
43
|
+
window: '10s',
|
|
44
|
+
limit: 5,
|
|
45
|
+
message: { error: 'Too many requests. Please slow down.' },
|
|
46
|
+
});
|
|
47
|
+
|
|
48
|
+
module.exports = defineApi({
|
|
49
|
+
middleware: [strictLimiter],
|
|
50
|
+
handler: async (req, res) => {
|
|
51
|
+
return res.json({ success: true });
|
|
52
|
+
},
|
|
53
|
+
});
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### 3. Distributed Redis Rate Limiting (Scale-Out)
|
|
57
|
+
|
|
58
|
+
```javascript
|
|
59
|
+
const { rateLimit, RedisRateLimitStore } = require('webspresso');
|
|
60
|
+
const Redis = require('ioredis');
|
|
61
|
+
|
|
62
|
+
const redis = new Redis(process.env.REDIS_URL);
|
|
63
|
+
|
|
64
|
+
const limiter = rateLimit({
|
|
65
|
+
window: '15m',
|
|
66
|
+
limit: 100,
|
|
67
|
+
store: new RedisRateLimitStore({
|
|
68
|
+
client: redis,
|
|
69
|
+
prefix: 'rl:auth:',
|
|
70
|
+
passOnStoreError: true, // Fail-open if Redis drops
|
|
71
|
+
}),
|
|
72
|
+
});
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### 4. Edge / Hono Application Usage
|
|
76
|
+
|
|
77
|
+
The middleware automatically detects Hono's `(c, next)` context signature:
|
|
78
|
+
|
|
79
|
+
```javascript
|
|
80
|
+
const { Hono } = require('hono');
|
|
81
|
+
const { rateLimit } = require('webspresso');
|
|
82
|
+
|
|
83
|
+
const app = new Hono();
|
|
84
|
+
const limiter = rateLimit({ limit: 100, window: '1m' });
|
|
85
|
+
|
|
86
|
+
app.use('*', (c, next) => limiter(c, next));
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## ⚙️ Options Reference
|
|
92
|
+
|
|
93
|
+
| Option | Type | Default | Description |
|
|
94
|
+
| :--- | :--- | :--- | :--- |
|
|
95
|
+
| `limit` / `max` | `number` | `60` | Maximum number of allowed requests within the time window. |
|
|
96
|
+
| `window` / `windowMs`| `string\|number` | `'1m'` | Window duration in ms or string shorthand (`'500ms'`, `'10s'`, `'1m'`, `'1h'`, `'1d'`). |
|
|
97
|
+
| `store` | `RateLimitStore` | `MemoryRateLimitStore` | Underlying storage engine. Defaults to sliding window in-memory store. |
|
|
98
|
+
| `keyGenerator` | `(req) => string` | `extractClientIp` | Function to identify client. Extracts Cloudflare, X-Forwarded-For, or socket IP by default. |
|
|
99
|
+
| `skip` | `(req) => boolean` | `() => false` | Condition to skip rate limiting (e.g. internal healthchecks). |
|
|
100
|
+
| `standardHeaders` | `boolean` | `true` | Send standard RFC headers (`RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, `RateLimit-Policy`). |
|
|
101
|
+
| `legacyHeaders` | `boolean` | `false` | Send legacy `X-RateLimit-*` headers. |
|
|
102
|
+
| `skipFailedRequests` | `boolean` | `false` | When `true`, failed requests (`statusCode >= 400`) decrement hit count upon finish. |
|
|
103
|
+
| `skipSuccessfulRequests`| `boolean` | `false` | When `true`, successful requests (`statusCode < 400`) decrement hit count upon finish. |
|
|
104
|
+
| `statusCode` | `number` | `429` | HTTP status code returned when limit is exceeded. |
|
|
105
|
+
| `message` | `string\|object` | `{ error: 'Too Many Requests', ... }` | Response payload on limit exceeded. |
|
|
106
|
+
| `handler` | `Function` | `null` | Custom callback when limit is exceeded `(req, res, next, info) => void`. |
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## 📡 HTTP Response Headers
|
|
111
|
+
|
|
112
|
+
When enabled, the rate limiter emits standard headers:
|
|
113
|
+
|
|
114
|
+
```http
|
|
115
|
+
HTTP/1.1 200 OK
|
|
116
|
+
RateLimit-Limit: 60
|
|
117
|
+
RateLimit-Remaining: 58
|
|
118
|
+
RateLimit-Reset: 42
|
|
119
|
+
RateLimit-Policy: 60;w=60
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
When the limit is exceeded (HTTP 429 Too Many Requests):
|
|
123
|
+
|
|
124
|
+
```http
|
|
125
|
+
HTTP/1.1 429 Too Many Requests
|
|
126
|
+
Retry-After: 42
|
|
127
|
+
RateLimit-Limit: 60
|
|
128
|
+
RateLimit-Remaining: 0
|
|
129
|
+
RateLimit-Reset: 42
|
|
130
|
+
RateLimit-Policy: 60;w=60
|
|
131
|
+
Content-Type: application/json
|
|
132
|
+
|
|
133
|
+
{"error":"Too Many Requests","message":"Too many requests, please try again later.","statusCode":429}
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## 🛡️ Reverse Proxy & Client IP Extraction
|
|
139
|
+
|
|
140
|
+
`extractClientIp(req)` extracts client IP in the following priority order:
|
|
141
|
+
1. `req.ip` (Express `trust proxy`)
|
|
142
|
+
2. `CF-Connecting-IP` (Cloudflare)
|
|
143
|
+
3. `Fastly-Client-IP` / `True-Client-IP` (Fastly / Akamai)
|
|
144
|
+
4. `X-Forwarded-For` (First client IP in comma-separated list)
|
|
145
|
+
5. `X-Real-IP` (Nginx / HAProxy)
|
|
146
|
+
6. `socket.remoteAddress` (Normalized IPv4 / IPv6)
|
|
147
|
+
7. Fallback `127.0.0.1`
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
# Stateless Scale-Out & Database Connection Pool Management Guide
|
|
2
|
+
|
|
3
|
+
When deploying Webspresso to serverless or auto-scaling cloud environments (Google Cloud Run, AWS App Runner, ECS Fargate, AWS Lambda, Kubernetes, or PM2 Cluster), multiple container instances run concurrently and scale up/down dynamically.
|
|
4
|
+
|
|
5
|
+
To ensure seamless operation without lost user sessions or database exhaustion, Webspresso provides native zero-dependency **Stateless Scale-Out** primitives:
|
|
6
|
+
1. **Distributed Redis Session Store** (`createRedisSessionStore`)
|
|
7
|
+
2. **Distributed Redis Session & Device Tracker** (`createRedisSessionTracker`)
|
|
8
|
+
3. **Built-in Lightweight RESP Redis Client** (`createMiniRedisClient`) & `InMemoryRedisClient`
|
|
9
|
+
4. **Serverless Database Connection Pool Optimizer** (`normalizeServerlessPoolConfig`, `isServerlessEnvironment`)
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## 🎯 Key Capabilities
|
|
14
|
+
|
|
15
|
+
### 1. Distributed Redis Session Store (`createRedisSessionStore`)
|
|
16
|
+
- Drop-in replacement for `express-session` MemoryStore.
|
|
17
|
+
- Stores session data in Redis with automatic TTL expiration synced to session cookies (`maxAge`).
|
|
18
|
+
- Implements the complete `express-session` Store contract: `get`, `set`, `destroy`, `touch`, `all`, `length`, `clear`.
|
|
19
|
+
- Zero external dependencies: works with any standard client (`ioredis`, `redis` v4/v3, or the built-in `createMiniRedisClient`).
|
|
20
|
+
- **Auto-Detection**: Setting `REDIS_URL=redis://...` or `SESSION_STORE=redis` in your environment automatically wires the Redis session store without manual boilerplate!
|
|
21
|
+
|
|
22
|
+
### 2. Distributed Redis Device & Session Tracker (`createRedisSessionTracker`)
|
|
23
|
+
- Replaces or supplements Knex database session tracking across ephemeral instances.
|
|
24
|
+
- Maintains user active sessions (`user:sessions:<userId>`) and device metadata (`sess:device:<sessionId>`).
|
|
25
|
+
- Supports remote single-session revocation, "revoke other sessions", and "force logout from all devices".
|
|
26
|
+
|
|
27
|
+
### 3. Serverless DB Connection Pool Optimizer (`normalizeServerlessPoolConfig`)
|
|
28
|
+
- Prevents database connection exhaustion on managed databases (Google Cloud SQL, AWS RDS / Aurora) when container replicas scale from 1 to 50+.
|
|
29
|
+
- Automatically detects serverless platforms (`K_SERVICE` on Cloud Run, `AWS_EXECUTION_ENV` / `AWS_LAMBDA_FUNCTION_NAME` on AWS):
|
|
30
|
+
- Sets `pool.min: 0` (so idle containers release database sockets).
|
|
31
|
+
- Clamps `pool.max: 5` (or `2` on Lambda) per container instance.
|
|
32
|
+
- Reduces `acquireTimeoutMillis` from Knex's 60s default to 10s.
|
|
33
|
+
- Sets safe `idleTimeoutMillis: 30000`, `createTimeoutMillis: 5000`.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## 🚀 Quick Start & Usage
|
|
38
|
+
|
|
39
|
+
### 1. Zero-Configuration Cloud Run / ECS Deployment
|
|
40
|
+
|
|
41
|
+
Simply provide `REDIS_URL` in your container environment variables:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
# In Cloud Run or AWS ECS Task Definition
|
|
45
|
+
REDIS_URL=redis://default:secret@memorystore-or-elasticache:6379
|
|
46
|
+
SESSION_SECRET=a_very_secure_random_production_secret_32_chars
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Webspresso automatically detects `REDIS_URL` and activates both `createRedisSessionStore` and `normalizeServerlessPoolConfig`.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
### 2. Explicit Auth Session Configuration
|
|
54
|
+
|
|
55
|
+
```javascript
|
|
56
|
+
// server.js
|
|
57
|
+
const { createApp, createRedisSessionStore } = require('webspresso');
|
|
58
|
+
|
|
59
|
+
const app = createApp({
|
|
60
|
+
auth: {
|
|
61
|
+
session: {
|
|
62
|
+
secret: process.env.SESSION_SECRET,
|
|
63
|
+
// Pass custom store or let Webspresso auto-detect REDIS_URL
|
|
64
|
+
store: createRedisSessionStore({
|
|
65
|
+
url: process.env.REDIS_URL,
|
|
66
|
+
prefix: 'myapp:sess:',
|
|
67
|
+
ttl: 86400, // 24 hours
|
|
68
|
+
}),
|
|
69
|
+
},
|
|
70
|
+
},
|
|
71
|
+
server: {
|
|
72
|
+
compression: true,
|
|
73
|
+
},
|
|
74
|
+
});
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
### 3. Explicit Database Connection Pool Configuration
|
|
80
|
+
|
|
81
|
+
```javascript
|
|
82
|
+
// webspresso.db.js
|
|
83
|
+
module.exports = {
|
|
84
|
+
client: 'pg',
|
|
85
|
+
connection: process.env.DATABASE_URL,
|
|
86
|
+
// Enable serverless pool optimization explicitly
|
|
87
|
+
serverlessPool: true,
|
|
88
|
+
pool: {
|
|
89
|
+
// Override defaults if using a pooler like PgBouncer or RDS Proxy
|
|
90
|
+
min: 0,
|
|
91
|
+
max: 5,
|
|
92
|
+
},
|
|
93
|
+
};
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
### 4. Production Preflight Audit (`webspresso deploy:check`)
|
|
99
|
+
|
|
100
|
+
Run the preflight check before deploying to verify that your session store and connection pool are properly configured for scale-out:
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
npx webspresso deploy:check
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
The preflight check will:
|
|
107
|
+
- ✅ Verify distributed session store (Redis) is active in production instead of ephemeral MemoryStore.
|
|
108
|
+
- ⚠️ Flag database connection pools with excessively large `max` (> 10) on autoscaling serverless containers.
|
|
109
|
+
- ⚠️ Check that `pool.min` is set to 0 for serverless scale-to-zero workloads.
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# Cloud Secrets Management Guide (`core/secrets` & `secrets:pull`)
|
|
2
|
+
|
|
3
|
+
Webspresso features an enterprise, zero-dependency Cloud Secrets Management system (`core/secrets`) designed to eliminate plain-text `.env` files and secret leakage in containerized and serverless environments (Google Cloud Run, AWS App Runner, ECS Fargate, Kubernetes).
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 🎯 Key Capabilities
|
|
8
|
+
|
|
9
|
+
1. **Zero External Dependencies**:
|
|
10
|
+
- Interacts directly with cloud vault REST APIs using native Node.js standard libraries (`https`, `crypto`).
|
|
11
|
+
- **Google Cloud Secret Manager**: Automatically acquires OAuth2 tokens from the local GCP Metadata Server in Cloud Run/GKE, exchanges RSA-SHA256 signed JWTs from `GOOGLE_APPLICATION_CREDENTIALS`, or uses static tokens.
|
|
12
|
+
- **AWS Secrets Manager & SSM Parameter Store**: Employs an embedded, standards-compliant AWS Signature Version 4 (SigV4) signer with HMAC-SHA256.
|
|
13
|
+
2. **In-Memory Caching with TTL & Invalidation**:
|
|
14
|
+
- Caches secret values in RAM with a configurable TTL (default: 5 minutes) to avoid redundant HTTP roundtrips.
|
|
15
|
+
- Supports single secret cache busting (`secrets.invalidate(name)`) and full cache clearing (`secrets.clearCache()`) for automated secret rotation.
|
|
16
|
+
3. **JSON Secret Bundle Unpacking & Dot-Notation Subkeys**:
|
|
17
|
+
- Access specific fields within JSON bundles using dot-notation:
|
|
18
|
+
`await secrets.get('prod/app/config.DATABASE_URL')`
|
|
19
|
+
- Unpack entire JSON bundles into `process.env`:
|
|
20
|
+
`await secrets.loadBundle('prod/app/config')`
|
|
21
|
+
4. **Local Developer Synchronization (`webspresso secrets:pull`)**:
|
|
22
|
+
- Pulls secrets from Google Cloud or AWS into a local `.env.local` file for local development without committing credentials to Git.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 🚀 Usage in Application Bootstrap
|
|
27
|
+
|
|
28
|
+
### 1. Hydrating Environment at Application Boot
|
|
29
|
+
|
|
30
|
+
Call `loadSecrets()` before initializing your database or server:
|
|
31
|
+
|
|
32
|
+
```javascript
|
|
33
|
+
// server.js
|
|
34
|
+
const { createApp, loadSecrets } = require('webspresso');
|
|
35
|
+
|
|
36
|
+
async function start() {
|
|
37
|
+
// Load secrets from Google Cloud Secret Manager or AWS before bootstrapping
|
|
38
|
+
await loadSecrets({
|
|
39
|
+
provider: 'auto', // 'cloudrun' | 'gcp' | 'aws' | 'auto'
|
|
40
|
+
secrets: {
|
|
41
|
+
SESSION_SECRET: 'session-secret:latest',
|
|
42
|
+
DATABASE_URL: 'rds-database-url:latest',
|
|
43
|
+
},
|
|
44
|
+
// Or unpack a single JSON bundle holding all config:
|
|
45
|
+
// bundle: 'prod/webspresso/app-secrets',
|
|
46
|
+
});
|
|
47
|
+
|
|
48
|
+
const app = createApp({
|
|
49
|
+
server: { port: process.env.PORT || 3000 },
|
|
50
|
+
});
|
|
51
|
+
|
|
52
|
+
await app.listen();
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
start();
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
---
|
|
59
|
+
|
|
60
|
+
## 🛠️ Standalone `SecretManager` Usage
|
|
61
|
+
|
|
62
|
+
```javascript
|
|
63
|
+
const { createSecretManager } = require('webspresso/core/secrets');
|
|
64
|
+
|
|
65
|
+
const secrets = createSecretManager({
|
|
66
|
+
provider: 'auto', // Auto-detects GCP or AWS based on environment markers
|
|
67
|
+
cacheTtl: '10m', // 10 minutes cache TTL
|
|
68
|
+
projectId: 'my-gcp-project', // optional for GCP
|
|
69
|
+
region: 'us-east-1', // optional for AWS
|
|
70
|
+
});
|
|
71
|
+
|
|
72
|
+
// 1. Fetch a single secret
|
|
73
|
+
const apiKey = await secrets.get('STRIPE_SECRET_KEY');
|
|
74
|
+
|
|
75
|
+
// 2. Fetch with fallback or enforce required
|
|
76
|
+
const optionalToken = await secrets.get('SLACK_WEBHOOK', { fallback: null });
|
|
77
|
+
const mustExist = await secrets.get('APP_KEY', { required: true });
|
|
78
|
+
|
|
79
|
+
// 3. Dot-notation subkey access inside JSON secret bundles
|
|
80
|
+
const dbPassword = await secrets.get('prod/database.password');
|
|
81
|
+
|
|
82
|
+
// 4. Invalidate cache on rotation
|
|
83
|
+
secrets.invalidate('STRIPE_SECRET_KEY');
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
## 💻 CLI Synchronization Tool (`secrets:pull`)
|
|
89
|
+
|
|
90
|
+
Pulls cloud secrets into `.env.local` for local development:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
# Auto-detect cloud platform and pull expected secrets into .env.local
|
|
94
|
+
npx webspresso secrets:pull
|
|
95
|
+
|
|
96
|
+
# Explicitly pull from Google Cloud Run / Secret Manager
|
|
97
|
+
npx webspresso secrets:pull --platform cloudrun -s SESSION_SECRET DATABASE_URL
|
|
98
|
+
|
|
99
|
+
# Explicitly pull from AWS Secrets Manager
|
|
100
|
+
npx webspresso secrets:pull --platform aws -s SESSION_SECRET DATABASE_URL
|
|
101
|
+
|
|
102
|
+
# Unpack a JSON bundle to stdout (dry run)
|
|
103
|
+
npx webspresso secrets:pull --bundle prod/app/config --dry-run
|
|
104
|
+
|
|
105
|
+
# Output as JSON
|
|
106
|
+
npx webspresso secrets:pull -s API_KEY DB_PASS --format json -o secrets.json
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## 🔒 Security Best Practices
|
|
112
|
+
|
|
113
|
+
1. **Never commit `.env` or `.env.local`**: Ensure `.env.local` is listed in `.gitignore` and `.gcloudignore`.
|
|
114
|
+
2. **Minimal IAM Permissions**:
|
|
115
|
+
- GCP: Grant the Cloud Run service account `roles/secretmanager.secretAccessor`.
|
|
116
|
+
- AWS: Grant the ECS Task role `secretsmanager:GetSecretValue` and `ssm:GetParameter`.
|
|
117
|
+
3. **Use Short TTL for Rotating Secrets**: If secrets rotate frequently (e.g. daily), configure `cacheTtl: '15m'` or invalidate on rotation webhook triggers.
|
package/AGENTS.md
CHANGED
|
@@ -19,6 +19,12 @@ When tasked with a feature or bugfix, **immediately consult the corresponding to
|
|
|
19
19
|
| **Background Jobs & Worker Queue** | [`.agents/QUEUE.md`](.agents/QUEUE.md) | `jobs/` |
|
|
20
20
|
| **Decoupled Events & Signals** | [`.agents/EVENTS.md`](.agents/EVENTS.md) | `core/events/`, `listeners/` |
|
|
21
21
|
| **Fault Tolerance / Circuit Breaker / Retry** | [`.agents/RESILIENCE.md`](.agents/RESILIENCE.md) | `core/resilience/` |
|
|
22
|
+
| **Stateless Scale-Out (Redis Sessions & DB Pool)** | [`.agents/SCALE_OUT.md`](.agents/SCALE_OUT.md) | `core/auth/redis-session.js`, `core/orm/pool.js` |
|
|
23
|
+
| **Cloud Secrets (GCP & AWS Secret Manager)** | [`.agents/SECRETS.md`](.agents/SECRETS.md) | `core/secrets/`, `bin/commands/secrets-pull.js` |
|
|
24
|
+
| **Cloud Metrics & Distributed Tracing** | [`.agents/METRICS.md`](.agents/METRICS.md) | `core/metrics/`, `/metrics` |
|
|
25
|
+
| **Cloud Migration Jobs (Zero-Downtime DB Deploy)** | [`.agents/MIGRATION_JOBS.md`](.agents/MIGRATION_JOBS.md) | `bin/commands/add-migration-job.js`, `core/orm/migrate-cloud.js` |
|
|
26
|
+
| **Rate Limiting Engine (Zero-Dependency & Redis)** | [`.agents/RATE_LIMIT.md`](.agents/RATE_LIMIT.md) | `core/rate-limit/` |
|
|
27
|
+
| **Feature Flags & Canary Splitter (`core/flags`)** | [`.agents/FLAGS.md`](.agents/FLAGS.md) | `core/flags/` |
|
|
22
28
|
| **API Signatures & Types Verification** | [`index.d.ts`](index.d.ts) | **Canonical Ground Truth** |
|
|
23
29
|
|
|
24
30
|
---
|
|
@@ -63,6 +69,10 @@ Every coding agent working on Webspresso **MUST** follow this 4-step execution d
|
|
|
63
69
|
- **Native Cloud & Kubernetes Health & Readiness Probes**: Zero-dependency RFC-compliant liveness (`/healthz`), readiness (`/readyz`), and combined diagnostic (`/health`) endpoints configured via `createApp({ healthCheck: true })`. Includes non-blocking event loop lag monitoring (`perf_hooks.monitorEventLoopDelay`), database/queue connectivity tests, and automatic HTTP 503 draining integration with `ShutdownManager` when pods enter graceful termination.
|
|
64
70
|
- **High-Performance In-Memory LRU & Tagged Cache Engine**: Zero-dependency memory-bounded caching (`core/cache`) configured via `createApp({ cache: { maxEntries: 1000, maxMemoryMb: 50, defaultTtl: '15m' } })` or shorthand `cache: true|false`. Provides O(1) LRU eviction, byte budget constraints preventing OOM, leak-free tag invalidation (`TaggedLruCache`), stampede protection (`getOrSet`), and powers ORM query caching and Zod schema caching.
|
|
65
71
|
- **Enterprise Resilience & Fault Tolerance (Circuit Breaker & Retry Engine)**: Zero-dependency resilience system (`core/resilience`) featuring a three-state `CircuitBreaker` (`CLOSED`, `OPEN`, `HALF_OPEN`), rolling window failure rate tracking, per-call execution timeouts, graceful fallbacks, and an exponential backoff retry engine (`retry`, `withRetry`) with randomized jitter and transient error detection (`ECONNRESET`, 429, 5xx). Seamlessly integrated into the Services Layer (`defineService({ circuitBreaker, retry })`) and exposed via `app.breakers`, `app.getBreaker()`, `req.breaker()`, and `req.retry()`.
|
|
72
|
+
- **Stateless Scale-Out & Database Connection Pool Management**: Zero-dependency Redis session store (`createRedisSessionStore`), distributed device tracking (`createRedisSessionTracker`), lightweight RESP Redis client, serverless DB connection pool optimizer (`normalizeServerlessPoolConfig`), and scale-out audits in `webspresso deploy:check`. Automatically activates distributed sessions when `REDIS_URL` or `SESSION_STORE=redis` is configured.
|
|
73
|
+
- **Enterprise Cloud Secrets Management**: Zero-dependency integration with Google Cloud Secret Manager & AWS Secrets Manager/SSM Parameter Store (`core/secrets`), in-memory caching with TTL (`cacheTtl: '5m'`), JSON secret bundle unpacking, dot-notation subkeys (`app.DB_PASSWORD`), bootstrap environment hydration (`loadSecrets()`), and local sync CLI (`webspresso secrets:pull`).
|
|
74
|
+
- **Cloud Metrics & Distributed Tracing**: Zero-dependency Prometheus exposition format (`/metrics`), JSON telemetry (`/metrics.json`), HTTP latency histograms, Node.js event loop/memory/pool/cache collectors, and W3C `traceparent` & `x-trace-id` propagation (`core/metrics`) configured via `createApp({ metrics: true })`.
|
|
75
|
+
- **Cloud Migration Jobs & Zero-Downtime Database Deploy**: Pre-deploy Cloud Run Jobs, AWS ECS Fargate one-off tasks, and Kubernetes Helm hooks (`add:migration-job`), safe retry connections, and automatic stale lock healing (`core/orm/migrate-cloud`, `webspresso db:migrate --safe`).
|
|
66
76
|
- **Custom Error & 404 Pages**: Configured via `createApp({ errorPages: { notFound, serverError, timeout } })`. Supports template paths (e.g. `notFound: '404.njk'`) or custom handler functions, passing `{ fsy, locale, isDev, url, method }` to templates.
|
|
67
77
|
- **Cold Start & High-Performance Optimizations**:
|
|
68
78
|
- Unified single-pass route scanner (`discoverAppRoutes`).
|