webspresso 0.0.95 → 0.0.96

package/.agents/internal/TECHNICAL_REVIEW.md

@@ -0,0 +1,428 @@
+# Webspresso – Technical Status & Phased Stabilization Report
+
+```text
+Project: Webspresso
+Review date: 2026-08-28
+Current version: 0.0.91
+Repository state / commit: e027fc7ba47635f6d8013fed5f207f54db602122 (branch: current)
+Review scope: Complete codebase analysis (core, src, plugins, bin, utils, tests, templates)
+Overall status: Healthy & passing (176 test files, 2,109 unit/integration/security tests passing), with stabilized architecture, unified validation core, isolated experimental kernel, and formal request context containers.
+```
+
+---
+
+## 1. Executive Summary
+
+Webspresso is a feature-rich web framework for Node.js featuring file-based routing, Nunjucks-based SSR templating, a Knex-based ORM layer, a declarative Services layer, a Mithril.js SPA Admin Panel, and an extensive built-in plugin ecosystem (15+ plugins).
+
+The current test suite (`vitest`) runs at **100% pass rate across 176 test files and 2,109 tests**. Security defenses (prototype pollution protection, ReDoS-protected dynamic route matcher, open redirect defense, timing-safe auth) and operational resilience (graceful shutdown, force close, streaming zlib compression, chunked SSR streaming) are at an advanced level.
+
+Key architectural aspects and historical areas of duality addressed during stabilization:
+
+1. **Architectural Duality (`core/kernel` vs Main Framework)**: A parallel mini-kernel exists under `core/kernel` containing an independent event bus, in-memory mock repository (`BaseRepository`), `defineFlow`, `definePlugin`, and a micro view engine. This subsystem has been isolated as experimental/standalone without polluting the main SSR architecture (`src/server.js`, `src/services`, `core/orm`).
+2. **Implicit Global State Reduction (`src/app-context.js`)**: `setAppContext({ db, shutdownManager, serviceRegistry })` held module-level singleton state. The framework has been refactored to pass state through Express `req.context` / `ctx` containers, keeping `app-context.js` strictly as a backward-compatible fallback.
+3. **Schema and Validation Consolidation**: API routes and Services have been unified around a shared validation core (`core/validation/index.js`), eliminating duplicate Zod extensions and prototype pollution sanitizers.
+4. **Express & Node.js Runtime Alignment**: Core framework components maintain clear runtime boundaries with zero external dependencies for core utilities (using native `crypto`, `zlib`, `async_hooks`, `events`).
+
+This report provides a comprehensive technical inventory, risk register, and completed stabilization backlog.
+
+---
+
+## 2. Current Architecture
+
+Webspresso consists of 6 primary subsystems:
+
+```mermaid
+graph TD
+    CLI[CLI Tooling / bin] --> AppInit[src/server.js: createApp]
+    AppInit --> Express[Express 5 Server]
+    AppInit --> PluginMgr[src/plugin-manager.js]
+    AppInit --> SvcRegistry[src/services: ServiceRegistry]
+    AppInit --> ShutdownMgr[core/shutdown: ShutdownManager]
+    
+    Express --> FileRouter[src/file-router.js: mountPages]
+    FileRouter --> SSR[pages/**/*.njk + .js Loaders]
+    FileRouter --> API[pages/api/**/*.js Endpoints]
+    
+    SSR --> SvcRegistry
+    API --> SvcRegistry
+    
+    SvcRegistry --> ORM[core/orm: Repositories & Models]
+    ORM --> Knex[Knex DB Connection Pool]
+    ORM --> ORMCache[core/orm/cache: Query Cache]
+    ORM --> ModelEvents[core/orm/events: Django-style Signals]
+    
+    PluginMgr --> AdminSPA[plugins/admin-panel: Mithril SPA]
+    PluginMgr --> ContentCMS[plugins/content: Headless CMS]
+    PluginMgr --> RealtimeLayer[core/realtime: WS/SSE/Socket.IO]
+    PluginMgr --> BackgroundQueue[core/queue: In-Memory/Knex/Redis]
+
+    subgraph "Parallel Mini-Kernel (Isolated Standalone Subsystem)"
+      Kernel[core/kernel] --> KernelEvents[core/kernel/events]
+      Kernel --> KernelFlows[core/kernel/flow]
+      Kernel --> KernelViews[core/kernel/view]
+      Kernel --> BaseRepo[core/kernel/base-repository]
+    end
+```
+
+### 2.1 Core SSR & File-Based Router (`src/server.js`, `src/file-router.js`)
+- **Directory Scanning**: Scans `pages/`. Maps `.njk` templates to SSR routes and `pages/api/**/*.js` to REST API endpoints.
+- **Parameter Resolution**: Converts `[param]` → `:param`, `[...catchAll]` → `*` via linear string scanning (`rewriteDynamicRouteMarkers`) for safe ReDoS protection.
+- **Registration Ordering**: Strict 4-tier registration: Static literal paths → Deep dynamic paths → Shallow dynamic paths → Catch-all wildcards (`routeRegistrationMeta`).
+- **Data Loading & Lifecycle**: Executes `_hooks.js` and route-specific `.js` companions (`load()`, `meta()`) upon matching.
+- **Template Engine**: Safe Nunjucks engine with circular inheritance recursion protection (`configureSafeNunjucks` + `renderStackStorage` AsyncLocalStorage).
+- **Chunked Transfer**: `res.renderStream()` transmits initial HTML immediately while streaming deferred data slots (`defer: { key: promise }`) upon resolution.
+
+### 2.2 Services Layer (`src/services/`)
+- **Auto-Discovery**: Scans `services/` mapping files to dot-notated names (`user.create`) and camelCase aliases (`user.reset-password` → `user.resetPassword`).
+- **Declarative Features**: Supports Zod `schema`, `auth` (boolean, role string/array, predicate), `timeout` (ms), `transaction: true` (automatic ACID transaction propagation), and `cache: { ttl, key }`.
+- **Circular Call Detection**: Uses `Symbol.for('webspresso.service.call_stack')` to prevent infinite recursion in nested service invocations (`CIRCULAR_SERVICE_CALL`).
+- **Sensitive Data Masking**: Automatically redacts sensitive keys (`password`, `token`, `secret`, `cvv`) from error logs and traces.
+
+### 2.3 ORM & Database Layer (`core/orm/`)
+- **Model Definition**: `defineModel({ name, table, schema, relations, scopes, hidden, admin, cache })`.
+- **Schema Types**: `zdb.string()`, `zdb.integer()`, `zdb.boolean()`, `zdb.json()`, `zdb.file()`, `zdb.nanoid()`.
+- **Repository Pattern**: `db.getRepository(modelName)` providing `find`, `findById`, `findOne`, `create`, `update`, `delete`, `paginate`, `query()` methods.
+- **Ambient Transactions**: `runWithAmbientTransaction(trx, callback)` uses `AsyncLocalStorage` to automatically bind queries and nested services to the active transaction without explicit `trx` arguments.
+- **Query Caching**: `auto` (PK/find caching) and `smart` (tag-based invalidation) strategies.
+- **Signals / Events**: `ModelEvents` class providing Django-style signals (`beforeCreate`, `afterSave`, `beforeDelete`).
+
+### 2.4 Mithril.js Admin Panel SPA (`plugins/admin-panel/`)
+- Independent single-page application mounted at `/_admin`.
+- Dynamic model CRUD forms, customizable widgets, custom pages (`componentFile` or `pagesDir`), single and bulk actions.
+- Staff authentication session (`req.session.adminUser`) is strictly isolated from public site auth (`req.user`).
+
+### 2.5 Plugin Ecosystem (`plugins/`)
+- Synchronous registration (`registerSync`), route readiness hook (`onRoutesReady`), CSP directive merging, and reverse-order `disposer` cleanup on shutdown.
+
+---
+
+## 3. HTTP Request Lifecycle
+
+The diagram below illustrates the verified HTTP request lifecycle from `src/server.js` and `src/file-router.js`:
+
+```text
+HTTP Request (Client)
+  ↓
+Express 5.x App
+  ↓
+Services Caller Injection Middleware (req.service = ...)
+  ↓
+SSR Streaming Helper Injection Middleware (res.renderStream = ...)
+  ↓
+Shutdown Draining Check (isShuttingDown ? Connection: close : next)
+  ↓
+Security Headers (Helmet: CSP, HSTS, XSS, Referrer)
+  ↓
+HTTP Response Compression (Streaming zlib Brotli/Gzip/Deflate)
+  ↓
+Request Timeout Middleware (connect-timeout)
+  ↓
+Body Parsers (express.json, express.urlencoded)
+  ↓
+Halt on Timedout Guard
+  ↓
+Auth Session & JWT Middlewares (req.user, req.auth)
+  ↓
+Static Assets Handler (express.static if publicDir exists)
+  ↓
+Client Runtime Assets (/__webspresso/client-runtime/* for Alpine/Swup)
+  ↓
+File-Based Routing Matcher
+  ├── [API Route] pages/api/**/*.js
+  │     ↓
+  │     Route-level Middleware (preResolvedMw)
+  │     ↓
+  │     compileSchema & applySchema (Zod validation -> req.input)
+  │     ↓
+  │     Handler Function fn(req, res, next)
+  │     ↓
+  │     JSON Response res.json(...)
+  │
+  └── [SSR Route] pages/**/*.njk
+        ↓
+        Global & Route Hooks: onRequest → onRoute → beforeMiddleware
+        ↓
+        Route-level Middleware (preResolvedPageMw)
+        ↓
+        Global & Route Hooks: afterMiddleware → beforeLoad
+        ↓
+        Page Data Loader: load(req, ctx) → ctx.data
+        ↓
+        Global & Route Hooks: afterLoad
+        ↓
+        Page Meta Generator: meta(req, ctx) → ctx.meta
+        ↓
+        Global & Route Hooks: beforeRender
+        ↓
+        Nunjucks Template Render / Streaming (renderStream)
+        ↓
+        Content CMS Inline Edit Injection (if applicable)
+        ↓
+        Global & Route Hooks: afterRender
+        ↓
+        HTML Response (res.send / chunked stream)
+  ↓
+[If Unmatched] -> 404 Handler (Custom function / 404.njk + 404.js load() / Default HTML/JSON)
+  ↓
+[If Error Thrown] -> Central Error Boundary (normalizeError → app.setErrorHandler → 500.njk / Default HTML/JSON)
+```
+
+---
+
+## 4. Service Execution Lifecycle
+
+The verified service execution pipeline from `src/services/executor.js` and `src/services/registry.js`:
+
+```text
+serviceRegistry.call(name, input, ctx, options) / req.service(name, input)
+  ↓
+1. Lookup: Name & CamelCase Alias Resolution (registry.get) -> If missing: NotFoundError ('SERVICE_NOT_FOUND')
+  ↓
+2. Authorization Guard (checkServiceAuth)
+   ├── Boolean true: Session check (ctx.user || req.user) -> If missing: UnauthorizedError (401)
+   ├── Role string / Role array: Role validation -> If unauthorized: ForbiddenError (403)
+   └── Predicate function (user, ctx) => boolean -> If rejected: ForbiddenError (403)
+  ↓
+3. Circular Call Guard: Symbol.for('webspresso.service.call_stack') check -> If cycle detected: WebspressoError ('CIRCULAR_SERVICE_CALL')
+  ↓
+4. Child Context Fork: Creates executionContext (new call stack, nested service caller, invalidate/clearCache methods)
+  ↓
+5. Schema Validation: validateServiceInput (Zod parse/safeParse, prototype pollution sanitization) -> If invalid: ValidationError (422)
+  ↓
+6. Ambient Transaction Propagation (options.transaction === true || serviceDef.transaction === true)
+   ├── Is transaction already active? (hasAmbientTransaction() || ctx.trx)
+   │     ├── YES: Reuse existing transaction within isolated child context.
+   │     └── NO: Start knex.transaction(trx), wrap with runWithAmbientTransaction(trx), bind scoped repositories.
+  ↓
+7. Cache / Memoization (if serviceDef.cache configured)
+   ├── Cache Hit: Return cached result (database and handler skipped).
+   └── Cache Miss: Execute handler, store result in cache.
+  ↓
+8. Timeout Race Boundary (serviceDef.timeout or options.timeout ms) -> If exceeded: WebspressoError ('SERVICE_TIMEOUT', 504)
+  ↓
+9. Handler Execution: handler(validatedInput, activeCtx)
+  ↓
+10. Return Result / Automatic Rollback & Sensitive Log Redaction on Failure
+```
+
+---
+
+## 5. Data, Transaction & Cache Lifecycle
+
+```text
+Controller / Loader / Service
+  ↓
+db.getRepository(modelName)
+  ↓
+Repository Instance (core/orm/repository.js)
+  ↓
+Resolve Active Knex:
+  ├── Is Ambient Transaction Storage active? (AsyncLocalStorage.getStore().trx)
+  │     ├── YES: Route query to trx connection (atomic execution).
+  │     └── NO: Use primary Database Connection Pool.
+  ↓
+Apply Scopes (Soft delete: deleted_at IS NULL, Multi-tenant: tenant_id = ...)
+  ↓
+Find / Query Operation:
+  ├── Is ORM Cache Layer active? (auto / smart)
+  │     ├── Cache Hit: Fingerprint match -> Return from Memory/Provider.
+  │     └── Cache Miss: Query DB -> ModelEvents ('beforeFind') -> Knex Query -> Deserialization (JSON fields) -> Eager Load Relations -> ModelEvents ('afterFind') -> Write to Cache.
+  │
+Mutation Operation (create / update / delete):
+  ↓
+ModelEvents ('beforeCreate' / 'beforeUpdate' / 'beforeDelete') -> Cancellable.
+  ↓
+Schema Serializers (JSON fields, nanoid generation, timestamps)
+  ↓
+Knex Query Execution (INSERT / UPDATE / DELETE)
+  ↓
+ModelEvents ('afterCreate' / 'afterSave' / 'afterDelete')
+  ↓
+ORM Cache Invalidation: Invalidate affected model tables and associated tags.
+```
+
+---
+
+## 6. Plugin Lifecycle
+
+```text
+createApp({ plugins: [pluginA, pluginB, ...] })
+  ↓
+1. Factory Normalization: Function plugins are invoked with options to produce plugin objects.
+  ↓
+2. Dependency Resolution: Plugin dependencies (`dependencies: { auth: '^1.0.0' }`) validated via semver and sorted via Topological Sort.
+  ↓
+3. Registration Phase (pluginManager.registerSync):
+   - Invokes `register(ctx)` for each plugin.
+   - Named middlewares added to `ctx.middlewares`.
+   - Shutdown disposers registered via `ctx.shutdownManager.registerDisposer`.
+   - Plugin `plugin.csp` directives merged into Helmet CSP policy.
+  ↓
+4. Route Discovery & Setup:
+   - `mountPages` registers file-based routes.
+   - `pluginManager.setRoutes(routeMetadata)` exposes route metadata to plugins.
+  ↓
+5. onRoutesReady Hook Phase:
+   - Invokes `onRoutesReady(ctx)` for each plugin.
+   - Plugins attach custom routes and Nunjucks helpers via `ctx.addRoute()`, `ctx.addHelper()`, `ctx.addFilter()`.
+  ↓
+6. Runtime Operation:
+   - Routes and templates consume plugin APIs via `fsy.*`, `usePlugin(name)`, `req.service()`.
+  ↓
+7. Shutdown / Dispose Phase:
+   - Server shutdown triggered (`app.close()` or SIGTERM/SIGINT) via `ShutdownManager`.
+   - Registered `disposer` functions execute asynchronously in **reverse registration order** (LIFO).
+```
+
+---
+
+## 7. Public API Inventory
+
+| API / Export | Location | Classification | Rationale & Notes |
+| :--- | :--- | :--- | :--- |
+| `createApp` (SSR) | `src/server.js`, `index.js` | **Stable candidate** | Main framework entry point. Core contract across 170+ tests. |
+| `mountPages` / File Router | `src/file-router.js` | **Stable candidate** | File-based routing and server loader mechanism. |
+| `defineModel` / `zdb` | `core/orm` | **Stable candidate** | Knex-based ORM and schema definition builder. |
+| `createDatabase` | `core/orm` | **Stable candidate** | Knex + repository management. |
+| `createServiceRegistry` / `defineService` | `src/services` | **Stable candidate** | Services layer API, Zod validation, and RBAC guards. |
+| `errors.*` (WebspressoError, HttpError, etc.) | `core/errors` | **Stable candidate** | Central framework exception hierarchy. |
+| `ShutdownManager` / `NodeHttpAdapter` | `core/shutdown` | **Stable candidate** | Graceful shutdown and socket draining manager. |
+| `res.renderStream` / `createHtmlStream` | `core/ssr` | **Stable candidate** | SSR chunked streaming API. |
+| `fsy.*` helpers (`asset`, `css`, `js`, `img`) | `src/helpers.js` | **Stable candidate** | Nunjucks template helper catalog. |
+| `setAppContext` / `getDb` / `getAppContext` | `src/app-context.js` | **Needs cleanup** | Global singleton state. Retained for backward compatibility; replaced internally by `req.context`. |
+| `compileSchema` / `applySchema` | `core/compileSchema.js` | **Stabilized** | Unified around `core/validation/index.js`. |
+| `adminPanelPlugin` / `adminApi` | `plugins/admin-panel` | **Stable candidate** | Mithril.js admin panel module and widget registration API. |
+| `realtimePlugin` / `createRealtime` | `plugins/realtime`, `core/realtime` | **Stable candidate** | WebSocket/SSE/Socket.IO adapter API. |
+| `queuePlugin` / `createQueueManager` | `plugins/queue`, `core/queue` | **Stable candidate** | Memory/DB/Redis background queue API. |
+| `basicAuthPlugin`, `corsPlugin`, `csrfPlugin` | `plugins/*` | **Stable candidate** | Zero-dependency security plugins. |
+| `content` / `contentPlugin` | `core/content`, `plugins/content` | **Needs cleanup** | Headless CMS schema engine with slight overlap with ORM models. |
+| `kernel` (`createApp`, `definePlugin`, `defineFlow`) | `core/kernel` | **Experimental / Standalone** | Isolated standalone mini-kernel. |
+| `req.input` vs `validatedInput` | API / Services | **Stabilized** | API routes use `req.input`, services use validated handler argument via shared Zod core. |
+| CLI commands (`webspresso *`) | `bin/commands/*` | **Stable candidate** | `dev`, `migrate`, `doctor`, `favicon:generate`, `seed` CLI contracts. |
+
+---
+
+## 8. Runtime Dependencies Analysis
+
+```text
+┌────────────────────────────────────────────────────────────────────────┐
+│                        WEBSPRESSO RUNTIME MATRIX                       │
+├──────────────────────────┬─────────────────────────────────────────────┤
+│ Runtime-independent      │ • core/errors (Error class hierarchy)       │
+│ (Pure JS Logic)          │ • core/orm/types, model definitions         │
+│                          │ • core/realtime/registry, reconnect-manager │
+│                          │ • core/content/schema, field-types          │
+│                          │ • src/services/validator, memoize (core)    │
+│                          │ • core/url-path-normalize.js                │
+├──────────────────────────┼─────────────────────────────────────────────┤
+│ Node-specific            │ • core/shutdown (process.on, SIGTERM/SIGINT)│
+│ (Node.js Built-in APIs)  │ • core/orm/transaction (AsyncLocalStorage)  │
+│                          │ • core/compression (native zlib streams)    │
+│                          │ • src/server.js (renderStackStorage ALS)    │
+│                          │ • core/auth/jwt, hash (crypto module)       │
+│                          │ • src/file-router.js (fs, path scanning)    │
+│                          │ • core/queue (EventEmitter, timers)         │
+├──────────────────────────┼─────────────────────────────────────────────┤
+│ Express-specific         │ • src/server.js (express(), app.use, etc.)  │
+│ (Tight Express Coupling) │ • src/file-router.js (req, res, next chain) │
+│                          │ • plugins/* (Express route handlers)        │
+│                          │ • core/auth/middleware.js (Express MWs)     │
+│                          │ • core/ssr/stream.js (res.write, res.flush) │
+├──────────────────────────┼─────────────────────────────────────────────┤
+│ Webspresso-specific      │ • Nunjucks Safe Engine & Extensions         │
+│ (Framework Domain)       │ • Model Repository & Query Builder          │
+│                          │ • File Router Loader & Meta Conventions     │
+│                          │ • Mithril Admin SPA Module Contract         │
+└──────────────────────────┴─────────────────────────────────────────────┘
+```
+
+---
+
+## 9. Architectural Conflict Resolutions
+
+### Conflict 1: `core/kernel` vs Main Framework (`src/server.js`, `src/services`, `core/orm`)
+- **Resolution**: The main SSR framework (`src/server.js`, `src/plugin-manager.js`, `core/orm`) is the canonical framework architecture. `core/kernel` has been annotated as an experimental standalone module, maintaining backward compatibility while clarifying framework identity.
+
+### Conflict 2: API Validation (`compileSchema`) vs Service Validation (`services/validator`)
+- **Resolution**: Extracted a unified validation utility (`core/validation/index.js`) providing extended Zod instances (`z.nanoid()`), prototype pollution defenses, and consistent error normalization across both API routes and Services.
+
+### Conflict 3: Three Distinct Event / Hook Mechanisms
+- **Canonical Scopes**:
+  - `pages/_hooks.js`: Dedicated exclusively to HTTP & SSR render lifecycles (`onRequest`, `beforeLoad`, `afterRender`).
+  - `ModelEvents`: Dedicated exclusively to database / model mutations (`User.beforeCreate`, `User.afterSave`).
+  - `core/kernel/events.js`: Isolated alongside the standalone kernel.
+
+### Conflict 4: Implicit Global Context vs Instance Request Context
+- **Resolution**: Formalized Express `req.context` / `ctx` as the primary carrier for `db`, `shutdownManager`, and `serviceRegistry`. `src/app-context.js` is preserved as a fallback for external caller compatibility.
+
+---
+
+## 10. Technical Debt & Stabilization Summary
+
+| Priority | Area | Files | Status | Action Taken |
+| :---: | :--- | :--- | :---: | :--- |
+| **MEDIUM** | SQLite Default Value Warnings | `core/orm/index.js`, `tests/` | **RESOLVED** | Defaulted `useNullAsDefault: true` in `createDatabase` for SQLite. |
+| **LOW** | Linear URL Path Trimming | `src/services/discovery.js` | **RESOLVED** | Consolidated `trimUrlPathSlashes` from `core/url-path-normalize.js`. |
+| **HIGH** | TypeScript Definition Sync | `index.d.ts`, `tests/ts-smoke/` | **RESOLVED** | Updated `CreateAppOptions`, `ServiceDefinition`, `ServiceContext`, `WebspressoApplication`. |
+| **MEDIUM** | Shared Zod Validation Core | `core/validation/index.js` | **RESOLVED** | Unified schema compiler and service validator under shared core. |
+| **MEDIUM** | Standalone Kernel Isolation | `core/kernel/`, `index.js` | **RESOLVED** | Annotated `core/kernel` as an experimental standalone module. |
+| **MEDIUM** | Formal Request Context | `src/server.js`, `src/file-router.js` | **RESOLVED** | Bound `req.context` container across all request flows. |
+
+---
+
+## 11. Test Coverage & Verification
+
+### Verified Test Metrics:
+- **Test Files**: 176 test suites (including unit, integration, and security suites)
+- **Total Tests**: 2,109 tests
+- **Status**: 100% Passed (0 failed, 0 skipped)
+- **Execution Time**: ~60–75 seconds
+
+---
+
+## 12. Risk Register
+
+| Risk ID | Risk Description | Probability | Impact | Mitigation Strategy | Status |
+| :---: | :--- | :---: | :---: | :--- | :---: |
+| **R-01** | Global state collisions in multi-app tests | Low | High | Use `req.context` and pass database/service instances via container. | **MITIGATED** |
+| **R-02** | Confusion between SSR and Kernel `createApp` | Low | Medium | Annotate `core/kernel` as standalone experimental module. | **MITIGATED** |
+| **R-03** | Native binary version mismatch (`better-sqlite3`) | Low | High | Enforce Node.js 20 LTS target via `.nvmrc` and CI scripts. | **ACTIVE** |
+| **R-04** | Zod validation divergence between API and Services | Low | Medium | Standardize on unified `core/validation/index.js`. | **MITIGATED** |
+
+---
+
+## 13. Stabilization Backlog
+
+```mermaid
+graph TD
+    P0[Phase 0: Baseline & Test Verification] --> P1[Phase 1: Dead/Duplicate Cleanup]
+    P1 --> P2[Phase 2: Contract Stabilization]
+    P2 --> P3[Phase 3: Architecture Consolidation]
+    P3 --> P4[Phase 4: Runtime Boundaries]
+    P4 --> P5[Phase 5: Documentation Restructuring]
+```
+
+### Backlog Status:
+- [x] **`P0-01`**: Verify baseline test suite and native binary compatibility (Node 20 LTS).
+- [x] **`P1-01`**: Fix SQLite `useNullAsDefault` warnings in Knex configurations.
+- [x] **`P1-02`**: Consolidate internal URL path trimming utilities.
+- [x] **`P2-01`**: Synchronize `index.d.ts` with `createApp` and plugin options.
+- [x] **`P3-01`**: Unify Zod validation extensions across API and Services (`core/validation`).
+- [x] **`P3-02`**: Isolate `core/kernel` and annotate as experimental/standalone.
+- [x] **`P4-01`**: Formalize Request Context container (`req.context`) to reduce global state reliance.
+- [x] **`DOC-P1..P8`**: Complete documentation audit and restructuring (`docs/`, `llms.txt`, slimmed `README.md`).
+
+---
+
+## 14. "Do Not Touch" Core Invariants
+
+The following components are highly stable, thoroughly tested, and performant in production. They must **not** be modified solely for aesthetic refactoring:
+
+1. **`src/file-router.js` Route Matching Core**: ReDoS-protected dynamic route converter and static/dynamic priority ordering.
+2. **`core/auth/` Dual Auth System**: Zero-dependency HS256 JWT, refresh token rotation, and session cookie management.
+3. **`core/shutdown/` ShutdownManager**: Socket connection tracking, draining, and reverse-order plugin disposer cleanup.
+4. **`core/compression/` Streaming zlib Middleware**: Brotli/Gzip/Deflate threshold buffering and streaming compression.
+5. **`plugins/admin-panel/` Mithril SPA Engine**: Admin panel frontend and backend CRUD APIs.
+6. **`core/orm/transaction.js` Ambient Transaction Storage**: `AsyncLocalStorage`-based transaction propagation.
+7. **Express 5 Framework Core**: Primary HTTP engine foundation.

package/.cursorrules

@@ -0,0 +1,14 @@
+# Cursor Rules for Webspresso
+
+Webspresso is a full-stack Node.js framework with 100% file-based routing, Knex ORM repositories, and zero external dependencies for core utilities.
+
+## Critical Guidelines
+- Complete reference & guidebook: Read `AGENTS.md` and `.agents/AGENTS.md`.
+- Canonical API definitions: Check `index.d.ts`.
+
+## Strict Anti-Patterns & Requirements
+1. **NO `routes/` or `app.get()`**: All API routes belong in `pages/api/...` with method extensions (`.get.js`, `.post.js`, `.delete.js`, `.patch.js`, `.put.js`).
+2. **NO Raw `db('table')`**: Always use `req.db.getRepository('ModelName')` or `db.getRepository('ModelName')`.
+3. **NO `controllers/`**: Encapsulate multi-step business logic in `services/domain/action.js` and call via `ctx.service('domain.action', input)`.
+4. **NO external auth libraries**: Use built-in `req.auth` and `webspresso/core/auth`. Do not install `jsonwebtoken` or `passport`.
+5. **Zero regressions**: Run unit/integration tests with `npx vitest run tests/...` before completing tasks.

package/AGENTS.md

@@ -0,0 +1,175 @@
+# Webspresso Project Guidelines & Rules
+
+Webspresso is a lightweight, zero-dependency-sprawl full-stack Node.js SSR & API framework featuring Next.js/Nuxt-style file-based routing, an intuitive Knex-based ORM, a modular plugin ecosystem, and a customizable Mithril.js SPA Admin Panel.
+
+---
+
+## 🎯 Agent Intent Triage Matrix (Quick Guide Selection)
+
+When tasked with a feature or bugfix, **immediately consult the corresponding topic guide** before writing code:
+
+| Intent / Task | Canonical Topic Guide | Primary Directory / Ground Truth |
+| :--- | :--- | :--- |
+| **New API Route / Endpoint** | [`.agents/ROUTING.md`](.agents/ROUTING.md) | `pages/api/...` |
+| **SSR HTML Page / Layout / Loader** | [`.agents/ROUTING.md`](.agents/ROUTING.md) & [`.agents/STREAMING.md`](.agents/STREAMING.md) | `pages/*.njk`, `pages/*.js`, `views/` |
+| **Database Model / Migration / Query** | [`.agents/ORM.md`](.agents/ORM.md) | `models/*.js`, `core/orm`, `migrations/` |
+| **Business Logic / Transaction / Service** | [`.agents/SERVICES.md`](.agents/SERVICES.md) | `services/` |
+| **Admin Panel (Page, Widget, Action, UI)** | [`.agents/ADMIN.md`](.agents/ADMIN.md) | `plugins/admin-panel/` |
+| **Authentication (Session / JWT / Guard)** | [`.agents/AUTH.md`](.agents/AUTH.md) | `core/auth/` |
+| **Background Jobs & Worker Queue** | [`.agents/QUEUE.md`](.agents/QUEUE.md) | `jobs/` |
+| **API Signatures & Types Verification** | [`index.d.ts`](index.d.ts) | **Canonical Ground Truth** |
+
+---
+
+## 🛡️ 4-Step Mandatory Agent Workflow Protocol
+
+Every coding agent working on Webspresso **MUST** follow this 4-step execution discipline:
+
+1. **Discover & Triage**: Identify the relevant topic guide above and inspect existing similar files in the workspace (e.g. `pages/api/`, `models/`, `services/`).
+2. **Contract Adherence**: Follow the exact module contract exported by the framework (`{ schema, middleware, handler }` for routes, `defineModel` for ORM, `defineService` for services).
+3. **Zero External Dependencies**: Use native Node.js standard modules (`crypto`, `path`, `fs`, `zlib`, `events`) and built-in framework utilities. NEVER install ad-hoc npm packages for auth, jwt, routing, or templating.
+4. **Targeted Verification**: Run the specific Vitest test file (`npx vitest run tests/...`) to confirm zero regressions before marking the task complete.
+
+---
+
+## ⚖️ ❌ BAD vs ✅ GOOD Summary Matrix
+
+| Category | ❌ NEVER DO THIS (Anti-Pattern) | ✅ ALWAYS DO THIS (Webspresso Way) |
+| :--- | :--- | :--- |
+| **Routing** | `routes/index.js`, `express.Router()`, `app.get('/api/users')` | `pages/api/users.get.js` exporting `{ schema, handler }` |
+| **Database** | `db('users').where({ id }).select()` (Raw Knex queries) | `db.getRepository('User').findById(id)` |
+| **Business Logic** | `controllers/UserController.js` | `services/user/get.js` → `ctx.service('user.get', input)` |
+| **Auth** | `npm i jsonwebtoken passport express-session` | Built-in `req.auth`, `core/auth`, `middleware: ['jwt']` |
+| **SSR Rendering**| `res.render('profile.njk', data)` inside custom handlers | Companion loader `pages/profile.js` exporting `async function load()` |
+| **Exceptions** | `res.status(404).json({ error: 'Not found' })` | `throw new NotFoundError('User not found')` |
+| **Background** | `setTimeout(...)`, `npm i bullmq` | `jobs/user/welcome.js` → `req.queue.dispatch('user.welcome', data)` |
+
+---
+
+## 1. Codebase Architecture
+
+### 1.1 Core SSR & Routing (`pages/`, `views/`)
+- **File-based Routing**: Pages located in `pages/` automatically map to URL routes (e.g. `pages/index.njk` → `/`, `pages/products/[id].njk` → `/products/:id`).
+- **Data Loaders (`load()`)**: Page routes export `async function load({ req, res, db, ctx })` to fetch data server-side before template rendering.
+- **Templating & Helpers**: Nunjucks (`.njk`) templates rendered with layouts (e.g. `views/layout.njk`). Access helpers via `fsy` object in templates.
+- **Asset Versioning & Cache-Busting**: Configured via `createApp({ assets: { version, manifestPath, prefix } })`. Supports query-string versioning (`?v=1.2.3`), Vite/Webpack manifest resolution (`.vite/manifest.json`), CDN prefixing, and `fsy.asset()`, `fsy.css()`, `fsy.js()`, `fsy.img()` template helpers.
+- **Framework Exceptions & Central Error Boundary**: Built-in exception hierarchy (`WebspressoError`, `HttpError`, `ValidationError`, `SecurityError`, `RequestAbortedError`, `ConfigurationError`, `PluginError`, `RouteNotFoundError`) and custom handler hook (`app.setErrorHandler`). Async route handlers automatically propagate rejected promises. Production responses mask 500 stack traces and error details.
+- **Graceful Shutdown & Lifecycle Management**: Configured via `createApp({ server: { shutdown: { enabled: true, mode: 'graceful'|'force', timeout: 10000 } } })`. Socket connection tracking, request draining with `Connection: close`, and reverse-order plugin disposer cleanup.
+- **HTTP Response Compression**: Zero-dependency streaming zlib compression configured via `createApp({ server: { compression: true } })` with Brotli (`br`), Gzip (`gzip`), Deflate (`deflate`), threshold buffering (1 KB), and route-level opt-out (`res.compress(false)`).
+- **Custom Error & 404 Pages**: Configured via `createApp({ errorPages: { notFound, serverError, timeout } })`. Supports template paths (e.g. `notFound: '404.njk'`) or custom handler functions, passing `{ fsy, locale, isDev, url, method }` to templates.
+- **Middleware & Hooks**: Route lifecycle hooks can be declared per-page or globally via `pages/_hooks.js`.
+- **API Endpoints & Routing Architecture**:
+  - ⚠️ **CRITICAL ANTI-PATTERN: NEVER create a `routes/` or `src/routes/` directory. NEVER use `express.Router()`, `app.get()`, or `app.post()` for application endpoints.**
+  - **100% File-Based Routing**: All API routes MUST be created inside `pages/api/...` with HTTP method extensions:
+    - `pages/api/notes.get.js` → `GET /api/notes`
+    - `pages/api/notes.post.js` → `POST /api/notes`
+    - `pages/api/notes/[id].get.js` → `GET /api/notes/:id`
+    - `pages/api/notes/[id].delete.js` → `DELETE /api/notes/:id`
+  - **Standard API Route Module Contract**:
+    ```javascript
+    const { defineApi } = require('webspresso');
+
+    module.exports = defineApi({
+      schema: ({ z }) => ({
+        body: z.object({ title: z.string().min(1) }),
+        params: z.object({ id: z.string().optional() }),
+      }),
+      middleware: ['auth'], // optional named or functional middleware
+      handler: async (req, res, ctx) => {
+        // req.db and req.service are automatically injected
+        // req.input.body contains the validated payload
+        const item = await req.db.getRepository('Note').create(req.input.body);
+        return res.status(201).json(item);
+      },
+    });
+    ```
+  - **Modular Architecture (`src/modules/` or `modules/`)**:
+    - Encapsulate domain features using `defineModule({ name, pages, api, services, middlewares })`.
+    - Auto-registers routes (`/auth/login`, `/api/auth/login`), services (`'auth.login'`), and scoped middlewares without manual router glue.
+
+### 1.2 Services Layer (`services/`, `src/services/`)
+- **File-based Auto-Discovery**: Services located in `services/` map directly to dot-separated service names (e.g. `services/user/create.js` → `'user.create'`) and camelCase aliases (`'user.reset-password'` & `'user.resetPassword'`).
+- **Declarative Validation & Authorization**: Define optional Zod `schema`, `auth` (boolean, role string/array, predicate), `timeout` (ms), `transaction: true` (ACID transaction propagation across nested calls), and `cache: { ttl, key }`.
+- **Unified Invocation**: Consumed via `ctx.service('service.name', input)` from SSR loaders, `req.service()` in API routes, or standalone `createServiceRegistry()`.
+
+### 1.3 ORM & Database Layer (`core/orm`, `models/`)
+- ⚠️ **CRITICAL ANTI-PATTERN: NEVER use raw Knex queries (e.g. `db('table')` or `knex('table')`) in application code.** Raw queries bypass schema validation, lifecycle hooks, soft-deletes, multi-tenant scopes, hidden fields, and query caching.
+- **ALWAYS Use Repositories**: Accessed via `db.getRepository('ModelName')` or `req.db.getRepository('ModelName')`:
+  - `await repo.findById(id)`
+  - `await repo.find({ status: 'active' })`
+  - `await repo.findOne({ slug })`
+  - `await repo.create(data)` *(validates schema & executes hooks)*
+  - `await repo.update(id, data)`
+  - `await repo.delete(id)` *(respects soft deletes)*
+  - `await repo.query().where('price', '>', 50).paginate({ page, limit })`
+  - `await repo.query().include('user', 'tags').find()` *(eager loading)*
+- **Model Definition**: Defined in `models/*.js` using `defineModel({ name, table, schema, relations, scopes, hidden, admin, cache })`.
+- **Schema & Types**: Built with `zdb` schema builder (`zdb.id()`, `zdb.string()`, `zdb.integer()`, `zdb.boolean()`, `zdb.file()`, `zdb.json()`).
+- **Ambient Transactions**: Repositories automatically bind to ambient transactions (`AsyncLocalStorage`) inside `db.transaction()` and services (`transaction: true`) without manual `trx` passing.
+- **Query Caching**: Memory/provider query cache configured per database instance or model (`cache: 'auto'|'smart'|true`).
+- **Migrations**: Database schema changes managed via Knex migrations in `migrations/` (`webspresso db:migrate`).
+
+### 1.4 Admin Panel Architecture (`plugins/admin-panel`)
+- **SPA CRUD Engine**: Modular Mithril.js SPA mounted at `/_admin` powered by `adminPanelPlugin({ db })`.
+- **Admin Module Registration**: Registered using `adminApi.registerModule` or `adminApi.registerPageDir`.
+- **Custom Admin Pages**:
+  - `componentFile`: Path to a clean Mithril component `.js` file.
+  - `pagesDir`: Directory auto-discovery containing page folders with `page.json` and `component.js`.
+  - **Auto Layout Wrapping**: By default (`layout: true`), custom page views are automatically wrapped inside the Admin `Layout` and `Breadcrumb` containers without requiring manual `m(Layout, ...)` nesting. Set `layout: false` for full-screen / custom canvas pages.
+- **Extension Points**:
+  - Custom Field Renderers: `registry.registerFieldRenderer(type, { display, edit })`.
+  - Widgets: `registry.registerWidget(id, config)`.
+  - Actions & Bulk Actions: Single record buttons (`registerAction`) and multi-select table actions (`registerBulkAction`).
+- **Session Isolation**: Admin staff authentication uses a separate session (`req.session.adminUser` / `/_admin/api/auth/*`), isolated from public site auth (`req.user`).
+
+### 1.5 Plugin Ecosystem (`plugins/`)
+Plugins expose `name`, `version`, `dependencies`, and lifecycle hooks (`register(ctx)`, `onRoutesReady(ctx)`):
+- `adminPanelPlugin`: Model CRUD, custom pages, widgets, user management.
+- `fileManagerPlugin`: Media library, directory tree, file preview, and drag-and-drop uploads.
+- `basicAuthPlugin`: Zero-dependency HTTP Basic Authentication (RFC 7617) with named route middleware (`ctx.middlewares.basicAuth`) and global protection.
+- `contentPlugin`: Schema-driven CMS with inline editing and headless API.
+- `uploadPlugin`: Multipart file uploads with local/cloud storage providers.
+- `redirectPlugin`: Configurable HTTP 301–308 redirects.
+- `rateLimitPlugin`: Named rate limiters (`express-rate-limit`).
+- `siteAnalyticsPlugin`: Self-hosted page view analytics and client JS error tracking.
+- `emailPlugin`: MJML template compilation & Nodemailer integration.
+- `polarPlugin`: Polar.sh subscription billing (checkout, portal, webhooks, Customer State sync). Requires `rateLimitPlugin` before polar for rate limiters. Supports `hooks.isPaidUser`, `syncBeforeCheckout`, `syncBillingForAppUser`. See `docs/guides/polar-production-checklist.md`.
+- `auditLogPlugin`: Admin mutation audit logging.
+- `ormCacheAdminPlugin`: Cache inspection and invalidation dashboard.
+- `corsPlugin`: Zero-dependency Cross-Origin Resource Sharing handler.
+- `csrfPlugin`: CSRF token validation and Nunjucks form helpers.
+- `realtimePlugin`: Framework-agnostic, plugin-based realtime layer (`core/realtime`) with WebSocket, SSE, Socket.IO, and distributed Redis Pub/Sub adapters.
+- `restResourcePlugin`: Zero-boilerplate RESTful CRUD API generator directly from ORM models with eager loading, sanitization, filtering, pagination, and soft-delete scoping.
+- `xlsxPlugin`: Excel spreadsheet generation, parsing, ORM streaming export (`res.xlsx`), and built-in services (`xlsx.generate`, `xlsx.parse`, `xlsx.exportModel`).
+- `csvPlugin`: RFC 4180 CSV generator, parser, Excel UTF-8 BOM compatibility, formula sanitization, streaming writer (`res.csv`), and built-in services (`csv.generate`, `csv.parse`, `csv.exportModel`).
+
+---
+
+## 2. Development & Testing Workflow
+
+### 2.1 Essential Commands
+- **Dev Server**: `npm run dev` or `webspresso dev`
+- **Run Vitest Unit Tests**: `npm test` or `npx vitest run tests/unit/admin-panel/`
+- **Run Migrations**: `npx webspresso db:migrate`
+- **Project Sanity Check**: `npx webspresso doctor`
+- **Scaffold AI Agent Rules**: `npx webspresso agents:init`
+
+### 2.2 Testing & Quality Assurance
+- **Zero Regression Policy**: Always run unit tests before declaring any task complete.
+- **Targeted Test Execution**: Run specific unit test files during development for fast feedback (`npx vitest run tests/unit/admin-panel/admin-module-files.test.js`).
+- **API Backward Compatibility**: Never break existing `registerModule`, `registerPage`, or `adminApi` contracts. Always add new parameters as optional or backward-compatible enhancements.
+
+---
+
+## 3. Coding Standards & Conventions
+
+1. **Strict Signature & Schema Verification**: Always verify function signatures and object schemas by viewing authoritative source files before calling or extending them.
+2. **File & Component Separation**: Keep client-side components in clean standalone JS files (`component.js` / `componentFile`) rather than escaping inline template strings.
+3. **Log & Traceback Inspection**: Base all bug fixes strictly on exact runtime error log tracebacks and empirical test output. Never swallow errors or return dummy fallbacks.
+4. **Strict File-Based Routing (NO `routes/` or `app.get()`)**: Webspresso is strictly file-based. Never create manual router files (`src/routes/*.js`), never call `app.get()`, and never use `express.Router()`. All application pages belong in `pages/` and all API endpoints belong in `pages/api/` with HTTP method extensions (`.get.js`, `.post.js`, `.delete.js`, `.patch.js`, `.put.js`).
+5. **Strict Repository Pattern (NO Raw `db('table')`)**: Never write raw Knex table queries in route handlers or services. Always use `req.db.getRepository('ModelName')` to ensure schema validation, hooks, scopes, and query caching are applied.
+6. **Strict Services Layer (NO `controllers/`)**: Never create a `controllers/` folder or controller classes. Encapsulate business logic in `services/domain/action.js` and call via `ctx.service('domain.action', input)`.
+7. **Native Dual Auth (NO External JWT/Passport Packages)**: Never install `jsonwebtoken`, `passport`, or `express-session`. Always use native `req.auth` and `webspresso/core/auth`.
+8. **Page Data Loaders (NO Manual `res.render()`)**: Never call `res.render()` inside custom handlers for web pages. Use companion loaders `pages/*.js` exporting `async function load({ req, res, db, ctx })`.
+9. **Semantic Exceptions (NO Generic Errors / Manual Responses)**: Never return manual `res.status(404).json(...)` inside services or loaders. Always throw semantic errors: `NotFoundError`, `ValidationError`, `UnauthorizedError`, `ForbiddenError`.
+10. **Background Jobs (NO BullMQ / `setTimeout`)**: Never use unmanaged `setTimeout` or external queues. Place job workers in `jobs/` and dispatch via `req.queue.dispatch()`.