@dboio/cli 0.11.4 → 0.13.2

package/plugins/claude/dbo/docs/_audit_required/dbo-problems-code.md

@@ -0,0 +1,111 @@
+# DBO.io: Implementation Problems in the Codebase
+
+## 1. Purpose
+
+This document is the companion to the DBO.io white paper. Where the white paper describes the architecture — schema-driven applications, template composition, cell-level security, the eliminated middle tier — this document examines the implementation that grew around those ideas over fifteen years of organic development. The architecture is sound. The problem is that the codebase never enforced the boundaries the architecture implies. What follows names the structural problems precisely, grounded in specific files and patterns, so the rewrite can preserve the architectural ideas while eliminating the implementation failures.
+
+## 2. Ambient Static State as the Foundation
+
+This is the highest-impact problem in the codebase and the one from which most other problems follow.
+
+Three ambient singletons permeate the application. DBOIOC.Current provides the HTTP context and debug/strict mode flags. DBOIOReqI.Current provides the request context: the account, the core data source, the request stack, the current user. DBOIOResI.Current provides the response context: messages, payload, output state. These are not injected. They are reached for, from anywhere, by any layer.
+
+The numbers tell the story: DBOIOC appears in 145 files, DBOIOReqI in 191 files, DBOIOResI in 72 files. Across the codebase, 187 unique files reference at least one of the three. The data layer reaches up into request context. Property getters perform database queries through global state — Entity.SchemaPhysicalName, for instance, calls Entity.select using DBOIOReqI.Current.DataSource_core inside a getter, meaning that reading a property triggers a database round-trip through ambient state with no indication at the call site. The dboioContext class, which defines these singletons in 534 lines, mixes HTTP context management, debugging flags, profiling, and session management into a single static surface.
+
+The consequences compound. There is no dependency injection because every class already has access to everything through the ambient singletons. There is no unit testing because instantiating a domain object requires the full HTTP stack. There is no thread safety because the pattern mixes thread-static storage (CallContext.LogicalGetData) with instance state. And there is no clean path to .NET Core because the singletons are backed by HttpContext.Current and HttpContext.Items.
+
+A migration abstraction was introduced. An IDboioContext interface was defined, with DboioContextFramework and DboioContextCore implementations. A DboioContextProvider was written to manage the switchover. A README_ContextMigration.md was created with a detailed migration checklist. But DboioContextProvider.cs is entirely commented out — all 115 lines. Every item on the migration checklist is unchecked. The migration infrastructure exists on paper, but the codebase never moved toward it because the ambient singletons are load-bearing in 187 files.
+
+## 3. Objects That Are Everything at Once
+
+The god-class problem is pervasive, but three classes illustrate it most clearly.
+
+Entity, at 1,218 lines, is simultaneously a schema definition, a mutable data container, and a CRUD executor. Its RecordStates enum (Template, SingleRecord, MultiRecord) reveals the confusion: the same object represents a table definition when it has no data, a single record when one is loaded, and a result set when multiple are loaded. Its static template method is a 15-case switch statement that returns a different preconfigured Entity depending on which system type is requested — Output, Site, User, Media, Content, and ten others. The Properties accessor throws an exception if the object is in the wrong state. Six overloaded select methods handle reading. A save method (recently delegated to a separate EntitySave class, but still orchestrated by Entity) handles writing. A delete method handles removal. Schema metadata, record data, and persistence logic all live in the same class, which means changing how records are stored risks breaking how schema is resolved, and testing any one concern requires standing up all the others.
+
+EntityActionCollection, at 2,451 lines across 39 region blocks, handles the entire lifecycle of an input operation: parsing the add/edit/delete syntax from request payloads, validating column values against entity metadata, enforcing security rules, resolving foreign key references between new records, handling media uploads, performing structural database operations, managing transactions, and executing the final persistence. Nine behavioral flags (BypassSecurity, PerformAppAssetAbleToUpdateChecks, DisableForeignKeyConstraintsOnPersistence, and six others) interact throughout the file in ways that cannot be understood locally. Twenty-nine exception catches suggest business logic so entangled that it requires constant defensive error handling.
+
+DataSource_ANSI, at 3,279 lines, contains all SQL generation for ANSI-compatible databases. Identifier quoting, type conversion, SELECT/INSERT/UPDATE/DELETE assembly, pagination, sorting, filtering, metadata queries, and transaction management all occupy a single file. This is the class that every datasource subclass extends, which means a change to SQL generation for one operation risks every other operation in the system.
+
+## 4. The Rendering Pipeline's Compounding Complexity
+
+The rendering pipeline is the most important subsystem in the platform — it is the mechanism by which schema becomes visible output — and it is the hardest to modify with confidence.
+
+TokenReplacement has 14 active subclasses: Content, CustomSqlOutputValues, Include, Input, Meta, OutputMetadata, OutputValue, OutputValueMetadata, Payload, Request, ResultSetAccess, Session, Site, and Unique (plus two more that are commented out). Token processing order is critical: certain tokens must resolve before others, and incorrect ordering causes silent failures where tokens pass through unresolved or resolve against the wrong data context. The base TokenReplacement class alone is large enough to contain the ordering logic, the replacement patterns, and the context management for all subclasses.
+
+EmbedToken processing adds another layer of complexity. Nine EmbedToken classes form a three-level inheritance hierarchy (EmbedToken base, EmbedToken_ResultSettable and EmbedToken_ValidatableExecutableOperation as intermediates, and five leaf types for Content, Output, OutputByEntity, Input, and Message). These total 2,267 lines. The EmbedProcessor_RegEx file, at 688 lines, manages embed resolution through a RequestStack that tracks nested embed depth, handles push and pop operations for execution context, and prevents infinite recursion when embeds reference each other. The interaction between embed resolution, token replacement ordering, and request stack management creates a system where the only way to understand what a template will produce is to trace the full execution path mentally.
+
+The TemplateParser uses a two-phase approach: regex preprocessing to identify and normalize template tags, followed by XML parsing via XDocument.Parse. This means templates must survive both regex matching and XML validation, and failures in either phase produce errors that are difficult to trace to their source.
+
+Developer confidence in this subsystem is low, and the code says so explicitly. The string ":-/" appears in 146 files across the codebase, with heavy concentration in the rendering pipeline: TokenReplacement.cs alone contains it on fifteen separate lines. "MICKEY-MOUSE" appears in five files, including TokenReplacement.cs and ExternalMessage_ChatBot.cs. These are not TODO markers — they are expressions of active uncertainty by the original developer about whether the current approach is correct. When the most critical subsystem is annotated with its own author's doubt, modification carries risk that no one can accurately assess.
+
+## 5. Three Syntaxes for Everything
+
+The EmbedProcessor.cs file defines three distinct tag types in a static array: "#_embed" (underscore-prefixed system syntax), "#embed" (non-prefixed syntax), and "EmbedToken" (XML element syntax). Each requires its own regex patterns, its own closing tag handlers, and its own parsing paths. This triple syntax exists because the tag system evolved over time without deprecating earlier forms.
+
+The consequences multiply. The template parser must recognize and normalize all three forms. Message templates support only the underscore-prefixed syntax while other template types support both prefixed and non-prefixed forms, creating inconsistency that authors encounter as unexplained failures when reusing templates across contexts. The API documentation must describe three ways to express the same operation. The testing matrix triples for any change that touches embed processing.
+
+Simplifying this requires auditing every deployed template across every tenant instance to determine which syntaxes are in active use — a prerequisite that has, so far, prevented any simplification from occurring.
+
+## 6. The ASP.NET Framework Ceiling
+
+Two hundred and four files import System.Web. The web application is structured around App_Code dynamic compilation. The request pipeline depends on thirteen IHttpModule implementations (CacheModule, AuthenticationModule, LoggingModule, Rewriter, and nine others) that hook into the ASP.NET Framework pipeline. Session management uses a custom MySQL session store built on System.Web.SessionState. The Account_Request cache tier uses HttpContext.Items for request-scoped storage. The caching layer mixes Redis for distributed state with HttpRuntime.Cache for local state. Authentication includes a SAML 2.0 implementation built on the ASP.NET Framework authentication pipeline.
+
+ASP.NET Framework is end-of-life. It cannot run on Linux or in containers. The C# 7.3 language version constraint (required by netstandard2.0 targeting for the shared libraries) prevents using modern language features that would simplify the code. The platform cannot scale horizontally without containerization, cannot deploy to modern cloud infrastructure without Linux support, and cannot adopt current .NET runtime improvements without migrating off Framework.
+
+The migration abstraction exists — IDboioContext, the Framework and Core implementations, the provider, the migration guide. But as described in section 2, the provider is commented out and the migration checklist is untouched. The gap between the migration plan and the codebase reality is the 204 files importing System.Web and the 187 files reaching for ambient singletons that are backed by HttpContext.Current.
+
+## 7. SQL Generation by String Assembly
+
+Filter.cs (631 lines), FilterBuilder.cs (1,119 lines), and EnforceSecurityFilter.cs (215 lines) build SQL WHERE clauses through StringBuilder concatenation. FilterBuilder appends field names directly — the targetField parameter is concatenated into the SQL string without identifier escaping or quoting. Filter.cs provides five overloaded factory methods that construct filter objects from different parameter combinations, each assembling SQL through a different path. EnforceSecurityFilter builds security-scoped WHERE clauses by constructing parameter names dynamically using DateTime.Now.Ticks.
+
+Parameter values are handled safely through parameterized queries. But identifier names — table names, column names, field references — are assembled as strings. Protection against SQL injection in identifiers relies on the assumption that field names always come from trusted sources (entity metadata), not on structural guarantees. This is not automatically a vulnerability, but it creates an audit burden: proving the system is safe requires tracing every path through which a field name can reach the SQL assembly methods, across thousands of lines of concatenation logic. In a system designed for multi-tenant use where tenants define their own schemas, the distance between "trusted metadata" and "user-influenced input" is shorter than it first appears.
+
+## 8. No Path to Automated Testing
+
+The Test project targets .NET Framework 4.7.2, uses packages.config for NuGet dependencies, and requires MSBuild or Visual Studio to build — it cannot use dotnet test. Ten test source files exist for a codebase of this size, with no controller tests despite seventeen controllers.
+
+But the project structure is the symptom, not the cause. The real barrier to testing is the ambient state described in section 2. Business logic in Entity, EntityActionCollection, DataSource_ANSI, the rendering pipeline, and the messaging subsystem all reach for DBOIOC.Current, DBOIOReqI.Current, or DBOIOResI.Current instead of accepting dependencies through constructors or method parameters. There is no seam where a test can substitute a mock data source, a mock request context, or a mock cache. To test a token replacement, you would need to stand up the full singleton graph — which means you need an HTTP context, a database connection, an account record, and a populated cache.
+
+The result is self-reinforcing: without tests, changes carry risk. Because changes carry risk, development is slow and conservative. Because development is slow, there is never time to introduce the testing infrastructure that would make development faster. The few tests that exist appear to be integration tests that require database connectivity, confirming that even the test authors could not find a way to test the system's components in isolation.
+
+## 9. Build System and Deployment Fragility
+
+The repository maintains two solution files (dboioApi.sln and dboioComponents.sln) with source files in src/ and project files in project/, connected by relative Compile Include paths with Link elements. The dboioApi.sln solution references external projects three directories above the repository root (Host.csproj and Shared.csproj in a sibling dboio/Development/ExternalCode path), meaning the solution will not build on a clean checkout without the external repositories present. Mixed target frameworks — netstandard2.0 for shared libraries, .NET Framework 4.7.2 for the webapp and test project — require different toolchains. A literal copy artifact, AppOperation_Import - Copy.cs, sits in the source tree with its entire content commented out — a file that was duplicated during development and never cleaned up.
+
+This fragility is a drag multiplier. Every new developer must discover and configure the external dependencies. CI/CD requires either reproducing the full external directory structure or maintaining a parallel build configuration. Framework upgrades must account for both the netstandard2.0 and Framework 4.7.2 targets simultaneously. The cognitive overhead of understanding which source file belongs to which project through the linking model slows every modification.
+
+## 10. Compile-Time Feature Decisions
+
+Twilio SMS/MMS support is wrapped in #if TWILIO conditional compilation across three files with ten total occurrences. The ExternalMessage base class conditionally includes the Twilio message type in its enum, meaning the set of message types the system understands changes depending on which build configuration was used. ExternalMessage_Twilio.cs is bracketed with the conditional. MessageController.cs conditionally includes Twilio message reception handling.
+
+The messaging abstraction is not truly polymorphic. Whether the system can send SMS is not a runtime decision based on configuration — it is a compile-time decision based on preprocessor symbols. This creates a build matrix (with and without TWILIO), prevents single-binary deployment, and means that testing the SMS path requires a separate build from testing the non-SMS path.
+
+## 11. Caching Without Safety Guarantees
+
+DataCache.cs implements a singleton pattern without thread safety. The instance method checks if the static field is null and creates a new DataCache if so, with no lock or memory barrier. The Initialize method has the same pattern: a null check followed by assignment and property configuration, with no synchronization. The nullify method creates a new instance, calls refresh on it, then sets the field to null — three separate operations on shared state with no atomicity guarantee.
+
+Cache keys are built through string concatenation, with account and instance identifiers prepended to prevent cross-tenant leakage. But key construction has no type safety — a malformed key is a runtime failure, not a compile-time error. The Account_Request cache tier relies on HttpContext.Items, which ties request-scoped caching to the ASP.NET Framework pipeline. Cache invalidation is manual-only: there is no mechanism to automatically invalidate cached entity metadata when the underlying entity definition changes, which means stale metadata is a class of bug that exists by design and is addressed operationally rather than architecturally.
+
+In a multi-tenant system where cache isolation is a security requirement — where one tenant's data must never be served from another tenant's cache — these weaknesses represent real risk.
+
+## 12. Commented-Out Code as Version Control
+
+The ":-/" marker appears in 146 files — not as a TODO or a FIXME, but as the original developer's inline expression of discomfort with a solution. "MICKEY-MOUSE" appears in five files. Beyond these markers, the codebase carries substantial volumes of commented-out code that preserves old approaches inline. Filter.cs contains commented-out implementations alongside their replacements. EntityColumn.cs carries over a hundred lines of commented-out operator overloads and data type conversion methods. RenderableController contains a 40-line commented-out block with notes about a reconciliation that never happened. The _dboioController dispatch method includes three different commented-out CORS implementations labeled "OLD OLD OLD."
+
+Commented-out code inflates files, obscures current behavior, and forces every reader to determine which lines are active and which are artifacts. In a codebase without comprehensive tests, it also represents a temptation: when a change breaks something, the old approach is right there, ready to be uncommented, creating a cycle where code is never truly removed.
+
+## 13. What This Means for the Rewrite
+
+These are not independent problems. They are facets of a single underlying issue: the codebase grew for fifteen years without enforcing the architectural boundaries that the design implies.
+
+Dependency injection must replace ambient static state. This is not one item on a list — it is the prerequisite for every other improvement. Until classes receive their dependencies explicitly, you cannot test them, you cannot run them outside the ASP.NET Framework pipeline, and you cannot reason about their behavior without tracing global state through 187 files.
+
+Entity must be decomposed into separate concerns: a schema definition that describes a table, a data container that holds records, and a persistence service that reads and writes. EntityActionCollection must be broken into a parser, a validator, and an executor that can be tested and modified independently.
+
+The rendering pipeline must formalize token replacement ordering into an explicit, documented, testable sequence rather than an implicit convention. The three template tag syntaxes must be collapsed to one from day one, before any tenant templates are created against the new system.
+
+SQL generation must use parameterized queries for identifiers as well as values, or at minimum use a query builder that makes injection structurally impossible rather than conventionally avoided.
+
+The build must target a single modern .NET runtime. Conditional compilation for feature toggling must be replaced with runtime configuration and genuine polymorphism.
+
+The codebase that emerges from the rewrite should be one where the architecture described in the white paper is visible in the code — where the boundaries between schema, data, persistence, rendering, and security are enforced by the type system and the dependency graph, not just by convention and discipline.

package/plugins/claude/dbo/docs/_audit_required/dbo-problems-performance.md

@@ -0,0 +1,109 @@
+# DBO.io: Performance and Scalability Problems in the Product Requirements
+
+## 1. Purpose
+
+The implementation problems document examined the code — god classes, ambient state, string-assembled SQL. This document examines something different: the product requirements themselves.
+
+The platform's performance ceiling is not primarily a code quality issue. It is a consequence of the features that define DBO.io's value proposition. Template composition, cell-level security, live schema changes, token resolution, multi-tenancy with physical database isolation — the same features that differentiate the platform are the ones that create its scaling bottlenecks. A page that assembles itself from reusable embedded components is powerful until the embed resolution triggers sequential database round-trips. Cell-level security that is transparent to template authors is a genuine differentiator until the implementation makes every query unique and uncacheable. Live schema changes that require no deployment are compelling until the cache invalidation strategy is all-or-nothing.
+
+The rewrite cannot fix performance by writing better code for the same design. It must redesign how these features work. The architecture must change so that composition is batchable, security produces stable query plans, token resolution is single-pass, metadata loading is incremental, and the rendering pipeline can stream. What follows names each bottleneck precisely, grounded in the codebase, so the rewrite addresses the structural constraints — not just the code smell.
+
+## 2. Template Composition Creates N+1 Query Cascades
+
+The product promises that any content can embed any other content, and embeds can nest arbitrarily. This is the mechanism by which pages are assembled from reusable components without code. It is the core authoring primitive. And it is inherently expensive.
+
+EmbedProcessor_RegEx.processEmbeds (lines 141-295) processes embeds sequentially in a while loop. For each iteration: find the next embed via regex match, push it onto the RequestStack, call et.render() which triggers database queries to load and execute the embedded content, pop it off the stack, then replace the embed tag in the content string using StringBuilder.Remove and StringBuilder.Insert (lines 234-236). After replacement, convert the StringBuilder back to a string (line 236), then scan forward for the next embed starting after the inserted content (line 260).
+
+There is no batching. There is no parallel execution. A page with 10 embeds triggers 10 or more sequential database round-trips — each embed loads its content record, resolves its output queries, and renders its template before the next embed is even discovered. With three levels of nesting, a realistic page can produce 50 to 100 queries, each waiting for the previous embed to fully render before the next one starts.
+
+The StringBuilder operations compound the cost. Each Remove/Insert pair on a large content string creates O(n) memory operations for the removal and O(n) for the insertion, and this happens once per embed. For a page with substantial content, the string manipulation alone becomes significant.
+
+The sequential processing is not merely a lazy implementation choice. Inner embeds can depend on outer context through the RequestStack parameter inheritance mechanism. When an inner embed uses #{request@key} tokens, those values are resolved by walking the RequestStack — which includes all parent embeds currently in scope. This context dependency between nesting levels makes parallelization architecturally difficult under the current design. The product requirement that made page composition powerful — embeds can inherit context from their parents — is the same one that forces sequential execution.
+
+## 3. Cell-Level Security Makes Every Query Unique
+
+The product promises entity-level, column-level, and row-level security enforced transparently before data reaches templates. The author writes a template; the platform ensures the user only sees what they are authorized to see. This is the right requirement. The implementation strategy is the problem.
+
+EnforceSecurityFilter.getWhere (lines 142-211) generates WHERE IN clauses for row-level security. For each security operation, it retrieves the list of accessible row IDs, then builds a parameterized IN clause. Line 198 constructs each parameter name as:
+
+    @security_ + DateTime.Now.Ticks + _ + operation + index
+
+The developer's own assessment of this approach is embedded in the comment on the same line: "What could go wrong? :-/"
+
+What goes wrong is that every security-filtered query string is unique to the millisecond. The database engine cannot reuse execution plans because the parameter names change on every request. A user with access to 1,000 rows produces a WHERE clause with 1,000 uniquely-named parameters. Two requests from the same user, one millisecond apart, produce two completely different query strings that the database optimizer treats as unrelated queries.
+
+Security evaluation runs fresh on every request through dboioSecurity.instance().accessOnEntity (line 164). There is no cross-request caching of security evaluation results. The row-level access check produces a list of accessible row IDs, but that list is never stored for reuse — it is computed, used to generate a unique query, and discarded.
+
+The combination is devastating for database performance: uncacheable security evaluation produces uncacheable queries with uncacheable execution plans. Three requests from the same user for the same data produce three different query strings, each requiring full optimization. The rewrite needs equally granular security that generates stable, cacheable query patterns — normalized parameter names, security results cached per user per session, and row-level access expressed as subqueries rather than enumerated IN lists.
+
+## 4. Token Resolution Scales as O(Tokens x Stack Depth)
+
+Dynamic tokens that resolve session, request, output, site, and content values are central to the template authoring experience. The resolution cost scales multiplicatively with content complexity.
+
+TokenReplacement.replaceFloatingTokens (lines 1091-1149) performs the final-pass token resolution over the rendered content. This single method executes: SecurityContentTag.checkAllSecurityTags scanning the full content (line 1101), replaceTokensIn for the CacheableResponse token types (line 1105), replaceCachedTokens for cached token replacement (line 1116), RequestUnique regex replacement (line 1138), and Token.replaceRawTokenValues for raw tokens (line 1144). That is five full-string scans over content that can be 100KB or more.
+
+Before this final pass, replaceTokensIn itself (lines 879-919) first does a recursive call to replace include tokens (line 892), then builds or retrieves a regex (line 905) and runs a full replacement pass (line 910). Each regex match callback calls Token.getToken to re-parse the matched token string (line 968), then _replaceAnyToken routes to getTokenReplacementForToken (line 1039), which iterates through the list of 13 TokenReplacement subclass instances (lines 1195-1214) calling acceptsToken on each until one matches (lines 1227-1229). Thirteen type checks per token match.
+
+RequestStack parameter resolution adds another dimension. getParameter_Overridable (lines 445-456) iterates the stack from bottom to top; getParameter_Restrictive (lines 469-486) iterates from top to bottom. Each lookup is O(stack depth). For a page with 20 levels of embed nesting — each embed adding an entry to the stack — every request token resolution traverses 20 entries. Five hundred tokens across a complex page, each requiring a stack traversal, produces 10,000 list iterations per render. There is no caching of parameter lookups within a render pass; the same key resolved 50 times traverses the stack 50 times.
+
+## 5. Metadata Loading Is All-or-Nothing
+
+Schema changes take effect immediately — no deployment, no recompilation. This is a defining product feature. It is also what makes aggressive caching dangerous.
+
+EntityMetadata.getEntityMetadata_internal (lines 57-68) executes a single query: SELECT * FROM entity; SELECT * FROM entity_column. This loads all metadata for every entity and every column across the entire tenant schema, every time the cache is cold. Cached at the System_Account level through dboioDataCache (line 74), but the first request per account after any cache invalidation loads the complete catalog. There is no selective loading by entity. There is no incremental invalidation — changing one column's description on one entity flushes the entire metadata cache for that account, forcing the next request to reload every entity and every column.
+
+TemplateParser compounds the problem. The constructor (lines 133-147) calls initialize(), which runs regex preprocessing to normalize template tags through a state machine (lines 165-293), then constructs an XML document string, and parse() runs XDocument.Parse on the result (line 314). This happens fresh on every request for every template encountered during rendering. There is no template compilation cache. The same template parsed identically on 1,000 consecutive requests gets regex-preprocessed, XML-wrapped, and XDocument.Parse'd 1,000 times.
+
+The product requirement for immediate schema effect means the current architecture cannot pre-compile templates — a metadata change could invalidate any template that references the changed entity. But the implementation provides no mechanism to determine which templates are affected by a specific change. The result is that nothing is cached because anything could be stale, and everything is recomputed because nothing can be trusted.
+
+## 6. Multi-Tenancy Multiplies Every Problem
+
+Physical database isolation per tenant is a product requirement. Each tenant has its own database, its own schema, its own metadata. This means every per-request bottleneck described above is multiplied by tenant count.
+
+Metadata caching, security evaluation, template parsing, and connection establishment all happen per-tenant. Cache keys include tenant identifiers through string concatenation — the cacheKeyPrefixWithAccount method (lines 465-491 of dboioDataCache.cs) builds keys by concatenating the cache part, account short name, and core data source identifier with delimiter strings. Every cache lookup constructs this composite key string.
+
+Cache invalidation is where multi-tenancy becomes punitive. The getKeys method (lines 178-210) loads the full list of all cached keys across all tenants. The refresh method (lines 380-425) iterates that full list for each key being removed. The getKeys(Parts part, Account account) overload (lines 233-249) retrieves all keys, then iterates the full list comparing prefixes to find keys belonging to a specific tenant — building a removal list through iteration rather than direct lookup. For 100 tenants with 100 cached items each, removing one key for one tenant loads and scans 10,000 entries.
+
+The HttpRuntimeCacheProvider.RemoveKeys method (lines 139-167) makes this worse: for each key to remove, it enumerates the entire cache dictionary doing case-insensitive string comparison to find the matching entry. Removing 100 keys from a cache with 10,000 entries produces 1,000,000 string comparisons.
+
+The singleton cache instance (dboioDataCache, lines 110-112) uses a null check without locking: if me is null, create a new instance. Under concurrent request load at application startup, multiple threads can create separate instances, each with its own state, before one wins the race. Shared cache infrastructure bears cumulative load across all tenants with no partitioning, no isolation, and no protection against one tenant's cache operations degrading another's.
+
+## 7. The Request Pipeline Cannot Stream
+
+Everything happens within a single synchronous HTTP request. The full response is assembled in memory before any bytes reach the client.
+
+For the chatbot subsystem, ExternalMessage_ChatBot._sendMessage (line 216) calls conversationManager.ExecuteConversationAsync(...).GetAwaiter().GetResult(). This blocks a thread-pool thread for the entire duration of a multi-turn AI conversation — potentially seconds or tens of seconds of synchronous blocking while the thread does nothing but wait for HTTP responses from an external API.
+
+For large data outputs, a query returning 1,000 rows with complex templates renders every row completely into a single in-memory string before any part of the response can be delivered to the client. The rendering pipeline has no concept of chunked or progressive delivery.
+
+The message_daemon background job (message_daemon.aspx.cs, line 575) validates the Type parameter against an allowlist of "email" and "sms/mms" only. Chatbot messages — which are the most expensive message type, involving multi-turn conversations with external AI APIs — cannot be processed by the background daemon. They must be processed inline during the HTTP request.
+
+The request pipeline itself runs 15 or more HTTP modules sequentially on every request before the controller is reached. DomainModule initializes the account context, which requires a database query. AuthenticationModule executes on both OnBeginRequest and OnPostAcquireRequestState, running authentication logic twice per request. CacheModule checks for invalidation tokens. Each module runs to completion before the next begins. There is no infrastructure for early-return, progressive delivery, or async operations that produce results incrementally.
+
+## 8. Cache Architecture Has No Invalidation Strategy
+
+Five tiers — System, System_Account, Account, Account_Session, Account_Request — with no automatic invalidation logic. Entity definition changes require manual metadata cache refresh via the /api/cache/refresh endpoint. Security evaluation results are never cached at all. Templates are re-parsed every request with no caching.
+
+The HttpRuntimeCacheProvider's thundering herd protection (lines 50-71) waits a total of 100 milliseconds (10 retries at 10ms each) for another thread to finish generating a cache entry. If generation takes longer — which it will for any non-trivial metadata or query result — the waiting thread gives up and returns default(T), causing the caller to either fail or regenerate the same data simultaneously. Under load, N concurrent requests for the same uncached item produce N parallel regenerations instead of one.
+
+The Account_Request tier is stored in HttpContext.Items (dboioDataCache.cs, lines 318-336), which is an ASP.NET Framework dependency that cannot survive migration. The Redis provider operates in hybrid mode — Redis for invalidation tokens, HttpRuntime.Cache for actual data — adding a network round-trip per cache access for freshness verification.
+
+The product requirement for live schema changes means caches can never be trusted indefinitely. But the implementation provides no mechanism to invalidate surgically. The choice is: trust everything until manually refreshed, or flush everything when anything changes. There is no per-entity invalidation, no dependency tracking between cached items, and no way to know which cache entries are affected by a specific schema change.
+
+## 9. What This Means for the Rewrite
+
+Six priorities emerge from the structural bottlenecks described above.
+
+First, embed resolution must be declarative and batchable. The rewrite should parse the full embed tree before executing any of it — identify all embeds, determine their dependencies, batch database queries for independent embeds, and execute independent branches in parallel. Sequential processing should only occur where context inheritance genuinely requires it.
+
+Second, security must produce stable, cacheable SQL. Parameter names must be normalized and deterministic. Security evaluation results must be cached per user per session, not recomputed on every request. Row-level access should be expressed as subqueries or cached temporary tables rather than enumerated IN lists with thousands of parameters.
+
+Third, token resolution must be single-pass or pre-compiled. The template compilation step should parse all tokens once, build a resolution plan, and execute it in a single traversal — not five to six regex passes over the full content string. RequestStack lookups should be indexed or cached within a render pass.
+
+Fourth, metadata loading must be incremental and surgically invalidatable. Load metadata on demand per entity, not all-or-nothing for the entire schema. Invalidate per entity when a specific entity changes. Compile templates into reusable structures that survive unrelated metadata changes.
+
+Fifth, the cache must support concurrent population and surgical invalidation. Proper stampede protection that blocks waiters until generation completes rather than timing out after 100ms. Tenant-partitioned storage and invalidation so one tenant's cache operations do not scan another's keys. Background refresh for expensive cache entries before they expire.
+
+Sixth, the rendering pipeline must support streaming. Progressive delivery so the client receives content as it renders. Async execution so chatbot conversations and external API calls do not block thread-pool threads. Chunked responses for large data outputs.
+
+The performance problems documented here are structural consequences of the product requirements that define DBO.io. Template composition, cell-level security, live schema, format-agnostic output — these are the differentiators. The rewrite cannot eliminate them. It must implement them with algorithms and data structures that scale.

package/plugins/claude/dbo/docs/_audit_required/dbo-problems-syntax.md

@@ -0,0 +1,97 @@
+# DBO.io: Syntax Problems in the Platform
+
+## 1. Purpose
+
+The white paper describes what DBO.io is and why the architecture works. The code problems document names the structural implementation failures — ambient static state, god classes, the ASP.NET Framework ceiling. This document examines the syntax: the surface that every template author, API consumer, and administrator works with daily. Syntax that evolved over fifteen years without deprecation. Delimiters overloaded across contexts. Implicit ordering dependencies that template authors must memorize. Inconsistencies between template parsers that cause the same template to work in one context and silently fail in another. The code problems are invisible to users. The syntax problems are the ones they experience.
+
+## 2. The Token Grammar's Six Delimiters
+
+The core render token format is `#{type$target@reference:modifiers!default;comment}`. Six single-character delimiters — dollar sign, at sign, colon, exclamation, semicolon, and dot — each carry positional significance within a single pair of braces. The dot delimiter is the most context-sensitive: it separates the attribute property, but only when it appears before any of the other five delimiters. Once an at sign, dollar sign, exclamation, colon, or semicolon has been encountered, dots become literal characters. The parsing code in Token.cs handles this with a special case that checks whether the current property is already the attribute property before deciding whether to switch — a rule that is invisible at the authoring level and documented only in comments that begin with "Oh jeeeeZUS."
+
+The colon delimiter carries the heaviest overloading. In a token, it separates modifiers. In a filter key, it separates the column name from the filter type. In URL parameters, it creates sub-keys like `_debug:sql` and `_debug:cache`. In the input payload format, it separates the row identifier prefix from its value and the column keyword from the entity-column pair. Four different roles for the same character across four different syntax surfaces, with no escaping mechanism to distinguish them.
+
+Nested tokens are supported through balanced-brace matching. A default value can itself be a token: `#{value@RequestPathSecondary:cached!#{request@EntityUID!}}`. The regex that handles this uses .NET's balancing group construct — named captures for DEPTH that increment on opening braces and decrement on closing braces — defined in MediaMetadataToken.cs. Literal braces within token content require HTML entity escaping (`{` and `}`), a convention borrowed from XML that is nowhere documented in the API reference.
+
+Single and double quotes act as escape wrappers within the token parser, pausing delimiter switching while active. A hash followed by an opening brace starts nested-token escaping mode. These behaviors are parsing implementation details in Token.cs that template authors encounter only when something breaks. The grammar is powerful enough to express complex dynamic content. It has no formal specification, no railroad diagram, no test suite that validates the delimiter precedence rules. The specification is the parsing code itself, spread across Token.cs in the webapp-framework and MediaMetadataToken.cs in dboioStandard.
+
+## 3. Three Tag Prefixes, Four Parsers, Inconsistent Support
+
+Template tags exist in three forms. The preferred form uses `<#_name>` with an underscore prefix for system tags. The legacy form uses `<#name>` without the underscore. Email templates additionally accept `<~name>` as a third prefix from an even earlier era. Each of the four template parsers declares which prefixes it recognizes through a RawTagPrefixes property, and each declaration is different.
+
+OutputTemplateParser and InputTemplateParser accept both `#` and `#_`. When the parser encounters `<#_row>`, it normalizes the prefix to `#_` and the tag name to `row`. When it encounters `<#row>`, both prefixes match, and the same normalization occurs. These two parsers are permissive — legacy and preferred syntax are interchangeable.
+
+MessageTemplateParser accepts only `#_`. A template tag written as `<#row>` that works perfectly in an output context will pass through unrecognized in a message context. There is no error. The tag appears as literal text in the rendered output. The failure is silent, discovered only when a template author notices that a message looks wrong.
+
+EmailTemplateParser accepts `~` and `#_`. The tilde prefix is a remnant of the original email template syntax. EmailTemplateParser supports `#_html` as a tag but MessageTemplateParser explicitly does not — the comment in MessageTemplateParser.cs reads "Conflicted with #_html in embed/output templates! Use 'body' instead." The two message-oriented parsers disagree on the name of the tag that wraps message body content.
+
+User-defined template tags use `<#name>` without the underscore, intentionally overlapping with the legacy system tag syntax. The platform distinguishes them by checking against a list of known system tag names. If a user creates a custom tag whose name collides with a system tag — which is possible because the not-allowed list contains only four entries — the behavior depends on which parser processes the template. The base TemplateParser includes a deprecated comment acknowledging this overlap and noting the intention to rely on the underscore convention, but the convention is not enforced.
+
+Internally, all tags are normalized to an XML representation using a `_dboio_` prefix. The tag `<#_row>` becomes `<_dboio_row>` in the parsed XDocument. This third representation is the one that appears in stack traces and error messages, meaning a template author debugging a failure sees a prefix they never wrote and that does not appear in any documentation.
+
+## 4. Embed Syntax: Three Forms, Two Closing Styles, Plus Serialization
+
+The EmbedProcessor defines three tag type strings in a static array: `#_embed`, `#embed`, and `EmbedToken`. Each requires its own regex for opening tags, its own regex for closing tags, and its own normalization path. The closing tag regexes in EmbedProcessor_RegEx.cs are defined separately for each form — `</#_embed>`, `</#embed>`, and `</EmbedToken>` — because the patterns differ in what whitespace and attributes they tolerate.
+
+Both self-closing and wrapping forms are supported. A self-closing embed ends with `/>`. A wrapping embed has a closing tag and can contain content between the open and close tags. With three tag names times two closing styles, there are six syntactic forms for expressing the same embed operation. The API documentation in dbo-api-syntax.md demonstrates all three tag forms on every endpoint page — preferred, alternative, and legacy — tripling the documentation surface for a single concept.
+
+Serialization adds another dimension. When embeds are nested — an embed containing another embed of the same type — the parser must match opening and closing tags correctly. The auto-serialization mechanism in EmbedProcessor_RegEx.cs appends a dot-qualified suffix to the tag name, producing forms like `<#_embed.abc123>` paired with `</#_embed.abc123>`. This is an internal mechanism that template authors should never see, but it surfaces in debug output and error messages when nested embed resolution fails, presenting yet another tag form that the author must understand to diagnose the problem.
+
+The embed type is determined by the URL within the tag, not by the tag syntax itself. A URL starting with `/api/content/` creates an EmbedToken_Content. A URL starting with `/api/output/entity/` creates an EmbedToken_OutputByEntity. A URL starting with `/api/output/` (without `entity/`) creates an EmbedToken_Output. The matching is prefix-based and order-dependent — the entity check must come before the general output check, or every entity embed would be misclassified. Six URL prefix patterns map to six embed token types, but this mapping is implicit in the order of if-else branches in a single method, not in any declared routing table.
+
+## 5. Token Processing Order as Implicit Contract
+
+Token replacement follows a specific sequence: User tokens first, then Request tokens, then Request Meta, then Site, then Cached Value, then Display. This ordering matters because tokens can nest. A default value inside a token is itself a token, and it resolves only if its type has already been processed when the outer token fails to find a value.
+
+The API documentation provides a concrete example. The token `#{value@RequestPathSecondary:cached!#{request@EntityUID!}}` works because the cached value type processes after the request type — by the time the outer token needs its default, the inner request token has already resolved. Reversing the nesting — `#{request@EntityUID!#{value@RequestPathPrimary:cached}}` — fails because the request type processes before the cached value type, so the default value passes through as a literal string with unresolved braces.
+
+This ordering is documented in one place in the API reference and nowhere else. It is not enforced structurally — there is no compile-time or parse-time check that warns when a token's default value references a type that has not yet been processed. The failure mode is silent: the unresolved token appears as literal text in the output, indistinguishable from intentional literal text unless the author knows to look for it. Template authors who do not memorize the six-step ordering produce templates that fail in ways that require reading rendering pipeline code to diagnose.
+
+The fourteen active TokenReplacement subclasses each handle one or more token types, and the ordering in which they execute determines correct behavior. The base TokenReplacement class manages this ordering, but it is convention — changing the order of subclass registration would silently break existing templates across every tenant.
+
+## 6. The Input Payload Format
+
+The input submission format packs record identification, column targeting, and values into a single delimited string: `RowID:add1;column:entity.ColumnName=value`. Three delimiters — colon, semicolon, and equals — serve structural roles, while the at sign introduces file-based values (`@filepath`). The format is configurable: the `_input_field_delimiter` and `_input_descriptor_delimiter` parameters allow changing the semicolon and colon, an adaptation for environments where these characters conflict with locale conventions.
+
+Row identification prefixes are inconsistent across operation types. Add operations use `add` concatenated directly with a numeric key — `add1`, `add2`, `addUser123` — with no separator between the keyword and the key. Delete operations use `del` followed by an underscore and the row ID — `del_10062`. Edit operations use a plain numeric ID or alphanumeric UID with no prefix at all. Three operations, three prefix conventions, sharing a single RowID field. The delete syntax additionally requires a separate `entity:entityName=true` parameter to confirm the target entity, a requirement that does not apply to adds or edits.
+
+Foreign key references between add operations create ordering dependencies. When two records are being created in a single request and one references the other, the add key of the referenced record is used as the foreign key value. EntityActionCollection resolves these references after parsing, but the resolution depends on all add operations being present in the same request and processed in the correct order. This is a transactional semantic encoded in string syntax — a concern that belongs in a structured request format, not in a delimiter-separated flat string.
+
+## 7. Filter Syntax: Four Ways to Say the Same Thing
+
+Filters use the token delimiter structure applied to URL parameters: `_filter$target@reference:modifiers=value`. Named filter operators appear as modifiers — `:contains`, `:startswith`, `:endswith`, `:exact`, `:GreaterThan`, `:LessThan`, `:GreaterThanOrEqualTo`, `:LessThanOrEqualTo`. Shorthand operators replace the modifier syntax with a symbol appended to the equals sign — `*=` for contains, `^=` for starts with, `$=` for ends with, `!=` for negation.
+
+Only four of the eight named filter types have shorthand equivalents. Contains, starts with, ends with, and negation have shorthands. Greater than, less than, and their or-equal-to variants do not. FilterBuilder.cs contains commented-out code for `<` (less than), `>` (greater than), and `<~` (less than or equal to) shorthands that were considered and abandoned. A comment on another line reads "too much thinking for these right now" — the incomplete shorthand set is not a design decision but an implementation that stopped partway through.
+
+The colon does triple duty in filter syntax. It separates the filter prefix from the column name (`_filter:ColumnName`), it separates the column name from the filter type (`_filter:ColumnName:contains`), and it chains logic modifiers (`_filter@Column:and:contains`). Filter modifiers include logic operators (`and`, `or`), inclusion operators (`include`, `exclude`), and match operators (`contains`, `exact`), all chained with colons in a sequence whose ordering is determined by parsing code in FilterBuilder.cs. The dollar sign delimiter also collides: `$=` means "ends with" in filter shorthand syntax, while `$` means "target" in the token grammar. A template author working with both filters and tokens must track which meaning applies in which context.
+
+## 8. URL Parameter Proliferation and Naming Inconsistency
+
+Pagination alone offers four overlapping approaches. `_maxrows` limits the total row count. `_rows` takes a range (`_rows=1-10`). `_page` combined with `_RowsPerPage` provides page-based navigation. `_RowsPerPage` overrides `_maxrows` when both are present, an implicit precedence rule that is documented only by the behavior of the code. The token type enum includes `maxrows`, `rows`, and `rowsperpage` as backwards-compatible aliases that all resolve to the `limit` token type — three parameter names for one concept.
+
+Format and template are split between `_format` and `_template`, which interact without a clear precedence. Minification has two parameters: `_minify=false` to disable it, and `_no_minify=true` to disable it with opposite boolean polarity. Both exist simultaneously in the API surface, both are documented, and both do the same thing.
+
+Debug parameters follow two naming conventions. The colon-separated form — `_debug:sql`, `_debug:cache`, `_debug:embeds`, `_debug:tokens`, `_debug:security` — uses the colon as a sub-key delimiter, consistent with the token modifier syntax. The underscore-separated form — `_debug_sql` — uses an underscore, consistent with standard URL parameter naming. Both forms appear in the API documentation. Authentication parameters have short aliases (`_un` for `_username`, `_pw` for `_password`, `_em` for `_email`, `_pn` for `_phonenumber`), but no other parameter category provides aliases, making the aliasing system an exception rather than a pattern.
+
+The backwards-compatible token type enum in Token.cs tells the story of this proliferation. `SessionUID` maps to `session`. `currentuser` and `loggedinuser` map to session. `CurrentSiteID` maps to `site`. `getdate` and `now` map to `date`. `response` and `response_guid` map to `unique`. `TotalRowCount`, `ReturnedRowCount`, and `GlobalRowCount` are all row count variants. Each alias represents a naming decision from a specific era that was never retired, and each remains a valid parameter that the platform will silently accept.
+
+## 9. The Restrictive/Overridable/Expansive Resolution Model
+
+When embeds nest — a content page embedding an output that embeds another output — URL parameters accumulate through the RequestStack. Each parameter carries an implicit merge strategy expressed as a token modifier: `:restrictive` (the default), `:overridable`, or `:expansive`. These use the same colon delimiter as token modifiers, filter operators, and debug sub-keys.
+
+Restrictive means the highest embed in the stack wins — parameters set by an outer embed cannot be overridden by inner embeds or by the URL. Overridable means the lowest level wins — the URL request can override what an embed specified. Expansive means values merge rather than replace. The defaults vary by token type: filters and most parameters are restrictive, display tokens and request tokens are overridable, and the default for each type is defined in a switch statement in Token.cs.
+
+The resolution is invisible to the template author. When a parameter set at the URL level is silently restricted by an outer embed's parameter, there is no error, no warning, no indication that the value was ignored. The author's URL parameter simply has no effect. Diagnosing this requires understanding the full embed nesting depth, the parameter's relater modifier (explicit or default), and the RequestStack walk direction — upward for overridable, downward for restrictive. This is architectural state management expressed as syntax modifiers on URL parameters, hidden behind a colon delimiter that already means three other things.
+
+## 10. Display Modifier Syntax: A Fourth Tag System
+
+Beyond system tags, user-defined tags, and embed tags, content templates have display modifiers using `<@key>` syntax. These are parsed by SecurityContentTag and ContentTag, not by any of the four template parsers. The `@` prefix is unrelated to the `@` in tokens (where it means reference) or in filters (where it also means reference). It is a separate mini-syntax with its own regex, its own attribute parsing, and its own URL parameter integration through the `_display@key` parameter format.
+
+Display modifiers interact with the content rendering pipeline independently of template parsing. A content record can contain `<@section>` tags that are shown or hidden based on URL parameters, security rules, or embed attributes. The security variant `<#_security>` adds group-based, entity-based, and operation-based visibility rules with comma-separated attribute values. This is a fourth tag system — system tags, user tags, embed tags, and display/security tags — each with its own prefix character, its own parser, and its own resolution rules, all coexisting in the same template content.
+
+## 11. What This Means for the Rewrite
+
+The token grammar needs a formal specification — a grammar definition that a parser generator could consume, with explicit delimiter precedence, escaping rules, and nesting semantics. Tags must collapse to one prefix. The underscore-prefixed `<#_name>` form should be the only system tag syntax from day one, with no legacy alternatives. Embed syntax must have one tag name and one closing style. The serialization mechanism for nested embeds should be internal, never visible in template content or error messages.
+
+Filters must choose between named operators and shorthand operators and implement one set completely. The colon must stop serving four roles — a dedicated sub-key delimiter for URL parameters, a separate modifier delimiter for tokens, and neither character reused for filter chaining. Input payloads should use structured JSON with explicit fields for operation type, entity, row identifier, and column-value pairs, replacing the delimited string format entirely. Token processing order must either be eliminated as a concern through simultaneous resolution, or made structurally enforceable through dependency declarations that the parser validates before rendering begins.
+
+The syntax that emerges from the rewrite should be one where a template author can learn the rules from a single reference page, where every context uses the same delimiters for the same purposes, and where failures are reported rather than silently absorbed.

package/plugins/claude/dbo/docs/_audit_required/dbo-product-market.md

@@ -0,0 +1,119 @@
+# DBO.io: Product-Market Fit in the Age of AI Agents
+
+## 1. Purpose
+
+The market DBO.io is entering with the rewrite has been reshaped by AI in ways that both threaten the original value proposition and validate the architectural thesis more strongly than at any point in the platform's fifteen-year history. What follows examines that market, argues that the rewrite's positioning must shift from what DBO.io replaces to what DBO.io enables, and identifies the specific capabilities the rewrite must prioritize to position the platform as infrastructure for the AI-native era rather than an artifact of the one that preceded it.
+
+## 2. The Original Value Proposition and Its Expiration Date
+
+DBO.io's founding pitch: eliminate development time by eliminating the middle tier. Define your tables and columns, and the platform provides the API, the rendering, the security, and the messaging automatically. The schema becomes the application.
+
+For fifteen years, this was a strong differentiator because the alternative was expensive custom development — hiring engineers to write the controllers, the service layers, the repositories, the migration scripts by hand. Every organization that could not afford that investment was stuck choosing between rigid SaaS platforms and no software at all.
+
+Two shifts have weakened this specific pitch. Low-code and no-code platforms now serve the non-developers-building-applications market with consumer-grade visual interfaces and hosted deployment. AI code assistants have made writing middle-tier code cheap and fast — an engineer with a code generation tool can produce an ORM, controllers, and a service layer in minutes rather than weeks.
+
+The original value proposition has not become wrong. It has become insufficient. "Saves you development time" is no longer a defensible market position on its own.
+
+## 3. Why AI Code Generation Does Not Solve the Problem DBO.io Solves
+
+AI code assistants make boilerplate cheaper to produce. They do not make boilerplate cheaper to own.
+
+An AI can generate a data access layer, controllers, validation, and serialization in a single session. The output compiles and the tests pass. But that generated code is now part of the codebase. It must be maintained when requirements change, debugged when it fails in production, migrated when the schema evolves, audited when a security review occurs, and understood by the next developer who inherits it.
+
+The ongoing cost of middle-tier code is not the writing but the owning. Every generated controller must be updated when a column is added. Every generated model class must be kept in sync with the table it represents. Every generated validation rule must be reviewed when business logic changes. Code generation reduces upfront cost. It does nothing about lifetime cost.
+
+DBO.io eliminates the middle tier entirely. There is no controller to maintain because there is no controller. The entity metadata is the model. Validation rules are properties of the column definition, enforced by the platform. When a column is added, the API surface updates automatically. When a validation rule changes, it changes in one place and takes effect everywhere.
+
+Code generation reduces upfront cost. Schema-driven architecture reduces lifetime cost. They address different cost centers.
+
+## 4. The Shift: From "Saves Developer Time" to "AI-Operable Infrastructure"
+
+The rewrite's positioning must shift from a platform that replaces developer effort to a platform that AI agents can build on, operate within, and reason about.
+
+The relevant market trend is not AI as a coding assistant that helps developers write applications faster. It is AI as an autonomous agent that performs tasks, makes decisions, and operates systems on behalf of users. The trajectory is clear across every category of AI tooling: from completion to conversation to action. Code assistants that suggest the next line are giving way to agents that plan and execute multi-step workflows, interact with APIs, manage data, and report results.
+
+Agents need specific properties in the systems they operate: queryable metadata describing available operations, predictable APIs with uniform response shapes, declarative configuration they can read and modify without tracing control flow, and structured security models they can reason about.
+
+DBO.io's architecture provides exactly these properties. Entity definitions, column metadata, security rules, and output definitions are all queryable data records. The API surface is uniform and generated from metadata. Configuration is declarative: adding a table or defining a security rule is a data operation, not a code change. The security model is explicit, granular, and enforced at the data level.
+
+The rewrite should position DBO.io as infrastructure AI agents consume — a structured, fully-described platform that agents can operate within as naturally as they operate within a database.
+
+## 5. Why Declarative Systems Are Natural Substrates for AI
+
+AI models struggle with large imperative codebases. Tracing control flow through layers of abstraction, understanding side effects of method calls, predicting the impact of changes across coupled components — these are tasks that require maintaining vast amounts of state context simultaneously, exactly the kind of reasoning where current models are weakest. A model asked to modify a conventional three-tier application must understand the controller, the service, the repository, the ORM configuration, the migration history, and their interactions. The surface area is enormous, and the failure modes are subtle.
+
+AI models excel at structured, queryable configuration. JSON schemas, relational structures, declarative rule sets — these are representations that models were trained extensively on and can manipulate reliably. A model does not need to trace control flow through a schema definition. It reads a record, understands the structure, and produces a valid modification.
+
+DBO.io's entity system is a declarative system. Adding a table is a data operation that produces API endpoints, security enforcement, and rendering capabilities automatically. Defining a security rule is inserting a record. Creating a query is configuring an output definition. An AI agent working within DBO.io does not need to understand a codebase. It needs to understand a schema — a relational structure that is precisely the kind of representation models handle best.
+
+The `/api/app/object/<appShortName>` endpoint demonstrates this concretely. It takes an application short name and returns a complete JSON hierarchy of the application — entities, outputs, content records, security rules, and their relationships, grouped by object type. This is the application definition itself as structured, navigable data. The `index_search` stored procedure goes further: it accepts search terms and performs weighted matching against asset UIDs, names, display titles, subtitles, attributes, and descriptions, with per-column weights from 1000 (exact UID match) down to 10 (description substring). Schema metadata as searchable, ranked, structured data — exactly the representation an AI agent needs to locate and operate on the right platform objects.
+
+## 6. The Chatbot Integration as Proof of Thesis
+
+The existing chatbot integration is not a feature bolted onto the side of the platform. It is architecturally integrated, built on the same subsystems that serve human users.
+
+The integration uses the messaging subsystem — `ExternalMessage_ChatBot` is a subclass of the same `ExternalMessage` hierarchy that handles email and SMS. AI configuration — model, system prompt, tool definitions — is stored in Content records using the template token system and a `body_wrapper` tag that `ChatBotConversationManager` parses as JSON to construct API requests. AI behavior is configured the same way every other platform behavior is configured: as data records with token-resolved dynamic values.
+
+Function calling is where the integration most directly proves the thesis. Three mechanisms exist, each routing AI tool calls back through the platform's own infrastructure. Convention-based routing: function names prefixed with `output_` or `content_` auto-resolve to DBO.io API endpoints — `output_dashboard` becomes `/api/o/dashboard`, `content_report` becomes `/api/c/report` — with arguments appended as URL parameters and results processed through the embed pipeline. `#include` content tag aliases: the `IncludeContentTag` system maps function names to content records, with a `chatbot_halt` attribute for flow control that returns results directly to the user instead of back to the model. Development stubs for functions like `get_weather` and `search_database` that demonstrate the extension pattern.
+
+Function arguments flow through the platform's token resolution pipeline via the `IRequestStackable` interface. The `BasicFunctionExecutor` implements `IRequestStackable`, exposing parsed JSON arguments as request tokens. Any template rendered during a function call resolves function arguments using the same `#{request@key}` syntax used in every other template context. The AI agent's tool call arguments become indistinguishable from URL parameters from a human user's browser.
+
+Conversation history is persisted to `message` and `message_log` tables with ThreadID-based threading, creating a queryable, auditable conversation trail stored in the same relational structure as every other platform record. The architecture did not need to be redesigned to support AI — the integration uses the same embed processing, token resolution, and security enforcement that already existed.
+
+## 7. What the Integration Gets Wrong
+
+The chatbot integration proves the architectural thesis. It does not reach production quality for the agentic pattern it partially implements.
+
+The conversation loop is synchronous. `ExecuteConversationAsync` is called with `.GetAwaiter().GetResult()`, blocking a thread for the entire conversation — every API call, every function execution, every database save. For agentic workflows with multiple tool calls across multiple iterations, a thread is blocked for what could be a multi-minute operation.
+
+Single tool call per response. `ParseChatBotResponse` reads `message["tool_calls"][0]` only — a comment reads "TODO: This is where we're just defaulting to the first choice, and ignoring any others!" No streaming — responses arrive as complete JSON payloads. No token counting, no cost attribution, no rate limiting — in a multi-tenant platform where AI usage has real monetary cost. No retry or backoff. No multi-model support — the server address is a single configured endpoint with no fallback capability.
+
+The `MICKEY-MOUSE` markers in `ExternalMessage_ChatBot.cs` are the developer's own assessment. They appear on the lines where function call results are formatted as embed URLs — the exact mechanism that routes AI tool calls through the platform's rendering pipeline. The developer built the integration, recognized its architectural significance, and simultaneously flagged it as not yet ready.
+
+The `message_daemon` confirms the background processing gap. Its `getMessagesAndRecipients` method validates the `Type` parameter against an explicit allowlist: `"email"` and `"sms/mms"`. Chatbot messages, which use `Type = "Chat"`, cannot pass this validation. Chatbot conversations cannot be queued and processed in the background by the same daemon that handles email and SMS delivery. Every chatbot interaction must be initiated and completed synchronously within a single HTTP request.
+
+The integration was built rapidly over four months in 2025, deepening with each version — from basic completion to function calling to `#include` aliases to `chatbot_halt` flow control to `body_wrapper` template configuration. Each iteration added capability without revisiting the synchronous foundation. The result is an integration that demonstrates everything the architecture can support and nothing that production agentic workloads require.
+
+## 8. AI as a First-Class Consumer of the Platform
+
+The rewrite should treat AI agents as first-class API consumers alongside human users. Not a special integration path. Not a separate endpoint. The same API, the same security, the same audit trail.
+
+An AI agent authenticates, receives a session, and is subject to entity-level, column-level, and row-level security. Its queries are filtered before results are returned. Its operations are logged, its data access is auditable, its permissions are configurable through the same security data records that govern human access.
+
+The architecture already supports this because security is enforced at the data level, not at the controller level. The security model does not care who is asking — it cares what permissions the authenticated identity holds. This holds whether the consumer is a browser, a mobile application, or an AI agent.
+
+What is missing is an identity model that represents AI agents explicitly. The chatbot integration currently operates within the human user's context — inheriting the user's session, permissions, and audit identity. This is adequate for a chatbot assisting a specific user. It is inadequate for autonomous agents that operate on behalf of an organization, perform scheduled tasks, or interact with the platform without a human in the loop.
+
+The rewrite needs an identity type for AI agents that supports group memberships, security rule assignments, and audit trail attribution distinct from any human user. An agent that processes invoices should have access to invoice entities and nothing else, enforced by the same security rules that restrict a human accounts payable clerk. AI agents are users of the platform with the same guarantees and the same constraints.
+
+## 9. Natural Language as an Interface to the Schema
+
+If the schema is the application, then natural language becomes a natural interface to the schema.
+
+The platform already has the metadata to support this translation. Entity names describe the data. Column names describe the fields. Column metadata describes types, constraints, relationships, and display titles. Security rules describe who can access what. Output definitions describe available queries. The `index_search` stored procedure already demonstrates search against this metadata — accepting free-text search terms and returning ranked results matched against asset UIDs, names, display titles, subtitles, attributes, and descriptions with weighted scoring that distinguishes exact matches from partial matches.
+
+What `index_search` does with keyword matching, embedding-based semantic search could do with natural language. A user who asks "show me all overdue invoices" does not need to know that the entity is called `invoice`, the column is `DueDate`, and the filter syntax is `_filter:DueDate:LessThan=2026-02-14`. Given vector embeddings of entity names, column names, column descriptions, and relationship metadata, a semantic search could resolve "overdue invoices" to the correct entity and construct the correct API call — or present the user with a disambiguation if multiple entities match.
+
+This is not general-purpose natural language to SQL. General-purpose natural language to SQL is an unsolved problem because it requires understanding arbitrary database schemas with no semantic context. What DBO.io offers is constrained translation against a known, fully-described schema — every entity has a name and a description, every column has a display title and a data type, every relationship is explicit in the metadata. The search space is bounded. The metadata provides the semantic context that general-purpose approaches lack. This is a dramatically simpler problem, and it is a problem that the platform's existing metadata architecture is uniquely positioned to solve.
+
+## 10. What This Means for the Rewrite
+
+Seven priorities, in dependency order. Each builds on the ones before it.
+
+**Async and streaming infrastructure.** Server-sent events for response streaming. Background processing for agentic conversations. Non-blocking I/O throughout the conversation pipeline. The `message_daemon` type validation must expand to include chatbot messages, enabling background-processed agentic workflows with the same queuing and retry semantics that email delivery already has.
+
+**Multi-model and multi-provider support.** Model selection as configuration data stored in message server records, consistent with the platform's pattern of making infrastructure choices into data. Fallback providers for resilience. The single hardcoded server address replaced with a provider abstraction resolved from configuration, the same way data source selection is resolved from entity metadata today.
+
+**AI identity and security.** Agents as first-class identities with group memberships, security rules, and audit trails. Permissions configurable through the same security data records that govern human access. Audit log entries that attribute operations to specific agent identities. Session management that supports agent-initiated operations without a human user context.
+
+**Parallel tool calling and structured outputs.** Handle multiple tool calls per response, executing them concurrently when they are independent. Structured output schemas for reliable agentic workflows where the model's response must conform to a defined shape. The current single-tool-call limitation removed entirely.
+
+**Embeddings and semantic search.** Vector storage for schema metadata — entity names, column descriptions, relationship definitions. Integration with the existing `index` and `object` tables so that semantic search augments rather than replaces `index_search`. The vector store as another data source type within the existing abstraction.
+
+**Observability and cost management.** Token counting per request and per conversation. Cost attribution per tenant, per agent, per conversation type. Rate limiting as a configurable property of message server records — a new dimension of multi-tenant isolation where AI consumption is bounded the same way database connections and cache sizes are bounded.
+
+**Natural language schema interface.** The capstone: natural language resolving to platform operations against known metadata. Each prior capability feeds into this one — embeddings provide the search, structured outputs provide reliable translation, AI identity provides the security context, async infrastructure provides the execution model.
+
+DBO.io's thesis — the schema is the application — was ahead of the market it entered. The competitive threat is not that someone else eliminated the middle tier. No one has. The competitive threat is that AI has made the middle tier cheap enough to tolerate. The cost of writing boilerplate code has dropped so dramatically that "we eliminate the boilerplate" no longer commands attention on its own.
+
+The rewrite must demonstrate that a declarative, schema-driven platform is a better substrate for the AI-native future than AI-generated imperative code. Generated code must be maintained by the humans or agents that generated it. A schema-driven platform maintains itself — add a column, and the API updates; add a security rule, and it enforces everywhere; add an entity, and every AI agent on the platform can discover and operate on it through the same structured metadata. The schema is still the application. Now it is also the substrate AI agents build on.