@loadbare/app 0.4.0 → 0.5.0

package/docs/theory.md

@@ -1,226 +1,347 @@
 # Theory of Loadbare App
 
-Loadbare App is a web framework that answers the author's pain points with modern
-web development: a paradoxical situation in which developer ergonomics appear
-to dominate framework architecture but we end up with developer tools that
-inflict increasing pain and expense at scale.  User
-experience, especially speed, is left as "an exercise for the reader" or waved
-away as irrelevant due to "powerful modern hardware."  
-
-The result is that it is incredibly expensive to make a web app, and the result
-is usually very slow. 
-
-The iterative effort began with Axiom Zero: The best
-developer experience is creating an application that users appreciate. 
-A framework for application development must put user needs first, and
-craft the resulting solution patterns to developer needs after the best
-user result is identified and achieved.
-
-So we begin with user experience, a fancy way of saying, "why we built this
-darn app in the first place, and what makes people likely to come back."
-Modern expectations for a web app are numerous and span multiple domains.  The
-one domain that seems to have been forgotten as inconvenient to --toy-making-- 
-developer tool forging is throughput, or performance.  Loadbare App therefore puts
-performance first, every decision must result in a performant application.
-
-This leads to Axiom 1: Performance drives all architecture.  Primary
-decisions drive performance, secondary decisions do not subvert it.
-
-## The Performance Budget
-
-Research over the past 50 years concludes that users appreciate a tight average
-of ~300ms response time from a request to a completed paint.  
+by Ken Downs, August 29, 2026.
 
-As the round trip is the critical path, we look at a round trip and break down
-the budget into wire 
-time, browser time, and server time.  Wire time is limited by the speed of light,
-so we pick 200ms as a strong median for a browser and server located in roughly
-the same region.
-
-With 200ms consumed on the wire, that leaves 100ms for the server and the browser.
-Within that 100ms, there is browser paint, which we control, and request handling,
-which the framework can structure, and then database response time.  Since the
-database is the one thing outside of our control, we come to a simple conclusion:
-
-> Global Requirement 1: The framework must consume as little of the
-> 100ms as possible, leaving
-> as much of the 100ms budget as possible to the database.
-
-
-## The Split
-
-Since the author knew the fat browser frameworks could never be twisted into
-shape to satisfy the ground requirements, he experimented mostly with htmx, and
-investigated dataStar, Turbo, and others.
-
-What became clear is that all modern offerings have two things in common:
-
-COMMEN ELEMENT 1: They all assume that any DOM node can be mutated, added, or
-removed at any time.  Most of the run-time expense and framework complexity
-is a direct result of this assumption.  
-
-COMMON ELEMENT 2: Their templating systems mix the shape of a response (the
-HTML/CSS and some JS for behavior), with the content of the response (the stuff
-we got from a data store, search engine, relational database or what have you).
-
-> *Note: Phoenix LiveView is the only exception I found, it was rejected
-> for reasons explained in [prior-art.md](../docs-llm-slop/prior-art.md).
-
-The final iteration, which led to Release 1.0, only got off the ground
-once the fundamental split was identified:
-
-> The UI is shipped as a static and permanent "host" for data that is supplied
-> on a separate channel.  Page navigation swaps in the page "host" code, and
-> user interaction refreshes data that the framework must push into the DOM.
-
-This split does not preclude conditional rendering or list processing, but it
-significantly changes their shape.  There are plentiful examples in the
-[guide.md](./guide.md).
-
-## Some Practical Wisdom
-
-Given a collection of ideals, a tight budget, and one lonely architectural
-decision, a few more ideas were needed to maintain alignment with Axiom zero,
-that user experience and developer ergonomics must be aligned.
-
-0. Use the fewest mechanisms that are most expressive
-1. Leverage the browser, don't fight it or supersede it.
-2. Stick with HTML, don't invent a templating system
-3. Ship example CSS, but don't force our solution path
-4. Find a sane default for shipping Javascript
-5. Use community solutions where they exist (eg, Express), 
-   don't reinvent a building block one exists that is not going
-   to chew too much of our 100ms budget
-6. The build is the developer-equivalent "speed trumps everything", the
-   fastest build has the fewest steps, and the easiest build has the
-   least configurations
-7. Prefer explicit over magic, except in the build, we do not want
-   the build to force boilerplate into the code.
-8. Authentication and Authorization should be demonstrable but are
-   squarely outside of scope.
-9. Keep architectural elements loosely coupled, but don't over-force
-   that to the detriment of a coherent solution that can be used
-   as-is. 
-10. For 1.0, create a minimal working solution whose elements can
-    be extended without fundamental refactoring.
-
-## Implementation Decisions
-
-Custom Elements with Light DOM.  Loadbare App connects javascript to HTML in
-the simplest way possible.  Custom elements and light DOM
-leverage the browser instead of fighting it, and they naturally fit
-the role of code in a tree, which is to manage itself and its descendants.
-
-A browser "hub" with two jobs.  Job 1 is to respond to navigation requests
-by swapping in the HTML "host" for the page.  Job 2 is to catch events,
-send requests to the server, and update host elements that display results.
-
-Provide a fixed set of data shapes, from cell (scalar), to a tuple (a
-single collection of cells), or a projection, a collection of tuples.
-
-Have the hub locate nodes to update with attributes like `lb-query`,
-`lb-key` and `lb-cell`.  Supplement the vocabulary with special ops
-for row deletions or adds, and both CRUD and non-CRUD requests to the
-server.
-
-A page system that organizes the HTML host template and the server-side
-code with minimal or zero boilerplate. 
-
-
-
-## Settled Implementation Approaches
-
-### Widgets
-
-A widget is an HTML custom element without Shadow DOM. Widgets compose,
-up to the level of a page.
-
-### Defining a Page
-
-A "page" is the static host of the HTML main landmark. It can be written into
-a plain old HTML document. How it is packaged, delivered and cached is settled
-in [bundling.md](../docs-llm-slop/bundling.md).
-
-We assume a page-level server-side state system.
-
-A page also has a set of declared queries in a data structure, mapped to
-the server-side state variables they require. A page GET must execute
-all queries and deliver all results.
-
-A page is a set of files sharing a basename: the host, the queries it
-declares, and the hooks it declares. The name of the file says which page
-it belongs to, so nothing inside it has to.
-
-User interactions break down into three groups:
-
-- Fire-and-forget. The user changed sort order on an HTML TABLE that
-  is smart enough to sort its own rows. It still fires a fire-and-forget
-  update to server state, so that user choice can be fetched on the next
-  page load
-- CRUD operations with hooks, declared per page: the table, the operation,
-  whatever custom code it runs, and the queries it refreshes.
-- Declared actions, for work that is not CRUD. The page names what it can be
-  asked to do and what each one refreshes. The browser sends a declared name
-  and nothing else, so the server decides every value.
-
-A hook always states which queries it refreshes. There is no automatic
-mapping from an operation to the projections it affects. The exception is a
-hook that runs before a page GET, which needs no such declaration because
-the whole query set runs after it.
-
-### Data Updates
-
-A data update can be a full hydration or rehydration of page, or a refresh of
-some elements based on a user action. In all cases, the following statements hold.
-
-We assume the host HTML is fully loaded and upgraded with JS.
-
-We define an "addressable" widget as one with custom attributes that allow the
-hub to identify which widgets are responsible for the data fragments it has
-received. A widget is addressed by three coordinates: the declared query, the
-tuple key, and the cell.
-
-A cell lands as text when it addresses a native element and as an attribute
-when it addresses a widget. A native element has no behavior of its own, so
-the value is simply its text; a widget owns whatever control it wraps, so it
-receives the value and renders it. The test is the browser's own rule for what
-a custom element is, so the hub holds no knowledge of any particular element.
-
-Projections land by method call, and tuples get no mechanism of their own.
-
-Because the browser runs `attributeChangedCallback` for attributes already
-present when a widget upgrades, hydration and refresh are the same operation,
-and hydration timing is invisible to widget code.
-
-The attribute vocabulary and the binding mechanism are specified in
-[data-binding.md](../docs-llm-slop/data-binding.md).
-
-## Open Implementation Topics
-
-Binding, addressing and the update mechanism are settled in
-[data-binding.md](../docs-llm-slop/data-binding.md). The request vocabulary is settled in
-[requests.md](../docs-llm-slop/requests.md). Packaging of the static half is settled in
-[bundling.md](../docs-llm-slop/bundling.md). Expansion of authored markup is settled in
-[expansion.md](../docs-llm-slop/expansion.md). What remains:
-
-topics to expand:
-
-- values bound to executable attributes. Markup injection is closed:
-  values are applied through the DOM and never parsed. An attribute that
-  is executable in its own right — href, src, style, on\* — is not, and
-  the first widget that binds a URL has to answer for it.
-- navigation replaces a static host rather than data, the one operation
-  that does replace host DOM. What makes something a new host versus new
-  data for an existing host is the daily modeling decision.
-- nesting. Whether a tuple may contain a projection (master-detail, an
-  expanding row). If it may not, that limit is load bearing.
-- staleness and concurrent writers. Two tabs or two users against one
-  projection. Do tuples carry a version, or is last-write-wins the
-  stated position?
-- validation placement. Per-keystroke feedback cannot afford a round
-  trip, so the budget forces some validation into the widget while the
-  server remains the source of truth.
-- the declarative path for setting a page-state variable to a literal. A
-  native element can send a declared action, because an action carries
-  nothing; a control that writes a literal directly has no markup for it
-  yet.
+Loadbare is one man's answer to the extreme expense and slow performance
+of modern web apps.  When using modern frameworks, it seems to me that
+they are composed entirely of accidental complexity, I seem to be always working
+for the tools instead of the other way around.
+
+I wanted a new way to develop web apps that reduced accidental complexity
+as far as possible to zero, and gave me snappy performant apps.
+
+This essay describes the analysis and conclusions that led to the
+design and implementation of the Loadbare application layer.
+
+## The Principle of Developer Experience
+
+First I had to find my way out of the cruft that clutters
+the front end discussions of today, and establish a "Principle zero" that
+could be a starting point.  This is it:
+
+> The Principle of Developer Experience: The best developer experience is
+> creating an application that people use and appreciate.
+
+This principle combines and aligns business concerns with engineering
+concerns, grounding any solution in why we do what we do.
+
+This principle also
+pre-empts arguments for seemingly elegant abstractions and tooling that
+cannot be obviously and easily shown to contribute directly to an
+application that end users use and appreciate.
+
+## The Principle of User Experience
+
+The Principle of Developer experience grounds us in the end user,
+and from here I considered my own largest single complaint about
+web apps: they are so damn slow. 
+
+It seems to me that the front end industry, as represented by the
+fat browser frameworks, has abandoned performance completely.
+
+I wanted to put performance front and center, and design around
+it instead of optimizing for it later.
+
+After some research into what people perceive as performant, I came
+upon the number of 300ms.  Faster than that seems abrupt, slower 
+seems sluggish.  This led to the next foundation principle:
+
+> The Principle of User Experience: a full server round trip should
+> complete, from user action through durable persitence to final
+> paint, in a median time of 300ms.
+
+This scopes the goal to performance and performance only.  All other
+aspects of the UI are mature, we have a mature inventory of widgets
+for user display and interaction.  We do not need to invent new
+solutions for accessibility, tabs, buttons and so forth.  What we need
+is a way to use the mature solution and respect the user's time.
+
+But we do have a practical implication: if we have a 300ms budget,
+and roughly assume 200ms median wire time, we must split the remaining
+100ms between database, non-db server, and browser.  Since the database
+in the limiting case must provide durable persistence, our ideal
+processing time for browser and non-db server activity must
+approach zero.  In other words, our solution must be a model citizen,
+chewing up as few cycles as possible while providing the full modern
+experience of a web app, leaving
+as much of the 100ms as possible to the database.
+
+## The Primacy of Essential Complexity
+
+Now we move on to developer efficiency.  Fred Brooks gave us a powerful
+way to consider this, when he made the distinction between "essential difficulties"
+and "accidental difficulties".  These terms have since been glossed into
+"essential complexity" and "accidental complexity", and I will
+use the complexity versions to be intelligble to the casual reader.
+
+Essential complexity is the inherent difficulty of solving the user's
+problem.  This is where we want to spend our time and effort.
+Accidental complexity is the work demanded by the tools we use to solve
+the problem.  We want to minimize this to being vanishingly small.
+
+Because I believe there is some objective truth to my subjective feeling
+that the fat browser frameworks have become nothing but accidental complexity, 
+it seemed necessary to state the princple of the Primacy of 
+Essential Complexity, which borrows phrasing from both Brooks and
+Pike to establish that:
+
+> Essential Complexity Dominates.  Show me your abstractions and tooling and
+> I shall remain mystified.  Show me your tables and your visual vocabulary
+> and I shall be enlightened - the organization of the code will be self-evident.
+
+
+## The Search For a Solution
+
+The three principles stated above, once articulated, allowed me to
+clarify my rejection of the fat browser frameworks: nowhere do they
+state the same motivations that I have, and their illustrations and
+examples make clear their motivations are irrelevent to mine and
+vice-versa.  In fact, my
+extensive experience with React, Angular and VueJS contributed to 
+my articulation of these principles, the principles state what I found
+lacking in the fat browser frameworks.
+
+This led me to look at the Hypermedia libraries.  After several weeks
+of iterations, I found myself frustrated.  These were very clearly
+closer to what I wanted, and the various authors' motivations seemed
+very close to my own, but I was still seeing more tooling concerns
+than I wanted.  I do not blame the hypermedia libraries, I think I was
+fighting them.  But that simply told me we differed on an assumption,
+and that assumption remained unidentified and unstated. 
+
+It turns out that all of the tools and frameworks I could find are
+supporting something I don't need: a DOM that is fully mutatable at
+any time, for any reason deemed necessary by the developer.  They
+all provide or expect an HTML templating system that combines
+interpolation of data values with conditional and list rendering,
+controlled by run-time attributes.  All of them must chase down the
+consequences of the decision to allow a fully mutatable DOM.  Those
+consequences spill into the usage and tooling around the frameworks
+and libraries.  The arbitrarily mutatable DOM was the source of my
+intuitive sense that I was dealing with too much accidental complexity.
+
+## The Loadbare Split
+
+Finally I had the foundational idea for a framework that splits code
+between a static "host" application and an active data channel, with
+some small framework library updating only data values in the DOM
+by following data-binding attributes in the HTML.
+
+(As a side note, I originally considered calling Loadbare "liveload"
+to draw the association to a bridge.  The static HTML is the bridge,
+the data that refreshes the page is the "live load" (traffic) travelling
+on it.  Alas, it seemed too close to 'livereload' and I gave it up.)
+
+The static host application is pure HTML, including custom elements
+with Light DOM.  User interactions send requests to the server which
+returns data objects.  The HTML contains attributes identifying how
+elements are bound to data, so the framework can update them.
+
+This should be self-evidently efficacious for scalar cells and
+tuples.  Something like `<span lb-query="siteStats" lb-cell="visitCount"></span>`
+or the same type of addressing for an input makes it trivial to 
+both hyrdrate and refresh forms, and to gather a form's values
+to send to the server.
+
+List processing, such as for an HTML SELECT and complex tables took a bit
+longer to figure out, but the principle could be extended.  Any list
+element can accept a `<template>` that describes how to build its 
+items.  The data channel must expand to allow "patch" operations,
+removing or adding one row instead of reshipping the entire 
+query, but that is also straightforward.
+
+So a purist might say, "hey you are still manipulating the DOM".
+Of course we are, it is an application, not a static site.  But the
+point is that the HTML can be written, read, and reasoned
+about as a static artefact, while still being dynamic at runtime.
+
+The model can also be extended for trees, but my current projects
+do not require this, so it is not implemented as of this writing.
+
+## Choosing the Mechanisms
+
+So now I had three principles and one architectural decision that 
+seemed to make it all possible. The next step was to pick the
+fewest number of mechanisms that would work.  
+
+On the benefit side, the fewer mechanisms there are the easier
+it would be to create the solution and to use the solution.  Fewer
+mechanisms makes for a system that is easier to learn, and easier
+to teach to an LLM.
+
+On the cost-avoidance side, each mechanism is
+an invitation to accidental complexity around the use of the
+mechanism, so they must be kept at a minimum. 
+
+### Browser Application Code
+
+Using plain HTML, extended with custom elements that do not use Shadow
+DOM has so far, apparently, completely solved the browser for me.
+
+In my humble opinion,
+Javascript belongs in the browser to enhance default behavior, and
+the enhancements ought to always be scoped to a custom element and
+its children.  A custom element neatly satisfies this.  
+
+Custom elements give an nice answer to the question, why can't we just
+use HTML like we used to?  The Loadbare answer is: you can, go ahead.
+If you need something special, make a custom element, it is still
+just HTML.
+
+A simple builder also gives server-side HTML includes for free with
+no extra syntax.  The builder scans for one fixed file (the chrome),
+and all pages.  Anywhere a custom element is found, it looks for
+an HTML file to match, and inserts it into the built HTML.  This is recursive
+of course.
+
+The builder became "tree shaking" for free.  During HTML assembly
+and composition, any Typescript file whose name matches a custom
+element that was actually used gets added to the monolithic 
+`client.js`.  No import statement required.  The Loadbare code is 
+loading no extra Javascript outside of the single `client.js` named
+in the main HTML document.
+
+The validation during a build also fell out for free: if a custom
+element is used, it must have either a matching Typescript file
+or a matching HTML file, or both.  The only error is if neither
+are present.
+
+### Data Binding
+
+For the hub to know what data to apply where and how, we need 
+custom attributes on the HTML.   It was possible to define a small
+and coherent vocabulary because the actual number of operations
+afforded by a database is small and coherent.
+
+### Browser Framework Code
+
+The entire browser framework code is in the hub `<lb-hub>`, and its
+only real job is to handle page navigation and the data channel.  
+
+It also contains addressable methods to update DOM elements with
+data objects, so that custom elements do not have to reproduce that
+code.
+
+### Application Server Code
+
+For server page code, this is where we truly got accidental complexity
+to zero.  A page is just its HTML, its queries, and its request handlers.
+
+For the Express server, I was rather surprised myself to find that the
+simplest thing was to show example code for an Express server.  Nowadays
+most frameworks provide their own Express server that is so completely
+invisible that a newcomer to the industry can be forgiven for not even
+knowing that Express exists.  I wondered if pushing this concern to the
+dev team violated the Principle of Essential Complexity, but I really
+don't think so.
+
+The reasoning for not providing a shrink-wrapped
+server is simple: the boilerplate code is one-time effort, and the dev
+team will need to add in their own database handler, security and 
+other concerns.  A one-time effort may be seen as accidental 
+complexity to a skeptic, but its one-time set up means it is not a 
+continual tax being paid every time you write a new page.  It is that
+sense of paying an ongoing tax that I'm really trying to avoid.
+
+### The Database
+
+Database concerns are mostly out of scope for the application layer,
+but we must make mention of them because Loadbare is supposed to
+be optimized for database apps.
+
+Loadbare satisfies the minimal principle of not being rude by not
+requiring a specific database, neither prohibiting nor requiring
+the ORM of your choice if you use one.  
+
+However, I should mention that I personally always use
+the `@loadbare/db` package to build
+my databases, which enriches them with calculated values that are
+declarative in the Loadbare db schema.  For this reason, I need no
+ORM because the access patterns are very simple: most database reads
+come from one table or a fairly simple view, and user actions that
+update a single table automatically cascade calculations 
+to related tables.  For this reason, I do not need an ORM and
+my `@loadbare/app` applications connect directly to the database
+with no "business logic" code at all.
+
+I must also state that a team not using `@loadbare/db` will probably
+use an ORM or craft a business logic layer, and `@loadbare/app` does
+nothing special for this.  It simply provides the empty slot where
+you plug it in.
+
+### The Builder
+
+The builder kind of fell out for free from the architecture and 
+mechanism choices.
+
+In addition to what was noted above, it is a very straightforward
+piece of code: it walks the `src/` tree, requiring only a 'chrome.html',
+and picking up all '*.page.html' files.  It builds one Javascript file,
+one CSS file, and one HTML file.  Having the files be sparse, containing
+only what was used, is trivial.
+
+It also builds the application's API from the page's processing
+code, and 4 lines in the Express server end up handling the entire 
+API.  
+
+In short, the builder satisfied the Principle of Essential Complexity, that
+if the code is organized around just application concerns, the builder
+algorithm is self-evident, straightforward, and easy to decorate with
+nice-to-haves.
+
+## Assessing the Result
+
+I am a biased examiner, but I can say that `@loadbare/app` solves mutliple
+problems for me and achieves compliance with the principles stated
+at the top of this essay.
+
+The Principle of Developer Experience says my goal is to make something
+that users actually use and appreciate.  Loadbare cannot do that for me,
+but it does free me from so many accidental concerns that my time is 
+free to pursue that quality application.  It reduces the cost of iteration
+and increases my ability to focus on the shape of the solution. That is
+as much as I can ask for from a productivity tool.
+
+In terms of the Principle of User Experience, that we must complete
+round trips from click to paint in a median time of 300ms, Loadbare gives
+me a better chance of doing this than any other tool I have seen.
+The Loadbare app is so light and does so little that if we allow a median
+time of 200ms for wire time, Loadbare likely needs 10ms or less between
+the browser and the server, leaving 90ms for the database and business
+logic.  Making the budget in production can be reduced to careful modeling,
+tuning, and hardware spend.  (Note: using `@loadbare/db` makes this almost
+a guaranteed win, provided we don't skimp on hardware and know how to tune
+Postgres for production).
+
+Finally, in terms of the Primacy of Essential Complexity, I may be a biased
+examiner.  Because I wrote it, I see zero accidental complexity, inasmuch
+as everything seems to be there to serve the major principles.  But I cannot
+trust my judgment there for obvious reasons.  Instead I will propose that
+there are some reasonably obvious wins for Essential Complexity that even
+a skeptic might agree with.
+
+One big win is the ability to just write plain HTML.  Mixing HTMl and
+Javascript seems cool if you need a fully mutatable DOM, but when you realize
+you don't need that, it's back to plain old reliable HTML.
+
+Another big win is the server code for pages.  The clean organization
+of hooks, actions, CRUD and queries without boilerplate is pure
+Essential Complexity.  The automatic building of the API and 4 lines to
+handle it in Express is lower than I have seen in any other tool.
+
+Loadbare satisfies many other principles that do not merit a full explanation,
+such as using technology instead of fighting it, organizing the use of
+browser mechanisms instead of reimplementing them, and encouraging code
+that is likely to have a very long shelf life.  Loadbare does not
+interfere with normal CSS usage, so you can bring any solution you like.
+
+In closing I would add one more claim.  Loadbare is easy to learn because
+it has a small cognitive surface area.  The main activity, coding pages,
+is highly repetive and driven by a handful of patterns.  It can be learned
+by a person and taught to an LLM with just a few skills.  
+
+I wrote Loadbare originally in 2003, as "Andromeda", before Node existed
+and even 4 years before we had JQuery.  Andromeda was not nearly as
+optimized as Loadbare is now, but the principles were the same then as 
+they are now.  It has always worked for me, and I hope that it will work
+for you.