brut 0.0.21 → 0.0.23

data/brutrb.com/database-access.md

@@ -0,0 +1,201 @@
+# Database Access / Data Models
+
+Brut provides access to the database via the [Sequel library](https://sequel.jeremyevans.net/).  Sequel is fully featured and provides a lot of ways of interacting with and managing your database.  Brut includes several plugins and extensions to provide opinionated default behavior or additional features.
+
+One thing to keep in mind is that Brut refers to your database layer as *database models* (notably not the un-qualified "models").  Brut treats this layer as a *model* of your database, not a model of your *domain* (though you are free to conflate the two).
+
+This section details how to access data in your database.
+
+> [!NOTE]
+> Brut currently only supports Postgres. Sequel supports many database systems, however Brut's extensions are
+> currently geared toward Postgres only.
+
+## Overview
+
+Accessing your database in Brut uses Sequel's `Sequel::Model`.  A base class called `AppDataModel` exists in your
+app from which all other data models extend:
+
+```ruby
+# app/src/back_end/data_models/app_data_model.rb
+AppDataModel = Class.new(Sequel::Model)
+class AppDataModel
+  # You can insert your own shared methods here
+end
+
+# app/src/back_end/data_models/db/account.rb
+class DB::Account < AppDataModel
+end
+```
+
+All data models are in the `DB` namespace.  This clearly identifies a model as a model of your database and
+not your domain.
+
+Inside a data model, you can use all of Sequel's API. In particular, you will want to use its API for
+[associations](https://sequel.jeremyevans.net/rdoc/files/doc/association_basics_rdoc.html) so that you can create
+relationships between models.
+
+In your business logic or front-end code, you can access your data using these models:
+
+```ruby
+account = DB::Account.find(email: form.email)
+```
+
+> [!IMPORTANT]
+> Sequel's `Sequel::Model` is different from Active Record, especially when it comes to associations.
+> `account.organizations` would return an `Array` of `DB::Organization` records, all fetched from the database.
+> `account.organizations_dataset` would return a active-relation style object to allow stacking
+> quieries.  **Please** familiarize yourself with Sequel's API.
+
+## Testing
+
+Testing, as it applies to data models, is made up of two parts: managing test *data* and testing the models
+themselves.
+
+### Test Data is Managed with FactoryBot
+
+Brut apps come with [FactoryBot](https://github.com/thoughtbot/factory_bot) installed, and this is how you should
+create test (and seed) data.
+
+Factories for data models live in `specs/factories/db`. Because data models are in the `DB` namespace, you will
+need to explicitly state the `class:` in the factory, but otherwise, you can use FactoryBot in a conventional
+way. [Faker](https://github.com/faker-ruby/faker) is also installed to allow you to create realistic and
+randomized data.
+
+Here is a factory for our hypothetical account:
+
+```ruby
+# specs/factories/db/account.factory.rb
+FactoryBot.define do
+  factory :account, class: "DB::Account" do
+    email         { Faker::Internet.unique.email }
+    organization
+
+    trait :inactive do
+      deactivated_at { Time.now }
+    end
+  end
+end
+```
+
+The `spec_support.rb` file generated when you created your Brut app should ensure that `FactoryBot::Syntax::Methods` is included in all specs, meaning you can do `create(:account)` to create an instance of `DB::Account`.
+
+See [Unit Tests](/unit-tests) for more details on testing and Factory Bot setup.
+
+### Testing Your Data Models
+
+In general, you don't want to test the configuration in your data models. For example, testing that
+`account.organization = organization` works is largely pointless, since this is provided by Sequel.
+
+That said, if you have complex or unusual database constraints, having a test for them can be valuable.
+
+Suppose our `DB::Account` has the following check constraint that requires an email end with `@example.com`:
+
+```ruby {13-16}
+Sequel.migration do
+  up do
+    create_table :accounts,
+      comment: "People or systems who can access this system",
+      external_id: true do
+
+      column :email, :text, unique: true
+      foreign_key :organization_id, :organizations
+      column :deactivated_at, :timestamptz, null: true
+
+      key [:email, :organization_id]
+
+      constraint(
+        :email_must_be_domain,
+        "email ~* '@example.com$'"
+      )
+    end
+  end
+end
+```
+
+To test this, you would try to write invalid data into the database an ensure the expected exception is raised:
+
+```ruby {11}
+# specs/back_end/data_models/db/account.spec.rb
+require "spec_helper"
+RSpec.describe DB::Account do
+  describe "email" do
+    it "must end in @example.com" do
+      expect {
+        DB::Account.create(
+          email: "pat@example.net",
+          organization: create(:organization)
+        )
+      }.to raise_error(Sequel::CheckConstraintViolation)
+    end
+  end
+end
+```
+
+If you don't want to be overly coupled to Sequel's exceptions, you can also assert on the message Postgres will
+produce, which would include the name of the violated constraint:
+
+```ruby {11}
+# specs/back_end/data_models/db/account.spec.rb
+require "spec_helper"
+RSpec.describe DB::Account do
+  describe "email" do
+    it "must end in @example.com" do
+      expect {
+        DB::Account.create(
+          email: "pat@example.net",
+          organization: create(:organization)
+        )
+      }.to raise_error(/email_must_be_domain/)           
+    end
+  end
+end
+```
+
+## Recommended Practices
+
+### Do Not Put Business Logic On Your Database Models
+
+There's no reason to, or benefit to doing so.  What you'll find is that any app of even moderate complexity will
+not have a strict mapping from page to business concept to database table.  Rather these things will all differ
+greatly, and each serves a different purpose.
+
+The job of your data models—and the tables they provide access to—is to store reliable and unambiguous data.
+Their job is to ensure there is no bad data such that when you ask the database a question, you get a reliable
+and correct answer.
+
+Your views and business logic do not have this exact same job.
+
+As such, your models should only contain:
+
+* configuration to allow navigating the database.
+* methods to manage type conversions between your types and the strings or numbers required in the database
+* methods to query the data based on data definitions (not business logic).
+
+Business logic and data models *do* overlap at times, so there is some judgement in maintaining a clear
+separation of concerns.  One way to manage this is to always put all logic elsewhere until you see a pattern of
+re-use that leads you to extract that logic to a data model.
+
+### Do Not Use Validations on Models Unless There is No Other Choice
+
+Sequel provides a validation layer for use on models.  You should not generally use this, since a) data integrity
+is baked into your database design, and b) user interactions and constraints are part of the front-end.
+
+That said, there are times when you have data constraints that cannot be modeled in the database.  In that case,
+a validation on the data model is better than nothing.  Since all data access for your app should go through your
+data models, a validation on a data model has a high chance of being checked.
+
+> [!NOTE]
+> Since any process, app, or tool can manipulate your database, model-based validations won't be 
+> in effect, and therefore won't be applied.  This is why you design your schema to avoid invalid
+> data wherever possible.
+
+
+## Technical Notes
+
+> [!IMPORTANT]
+> Technical Notes are for deeper understanding and debugging. While we will try to keep them up-to-date with changes to Brut's
+> internals, the source code is always more correct.
+
+_Last Updated May 8, 2025_
+
+None at this time

data/brutrb.com/database-schema.md

@@ -0,0 +1,312 @@
+# Database Schema / Migrations
+
+Brut provides access to the database via the [Sequel library](https://sequel.jeremyevans.net/).  Sequel is fully featured and provides a lot of ways of interacting with and managing your database.  Brut includes several plugins and extensions to provide opinionated default behavior or additional features.
+
+One thing to keep in mind is that Brut refers to your database layer as *database models* (notably not the un-qualified "models").  Brut treats this layer as a *model* of your database, not a model of your *domain* (though you are free to conflate the two).
+
+This section details how to manage your database schema.
+
+> [!NOTE]
+> Brut currently only supports Postgres. Sequel supports many database systems, however Brut's extensions are
+> currently geared toward Postgres only.
+
+## Overview
+
+Brut uses *migrations* to control and manage the schema of your database. Migrations are changes to the
+schema that depend on the changes before them.  In a running production database, you will not be able to
+create the database schema from scratch—you will have to modify the existing schema to produce the schema
+you want.
+
+For example, if you have a table `widgets` that has a `name` and `description`, to add a `status` field,
+you cannot `drop table widgets` and then `create table widgets(...)` with the fields. You must instead
+`alter table widgets(...)` to add the new column.
+
+Thus, each migration file is a change to the schema produced by all previous migration files. 
+
+Brut's provides this via Sequels. See [both](https://sequel.jeremyevans.net/rdoc/files/doc/schema_modification_rdoc.html) [docs](https://sequel.jeremyevans.net/rdoc/files/doc/migration_rdoc.html) for details on the API.  Any schema modification method Sequel documents is available, however some default behavior has changed.
+
+Schema files are located in `app/src/back_end/data_models/migrations` and are named using a timestamp-based
+scheme.  This means that when you create a new migration, its name will be based on the time and date you
+created it, and any migrations that have not been applied will be applied in timestamp order.
+
+### Creating Migrations
+
+To create a migration, use `bin/db new-migration`. It accepts any  number of arguments that will be joined
+together to form the filename:
+
+```
+> bin/db new-migration user accounts
+[ bin/db ] Migration created:
+    app/src/back_end/data_models/migrations/20250508132646_user-accounts.rb
+```
+
+The file is created mostly blank:
+
+```ruby
+Sequel.migration do
+  up do
+  end
+end
+```
+
+Sequels' migration system is similar to Active Record's/Rails' in design and spirit, but the API is different.
+Please consult the documentation and don't assume Active Record's DSL will work. It will not.
+
+One thing to note is that Brut encourages the creation of only "up" migrations. That is, migrations that change a
+database. "Down" migrations, which revert a change, are discouraged.  See *Recommended Practices* for a detailed
+explanation.
+
+This is also why Sequel's `change` method is not included in the scaffolded code.  `change`, like Active Record's
+method of the same name, automagically creates both "up" and "down" migrations, but *only* if you use the DSL. If
+you use raw SQL, `change` doesn't work.  But that doesn't matter for Brut (again, see *Recommended Practices*).
+
+Let's create a user accounts table that has an email field, a `deactivated_at` timestamp, and a `created_at`
+timestamp:
+
+```ruby
+Sequel.migration do
+  up do
+    create_table :accounts,
+      comment: "People or systems who can access this system" do
+
+      column :email, :text, unique: true
+      column :deactivated_at, :timestamptz, null: true
+
+    end
+  end
+end
+```
+
+A few notes that aren't obvious without knowing about Brut's extensions:
+
+* `comment:` is required. You must provide documentation about what table is for
+* The table has a primary key named `id` of type `int` that is a serial.
+* `created_at` is created by default, with time `timestamptz` (AKA `timestamp with time zone`, see [Space/Time Continuum](/space-time-continuum)).
+* `email` is not null by default.  `deactivated_at` *is* null because it's specified as such.
+
+To apply this migration use `bin/db migrate`
+
+```
+> bin/db migrate
+```
+
+If you create a new migration, it will use a timestamp that is alphanumerically greater than the one we just made
+and thus that migration will be applied after this one.  Thus, you can rely on previous migrations having been
+applied when authoring new ones.
+
+### Managing Migrations
+
+Sequel uses a special database table to understand which migrations have been run.  This table will exist in
+production and prevent you from applying migrations twice or skipping a migration.
+
+Note that managing a production database in this way requires knowledge of both your database system and the data
+itself.  Brut can only provide so much to make this process manageable.  You should consult [Strong Migrations'
+README](https://github.com/ankane/strong_migrations?tab=readme-ov-file) and learn it deeply. Although it's
+targeted at Rails developers, the information here applies to any database management system.
+
+### Brut Extensions and Changes in Sequel's Behavior
+
+Brut includes the following standard plugins and extensions:
+
+* [`pg_array`](https://sequel.jeremyevans.net/rdoc-plugins/files/lib/sequel/extensions/pg_array_rb.html)
+* [`pg_json`](https://sequel.jeremyevans.net/rdoc-plugins/files/lib/sequel/extensions/pg_json_rb.html)
+* [`table_select`](https://sequel.jeremyevans.net/rdoc-plugins/classes/Sequel/Plugins/TableSelect.html), which
+changes queries to prepend `*` with the table name, e.g. `select accounts.*` instead of `select *`.
+* [`skip_saving_columns`](https://sequel.jeremyevans.net/rdoc-plugins/classes/Sequel/Plugins/SkipSavingColumns.html) which will skip saving columns that the database generates.
+
+Brut also provides the following plugins and behavior changes:
+
+* `Sequel::Extensions::BrutInstrumentation`, which adds OpenTelemetry instrumentation to Sequel (see [Instrumentation](/instrumentation)).
+* `Sequel::Plugins::FindBang`, which adds `find!` to all models. This wraps Sequel's `first!` method, but
+provides a more helpful error message when no records are found
+* `Sequel::Plugins::CreatedAt`, which automatically sets `created_at` when a record is created.
+* `Sequel::Plugins::ExternalId`, which adds support for external IDs (see below)
+* `Sequel::Extensions::BrutMigrations`, which enhances the migrations API (see below)
+
+#### External IDs
+
+It's often useful to provide a unique identifier for a record that is not the database primary key.  There are
+many advantages to doing so, but the core value Brut has regarding this is that database primary and foreign keys
+are considered private and internal and for developer use only.  It is trivial to produce externalizable keys, as
+you'll see, so there's no reason to expose primary keys.
+
+> [!NOTE]
+> Take care to differentiate the terms *primary key* and *key*.  In relational database literature, and thus
+> in Brut, a *key* is any value that uniquely identifies a record. `email` in the `accounts` table above
+> is a key.  The *primary key* is single source of truth for identifying records internally in the database.
+> Thus, it can be used as a *foreign key* to create relationships between tables.  Brut further implements
+> this as a *surrogate* or *synthetic* key, which means the value itself has  no business  or domain meaning.
+
+In Brut, an external ID is automatically generated by the database when a record is created.  By convention, it
+is prefixed with a short string representing your app and a short string representing the table, followed by a
+unique hash.
+
+For example, if our app's prefix is, say, "my" (for "my app"), and the accounts table's prefix is "ac" (for "accounts"), an external ID might look like `myac_3457238947239487`.  This double-prefixing is extremely useful when sharing these values with the outside world.  You can immediately identify an ID from your app *and* know what sort of thing it refers to.
+
+To use external IDs in Brut, you must do three things:
+
+1. You must set your external ID prefix in `app/src/app.rb`. This should have been done when you created your
+   Brut app, but it looks like so:
+
+   ```ruby {5}
+   class App < Brut::App
+     # ...
+     def initialize
+       # ...
+       Brut.container.override("external_id_prefix","my")
+     end
+    
+     # ...
+   end
+   ```
+2. When creating the table in a migration, use `external_id: true`:
+   ```ruby {5}
+   Sequel.migration do
+     up do
+       create_table :accounts,
+         comment: "People or systems who can access this system",
+         external_id: true do
+
+         column :email, :text, unique: true
+         column :deactivated_at, :timestamptz, null: true
+
+       end
+     end
+   end
+   ```
+3. In your [data model class](/database-access), use `has_external_id` to specify the prefix for this table:
+
+   ```
+   class DB::Account < AppDataModel
+     has_external_id :ac
+
+     # ...
+   end
+   ```
+
+Brut creates the external ID using Ruby code as part of Sequel's lifecycle hooks.  It's only set a) on creation, and b) if there is no value provided when creating the record.
+
+This means that you can set values explicitly if you like, *and* you can change them later.  This is useful if you shared the value with someone you didn't mean to.  Because these external IDs aren't use for referential integrity/foreign keys, they can be changed at any time, as long as the value is unique (which will be enforced by the database).
+
+### Brut Migration Changes and Enhancement
+
+Brut attempts to set default behavior for migrations to encourage a modicum of best practices.  This lists out
+the changes and a brief explanation for the purpose of the change.
+
+* **Automatic synthetic primary key named `id` of type `int`.**  You almost always want this. You can change the
+primary key configuration per table if you like, but if you do nothing, you get a primary key that works for 99%
+of your needs.
+* **Automatic `created_at` of type `timestamptz`.** It's a good practice to store the date a record was created. This can help with debugging and provide a reliable sort key for data that otherwise has none.  It uses `timestamp with time zone`, which you are encouraged to use always. See [Space/Time Continuum](/space-time-continuum) for details.
+* **No automatic `updated_at`.** While you are free to add `updated_at`, in practice this column creates more problems than it solves.  If you need to know when data has changed, it is almost always better to do this with an audit table, event log, or special-purpose field.
+* **`create_table` requires `comment:`.** Just document your tables.  It takes two seconds and can save a lot of
+time later.
+* **Support for external IDs via `external_id:`.** As discussed above, this will create a unique `external_id`
+column on your table and ensure it has a value on creation.
+* **Columns are `NOT NULL` by default.** Null is not a valid value.  In many cases, your columns should not allow
+`NULL` (`nil`), so in Brut apps, you must opt into nullable columns. You can use `null: true` to make a column
+nullable.
+* **Foreign keys are `NOT NULL` and have an index created for them by default.** Foreign keys should rarely be
+`NULL` and you almost always want an index on them, since you are likely to using them in queries, e.g.
+`account.widgets` would join on `accounts.widget_id`.  You can opt out of either via `null: true` and `index:
+false`.
+* **The method `key` allows you to specify a non-primary key, AKA a unique index**.  Suppose our `accounts` table
+allowed duplicate email addresses, but only one per `organization_id`. You'd model this by creating a unique
+index on `(email,organization_id)`.  In Brut:
+  ```ruby {11}
+  Sequel.migration do
+    up do
+      create_table :accounts,
+        comment: "People or systems who can access this system",
+        external_id: true do
+
+        column :email, :text, unique: true
+        foreign_key :organization_id, :organizations
+        column :deactivated_at, :timestamptz, null: true
+
+        key [:email, :organization_id]
+
+      end
+    end
+  end
+  ```
+
+  This allows your migrations to be more expressive *and* make it easier to set up unique constraints that relate
+  to your business logic or domain.
+
+## Testing
+
+Generally, you don't test database migrations, however you may want to test constraints or other logic you have
+set up.  Techniques for doing this are in the [database access](/database-access#testing) section.
+
+## Recommended Practices
+
+### Ephemeral Dev Database
+
+Brut intends for your develompent database to be ephemeral.  Your entire workflow should be built around it
+being OK and normal to completely blow away your development database and recreate it.  This is why down
+migrations (and the  use of `change`) are discouraged. You really don't need them.
+
+Assuming you have [seed data](/seed-data) set up properly, you can reliably reset everything like so:
+
+```
+> bin/db rebuild
+> bin/db seed
+> bin/db rebuild -e test
+```
+
+As long as you don't change migrations that have been applied in production, you can safely run the above
+commands to iterate on a schema change.
+
+This does imply that you should not run business logic or make *data* changes in your migration files.  No migration
+should rely on specific data being in the database.
+
+This workflow my be much different from what you are used to, but you will be quite happy when you adopt it.  The days
+of downloading a carefully-curated database image and taking great care to never delete it are over.
+
+### Use Your Database, It is Awesome
+
+Your database is the only part of the system that has any chance of ensuring data integrity.  You can use
+constraints, types, foreign keys, etc. to ensure that the data in your database is correct, based on your current
+understandings.  Code-based validation systems **cannot achieve this on any level**.
+
+Thus, you are encouraged to learn about your database's features and use them!
+
+For example, here's a way to add full text search to an existing table in Postgres:
+
+```ruby
+add_column :full_text_search,
+  :tsvector,
+  generated_always_as: Sequel.lit(%{
+    (
+      setweight(to_tsvector('english', name),'A') ||
+      setweight(to_tsvector('english', coalesce(description,'')),'B')
+    )
+  }),
+  generated_type: :stored
+```
+
+If you are using Postgtes, why *not* use its features?  Unless your app is database-agnostic, you should be using
+the features of your database, even if they aren't explicitly exposed via Sequel's Ruby API (that's why `Sequel.lit` exists).
+
+## Technical Notes
+
+> [!IMPORTANT]
+> Technical Notes are for deeper understanding and debugging. While we will try to keep them up-to-date with changes to Brut's
+> internals, the source code is always more correct.
+
+_Last Updated May 8, 2025_
+
+As mentioned, Brut uses Sequel under the covers.  This is unlikely to change.
+
+As also mentioned, Brut's extensions often rely on Postgres.  While we can all dream of a world where every
+developer uses the same database server, we don't live in that world.  Brut should, some day, support all the
+databases that Sequel supports.  For now, however, it only supports Postgres.
+
+This hard-coded support is due to:
+
+* `pg_array`
+* `pg_json`
+* Reliance on `citext` and `comment`
+* Reliance on `timestamptz`
+
+Brut is likely to add more Postgres-specific features before adding support for other databases.

data/brutrb.com/deployment.md

@@ -0,0 +1,66 @@
+# Deployment
+
+Brut apps are Rack apps, so they can be deployed in conventional
+ways.   Brut apps are 12-factor apps and the scripts used for
+development are inteded to work for production as well.
+
+## Overview
+
+Everyone deploys apps in different ways.  Brut can't provide a
+simple solution for all deployment setups, so this document will
+outline considerations when setting up deployment.
+
+The most direct way to understand what needs to happen is to look
+at `deploy/Dockerfile`, which is the foundation of a `Dockerfile`
+you can use.  In particular, it shows you the commands needed to
+setup and run the app in production:
+
+Beyond installing system software to run any Ruby web app, as well
+as whatever is needed for NodeJS and Postgres, the Brut-specific
+parts look like so:
+
+1. Install Ruby Gems with `bundle install`
+2. Install Node modules with `npm clean-install`
+3. Build all assets with `bin/build-assets` (this will bundle all
+   CSS and Javascript, plus copy over any other [assets](/assets)
+   to the locations from where the Brut app will serve them)
+4. Run the app with `bin/run`
+
+Your Brut app also includes `bin/release` which is a script
+intended to run in the production environment after the code has
+been deployed, but before the app starts up.  By default, it
+applies any needed migrations to the database.
+
+## Testing
+
+If you are using Docker, you can create the `Dockerfile`s and run
+them locally to see how they work.  You will need to have local
+versions of all infrastructure (database, Redis, etc.), but if
+these work locally, there is a high chance they work in
+production.
+
+If you are not using Docker, you will need to apply various
+techniques that are beyond the scope of this documentation.
+
+## Recommended Practices
+
+Brut goes to great lengths to avoid environment-specific code.
+Much of Brut's behavior works the same in dev as it does in
+production. For example, assets are hashed in all environments.
+
+Assuming your code does the same thing, there should be a minium
+of surprises.  That all being said, here are some recommendations:
+
+* Create a way to interact with external services in a testing
+capacity. For example, ensure you have a test user with a known
+email address and trigger an email to them.  Or a company credit
+card you charge and refund.
+* Configure observability so you know what your app is doing at
+all times.
+* Configure a URL that, when accessed, produces an error. This
+allows you to check your error reporting system.
+* Create a page somewhere that shows the git SHA of your
+deployment, or some other unique, unambiguous version number. This
+will clarify what version of the code is actually running.
+
+