cybertrain 0.2.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -30,24 +30,27 @@ template engine, SQLite-backed models with validations, callbacks and
30
30
  schema-derived associations, migrations, the `cybertrain` CLI, and a
31
31
  development loop that rebuilds and re-execs the server on code changes.
32
32
  [examples/blog](examples/blog) is the Rails "Getting Started" blog (articles
33
- with comments) built on it, exercised by CI on Linux and macOS. Not built:
34
- authentication, mailers, jobs, WebSockets/ActionCable, an asset pipeline,
35
- i18n, and anything beyond a minimal `render json:` — see "Differences from
36
- Rails" below.
33
+ with comments) built on it, exercised by CI on Linux and macOS.
34
+ `cybertrain generate authentication` adds email-and-password sign-in the way
35
+ Rails 8's generator does, and `cybertrain generate attachment` (or a
36
+ `cover:attachment` scaffold field) adds a file upload to a model. `Cybertrain::Mailer` sends plain-text e-mail over SMTP
37
+ (see "Sending e-mail"). Not built: Action Mailer-style mailer classes and templates, jobs, WebSockets/ActionCable,
38
+ an asset pipeline, i18n, and anything beyond a minimal `render json:` — see
39
+ "Differences from Rails" below.
37
40
 
38
41
  ## Try it in the browser
39
42
 
40
43
  [Open the playground in GitHub Codespaces](https://codespaces.new/saeki-mototsune/CyberTrain?quickstart=1)
41
44
  to try cybertrain without installing anything: VS Code opens in your browser with
42
- Spinel 2026.09.12, the `cybertrain` CLI and the blog from the walkthrough below
45
+ Spinel 2026.10.26.7019, the `cybertrain` CLI and the blog from the walkthrough below
43
46
  already created (`cybertrain new blog`, the article scaffold, the root route and
44
47
  the first migration, built once so the server starts without compiling), its
45
48
  development server running in a terminal and the app, showing the article list,
46
- in the editor's preview. The app is a git repository with one commit, so Source
49
+ in a new browser tab. The app is a git repository with one commit, so Source
47
50
  Control shows what you change. `PLAYGROUND.md` in the app lists what to try, from
48
51
  editing a view to carrying on with the walkthrough's comments. A view edit shows
49
52
  on the next reload; a Ruby edit is a full rebuild, about a minute, after which the
50
- server restarts by itself; the preview does not reload by itself. `cybertrain new`
53
+ server restarts by itself; the app's tab does not reload by itself. `cybertrain new`
51
54
  works there with no network.
52
55
 
53
56
  - A GitHub account is required, and the codespace runs on your own Codespaces
@@ -55,8 +58,12 @@ works there with no network.
55
58
  month, about 60 hours on the default 2-core machine the playground uses. GitHub
56
59
  stops an idle codespace after 30 minutes by default; its storage counts until
57
60
  you delete it.
58
- - Inside the preview, pop-ups and `confirm()` dialogs do not work; the Ports
59
- view's "Open in Browser" shows the app in a normal tab.
61
+ - If VS Code asks whether you trust the authors of the files in this folder,
62
+ choose "Trust Folder & Continue": the server's terminal starts only then.
63
+ - If the app's tab did not open (a pop-up blocker, or the codespace was stopped
64
+ and restarted), click "Open in Browser" in VS Code's notification for port
65
+ 3000 or in the Ports view, or Ctrl-click the app's URL in the `server`
66
+ terminal. `PLAYGROUND.md` shows where each is.
60
67
  - The same image runs locally (it is published for linux/amd64; on arm64, build
61
68
  it from a checkout):
62
69
 
@@ -69,8 +76,11 @@ works there with no network.
69
76
 
70
77
  ## Requirements
71
78
 
72
- - Spinel `2026.09.12` (`Cybertrain::SPINEL_TAG`, the same release as
73
- `SPINEL_TAG` in [.github/workflows/ci.yml](.github/workflows/ci.yml));
79
+ - Spinel `2026.10.26.7019` (`Cybertrain::SPINEL_TAG`, the same release as
80
+ `SPINEL_TAG` in [.github/workflows/ci.yml](.github/workflows/ci.yml)), a
81
+ release of [saeki-mototsune/spinel](https://github.com/saeki-mototsune/spinel)
82
+ (`Cybertrain::SPINEL_REPOSITORY`), the fork of matz/spinel that carries
83
+ the fixes cybertrain needs ahead of an upstream release;
74
84
  untested against other versions. You do not install it by hand: the
75
85
  `cybertrain` command builds that release into `~/.cybertrain` the first
76
86
  time it needs it (`cybertrain setup` does that on request), unless a
@@ -80,10 +90,24 @@ works there with no network.
80
90
  - SQLite 3 headers/library (`libsqlite3-dev` on Debian/Ubuntu; present with
81
91
  Xcode's command line tools on macOS) — models use Spinel's FFI directly,
82
92
  not a gem.
93
+ - zlib headers/library (`zlib1g-dev` on Debian/Ubuntu; in the macOS SDK) —
94
+ gzip request bodies (`request.decoded_body`) are inflated through the
95
+ system zlib, bound the same way.
83
96
  - Ruby 3.2+ and RubyGems, for the `cybertrain` command only (applications
84
- never run on CRuby).
97
+ never run on CRuby). Building Spinel also needs Ruby's `prism` gem, which
98
+ Ruby 3.3 and newer bundle; on Ruby 3.2, `gem install prism` (with Ruby's
99
+ development headers, `ruby-dev` on Debian/Ubuntu). `cybertrain doctor`
100
+ checks it.
85
101
  - OpenSSL's headers (`libssl-dev`) are optional: without them Spinel builds
86
- without its `openssl` package, which cybertrain does not use.
102
+ without its `openssl` package, which cybertrain does not use. On macOS an
103
+ application that requires `openssl` needs Homebrew's (keg-only) OpenSSL on
104
+ the compiler's paths: `export LIBRARY_PATH="$(brew --prefix openssl@3)/lib"
105
+ CPATH="$(brew --prefix openssl@3)/include"`.
106
+ - jemalloc's development package (`libjemalloc-dev`; `brew install jemalloc`)
107
+ is optional: an app needs it only once its `spin.toml` says
108
+ `allocator = "jemalloc"`, which `cybertrain new` writes commented out. It
109
+ is worth turning on for production: 1.1 to 1.4 times the requests per second on
110
+ examples/blog ([docs/benchmark.md](docs/benchmark.md)).
87
111
 
88
112
  `cybertrain doctor` checks these tools and headers and the Spinel install;
89
113
  when a required one is missing it also prints the install command for your
@@ -128,6 +152,18 @@ commands (`cybertrain help` prints them):
128
152
  `spin run gen` in it, installing Spinel first when it is needed.
129
153
  - `cybertrain generate scaffold NAME field:type ...` (alias `g scaffold`)
130
154
  adds a resource to the application in the current directory.
155
+ - `cybertrain generate authentication` adds email-and-password sign-in to
156
+ it: users and sessions tables, `User` and `Session` models, the sign-in
157
+ and sign-up controllers and views, the routes, a `before_action` in
158
+ `ApplicationController` and a test (see "Add authentication").
159
+ With `--invite-only` there is no sign-up page: users are created with
160
+ `./NAME user EMAIL`, and the first one also at a `/setup` page that
161
+ answers only to the server's own machine, with the one-time token its
162
+ boot warning prints, while no user exists.
163
+ - `cybertrain generate attachment MODEL FIELD [--any]` adds a file upload
164
+ to a scaffolded model: five metadata columns, model methods and
165
+ callbacks, a download action, the route, the form field and a test (see
166
+ "Add a file upload"). A scaffold field `cover:attachment` does the same.
131
167
  - `cybertrain db migrate` generates, applies pending migrations, and
132
168
  generates again (`spin run gen; spin run db -- migrate; spin run gen`);
133
169
  `cybertrain migration` is kept as an alias.
@@ -141,7 +177,7 @@ commands (`cybertrain help` prints them):
141
177
  `dist/` (the binary, `public/`, `storage/`, `tmp/`).
142
178
  - `cybertrain spin ARGS...` runs `spin` with the pinned Spinel from any
143
179
  directory (`cybertrain spin test`, `cybertrain spin run gen -- --check`).
144
- - `cybertrain setup [--force]` installs Spinel `2026.09.12` into
180
+ - `cybertrain setup [--force]` installs Spinel `2026.10.26.7019` into
145
181
  `~/.cybertrain` unless a `spinel` of that release is already on `PATH`;
146
182
  `--force` rebuilds the copy under `~/.cybertrain`.
147
183
  - `cybertrain doctor` checks the C toolchain, the SQLite headers and the
@@ -155,10 +191,11 @@ Every command that runs `spin` (`new`, `db`, `server`, `build`, `spin`) first ma
155
191
  sure a Spinel of the release in `Cybertrain::SPINEL_TAG` is available. A
156
192
  `spinel` on `PATH` (with a `spin`) whose `spinel --version` reports that
157
193
  release is used as it is; otherwise the copy `cybertrain` keeps under
158
- `~/.cybertrain/spinel/2026.09.12/` is used; otherwise `cybertrain` builds
159
- that copy from the release tag (`git clone`, `make deps`, `make -j`, `make
160
- install PREFIX=...`), once, in a few minutes. The build log is
161
- `~/.cybertrain/log/spinel-2026.09.12-build.log` (its last lines are printed
194
+ `~/.cybertrain/spinel/2026.10.26.7019/` is used; otherwise `cybertrain`
195
+ builds that copy from the release tag of `Cybertrain::SPINEL_REPOSITORY`
196
+ (`git clone`, `make deps`, `make -j`, `make install PREFIX=...`), once, in a
197
+ few minutes. The build log is
198
+ `~/.cybertrain/log/spinel-2026.10.26.7019-build.log` (its last lines are printed
162
199
  if a step fails); if a tool the build needs is missing, `cybertrain` says
163
200
  which and prints the install command for your platform before it starts. A
164
201
  `spinel` of another release on `PATH` is never used or touched:
@@ -183,7 +220,7 @@ if they are not that release.
183
220
 
184
221
  `cybertrain new` writes the application with its `spin.toml` pointing at
185
222
  the release matching the CLI (`cybertrain = { git =
186
- "https://github.com/saeki-mototsune/cybertrain", ref = "v0.2.1" }`), makes
223
+ "https://github.com/saeki-mototsune/cybertrain", ref = "v0.3.0" }`), makes
187
224
  sure Spinel is installed, then runs `spin lock` (spin fetches the framework
188
225
  into its cache, `~/.cache/spin/packages/`, and pins the commit in
189
226
  `spin.lock`) and `spin run gen`, much as `rails new` runs `bundle install`,
@@ -235,6 +272,30 @@ uses it, since `bin/blog.rb` cannot compile until the first migration has
235
272
  generated `gen/models/`). `gen/` already holds `views.rb`, the committed,
236
273
  empty table of embedded views (see "How it works").
237
274
 
275
+ Operational commands of your own (create a user, prune old rows, take a
276
+ backup) go on the same binary: register them in `bin/blog.rb` before
277
+ `Main.run`.
278
+
279
+ ```ruby
280
+ Cybertrain::Main.command("prune", "delete comments older than DAYS") do |argv|
281
+ days = argv.empty? ? 30 : argv[0].to_i
282
+ Cybertrain::DB.with do |c|
283
+ c.execute("DELETE FROM comments WHERE created_at < datetime('now', ?)", ["-#{days} days"])
284
+ end
285
+ 0 # the exit status
286
+ end
287
+ ```
288
+
289
+ `./blog prune 90` then runs the block with the arguments after the name
290
+ (`["90"]`), with `Cybertrain::DB` already connected to the configured
291
+ database (a pool of one connection), and exits with the block's value; a
292
+ `Cybertrain::DB::Error` is printed as `prune failed: ...` and exits 1, and
293
+ `./blog help` lists the command. The block takes `argv` explicitly, as a
294
+ `before_action` block takes the controller (Spinel has no `instance_eval`).
295
+ A built-in word (`server`, `migrate`, `db`, `help`), a number (the port
296
+ argument) or a name already registered raises `ArgumentError`. The server,
297
+ views and session secret are not set up for a command.
298
+
238
299
  Apps created before `cybertrain build` existed have `bin/server.rb` and
239
300
  no `bin/<name>.rb`: copy `bin/<name>.rb` from a fresh `cybertrain new` (with
240
301
  the same NAME), delete `bin/server.rb`, add `/dist/` to `.gitignore` (the old
@@ -422,6 +483,309 @@ child routes to the nested collection (`article_comments_path`), a persisted
422
483
  one to the nested member (`article_comment_path`) — generated into
423
484
  `gen/routes.rb` from the nested `resources` block above.
424
485
 
486
+ ### Add authentication
487
+
488
+ ```sh
489
+ cybertrain generate authentication
490
+ cybertrain db migrate
491
+ cybertrain server # then open http://127.0.0.1:3000/sign_up
492
+ ```
493
+
494
+ Like Rails 8's `bin/rails generate authentication`, this writes code the
495
+ app owns rather than installing an engine: migrations for `users`
496
+ (`email_address`, `password_digest`, a unique index) and `sessions` (one
497
+ row per signed-in browser, with `ip_address` and `user_agent`), the `User`
498
+ and `Session` models, `SessionsController` (`/session/new`, `POST
499
+ /session`, `DELETE /session`), `RegistrationsController` (`/sign_up`,
500
+ because there is no console to create the first user in; delete it for an
501
+ invitation-only app), their views, a `sessions/_nav` partial rendered at
502
+ the top of the layout, and `test/authentication.rb` with its
503
+ `test/support/app_test.rb`. It also inserts a block after the class line of
504
+ `ApplicationController`:
505
+
506
+ ```ruby
507
+ class ApplicationController < Cybertrain::Controller
508
+ before_action :resume_session # sets @current_user and @current_session
509
+ before_action :require_authentication # 303 to /session/new unless signed in
510
+
511
+ def self.allow_unauthenticated_access(only: [], except: [])
512
+ skip_before_action(:require_authentication, only: only, except: except)
513
+ end
514
+ # authenticated?, start_new_session_for(user), terminate_session,
515
+ # after_authentication_url ...
516
+ ```
517
+
518
+ so every action needs a signed-in user unless its controller says
519
+ `allow_unauthenticated_access only: [:new, :create]`, as the two generated
520
+ controllers do. Templates read `@current_user` (`<% if @current_user %>`).
521
+ Passwords are hashed by `Cybertrain::Password` (PBKDF2-HMAC-SHA256, 100,000
522
+ iterations by default, about 0.2 s compiled; raise it with
523
+ `Cybertrain::Password.iterations = 600_000` in `config/app.rb`, and
524
+ `User#authenticate` rehashes each user as they sign in). The `User` model
525
+ spells out what `has_secure_password` would define (`password=`,
526
+ `authenticate`, `User.authenticate_by(email_address:, password:)`), since
527
+ Spinel cannot define methods at run time; edit it like any other model.
528
+ Password reset is not generated (no mailers). Running the generator again
529
+ writes nothing. The design, and why not a package: [docs/design.md](docs/design.md) D15.
530
+
531
+ Sign-in is protected from password guessing the way Rails 8's generator
532
+ does it: `SessionsController` declares `rate_limit to: 10, within: 3 * 60,
533
+ only: [:create], redirect_to: "/session/new", alert: "Try again later."`,
534
+ so an address that has tried ten times in three minutes goes back to the
535
+ form until the window closes. Behind a reverse proxy on the same machine,
536
+ set `CYBERTRAIN_TRUSTED_PROXY=1`, or every visitor shares the proxy's
537
+ address (see [Rate limiting](#rate-limiting-and-the-client-address)); behind
538
+ kamal-proxy, the generated `config/deploy.yml` already sets
539
+ `CYBERTRAIN_TRUSTED_PROXIES` to the Docker network.
540
+ `AppTest.sign_in` forgets the counts first, since every test client comes
541
+ from 127.0.0.1.
542
+
543
+ #### An invitation-only app: `--invite-only`
544
+
545
+ The sign-up page lets anyone who reaches the server register, which is the
546
+ wrong default for an internal tool: whoever gets there before the admin
547
+ does has an account. `--invite-only` writes no `RegistrationsController`
548
+ and no sign-up link; the first user comes from the command line or from a
549
+ page only the server's own machine can open, with a link the server prints
550
+ when it starts:
551
+
552
+ ```sh
553
+ cybertrain generate authentication --invite-only
554
+ cybertrain db migrate
555
+ cybertrain server # then open the http://127.0.0.1:3000/setup?token=... link it prints, on the same machine
556
+ ```
557
+
558
+ In production, from the directory the binary runs in:
559
+
560
+ ```sh
561
+ ./shop user ann@example.com # reads the password from stdin (one line)
562
+ printf '%s' "$PASSWORD" | ./shop user ann@example.com
563
+ ./shop user ann@example.com 's3cr3t-s3cr3t' # or as an argument (visible in ps)
564
+ ```
565
+
566
+ - `app/commands/user_command.rb` registers the `user` command with
567
+ `Cybertrain::Main.command` (so `./shop help` lists it) and is required
568
+ from `bin/shop.rb`. `EMAIL [PASSWORD]` creates the user, or sets the
569
+ password of the one with that address (which signs that user's browsers
570
+ out, as any password change does); it prints `created ADDRESS` or
571
+ `updated ADDRESS` and exits 0. Without `PASSWORD` it reads one line of
572
+ standard input (prompting when that is a terminal; the typing is echoed)
573
+ and drops the trailing newline, so the password stays out of `ps` and
574
+ the shell history. An empty password, a record that does not validate
575
+ (a password under 8 characters) or bad arguments print an error or the
576
+ usage line and exit 1.
577
+ - `SetupController` (`GET`/`POST /setup`, `app/views/setup/new.html.erb`)
578
+ creates the first user and signs them in. It answers only while no user
579
+ exists, only with the one-time token (below), and only to a connection
580
+ whose socket peer is the loopback address (`request.loopback?`:
581
+ `127.0.0.0/8` or `::1`) and that carries no `X-Forwarded-For` header (a
582
+ proxy on the same host makes every request a loopback one, and it adds
583
+ that header). A header can close the page, never open it. Everyone else,
584
+ and a missing or wrong token, gets a 404.
585
+ - The token: `SetupController::SETUP_TOKEN`, 32 random hex characters
586
+ (`Cybertrain::Crypto.random_token(16)`) made when the program starts and
587
+ kept until it exits, so a restart makes a new one. `GET /setup` takes it
588
+ as `?token=`, the form carries it back in a hidden field, and both
589
+ compare it in constant time (`Cybertrain::Crypto.secure_compare`). It is
590
+ what keeps the page closed when the address check cannot: a proxy on the
591
+ same host that adds no `X-Forwarded-For` (nginx without
592
+ `proxy_set_header X-Forwarded-For ...`) makes every client look like
593
+ loopback, and only someone who can read the server's log has the token.
594
+ The request log's `Started GET "/setup?token=..."` line writes it to the
595
+ same log; it opens nothing once the user exists.
596
+ - For a first run behind a proxy on another host, `SHOP_SETUP_OPEN=1`
597
+ (declared in `config/app.rb` with `c.setting_bool :setup_open`) lifts the
598
+ address check, still only while no user exists and still only with the
599
+ token. Unset it once the user is there.
600
+ - While no user exists, the server logs `[WARN] no user exists yet: run
601
+ ./shop user EMAIL, or open /setup?token=<token> from this machine` before
602
+ it boots (`UserCommand.warn_if_no_user`, called from `bin/shop.rb` when
603
+ `Cybertrain::Main.server?(ARGV)`; it connects for that one question and
604
+ says nothing when the users table is not migrated yet). That line is the
605
+ only place the token is printed.
606
+
607
+ The generated `test/authentication.rb` covers both: `/setup` with and
608
+ without the token (`SetupController.setup_token`) or with a wrong one, from
609
+ another address (`client.peer_address = "203.0.113.7"`), through a proxy,
610
+ once a user exists and with the switch on, and the `user` command run in-process
611
+ through `Cybertrain::Main.run`. All of it is application code, yours to
612
+ change (docs/design.md D29); the framework only provides
613
+ `request.loopback?`, `Main.command`, `Main.server?`,
614
+ `Crypto.random_token` / `Crypto.secure_compare` and the test client's
615
+ `peer_address`.
616
+
617
+ In the blog, open the pages anyone may read again, as Rails' guide does, and
618
+ hide what a visitor cannot use (`resume_session` still runs on those pages,
619
+ so `@current_user` is set):
620
+
621
+ ```ruby
622
+ # app/controllers/articles_controller.rb
623
+ class ArticlesController < ApplicationController
624
+ allow_unauthenticated_access only: [:index, :show]
625
+ before_action :set_article, only: [:show, :edit, :update, :destroy]
626
+ ```
627
+
628
+ ```erb
629
+ <%# app/views/articles/show.html.erb (excerpt) %>
630
+ <% if @current_user %>
631
+ <%= button_to "Destroy this article", article_path(@article), method: :delete %>
632
+ <% end %>
633
+
634
+ <% if @current_user %>
635
+ <h2>Add a comment:</h2>
636
+ <%= render "comments/form" %>
637
+ <% end %>
638
+ ```
639
+
640
+ A test that expects a form, submits one, or reaches an action other than
641
+ `index` and `show` now meets the sign-in page: sign the client in first
642
+ with the helpers in `test/support/app_test.rb`.
643
+
644
+ ```ruby
645
+ require_relative "support/app_test" # written by generate authentication: APP, AppTest
646
+
647
+ test "a signed-in user creates an article" do
648
+ client = Cybertrain::Test::Client.new(APP)
649
+ AppTest.user("ann@example.com", "s3cr3t-s3cr3t")
650
+ AppTest.sign_in(client, "ann@example.com", "s3cr3t-s3cr3t")
651
+ client.get("/articles/new")
652
+ res = client.post("/articles", { "authenticity_token" => AppTest.form_token(client),
653
+ "article[title]" => "Hello Rails",
654
+ "article[body]" => "I am on Rails! This is my first article." })
655
+ assert_redirected_to res, "/articles/#{Article.last.id}"
656
+ end
657
+ ```
658
+
659
+ `examples/blog` itself stops before this step.
660
+
661
+ ## Authentication: what the framework provides
662
+
663
+ The generator above writes the flow; the pieces it is built from are
664
+ framework API, usable without it (for an API token, an admin password, a
665
+ hand-written sign-in):
666
+
667
+ - **`Cybertrain::Password`**: `hash(plain)` makes a salted
668
+ PBKDF2-HMAC-SHA256 digest that records its own parameters
669
+ (`$pbkdf2-sha256$i=100000$salt$key`), `verify?(plain, digest)` checks one
670
+ in constant time and never raises on a malformed digest, and
671
+ `needs_rehash?(digest)` says when one is behind the current
672
+ `Password.iterations`. For a sign-in action:
673
+ - `verify_serialized(plain, digest)` is `verify?` one at a time per
674
+ process (a Mutex). A hash takes about 0.2 s of CPU; a burst of attempts
675
+ side by side would hold every worker, through this method they queue and
676
+ the other workers keep serving pages.
677
+ - `dummy_digest` is a digest of a random password, made once per process
678
+ at the current cost. Verify against it when the address is unknown, so
679
+ the response time does not tell which addresses have accounts:
680
+ `Password.verify_serialized(password, user.nil? ? Password.dummy_digest : user.password_digest)`.
681
+ - `session_key(digest)` is an HMAC of the password digest under
682
+ `secret_key_base`; keep it in the session at sign-in and check it with
683
+ `session_key_matches?(session[:password_key], user.password_digest)` on
684
+ each request. A password change makes a new digest, so every session
685
+ started before it is signed out.
686
+ - `iterations` is `Password::TEST_ITERATIONS` (1,000) when
687
+ `Cybertrain.config.test?`, so test suites do not pay the production cost
688
+ per user; digests record their count, so nothing else changes.
689
+ - **The session** (`session[:key]`, a signed cookie): `clear` empties it
690
+ (CSRF token included) on sign-out, and `config.session_absolute_max_age`
691
+ (two weeks; `0` turns it off) bounds a session's whole life however often
692
+ it is used: the cookie records when it was first written, a cookie older
693
+ than that loads as an empty session, so a copied cookie cannot be
694
+ replayed forever. `session.restart_lifetime!` starts the count again (at
695
+ sign-in).
696
+ - **`Controller#local_path?(text)` and `safe_redirect_path(text, fallback)`**:
697
+ a path that came from the request (a `return_to` parameter, a remembered
698
+ page) is redirected to only when it starts with `/` but not `//` or `/\`,
699
+ holds printable ASCII only and is at most 2,048 bytes:
700
+ `redirect_to safe_redirect_path(params[:return_to], "/"), status: :see_other`.
701
+ - **`skip_before_action(name, only:, except:)`** on a controller class drops
702
+ an inherited `before_action` for some actions; the generated
703
+ `allow_unauthenticated_access` is a one-line wrapper around it.
704
+ - **`cybertrain generate authentication`** writes, as code the app owns: the
705
+ `users` and `sessions` migrations and models, `SessionsController`,
706
+ `RegistrationsController`, their views and nav partial, the
707
+ `ApplicationController` block (`resume_session`, `require_authentication`,
708
+ `start_new_session_for`, `terminate_session`, `after_authentication_url`)
709
+ and `test/authentication.rb`. That code uses everything in this list:
710
+ `User.authenticate_by` verifies with `verify_serialized` and the dummy
711
+ digest, `start_new_session_for` stores the password key and restarts the
712
+ lifetime, `resume_session` ends a Session row whose key no longer matches
713
+ or that is older than `session_absolute_max_age`, and
714
+ `after_authentication_url` is `safe_redirect_path`. To keep the browser
715
+ that changed a password signed in, store the new key after saving:
716
+ `session[:password_key] = Cybertrain::Password.session_key(user.password_digest)`.
717
+ The design: [docs/design.md](docs/design.md) D25.
718
+
719
+ ### Add a file upload
720
+
721
+ ```sh
722
+ cybertrain generate attachment article cover # or: generate scaffold article title:string cover:attachment
723
+ cybertrain db migrate
724
+ ```
725
+
726
+ The same split as authentication: the framework parses
727
+ `multipart/form-data` and stores bytes, and the generator writes the rest
728
+ as code the app owns. It writes a migration adding five columns to
729
+ `articles` (`cover_key`, `cover_filename`, `cover_content_type`,
730
+ `cover_byte_size`, `cover_checksum`; the file itself is not in SQLite) and
731
+ a block in `Article` (`cover_file=`, `cover_attached?`, `cover_url`, a
732
+ validation and save/destroy callbacks). In `ArticlesController` it adds a
733
+ `cover` action behind `before_action :set_article` and one line in
734
+ `create` and `update`:
735
+
736
+ ```ruby
737
+ @article.cover_file = params.nested(:article).upload(:cover)
738
+ ```
739
+
740
+ It also adds `get "/articles/:id/cover"` to `config/routes.rb`, and to the
741
+ form `form_with(model: article, multipart: true)` with `f.file_field :cover`.
742
+ The show page gets an `<img>`, and `test/attachment_article_cover.rb` tests
743
+ the whole flow.
744
+
745
+ What is accepted and how it is served:
746
+
747
+ - **Types.** By default only PNG, JPEG, GIF and WebP up to 2 MB are
748
+ accepted (`COVER_TYPES` / `COVER_MAX_BYTES` in the model; `--any` accepts
749
+ any type). The type is detected from the file's first bytes, never taken
750
+ from the browser.
751
+ - **Storage.** The bytes go to `storage/uploads/<2 hex>/<32 hex key>`
752
+ (`Cybertrain::BlobStore`, mode 0600). Replacing a file deletes the old
753
+ one, and destroying the record deletes its file.
754
+ - **Serving.** The `cover` action serves files with `send_blob`, so the
755
+ controller's `before_action`s (sign-in, scoping) apply to downloads too.
756
+ The four image types are served inline; anything else is a download
757
+ (`application/octet-stream`). Like every response, they carry
758
+ `X-Content-Type-Options: nosniff` and `X-Frame-Options: SAMEORIGIN` (the
759
+ framework-wide defaults, below), and `send_blob` adds
760
+ `Content-Security-Policy: default-src 'none'; sandbox`. Never put uploads
761
+ under `public/`. The download is exactly as public as show: in the blog
762
+ above, where `allow_unauthenticated_access only: [:index, :show]` lets
763
+ visitors read articles, the generator adds `:cover` to that list, so the
764
+ image on a public page loads (only when the download loads the record
765
+ through the same `before_action :set_article` as show, and show checks
766
+ nothing more). The generator checks inherited callbacks too. If a policy
767
+ applies only to show, show contains inline checks, or the policy cannot
768
+ be read safely, generation stops before writing any files. Put authorization
769
+ in an explicit callback shared by show and the download, for example
770
+ `before_action :authorize_article, only: [:show, :cover]`, and load both
771
+ through `before_action :set_article`, then rerun the generator. Keeping a
772
+ download behind sign-in alone does not preserve record or role permissions.
773
+ Where show needs a sign-in, so does the download.
774
+ - **Params and CSRF.** `params.nested(:article).upload(:cover)` is a
775
+ `Cybertrain::UploadedFile` (`original_filename`, `byte_size`,
776
+ `detected_content_type`, `body_bytes`, `sha256_hex`), or nil when no
777
+ file was chosen. `permit` never returns files. The CSRF token and
778
+ `_method` are read from multipart bodies like any form.
779
+ - **Limits.** Three config keys bound uploads: `max_upload_bytes`,
780
+ `max_multipart_parts` and `upload_root` (see "Configuration"). A broken
781
+ or oversize body is a 400.
782
+ - **Not built yet:** several files per field, S3 or other remote storage,
783
+ image variants, and streaming large bodies to disk (the whole body is
784
+ read into memory, up to `max_body_bytes`).
785
+
786
+ Back up `storage/uploads/` together with the SQLite file. The design, and
787
+ why it is not a separate library: [docs/design.md](docs/design.md) D16.
788
+
425
789
  ### Build for deployment
426
790
 
427
791
  ```sh
@@ -435,7 +799,7 @@ template embedded (rendered from memory, error messages still say
435
799
  links the system SQLite) and run `./blog migrate && ./blog`; a built binary
436
800
  is production by default (`CYBERTRAIN_ENV` overrides). Set
437
801
  `CYBERTRAIN_SECRET_KEY_BASE`. Static files can also be served by the reverse
438
- proxy from `dist/public/` — see [docs/deploy.md](docs/deploy.md).
802
+ proxy from `dist/public/` — see [docs/deploy.en.md](docs/deploy.en.md).
439
803
 
440
804
  A running `cybertrain server` skips its rebuilds while `cybertrain build` is
441
805
  in progress (the build holds `tmp/cybertrain-build.lock`, since both write
@@ -479,16 +843,43 @@ helpers and gaps: [docs/template-language.md](docs/template-language.md).
479
843
 
480
844
  **The HTTP server.** `Cybertrain::Server` is plain-Ruby HTTP/1.1:
481
845
  `TCPServer#accept` plus one Spinel green thread per connection (M:N
482
- scheduled, no GVL), typed `Request`/`Response`, no Rack layer. TLS/HTTP/2 are
846
+ scheduled, no GVL), typed `Request`/`Response`, no Rack layer. It calls an `HttpHandler` (`call(request) -> Response`); `ContextHandler` joins it to the middleware stack, so the HTTP layer (`cybertrain/http/`, plus `Params` and `Logger`) does not depend on the rest of the framework; `script/check-http-layer` enforces it in CI. TLS/HTTP/2 are
483
847
  left to a reverse proxy. `SPINEL_WORKERS` defaults to `1` — spikes found that
484
848
  faster and stall-free for this I/O-bound shape at 100 concurrent connections.
485
849
 
486
850
  **SQLite via FFI.** Models talk to SQLite through Spinel's
487
851
  `ffi_func`/`ffi_lib` directly — no C extension, no gem. A pool (`SizedQueue`,
488
- 4 connections by default) hands out connections with WAL, a `busy_timeout`,
489
- and foreign keys on; every query is bound, never interpolated. It's the only
852
+ 4 connections by default) hands out connections with WAL, a busy timeout of
853
+ about five seconds (a busy handler retrying every 100 µs, where SQLite's
854
+ `busy_timeout` sleeps a millisecond or more), up to 64 cached prepared
855
+ statements and foreign keys on; every query is bound, never interpolated. It's the only
490
856
  adapter today (PostgreSQL via `libpq` FFI is noted as possible future work).
491
857
 
858
+ **Transactions.** `Cybertrain::DB.transaction { |c| ... }` (or
859
+ `Article.transaction`, the same thing under the Rails name) checks a
860
+ connection out, runs `BEGIN`, yields the connection, `COMMIT`s when the block
861
+ returns and rolls back and re-raises when it raises. `immediate: true` runs
862
+ `BEGIN IMMEDIATE`, which takes SQLite's write lock at the start: use it for a
863
+ transaction that reads and then writes, so a concurrent writer makes it wait
864
+ at `BEGIN` rather than fail with "database is locked" at its first write.
865
+ While the block runs, everything on that thread that goes through the pool
866
+ (`save`, `update`, `destroy`, Relation queries, `DB.with`) uses the
867
+ transaction's connection, so it is part of the transaction:
868
+
869
+ ```ruby
870
+ Article.transaction(immediate: true) do |c|
871
+ article.save!
872
+ c.execute("UPDATE authors SET articles_count = articles_count + 1 WHERE id = ?", [article.author_id])
873
+ end
874
+ ```
875
+
876
+ A `DB.transaction` nested in the block joins the outer one (only the
877
+ outermost commits; a raise anywhere rolls everything back; no savepoints).
878
+ Code that must keep its `rescue` in a plain method uses the block-free form,
879
+ `c.begin!(immediate: true)` ... `c.commit!`, with `c.rollback!` in the rescue.
880
+ It returns nil rather than the block's value, and a `DB.transaction` called
881
+ inside a `DB.with` block (which is not a transaction) gets a second connection.
882
+
492
883
  **The development loop.** `cybertrain server` (`spin run gen`, then `spin run NAME`) in
493
884
  `development` (the default `CYBERTRAIN_ENV`, except in a binary from
494
885
  `cybertrain build`) also polls `app/**/*.rb`, `config/**/*.rb`, `db/schema.rb`
@@ -500,11 +891,378 @@ main thread `execv`s the new binary on the same port and PID, invisible to a
500
891
  client mid-session. A failed build keeps serving the old binary and banners the
501
892
  compiler output on every HTML response. None of this loads in production.
502
893
 
894
+ ## Migrations
895
+
896
+ A migration in `db/migrate/<version>_<name>.rb` is a `Cybertrain::Migration::Base`
897
+ subclass. Its `change` records schema operations (`create_table`,
898
+ `add_column`, `add_index`, `rename_column`, `add_reference`, `remove_column`,
899
+ `remove_index`, `drop_table`); `cybertrain db migrate` applies each pending
900
+ version in its own transaction and rewrites `db/schema.rb` from the database,
901
+ and `cybertrain db rollback [N]` runs the inverse of `change`. When `change`
902
+ holds an operation with no automatic inverse, write `down` (and `up`).
903
+
904
+ A schema change that needs a data step to be safe uses `execute(sql, binds =
905
+ [])`. It runs at its place in the migration, on the migrator's own
906
+ connection and in the same transaction as the version's schema operations,
907
+ so the column added before it exists when it runs, and a failure undoes the
908
+ whole version:
909
+
910
+ ```ruby
911
+ class AddRetentionDaysToBackups < Cybertrain::Migration::Base
912
+ def change
913
+ add_column "backups", "retention_days", :integer, null: false, default: "90"
914
+ execute "UPDATE backups SET retention_days = ?", [0] # existing rows keep "forever"
915
+ end
916
+
917
+ def down
918
+ remove_column "backups", "retention_days"
919
+ end
920
+ end
921
+ ```
922
+
923
+ Without binds the SQL may be several `;`-separated statements; with binds
924
+ it is one statement with `?` placeholders. `execute` has no inverse: rolling
925
+ back a `change` that executes SQL, with no `down` of its own, stops with
926
+ `version N is irreversible: it executes SQL in change; write down` before
927
+ anything is reverted. A data statement never appears in `db/schema.rb`,
928
+ which is dumped from the database. Recording a migration runs nothing, so
929
+ the statement runs exactly once per `migrate` of its version.
930
+
931
+ ## Querying beyond `where`
932
+
933
+ The generated relations cover `where` / `where_sql` / `order` / `limit` /
934
+ `offset` / `count` / `exists?` / `delete_all`. For the counters, statistics
935
+ and search boxes most applications also need, there are typed additions
936
+ (issue #74, `docs/design.md` D22) rather than a query builder; each takes
937
+ SQL column names, checked and quoted (anything else is an `ArgumentError`),
938
+ and binds every value:
939
+
940
+ ```ruby
941
+ # INSERT ... ON CONFLICT (project_id, key) DO UPDATE SET ...; returns rows written
942
+ Stat.upsert({ project_id: project.id, key: path, calls: 1, max_ms: ms, seen_at: Time.now },
943
+ conflict: [:project_id, :key],
944
+ update: { calls: "calls + excluded.calls", max_ms: "MAX(max_ms, excluded.max_ms)",
945
+ seen_at: "excluded.seen_at" })
946
+ Article.where(id: article.id).increment(:views) # UPDATE ... SET views = views + 1
947
+ Comment.where(article_id: article.id).pluck_int(:id) # => [3, 7] (also pluck_str, pluck_float)
948
+ Comment.all.count_by(:article_id) # => {"1" => 2, "4" => 1}
949
+ Comment.all.count_by_sql("strftime('%Y-%m-%dT%H', created_at)") # hour buckets
950
+ Stat.where(project_id: 1).sum_int(:calls) # also sum_float
951
+ Article.where_like(:title, params[:q].to_s) # `title` LIKE '%...%' ESCAPE '\'
952
+ ```
953
+
954
+ - `upsert`'s `update` values and `count_by_sql`'s expression are SQL text, like
955
+ `order_sql`: never build them from request data. An empty `update` is
956
+ `DO NOTHING`. `upsert` and `increment` load no model and run no
957
+ validation or callback, and fill in no timestamps.
958
+ - `count_by` keys are Strings (through `Cast.str`), ascending; NULL and `""`
959
+ share the key `""`. With a `limit` / `offset`, `count_by`, `sum_*` and
960
+ `increment` work on that window, as `count` and `delete_all` do.
961
+ - `where_like` is a substring match (case-insensitive for ASCII, as SQLite's
962
+ LIKE is). A prefix search writes its own pattern:
963
+ `Article.all.where_sql("title LIKE ? ESCAPE '\\'", [Cybertrain::Cast.like_escape(q) + "%"])`.
964
+ - For hand-written SQL, `Cybertrain::Cast.binds(a, b, c)` builds the bind
965
+ Array `execute` takes (a Time as UTC ISO 8601 text, booleans as 1/0):
966
+ `Cybertrain::DB.with { |c| c.execute(sql, Cybertrain::Cast.binds(id, since)) }`.
967
+ - `Cybertrain::Cast.hash(v)` / `Cast.array(v)` narrow a parsed JSON value to
968
+ a Hash / Array, or a fresh empty one when it is not one (a Time is checked
969
+ first, spikes/NOTES.md rule 8). They return a typed shallow copy, not `v`
970
+ itself; `Cast.empty_hash` / `Cast.empty_array` are the seeded empty
971
+ containers.
972
+
973
+ ## Pagination
974
+
975
+ A list page reads the page number from `?page=N`, windows the relation, and
976
+ renders the links:
977
+
978
+ ```ruby
979
+ # app/controllers/articles_controller.rb
980
+ def index
981
+ @articles = Article.all.order("created_at DESC")
982
+ @page = @articles.page(params.page_number, per: 20)
983
+ end
984
+ ```
985
+
986
+ ```erb
987
+ <%# app/views/articles/index.html.erb %>
988
+ <% @articles.each do |article| %>
989
+ <p><%= link_to article.title, article %></p>
990
+ <% end %>
991
+ <%= paginate @page, articles_path %>
992
+ ```
993
+
994
+ `params.page_number` is the 1-based `page` parameter, 1 when it is absent or
995
+ not a number, clamped to `1..max_page` (`config.max_page`, 10000 by default,
996
+ `CYBERTRAIN_MAX_PAGE`); it never hands unbounded text to `String#to_i`, which
997
+ raises `RangeError` past 2**63-1 under Spinel. `params.int(:per, default: 20,
998
+ min: 1, max: 100)` is the same bounded read for any key (an optional `-` and
999
+ at most 18 digits, else the default; clamped to `min..max`).
1000
+ `relation.page(n, per: 50)` sets `LIMIT per OFFSET (n - 1) * per` on the
1001
+ relation (replacing an earlier `limit`/`offset`, so call it last), clamps `n`
1002
+ the same way, so the offset never passes `(max_page - 1) * per`, and returns a
1003
+ `Cybertrain::Page`: `number`, `per`, `row_offset`, `next?` (asked with one
1004
+ `SELECT 1 ... LIMIT per + 1` over the window, no `COUNT(*)`), `prev?`,
1005
+ `next_number`, `prev_number`. The rows stay on the relation, so
1006
+ `@articles.each` yields typed `Article`s; `per` outside 1..1000 is an
1007
+ `ArgumentError`. `paginate @page, path` renders `<nav class="pager">` with a
1008
+ "Newer" link (`rel="prev"`), "Page N" and an "Older" link (`rel="next"`),
1009
+ each left out when there is no such page; the links keep the current query
1010
+ string with `page` replaced (page 1 drops it), re-encoded and escaped, so a
1011
+ search or filter survives paging, and a path with its own query string
1012
+ (`"/search?q=x"`) uses that one instead. Offset pagination only: keyset
1013
+ pagination (`after: id`) for very large tables is not built yet. The scaffold's
1014
+ index view lists every record; add the three lines above to page it.
1015
+
1016
+ ## Rate limiting and the client address
1017
+
1018
+ `rate_limit` in a controller caps how often one client may call some
1019
+ actions, counted before the action runs:
1020
+
1021
+ ```ruby
1022
+ class SessionsController < ApplicationController
1023
+ # a 303 back to the form, with flash[:alert], once exhausted
1024
+ rate_limit to: 10, within: 3 * 60, only: [:create],
1025
+ redirect_to: "/session/new", alert: "Try again later."
1026
+ end
1027
+
1028
+ class ApiKeysController < ApplicationController
1029
+ rate_limit to: 100, within: 60, only: [:create, :update] # 429 Too Many Requests + Retry-After
1030
+ end
1031
+ ```
1032
+
1033
+ Each declaration counts at most `to` requests per `within` seconds for each
1034
+ client address and controller (a fixed window that opens with the first
1035
+ request), then answers 429 with a `Retry-After` header, or redirects when
1036
+ given `redirect_to:`. `within:` is in seconds, `only:`/`except:` take
1037
+ Arrays, and there is no `with:` block. The counts live in the process, in a
1038
+ table of at most 10,000 addresses (when it is full, expired windows and
1039
+ then the oldest ones make room), so they are lost on restart and each
1040
+ process counts on its own. Every request to a limited action takes one lock:
1041
+ name the actions with `only:`. In a test, `Cybertrain::RateLimit.reset_all!`
1042
+ forgets every count and `Cybertrain::RateLimit.current_time = n` fixes the
1043
+ clock (`-1` restores it).
1044
+
1045
+ The address counted is `request.client_ip` (`by: :remote_addr` counts the
1046
+ connection's peer instead). It is `request.remote_addr` unless
1047
+ `config.trusted_proxy` is on (`CYBERTRAIN_TRUSTED_PROXY=1`) **and** the
1048
+ connection comes from loopback, that is, from a reverse proxy on the same
1049
+ host (nginx, Caddy, kamal-proxy); then it is the last address in
1050
+ `X-Forwarded-For`, the one the proxy appended. Earlier entries are whatever
1051
+ the client sent and are never used, and a last entry that is empty, longer
1052
+ than 45 bytes or not printable ASCII gives `remote_addr`. Without the
1053
+ setting, every visitor behind such a proxy shares its address, and ten bad
1054
+ passwords lock everyone out; with it, a client connecting directly from
1055
+ elsewhere still cannot choose its own address. `request.loopback?` tells
1056
+ whether the connection comes from this machine (127.0.0.0/8, `::1`,
1057
+ `::ffff:127.x.x.x`), for pages only an operator on the server should
1058
+ reach; behind a proxy on the same host every request is loopback, so check
1059
+ `client_ip` too there. Design: [docs/design.md](docs/design.md) D26.
1060
+
1061
+ A proxy that reaches the app over a network, not loopback, is listed by
1062
+ address instead: `config.trusted_proxies`, from `CYBERTRAIN_TRUSTED_PROXIES`,
1063
+ takes comma-separated IPv4 or IPv6 addresses or CIDRs
1064
+ (`CYBERTRAIN_TRUSTED_PROXIES=172.16.0.0/12,10.0.0.0/8`, or a single
1065
+ `172.18.0.2`). A connection from inside one of them gets the same treatment
1066
+ as a loopback one above: the last `X-Forwarded-For` address, validated the
1067
+ same way. An IPv4 entry also matches the IPv4-mapped form of its addresses
1068
+ (`::ffff:172.18.0.2`), and a peer's `%zone` is ignored. An entry that is not
1069
+ an address or CIDR, or has a prefix past `/32` (IPv4) or `/128` (IPv6),
1070
+ stops the server at boot. `CYBERTRAIN_TRUSTED_PROXY=1` is the shorthand for
1071
+ `127.0.0.0/8,::1`, and the two add up. This is the Kamal case: kamal-proxy
1072
+ runs in its own Docker container and connects from its address on the Docker
1073
+ network, so the `config/deploy.yml` that `cybertrain new` writes sets
1074
+ `CYBERTRAIN_TRUSTED_PROXIES: 172.16.0.0/12` (Docker's default range; change
1075
+ it if your networks use another, `docker network inspect kamal`). Leave
1076
+ Kamal's `proxy.forward_headers` unset: with `ssl: true`, kamal-proxy sets
1077
+ `X-Forwarded-For` itself rather than passing on the client's. List only
1078
+ networks that clients cannot connect from: anyone inside a listed network
1079
+ can choose the address a rate limit counts.
1080
+
1081
+ ## Parameterised headers
1082
+
1083
+ API clients often send `Scheme k1=v1, k2="v2"` headers (`Authorization:
1084
+ Digest ...`, a vendor `X-Example-Auth`) or `k1=v1;k2=v2, k3` lists
1085
+ (`Forwarded`, `Prefer`). `request.header_params(name)` parses one into a
1086
+ `Hash` of lowercase names to values, so an API controller does not split it
1087
+ by hand:
1088
+
1089
+ ```ruby
1090
+ # Authorization: Digest username="Mufasa", realm="api@example.com", nc=00000001, qop=auth
1091
+ auth = request.header_params("authorization")
1092
+ auth["username"] # => "Mufasa"
1093
+ request.header_scheme("authorization") # => "Digest"
1094
+
1095
+ # Forwarded: for=192.0.2.60;proto=https;by=203.0.113.43
1096
+ request.header_params("forwarded")["proto"] # => "https"
1097
+
1098
+ # Prefer: respond-async, wait=5
1099
+ request.header_params("prefer").key?("respond-async") # => true
1100
+ ```
1101
+
1102
+ `,` and `;` both separate parameters and the first occurrence of a name
1103
+ wins; a quoted value is unquoted (`\"` and `\\` undone), an unquoted one
1104
+ has the spaces around it trimmed, and a name without `=` maps to `""`. A
1105
+ leading scheme word (`Digest`, `Bearer`) is not a parameter:
1106
+ `request.header_scheme(name)` returns it (`""` when there is none). A header
1107
+ that is missing or longer than 8 KiB gives `{}`. Values are the bytes as sent,
1108
+ never percent-decoded and possibly not valid UTF-8. A `Bearer <token>` or
1109
+ `Basic <base64>` credential is not a parameter: check the scheme, then read
1110
+ the credential from `request.header("authorization")`. Design:
1111
+ [docs/design.md](docs/design.md) D31.
1112
+
1113
+ ## Background work
1114
+
1115
+ An application that has to do something outside a request (send a
1116
+ notification after the response went out, sweep old rows every hour, retry a
1117
+ call that failed) registers it with the framework, which runs it on one
1118
+ background thread of the served process:
1119
+
1120
+ ```ruby
1121
+ # config/app.rb (or a file bin/<app>.rb requires before Cybertrain::Main.run)
1122
+ Cybertrain::Scheduler.every(3600, "prune sessions") do
1123
+ Cybertrain::DB.with { |c| c.execute("DELETE FROM sessions WHERE updated_at < datetime('now', '-30 days')") }
1124
+ end
1125
+
1126
+ Cybertrain::Job.handle("welcome_mail") do |payload|
1127
+ data = Cybertrain::Cast.hash(JSON.parse(payload))
1128
+ WelcomeMail.deliver(data["email"].to_s) # raise to have it retried
1129
+ end
1130
+ ```
1131
+
1132
+ ```ruby
1133
+ # in an action, after saving the user
1134
+ Cybertrain::Job.enqueue("welcome_mail", JSON.generate({ "email" => @user.email }))
1135
+ Cybertrain::Job.enqueue("reminder", @order.id.to_s, run_at: Time.now + 600)
1136
+ ```
1137
+
1138
+ - **When it runs.** The thread starts when the binary serves (`./<app>`,
1139
+ `./<app> server`), after boot has connected the database, bound the port
1140
+ and set `SPINEL_WORKERS`; `migrate`, `db ...` and `Main.command` commands
1141
+ never start it, and neither does `Cybertrain::Test::Client`. A task runs
1142
+ first `seconds` after the start, then `seconds` after the start of each
1143
+ run. Each run is wrapped in a rescue that logs `[ERROR] Scheduled task
1144
+ "name" failed: ...` and the loop goes on. The block takes no arguments
1145
+ (Spinel has no `instance_eval`). `Cybertrain::Scheduler.last_tick` is the
1146
+ epoch second of the last tick, for a health check of your own.
1147
+ - **Jobs.** `Job.enqueue` adds a row to the `cybertrain_jobs` table, which
1148
+ the framework creates itself (`CREATE TABLE IF NOT EXISTS`, on first use)
1149
+ and leaves out of `db/schema.rb`. The payload is a String: JSON-encode
1150
+ structured data. Inside a `DB.transaction` the row commits or rolls back
1151
+ with the rest. The worker, a scheduler task polling every
1152
+ `job_poll_interval` seconds (5), claims the oldest due job in a short
1153
+ `BEGIN IMMEDIATE` transaction, runs its handler outside it, and marks it
1154
+ `done`. A handler that raises is retried 1, 5, 15 and then 60 minutes
1155
+ later, with `Class: message` (up to 1000 characters) in `last_error`, until
1156
+ `job_max_attempts` (5) attempts have failed: the row is then `failed` and
1157
+ stays for you to look at. A job whose kind has no handler is `failed` at
1158
+ once; `Job.enqueue` refuses an unregistered kind. Rows a crashed process
1159
+ left `running` are requeued when the worker starts. `Job.state_count("failed")`
1160
+ and `Job.prune_done(before_epoch)` help with a health page and housekeeping.
1161
+ - **One worker per database.** Every served process runs the thread unless
1162
+ `CYBERTRAIN_WORKER=0` (`config.worker = false`): when several processes
1163
+ share one SQLite file, turn it off in all but one, or each scheduled task
1164
+ runs once per process.
1165
+ - **Keep tasks short.** Tasks and handlers run one at a time on that one
1166
+ thread, and a SQLite call holds its OS worker until it returns: with
1167
+ `SPINEL_WORKERS=1` (the default) a long query stalls every request for as
1168
+ long. Batch big work (delete a thousand rows per run, enqueue one job per
1169
+ item) rather than doing it in one go.
1170
+ - **Testing.** `Scheduler.run_due(now)` runs one tick and `Job.work_one(now)`
1171
+ one job, with an epoch time you choose, so a test never sleeps.
1172
+
1173
+ There is no cron syntax, no priority or uniqueness for jobs and no mailer:
1174
+ this covers "send later" and "run every N seconds" (`docs/design.md` D28).
1175
+
1176
+ ## Operations: backups, vacuum, health check
1177
+
1178
+ The data of a deployed application is one SQLite file, so the binary carries
1179
+ the three operational tools such an application needs.
1180
+
1181
+ **Backups.** `./<app> db backup PATH` writes a consistent copy of the
1182
+ configured database (`CYBERTRAIN_DATABASE`) while the server keeps running:
1183
+ SQLite's `VACUUM INTO` (3.27 or newer; the binary checks) reads one snapshot,
1184
+ and under WAL the server's writes go on meanwhile. The copy is written to a
1185
+ temporary file next to `PATH`, created exclusively with mode 0600 (the file
1186
+ holds password digests and session tokens), fsynced, and only then renamed
1187
+ to `PATH`, so a failed or interrupted copy never leaves a partial file under
1188
+ the real name (the temporary one is deleted). An existing `PATH` is refused
1189
+ unless you pass `--force`; the database itself never is a target, nor is a
1190
+ `:memory:` database a source.
1191
+
1192
+ ```sh
1193
+ ./blog db backup /srv/blog/backups/before-upgrade.sqlite3
1194
+ ./blog db backup /srv/blog/backups/ --keep 14 # blog-20261025T031500Z.sqlite3, newest 14 kept
1195
+ ```
1196
+
1197
+ A target ending in `/` (or an existing directory) gets a dated name,
1198
+ `<app>-<YYYYmmddTHHMMSSZ>.sqlite3` in UTC, in a directory created with mode
1199
+ 0700 if it does not exist; `--keep N` then deletes the oldest files of
1200
+ exactly that shape beyond N and touches nothing else there. Restoring is
1201
+ copying a backup over the database file with the server stopped (and
1202
+ deleting that file's `-wal` and `-shm` beside it).
1203
+
1204
+ **Vacuum.** Deleting rows leaves free pages in the file. `./<app> db vacuum`
1205
+ runs `VACUUM` and then `PRAGMA wal_checkpoint(TRUNCATE)`, so the rebuilt
1206
+ pages reach the database file, the file is cut to its new size and the WAL
1207
+ is emptied; it prints the size on disk (database plus WAL) before and after.
1208
+ `VACUUM` rewrites the whole file and holds the write lock while it does, so
1209
+ run it when the application is quiet. If a reader keeps the checkpoint from
1210
+ finishing it says so, and the WAL shrinks at a later checkpoint.
1211
+
1212
+ **Run backups outside the server.** A SQLite call holds its OS worker until
1213
+ it returns, so a large `VACUUM INTO` on a `Scheduler` task would stall
1214
+ requests (with `SPINEL_WORKERS=1`, all of them) for as long as the copy
1215
+ takes. Run the command from cron or a systemd timer instead, as a separate
1216
+ process:
1217
+
1218
+ ```sh
1219
+ # crontab of the app's user: every night at 03:15, keep two weeks
1220
+ 15 3 * * * cd /srv/blog/current && ./blog db backup /srv/blog/backups/ --keep 14
1221
+ ```
1222
+
1223
+ An application that wants to take backups itself can run its own binary as
1224
+ a child process from a scheduled task, rather than run the SQL on its own
1225
+ thread: `system` waits for the child on the scheduler's thread only, and
1226
+ requests go on being served meanwhile (Spinel has no `Process.wait`, and a
1227
+ `Process.spawn` without it leaves a zombie, so use `system` with an argument
1228
+ list, never a string with request data in it). A copy that takes longer than
1229
+ `health_worker_stale` makes the health check answer `worker stale` until it
1230
+ is done.
1231
+
1232
+ ```ruby
1233
+ # config/app.rb, in a served binary started from its own directory (dist/)
1234
+ Cybertrain::Scheduler.every(86_400, "nightly backup") do
1235
+ system("./blog", "db", "backup", "storage/backups/", "--keep", "14")
1236
+ end
1237
+ ```
1238
+
1239
+ **Health check.** While the health check is enabled (it is by default;
1240
+ an empty `config.health_path` turns it off), the served application
1241
+ answers `GET /up` (and `HEAD`) before the static files, the session, CSRF and the router: no controller
1242
+ is involved and no login `before_action` can redirect it, and it sets no
1243
+ cookie. It runs `SELECT count(*) FROM sqlite_master`, which has to read the
1244
+ schema page from the file (a bare `SELECT 1` never touches it), and answers
1245
+ in `text/plain` with `Cache-Control: no-store`:
1246
+
1247
+ | Status | Body | When |
1248
+ | --- | --- | --- |
1249
+ | 200 | `ok` | the query succeeded |
1250
+ | 503 | `database unavailable` | it raised `Cybertrain::DB::Error` (the file is missing or unreadable, the pool is closed) |
1251
+ | 503 | `worker stale` | the background `Scheduler` runs and its last tick is older than `health_worker_stale` seconds (300): a task is stuck, or runs far too long |
1252
+
1253
+ A poll is logged like any request (two lines). The path is
1254
+ `config.health_path` (`CYBERTRAIN_HEALTH_PATH`, default `/up`; an empty
1255
+ value turns the check off, and the path then goes on to `public/` and the
1256
+ router as before). A load balancer, kamal-proxy or a deploy script can poll
1257
+ it; `public/up.html`, which `cybertrain new` writes and its Kamal
1258
+ configuration still polls, only shows that the process answers HTTP.
1259
+
503
1260
  ## Differences from Rails
504
1261
 
505
1262
  | Rails | cybertrain |
506
1263
  | --- | --- |
507
1264
  | `rails console` | No console — Spinel has no `eval` |
1265
+ | `rails runner`, `lib/tasks/*.rake` | `Cybertrain::Main.command("name", "description") { \|argv\| ...; 0 }` in `bin/<app>.rb`: `./<app> name ARGS` runs it with the database connected |
508
1266
  | Edit code, the running app picks it up | Ruby needs a rebuild; `cybertrain server` in development does this for you (rebuild, stop, `execv`) |
509
1267
  | Edit a view, no reload needed | Same — views are parsed from disk per request in development |
510
1268
  | `def new` | `def new_action` (`new` would shadow `Klass.new(ctx)`) |
@@ -513,18 +1271,62 @@ compiler output on every HTML response. None of this loads in production.
513
1271
  | ERB compiles to a method; any object, any method | Interpreted against a fixed grammar/dispatch table — [docs/template-language.md](docs/template-language.md) |
514
1272
  | Session holds any marshalled object | Session values are Strings only, HMAC-signed cookie |
515
1273
  | Many database adapters | SQLite only, via FFI |
516
- | `has_many :through`, `includes`, `pluck`, `dependent:`, enums, STI, polymorphic associations | Not implemented; associations come from schema foreign keys only |
1274
+ | `Article.transaction { ... }` returns the block's value; `raise ActiveRecord::Rollback`; `requires_new: true` savepoints | `Article.transaction { \|c\| ... }` (= `Cybertrain::DB.transaction`) yields the connection and returns nil; leave it by raising (it re-raises after the rollback); nested calls join the outer transaction, no savepoints. `immediate: true` for SQLite's `BEGIN IMMEDIATE`. `c.transaction { }` cannot be called on the yielded `c` under Spinel: nest `DB.transaction`, or use `c.begin!` / `c.commit!` / `c.rollback!` |
1275
+ | `has_many :through`, `includes`, `dependent:`, enums, STI, polymorphic associations | Not implemented; associations come from schema foreign keys only |
1276
+ | `pluck(:a)`, `sum(:a)`, `group(:a).count`, `upsert`, `update_counters`, `where("title LIKE ?", "%#{q}%")` | Typed one-column flavours, because a Spinel method has one return type: `pluck_str` / `pluck_int` / `pluck_float` (one column only), `sum_int` / `sum_float`, `count_by(:a)` (a `Hash{String => Integer}`) and `count_by_sql(expr)` for expressions; `Model.upsert(attrs, conflict:, update:)` takes the `DO UPDATE SET` expressions as SQL text and runs no validations, callbacks or timestamps; `relation.increment(:col, by: n)`; `where_like(:col, text)` escapes `\`, `%` and `_` for you. See [Querying beyond `where`](#querying-beyond-where) |
517
1277
  | Any column name; an association can shadow `errors` | A column name must be an ASCII identifier (`[A-Za-z_][A-Za-z0-9_]*`): a schema with a non-ASCII column name (`名前`, `prénom`) generated under CRuby before PR #10 and `spin run gen` refuses it now, so rename the column when upgrading (a name like `first-name` was never a method name). A column named like a method the generated class or the framework calls on a record (`errors`, `save`, `attributes`, `hash`, `to_s`, `to_ary`, `raise`, ...) or like a Ruby keyword (`end`, `class`, `begin`) generates under `<column>_column` (`hash_column`, reader, writer and ivar) with a note in the generated file; its SQL name keeps working in `read_attribute`, params and `attributes["hash"]`, and a template reaches it under either name (`post.hash`, `post.hash_column`), except `errors` and `to_param`, which a template resolves as the model methods, so use `post.errors_column` there; the query API (`where`, `order`, `find_by`) always takes the SQL column name (`Post.where(hash: 1)`), the Ruby name being the reader/writer and the template name only. A column named like another Object method (`display`, `tap`, `methods`) generates under its own name with a note. `cybertrain generate scaffold` refuses keywords and reserved names (it would have to call a renamed reader) and accepts a shadowing name with a note, like the generator. An association whose plain name a column, another association or a `Model` method already owns is emitted under a fallback name with a comment in the generated file: `<table>_as_<column stem>` for a `has_many` (`comments_as_article`), `<stem>_as_<column>` for a `belongs_to` (`author_as_author_id`) |
518
1278
  | `namespace`, format/`respond_to`, `constraints`, `mount` | Not implemented — flat names, `render json:` only |
519
1279
  | `rescue StandardError` catches a bad `JSON.parse` | Under Spinel `JSON::ParserError` is not a `StandardError`: app code must `rescue JSON::ParserError, StandardError`. The server catches it as a last resort and answers 500 |
1280
+ | `skip_forgery_protection` in an API controller | CSRF is a middleware that runs before routing, so an API is exempted by path: `c.csrf_exempt_prefixes = ["/api/"]` in `config/app.rb`. The controller must then authenticate its callers itself (an API key in a header), and no session-authenticated route may match under the prefix |
1281
+ | Rack leaves a gzip request body as it is | A body sent with `Content-Encoding: gzip` (an SDK or API client past a few kilobytes) is `request.decoded_body`, inflated once through the system zlib under a ceiling on the output (10 MiB by default; an endpoint that expects more passes its own, `request.decoded_body(limit)`): a corrupt body or an encoding other than gzip/identity is `Cybertrain::BodyUndecodable`, one that would inflate past the limit `Cybertrain::BodyTooLarge`, both `QueryInvalid`s answered 400; `request.body` stays the bytes as sent |
520
1282
  | `order("lower(title)")`, `order("posts.title")`, `order(params[:sort])` | `order` takes only `column [ASC\|DESC]` lists (each column quoted) and raises `ArgumentError` (a 500: the string is the developer's) on anything else; request data must go through an allowlist first, e.g. `SORTS = { "title" => "title", "newest" => "created_at DESC" }` and `Post.order(SORTS.fetch(params[:sort].to_s, "id"))`, because a well-formed term can still name a missing column (a 500 from SQLite) or a column the client must not sort by; raw ORDER BY text goes through `order_sql` (`Post.order_sql("lower(title)")`, also on a relation), which must never see request data. A `limit`/`offset` on `delete_all` is honoured (a subselect), and request parameters nest at most 32 levels / 4096 pairs (400 past that). `Query.parse`, `Query.value_of`, `Query.decode` and `request.form_value` / `query_value` raise `Cybertrain::QueryMalformed` (a `StandardError` under `Cybertrain::QueryInvalid`, not an `ArgumentError`) on a malformed percent-escape, on both runtimes (`Cookies.parse` still keeps the raw value). They raise it too on an invalid UTF-8 byte sequence, raw (whatever the String's tag) or percent-encoded (Rails likewise answers 400 for an invalid parameter encoding), and so does a raw byte that only an escape would complete (`a=\xC3%A9`); `Cookies.parse` keeps such a value raw, and a path with an invalid sequence (raw, or percent-decoded in a segment) is a 400 as well (`request.path_segments`; a malformed escape in a path stays literal, a 404 by non-match). App code that rescued `ArgumentError` around them should rescue `Cybertrain::QueryInvalid` instead (upgrade note). The request keeps no parameter tree of its own (the Router builds `params`, the middleware read one key); an app that wants the whole tree parses it itself, `Query.parse(request.utf8_query_string)`, and owns the result |
521
1283
  | Full backtrace on an exception | Class, message, request line, template name/line — Spinel exposes no backtraces |
522
1284
  | Rack, its middleware, and any gem in a `Gemfile` | No Rack compatibility; a small fixed middleware set; Spinel's own `spin-index`, limited to what compiles under its Ruby subset |
523
1285
  | minitest / RSpec | `Cybertrain::Test` — reflection-based runners can't work ahead-of-time |
524
- | Auth, mailers, jobs, ActionCable, asset pipeline, i18n | Not built (MVP non-goals, `docs/design.md` §2.3) |
1286
+ | `bin/rails generate authentication`, `has_secure_password` + bcrypt | `cybertrain generate authentication` writes the same shape (User, Session, SessionsController, the before_action) plus a sign-up page, since there is no console to create a user in; `has_secure_password` is written out as methods on the User model over `Cybertrain::Password` (PBKDF2-HMAC-SHA256 in Ruby: bcrypt would need a C library the binary links, and `String#crypt` has no bcrypt on macOS). No password reset (no mailers); sign-in attempts are limited by `rate_limit` (counted per process; see [Rate limiting](#rate-limiting-and-the-client-address)) |
1287
+ | Active Storage, `has_one_attached :cover` | `cybertrain generate attachment post cover` writes five `cover_*` columns on the owner table and the model methods (`cover_file=`, `cover_url`, ...) into the app; files live on disk under `storage/uploads/` and are served by a generated action through `send_blob`. One file per field, local disk only, no variants or direct uploads |
1288
+ | `skip_before_action :x` | The same, for named before callbacks (a block callback has no name to skip); no `skip_after_action` |
1289
+ | `resource :session` | The same six routes to `SessionsController`; no nesting under it |
1290
+ | `get "/x"` then `post "/x"` | Both are fine: the second route's derived name is taken, so it is unnamed, as in Rails; a duplicate `as:` is still an error |
1291
+ | `Time.zone = "Asia/Tokyo"`, `time.in_time_zone`, ActiveSupport's bundled zone data | `Cybertrain::TimeZone.find("Asia/Tokyo")` reads the system's TZif files (`/usr/share/zoneinfo`, or `TZDIR`) and converts Integer epoch seconds (`utc_offset_at`, `utc_to_local`, `local_to_utc`); no per-request `Time.zone` and no `l(time, zone:)` yet ([Time zones](#time-zones)) |
1292
+ | `Net::HTTP`, Faraday | `Cybertrain::HttpClient` for webhooks and APIs: every wait bounded, non-public destinations refused by default, no redirects or keep-alive (see "Outbound HTTP"); Spinel's `Net::HTTP` blocks the whole server on a stalled peer |
1293
+ | Action Mailer (`deliver_later`, mailer classes, views) | `Cybertrain::Mailer.deliver(to:, subject:, body:)` sends plain text over SMTP (STARTTLS, implicit TLS or plain to loopback; AUTH PLAIN; every wait bounded); no templates, HTML, attachments or queue (see "Sending e-mail") |
1294
+ | Active Job, `perform_later`, `solid_queue`, recurring tasks | `Cybertrain::Job.handle(kind) { \|payload\| ... }` and `Job.enqueue(kind, payload, run_at:)` (a String payload in a `cybertrain_jobs` table, retried with backoff, then `failed`), `Cybertrain::Scheduler.every(seconds, name) { ... }`; one thread in the served process, `CYBERTRAIN_WORKER=0` elsewhere ([Background work](#background-work)) |
1295
+ | `config.filter_parameters += [:passw, /\Apin\z/, ->(k, v) { ... }]` | `config.filter_parameters` is an Array of Strings matched by substring, case-insensitively (no Regexps or lambdas); it applies to the request log and the development error page ([Filtered parameters](#configuration-and-environment-variables)) |
1296
+
1297
+ | `get "up" => "rails/health#show"` (a route to a controller) | `GET /up` is built in and answered before the static files, the session and the router, so no `before_action` runs; it queries the database and the background worker's last tick, `200 ok` or `503` ([Operations](#operations-backups-vacuum-health-check)) |
1298
+ | `bin/rails db:...` tasks for backups (none built in) | `./<app> db backup PATH` / `db backup DIR/ --keep N` (`VACUUM INTO`, mode 0600, renamed into place) and `./<app> db vacuum` |
1299
+ | ActionCable, asset pipeline, i18n | Not built (MVP non-goals, `docs/design.md` §2.3) |
525
1300
 
526
1301
  Not exhaustive — `docs/design.md` §8 ("捨てたもの") is the fuller record.
527
1302
 
1303
+ ## Performance
1304
+
1305
+ [docs/benchmark.md](docs/benchmark.md) measures examples/blog against the
1306
+ same blog written in Rails 8.1, [bench/rails_blog](bench/rails_blog) (the same
1307
+ views, routes, models and schema), both in production mode, each server
1308
+ pinned to the same CPU cores of one 4-vCPU cloud VM and loaded by wrk from
1309
+ two others. Medians of three 10-second runs, recorded 2026-10-09:
1310
+
1311
+ | 1 CPU core, 16 connections | CyberTrain 0.2.1 | Rails 8.1.4 (YJIT, Puma) | |
1312
+ | --- | --: | --: | --: |
1313
+ | `GET /articles` (30 articles) | 2,641 req/s | 468 req/s | 5.6x |
1314
+ | `GET /articles/1` (10 comments, 12 forms) | 2,450 req/s | 201 req/s | 12.2x |
1315
+ | `POST /articles/2/comments` | 1,507 req/s | 390 req/s | 3.9x |
1316
+ | `GET /style.css` | 13,300 req/s | 5,127 req/s | 2.6x |
1317
+ | Memory after the load (RSS) | 10.6 MB | 126 MB | |
1318
+ | Spawn to first response | 0.02 s | 1.3 s | |
1319
+ | What the server needs | 2.5 MB (one binary) | 109 MB (Ruby, 69 gems, the app) | |
1320
+
1321
+ With jemalloc in both (`allocator = "jemalloc"` in spin.toml; Rails'
1322
+ production image preloads it) the ratios are 2.8 to 14.6. On two cores the lead
1323
+ narrows to 2.0 to 5.5: a second Puma worker doubles Rails, while
1324
+ `SPINEL_WORKERS=2` adds 9 to 58% to CyberTrain (set it to the core count on a
1325
+ multi-core server; it defaults to 1). The binary also has to be built
1326
+ (`cybertrain build` took about three minutes there). docs/benchmark.md has
1327
+ latency, every configuration, the method and what was optimized to get here;
1328
+ `bench/run` repeats the measurement and `bench/report` prints its tables.
1329
+
528
1330
  ## Configuration and environment variables
529
1331
 
530
1332
  ```ruby
@@ -545,12 +1347,25 @@ end
545
1347
  | `CYBERTRAIN_SECRET_KEY_BASE` | `secret_key_base` | required in production; dev/test auto-generate one into `tmp/secret_key` |
546
1348
  | `SPINEL_WORKERS` | `workers` | `1` |
547
1349
  | `CYBERTRAIN_HOST` | `host` | `"127.0.0.1"`; `0.0.0.0` listens on every interface (containers) |
1350
+ | `CYBERTRAIN_ALLOWED_HOSTS` | `allowed_hosts` | Development accepts `localhost`, `127.0.0.1`, `[::1]`; comma-separated exact hostnames extend this list (no scheme, port or wildcard). Other hosts get 403 before static files, sessions, routing or diagnostics. This is separate from the listening address; production and test do not apply this development guard |
548
1351
  | `CYBERTRAIN_SESSION_SAME_SITE` | `session_same_site` | `"Lax"`; also `Strict` or `None` (`None` always adds `Secure`); anything else stops the server at boot |
549
1352
  | `CYBERTRAIN_SESSION_PARTITIONED` | `session_partitioned` | `false`; `1` or `true` adds `Partitioned` (and `Secure`) |
1353
+ | `CYBERTRAIN_TRUSTED_PROXY` | `trusted_proxy` | `false`; `1` or `true` makes `request.client_ip` (what `rate_limit` counts) the last `X-Forwarded-For` address of a connection from loopback, for a reverse proxy on the same host (see [Rate limiting](#rate-limiting-and-the-client-address)); shorthand for `CYBERTRAIN_TRUSTED_PROXIES=127.0.0.0/8,::1` |
1354
+ | `CYBERTRAIN_TRUSTED_PROXIES` | `trusted_proxies` | `[]`; comma-separated IPv4/IPv6 addresses or CIDRs (`172.16.0.0/12,10.0.0.0/8`) whose connections have `request.client_ip` taken from the last `X-Forwarded-For` address, for a proxy reached over a network, such as kamal-proxy in its container; a bad entry stops the server at boot |
1355
+ | `CYBERTRAIN_MAX_PAGE` | `max_page` | `10000`: the largest page `params.page_number` gives and `Relation#page` accepts (see [Pagination](#pagination)); a value that is not a number of at most 18 digits, or below 1, is the default |
1356
+ | `CYBERTRAIN_FILTER_PARAMETERS` | `filter_parameters` | `["password", "password_confirmation", "token", "secret", "secret_key_base", "session_key", "api_key", "authorization"]`; comma-separated names replace the list. A parameter whose name contains one of them (ASCII case-insensitively: `password` also matches `user[password]`) has its value written as `[FILTERED]` in the request log and on the development error page; `[]` in `config/app.rb` turns this off |
1357
+ | `CYBERTRAIN_WORKER` | `worker` | `true`; `0` or `false` keeps this process from running scheduled tasks and jobs (see [Background work](#background-work)); `scheduler_interval` (1 s), `job_poll_interval` (5 s) and `job_max_attempts` (5) are set in `config/app.rb` |
1358
+ | `CYBERTRAIN_HEALTH_PATH` | `health_path` | `"/up"`; the built-in health check's path (see [Operations](#operations-backups-vacuum-health-check)); an empty value turns it off; one without a leading `/` stops the server at boot. `health_worker_stale` (300 s) is set in `config/app.rb` |
1359
+ | `CYBERTRAIN_HTTP_ALLOW_PRIVATE` | `http_allow_private` | `false`; `1` or `true` lets `Cybertrain::HttpClient` connect to loopback, private and other non-public addresses (see "Outbound HTTP") |
1360
+ | `CYBERTRAIN_SMTP_HOST` | `smtp_host` | `""`; the SMTP server `Cybertrain::Mailer` sends through (see "Sending e-mail") |
1361
+ | `CYBERTRAIN_SMTP_PORT` | `smtp_port` | `0`: the mode's port, 587 for `starttls`, 465 for `tls`, 25 for `none`; anything but 1 to 65535 is 0 |
1362
+ | `CYBERTRAIN_SMTP_USER` | `smtp_user` | `""`: no AUTH; set, the mailer authenticates with AUTH PLAIN |
1363
+ | `CYBERTRAIN_SMTP_PASSWORD` | `smtp_password` | `""`; a secret, never logged, never in the boot banner or an error message |
1364
+ | `CYBERTRAIN_SMTP_FROM` | `smtp_from` | `""`: the sender `Mailer.deliver` uses when it is not given `from:` |
1365
+ | `CYBERTRAIN_SMTP_TLS` | `smtp_tls` | `"starttls"`; also `tls` (implicit TLS) or `none` (plain text, credentials to loopback only); both TLS modes need `require "cybertrain/net/tls"` |
550
1366
 
551
1367
  Use `None` (with `CYBERTRAIN_SESSION_PARTITIONED=1`) only for an app shown
552
- inside another site's frame, such as the playground's Codespaces preview; the
553
- browser must reach the app over HTTPS.
1368
+ inside another site's frame; the browser must reach the app over HTTPS.
554
1369
 
555
1370
  Other attributes with fixed, overridable defaults: `views_root`
556
1371
  (`"app/views"`), `public_root` (`"public"`), `layout`
@@ -558,13 +1373,61 @@ Other attributes with fixed, overridable defaults: `views_root`
558
1373
  `session_max_age` (2 weeks), `session_secure` (`true` in production, which
559
1374
  marks the session cookie `Secure`; `false` elsewhere, though `SameSite=None`
560
1375
  and `Partitioned` add `Secure` anyway), `pool_size` (4), `static_files`/`csrf`
561
- (`true`), `max_render_depth` (12 renders open at once: the page and its
1376
+ (`true`), `csrf_exempt_prefixes` (`[]`: path prefixes such as `"/api/"` whose
1377
+ non-GET requests skip the CSRF check, for an endpoint that programs call with
1378
+ a credential of their own; each must start and end with `/` and not be `/`
1379
+ alone, or boot fails; the prefix is matched on the path as sent, so a
1380
+ percent-encoded spelling is still checked; no other route may match a path
1381
+ under the prefix, and the exempt controller must not rely on the session
1382
+ for authentication, since the session is still loaded there), `max_body_bytes`
1383
+ (10 MiB: a longer request body, as sent, is answered 413 before it is read;
1384
+ raise it for large uploads; a gzip body's inflated size is bounded separately
1385
+ by `request.decoded_body(limit)`), `max_render_depth` (12 renders open at once: the page and its
562
1386
  partials; raise it for partials that legitimately recurse deeper; it must be
563
1387
  at least 1, or boot fails). Upgrade note: render nesting, unlimited before, is
564
1388
  now capped at 12 by default (the page is depth 1, so partials can nest 11
565
1389
  levels; the layout renders after the page and does not nest), and an app with
566
1390
  a deeper tree (threaded comments, a category menu) gets a template error until
567
- it sets `max_render_depth`.
1391
+ it sets `max_render_depth`. File uploads: `max_upload_bytes` (5 MiB per
1392
+ uploaded file; it must not exceed `max_body_bytes`), `max_multipart_parts`
1393
+ (128 parts, text fields included, at most 4096), `upload_root`
1394
+ (`"storage/uploads"`, where `Cybertrain::BlobStore` writes; created on the
1395
+ first upload, not at boot).
1396
+
1397
+ **Security headers:** every response the server writes (actions, the
1398
+ static-file middleware, error pages and the development diagnostics page,
1399
+ the server's own 400/411/413/500, and HEAD) carries
1400
+ `X-Content-Type-Options: nosniff` and `X-Frame-Options: SAMEORIGIN`, unless
1401
+ the response already sets a header of that name (any case): `Response#to_http`
1402
+ adds them after the response's own headers and only when missing, so
1403
+ `response.set_header("X-Frame-Options", "DENY")` in an action sends exactly
1404
+ your value, once. There is deliberately no `Config`
1405
+ switch to turn them off app-wide; an embeddable widget overrides the header
1406
+ on its own responses (a `Content-Security-Policy: frame-ancestors ...` header
1407
+ takes precedence over `X-Frame-Options` in current browsers). Because of
1408
+ `nosniff`, browsers take `Content-Type` literally: the static-file middleware
1409
+ knows the usual `public/` extensions (html, css, js/mjs, images including
1410
+ webp and avif, fonts, wasm, pdf, audio and video, json, csv, xml), and
1411
+ anything else is served as `application/octet-stream`, a download; add the
1412
+ extension to `Cybertrain::Static::EXTENSIONS` if an app needs another one
1413
+ inline. A reverse proxy that serves `public/` itself bypasses these headers
1414
+ (see [docs/deploy.en.md](docs/deploy.en.md)). `Cybertrain::Test::Client` calls the
1415
+ app in-process and never serializes the response, so it does not show the two
1416
+ defaults; assert them on `response.to_http`.
1417
+
1418
+ **Filtered parameters:** the request log's `Started GET "/login?..."` line
1419
+ and the development error page (its request line, and its dump of the
1420
+ params and of a JSON body) show the value of every parameter named in
1421
+ `config.filter_parameters` as `[FILTERED]`, as Rails' `filter_parameters`
1422
+ does, so a `?token=` in a URL or a password posted to a form that fails
1423
+ does not appear in the framework's request log or on the development
1424
+ error page. Only the values are replaced: the rest of the
1425
+ query string is logged as sent (control characters escaped). The list
1426
+ matches by substring, case-insensitively; extend it in `config/app.rb`
1427
+ (`c.filter_parameters = c.filter_parameters + ["otp", "ssn"]`) or replace
1428
+ it with `CYBERTRAIN_FILTER_PARAMETERS`. Values an action logs itself are not
1429
+ filtered; ask `Cybertrain::ParameterFilter.filter?(name)` or
1430
+ `ParameterFilter.filter_query(text)` there.
568
1431
 
569
1432
  **Deployment:** `cybertrain build` produces `dist/`: the binary `dist/NAME`
570
1433
  with the views embedded, `dist/public/` (static assets, or let a reverse
@@ -580,10 +1443,277 @@ plain-text 404, `head :not_found`, `render plain:`) is replaced by
580
1443
  or JSON pass through. `SIGTERM` stops accepting, lets requests already in
581
1444
  flight finish (answered with `Connection: close`; idle keep-alive connections
582
1445
  are closed at once) and exits when they are done, or after the server's
583
- `drain_timeout` (10 s). The watcher/rebuild loop never runs in production. [docs/deploy.md](docs/deploy.md)
584
- (Japanese) walks through it end to end: `cybertrain new`, an Ubuntu server,
1446
+ `drain_timeout` (10 s). The watcher/rebuild loop never runs in production. [docs/deploy.en.md](docs/deploy.en.md)
1447
+ (in Japanese: [docs/deploy.md](docs/deploy.md)) walks through it end to end: `cybertrain new`, an Ubuntu server,
585
1448
  systemd, Caddy with HTTPS, redeploys, rollbacks and backups.
586
1449
 
1450
+ **Kamal:** `cybertrain new` also writes what [Kamal 2](https://kamal-deploy.org)
1451
+ needs: a `Dockerfile` that runs `cybertrain setup` and `cybertrain build` in an
1452
+ Ubuntu build stage and copies `dist/` into a slim stage of the same release
1453
+ (migrate on boot, then serve on `0.0.0.0:3000`), a `.dockerignore`,
1454
+ `config/deploy.yml` (kamal-proxy with Let's Encrypt in front, the
1455
+ health check on `public/up.html`, the SQLite file on a volume at
1456
+ `/app/storage`) and `.kamal/secrets` (`KAMAL_REGISTRY_PASSWORD` and
1457
+ `CYBERTRAIN_SECRET_KEY_BASE` from the environment). Fill in the values marked
1458
+ `CHANGE ME`, commit, and `kamal setup`;
1459
+ [docs/deploy-kamal.md](docs/deploy-kamal.md) has the details (part 8 of
1460
+ the deployment tutorial).
1461
+ [examples/blog](examples/blog#deploying-with-kamal) carries the same setup,
1462
+ built against this checkout's framework.
1463
+
1464
+ ### Application settings
1465
+
1466
+ An application's own settings (a public base URL, a feature switch, an
1467
+ interval, an SMTP host) are declared on the same `Config`, with a type, an
1468
+ environment variable and a default:
1469
+
1470
+ ```ruby
1471
+ # config/app.rb
1472
+ Cybertrain.configure do |c|
1473
+ c.setting_string :base_url, env: "MYAPP_BASE_URL", default: ""
1474
+ c.setting_string :smtp_password, env: "MYAPP_SMTP_PASSWORD", default: "", secret: true
1475
+ c.setting_bool :worker, env: "MYAPP_WORKER", default: true
1476
+ c.setting_int :prune_interval, env: "MYAPP_PRUNE_INTERVAL", default: 3600, min: 1, max: 9_999_999
1477
+ end
1478
+
1479
+ Cybertrain.config.string_setting(:base_url)
1480
+ Cybertrain.config.bool_setting(:worker)
1481
+ Cybertrain.config.int_setting(:prune_interval)
1482
+ ```
1483
+
1484
+ Each value is read once, when `config/app.rb` runs at startup. An empty or
1485
+ unset variable means the default. A boolean takes `1`/`true` or
1486
+ `0`/`false`; an integer takes decimal digits (an optional leading `-`) within
1487
+ `min..max` (both optional). Any other value falls back to the default and
1488
+ logs a `[WARN]` line saying so. Reading an undeclared name, or one declared
1489
+ with another type, raises `ArgumentError`, as does declaring a name twice.
1490
+ Tests change a value with `set_string_setting`, `set_int_setting` and
1491
+ `set_bool_setting`. `./NAME help` lists every declared setting with its
1492
+ variable and default, and the boot banner prints the current values
1493
+ (`=> Settings: base_url="" worker=true ...`); a `secret: true` setting shows
1494
+ up in both only by name, as `[FILTERED]`. There is no
1495
+ `Cybertrain.config.base_url` accessor: Spinel has no `define_method`, so the
1496
+ readers are typed methods that take the name.
1497
+
1498
+ ## Time zones
1499
+
1500
+ Spinel ships no tz database, and setting `ENV["TZ"]` to convert a time
1501
+ changes it for every thread of the process. `Cybertrain::TimeZone` reads the
1502
+ system's zone files instead: `/usr/share/zoneinfo/<name>`, or `$TZDIR/<name>`
1503
+ when `TZDIR` is set (install the `tzdata` package in a slim container image).
1504
+ It parses TZif versions 1 to 4 (RFC 8536), including the POSIX TZ rule a
1505
+ "slim" file relies on after its last transition, and caches each zone for the
1506
+ life of the process (copy-on-write, so lookups from several workers take no
1507
+ lock).
1508
+
1509
+ Everything is Integer epoch seconds. A *local epoch* is a wall-clock reading
1510
+ counted as if it were UTC, so wall-clock arithmetic is plain addition and
1511
+ `TimeZone.civil_to_epoch` / `TimeZone.epoch_to_civil` move between it and
1512
+ `[year, month, day, hour, minute, second]`:
1513
+
1514
+ ```ruby
1515
+ tokyo = Cybertrain::TimeZone.find("Asia/Tokyo")
1516
+ tokyo.found? # false for an unknown name: the zone then behaves as UTC
1517
+ tokyo.utc_offset_at(Time.now.to_i) # => 32400
1518
+ tokyo.abbreviation_at(Time.now.to_i) # => "JST"
1519
+
1520
+ # The next 09:00 in Tokyo:
1521
+ now = Time.now.to_i
1522
+ today = Cybertrain::TimeZone.epoch_to_civil(tokyo.utc_to_local(now))
1523
+ run = tokyo.local_to_utc(Cybertrain::TimeZone.civil_to_epoch(today[0], today[1], today[2], 9, 0, 0))
1524
+ run = tokyo.local_to_utc(Cybertrain::TimeZone.civil_to_epoch(today[0], today[1], today[2] + 1, 9, 0, 0)) if run <= now
1525
+ ```
1526
+
1527
+ At a daylight-saving transition `local_to_utc` is deterministic: a wall time
1528
+ that a forward jump skips (02:30 on a US spring-forward day) is read with the
1529
+ offset before the jump, so it lands after it (03:30 EDT); a wall time that a
1530
+ backward jump repeats gives the earlier instant. `find` accepts only a
1531
+ relative name made of `A-Za-z0-9_+-./` with no empty, `.` or `..` segment, so
1532
+ a zone name taken from a user's profile cannot reach a file outside the zone
1533
+ directory; an unknown or malformed zone is not cached, and `"UTC"` needs no
1534
+ file. Formatting helpers for templates (`l(time, zone:)`) are not there yet.
1535
+
1536
+ ## Outbound HTTP
1537
+
1538
+ An application that calls another server (a webhook, a Slack incoming
1539
+ webhook, a third-party API) uses `Cybertrain::HttpClient`. It is not in
1540
+ `require "cybertrain"`; require it where it is used, and the TLS layer too
1541
+ when any URL is https:
1542
+
1543
+ ```ruby
1544
+ require "json"
1545
+ require "cybertrain/http_client"
1546
+ require "cybertrain/net/tls" # only for https://; links the system libssl
1547
+
1548
+ client = Cybertrain::HttpClient.new(timeout: 5, deadline: 15, max_body: 1 << 20)
1549
+ begin
1550
+ res = client.post(ENV["SLACK_WEBHOOK_URL"], JSON.generate({ "text" => "Deployed" }),
1551
+ content_type: "application/json")
1552
+ Cybertrain.logger.warn("slack answered #{res.status}") unless res.success?
1553
+ rescue Cybertrain::HttpClient::Error => e
1554
+ Cybertrain.logger.warn("slack: #{e.message}") # names host:port, never the path
1555
+ end
1556
+
1557
+ res = client.get("https://api.example.com/v1/status", headers: { "Authorization" => "Bearer #{token}" })
1558
+ res.status # 200
1559
+ res.header("ETag") # names in any case; res.headers has them lowercase
1560
+ res.body # the bytes, chunked coding removed
1561
+ ```
1562
+
1563
+ Why it exists: Spinel's `Net::HTTP` does the TLS handshake and every read
1564
+ after the first byte as blocking C calls, and each one holds an OS worker,
1565
+ which with the default `SPINEL_WORKERS=1` is the whole web server, for as
1566
+ long as a stalled endpoint likes. `HttpClient` never blocks a worker on the
1567
+ network:
1568
+
1569
+ - **Every wait is bounded.** Connecting (a non-blocking connect; Spinel's
1570
+ `TCPSocket.new` ignores `connect_timeout:`), each TLS handshake step, each
1571
+ read and each write waits at most `timeout` seconds (default 5), and the
1572
+ whole request at most `deadline` seconds (default 15). Past either,
1573
+ `HttpClient::TimeoutError`. Reads go through a byte buffer fed by
1574
+ `read_nonblock`, so bytes already received are never waited for.
1575
+ - **Name resolution blocks.** `getaddrinfo` has no non-blocking form in the
1576
+ runtime: the thread making the request waits for DNS (usually
1577
+ milliseconds, but a dead resolver can take the system's timeout, about 5 s
1578
+ per attempt). The deadline counts that time but cannot interrupt it.
1579
+ - **Non-public destinations are refused** before anything is sent:
1580
+ loopback (`127.0.0.0/8`, `::1`), private (`10/8`, `172.16/12`,
1581
+ `192.168/16`, `fc00::/7`), link-local (`169.254/16`, which includes cloud
1582
+ metadata services, `fe80::/10`), unspecified (`0.0.0.0/8`, `::`),
1583
+ multicast, broadcast, CGNAT (`100.64/10`), the documentation and
1584
+ benchmarking ranges, and IPv4-mapped, NAT64 and 6to4 forms of those
1585
+ (`Cybertrain::Net::Address.blocked_reason`). The host name is resolved
1586
+ once and the addresses it resolved to are the ones checked and dialled
1587
+ (nothing is looked up again), so a name that resolves to a private
1588
+ address (an `/etc/hosts` entry, DNS rebinding) is refused like the
1589
+ literal; the TLS certificate is still checked against the name. The
1590
+ check is per address: a refused address is skipped, and only when every
1591
+ address is refused does the request fail.
1592
+ `HttpClient::BlockedAddressError` says why. To call a service on your
1593
+ own network, or a local server in a test, set `c.http_allow_private =
1594
+ true` in `config/app.rb` (or `CYBERTRAIN_HTTP_ALLOW_PRIVATE=1`), or pass
1595
+ `allow_private: true` to `HttpClient.new`; leave it off whenever a URL
1596
+ can come from a user.
1597
+ - **Every resolved address is tried**, in the order the resolver gives
1598
+ them, IPv6 and IPv4 alike: when one refuses, is unreachable, resets or
1599
+ does not answer within `timeout`, the next is dialled, all of them
1600
+ within `deadline`, and when every one fails the last one's error is
1601
+ raised. This is what makes `localhost` work on macOS, where it resolves
1602
+ to `::1` first and a server listening on `127.0.0.1` refuses that.
1603
+ `Cybertrain::Mailer` dials the same way.
1604
+ - **TLS** is the openssl package's: `VERIFY_PEER` against the operating
1605
+ system's trust store (`SSL_CERT_FILE` / `SSL_CERT_DIR` point it
1606
+ elsewhere; the package has no `X509::Store` for a custom one), host-name
1607
+ checking, SNI, TLS 1.2 or newer. An https URL needs a host name: an IP
1608
+ address does not match a certificate's name. Without `require
1609
+ "cybertrain/net/tls"`, an https request raises `HttpClient::TLSError`
1610
+ saying so; `cybertrain/http_client` alone never links libssl.
1611
+ - **A notification client, not a browser:** HTTP/1.1 with `Connection:
1612
+ close`, one connection per request, no keep-alive, no redirects (a 3xx is
1613
+ returned as it is), no proxy, no cookies, no compression. Responses may be
1614
+ `Content-Length`, chunked or read to the close; the headers are capped at
1615
+ 64 KiB and 100 lines, the body at `max_body` (default 1 MiB), past which
1616
+ `HttpClient::TooLargeError`. A response that is not HTTP/1.x, or ends
1617
+ early, is `HttpClient::BadResponseError`; a refused or reset connection, or
1618
+ a name that does not resolve, `HttpClient::RefusedError`. A connection
1619
+ that is reset or closed during the TLS handshake (a server that does not
1620
+ speak TLS) is `HttpClient::TLSError` ("TLS handshake failed") on every
1621
+ platform, like a handshake OpenSSL rejects. All of them are
1622
+ `HttpClient::Error`s.
1623
+ - **Errors never contain the path or the query**, where a webhook URL
1624
+ carries its secret: messages name the host and port only. A malformed URL
1625
+ or a header with CR/LF is an `ArgumentError` that does not repeat it
1626
+ either. `Host`, `Connection`, `Content-Length` and `Transfer-Encoding` are
1627
+ the client's; `User-Agent` defaults to `cybertrain/<version>`.
1628
+
1629
+ The request runs on the calling thread, a green thread like the request
1630
+ being served, so other connections keep being served while it waits. The
1631
+ socket layer (`cybertrain/net/socket.rb`: bounded connect, buffered reads,
1632
+ bounded writes) is separate from the HTTP layer so that other protocols can
1633
+ use it. The design: [docs/design.md](docs/design.md) D24.
1634
+
1635
+ ## Sending e-mail
1636
+
1637
+ `Cybertrain::Mailer` sends plain-text e-mail through an SMTP server (your
1638
+ provider's submission server, or a local relay). Like the HTTP client it is
1639
+ not in `require "cybertrain"`, and TLS is a separate require:
1640
+
1641
+ ```ruby
1642
+ require "cybertrain/mailer"
1643
+ require "cybertrain/net/tls" # for starttls (the default) and tls; links the system libssl
1644
+
1645
+ # Settings from CYBERTRAIN_SMTP_HOST, _PORT, _USER, _PASSWORD, _FROM, _TLS:
1646
+ begin
1647
+ Cybertrain::Mailer.deliver(to: user.email, subject: "Welcome", body: "Hello #{user.name},\n...")
1648
+ rescue Cybertrain::Mailer::Error => e
1649
+ Cybertrain.logger.warn("mail: #{e.message}") # never contains the password
1650
+ end
1651
+
1652
+ # Or explicit settings, bypassing Config:
1653
+ mailer = Cybertrain::Mailer.new(host: "smtp.example.com", port: 587, tls: "starttls",
1654
+ user: "app", password: ENV["SMTP_PASSWORD"].to_s,
1655
+ from: "noreply@example.com", timeout: 10, deadline: 60)
1656
+ mailer.deliver(to: "a@example.com, b@example.com", subject: "Report", body: report)
1657
+ ```
1658
+
1659
+ | Variable | Attribute | Default |
1660
+ | --- | --- | --- |
1661
+ | `CYBERTRAIN_SMTP_HOST` | `smtp_host` | `""` (`Mailer.new` raises `ArgumentError` until it is set) |
1662
+ | `CYBERTRAIN_SMTP_PORT` | `smtp_port` | `0`: 587 for `starttls`, 465 for `tls`, 25 for `none` |
1663
+ | `CYBERTRAIN_SMTP_USER` | `smtp_user` | `""`: no AUTH |
1664
+ | `CYBERTRAIN_SMTP_PASSWORD` | `smtp_password` | `""` (secret) |
1665
+ | `CYBERTRAIN_SMTP_FROM` | `smtp_from` | `""` |
1666
+ | `CYBERTRAIN_SMTP_TLS` | `smtp_tls` | `"starttls"`; `tls` or `none` |
1667
+
1668
+ - **TLS modes.** `starttls` connects in plain text, requires the server to
1669
+ offer STARTTLS (else `Mailer::TLSError`) and upgrades before anything
1670
+ else is said; bytes the server sends after its STARTTLS reply are
1671
+ refused. `tls` speaks TLS from the first byte (port 465). Both check the
1672
+ certificate against `host` as the HTTP client does (give a host name, not
1673
+ an IP address), and both need `require "cybertrain/net/tls"`: without
1674
+ it, `deliver` raises `Mailer::TLSError` saying so, before connecting.
1675
+ `none` is plain text, for a relay on the same machine or a test server:
1676
+ with a user configured it refuses (`Mailer::InsecureError`, before
1677
+ connecting) unless the host resolves to a loopback address, and then
1678
+ dials only its loopback addresses.
1679
+ - **The password is never logged or shown.** It is not in the boot banner,
1680
+ `help`, the log or any error message; it is sent only inside the AUTH
1681
+ PLAIN command, over TLS (or to loopback). Do not `p` or `inspect` a
1682
+ `Mailer` or the `Config`, which hold it.
1683
+ - **Every wait is bounded**, as in "Outbound HTTP": connecting, each TLS
1684
+ step and each reply waits at most `timeout` seconds (default 10), the
1685
+ whole delivery at most `deadline` seconds (default 60), else
1686
+ `Mailer::TimeoutError`. Name resolution blocks the calling thread.
1687
+ - **The dialogue:** the 220 greeting, `EHLO` (the name is `helo:`, default
1688
+ `localhost`), `STARTTLS` and `EHLO` again, `AUTH PLAIN` when a user is
1689
+ set (the server must offer PLAIN), `MAIL FROM`, one `RCPT TO` per
1690
+ recipient, `DATA`, `QUIT`. A reply with another code is a
1691
+ `Mailer::RejectedError` whose `smtp_code` is the server's (421, 535,
1692
+ 550, ...) and whose message has the step and the server's text. The
1693
+ return value is the server's reply to the message (`"250 2.0.0 Ok: queued
1694
+ as ..."`).
1695
+ - **The message:** `Date`, `From`, `To`, `Subject`, `MIME-Version: 1.0`,
1696
+ `Content-Type: text/plain; charset=UTF-8`. A non-ASCII subject is RFC 2047
1697
+ encoded (`=?UTF-8?B?...?=`, split between characters). The body's line
1698
+ endings become CRLF and lines starting with `.` are dot-stuffed. Its
1699
+ `Content-Transfer-Encoding` is `7bit` for ASCII, `8bit` (with `MAIL FROM
1700
+ ... BODY=8BITMIME`) for UTF-8 when the server advertises 8BITMIME, and
1701
+ `base64` otherwise or when a line is longer than 998 bytes. No
1702
+ `Message-ID` is added (submission servers add one).
1703
+ - **What is refused** with an `ArgumentError`, before connecting: an
1704
+ address that is not a bare ASCII `local@domain` (no display name, no
1705
+ `<>`, spaces, commas, quotes, CR or LF; at most 254 bytes), a subject with
1706
+ CR, LF or NUL (header injection) or over 998 bytes, a body that is not
1707
+ valid UTF-8 or holds a NUL, more than 100 recipients. `to:` takes one
1708
+ address or several separated by commas.
1709
+ - **Not yet:** HTML and multipart messages, attachments, templates, display
1710
+ names, `Cc`/`Bcc` headers (send to several recipients instead), AUTH
1711
+ LOGIN/CRAM-MD5, SMTPUTF8 addresses, pipelining, and a delivery queue (a
1712
+ delivery runs on the calling thread, a green thread like a request).
1713
+ `cybertrain generate authentication` still writes no password reset.
1714
+
1715
+ The design: [docs/design.md](docs/design.md) D27.
1716
+
587
1717
  ## Testing an app
588
1718
 
589
1719
  App tests are plain Spinel programs under `test/`, like the framework's own:
@@ -626,6 +1756,11 @@ rendered form (`BlogTest.form_token`). End each test file with
626
1756
  `Cybertrain::Test.run!`; `examples/blog/test/articles.rb` and `comments.rb`
627
1757
  show the full pattern.
628
1758
 
1759
+ Every request a test client sends comes from `127.0.0.1` (`request.remote_addr`,
1760
+ so `request.loopback?` is true). To see what a page does for a visitor on
1761
+ another machine, set `client.peer_address = "203.0.113.7"` before the
1762
+ request.
1763
+
629
1764
  Run with `cybertrain spin test`: it compiles each `test/*.rb` into its own
630
1765
  program and diffs its output (stdout and stderr together) against
631
1766
  `test/<name>.rb.expected`. Regenerate with `cybertrain spin test --regen
@@ -638,6 +1773,79 @@ the capture when the compile failed, so the old snapshot is kept; in the
638
1773
  framework checkout, `script/regen-snapshot test/<name>.rb` does exactly this and
639
1774
  prints the diff stat).
640
1775
 
1776
+ ### Testing outbound calls
1777
+
1778
+ When the code under test sends a webhook ([Outbound HTTP](#outbound-http)) or
1779
+ an e-mail ([Sending e-mail](#sending-e-mail)), the test receives it
1780
+ in-process: `Cybertrain::Test::HttpReceiver` and `Cybertrain::Test::SmtpReceiver`
1781
+ listen on `127.0.0.1`, record what arrives and answer with a canned reply.
1782
+
1783
+ ```ruby
1784
+ require_relative "support/app_test"
1785
+ require "cybertrain/test/http_receiver"
1786
+ require "cybertrain/test/smtp_receiver"
1787
+
1788
+ test "a sign-up posts a webhook" do
1789
+ hook = Cybertrain::Test::HttpReceiver.new(status: 202, body: "queued")
1790
+ hook.start # binds a free port, returns it
1791
+ Cybertrain.config.http_allow_private = true # HttpClient refuses loopback otherwise
1792
+ SignupHook.post(hook.url("/hooks/signup"), "ann@example.com") # the app's code
1793
+ req = hook.received_request # waits at most 5 s; nil if nothing came
1794
+ hook.stop
1795
+ assert_equal "POST", req.http_method
1796
+ assert_equal "/hooks/signup", req.http_target
1797
+ assert_equal "application/json", req.http_header("Content-Type")
1798
+ assert_includes req.http_body, "ann@example.com"
1799
+ end
1800
+
1801
+ test "a refused recipient is reported" do
1802
+ smtp = Cybertrain::Test::SmtpReceiver.new(reject: { rcpt: "550 5.1.1 no such user" })
1803
+ port = smtp.start
1804
+ mailer = Cybertrain::Mailer.new(host: "127.0.0.1", port: port, tls: "none", from: "app@example.com")
1805
+ assert_raises("RejectedError") { mailer.deliver(to: "nobody@example.org", subject: "Hi", body: "Hello\n") }
1806
+ mail = smtp.received_message
1807
+ smtp.stop
1808
+ assert_equal ["EHLO localhost", "MAIL FROM:<app@example.com>", "RCPT TO:<nobody@example.org>"],
1809
+ mail.transcript.first(3)
1810
+ refute mail.delivered?
1811
+ end
1812
+ ```
1813
+
1814
+ `HttpReceiver.new` takes the canned `status:`, `headers:`, `body:` and
1815
+ `chunked:` (`respond_with` changes them between requests); the request comes
1816
+ back as a `ReceivedRequest` (`http_method`, `http_target`, `http_path`,
1817
+ `http_headers` with lowercased names, `http_header(name)`, `http_body` with a
1818
+ chunked body decoded). `SmtpReceiver` plays 220, EHLO (with 8BITMIME unless
1819
+ `eight_bit_mime: false`, AUTH PLAIN with `auth: true`), MAIL, RCPT, DATA and
1820
+ QUIT; `reject:` fails any of `:greeting`, `:ehlo`, `:mail`, `:rcpt`, `:data`
1821
+ or `:message` with the reply given. A `ReceivedMessage` has the
1822
+ `envelope_from`, `envelope_recipients`, the `message_data` (dot-stuffing
1823
+ undone), `message_header(name)`, `message_body` and the `transcript` of
1824
+ commands. Both accept one connection unless `max:` says more
1825
+ (`received_requests` / `received_messages` return them all). Every wait is
1826
+ bounded: a receiver gives up after `accept_window:` (10 s) without a
1827
+ connection, a silent client after `read_timeout:` (5 s), and `stop` (safe
1828
+ to call twice) ends and joins the receiver's thread and closes its port, so
1829
+ a receiver never takes a later test's connection. They need only Ruby's
1830
+ socket library, not openssl.
1831
+
1832
+ ### Continuous integration
1833
+
1834
+ `cybertrain new` writes `.github/workflows/ci.yml` for the app. It calls the
1835
+ `setup-cybertrain` action of this repository
1836
+ ([.github/actions/setup-cybertrain](.github/actions/setup-cybertrain/action.yml)),
1837
+ pinned to the CLI's release tag, which installs the SQLite, zlib and OpenSSL
1838
+ headers and Ruby, reads the framework commit the app is locked to out of
1839
+ `spin.lock`, fetches the framework at that commit and runs its `cybertrain
1840
+ doctor` and `cybertrain setup` (with `~/.cybertrain/spinel` cached per Spinel
1841
+ release and runner image). The workflow then checks that `spin.lock` is
1842
+ current (`spin lock && git diff --exit-code spin.lock`) and `gen/` is fresh
1843
+ (`spin run gen -- --check`), and runs `spin test` once `test/` has a test
1844
+ program. When the toolchain steps change, the action changes, not every app's
1845
+ copy. The commit is read from `spin.lock`, so the workflow assumes the
1846
+ default git dependency; an app made with `--path` passes the framework
1847
+ checkout to the action as `framework-path`.
1848
+
641
1849
  ## Learn more
642
1850
 
643
1851
  - [Homepage](https://saeki-mototsune.github.io/CyberTrain/),
@@ -657,13 +1865,25 @@ prints the diff stat).
657
1865
  decision, what was rejected and why, and the Spinel constraints behind it.
658
1866
  - [docs/template-language.md](docs/template-language.md) — the full template
659
1867
  grammar and helpers, and where it differs from Rails' ERB.
1868
+ - [docs/benchmark.md](docs/benchmark.md) — examples/blog measured against the
1869
+ same blog in Rails 8.1 ([bench/](bench/)), and how to run it again.
660
1870
  - [spikes/NOTES.md](spikes/NOTES.md) — the spikes that answered
661
1871
  `docs/design.md`'s open questions, and the compiler constraints they found.
662
- - [Spinel](https://github.com/matz/spinel) — the AOT Ruby compiler cybertrain targets.
1872
+ - [Spinel](https://github.com/matz/spinel) — the AOT Ruby compiler cybertrain targets;
1873
+ cybertrain pins a release of the
1874
+ [saeki-mototsune/spinel](https://github.com/saeki-mototsune/spinel) fork.
663
1875
  - [Roundhouse](https://github.com/rubys/roundhouse) (Sam Ruby) — transpiles
664
1876
  *existing* Rails apps to Spinel. cybertrain is not that: it's a native
665
1877
  framework you write directly against, not a compatibility layer.
666
1878
 
1879
+ ## Contributing
1880
+
1881
+ Bug reports, questions and pull requests are welcome. [CONTRIBUTING.md](CONTRIBUTING.md)
1882
+ explains how to set up a checkout, run the tests and the commit gate
1883
+ (`script/install-hooks` once per clone; `script/gate` runs the same checks CI
1884
+ does), and what a pull request needs. Security issues go through
1885
+ [SECURITY.md](SECURITY.md), not an issue.
1886
+
667
1887
  ## Releasing
668
1888
 
669
1889
  The gem and the framework are released from the same tag, and `cybertrain
@@ -682,9 +1902,11 @@ before the gem is pushed:
682
1902
 
683
1903
  `Cybertrain::SPINEL_TAG` (also in `cybertrain/version.rb`) is the Spinel
684
1904
  release `cybertrain setup` installs. It must equal `SPINEL_TAG` in
685
- [.github/workflows/ci.yml](.github/workflows/ci.yml), which CI checks, so
686
- change the two together when the framework moves to another Spinel release.
1905
+ [.github/workflows/ci.yml](.github/workflows/ci.yml) and in
1906
+ [script/claude-cloud-setup](script/claude-cloud-setup), which `script/gate`
1907
+ checks, so change the three together when the framework moves to another
1908
+ Spinel release.
687
1909
 
688
1910
  ## License
689
1911
 
690
- MIT.
1912
+ MIT; see [LICENSE](LICENSE). Contributions are accepted under the same license.