cybertrain 0.3.1 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -7,133 +7,57 @@
7
7
  <p align="center">
8
8
  <a href="https://saeki-mototsune.github.io/CyberTrain/">Homepage</a> ·
9
9
  <a href="https://saeki-mototsune.github.io/CyberTrain/tutorial.html">Tutorial</a> ·
10
+ <a href="https://saeki-mototsune.github.io/CyberTrain/guide/">Guides</a> ·
10
11
  <a href="https://saeki-mototsune.github.io/CyberTrain/api/">API reference</a>
11
12
  </p>
12
13
 
14
+
13
15
  # cybertrain
14
16
 
15
- cybertrain is a Rails-shaped web application framework written natively for
17
+ A Rails-shaped web framework written natively for
16
18
  [Spinel](https://github.com/matz/spinel), matz's ahead-of-time Ruby compiler.
17
- There is no Rack, no gems at runtime, and no Ruby interpreter in the deployed
18
- artifact: `cybertrain build` compiles an application — controllers, models,
19
- views and the framework itself — into a single native binary. Where Rails leans on
20
- runtime metaprogramming (`method_missing`, `define_method`, `eval`) for
21
- `before_action :set_post` or `Post.where(title: "x")`, cybertrain gets the
22
- same vocabulary by generating plain, committed Ruby from `db/schema.rb` and
23
- `config/routes.rb` at build time (`spin run gen`), plus a little runtime data
24
- for validations/callbacks and a runtime-interpreted template language.
25
-
26
- **Status:** pre-alpha. Every milestone in [docs/design.md](docs/design.md) is
27
- implemented end to end: an HTTP/1.1 server, routing and controllers with
28
- callbacks and `rescue_from`, cookie sessions/flash/CSRF, an ERB-flavored
29
- template engine, SQLite-backed models with validations, callbacks and
30
- schema-derived associations, migrations, the `cybertrain` CLI, and a
31
- development loop that rebuilds and re-execs the server on code changes.
32
- [examples/blog](examples/blog) is the Rails "Getting Started" blog (articles
33
- with comments) built on it, exercised by CI on Linux and macOS.
34
- `cybertrain generate authentication` adds email-and-password sign-in the way
35
- Rails 8's generator does, and `cybertrain generate attachment` (or a
36
- `cover:attachment` scaffold field) adds a file upload to a model. `Cybertrain::Mailer` sends plain-text e-mail over SMTP
37
- (see "Sending e-mail"), and `Cybertrain::WebSocket.subscribe` keeps a connection to an outside
38
- WebSocket feed open while the server runs (see "Consuming WebSocket feeds"). Not built: Action Mailer-style
39
- mailer classes and templates, a WebSocket server (ActionCable),
40
- an asset pipeline, i18n, and anything beyond a minimal `render json:` — see
41
- "Differences from Rails" below.
42
-
43
- ## Try it in the browser
44
-
45
- [Open the playground in GitHub Codespaces](https://codespaces.new/saeki-mototsune/CyberTrain?quickstart=1)
46
- to try cybertrain without installing anything: VS Code opens in your browser with
47
- Spinel 2026.10.27.7020, the `cybertrain` CLI and the blog from the walkthrough below
48
- already created (`cybertrain new blog`, the article scaffold, the root route and
49
- the first migration, built once so the server starts without compiling), its
50
- development server running in a terminal and the app, showing the article list,
51
- in a new browser tab. The app is a git repository with one commit, so Source
52
- Control shows what you change. `PLAYGROUND.md` in the app lists what to try, from
53
- editing a view to carrying on with the walkthrough's comments. A view edit shows
54
- on the next reload; a Ruby edit is a full rebuild, about a minute, after which the
55
- server restarts by itself; the app's tab does not reload by itself. `cybertrain new`
56
- works there with no network.
57
-
58
- - A GitHub account is required, and the codespace runs on your own Codespaces
59
- quota: GitHub's free plan includes 120 core-hours and 15 GB-month of storage a
60
- month, about 60 hours on the default 2-core machine the playground uses. GitHub
61
- stops an idle codespace after 30 minutes by default; its storage counts until
62
- you delete it.
63
- - If VS Code asks whether you trust the authors of the files in this folder,
64
- choose "Trust Folder & Continue": the server's terminal starts only then.
65
- - If the app's tab did not open (a pop-up blocker, or the codespace was stopped
66
- and restarted), click "Open in Browser" in VS Code's notification for port
67
- 3000 or in the Ports view, or Ctrl-click the app's URL in the `server`
68
- terminal. `PLAYGROUND.md` shows where each is.
69
- - The same image runs locally (it is published for linux/amd64; on arm64, build
70
- it from a checkout):
71
-
72
- ```sh
73
- docker run --rm -it --init -p 127.0.0.1:3000:3000 -e CYBERTRAIN_HOST=0.0.0.0 ghcr.io/saeki-mototsune/cybertrain-playground
74
- ```
75
-
76
- Then open http://localhost:3000. [playground/README.md](playground/README.md)
77
- describes the image, building it and the Codespaces setup.
78
-
79
- ## Requirements
80
-
81
- - Spinel `2026.10.27.7020` (`Cybertrain::SPINEL_TAG`, the same release as
82
- `SPINEL_TAG` in [.github/workflows/ci.yml](.github/workflows/ci.yml)), a
83
- release of [saeki-mototsune/spinel](https://github.com/saeki-mototsune/spinel)
84
- (`Cybertrain::SPINEL_REPOSITORY`), the fork of matz/spinel that carries
85
- the fixes cybertrain needs ahead of an upstream release;
86
- untested against other versions. You do not install it by hand: the
87
- `cybertrain` command builds that release into `~/.cybertrain` the first
88
- time it needs it (`cybertrain setup` does that on request), unless a
89
- `spinel` of that release is already on `PATH` — see "Installing the CLI".
90
- - To build it: a C toolchain (`cc`, or the compiler `CC` names) and `make`
91
- — Spinel compiles every program to C — plus `git` and `curl`.
92
- - SQLite 3 headers/library (`libsqlite3-dev` on Debian/Ubuntu; present with
93
- Xcode's command line tools on macOS) — models use Spinel's FFI directly,
94
- not a gem.
95
- - zlib headers/library (`zlib1g-dev` on Debian/Ubuntu; in the macOS SDK) —
96
- gzip request bodies (`request.decoded_body`) are inflated through the
97
- system zlib, bound the same way.
98
- - Ruby 3.2+ and RubyGems, for the `cybertrain` command only (applications
99
- never run on CRuby). Building Spinel also needs Ruby's `prism` gem, which
100
- Ruby 3.3 and newer bundle; on Ruby 3.2, `gem install prism` (with Ruby's
101
- development headers, `ruby-dev` on Debian/Ubuntu). `cybertrain doctor`
102
- checks it.
103
- - OpenSSL's headers (`libssl-dev`) are optional: without them Spinel builds
104
- without its `openssl` package, which cybertrain does not use. On macOS an
105
- application that requires `openssl` needs Homebrew's (keg-only) OpenSSL on
106
- the compiler's paths: `export LIBRARY_PATH="$(brew --prefix openssl@3)/lib"
107
- CPATH="$(brew --prefix openssl@3)/include"`.
108
- - jemalloc's development package (`libjemalloc-dev`; `brew install jemalloc`)
109
- is optional: an app needs it only once its `spin.toml` says
110
- `allocator = "jemalloc"`, which `cybertrain new` writes commented out. It
111
- is worth turning on for production: 1.1 to 1.4 times the requests per second on
112
- examples/blog ([docs/benchmark.md](docs/benchmark.md)).
113
-
114
- `cybertrain doctor` checks these tools and headers and the Spinel install;
115
- when a required one is missing it also prints the install command for your
116
- platform.
117
-
118
- The framework has no separate test runner; it tests itself. `spin` comes
119
- with Spinel, and `cybertrain spin` runs the copy `cybertrain` installs
120
- without touching `PATH` (or put that copy on `PATH` with the `export
121
- PATH=...` line `cybertrain setup` prints, and call `spin` directly):
122
-
123
- ```sh
124
- cybertrain spin test
125
- ```
126
-
127
- compiles and runs every program under `test/` and diffs its output against a
128
- committed `test/<name>.rb.expected` snapshot — minitest/RSpec can't run under
129
- Spinel, since they find test methods by reflection and an ahead-of-time
130
- compiler has nothing to reflect on at run time. `cybertrain spin test
131
- --regen test/<name>.rb` rewrites a snapshot from CRuby's output; a few tests that
132
- touch SQLite through FFI take theirs from the compiled binary instead
133
- ([spikes/NOTES.md](spikes/NOTES.md), rule 23); `script/regen-snapshot
134
- test/<name>.rb` does that build-then-capture step for you and prints the diff stat.
135
-
136
- ## Installing the CLI
19
+ `cybertrain build` compiles your controllers, models, views and the framework
20
+ into **one native binary**: no Rack, no gems at runtime, no Ruby interpreter
21
+ in production. The Rails vocabulary (`before_action`, `Article.where(...)`,
22
+ `resources :articles`) comes from plain Ruby generated at build time instead
23
+ of runtime metaprogramming.
24
+
25
+ **Status:** pre-alpha. Routing (namespaces, scopes, formats and
26
+ `respond_to`), controllers with concerns, `helper_method`, strong
27
+ parameters, HTTP caching, cookies and an encrypted session, CSRF, a Content
28
+ Security Policy, `force_ssl` and host checks, ERB-style templates with
29
+ components, application helpers, the number, form and Turbo helpers and
30
+ per-controller layouts, digested assets with an import map, SQLite models
31
+ (decimal, json, date and datetime columns; declared associations with
32
+ `includes`; Rails' validators; model concerns; encrypted columns; bulk
33
+ inserts and typed aggregates), migrations, i18n (Rails' `t` and `l`,
34
+ dictionaries compiled in, English and Japanese bundled), and the
35
+ authentication generator (sign-in, sign-up or invite-only, password reset,
36
+ confirmation, lockout, remember me, an account page, admin users) and the
37
+ file-upload generator all work end to end. So do background jobs (job classes with retry
38
+ policies, cron schedules, a jobs page), a cache store, Turbo Streams over
39
+ Server-Sent Events or WebSocket, and the outbound clients: HTTP (streamed,
40
+ gzip and Server-Sent Events responses), SMTP (plain or HTML mail, sent now
41
+ or from a job) and WebSocket feeds, and the tests: Rails' test classes
42
+ (`ActiveSupport::TestCase`, `ActionDispatch::IntegrationTest`,
43
+ `ViewComponent::TestCase`) with `setup` / `teardown`, fixtures,
44
+ `travel_to`, `assert_select`, broadcast, mail and job assertions, all run
45
+ by `cybertrain test` as one compiled program, and the time helpers
46
+ (`5.minutes.ago`, `beginning_of_day`) in a configured time zone. [examples/blog](examples/blog) is the
47
+ Rails "Getting Started" blog built on it, in English and Japanese, with an
48
+ admin user guarding its `/admin` pages. Since 2026-11-05 porting an
49
+ existing Rails application is a goal, in five phases
50
+ ([design record](docs/design.md), D1); a SQLite database that a Rails
51
+ application created can be taken over as it is. Not built: mailer
52
+ classes (a mailer is a module whose methods return a `MailMessage` to
53
+ `deliver_later`), mail attachments, HTTP/2
54
+ ([what differs from Rails](https://saeki-mototsune.github.io/CyberTrain/guide/how-it-works.html#differences-from-rails)).
55
+
56
+ ## Quick start
57
+
58
+ Needs Ruby 3.2+, a C toolchain, `make`, `git`, `curl` and the SQLite, zlib and
59
+ OpenSSL headers ([details](https://saeki-mototsune.github.io/CyberTrain/guide/install.html#requirements); `cybertrain doctor`
60
+ checks them).
137
61
 
138
62
  ```sh
139
63
  gem install cybertrain
@@ -142,1944 +66,53 @@ cd blog
142
66
  cybertrain g scaffold article title:string body:text
143
67
  cybertrain db migrate
144
68
  cybertrain server # http://127.0.0.1:3000
145
- cybertrain build # production: dist/
146
- ```
147
-
148
- The `cybertrain` command is plain Ruby and the gem runs it under CRuby
149
- (3.2 or newer); neither the framework nor Spinel is in the gem. Its
150
- commands (`cybertrain help` prints them):
151
-
152
- - `cybertrain new NAME [--path DIR | --version V | --git URL [--ref R]]
153
- [--skip-spin]` creates an application in NAME, then runs `spin lock` and
154
- `spin run gen` in it, installing Spinel first when it is needed.
155
- - `cybertrain generate scaffold NAME field:type ...` (alias `g scaffold`)
156
- adds a resource to the application in the current directory.
157
- - `cybertrain generate authentication` adds email-and-password sign-in to
158
- it: users and sessions tables, `User` and `Session` models, the sign-in
159
- and sign-up controllers and views, the routes, a `before_action` in
160
- `ApplicationController` and a test (see "Add authentication").
161
- With `--invite-only` there is no sign-up page: users are created with
162
- `./NAME user EMAIL`, and the first one also at a `/setup` page that
163
- answers only to the server's own machine, with the one-time token its
164
- boot warning prints, while no user exists.
165
- - `cybertrain generate attachment MODEL FIELD [--any]` adds a file upload
166
- to a scaffolded model: five metadata columns, model methods and
167
- callbacks, a download action, the route, the form field and a test (see
168
- "Add a file upload"). A scaffold field `cover:attachment` does the same.
169
- - `cybertrain db migrate` generates, applies pending migrations, and
170
- generates again (`spin run gen; spin run db -- migrate; spin run gen`);
171
- `cybertrain migration` is kept as an alias.
172
- - `cybertrain db COMMAND...` runs any other database command after `spin
173
- run gen`: `status`, `rollback [N]` (which generates again afterwards),
174
- `schema:dump`, `create`.
175
- - `cybertrain server [PORT]` generates, then starts the development server
176
- (`spin run gen; spin run NAME`, or `spin run NAME -- PORT`; default port
177
- 3000).
178
- - `cybertrain build` builds NAME with `app/views/` embedded and assembles
179
- `dist/` (the binary, `public/`, `storage/`, `tmp/`).
180
- - `cybertrain spin ARGS...` runs `spin` with the pinned Spinel from any
181
- directory (`cybertrain spin test`, `cybertrain spin run gen -- --check`).
182
- - `cybertrain setup [--force]` installs Spinel `2026.10.27.7020` into
183
- `~/.cybertrain` unless a `spinel` of that release is already on `PATH`;
184
- `--force` rebuilds the copy under `~/.cybertrain`.
185
- - `cybertrain doctor` checks the C toolchain, the SQLite headers and the
186
- Spinel install.
187
- - `cybertrain version` prints the version.
188
-
189
- `db`, `server` and `build` run inside an application; NAME is the
190
- `[package] name` in its `spin.toml`.
191
-
192
- Every command that runs `spin` (`new`, `db`, `server`, `build`, `spin`) first makes
193
- sure a Spinel of the release in `Cybertrain::SPINEL_TAG` is available. A
194
- `spinel` on `PATH` (with a `spin`) whose `spinel --version` reports that
195
- release is used as it is; otherwise the copy `cybertrain` keeps under
196
- `~/.cybertrain/spinel/2026.10.27.7020/` is used; otherwise `cybertrain`
197
- builds that copy from the release tag of `Cybertrain::SPINEL_REPOSITORY`
198
- (`git clone`, `make deps`, `make -j`, `make install PREFIX=...`), once, in a
199
- few minutes. The build log is
200
- `~/.cybertrain/log/spinel-2026.10.27.7020-build.log` (its last lines are printed
201
- if a step fails); if a tool the build needs is missing, `cybertrain` says
202
- which and prints the install command for your platform before it starts. A
203
- `spinel` of another release on `PATH` is never used or touched:
204
- `cybertrain` keeps its own copy (and says so when it installs it).
205
-
206
- `cybertrain setup` does the same on request and, whenever its own copy is
207
- the one in use, prints the line that puts it on `PATH`, for running `spin`
208
- yourself (`~/.cybertrain/bin` holds `spinel` and `spin` links to the release
209
- in use, so the line survives a release bump; `cybertrain spin ...` needs no
210
- `PATH` at all):
211
-
212
- ```sh
213
- export PATH="$HOME/.cybertrain/bin:$PATH"
214
- ```
215
-
216
- Two environment variables override all this: `CYBERTRAIN_HOME` replaces
217
- `~/.cybertrain` (a path without spaces, quotes, `$` or `:`, which Spinel's
218
- Makefile and `PATH` cannot take), and `CYBERTRAIN_SPINEL_HOME=PREFIX` forces
219
- an existing install — `cybertrain` then uses the `spinel` and `spin` in `PREFIX/bin` (or
220
- in `PREFIX` itself, if `spinel` is directly in it) and stops with an error
221
- if they are not that release.
222
-
223
- `cybertrain new` writes the application with its `spin.toml` pointing at
224
- the release matching the CLI (`cybertrain = { git =
225
- "https://github.com/saeki-mototsune/cybertrain", ref = "v0.3.1" }`), makes
226
- sure Spinel is installed, then runs `spin lock` (spin fetches the framework
227
- into its cache, `~/.cache/spin/packages/`, and pins the commit in
228
- `spin.lock`) and `spin run gen`, much as `rails new` runs `bundle install`,
229
- and prints the next steps. Nothing of the framework is copied into the app;
230
- commit `spin.toml` and `spin.lock`. If Spinel or one of the two `spin`
231
- steps fails, the files stay, `new` exits with 1 and prints what to run once
232
- the cause is fixed (for Spinel: `cybertrain setup`, then `cybertrain spin
233
- lock && cybertrain spin run gen` in the new directory).
234
-
235
- `--skip-spin` only writes the files: no Spinel install, no `spin lock`, no
236
- `spin run gen`. `--path DIR` depends on a local checkout instead,
237
- `--git URL [--ref R]` on the framework at URL (at branch or tag R), and
238
- `--version V` on an index version once cybertrain is on a `spin-index`.
239
- Give only one of `--path`, `--version` and `--git`; `--ref` needs `--git`.
240
-
241
- Working on cybertrain itself, run the CLI from the checkout (`spin install`
242
- builds `bin/cybertrain.rb` into `~/.local/bin/cybertrain`; `ruby -I.
243
- bin/cybertrain.rb` also works) and create apps against it with `cybertrain
244
- new NAME --path ~/src/cybertrain` (wherever the checkout is).
245
-
246
- ## Walkthrough: building a blog
247
-
248
- Mirrors Rails' own [Getting
249
- Started](https://guides.rubyonrails.org/getting_started.html): a blog with
250
- articles and comments.
251
-
252
- ### Create the application
253
-
254
- ```sh
255
- cybertrain new blog
256
- cd blog
257
- cybertrain server # works right away: `new` already locked the framework and ran `spin run gen`
258
- ```
259
-
260
- ```
261
- blog/
262
- spin.toml config/app.rb config/routes.rb
263
- db/schema.rb db/migrate/
264
- app/controllers/application_controller.rb app/models/ app/views/ app/helpers/
265
- public/ bin/blog.rb bin/gen.rb bin/db.rb
266
- gen/ storage/ tmp/ test/
267
- ```
268
-
269
- `bin/blog.rb` is the application's one entry point, the program `spin build
270
- blog` compiles: `./blog` serves, `./blog migrate` and `./blog db status`
271
- manage the database. `bin/gen.rb` runs the generator (`spin run gen`), and
272
- `bin/db.rb` runs the migrator during development (`cybertrain db migrate`
273
- uses it, since `bin/blog.rb` cannot compile until the first migration has
274
- generated `gen/models/`). `gen/` already holds `views.rb`, the committed,
275
- empty table of embedded views (see "How it works").
276
-
277
- Operational commands of your own (create a user, prune old rows, take a
278
- backup) go on the same binary: register them in `bin/blog.rb` before
279
- `Main.run`.
280
-
281
- ```ruby
282
- Cybertrain::Main.command("prune", "delete comments older than DAYS") do |argv|
283
- days = argv.empty? ? 30 : argv[0].to_i
284
- Cybertrain::DB.with do |c|
285
- c.execute("DELETE FROM comments WHERE created_at < datetime('now', ?)", ["-#{days} days"])
286
- end
287
- 0 # the exit status
288
- end
289
- ```
290
-
291
- `./blog prune 90` then runs the block with the arguments after the name
292
- (`["90"]`), with `Cybertrain::DB` already connected to the configured
293
- database (a pool of one connection), and exits with the block's value; a
294
- `Cybertrain::DB::Error` is printed as `prune failed: ...` and exits 1, and
295
- `./blog help` lists the command. The block takes `argv` explicitly, as a
296
- `before_action` block takes the controller (Spinel has no `instance_eval`).
297
- A built-in word (`server`, `migrate`, `db`, `help`), a number (the port
298
- argument) or a name already registered raises `ArgumentError`. The server,
299
- views and session secret are not set up for a command.
300
-
301
- Apps created before `cybertrain build` existed have `bin/server.rb` and
302
- no `bin/<name>.rb`: copy `bin/<name>.rb` from a fresh `cybertrain new` (with
303
- the same NAME), delete `bin/server.rb`, add `/dist/` to `.gitignore` (the old
304
- template did not ignore it) and run `cybertrain spin run gen`.
305
-
306
- `ApplicationController` starts with the one thing every controller inherits:
307
-
308
- ```ruby
309
- class ApplicationController < Cybertrain::Controller
310
- rescue_from Cybertrain::RecordNotFound, with: :record_not_found
311
-
312
- private
313
-
314
- def record_not_found
315
- render plain: "Not Found", status: :not_found
316
- end
317
- end
318
- ```
319
-
320
- ### Scaffold an article
321
-
322
- ```sh
323
- cybertrain generate scaffold article title:string body:text
324
- ```
325
-
326
- writes a migration, a model, a controller and five views, and inserts
327
- `resources :articles` into `config/routes.rb`. Point root at it and, as
328
- Rails' guide does, add a length validation by hand:
329
-
330
- ```ruby
331
- # config/routes.rb
332
- Cybertrain::Routes.draw do
333
- root "articles#index"
334
- resources :articles
335
- end
336
- ```
337
-
338
- ```ruby
339
- # app/models/article.rb -- only the first string field gets a default
340
- # presence validation; add anything past that yourself.
341
- class Article
342
- validates :title, presence: true
343
- validates :body, presence: true, length: { minimum: 10 }
344
- end
345
- ```
346
-
347
- The controller is full CRUD, Rails-shaped down to strong params and
348
- flash-then-redirect, except for two things Spinel forces everywhere: `new` is
349
- written `new_action` (a method literally named `new` would shadow
350
- `ArticlesController.new(ctx)`), and every action/callback is called by its
351
- literal name, never `send`:
352
-
353
- ```ruby
354
- # app/controllers/articles_controller.rb (index/show/edit are as plain as
355
- # Rails' own scaffold; update mirrors create)
356
- class ArticlesController < ApplicationController
357
- before_action :set_article, only: [:show, :edit, :update, :destroy]
358
-
359
- def new_action # a `new` method would shadow ArticlesController.new
360
- @article = Article.new
361
- end
362
-
363
- def create
364
- @article = Article.new(article_params)
365
- if @article.save
366
- flash[:notice] = "Article was successfully created."
367
- redirect_to article_path(@article), status: :see_other
368
- else
369
- render :new, status: :unprocessable_entity
370
- end
371
- end
372
-
373
- private
374
-
375
- def set_article
376
- @article = Article.find(params[:id])
377
- end
378
-
379
- def article_params
380
- params.require(:article).permit(:title, :body)
381
- end
382
- end
383
- ```
384
-
385
- ### Generate, migrate, run
386
-
387
- ```sh
388
- cybertrain db migrate # spin run gen; spin run db -- migrate; spin run gen
389
- cybertrain server # http://127.0.0.1:3000; rebuilds on Ruby edits, views reload without a rebuild
390
- ```
391
-
392
- `spin run gen` reads `db/schema.rb` and `config/routes.rb`, scans `app/` and
393
- writes `gen/`; `spin run db -- migrate` applies `db/migrate/*.rb` and
394
- rewrites `db/schema.rb`.
395
-
396
- Re-run `cybertrain spin run gen` (and commit `gen/`) after touching the
397
- schema, routes, or a controller/model's callbacks and ivars — the dev server does this for
398
- you automatically (see "How it works").
399
-
400
- ### Add comments, nested under articles
401
-
402
- ```sh
403
- cybertrain generate scaffold comment commenter:string body:text article:references
404
- cybertrain db migrate # the scaffold wrote a migration for the comments table
405
- ```
406
-
407
- `article:references` adds an `article_id` column, index and foreign key —
408
- enough for `Article`/`Comment` to get their association; there's no
409
- `has_many`/`belongs_to` to write, `spin run gen` reads it off the foreign
410
- key. This also inserts an independent `resources :comments` line; Rails'
411
- guide nests comments under articles instead, so edit `config/routes.rb`:
412
-
413
- ```ruby
414
- Cybertrain::Routes.draw do
415
- root "articles#index"
416
-
417
- # A nested block takes the mapper explicitly (Spinel's instance_eval
418
- # trampoline only rewires `self` for the outermost block): `articles.resources`.
419
- resources :articles do |articles|
420
- articles.resources :comments, only: [:create, :destroy]
421
- end
422
- end
423
- ```
424
-
425
- Trim `CommentsController` to the two actions the nested routes call,
426
- resolving the parent from the URL instead of `:id`:
427
-
428
- ```ruby
429
- class CommentsController < ApplicationController
430
- before_action :set_article
431
-
432
- def create
433
- @comment = Comment.new(comment_params)
434
- @comment.article_id = @article.id
435
- if @comment.save
436
- flash[:notice] = "Comment was successfully created."
437
- else
438
- flash[:alert] = "Comment could not be saved: #{@comment.errors.full_messages.join(", ")}"
439
- end
440
- redirect_to article_path(@article), status: :see_other
441
- end
442
-
443
- def destroy
444
- @comment = Comment.where(article_id: @article.id).find(params[:id])
445
- @comment.destroy
446
- flash[:notice] = "Comment was successfully destroyed."
447
- redirect_to article_path(@article), status: :see_other
448
- end
449
-
450
- private
451
-
452
- def set_article
453
- @article = Article.find(params[:article_id])
454
- end
455
-
456
- def comment_params
457
- params.require(:comment).permit(:commenter, :body)
458
- end
459
- end
460
- ```
461
-
462
- Delete the now-unreachable `index`/`show`/`new`/`edit` comment views, give
463
- `ArticlesController#show` a blank `@comment` to bind against, and add a list
464
- and a form to the article page:
465
-
466
- ```erb
467
- <%# app/views/articles/show.html.erb %>
468
- <% @article.comments.each do |comment| %>
469
- <%= render "comments/comment", comment: comment %>
470
- <% end %>
471
- <%= render "comments/form" %>
472
-
473
- <%# app/views/comments/_form.html.erb %>
474
- <%= form_with(model: [@article, @comment]) do |f| %>
475
- <%= f.label :commenter %> <%= f.text_field :commenter %>
476
- <%= f.label :body %> <%= f.text_area :body %>
477
- <%= f.submit %>
478
- <% end %>
479
- ```
480
-
481
- `comments/_comment.html.erb` destroys with `button_to "Destroy comment",
482
- [@article, comment], method: :delete`. Both that and `form_with(model:
483
- [@article, @comment])` resolve through the same `[parent, child]` rule: a new
484
- child routes to the nested collection (`article_comments_path`), a persisted
485
- one to the nested member (`article_comment_path`) — generated into
486
- `gen/routes.rb` from the nested `resources` block above.
487
-
488
- ### Add authentication
489
-
490
- ```sh
491
- cybertrain generate authentication
492
- cybertrain db migrate
493
- cybertrain server # then open http://127.0.0.1:3000/sign_up
494
- ```
495
-
496
- Like Rails 8's `bin/rails generate authentication`, this writes code the
497
- app owns rather than installing an engine: migrations for `users`
498
- (`email_address`, `password_digest`, a unique index) and `sessions` (one
499
- row per signed-in browser, with `ip_address` and `user_agent`), the `User`
500
- and `Session` models, `SessionsController` (`/session/new`, `POST
501
- /session`, `DELETE /session`), `RegistrationsController` (`/sign_up`,
502
- because there is no console to create the first user in; delete it for an
503
- invitation-only app), their views, a `sessions/_nav` partial rendered at
504
- the top of the layout, and `test/authentication.rb` with its
505
- `test/support/app_test.rb`. It also inserts a block after the class line of
506
- `ApplicationController`:
507
-
508
- ```ruby
509
- class ApplicationController < Cybertrain::Controller
510
- before_action :resume_session # sets @current_user and @current_session
511
- before_action :require_authentication # 303 to /session/new unless signed in
512
-
513
- def self.allow_unauthenticated_access(only: [], except: [])
514
- skip_before_action(:require_authentication, only: only, except: except)
515
- end
516
- # authenticated?, start_new_session_for(user), terminate_session,
517
- # after_authentication_url ...
518
- ```
519
-
520
- so every action needs a signed-in user unless its controller says
521
- `allow_unauthenticated_access only: [:new, :create]`, as the two generated
522
- controllers do. Templates read `@current_user` (`<% if @current_user %>`).
523
- Passwords are hashed by `Cybertrain::Password` (PBKDF2-HMAC-SHA256, 100,000
524
- iterations by default, about 0.2 s compiled; raise it with
525
- `Cybertrain::Password.iterations = 600_000` in `config/app.rb`, and
526
- `User#authenticate` rehashes each user as they sign in). The `User` model
527
- spells out what `has_secure_password` would define (`password=`,
528
- `authenticate`, `User.authenticate_by(email_address:, password:)`), since
529
- Spinel cannot define methods at run time; edit it like any other model.
530
- Password reset is not generated (no mailers). Running the generator again
531
- writes nothing. The design, and why not a package: [docs/design.md](docs/design.md) D15.
532
-
533
- Sign-in is protected from password guessing the way Rails 8's generator
534
- does it: `SessionsController` declares `rate_limit to: 10, within: 3 * 60,
535
- only: [:create], redirect_to: "/session/new", alert: "Try again later."`,
536
- so an address that has tried ten times in three minutes goes back to the
537
- form until the window closes. Behind a reverse proxy on the same machine,
538
- set `CYBERTRAIN_TRUSTED_PROXY=1`, or every visitor shares the proxy's
539
- address (see [Rate limiting](#rate-limiting-and-the-client-address)); behind
540
- kamal-proxy, the generated `config/deploy.yml` already sets
541
- `CYBERTRAIN_TRUSTED_PROXIES` to the Docker network.
542
- `AppTest.sign_in` forgets the counts first, since every test client comes
543
- from 127.0.0.1.
544
-
545
- #### An invitation-only app: `--invite-only`
546
-
547
- The sign-up page lets anyone who reaches the server register, which is the
548
- wrong default for an internal tool: whoever gets there before the admin
549
- does has an account. `--invite-only` writes no `RegistrationsController`
550
- and no sign-up link; the first user comes from the command line or from a
551
- page only the server's own machine can open, with a link the server prints
552
- when it starts:
553
-
554
- ```sh
555
- cybertrain generate authentication --invite-only
556
- cybertrain db migrate
557
- cybertrain server # then open the http://127.0.0.1:3000/setup?token=... link it prints, on the same machine
558
- ```
559
-
560
- In production, from the directory the binary runs in:
561
-
562
- ```sh
563
- ./shop user ann@example.com # reads the password from stdin (one line)
564
- printf '%s' "$PASSWORD" | ./shop user ann@example.com
565
- ./shop user ann@example.com 's3cr3t-s3cr3t' # or as an argument (visible in ps)
566
- ```
567
-
568
- - `app/commands/user_command.rb` registers the `user` command with
569
- `Cybertrain::Main.command` (so `./shop help` lists it) and is required
570
- from `bin/shop.rb`. `EMAIL [PASSWORD]` creates the user, or sets the
571
- password of the one with that address (which signs that user's browsers
572
- out, as any password change does); it prints `created ADDRESS` or
573
- `updated ADDRESS` and exits 0. Without `PASSWORD` it reads one line of
574
- standard input (prompting when that is a terminal; the typing is echoed)
575
- and drops the trailing newline, so the password stays out of `ps` and
576
- the shell history. An empty password, a record that does not validate
577
- (a password under 8 characters) or bad arguments print an error or the
578
- usage line and exit 1.
579
- - `SetupController` (`GET`/`POST /setup`, `app/views/setup/new.html.erb`)
580
- creates the first user and signs them in. It answers only while no user
581
- exists, only with the one-time token (below), and only to a connection
582
- whose socket peer is the loopback address (`request.loopback?`:
583
- `127.0.0.0/8` or `::1`) and that carries no `X-Forwarded-For` header (a
584
- proxy on the same host makes every request a loopback one, and it adds
585
- that header). A header can close the page, never open it. Everyone else,
586
- and a missing or wrong token, gets a 404.
587
- - The token: `SetupController::SETUP_TOKEN`, 32 random hex characters
588
- (`Cybertrain::Crypto.random_token(16)`) made when the program starts and
589
- kept until it exits, so a restart makes a new one. `GET /setup` takes it
590
- as `?token=`, the form carries it back in a hidden field, and both
591
- compare it in constant time (`Cybertrain::Crypto.secure_compare`). It is
592
- what keeps the page closed when the address check cannot: a proxy on the
593
- same host that adds no `X-Forwarded-For` (nginx without
594
- `proxy_set_header X-Forwarded-For ...`) makes every client look like
595
- loopback, and only someone who can read the server's log has the token.
596
- The request log's `Started GET "/setup?token=..."` line writes it to the
597
- same log; it opens nothing once the user exists.
598
- - For a first run behind a proxy on another host, `SHOP_SETUP_OPEN=1`
599
- (declared in `config/app.rb` with `c.setting_bool :setup_open`) lifts the
600
- address check, still only while no user exists and still only with the
601
- token. Unset it once the user is there.
602
- - While no user exists, the server logs `[WARN] no user exists yet: run
603
- ./shop user EMAIL, or open /setup?token=<token> from this machine` before
604
- it boots (`UserCommand.warn_if_no_user`, called from `bin/shop.rb` when
605
- `Cybertrain::Main.server?(ARGV)`; it connects for that one question and
606
- says nothing when the users table is not migrated yet). That line is the
607
- only place the token is printed.
608
-
609
- The generated `test/authentication.rb` covers both: `/setup` with and
610
- without the token (`SetupController.setup_token`) or with a wrong one, from
611
- another address (`client.peer_address = "203.0.113.7"`), through a proxy,
612
- once a user exists and with the switch on, and the `user` command run in-process
613
- through `Cybertrain::Main.run`. All of it is application code, yours to
614
- change (docs/design.md D29); the framework only provides
615
- `request.loopback?`, `Main.command`, `Main.server?`,
616
- `Crypto.random_token` / `Crypto.secure_compare` and the test client's
617
- `peer_address`.
618
-
619
- In the blog, open the pages anyone may read again, as Rails' guide does, and
620
- hide what a visitor cannot use (`resume_session` still runs on those pages,
621
- so `@current_user` is set):
622
-
623
- ```ruby
624
- # app/controllers/articles_controller.rb
625
- class ArticlesController < ApplicationController
626
- allow_unauthenticated_access only: [:index, :show]
627
- before_action :set_article, only: [:show, :edit, :update, :destroy]
628
- ```
629
-
630
- ```erb
631
- <%# app/views/articles/show.html.erb (excerpt) %>
632
- <% if @current_user %>
633
- <%= button_to "Destroy this article", article_path(@article), method: :delete %>
634
- <% end %>
635
-
636
- <% if @current_user %>
637
- <h2>Add a comment:</h2>
638
- <%= render "comments/form" %>
639
- <% end %>
640
- ```
641
-
642
- A test that expects a form, submits one, or reaches an action other than
643
- `index` and `show` now meets the sign-in page: sign the client in first
644
- with the helpers in `test/support/app_test.rb`.
645
-
646
- ```ruby
647
- require_relative "support/app_test" # written by generate authentication: APP, AppTest
648
-
649
- test "a signed-in user creates an article" do
650
- client = Cybertrain::Test::Client.new(APP)
651
- AppTest.user("ann@example.com", "s3cr3t-s3cr3t")
652
- AppTest.sign_in(client, "ann@example.com", "s3cr3t-s3cr3t")
653
- client.get("/articles/new")
654
- res = client.post("/articles", { "authenticity_token" => AppTest.form_token(client),
655
- "article[title]" => "Hello Rails",
656
- "article[body]" => "I am on Rails! This is my first article." })
657
- assert_redirected_to res, "/articles/#{Article.last.id}"
658
- end
659
- ```
660
-
661
- `examples/blog` itself stops before this step.
662
-
663
- ## Authentication: what the framework provides
664
-
665
- The generator above writes the flow; the pieces it is built from are
666
- framework API, usable without it (for an API token, an admin password, a
667
- hand-written sign-in):
668
-
669
- - **`Cybertrain::Password`**: `hash(plain)` makes a salted
670
- PBKDF2-HMAC-SHA256 digest that records its own parameters
671
- (`$pbkdf2-sha256$i=100000$salt$key`), `verify?(plain, digest)` checks one
672
- in constant time and never raises on a malformed digest, and
673
- `needs_rehash?(digest)` says when one is behind the current
674
- `Password.iterations`. For a sign-in action:
675
- - `verify_serialized(plain, digest)` is `verify?` one at a time per
676
- process (a Mutex). A hash takes about 0.2 s of CPU; a burst of attempts
677
- side by side would hold every worker, through this method they queue and
678
- the other workers keep serving pages.
679
- - `dummy_digest` is a digest of a random password, made once per process
680
- at the current cost. Verify against it when the address is unknown, so
681
- the response time does not tell which addresses have accounts:
682
- `Cybertrain::Password.verify_serialized(password, user.nil? ? Cybertrain::Password.dummy_digest : user.password_digest)`.
683
- - `session_key(digest)` is an HMAC of the password digest under
684
- `secret_key_base`; keep it in the session at sign-in and check it with
685
- `session_key_matches?(session[:password_key], user.password_digest)` on
686
- each request. A password change makes a new digest, so every session
687
- started before it is signed out.
688
- - `iterations` is `Password::TEST_ITERATIONS` (1,000) when
689
- `Cybertrain.config.test?`, so test suites do not pay the production cost
690
- per user; digests record their count, so nothing else changes.
691
- - **The session** (`session[:key]`, a signed cookie): `clear` empties it
692
- (CSRF token included) on sign-out, and `config.session_absolute_max_age`
693
- (two weeks; `0` turns it off) bounds a session's whole life however often
694
- it is used: the cookie records when it was first written, a cookie older
695
- than that loads as an empty session, so a copied cookie cannot be
696
- replayed forever. `session.restart_lifetime!` starts the count again (at
697
- sign-in).
698
- - **`Controller#local_path?(text)` and `safe_redirect_path(text, fallback)`**:
699
- a path that came from the request (a `return_to` parameter, a remembered
700
- page) is redirected to only when it starts with `/` but not `//` or `/\`,
701
- holds printable ASCII only and is at most 2,048 bytes:
702
- `redirect_to safe_redirect_path(params[:return_to], "/"), status: :see_other`.
703
- - **`skip_before_action(name, only:, except:)`** on a controller class drops
704
- an inherited `before_action` for some actions; the generated
705
- `allow_unauthenticated_access` is a one-line wrapper around it.
706
- - **`cybertrain generate authentication`** writes, as code the app owns: the
707
- `users` and `sessions` migrations and models, `SessionsController`,
708
- `RegistrationsController`, their views and nav partial, the
709
- `ApplicationController` block (`resume_session`, `require_authentication`,
710
- `start_new_session_for`, `terminate_session`, `after_authentication_url`)
711
- and `test/authentication.rb`. That code uses everything in this list:
712
- `User.authenticate_by` verifies with `verify_serialized` and the dummy
713
- digest, `start_new_session_for` stores the password key and restarts the
714
- lifetime, `resume_session` ends a Session row whose key no longer matches
715
- or that is older than `session_absolute_max_age`, and
716
- `after_authentication_url` is `safe_redirect_path`. To keep the browser
717
- that changed a password signed in, store the new key after saving:
718
- `session[:password_key] = Cybertrain::Password.session_key(user.password_digest)`.
719
- The design: [docs/design.md](docs/design.md) D25.
720
-
721
- ### Add a file upload
722
-
723
- ```sh
724
- cybertrain generate attachment article cover # or: generate scaffold article title:string cover:attachment
725
- cybertrain db migrate
726
- ```
727
-
728
- The same split as authentication: the framework parses
729
- `multipart/form-data` and stores bytes, and the generator writes the rest
730
- as code the app owns. It writes a migration adding five columns to
731
- `articles` (`cover_key`, `cover_filename`, `cover_content_type`,
732
- `cover_byte_size`, `cover_checksum`; the file itself is not in SQLite) and
733
- a block in `Article` (`cover_file=`, `cover_attached?`, `cover_url`, a
734
- validation and save/destroy callbacks). In `ArticlesController` it adds a
735
- `cover` action behind `before_action :set_article` and one line in
736
- `create` and `update`:
737
-
738
- ```ruby
739
- @article.cover_file = params.nested(:article).upload(:cover)
740
- ```
741
-
742
- It also adds `get "/articles/:id/cover"` to `config/routes.rb`, and to the
743
- form `form_with(model: article, multipart: true)` with `f.file_field :cover`.
744
- The show page gets an `<img>`, and `test/attachment_article_cover.rb` tests
745
- the whole flow.
746
-
747
- What is accepted and how it is served:
748
-
749
- - **Types.** By default only PNG, JPEG, GIF and WebP up to 2 MB are
750
- accepted (`COVER_TYPES` / `COVER_MAX_BYTES` in the model; `--any` accepts
751
- any type). The type is detected from the file's first bytes, never taken
752
- from the browser.
753
- - **Storage.** The bytes go to `storage/uploads/<2 hex>/<32 hex key>`
754
- (`Cybertrain::BlobStore`, mode 0600). Replacing a file deletes the old
755
- one, and destroying the record deletes its file.
756
- - **Serving.** The `cover` action serves files with `send_blob`, so the
757
- controller's `before_action`s (sign-in, scoping) apply to downloads too.
758
- The four image types are served inline; anything else is a download
759
- (`application/octet-stream`). Like every response, they carry
760
- `X-Content-Type-Options: nosniff` and `X-Frame-Options: SAMEORIGIN` (the
761
- framework-wide defaults, below), and `send_blob` adds
762
- `Content-Security-Policy: default-src 'none'; sandbox`. Never put uploads
763
- under `public/`. The download is exactly as public as show: in the blog
764
- above, where `allow_unauthenticated_access only: [:index, :show]` lets
765
- visitors read articles, the generator adds `:cover` to that list, so the
766
- image on a public page loads (only when the download loads the record
767
- through the same `before_action :set_article` as show, and show checks
768
- nothing more). The generator checks inherited callbacks too. If a policy
769
- applies only to show, show contains inline checks, or the policy cannot
770
- be read safely, generation stops before writing any files. Put authorization
771
- in an explicit callback shared by show and the download, for example
772
- `before_action :authorize_article, only: [:show, :cover]`, and load both
773
- through `before_action :set_article`, then rerun the generator. Keeping a
774
- download behind sign-in alone does not preserve record or role permissions.
775
- Where show needs a sign-in, so does the download.
776
- - **Params and CSRF.** `params.nested(:article).upload(:cover)` is a
777
- `Cybertrain::UploadedFile` (`original_filename`, `byte_size`,
778
- `detected_content_type`, `body_bytes`, `sha256_hex`), or nil when no
779
- file was chosen. `permit` never returns files. The CSRF token and
780
- `_method` are read from multipart bodies like any form.
781
- - **Limits.** Three config keys bound uploads: `max_upload_bytes`,
782
- `max_multipart_parts` and `upload_root` (see "Configuration"). A broken
783
- or oversize body is a 400.
784
- - **Not built yet:** several files per field, S3 or other remote storage,
785
- image variants, and streaming large bodies to disk (the whole body is
786
- read into memory, up to `max_body_bytes`).
787
-
788
- Back up `storage/uploads/` together with the SQLite file. The design, and
789
- why it is not a separate library: [docs/design.md](docs/design.md) D16.
790
-
791
- ### Build for deployment
792
-
793
- ```sh
794
- cybertrain build
795
- ```
796
-
797
- writes `dist/`: `dist/blog`, the app compiled with every `app/views/`
798
- template embedded (rendered from memory, error messages still say
799
- `articles/show.html.erb:12`), `dist/public/`, and empty `dist/storage/` and
800
- `dist/tmp/`. Copy `dist/` to a machine with the same OS and CPU (the binary
801
- links the system SQLite) and run `./blog migrate && ./blog`; a built binary
802
- is production by default (`CYBERTRAIN_ENV` overrides). Set
803
- `CYBERTRAIN_SECRET_KEY_BASE`. Static files can also be served by the reverse
804
- proxy from `dist/public/` — see [docs/deploy.en.md](docs/deploy.en.md).
805
-
806
- A running `cybertrain server` skips its rebuilds while `cybertrain build` is
807
- in progress (the build holds `tmp/cybertrain-build.lock`, since both write
808
- `build/bin/blog`), logs the skip, and rebuilds as usual on the next save
809
- once the build has finished. The lock records the build's PID; one left
810
- behind by a killed build is ignored and removed.
811
-
812
- ## How it works
813
-
814
- **Build-time code generation.** `spin run gen` (`bin/gen.rb`) executes
815
- `db/schema.rb`'s `create_table` DSL and `config/routes.rb`'s `Routes.draw`
816
- DSL to get table and route *data*, and separately does a **lexical scan**
817
- (regexes, not a parser — Spinel exposes none at run time) of
818
- `app/controllers/**/*.rb` and `app/models/**/*.rb` for `@ivar =`
819
- assignments, `before_action`/`after_action`/`rescue_from ... with:` names and
820
- zero-argument `def`s. From that it writes plain, readable Ruby under `gen/`:
821
- `gen/models/<name>.rb` (accessors, casts, finders, FK-derived associations),
822
- `gen/routes.rb` (route table, dispatcher, `*_path`/`*_url` helpers),
823
- `gen/controllers.rb` (`view_assigns` and `run_callback` per controller),
824
- `gen/migrations.rb`, `gen/views.rb` (the embedded views table, empty unless
825
- building for deployment) and `gen/app.rb` (the `require_relative` manifest).
826
- `gen/` is committed, not gitignored — `spin build`/`spin test` have no hook
827
- to generate first, so the checked-in output has to already be what compiles.
828
- CI re-runs `spin run gen` and fails on a diff, so stale generated code can't
829
- merge.
830
-
831
- **Views are interpreted, not compiled.** `app/views/**/*.html.erb` files look
832
- like Rails ERB but are parsed into an AST at request time, then walked by a
833
- tree-walking interpreter. Where the source comes from depends on the mode:
834
- in development (and test) the engine reads `app/views/` from disk, and in
835
- development re-parses a template when it changes, so view edits need no rebuild.
836
- `cybertrain build` runs `spin run gen -- --embed-views`, which writes every
837
- template's source into `gen/views.rb` as a string table for that one `spin
838
- build`, then runs `spin run gen` again to restore the empty table that is
839
- committed; the production binary renders only that table (parsing each
840
- template once) and refuses to boot when it is empty (`error: views are not
841
- embedded in this binary`). Spinel has no `eval`, so this is a closed grammar (literals, `@ivar`/local lookups, calls
842
- through fixed per-type dispatch tables, `if`/`unless`,
843
- `each`/`each_with_index`/one helper block), not real Ruby. Full grammar,
844
- helpers and gaps: [docs/template-language.md](docs/template-language.md).
845
-
846
- **The HTTP server.** `Cybertrain::Server` is plain-Ruby HTTP/1.1:
847
- `TCPServer#accept` plus one Spinel green thread per connection (M:N
848
- scheduled, no GVL), typed `Request`/`Response`, no Rack layer. It calls an `HttpHandler` (`call(request) -> Response`); `ContextHandler` joins it to the middleware stack, so the HTTP layer (`cybertrain/http/`, plus `Params` and `Logger`) does not depend on the rest of the framework; `script/check-http-layer` enforces it in CI. TLS/HTTP/2 are
849
- left to a reverse proxy. `SPINEL_WORKERS` defaults to `1` — spikes found that
850
- faster and stall-free for this I/O-bound shape at 100 concurrent connections.
851
-
852
- **SQLite via FFI.** Models talk to SQLite through Spinel's
853
- `ffi_func`/`ffi_lib` directly — no C extension, no gem. A pool (`SizedQueue`,
854
- 4 connections by default) hands out connections with WAL, a busy timeout of
855
- about five seconds (a busy handler retrying every 100 µs, where SQLite's
856
- `busy_timeout` sleeps a millisecond or more), up to 64 cached prepared
857
- statements and foreign keys on; every query is bound, never interpolated. It's the only
858
- adapter today (PostgreSQL via `libpq` FFI is noted as possible future work).
859
-
860
- **Transactions.** `Cybertrain::DB.transaction { |c| ... }` (or
861
- `Article.transaction`, the same thing under the Rails name) checks a
862
- connection out, runs `BEGIN`, yields the connection, `COMMIT`s when the block
863
- returns and rolls back and re-raises when it raises. `immediate: true` runs
864
- `BEGIN IMMEDIATE`, which takes SQLite's write lock at the start: use it for a
865
- transaction that reads and then writes, so a concurrent writer makes it wait
866
- at `BEGIN` rather than fail with "database is locked" at its first write.
867
- While the block runs, everything on that thread that goes through the pool
868
- (`save`, `update`, `destroy`, Relation queries, `DB.with`) uses the
869
- transaction's connection, so it is part of the transaction:
870
-
871
- ```ruby
872
- Article.transaction(immediate: true) do |c|
873
- article.save!
874
- c.execute("UPDATE authors SET articles_count = articles_count + 1 WHERE id = ?", [article.author_id])
875
- end
69
+ cybertrain build # production: one binary in dist/
876
70
  ```
877
71
 
878
- A `DB.transaction` nested in the block joins the outer one (only the
879
- outermost commits; a raise anywhere rolls everything back; no savepoints).
880
- Code that must keep its `rescue` in a plain method uses the block-free form,
881
- `c.begin!(immediate: true)` ... `c.commit!`, with `c.rollback!` in the rescue.
882
- It returns nil rather than the block's value, and a `DB.transaction` called
883
- inside a `DB.with` block (which is not a transaction) gets a second connection.
72
+ The binary runs every role a deployment needs: `./blog server` for HTTP and
73
+ the realtime endpoint (`turbo_stream_from @article` on a page,
74
+ `Cybertrain::Stream.broadcast_append_to` from an action or a job),
75
+ `./blog worker` for jobs and scheduled tasks (several may share the
76
+ database; jobs are delivered at least once), `./blog feeds` for WebSocket
77
+ subscriptions, and `./blog console` for a prompt over the models.
884
78
 
885
- **The development loop.** `cybertrain server` (`spin run gen`, then `spin run NAME`) in
886
- `development` (the default `CYBERTRAIN_ENV`, except in a binary from
887
- `cybertrain build`) also polls `app/**/*.rb`, `config/**/*.rb`, `db/schema.rb`
888
- and `gen/**/*.rb` every half second (views are excluded — the engine reloads
889
- those itself — and so is `gen/views.rb`, empty in development). A change runs `spin run gen && spin build NAME -- --incremental` in the
890
- background (Spinel's incremental build, which recompiles only the C units the
891
- edit changed; a Spinel release without `--incremental` rejects the flag, and the
892
- loop then builds without it); success requests a restart (the same flag an external `kill -HUP`
893
- sets): a monitor thread stops the listener and drains open requests, then the
894
- main thread `execv`s the new binary on the same port and PID, invisible to a
895
- client mid-session. A failed build keeps serving the old binary and banners the
896
- compiler output on every HTML response. None of this loads in production.
79
+ Or try it with nothing installed:
80
+ [open the playground in GitHub Codespaces](https://codespaces.new/saeki-mototsune/CyberTrain?quickstart=1)
81
+ ([what the playground holds](https://saeki-mototsune.github.io/CyberTrain/playground.html)).
897
82
 
898
- ## Migrations
83
+ On one CPU core, with 16 connections, the blog serves 2.2 to 8.8 times the requests per second of
84
+ the same app in Rails 8.1, in about a tenth of the memory
85
+ ([numbers](https://saeki-mototsune.github.io/CyberTrain/index.html#performance)).
899
86
 
900
- A migration in `db/migrate/<version>_<name>.rb` is a `Cybertrain::Migration::Base`
901
- subclass. Its `change` records schema operations (`create_table`,
902
- `add_column`, `add_index`, `rename_column`, `add_reference`, `remove_column`,
903
- `remove_index`, `drop_table`); `cybertrain db migrate` applies each pending
904
- version in its own transaction and rewrites `db/schema.rb` from the database,
905
- and `cybertrain db rollback [N]` runs the inverse of `change`. When `change`
906
- holds an operation with no automatic inverse, write `down` (and `up`).
87
+ ## Documentation
907
88
 
908
- A schema change that needs a data step to be safe uses `execute(sql, binds =
909
- [])`. It runs at its place in the migration, on the migrator's own
910
- connection and in the same transaction as the version's schema operations,
911
- so the column added before it exists when it runs, and a failure undoes the
912
- whole version:
913
-
914
- ```ruby
915
- class AddRetentionDaysToBackups < Cybertrain::Migration::Base
916
- def change
917
- add_column "backups", "retention_days", :integer, null: false, default: "90"
918
- execute "UPDATE backups SET retention_days = ?", [0] # existing rows keep "forever"
919
- end
920
-
921
- def down
922
- remove_column "backups", "retention_days"
923
- end
924
- end
925
- ```
926
-
927
- Without binds the SQL may be several `;`-separated statements; with binds
928
- it is one statement with `?` placeholders. `execute` has no inverse: rolling
929
- back a `change` that executes SQL, with no `down` of its own, stops with
930
- `version N is irreversible: it executes SQL in change; write down` before
931
- anything is reverted. A data statement never appears in `db/schema.rb`,
932
- which is dumped from the database. Recording a migration runs nothing, so
933
- the statement runs exactly once per `migrate` of its version.
934
-
935
- ## Querying beyond `where`
936
-
937
- The generated relations cover `where` / `where_sql` / `order` / `limit` /
938
- `offset` / `count` / `exists?` / `delete_all`. For the counters, statistics
939
- and search boxes most applications also need, there are typed additions
940
- (issue #74, `docs/design.md` D22) rather than a query builder; each takes
941
- SQL column names, checked and quoted (anything else is an `ArgumentError`),
942
- and binds every value:
943
-
944
- ```ruby
945
- # INSERT ... ON CONFLICT (project_id, key) DO UPDATE SET ...; returns rows written
946
- Stat.upsert({ project_id: project.id, key: path, calls: 1, max_ms: ms, seen_at: Time.now },
947
- conflict: [:project_id, :key],
948
- update: { calls: "calls + excluded.calls", max_ms: "MAX(max_ms, excluded.max_ms)",
949
- seen_at: "excluded.seen_at" })
950
- Article.where(id: article.id).increment(:views) # UPDATE ... SET views = views + 1
951
- Comment.where(article_id: article.id).pluck_int(:id) # => [3, 7] (also pluck_str, pluck_float)
952
- Comment.all.count_by(:article_id) # => {"1" => 2, "4" => 1}
953
- Comment.all.count_by_sql("strftime('%Y-%m-%dT%H', created_at)") # hour buckets
954
- Stat.where(project_id: 1).sum_int(:calls) # also sum_float
955
- Article.where_like(:title, params[:q].to_s) # `title` LIKE '%...%' ESCAPE '\'
956
- ```
957
-
958
- - `upsert`'s `update` values and `count_by_sql`'s expression are SQL text, like
959
- `order_sql`: never build them from request data. An empty `update` is
960
- `DO NOTHING`. `upsert` and `increment` load no model and run no
961
- validation or callback, and fill in no timestamps.
962
- - `count_by` keys are Strings (through `Cast.str`), ascending; NULL and `""`
963
- share the key `""`. With a `limit` / `offset`, `count_by`, `sum_*` and
964
- `increment` work on that window, as `count` and `delete_all` do.
965
- - `where_like` is a substring match (case-insensitive for ASCII, as SQLite's
966
- LIKE is). A prefix search writes its own pattern:
967
- `Article.all.where_sql("title LIKE ? ESCAPE '\\'", [Cybertrain::Cast.like_escape(q) + "%"])`.
968
- - For hand-written SQL, `Cybertrain::Cast.binds(a, b, c)` builds the bind
969
- Array `execute` takes (a Time as UTC ISO 8601 text, booleans as 1/0):
970
- `Cybertrain::DB.with { |c| c.execute(sql, Cybertrain::Cast.binds(id, since)) }`.
971
- - `Cybertrain::Cast.hash(v)` / `Cast.array(v)` narrow a parsed JSON value to
972
- a Hash / Array, or a fresh empty one when it is not one (a Time is checked
973
- first, spikes/NOTES.md rule 8). They return a typed shallow copy, not `v`
974
- itself; `Cast.empty_hash` / `Cast.empty_array` are the seeded empty
975
- containers.
976
-
977
- ## Pagination
978
-
979
- A list page reads the page number from `?page=N`, windows the relation, and
980
- renders the links:
981
-
982
- ```ruby
983
- # app/controllers/articles_controller.rb
984
- def index
985
- @articles = Article.all.order("created_at DESC")
986
- @page = @articles.page(params.page_number, per: 20)
987
- end
988
- ```
989
-
990
- ```erb
991
- <%# app/views/articles/index.html.erb %>
992
- <% @articles.each do |article| %>
993
- <p><%= link_to article.title, article %></p>
994
- <% end %>
995
- <%= paginate @page, articles_path %>
996
- ```
997
-
998
- `params.page_number` is the 1-based `page` parameter, 1 when it is absent or
999
- not a number, clamped to `1..max_page` (`config.max_page`, 10000 by default,
1000
- `CYBERTRAIN_MAX_PAGE`); it never hands unbounded text to `String#to_i`, which
1001
- raises `RangeError` past 2**63-1 under Spinel. `params.int(:per, default: 20,
1002
- min: 1, max: 100)` is the same bounded read for any key (an optional `-` and
1003
- at most 18 digits, else the default; clamped to `min..max`).
1004
- `relation.page(n, per: 50)` sets `LIMIT per OFFSET (n - 1) * per` on the
1005
- relation (replacing an earlier `limit`/`offset`, so call it last), clamps `n`
1006
- the same way, so the offset never passes `(max_page - 1) * per`, and returns a
1007
- `Cybertrain::Page`: `number`, `per`, `row_offset`, `next?` (asked with one
1008
- `SELECT 1 ... LIMIT per + 1` over the window, no `COUNT(*)`), `prev?`,
1009
- `next_number`, `prev_number`. The rows stay on the relation, so
1010
- `@articles.each` yields typed `Article`s; `per` outside 1..1000 is an
1011
- `ArgumentError`. `paginate @page, path` renders `<nav class="pager">` with a
1012
- "Newer" link (`rel="prev"`), "Page N" and an "Older" link (`rel="next"`),
1013
- each left out when there is no such page; the links keep the current query
1014
- string with `page` replaced (page 1 drops it), re-encoded and escaped, so a
1015
- search or filter survives paging, and a path with its own query string
1016
- (`"/search?q=x"`) uses that one instead. Offset pagination only: keyset
1017
- pagination (`after: id`) for very large tables is not built yet. The scaffold's
1018
- index view lists every record; add the three lines above to page it.
1019
-
1020
- ## Rate limiting and the client address
1021
-
1022
- `rate_limit` in a controller caps how often one client may call some
1023
- actions, counted before the action runs:
1024
-
1025
- ```ruby
1026
- class SessionsController < ApplicationController
1027
- # a 303 back to the form, with flash[:alert], once exhausted
1028
- rate_limit to: 10, within: 3 * 60, only: [:create],
1029
- redirect_to: "/session/new", alert: "Try again later."
1030
- end
1031
-
1032
- class ApiKeysController < ApplicationController
1033
- rate_limit to: 100, within: 60, only: [:create, :update] # 429 Too Many Requests + Retry-After
1034
- end
1035
- ```
1036
-
1037
- Each declaration counts at most `to` requests per `within` seconds for each
1038
- client address and controller (a fixed window that opens with the first
1039
- request), then answers 429 with a `Retry-After` header, or redirects when
1040
- given `redirect_to:`. `within:` is in seconds, `only:`/`except:` take
1041
- Arrays, and there is no `with:` block. The counts live in the process, in a
1042
- table of at most 10,000 addresses (when it is full, expired windows and
1043
- then the oldest ones make room), so they are lost on restart and each
1044
- process counts on its own. Every request to a limited action takes one lock:
1045
- name the actions with `only:`. In a test, `Cybertrain::RateLimit.reset_all!`
1046
- forgets every count and `Cybertrain::RateLimit.current_time = n` fixes the
1047
- clock (`-1` restores it).
1048
-
1049
- The address counted is `request.client_ip` (`by: :remote_addr` counts the
1050
- connection's peer instead). It is `request.remote_addr` unless
1051
- `config.trusted_proxy` is on (`CYBERTRAIN_TRUSTED_PROXY=1`) **and** the
1052
- connection comes from loopback, that is, from a reverse proxy on the same
1053
- host (nginx, Caddy, kamal-proxy); then it is the last address in
1054
- `X-Forwarded-For`, the one the proxy appended. Earlier entries are whatever
1055
- the client sent and are never used, and a last entry that is empty, longer
1056
- than 45 bytes or not printable ASCII gives `remote_addr`. Without the
1057
- setting, every visitor behind such a proxy shares its address, and ten bad
1058
- passwords lock everyone out; with it, a client connecting directly from
1059
- elsewhere still cannot choose its own address. `request.loopback?` tells
1060
- whether the connection comes from this machine (127.0.0.0/8, `::1`,
1061
- `::ffff:127.x.x.x`), for pages only an operator on the server should
1062
- reach; behind a proxy on the same host every request is loopback, so check
1063
- `client_ip` too there. Design: [docs/design.md](docs/design.md) D26.
1064
-
1065
- A proxy that reaches the app over a network, not loopback, is listed by
1066
- address instead: `config.trusted_proxies`, from `CYBERTRAIN_TRUSTED_PROXIES`,
1067
- takes comma-separated IPv4 or IPv6 addresses or CIDRs
1068
- (`CYBERTRAIN_TRUSTED_PROXIES=172.16.0.0/12,10.0.0.0/8`, or a single
1069
- `172.18.0.2`). A connection from inside one of them gets the same treatment
1070
- as a loopback one above: the last `X-Forwarded-For` address, validated the
1071
- same way. An IPv4 entry also matches the IPv4-mapped form of its addresses
1072
- (`::ffff:172.18.0.2`), and a peer's `%zone` is ignored. An entry that is not
1073
- an address or CIDR, or has a prefix past `/32` (IPv4) or `/128` (IPv6),
1074
- stops the server at boot. `CYBERTRAIN_TRUSTED_PROXY=1` is the shorthand for
1075
- `127.0.0.0/8,::1`, and the two add up. This is the Kamal case: kamal-proxy
1076
- runs in its own Docker container and connects from its address on the Docker
1077
- network, so the `config/deploy.yml` that `cybertrain new` writes sets
1078
- `CYBERTRAIN_TRUSTED_PROXIES: 172.16.0.0/12` (Docker's default range; change
1079
- it if your networks use another, `docker network inspect kamal`). Leave
1080
- Kamal's `proxy.forward_headers` unset: with `ssl: true`, kamal-proxy sets
1081
- `X-Forwarded-For` itself rather than passing on the client's. List only
1082
- networks that clients cannot connect from: anyone inside a listed network
1083
- can choose the address a rate limit counts.
1084
-
1085
- ## Parameterised headers
1086
-
1087
- API clients often send `Scheme k1=v1, k2="v2"` headers (`Authorization:
1088
- Digest ...`, a vendor `X-Example-Auth`) or `k1=v1;k2=v2, k3` lists
1089
- (`Forwarded`, `Prefer`). `request.header_params(name)` parses one into a
1090
- `Hash` of lowercase names to values, so an API controller does not split it
1091
- by hand:
1092
-
1093
- ```ruby
1094
- # Authorization: Digest username="Mufasa", realm="api@example.com", nc=00000001, qop=auth
1095
- auth = request.header_params("authorization")
1096
- auth["username"] # => "Mufasa"
1097
- request.header_scheme("authorization") # => "Digest"
1098
-
1099
- # Forwarded: for=192.0.2.60;proto=https;by=203.0.113.43
1100
- request.header_params("forwarded")["proto"] # => "https"
1101
-
1102
- # Prefer: respond-async, wait=5
1103
- request.header_params("prefer").key?("respond-async") # => true
1104
- ```
1105
-
1106
- `,` and `;` both separate parameters and the first occurrence of a name
1107
- wins; a quoted value is unquoted (`\"` and `\\` undone), an unquoted one
1108
- has the spaces around it trimmed, and a name without `=` maps to `""`. A
1109
- leading scheme word (`Digest`, `Bearer`) is not a parameter:
1110
- `request.header_scheme(name)` returns it (`""` when there is none). A header
1111
- that is missing or longer than 8 KiB gives `{}`. Values are the bytes as sent,
1112
- never percent-decoded and possibly not valid UTF-8. A `Bearer <token>` or
1113
- `Basic <base64>` credential is not a parameter: check the scheme, then read
1114
- the credential from `request.header("authorization")`. Design:
1115
- [docs/design.md](docs/design.md) D31.
1116
-
1117
- ## Background work
1118
-
1119
- An application that has to do something outside a request (send a
1120
- notification after the response went out, sweep old rows every hour, retry a
1121
- call that failed) registers it with the framework, which runs it on one
1122
- background thread of the served process:
1123
-
1124
- ```ruby
1125
- # config/app.rb (or a file bin/<app>.rb requires before Cybertrain::Main.run)
1126
- Cybertrain::Scheduler.every(3600, "prune sessions") do
1127
- Cybertrain::DB.with { |c| c.execute("DELETE FROM sessions WHERE updated_at < datetime('now', '-30 days')") }
1128
- end
1129
-
1130
- Cybertrain::Job.handle("welcome_mail") do |payload|
1131
- data = Cybertrain::Cast.hash(JSON.parse(payload))
1132
- WelcomeMail.deliver(data["email"].to_s) # raise to have it retried
1133
- end
1134
- ```
1135
-
1136
- ```ruby
1137
- # in an action, after saving the user
1138
- Cybertrain::Job.enqueue("welcome_mail", JSON.generate({ "email" => @user.email }))
1139
- Cybertrain::Job.enqueue("reminder", @order.id.to_s, run_at: Time.now + 600)
1140
- ```
1141
-
1142
- - **When it runs.** The thread starts when the binary serves (`./<app>`,
1143
- `./<app> server`), after boot has connected the database, bound the port
1144
- and set `SPINEL_WORKERS`; `migrate`, `db ...` and `Main.command` commands
1145
- never start it, and neither does `Cybertrain::Test::Client`. A task runs
1146
- first `seconds` after the start, then `seconds` after the start of each
1147
- run. Each run is wrapped in a rescue that logs `[ERROR] Scheduled task
1148
- "name" failed: ...` and the loop goes on. The block takes no arguments
1149
- (Spinel has no `instance_eval`). `Cybertrain::Scheduler.last_tick` is the
1150
- epoch second of the last tick, for a health check of your own.
1151
- - **Jobs.** `Job.enqueue` adds a row to the `cybertrain_jobs` table, which
1152
- the framework creates itself (`CREATE TABLE IF NOT EXISTS`, on first use)
1153
- and leaves out of `db/schema.rb`. The payload is a String: JSON-encode
1154
- structured data. Inside a `DB.transaction` the row commits or rolls back
1155
- with the rest. The worker, a scheduler task polling every
1156
- `job_poll_interval` seconds (5), claims the oldest due job in a short
1157
- `BEGIN IMMEDIATE` transaction, runs its handler outside it, and marks it
1158
- `done`. A handler that raises is retried 1, 5, 15 and then 60 minutes
1159
- later, with `Class: message` (up to 1000 characters) in `last_error`, until
1160
- `job_max_attempts` (5) attempts have failed: the row is then `failed` and
1161
- stays for you to look at. A job whose kind has no handler is `failed` at
1162
- once; `Job.enqueue` refuses an unregistered kind. Rows a crashed process
1163
- left `running` are requeued when the worker starts. `Job.state_count("failed")`
1164
- and `Job.prune_done(before_epoch)` help with a health page and housekeeping.
1165
- - **One worker per database.** Every served process runs the thread unless
1166
- `CYBERTRAIN_WORKER=0` (`config.worker = false`): when several processes
1167
- share one SQLite file, turn it off in all but one, or each scheduled task
1168
- runs once per process.
1169
- - **Keep tasks short.** Tasks and handlers run one at a time on that one
1170
- thread, and a SQLite call holds its OS worker until it returns: with
1171
- `SPINEL_WORKERS=1` (the default) a long query stalls every request for as
1172
- long. Batch big work (delete a thousand rows per run, enqueue one job per
1173
- item) rather than doing it in one go.
1174
- - **Testing.** `Cybertrain::Scheduler.run_due(now)` runs one tick and
1175
- `Cybertrain::Job.work_one(now)` one job, with an epoch time you choose, so
1176
- a test never sleeps.
1177
-
1178
- There is no cron syntax, no priority or uniqueness for jobs and no mailer:
1179
- this covers "send later" and "run every N seconds" (`docs/design.md` D28).
1180
-
1181
- ## Operations: backups, vacuum, health check
1182
-
1183
- The data of a deployed application is one SQLite file, so the binary carries
1184
- the three operational tools such an application needs.
1185
-
1186
- **Backups.** `./<app> db backup PATH` writes a consistent copy of the
1187
- configured database (`CYBERTRAIN_DATABASE`) while the server keeps running:
1188
- SQLite's `VACUUM INTO` (3.27 or newer; the binary checks) reads one snapshot,
1189
- and under WAL the server's writes go on meanwhile. The copy is written to a
1190
- temporary file next to `PATH`, created exclusively with mode 0600 (the file
1191
- holds password digests and session tokens), fsynced, and only then renamed
1192
- to `PATH`, so a failed or interrupted copy never leaves a partial file under
1193
- the real name (the temporary one is deleted). An existing `PATH` is refused
1194
- unless you pass `--force`; the database itself never is a target, nor is a
1195
- `:memory:` database a source.
1196
-
1197
- ```sh
1198
- ./blog db backup /srv/blog/backups/before-upgrade.sqlite3
1199
- ./blog db backup /srv/blog/backups/ --keep 14 # blog-20261025T031500Z.sqlite3, newest 14 kept
1200
- ```
1201
-
1202
- A target ending in `/` (or an existing directory) gets a dated name,
1203
- `<app>-<YYYYmmddTHHMMSSZ>.sqlite3` in UTC, in a directory created with mode
1204
- 0700 if it does not exist; `--keep N` then deletes the oldest files of
1205
- exactly that shape beyond N and touches nothing else there. Restoring is
1206
- copying a backup over the database file with the server stopped (and
1207
- deleting that file's `-wal` and `-shm` beside it).
1208
-
1209
- **Vacuum.** Deleting rows leaves free pages in the file. `./<app> db vacuum`
1210
- runs `VACUUM` and then `PRAGMA wal_checkpoint(TRUNCATE)`, so the rebuilt
1211
- pages reach the database file, the file is cut to its new size and the WAL
1212
- is emptied; it prints the size on disk (database plus WAL) before and after.
1213
- `VACUUM` rewrites the whole file and holds the write lock while it does, so
1214
- run it when the application is quiet. If a reader keeps the checkpoint from
1215
- finishing it says so, and the WAL shrinks at a later checkpoint.
1216
-
1217
- **Run backups outside the server.** A SQLite call holds its OS worker until
1218
- it returns, so a large `VACUUM INTO` on a `Scheduler` task would stall
1219
- requests (with `SPINEL_WORKERS=1`, all of them) for as long as the copy
1220
- takes. Run the command from cron or a systemd timer instead, as a separate
1221
- process:
1222
-
1223
- ```sh
1224
- # crontab of the app's user: every night at 03:15, keep two weeks
1225
- 15 3 * * * cd /srv/blog/current && ./blog db backup /srv/blog/backups/ --keep 14
1226
- ```
1227
-
1228
- An application that wants to take backups itself can run its own binary as
1229
- a child process from a scheduled task, rather than run the SQL on its own
1230
- thread: `system` waits for the child on the scheduler's thread only, and
1231
- requests go on being served meanwhile (Spinel has no `Process.wait`, and a
1232
- `Process.spawn` without it leaves a zombie, so use `system` with an argument
1233
- list, never a string with request data in it). A copy that takes longer than
1234
- `health_worker_stale` makes the health check answer `worker stale` until it
1235
- is done.
1236
-
1237
- ```ruby
1238
- # config/app.rb, in a served binary started from its own directory (dist/)
1239
- Cybertrain::Scheduler.every(86_400, "nightly backup") do
1240
- system("./blog", "db", "backup", "storage/backups/", "--keep", "14")
1241
- end
1242
- ```
1243
-
1244
- **Health check.** While the health check is enabled (it is by default;
1245
- an empty `config.health_path` turns it off), the served application
1246
- answers `GET /up` (and `HEAD`) before the static files, the session, CSRF and the router: no controller
1247
- is involved and no login `before_action` can redirect it, and it sets no
1248
- cookie. It runs `SELECT count(*) FROM sqlite_master`, which has to read the
1249
- schema page from the file (a bare `SELECT 1` never touches it), and answers
1250
- in `text/plain` with `Cache-Control: no-store`:
1251
-
1252
- | Status | Body | When |
1253
- | --- | --- | --- |
1254
- | 200 | `ok` | the query succeeded |
1255
- | 503 | `database unavailable` | it raised `Cybertrain::DB::Error` (the file is missing or unreadable, the pool is closed) |
1256
- | 503 | `worker stale` | the background `Scheduler` runs and its last tick is older than `health_worker_stale` seconds (300): a task is stuck, or runs far too long |
1257
-
1258
- A poll is logged like any request (two lines). The path is
1259
- `config.health_path` (`CYBERTRAIN_HEALTH_PATH`, default `/up`; an empty
1260
- value turns the check off, and the path then goes on to `public/` and the
1261
- router as before). A load balancer, kamal-proxy or a deploy script can poll
1262
- it; `public/up.html`, which `cybertrain new` writes and its Kamal
1263
- configuration still polls, only shows that the process answers HTTP.
1264
-
1265
- ## Differences from Rails
1266
-
1267
- | Rails | cybertrain |
89
+ | | |
1268
90
  | --- | --- |
1269
- | `rails console` | No console — Spinel has no `eval` |
1270
- | `rails runner`, `lib/tasks/*.rake` | `Cybertrain::Main.command("name", "description") { \|argv\| ...; 0 }` in `bin/<app>.rb`: `./<app> name ARGS` runs it with the database connected |
1271
- | Edit code, the running app picks it up | Ruby needs a rebuild; `cybertrain server` in development does this for you (rebuild, stop, `execv`) |
1272
- | Edit a view, no reload needed | Same — views are parsed from disk per request in development |
1273
- | `def new` | `def new_action` (`new` would shadow `Klass.new(ctx)`) |
1274
- | `before_action { do_thing }` (implicit `self`) | `before_action { \|c\| c.do_thing }` — no `instance_exec` on a stored block, so callbacks take the controller/record explicitly |
1275
- | `resources :posts do resources :comments end` | `resources :posts do \|posts\| posts.resources :comments end` — nested blocks take the mapper explicitly |
1276
- | ERB compiles to a method; any object, any method | Interpreted against a fixed grammar/dispatch table — [docs/template-language.md](docs/template-language.md) |
1277
- | Session holds any marshalled object | Session values are Strings only, HMAC-signed cookie |
1278
- | Many database adapters | SQLite only, via FFI |
1279
- | `Article.transaction { ... }` returns the block's value; `raise ActiveRecord::Rollback`; `requires_new: true` savepoints | `Article.transaction { \|c\| ... }` (= `Cybertrain::DB.transaction`) yields the connection and returns nil; leave it by raising (it re-raises after the rollback); nested calls join the outer transaction, no savepoints. `immediate: true` for SQLite's `BEGIN IMMEDIATE`. `c.transaction { }` cannot be called on the yielded `c` under Spinel: nest `DB.transaction`, or use `c.begin!` / `c.commit!` / `c.rollback!` |
1280
- | `has_many :through`, `includes`, `dependent:`, enums, STI, polymorphic associations | Not implemented; associations come from schema foreign keys only |
1281
- | `pluck(:a)`, `sum(:a)`, `group(:a).count`, `upsert`, `update_counters`, `where("title LIKE ?", "%#{q}%")` | Typed one-column flavours, because a Spinel method has one return type: `pluck_str` / `pluck_int` / `pluck_float` (one column only), `sum_int` / `sum_float`, `count_by(:a)` (a `Hash{String => Integer}`) and `count_by_sql(expr)` for expressions; `Model.upsert(attrs, conflict:, update:)` takes the `DO UPDATE SET` expressions as SQL text and runs no validations, callbacks or timestamps; `relation.increment(:col, by: n)`; `where_like(:col, text)` escapes `\`, `%` and `_` for you. See [Querying beyond `where`](#querying-beyond-where) |
1282
- | Any column name; an association can shadow `errors` | A column name must be an ASCII identifier (`[A-Za-z_][A-Za-z0-9_]*`): a schema with a non-ASCII column name (`名前`, `prénom`) generated under CRuby before PR #10 and `spin run gen` refuses it now, so rename the column when upgrading (a name like `first-name` was never a method name). A column named like a method the generated class or the framework calls on a record (`errors`, `save`, `attributes`, `hash`, `to_s`, `to_ary`, `raise`, ...) or like a Ruby keyword (`end`, `class`, `begin`) generates under `<column>_column` (`hash_column`, reader, writer and ivar) with a note in the generated file; its SQL name keeps working in `read_attribute`, params and `attributes["hash"]`, and a template reaches it under either name (`post.hash`, `post.hash_column`), except `errors` and `to_param`, which a template resolves as the model methods, so use `post.errors_column` there; the query API (`where`, `order`, `find_by`) always takes the SQL column name (`Post.where(hash: 1)`), the Ruby name being the reader/writer and the template name only. A column named like another Object method (`display`, `tap`, `methods`) generates under its own name with a note. `cybertrain generate scaffold` refuses keywords and reserved names (it would have to call a renamed reader) and accepts a shadowing name with a note, like the generator. An association whose plain name a column, another association or a `Model` method already owns is emitted under a fallback name with a comment in the generated file: `<table>_as_<column stem>` for a `has_many` (`comments_as_article`), `<stem>_as_<column>` for a `belongs_to` (`author_as_author_id`) |
1283
- | `namespace`, format/`respond_to`, `constraints`, `mount` | Not implemented — flat names, `render json:` only |
1284
- | `rescue StandardError` catches a bad `JSON.parse` | Under Spinel `JSON::ParserError` is not a `StandardError`: app code must `rescue JSON::ParserError, StandardError`. The server catches it as a last resort and answers 500 |
1285
- | `skip_forgery_protection` in an API controller | CSRF is a middleware that runs before routing, so an API is exempted by path: `c.csrf_exempt_prefixes = ["/api/"]` in `config/app.rb`. The controller must then authenticate its callers itself (an API key in a header), and no session-authenticated route may match under the prefix |
1286
- | Rack leaves a gzip request body as it is | A body sent with `Content-Encoding: gzip` (an SDK or API client past a few kilobytes) is `request.decoded_body`, inflated once through the system zlib under a ceiling on the output (10 MiB by default; an endpoint that expects more passes its own, `request.decoded_body(limit)`): a corrupt body or an encoding other than gzip/identity is `Cybertrain::BodyUndecodable`, one that would inflate past the limit `Cybertrain::BodyTooLarge`, both `QueryInvalid`s answered 400; `request.body` stays the bytes as sent |
1287
- | `order("lower(title)")`, `order("posts.title")`, `order(params[:sort])` | `order` takes only `column [ASC\|DESC]` lists (each column quoted) and raises `ArgumentError` (a 500: the string is the developer's) on anything else; request data must go through an allowlist first, e.g. `SORTS = { "title" => "title", "newest" => "created_at DESC" }` and `Post.order(SORTS.fetch(params[:sort].to_s, "id"))`, because a well-formed term can still name a missing column (a 500 from SQLite) or a column the client must not sort by; raw ORDER BY text goes through `order_sql` (`Post.order_sql("lower(title)")`, also on a relation), which must never see request data. A `limit`/`offset` on `delete_all` is honoured (a subselect), and request parameters nest at most 32 levels / 4096 pairs (400 past that). `Query.parse`, `Query.value_of`, `Query.decode` and `request.form_value` / `query_value` raise `Cybertrain::QueryMalformed` (a `StandardError` under `Cybertrain::QueryInvalid`, not an `ArgumentError`) on a malformed percent-escape, on both runtimes (`Cookies.parse` still keeps the raw value). They raise it too on an invalid UTF-8 byte sequence, raw (whatever the String's tag) or percent-encoded (Rails likewise answers 400 for an invalid parameter encoding), and so does a raw byte that only an escape would complete (`a=\xC3%A9`); `Cookies.parse` keeps such a value raw, and a path with an invalid sequence (raw, or percent-decoded in a segment) is a 400 as well (`request.path_segments`; a malformed escape in a path stays literal, a 404 by non-match). App code that rescued `ArgumentError` around them should rescue `Cybertrain::QueryInvalid` instead (upgrade note). The request keeps no parameter tree of its own (the Router builds `params`, the middleware read one key); an app that wants the whole tree parses it itself, `Query.parse(request.utf8_query_string)`, and owns the result |
1288
- | Full backtrace on an exception | Class, message, request line, template name/line — Spinel exposes no backtraces |
1289
- | Rack, its middleware, and any gem in a `Gemfile` | No Rack compatibility; a small fixed middleware set; Spinel's own `spin-index`, limited to what compiles under its Ruby subset |
1290
- | minitest / RSpec | `Cybertrain::Test` — reflection-based runners can't work ahead-of-time |
1291
- | `bin/rails generate authentication`, `has_secure_password` + bcrypt | `cybertrain generate authentication` writes the same shape (User, Session, SessionsController, the before_action) plus a sign-up page, since there is no console to create a user in; `has_secure_password` is written out as methods on the User model over `Cybertrain::Password` (PBKDF2-HMAC-SHA256 in Ruby: bcrypt would need a C library the binary links, and `String#crypt` has no bcrypt on macOS). No password reset (no mailers); sign-in attempts are limited by `rate_limit` (counted per process; see [Rate limiting](#rate-limiting-and-the-client-address)) |
1292
- | Active Storage, `has_one_attached :cover` | `cybertrain generate attachment post cover` writes five `cover_*` columns on the owner table and the model methods (`cover_file=`, `cover_url`, ...) into the app; files live on disk under `storage/uploads/` and are served by a generated action through `send_blob`. One file per field, local disk only, no variants or direct uploads |
1293
- | `skip_before_action :x` | The same, for named before callbacks (a block callback has no name to skip); no `skip_after_action` |
1294
- | `resource :session` | The same six routes to `SessionsController`; no nesting under it |
1295
- | `get "/x"` then `post "/x"` | Both are fine: the second route's derived name is taken, so it is unnamed, as in Rails; a duplicate `as:` is still an error |
1296
- | `Time.zone = "Asia/Tokyo"`, `time.in_time_zone`, ActiveSupport's bundled zone data | `Cybertrain::TimeZone.find("Asia/Tokyo")` reads the system's TZif files (`/usr/share/zoneinfo`, or `TZDIR`) and converts Integer epoch seconds (`utc_offset_at`, `utc_to_local`, `local_to_utc`); no per-request `Time.zone` and no `l(time, zone:)` yet ([Time zones](#time-zones)) |
1297
- | `Net::HTTP`, Faraday | `Cybertrain::HttpClient` for webhooks and APIs: every wait bounded, non-public destinations refused by default, no redirects or keep-alive (see "Outbound HTTP"); Spinel's `Net::HTTP` blocks the whole server on a stalled peer |
1298
- | Action Mailer (`deliver_later`, mailer classes, views) | `Cybertrain::Mailer.deliver(to:, subject:, body:)` sends plain text over SMTP (STARTTLS, implicit TLS or plain to loopback; AUTH PLAIN; every wait bounded); no templates, HTML, attachments or queue (see "Sending e-mail") |
1299
- | Active Job, `perform_later`, `solid_queue`, recurring tasks | `Cybertrain::Job.handle(kind) { \|payload\| ... }` and `Job.enqueue(kind, payload, run_at:)` (a String payload in a `cybertrain_jobs` table, retried with backoff, then `failed`), `Cybertrain::Scheduler.every(seconds, name) { ... }`; one thread in the served process, `CYBERTRAIN_WORKER=0` elsewhere ([Background work](#background-work)) |
1300
- | ActionCable client, faye-websocket, websocket-client-simple | `Cybertrain::WebSocket.subscribe(name, url).on_message { \|conn, data\| ... }` keeps a feed connected while the server runs (reconnects, pings, silence detection), `Cybertrain::WebSocket::Client` for one synchronous connection; outbound only, every wait bounded (see "Consuming WebSocket feeds") |
1301
- | `config.filter_parameters += [:passw, /\Apin\z/, ->(k, v) { ... }]` | `config.filter_parameters` is an Array of Strings matched by substring, case-insensitively (no Regexps or lambdas); it applies to the request log and the development error page ([Filtered parameters](#configuration-and-environment-variables)) |
1302
-
1303
- | `get "up" => "rails/health#show"` (a route to a controller) | `GET /up` is built in and answered before the static files, the session and the router, so no `before_action` runs; it queries the database and the background worker's last tick, `200 ok` or `503` ([Operations](#operations-backups-vacuum-health-check)) |
1304
- | `bin/rails db:...` tasks for backups (none built in) | `./<app> db backup PATH` / `db backup DIR/ --keep N` (`VACUUM INTO`, mode 0600, renamed into place) and `./<app> db vacuum` |
1305
- | ActionCable (the server), asset pipeline, i18n | Not built (MVP non-goals, `docs/design.md` §2.3) |
1306
-
1307
- Not exhaustive — `docs/design.md` §8 ("捨てたもの") is the fuller record.
1308
-
1309
- ## Performance
1310
-
1311
- [docs/benchmark.md](docs/benchmark.md) measures examples/blog against the
1312
- same blog written in Rails 8.1, [bench/rails_blog](bench/rails_blog) (the same
1313
- views, routes, models and schema), both in production mode, each server
1314
- pinned to the same CPU cores of one 4-vCPU cloud VM and loaded by wrk from
1315
- two others. Medians of three 10-second runs, recorded 2026-10-09:
1316
-
1317
- | 1 CPU core, 16 connections | CyberTrain 0.2.1 | Rails 8.1.4 (YJIT, Puma) | |
1318
- | --- | --: | --: | --: |
1319
- | `GET /articles` (30 articles) | 2,641 req/s | 468 req/s | 5.6x |
1320
- | `GET /articles/1` (10 comments, 12 forms) | 2,450 req/s | 201 req/s | 12.2x |
1321
- | `POST /articles/2/comments` | 1,507 req/s | 390 req/s | 3.9x |
1322
- | `GET /style.css` | 13,300 req/s | 5,127 req/s | 2.6x |
1323
- | Memory after the load (RSS) | 10.6 MB | 126 MB | |
1324
- | Spawn to first response | 0.02 s | 1.3 s | |
1325
- | What the server needs | 2.5 MB (one binary) | 109 MB (Ruby, 69 gems, the app) | |
1326
-
1327
- With jemalloc in both (`allocator = "jemalloc"` in spin.toml; Rails'
1328
- production image preloads it) the ratios are 2.8 to 14.6. On two cores the lead
1329
- narrows to 2.0 to 5.5: a second Puma worker doubles Rails, while
1330
- `SPINEL_WORKERS=2` adds 9 to 58% to CyberTrain (set it to the core count on a
1331
- multi-core server; it defaults to 1). The binary also has to be built
1332
- (`cybertrain build` took about three minutes there). docs/benchmark.md has
1333
- latency, every configuration, the method and what was optimized to get here;
1334
- `bench/run` repeats the measurement and `bench/report` prints its tables.
1335
-
1336
- ## Configuration and environment variables
1337
-
1338
- ```ruby
1339
- # config/app.rb
1340
- Cybertrain.configure do |c|
1341
- c.port = 3000
1342
- c.workers = 1
1343
- end
1344
- ```
1345
-
1346
- `Cybertrain::Config` reads the environment first; `config/app.rb` overrides:
1347
-
1348
- | Variable | Attribute | Default |
1349
- | --- | --- | --- |
1350
- | `CYBERTRAIN_ENV` | `env` | `"development"`; `"production"` in a binary from `cybertrain build` (also `"test"`) |
1351
- | `PORT` | `port` | `3000` |
1352
- | `CYBERTRAIN_DATABASE` | `database_path` | `storage/#{env}.sqlite3` |
1353
- | `CYBERTRAIN_SECRET_KEY_BASE` | `secret_key_base` | required in production; dev/test auto-generate one into `tmp/secret_key` |
1354
- | `SPINEL_WORKERS` | `workers` | `1` |
1355
- | `CYBERTRAIN_HOST` | `host` | `"127.0.0.1"`; `0.0.0.0` listens on every interface (containers) |
1356
- | `CYBERTRAIN_ALLOWED_HOSTS` | `allowed_hosts` | Development accepts `localhost`, `127.0.0.1`, `[::1]`; comma-separated exact hostnames extend this list (no scheme, port or wildcard). Other hosts get 403 before static files, sessions, routing or diagnostics. This is separate from the listening address; production and test do not apply this development guard |
1357
- | `CYBERTRAIN_SESSION_SAME_SITE` | `session_same_site` | `"Lax"`; also `Strict` or `None` (`None` always adds `Secure`); anything else stops the server at boot |
1358
- | `CYBERTRAIN_SESSION_PARTITIONED` | `session_partitioned` | `false`; `1` or `true` adds `Partitioned` (and `Secure`) |
1359
- | `CYBERTRAIN_TRUSTED_PROXY` | `trusted_proxy` | `false`; `1` or `true` makes `request.client_ip` (what `rate_limit` counts) the last `X-Forwarded-For` address of a connection from loopback, for a reverse proxy on the same host (see [Rate limiting](#rate-limiting-and-the-client-address)); shorthand for `CYBERTRAIN_TRUSTED_PROXIES=127.0.0.0/8,::1` |
1360
- | `CYBERTRAIN_TRUSTED_PROXIES` | `trusted_proxies` | `[]`; comma-separated IPv4/IPv6 addresses or CIDRs (`172.16.0.0/12,10.0.0.0/8`) whose connections have `request.client_ip` taken from the last `X-Forwarded-For` address, for a proxy reached over a network, such as kamal-proxy in its container; a bad entry stops the server at boot |
1361
- | `CYBERTRAIN_MAX_PAGE` | `max_page` | `10000`: the largest page `params.page_number` gives and `Relation#page` accepts (see [Pagination](#pagination)); a value that is not a number of at most 18 digits, or below 1, is the default |
1362
- | `CYBERTRAIN_FILTER_PARAMETERS` | `filter_parameters` | `["password", "password_confirmation", "token", "secret", "secret_key_base", "session_key", "api_key", "authorization"]`; comma-separated names replace the list. A parameter whose name contains one of them (ASCII case-insensitively: `password` also matches `user[password]`) has its value written as `[FILTERED]` in the request log and on the development error page; `[]` in `config/app.rb` turns this off |
1363
- | `CYBERTRAIN_WORKER` | `worker` | `true`; `0` or `false` keeps this process from running scheduled tasks and jobs (see [Background work](#background-work)); `scheduler_interval` (1 s), `job_poll_interval` (5 s) and `job_max_attempts` (5) are set in `config/app.rb` |
1364
- | `CYBERTRAIN_HEALTH_PATH` | `health_path` | `"/up"`; the built-in health check's path (see [Operations](#operations-backups-vacuum-health-check)); an empty value turns it off; one without a leading `/` stops the server at boot. `health_worker_stale` (300 s) is set in `config/app.rb` |
1365
- | `CYBERTRAIN_HTTP_ALLOW_PRIVATE` | `http_allow_private` | `false`; `1` or `true` lets `Cybertrain::HttpClient` connect to loopback, private and other non-public addresses (see "Outbound HTTP") |
1366
- | `CYBERTRAIN_SMTP_HOST` | `smtp_host` | `""`; the SMTP server `Cybertrain::Mailer` sends through (see "Sending e-mail") |
1367
- | `CYBERTRAIN_SMTP_PORT` | `smtp_port` | `0`: the mode's port, 587 for `starttls`, 465 for `tls`, 25 for `none`; anything but 1 to 65535 is 0 |
1368
- | `CYBERTRAIN_SMTP_USER` | `smtp_user` | `""`: no AUTH; set, the mailer authenticates with AUTH PLAIN |
1369
- | `CYBERTRAIN_SMTP_PASSWORD` | `smtp_password` | `""`; a secret, never logged, never in the boot banner or an error message |
1370
- | `CYBERTRAIN_SMTP_FROM` | `smtp_from` | `""`: the sender `Mailer.deliver` uses when it is not given `from:` |
1371
- | `CYBERTRAIN_SMTP_TLS` | `smtp_tls` | `"starttls"`; also `tls` (implicit TLS) or `none` (plain text, credentials to loopback only); both TLS modes need `require "cybertrain/net/tls"` |
1372
-
1373
- Use `None` (with `CYBERTRAIN_SESSION_PARTITIONED=1`) only for an app shown
1374
- inside another site's frame; the browser must reach the app over HTTPS.
1375
-
1376
- Other attributes with fixed, overridable defaults: `views_root`
1377
- (`"app/views"`), `public_root` (`"public"`), `layout`
1378
- (`"layouts/application"`), `log_level` (`:info`), `session_cookie_name`,
1379
- `session_max_age` (2 weeks), `session_secure` (`true` in production, which
1380
- marks the session cookie `Secure`; `false` elsewhere, though `SameSite=None`
1381
- and `Partitioned` add `Secure` anyway), `pool_size` (4), `static_files`/`csrf`
1382
- (`true`), `csrf_exempt_prefixes` (`[]`: path prefixes such as `"/api/"` whose
1383
- non-GET requests skip the CSRF check, for an endpoint that programs call with
1384
- a credential of their own; each must start and end with `/` and not be `/`
1385
- alone, or boot fails; the prefix is matched on the path as sent, so a
1386
- percent-encoded spelling is still checked; no other route may match a path
1387
- under the prefix, and the exempt controller must not rely on the session
1388
- for authentication, since the session is still loaded there), `max_body_bytes`
1389
- (10 MiB: a longer request body, as sent, is answered 413 before it is read;
1390
- raise it for large uploads; a gzip body's inflated size is bounded separately
1391
- by `request.decoded_body(limit)`), `max_render_depth` (12 renders open at once: the page and its
1392
- partials; raise it for partials that legitimately recurse deeper; it must be
1393
- at least 1, or boot fails). Upgrade note: render nesting, unlimited before, is
1394
- now capped at 12 by default (the page is depth 1, so partials can nest 11
1395
- levels; the layout renders after the page and does not nest), and an app with
1396
- a deeper tree (threaded comments, a category menu) gets a template error until
1397
- it sets `max_render_depth`. File uploads: `max_upload_bytes` (5 MiB per
1398
- uploaded file; it must not exceed `max_body_bytes`), `max_multipart_parts`
1399
- (128 parts, text fields included, at most 4096), `upload_root`
1400
- (`"storage/uploads"`, where `Cybertrain::BlobStore` writes; created on the
1401
- first upload, not at boot).
1402
-
1403
- **Security headers:** every response the server writes (actions, the
1404
- static-file middleware, error pages and the development diagnostics page,
1405
- the server's own 400/411/413/500, and HEAD) carries
1406
- `X-Content-Type-Options: nosniff` and `X-Frame-Options: SAMEORIGIN`, unless
1407
- the response already sets a header of that name (any case): `Response#to_http`
1408
- adds them after the response's own headers and only when missing, so
1409
- `response.set_header("X-Frame-Options", "DENY")` in an action sends exactly
1410
- your value, once. There is deliberately no `Config`
1411
- switch to turn them off app-wide; an embeddable widget overrides the header
1412
- on its own responses (a `Content-Security-Policy: frame-ancestors ...` header
1413
- takes precedence over `X-Frame-Options` in current browsers). Because of
1414
- `nosniff`, browsers take `Content-Type` literally: the static-file middleware
1415
- knows the usual `public/` extensions (html, css, js/mjs, images including
1416
- webp and avif, fonts, wasm, pdf, audio and video, json, csv, xml), and
1417
- anything else is served as `application/octet-stream`, a download; add the
1418
- extension to `Cybertrain::Static::EXTENSIONS` if an app needs another one
1419
- inline. A reverse proxy that serves `public/` itself bypasses these headers
1420
- (see [docs/deploy.en.md](docs/deploy.en.md)). `Cybertrain::Test::Client` calls the
1421
- app in-process and never serializes the response, so it does not show the two
1422
- defaults; assert them on `response.to_http`.
1423
-
1424
- **Filtered parameters:** the request log's `Started GET "/login?..."` line
1425
- and the development error page (its request line, and its dump of the
1426
- params and of a JSON body) show the value of every parameter named in
1427
- `config.filter_parameters` as `[FILTERED]`, as Rails' `filter_parameters`
1428
- does, so a `?token=` in a URL or a password posted to a form that fails
1429
- does not appear in the framework's request log or on the development
1430
- error page. Only the values are replaced: the rest of the
1431
- query string is logged as sent (control characters escaped). The list
1432
- matches by substring, case-insensitively; extend it in `config/app.rb`
1433
- (`c.filter_parameters = c.filter_parameters + ["otp", "ssn"]`) or replace
1434
- it with `CYBERTRAIN_FILTER_PARAMETERS`. Values an action logs itself are not
1435
- filtered; ask `Cybertrain::ParameterFilter.filter?(name)` or
1436
- `ParameterFilter.filter_query(text)` there.
1437
-
1438
- **Deployment:** `cybertrain build` produces `dist/`: the binary `dist/NAME`
1439
- with the views embedded, `dist/public/` (static assets, or let a reverse
1440
- proxy serve it), and `dist/storage/` (the SQLite file by default) and
1441
- `dist/tmp/`. Run it from `dist/` with `CYBERTRAIN_SECRET_KEY_BASE` set;
1442
- it is production unless `CYBERTRAIN_ENV` says otherwise, and `./NAME
1443
- migrate` applies the migrations compiled into it. The binary
1444
- speaks plain HTTP/1.1 only; put nginx/Caddy in front for TLS and HTTP/2 (the
1445
- session cookie is `Secure` in production, so serve it over HTTPS). In
1446
- production an exception answers 500 and a bare error response (the router's
1447
- plain-text 404, `head :not_found`, `render plain:`) is replaced by
1448
- `public/<status>.html` when that file exists; errors an action renders as HTML
1449
- or JSON pass through. `SIGTERM` stops accepting, lets requests already in
1450
- flight finish (answered with `Connection: close`; idle keep-alive connections
1451
- are closed at once) and exits when they are done, or after the server's
1452
- `drain_timeout` (10 s). The watcher/rebuild loop never runs in production. [docs/deploy.en.md](docs/deploy.en.md)
1453
- (in Japanese: [docs/deploy.md](docs/deploy.md)) walks through it end to end: `cybertrain new`, an Ubuntu server,
1454
- systemd, Caddy with HTTPS, redeploys, rollbacks and backups.
1455
-
1456
- **Kamal:** `cybertrain new` also writes what [Kamal 2](https://kamal-deploy.org)
1457
- needs: a `Dockerfile` that runs `cybertrain setup` and `cybertrain build` in an
1458
- Ubuntu build stage and copies `dist/` into a slim stage of the same release
1459
- (migrate on boot, then serve on `0.0.0.0:3000`), a `.dockerignore`,
1460
- `config/deploy.yml` (kamal-proxy with Let's Encrypt in front, the
1461
- health check on `public/up.html`, the SQLite file on a volume at
1462
- `/app/storage`) and `.kamal/secrets` (`KAMAL_REGISTRY_PASSWORD` and
1463
- `CYBERTRAIN_SECRET_KEY_BASE` from the environment). Fill in the values marked
1464
- `CHANGE ME`, commit, and `kamal setup`;
1465
- [docs/deploy-kamal.md](docs/deploy-kamal.md) has the details (part 8 of
1466
- the deployment tutorial).
1467
- [examples/blog](examples/blog#deploying-with-kamal) carries the same setup,
1468
- built against this checkout's framework.
1469
-
1470
- ### Application settings
1471
-
1472
- An application's own settings (a public base URL, a feature switch, an
1473
- interval, an SMTP host) are declared on the same `Config`, with a type, an
1474
- environment variable and a default:
1475
-
1476
- ```ruby
1477
- # config/app.rb
1478
- Cybertrain.configure do |c|
1479
- c.setting_string :base_url, env: "MYAPP_BASE_URL", default: ""
1480
- c.setting_string :smtp_password, env: "MYAPP_SMTP_PASSWORD", default: "", secret: true
1481
- c.setting_bool :worker, env: "MYAPP_WORKER", default: true
1482
- c.setting_int :prune_interval, env: "MYAPP_PRUNE_INTERVAL", default: 3600, min: 1, max: 9_999_999
1483
- end
1484
-
1485
- Cybertrain.config.string_setting(:base_url)
1486
- Cybertrain.config.bool_setting(:worker)
1487
- Cybertrain.config.int_setting(:prune_interval)
1488
- ```
1489
-
1490
- Each value is read once, when `config/app.rb` runs at startup. An empty or
1491
- unset variable means the default. A boolean takes `1`/`true` or
1492
- `0`/`false`; an integer takes decimal digits (an optional leading `-`) within
1493
- `min..max` (both optional). Any other value falls back to the default and
1494
- logs a `[WARN]` line saying so. Reading an undeclared name, or one declared
1495
- with another type, raises `ArgumentError`, as does declaring a name twice.
1496
- Tests change a value with `set_string_setting`, `set_int_setting` and
1497
- `set_bool_setting`. `./NAME help` lists every declared setting with its
1498
- variable and default, and the boot banner prints the current values
1499
- (`=> Settings: base_url="" worker=true ...`); a `secret: true` setting shows
1500
- up in both only by name, as `[FILTERED]`. There is no
1501
- `Cybertrain.config.base_url` accessor: Spinel has no `define_method`, so the
1502
- readers are typed methods that take the name.
1503
-
1504
- ## Time zones
1505
-
1506
- Spinel ships no tz database, and setting `ENV["TZ"]` to convert a time
1507
- changes it for every thread of the process. `Cybertrain::TimeZone` reads the
1508
- system's zone files instead: `/usr/share/zoneinfo/<name>`, or `$TZDIR/<name>`
1509
- when `TZDIR` is set (install the `tzdata` package in a slim container image).
1510
- It parses TZif versions 1 to 4 (RFC 8536), including the POSIX TZ rule a
1511
- "slim" file relies on after its last transition, and caches each zone for the
1512
- life of the process (copy-on-write, so lookups from several workers take no
1513
- lock).
1514
-
1515
- Everything is Integer epoch seconds. A *local epoch* is a wall-clock reading
1516
- counted as if it were UTC, so wall-clock arithmetic is plain addition and
1517
- `TimeZone.civil_to_epoch` / `TimeZone.epoch_to_civil` move between it and
1518
- `[year, month, day, hour, minute, second]`:
1519
-
1520
- ```ruby
1521
- tokyo = Cybertrain::TimeZone.find("Asia/Tokyo")
1522
- tokyo.found? # false for an unknown name: the zone then behaves as UTC
1523
- tokyo.utc_offset_at(Time.now.to_i) # => 32400
1524
- tokyo.abbreviation_at(Time.now.to_i) # => "JST"
1525
-
1526
- # The next 09:00 in Tokyo:
1527
- now = Time.now.to_i
1528
- today = Cybertrain::TimeZone.epoch_to_civil(tokyo.utc_to_local(now))
1529
- run = tokyo.local_to_utc(Cybertrain::TimeZone.civil_to_epoch(today[0], today[1], today[2], 9, 0, 0))
1530
- run = tokyo.local_to_utc(Cybertrain::TimeZone.civil_to_epoch(today[0], today[1], today[2] + 1, 9, 0, 0)) if run <= now
1531
- ```
1532
-
1533
- At a daylight-saving transition `local_to_utc` is deterministic: a wall time
1534
- that a forward jump skips (02:30 on a US spring-forward day) is read with the
1535
- offset before the jump, so it lands after it (03:30 EDT); a wall time that a
1536
- backward jump repeats gives the earlier instant. `find` accepts only a
1537
- relative name made of `A-Za-z0-9_+-./` with no empty, `.` or `..` segment, so
1538
- a zone name taken from a user's profile cannot reach a file outside the zone
1539
- directory; an unknown or malformed zone is not cached, and `"UTC"` needs no
1540
- file. Formatting helpers for templates (`l(time, zone:)`) are not there yet.
1541
-
1542
- ## Outbound HTTP
1543
-
1544
- An application that calls another server (a webhook, a Slack incoming
1545
- webhook, a third-party API) uses `Cybertrain::HttpClient`. It is not in
1546
- `require "cybertrain"`; require it where it is used, and the TLS layer too
1547
- when any URL is https:
1548
-
1549
- ```ruby
1550
- require "json"
1551
- require "cybertrain/http_client"
1552
- require "cybertrain/net/tls" # only for https://; links the system libssl
1553
-
1554
- client = Cybertrain::HttpClient.new(timeout: 5, deadline: 15, max_body: 1 << 20)
1555
- begin
1556
- res = client.post(ENV["SLACK_WEBHOOK_URL"], JSON.generate({ "text" => "Deployed" }),
1557
- content_type: "application/json")
1558
- Cybertrain.logger.warn("slack answered #{res.status}") unless res.success?
1559
- rescue Cybertrain::HttpClient::Error => e
1560
- Cybertrain.logger.warn("slack: #{e.message}") # names host:port, never the path
1561
- end
1562
-
1563
- res = client.get("https://api.example.com/v1/status", headers: { "Authorization" => "Bearer #{token}" })
1564
- res.status # 200
1565
- res.header("ETag") # names in any case; res.headers has them lowercase
1566
- res.body # the bytes, chunked coding removed
1567
- ```
1568
-
1569
- Why it exists: Spinel's `Net::HTTP` does the TLS handshake and every read
1570
- after the first byte as blocking C calls, and each one holds an OS worker,
1571
- which with the default `SPINEL_WORKERS=1` is the whole web server, for as
1572
- long as a stalled endpoint likes. `HttpClient` never blocks a worker on the
1573
- network:
1574
-
1575
- - **Every wait is bounded.** Connecting (a non-blocking connect; Spinel's
1576
- `TCPSocket.new` ignores `connect_timeout:`), each TLS handshake step, each
1577
- read and each write waits at most `timeout` seconds (default 5), and the
1578
- whole request at most `deadline` seconds (default 15). Past either,
1579
- `HttpClient::TimeoutError`. Reads go through a byte buffer fed by
1580
- `read_nonblock`, so bytes already received are never waited for.
1581
- - **Name resolution blocks.** `getaddrinfo` has no non-blocking form in the
1582
- runtime: the thread making the request waits for DNS (usually
1583
- milliseconds, but a dead resolver can take the system's timeout, about 5 s
1584
- per attempt). The deadline counts that time but cannot interrupt it.
1585
- - **Non-public destinations are refused** before anything is sent:
1586
- loopback (`127.0.0.0/8`, `::1`), private (`10/8`, `172.16/12`,
1587
- `192.168/16`, `fc00::/7`), link-local (`169.254/16`, which includes cloud
1588
- metadata services, `fe80::/10`), unspecified (`0.0.0.0/8`, `::`),
1589
- multicast, broadcast, CGNAT (`100.64/10`), the documentation and
1590
- benchmarking ranges, and IPv4-mapped, NAT64 and 6to4 forms of those
1591
- (`Cybertrain::Net::Address.blocked_reason`). The host name is resolved
1592
- once and the addresses it resolved to are the ones checked and dialled
1593
- (nothing is looked up again), so a name that resolves to a private
1594
- address (an `/etc/hosts` entry, DNS rebinding) is refused like the
1595
- literal; the TLS certificate is still checked against the name. The
1596
- check is per address: a refused address is skipped, and only when every
1597
- address is refused does the request fail.
1598
- `HttpClient::BlockedAddressError` says why. To call a service on your
1599
- own network, or a local server in a test, set `c.http_allow_private =
1600
- true` in `config/app.rb` (or `CYBERTRAIN_HTTP_ALLOW_PRIVATE=1`), or pass
1601
- `allow_private: true` to `HttpClient.new`; leave it off whenever a URL
1602
- can come from a user.
1603
- - **Every resolved address is tried**, in the order the resolver gives
1604
- them, IPv6 and IPv4 alike: when one refuses, is unreachable, resets or
1605
- does not answer within `timeout`, the next is dialled, all of them
1606
- within `deadline`, and when every one fails the last one's error is
1607
- raised. This is what makes `localhost` work on macOS, where it resolves
1608
- to `::1` first and a server listening on `127.0.0.1` refuses that.
1609
- `Cybertrain::Mailer` dials the same way.
1610
- - **TLS** is the openssl package's: `VERIFY_PEER` against the operating
1611
- system's trust store (`SSL_CERT_FILE` / `SSL_CERT_DIR` point it
1612
- elsewhere; the package has no `X509::Store` for a custom one), host-name
1613
- checking, SNI, TLS 1.2 or newer. An https URL needs a host name: an IP
1614
- address does not match a certificate's name. Without `require
1615
- "cybertrain/net/tls"`, an https request raises `HttpClient::TLSError`
1616
- saying so; `cybertrain/http_client` alone never links libssl.
1617
- - **A notification client, not a browser:** HTTP/1.1 with `Connection:
1618
- close`, one connection per request, no keep-alive, no redirects (a 3xx is
1619
- returned as it is), no proxy, no cookies, no compression. Responses may be
1620
- `Content-Length`, chunked or read to the close; the headers are capped at
1621
- 64 KiB and 100 lines, the body at `max_body` (default 1 MiB), past which
1622
- `HttpClient::TooLargeError`. A response that is not HTTP/1.x, or ends
1623
- early, is `HttpClient::BadResponseError`; a refused or reset connection, or
1624
- a name that does not resolve, `HttpClient::RefusedError`. A connection
1625
- that is reset or closed during the TLS handshake (a server that does not
1626
- speak TLS) is `HttpClient::TLSError` ("TLS handshake failed") on every
1627
- platform, like a handshake OpenSSL rejects. All of them are
1628
- `HttpClient::Error`s.
1629
- - **Errors never contain the path or the query**, where a webhook URL
1630
- carries its secret: messages name the host and port only. A malformed URL
1631
- or a header with CR/LF is an `ArgumentError` that does not repeat it
1632
- either. `Host`, `Connection`, `Content-Length` and `Transfer-Encoding` are
1633
- the client's; `User-Agent` defaults to `cybertrain/<version>`.
1634
-
1635
- The request runs on the calling thread, a green thread like the request
1636
- being served, so other connections keep being served while it waits. The
1637
- socket layer (`cybertrain/net/socket.rb`: bounded connect, buffered reads,
1638
- bounded writes) is separate from the HTTP layer so that other protocols can
1639
- use it. The design: [docs/design.md](docs/design.md) D24.
1640
-
1641
- ## Consuming WebSocket feeds
1642
-
1643
- An application that listens to an outside feed (market ticks, device
1644
- telemetry, an event stream) while the server runs subscribes to it with
1645
- `Cybertrain::WebSocket.subscribe`. Like the HTTP client it is not in
1646
- `require "cybertrain"`; require it where the subscription is written, and the
1647
- TLS layer too for `wss://`:
1648
-
1649
- ```ruby
1650
- # config/app.rb (or a file bin/<app>.rb requires before Cybertrain::Main.run)
1651
- require "json"
1652
- require "cybertrain/websocket"
1653
- require "cybertrain/net/tls" # only for wss://; links the system libssl
1654
-
1655
- Cybertrain::WebSocket.subscribe("trades", ENV["FEED_URL"],
1656
- headers: { "Authorization" => "Bearer #{ENV["FEED_TOKEN"]}" },
1657
- stale_after: 30)
1658
- .on_open { |conn| conn.send_text(JSON.generate({ "op" => "subscribe", "channel" => "trades" })) }
1659
- .on_message { |conn, text| Cybertrain::DB.with { |c| c.execute("INSERT INTO trades(raw) VALUES (?)", [text]) } }
1660
- ```
1661
-
1662
- Requiring `cybertrain/websocket` registers the feeds with
1663
- `Cybertrain::Background`, which `Application#serve` starts after the
1664
- scheduler and stops at shutdown, so there is nothing to call. Each
1665
- subscription gets one thread. It connects, runs `on_open` (send your
1666
- subscribe message there: it runs again after every reconnect), then hands
1667
- each message to `on_message` with the `Connection` (`send_text`,
1668
- `send_binary`, `close`, and `last_opcode`, `:text` or `:binary`). The
1669
- blocks take their arguments and cannot use `self` (Spinel has no
1670
- `instance_eval`).
1671
-
1672
- - **When it runs.** The feeds start when the binary serves, like the
1673
- scheduler ([Background work](#background-work)); `migrate`, `db ...`,
1674
- `Main.command` commands and `Cybertrain::Test::Client` never start them.
1675
- With `CYBERTRAIN_WORKER=0` (`config.worker = false`) they do not run in
1676
- that process (they simply do not start; the only notice is the
1677
- scheduler's "run elsewhere" line, and `WebSocket.start!` called directly
1678
- logs its own): turn it off in all but one when
1679
- several processes serve, or each one receives the whole feed.
1680
- `subscribe` must be called before the server starts; there is no adding or
1681
- removing a feed while it runs. A name is unique (`ArgumentError`), and a
1682
- feed without `on_message` fails the boot before the port is bound
1683
- (`error: WebSocket subscription "trades" has no on_message`, exit 1).
1684
- - **Options.** `headers:` (extra handshake headers), `timeout:` (5 seconds
1685
- per wait while connecting and per write), `max_message:` (1 MiB; a larger
1686
- message closes the connection with 1009), `ping_interval:` (15) and
1687
- `stale_after:` (45), and `allow_private:` (as `HttpClient`, below).
1688
- - **Waiting and reconnecting.** After any failure the thread waits
1689
- `min(60, 2 ** (attempt - 1))` seconds, with 20 % of jitter either way: 1, 2,
1690
- 4, ... 60. A connection that lasted 60 seconds starts again from 1. Every
1691
- wait is bounded. `WebSocket.stop!` returns within about 2 seconds
1692
- whatever the servers do: it waits at most 2 seconds for all the threads
1693
- together, a connected feed sends a close (1000) and waits at most 1 second
1694
- for the server's answer, and every feed's `connected?` is false once it
1695
- returns (a thread still closing finishes on its own).
1696
- - **Silence.** A feed can go quiet without ever closing the connection, so
1697
- the thread pings after `ping_interval` seconds without any frame (and then
1698
- once per interval), and when no frame at all, a pong included, has arrived
1699
- for `stale_after` seconds it closes the connection and reconnects.
1700
- `stale_after` must be more than `ping_interval`, which must be at least 1
1701
- (`ArgumentError` from `subscribe`).
1702
- - **Errors in your blocks.** An exception in `on_message` is logged and the
1703
- connection stays. An exception in `on_open` is logged and the connection is
1704
- closed and retried, since a feed that did not subscribe is of no use.
1705
- - **What is logged.**
1706
-
1707
- | When | Level | Line |
1708
- | --- | --- | --- |
1709
- | start | INFO | `WebSocket started: trades (wss://feed.example.com:443)` |
1710
- | connected | INFO | `WebSocket "trades" connected to feed.example.com:443` |
1711
- | disconnected | WARN | `WebSocket "trades" disconnected: Cybertrain::WebSocket::ClosedError: feed.example.com:443: closed by the server (1001) (retry in 2s)` |
1712
- | silent | WARN | `WebSocket "trades" silent for 45s; reconnecting` |
1713
- | a block raised | ERROR | `WebSocket "trades" on_message failed: KeyError: ...` (or `on_open`) |
1714
-
1715
- The path and the query are never logged or put in an error message: they
1716
- often carry a token.
1717
- - **Errors.** All are `Cybertrain::WebSocket::Error`s, with host and port
1718
- only in the message. Once the handshake is done, a lost connection or a
1719
- close by the server is a `ClosedError` (`close_code`, `close_reason`; a
1720
- reset or an end without a close frame is 1006); a write that times out or
1721
- fails in TLS is still a `TimeoutError` or `TLSError`. While connecting it
1722
- is `RefusedError`, `TimeoutError`, `TLSError`, `BlockedAddressError` (a non-public address,
1723
- refused by default exactly as in [Outbound HTTP](#outbound-http), with
1724
- the same `allow_private:` and `http_allow_private` setting) or
1725
- `HandshakeError` (not a 101, a wrong `Sec-WebSocket-Accept`, or the
1726
- server answering with an extension or subprotocol). `ProtocolError`
1727
- (the server broke the framing; the client sends 1002) and
1728
- `TooLargeError` can come at any time.
1729
- - **Health.** `Cybertrain::WebSocket.feed_status("trades")` answers a
1730
- `Status`: `connected?`, `last_message_at` (epoch seconds, 0 before the
1731
- first message) and `reconnects`. It is not part of `GET /up`; use it in
1732
- your own check. The three are read one after the other, not as one
1733
- snapshot, and `connected?` turns true when the connection is made, before
1734
- `on_open` has run.
1735
- - **Keep `on_message` short.** The thread waits with `wait_readable` or
1736
- `sleep`, so a stalled feed does not hold an OS worker while it waits. Two
1737
- things do hold one: each connect attempt resolves the host name with a
1738
- blocking `getaddrinfo` (see "Name resolution blocks" in [Outbound
1739
- HTTP](#outbound-http); with `SPINEL_WORKERS=1` a dead resolver stalls
1740
- every request for the resolver's timeout, once per attempt per feed, and
1741
- an IP address in the URL avoids the lookup), and a SQLite call inside
1742
- `on_message` until it returns, as in a scheduled task ([Background
1743
- work](#background-work)): with `SPINEL_WORKERS=1` a long write stalls every
1744
- request. For a busy feed insert one small row, or collect messages in
1745
- memory and write them in batches from a `Scheduler.every` task.
1746
- - **Not built.** Extensions (`permessage-deflate`) and subprotocols are not
1747
- negotiated. There is no proxy, redirect or cookie support, no automatic
1748
- saving of messages, and no WebSocket server (ActionCable). Text frames are
1749
- not checked for valid UTF-8 (a departure from RFC 6455 section 8.1): the
1750
- bytes are passed on as they came. The `Connection` is valid only during
1751
- the callback; sending from a request or a task is not supported.
1752
-
1753
- For one connection without the reconnect loop (a script, a one-shot read)
1754
- use the synchronous client:
1755
-
1756
- ```ruby
1757
- require "cybertrain/websocket"
1758
- require "cybertrain/net/tls" # only for wss://
1759
-
1760
- client = Cybertrain::WebSocket::Client.connect("wss://feed.example.com/v1", timeout: 5)
1761
- client.send_text("hello")
1762
- msg = client.receive(10) # waits at most 10 s; nil when nothing came
1763
- msg.ws_opcode # :text or :binary
1764
- msg.ws_data # the bytes
1765
- client.close
1766
- ```
1767
-
1768
- The design: [docs/design.md](docs/design.md) D33.
1769
-
1770
- ## Sending e-mail
1771
-
1772
- `Cybertrain::Mailer` sends plain-text e-mail through an SMTP server (your
1773
- provider's submission server, or a local relay). Like the HTTP client it is
1774
- not in `require "cybertrain"`, and TLS is a separate require:
1775
-
1776
- ```ruby
1777
- require "cybertrain/mailer"
1778
- require "cybertrain/net/tls" # for starttls (the default) and tls; links the system libssl
1779
-
1780
- # Settings from CYBERTRAIN_SMTP_HOST, _PORT, _USER, _PASSWORD, _FROM, _TLS:
1781
- begin
1782
- Cybertrain::Mailer.deliver(to: user.email, subject: "Welcome", body: "Hello #{user.name},\n...")
1783
- rescue Cybertrain::Mailer::Error => e
1784
- Cybertrain.logger.warn("mail: #{e.message}") # never contains the password
1785
- end
1786
-
1787
- # Or explicit settings, bypassing Config:
1788
- mailer = Cybertrain::Mailer.new(host: "smtp.example.com", port: 587, tls: "starttls",
1789
- user: "app", password: ENV["SMTP_PASSWORD"].to_s,
1790
- from: "noreply@example.com", timeout: 10, deadline: 60)
1791
- mailer.deliver(to: "a@example.com, b@example.com", subject: "Report", body: report)
1792
- ```
1793
-
1794
- | Variable | Attribute | Default |
1795
- | --- | --- | --- |
1796
- | `CYBERTRAIN_SMTP_HOST` | `smtp_host` | `""` (`Mailer.new` raises `ArgumentError` until it is set) |
1797
- | `CYBERTRAIN_SMTP_PORT` | `smtp_port` | `0`: 587 for `starttls`, 465 for `tls`, 25 for `none` |
1798
- | `CYBERTRAIN_SMTP_USER` | `smtp_user` | `""`: no AUTH |
1799
- | `CYBERTRAIN_SMTP_PASSWORD` | `smtp_password` | `""` (secret) |
1800
- | `CYBERTRAIN_SMTP_FROM` | `smtp_from` | `""` |
1801
- | `CYBERTRAIN_SMTP_TLS` | `smtp_tls` | `"starttls"`; `tls` or `none` |
1802
-
1803
- - **TLS modes.** `starttls` connects in plain text, requires the server to
1804
- offer STARTTLS (else `Mailer::TLSError`) and upgrades before anything
1805
- else is said; bytes the server sends after its STARTTLS reply are
1806
- refused. `tls` speaks TLS from the first byte (port 465). Both check the
1807
- certificate against `host` as the HTTP client does (give a host name, not
1808
- an IP address), and both need `require "cybertrain/net/tls"`: without
1809
- it, `deliver` raises `Mailer::TLSError` saying so, before connecting.
1810
- `none` is plain text, for a relay on the same machine or a test server:
1811
- with a user configured it refuses (`Mailer::InsecureError`, before
1812
- connecting) unless the host resolves to a loopback address, and then
1813
- dials only its loopback addresses.
1814
- - **The password is never logged or shown.** It is not in the boot banner,
1815
- `help`, the log or any error message; it is sent only inside the AUTH
1816
- PLAIN command, over TLS (or to loopback). Do not `p` or `inspect` a
1817
- `Mailer` or the `Config`, which hold it.
1818
- - **Every wait is bounded**, as in "Outbound HTTP": connecting, each TLS
1819
- step and each reply waits at most `timeout` seconds (default 10), the
1820
- whole delivery at most `deadline` seconds (default 60), else
1821
- `Mailer::TimeoutError`. Name resolution blocks the calling thread.
1822
- - **The dialogue:** the 220 greeting, `EHLO` (the name is `helo:`, default
1823
- `localhost`), `STARTTLS` and `EHLO` again, `AUTH PLAIN` when a user is
1824
- set (the server must offer PLAIN), `MAIL FROM`, one `RCPT TO` per
1825
- recipient, `DATA`, `QUIT`. A reply with another code is a
1826
- `Mailer::RejectedError` whose `smtp_code` is the server's (421, 535,
1827
- 550, ...) and whose message has the step and the server's text. The
1828
- return value is the server's reply to the message (`"250 2.0.0 Ok: queued
1829
- as ..."`).
1830
- - **The message:** `Date`, `From`, `To`, `Subject`, `MIME-Version: 1.0`,
1831
- `Content-Type: text/plain; charset=UTF-8`. A non-ASCII subject is RFC 2047
1832
- encoded (`=?UTF-8?B?...?=`, split between characters). The body's line
1833
- endings become CRLF and lines starting with `.` are dot-stuffed. Its
1834
- `Content-Transfer-Encoding` is `7bit` for ASCII, `8bit` (with `MAIL FROM
1835
- ... BODY=8BITMIME`) for UTF-8 when the server advertises 8BITMIME, and
1836
- `base64` otherwise or when a line is longer than 998 bytes. No
1837
- `Message-ID` is added (submission servers add one).
1838
- - **What is refused** with an `ArgumentError`, before connecting: an
1839
- address that is not a bare ASCII `local@domain` (no display name, no
1840
- `<>`, spaces, commas, quotes, CR or LF; at most 254 bytes), a subject with
1841
- CR, LF or NUL (header injection) or over 998 bytes, a body that is not
1842
- valid UTF-8 or holds a NUL, more than 100 recipients. `to:` takes one
1843
- address or several separated by commas.
1844
- - **Not yet:** HTML and multipart messages, attachments, templates, display
1845
- names, `Cc`/`Bcc` headers (send to several recipients instead), AUTH
1846
- LOGIN/CRAM-MD5, SMTPUTF8 addresses, pipelining, and a delivery queue (a
1847
- delivery runs on the calling thread, a green thread like a request).
1848
- `cybertrain generate authentication` still writes no password reset.
1849
-
1850
- The design: [docs/design.md](docs/design.md) D27.
1851
-
1852
- ## Testing an app
1853
-
1854
- App tests are plain Spinel programs under `test/`, like the framework's own:
1855
-
1856
- ```ruby
1857
- require_relative "support/blog_test" # the app, BLOG and Cybertrain::Test
1858
-
1859
- test "title must be present" do
1860
- article = Article.new(body: "a body long enough to pass length")
1861
- refute article.save
1862
- assert_includes article.errors.full_messages, "Title can't be blank"
1863
- end
1864
- ```
1865
-
1866
- Assertions: `assert`, `refute`, `assert_equal`, `assert_nil`,
1867
- `assert_includes`, `assert_raises("Class") { }`, `flunk`. For controllers and
1868
- full HTTP flows, `Cybertrain::Test::Client` drives an app in-process (no
1869
- socket) with a cookie jar, so sessions and CSRF behave as behind a browser:
1870
-
1871
- ```ruby
1872
- require_relative "support/blog_test" # the app, BLOG and Cybertrain::Test
1873
-
1874
- test "POST /articles creates an article and redirects to it" do
1875
- client = Cybertrain::Test::Client.new(BLOG)
1876
- client.get("/articles/new")
1877
- res = client.post("/articles", { "authenticity_token" => BlogTest.form_token(client),
1878
- "article[title]" => "Hello Rails",
1879
- "article[body]" => "I am on Rails! This is my first article." })
1880
- assert_redirected_to res, BlogTest.article_path(Article.last)
1881
- assert_response res, :see_other
1882
- end
1883
- ```
1884
-
1885
- Both snippets start from the example's helper,
1886
- [`examples/blog/test/support/blog_test.rb`](examples/blog/test/support/blog_test.rb):
1887
- copy it into an app's `test/support/`, where files are required, not run.
1888
- It migrates a fresh `storage/test.sqlite3`, boots the app as `BLOG`, loads
1889
- `Cybertrain::Test` and its client, and reads the CSRF token out of the last
1890
- rendered form (`BlogTest.form_token`). End each test file with
1891
- `Cybertrain::Test.run!`; `examples/blog/test/articles.rb` and `comments.rb`
1892
- show the full pattern.
1893
-
1894
- Every request a test client sends comes from `127.0.0.1` (`request.remote_addr`,
1895
- so `request.loopback?` is true). To see what a page does for a visitor on
1896
- another machine, set `client.peer_address = "203.0.113.7"` before the
1897
- request.
1898
-
1899
- Run with `cybertrain spin test`: it compiles each `test/*.rb` into its own
1900
- program and diffs its output (stdout and stderr together) against
1901
- `test/<name>.rb.expected`. Regenerate with `cybertrain spin test --regen
1902
- test/<name>.rb`; FFI/database tests can't run under CRuby, so their snapshot
1903
- comes from the compiled binary instead: `rm -f build/test/<name>; cybertrain
1904
- spin test test/<name>.rb; [ -x build/test/<name> ] && ./build/test/<name> >
1905
- test/<name>.rb.expected 2>&1` (`cybertrain spin test` exits non-zero while the
1906
- snapshot is stale, so it is followed by `;`, not `&&`; the `[ -x ]` check skips
1907
- the capture when the compile failed, so the old snapshot is kept; in the
1908
- framework checkout, `script/regen-snapshot test/<name>.rb` does exactly this and
1909
- prints the diff stat).
1910
-
1911
- ### Testing outbound calls
1912
-
1913
- When the code under test sends a webhook ([Outbound HTTP](#outbound-http)) or
1914
- an e-mail ([Sending e-mail](#sending-e-mail)), the test receives it
1915
- in-process: `Cybertrain::Test::HttpReceiver` and `Cybertrain::Test::SmtpReceiver`
1916
- listen on `127.0.0.1`, record what arrives and answer with a canned reply.
1917
-
1918
- ```ruby
1919
- require_relative "support/app_test"
1920
- require "cybertrain/test/http_receiver"
1921
- require "cybertrain/test/smtp_receiver"
1922
-
1923
- test "a sign-up posts a webhook" do
1924
- hook = Cybertrain::Test::HttpReceiver.new(status: 202, body: "queued")
1925
- hook.start # binds a free port, returns it
1926
- Cybertrain.config.http_allow_private = true # HttpClient refuses loopback otherwise
1927
- SignupHook.post(hook.url("/hooks/signup"), "ann@example.com") # the app's code
1928
- req = hook.received_request # waits at most 5 s; nil if nothing came
1929
- hook.stop
1930
- assert_equal "POST", req.http_method
1931
- assert_equal "/hooks/signup", req.http_target
1932
- assert_equal "application/json", req.http_header("Content-Type")
1933
- assert_includes req.http_body, "ann@example.com"
1934
- end
1935
-
1936
- test "a refused recipient is reported" do
1937
- smtp = Cybertrain::Test::SmtpReceiver.new(reject: { rcpt: "550 5.1.1 no such user" })
1938
- port = smtp.start
1939
- mailer = Cybertrain::Mailer.new(host: "127.0.0.1", port: port, tls: "none", from: "app@example.com")
1940
- assert_raises("RejectedError") { mailer.deliver(to: "nobody@example.org", subject: "Hi", body: "Hello\n") }
1941
- mail = smtp.received_message
1942
- smtp.stop
1943
- assert_equal ["EHLO localhost", "MAIL FROM:<app@example.com>", "RCPT TO:<nobody@example.org>"],
1944
- mail.transcript.first(3)
1945
- refute mail.delivered?
1946
- end
1947
- ```
1948
-
1949
- `HttpReceiver.new` takes the canned `status:`, `headers:`, `body:` and
1950
- `chunked:` (`respond_with` changes them between requests); the request comes
1951
- back as a `ReceivedRequest` (`http_method`, `http_target`, `http_path`,
1952
- `http_headers` with lowercased names, `http_header(name)`, `http_body` with a
1953
- chunked body decoded). `SmtpReceiver` plays 220, EHLO (with 8BITMIME unless
1954
- `eight_bit_mime: false`, AUTH PLAIN with `auth: true`), MAIL, RCPT, DATA and
1955
- QUIT; `reject:` fails any of `:greeting`, `:ehlo`, `:mail`, `:rcpt`, `:data`
1956
- or `:message` with the reply given. A `ReceivedMessage` has the
1957
- `envelope_from`, `envelope_recipients`, the `message_data` (dot-stuffing
1958
- undone), `message_header(name)`, `message_body` and the `transcript` of
1959
- commands. Both accept one connection unless `max:` says more
1960
- (`received_requests` / `received_messages` return them all). Every wait is
1961
- bounded: a receiver gives up after `accept_window:` (10 s) without a
1962
- connection, a silent client after `read_timeout:` (5 s), and `stop` (safe
1963
- to call twice) ends and joins the receiver's thread and closes its port, so
1964
- a receiver never takes a later test's connection. They need only Ruby's
1965
- socket library, not openssl.
1966
-
1967
- A feed is tested the same way with `Cybertrain::Test::WebSocketReceiver`, a
1968
- scripted WebSocket server on `127.0.0.1`: each connection plays a list of
1969
- `Cybertrain::Test::WsStep`s (`"accept"`, `"send_text"`, `"expect"`,
1970
- `"silent"`, `"close"`, `"drop"`, `"echo_until_close"`, ...), and the frames
1971
- the client sent come back unmasked from `ws_received`.
1972
-
1973
- ```ruby
1974
- require "cybertrain/websocket"
1975
- require "cybertrain/test/websocket_receiver"
1976
-
1977
- TICKS = Array.new(0) { "" } # a constant: the blocks cannot see locals
1978
-
1979
- # The blocks are written in a method, where their parameters are used,
1980
- # not inside the test block.
1981
- def subscribe_trades(url)
1982
- Cybertrain::WebSocket.subscribe("trades", url, allow_private: true)
1983
- .on_open { |conn| conn.send_text("subscribe") }
1984
- .on_message { |conn, text| TICKS << text }
1985
- end
1986
-
1987
- test "a trade message is stored" do
1988
- rx = Cybertrain::Test::WebSocketReceiver.new
1989
- rx.ws_script([Cybertrain::Test::WsStep.new("accept"), Cybertrain::Test::WsStep.new("expect"),
1990
- Cybertrain::Test::WsStep.new("send_text", "tick"), Cybertrain::Test::WsStep.new("echo_until_close")])
1991
- rx.start
1992
- subscribe_trades(rx.url("/trades"))
1993
- Cybertrain::WebSocket.start! # Test::Client never starts feeds
1994
- # ... poll TICKS for at most a few seconds ...
1995
- Cybertrain::WebSocket.stop!
1996
- assert_equal ["tick"], TICKS
1997
- assert_equal "subscribe", rx.ws_received[0].rec_data
1998
- rx.stop
1999
- end
2000
- ```
2001
-
2002
- `Cybertrain::WebSocket.random_source = -> { 0.5 }` makes the retry delays exact.
2003
- `test/websocket_integration.rb` in this repository shows a complete test.
2004
-
2005
- ### Continuous integration
2006
-
2007
- `cybertrain new` writes `.github/workflows/ci.yml` for the app. It calls the
2008
- `setup-cybertrain` action of this repository
2009
- ([.github/actions/setup-cybertrain](.github/actions/setup-cybertrain/action.yml)),
2010
- pinned to the CLI's release tag, which installs the SQLite, zlib and OpenSSL
2011
- headers and Ruby, reads the framework commit the app is locked to out of
2012
- `spin.lock`, fetches the framework at that commit and runs its `cybertrain
2013
- doctor` and `cybertrain setup` (with `~/.cybertrain/spinel` cached per Spinel
2014
- release and runner image). The workflow then checks that `spin.lock` is
2015
- current (`spin lock && git diff --exit-code spin.lock`) and `gen/` is fresh
2016
- (`spin run gen -- --check`), and runs `spin test` once `test/` has a test
2017
- program. When the toolchain steps change, the action changes, not every app's
2018
- copy. The commit is read from `spin.lock`, so the workflow assumes the
2019
- default git dependency; an app made with `--path` passes the framework
2020
- checkout to the action as `framework-path`.
2021
-
2022
- ## Learn more
2023
-
2024
- - [Homepage](https://saeki-mototsune.github.io/CyberTrain/),
2025
- [tutorial](https://saeki-mototsune.github.io/CyberTrain/tutorial.html) and
2026
- [playground](https://saeki-mototsune.github.io/CyberTrain/playground.html) —
2027
- the source lives in [site/](site/), published by
2028
- [.github/workflows/pages.yml](.github/workflows/pages.yml); the logo files and
2029
- brand notes are in [site/assets/brand/](site/assets/brand/BRAND.md).
2030
- - [API reference](https://saeki-mototsune.github.io/CyberTrain/api/) — every
2031
- class and method an application uses (controllers, params, session and
2032
- flash, models and relations, routes, migrations, configuration, tests),
2033
- including what `spin run gen` generates. It is built by YARD
2034
- (`script/api-docs`, into `site/api/`) from the doc comments marked
2035
- `@api public` in `cybertrain/` and from [docs/api/](docs/api/); a method
2036
- without that tag is internal and stays out. CI fails on any YARD warning.
2037
- - [docs/design.md](docs/design.md) — the design record (Japanese): every
2038
- decision, what was rejected and why, and the Spinel constraints behind it.
2039
- - [docs/template-language.md](docs/template-language.md) — the full template
2040
- grammar and helpers, and where it differs from Rails' ERB.
2041
- - [docs/benchmark.md](docs/benchmark.md) — examples/blog measured against the
2042
- same blog in Rails 8.1 ([bench/](bench/)), and how to run it again.
2043
- - [spikes/NOTES.md](spikes/NOTES.md) — the spikes that answered
2044
- `docs/design.md`'s open questions, and the compiler constraints they found.
2045
- - [Spinel](https://github.com/matz/spinel) — the AOT Ruby compiler cybertrain targets;
2046
- cybertrain pins a release of the
2047
- [saeki-mototsune/spinel](https://github.com/saeki-mototsune/spinel) fork.
2048
- - [Roundhouse](https://github.com/rubys/roundhouse) (Sam Ruby) — transpiles
2049
- *existing* Rails apps to Spinel. cybertrain is not that: it's a native
2050
- framework you write directly against, not a compatibility layer.
91
+ | [Tutorial](https://saeki-mototsune.github.io/CyberTrain/tutorial.html) | Build the blog step by step: scaffold, comments, authentication, uploads, deployment |
92
+ | [Installing](https://saeki-mototsune.github.io/CyberTrain/guide/install.html) | Requirements, the `cybertrain` command, how Spinel and the framework are installed |
93
+ | [How it works](https://saeki-mototsune.github.io/CyberTrain/guide/how-it-works.html) | Build-time code generation, interpreted views, the server, the development loop, and a table of the differences from Rails |
94
+ | [Models and the database](https://saeki-mototsune.github.io/CyberTrain/guide/models.html) | Migrations, column types, queries, associations, validations, encrypted columns, transactions, taking over a Rails database |
95
+ | [Requests](https://saeki-mototsune.github.io/CyberTrain/guide/requests.html) | Routing, formats and `respond_to`, concerns, strong parameters, HTTP caching, cookies and the session, rate limiting, CSP, `force_ssl` and hosts, headers, query strings and request bodies |
96
+ | [Authentication](https://saeki-mototsune.github.io/CyberTrain/guide/authentication.html) | `generate authentication`: sign-in, sign-up or invite-only, password reset, confirmation, lockout, remember me, the account page, admin users, upgrading from 0.4 |
97
+ | [Assets](https://saeki-mototsune.github.io/CyberTrain/guide/assets.html) | Digested stylesheets, images and JavaScript, CSS `url()`, the helpers, the import map, serving |
98
+ | [Internationalization](https://saeki-mototsune.github.io/CyberTrain/guide/i18n.html) | Dictionaries in `config/locales`, `t` and `l`, lazy lookup, plural forms, the locale per request, translated validation messages |
99
+ | [Configuration](https://saeki-mototsune.github.io/CyberTrain/guide/configuration.html) | `config/app.rb`, environment variables, time zones |
100
+ | [Background work and operations](https://saeki-mototsune.github.io/CyberTrain/guide/background-and-operations.html) | Job classes, retries and concurrency keys (delivered at least once, so make jobs idempotent), `every` and `cron` tasks, the jobs page, the `worker` and `feeds` process roles, the cache store, the console, backups, the health check |
101
+ | [Realtime](https://saeki-mototsune.github.io/CyberTrain/guide/realtime.html) | Turbo Streams over Server-Sent Events or WebSocket, broadcasts from actions, jobs and feeds, signed stream names, feed health |
102
+ | [Outbound HTTP, WebSocket feeds and e-mail](https://saeki-mototsune.github.io/CyberTrain/guide/external-services.html) | `HttpClient` (streaming, gzip, redirects, Server-Sent Events), `WebSocket.subscribe` (timers, restarts), `Mailer` (HTML mail, `deliver_later`) |
103
+ | [Testing](https://saeki-mototsune.github.io/CyberTrain/guide/testing.html) | Test classes, fixtures, `cybertrain test`, time travel, components and streams, mail, jobs, outbound calls, CI |
104
+ | [Deploying](https://saeki-mototsune.github.io/CyberTrain/deploy.html) | To a server with systemd, or with Kamal |
105
+ | [API reference](https://saeki-mototsune.github.io/CyberTrain/api/) | Every class and method an application uses; also [the template language](https://saeki-mototsune.github.io/CyberTrain/guide/template-language.html) |
106
+ | [Design record](docs/design.md) · [benchmark](docs/benchmark.md) · [spikes](spikes/NOTES.md) | Why it is built this way, and the Spinel constraints behind it |
107
+
108
+ The guides' Markdown sources are in [docs/guide/](docs/guide/).
2051
109
 
2052
110
  ## Contributing
2053
111
 
2054
- Bug reports, questions and pull requests are welcome. [CONTRIBUTING.md](CONTRIBUTING.md)
2055
- explains how to set up a checkout, run the tests and the commit gate
2056
- (`script/install-hooks` once per clone; `script/gate` runs the same checks CI
2057
- does), and what a pull request needs. Security issues go through
2058
- [SECURITY.md](SECURITY.md), not an issue.
2059
-
2060
- ## Releasing
2061
-
2062
- The gem and the framework are released from the same tag, and `cybertrain
2063
- new` depends on the tag `v` + `Cybertrain::VERSION`, so the tag must exist
2064
- before the gem is pushed:
2065
-
2066
- 1. Bump `Cybertrain::VERSION` (`cybertrain/version.rb`) and `version` in
2067
- `spin.toml` together (CI checks they match); spin caches a git
2068
- dependency by that version.
2069
- 2. Merge to `main`, then tag and push: `git tag vX.Y.Z && git push origin vX.Y.Z`.
2070
- 3. `gem build cybertrain.gemspec && gem push cybertrain-X.Y.Z.gem`.
2071
- 4. The tag also publishes the playground image
2072
- `ghcr.io/saeki-mototsune/cybertrain-playground:X.Y.Z` (and `latest`) through
2073
- [.github/workflows/playground-image.yml](.github/workflows/playground-image.yml);
2074
- check that run.
2075
-
2076
- `Cybertrain::SPINEL_TAG` (also in `cybertrain/version.rb`) is the Spinel
2077
- release `cybertrain setup` installs. It must equal `SPINEL_TAG` in
2078
- [.github/workflows/ci.yml](.github/workflows/ci.yml) and in
2079
- [script/claude-cloud-setup](script/claude-cloud-setup), which `script/gate`
2080
- checks, so change the three together when the framework moves to another
2081
- Spinel release.
112
+ Bug reports, questions and pull requests are welcome; see
113
+ [CONTRIBUTING.md](CONTRIBUTING.md) (setup, tests, the commit gate, releasing).
114
+ Security issues go through [SECURITY.md](SECURITY.md).
2082
115
 
2083
116
  ## License
2084
117
 
2085
- MIT; see [LICENSE](LICENSE). Contributions are accepted under the same license.
118
+ MIT; see [LICENSE](LICENSE).