cybertrain 0.3.1 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +82 -2049
- data/cybertrain/cli/attachment.rb +246 -164
- data/cybertrain/cli/authentication.rb +2405 -898
- data/cybertrain/cli/authentication_tests.rb +3175 -0
- data/cybertrain/cli/build.rb +93 -7
- data/cybertrain/cli/component.rb +118 -0
- data/cybertrain/cli/helper.rb +143 -0
- data/cybertrain/cli/job.rb +159 -0
- data/cybertrain/cli/line_diff.rb +216 -0
- data/cybertrain/cli/new_app.rb +17 -7
- data/cybertrain/cli/scaffold.rb +29 -1
- data/cybertrain/cli/templates.rb +405 -29
- data/cybertrain/cli/test_command.rb +301 -0
- data/cybertrain/cli/toolchain.rb +12 -12
- data/cybertrain/cli.rb +145 -24
- data/cybertrain/crypto.rb +136 -0
- data/cybertrain/ident.rb +9 -1
- data/cybertrain/shell.rb +13 -0
- data/cybertrain/test/args.rb +286 -0
- data/cybertrain/test_files.rb +48 -0
- data/cybertrain/version.rb +2 -2
- metadata +11 -2
data/README.md
CHANGED
|
@@ -7,133 +7,57 @@
|
|
|
7
7
|
<p align="center">
|
|
8
8
|
<a href="https://saeki-mototsune.github.io/CyberTrain/">Homepage</a> ·
|
|
9
9
|
<a href="https://saeki-mototsune.github.io/CyberTrain/tutorial.html">Tutorial</a> ·
|
|
10
|
+
<a href="https://saeki-mototsune.github.io/CyberTrain/guide/">Guides</a> ·
|
|
10
11
|
<a href="https://saeki-mototsune.github.io/CyberTrain/api/">API reference</a>
|
|
11
12
|
</p>
|
|
12
13
|
|
|
14
|
+
|
|
13
15
|
# cybertrain
|
|
14
16
|
|
|
15
|
-
|
|
17
|
+
A Rails-shaped web framework written natively for
|
|
16
18
|
[Spinel](https://github.com/matz/spinel), matz's ahead-of-time Ruby compiler.
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
quota: GitHub's free plan includes 120 core-hours and 15 GB-month of storage a
|
|
60
|
-
month, about 60 hours on the default 2-core machine the playground uses. GitHub
|
|
61
|
-
stops an idle codespace after 30 minutes by default; its storage counts until
|
|
62
|
-
you delete it.
|
|
63
|
-
- If VS Code asks whether you trust the authors of the files in this folder,
|
|
64
|
-
choose "Trust Folder & Continue": the server's terminal starts only then.
|
|
65
|
-
- If the app's tab did not open (a pop-up blocker, or the codespace was stopped
|
|
66
|
-
and restarted), click "Open in Browser" in VS Code's notification for port
|
|
67
|
-
3000 or in the Ports view, or Ctrl-click the app's URL in the `server`
|
|
68
|
-
terminal. `PLAYGROUND.md` shows where each is.
|
|
69
|
-
- The same image runs locally (it is published for linux/amd64; on arm64, build
|
|
70
|
-
it from a checkout):
|
|
71
|
-
|
|
72
|
-
```sh
|
|
73
|
-
docker run --rm -it --init -p 127.0.0.1:3000:3000 -e CYBERTRAIN_HOST=0.0.0.0 ghcr.io/saeki-mototsune/cybertrain-playground
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
Then open http://localhost:3000. [playground/README.md](playground/README.md)
|
|
77
|
-
describes the image, building it and the Codespaces setup.
|
|
78
|
-
|
|
79
|
-
## Requirements
|
|
80
|
-
|
|
81
|
-
- Spinel `2026.10.27.7020` (`Cybertrain::SPINEL_TAG`, the same release as
|
|
82
|
-
`SPINEL_TAG` in [.github/workflows/ci.yml](.github/workflows/ci.yml)), a
|
|
83
|
-
release of [saeki-mototsune/spinel](https://github.com/saeki-mototsune/spinel)
|
|
84
|
-
(`Cybertrain::SPINEL_REPOSITORY`), the fork of matz/spinel that carries
|
|
85
|
-
the fixes cybertrain needs ahead of an upstream release;
|
|
86
|
-
untested against other versions. You do not install it by hand: the
|
|
87
|
-
`cybertrain` command builds that release into `~/.cybertrain` the first
|
|
88
|
-
time it needs it (`cybertrain setup` does that on request), unless a
|
|
89
|
-
`spinel` of that release is already on `PATH` — see "Installing the CLI".
|
|
90
|
-
- To build it: a C toolchain (`cc`, or the compiler `CC` names) and `make`
|
|
91
|
-
— Spinel compiles every program to C — plus `git` and `curl`.
|
|
92
|
-
- SQLite 3 headers/library (`libsqlite3-dev` on Debian/Ubuntu; present with
|
|
93
|
-
Xcode's command line tools on macOS) — models use Spinel's FFI directly,
|
|
94
|
-
not a gem.
|
|
95
|
-
- zlib headers/library (`zlib1g-dev` on Debian/Ubuntu; in the macOS SDK) —
|
|
96
|
-
gzip request bodies (`request.decoded_body`) are inflated through the
|
|
97
|
-
system zlib, bound the same way.
|
|
98
|
-
- Ruby 3.2+ and RubyGems, for the `cybertrain` command only (applications
|
|
99
|
-
never run on CRuby). Building Spinel also needs Ruby's `prism` gem, which
|
|
100
|
-
Ruby 3.3 and newer bundle; on Ruby 3.2, `gem install prism` (with Ruby's
|
|
101
|
-
development headers, `ruby-dev` on Debian/Ubuntu). `cybertrain doctor`
|
|
102
|
-
checks it.
|
|
103
|
-
- OpenSSL's headers (`libssl-dev`) are optional: without them Spinel builds
|
|
104
|
-
without its `openssl` package, which cybertrain does not use. On macOS an
|
|
105
|
-
application that requires `openssl` needs Homebrew's (keg-only) OpenSSL on
|
|
106
|
-
the compiler's paths: `export LIBRARY_PATH="$(brew --prefix openssl@3)/lib"
|
|
107
|
-
CPATH="$(brew --prefix openssl@3)/include"`.
|
|
108
|
-
- jemalloc's development package (`libjemalloc-dev`; `brew install jemalloc`)
|
|
109
|
-
is optional: an app needs it only once its `spin.toml` says
|
|
110
|
-
`allocator = "jemalloc"`, which `cybertrain new` writes commented out. It
|
|
111
|
-
is worth turning on for production: 1.1 to 1.4 times the requests per second on
|
|
112
|
-
examples/blog ([docs/benchmark.md](docs/benchmark.md)).
|
|
113
|
-
|
|
114
|
-
`cybertrain doctor` checks these tools and headers and the Spinel install;
|
|
115
|
-
when a required one is missing it also prints the install command for your
|
|
116
|
-
platform.
|
|
117
|
-
|
|
118
|
-
The framework has no separate test runner; it tests itself. `spin` comes
|
|
119
|
-
with Spinel, and `cybertrain spin` runs the copy `cybertrain` installs
|
|
120
|
-
without touching `PATH` (or put that copy on `PATH` with the `export
|
|
121
|
-
PATH=...` line `cybertrain setup` prints, and call `spin` directly):
|
|
122
|
-
|
|
123
|
-
```sh
|
|
124
|
-
cybertrain spin test
|
|
125
|
-
```
|
|
126
|
-
|
|
127
|
-
compiles and runs every program under `test/` and diffs its output against a
|
|
128
|
-
committed `test/<name>.rb.expected` snapshot — minitest/RSpec can't run under
|
|
129
|
-
Spinel, since they find test methods by reflection and an ahead-of-time
|
|
130
|
-
compiler has nothing to reflect on at run time. `cybertrain spin test
|
|
131
|
-
--regen test/<name>.rb` rewrites a snapshot from CRuby's output; a few tests that
|
|
132
|
-
touch SQLite through FFI take theirs from the compiled binary instead
|
|
133
|
-
([spikes/NOTES.md](spikes/NOTES.md), rule 23); `script/regen-snapshot
|
|
134
|
-
test/<name>.rb` does that build-then-capture step for you and prints the diff stat.
|
|
135
|
-
|
|
136
|
-
## Installing the CLI
|
|
19
|
+
`cybertrain build` compiles your controllers, models, views and the framework
|
|
20
|
+
into **one native binary**: no Rack, no gems at runtime, no Ruby interpreter
|
|
21
|
+
in production. The Rails vocabulary (`before_action`, `Article.where(...)`,
|
|
22
|
+
`resources :articles`) comes from plain Ruby generated at build time instead
|
|
23
|
+
of runtime metaprogramming.
|
|
24
|
+
|
|
25
|
+
**Status:** pre-alpha. Routing (namespaces, scopes, formats and
|
|
26
|
+
`respond_to`), controllers with concerns, `helper_method`, strong
|
|
27
|
+
parameters, HTTP caching, cookies and an encrypted session, CSRF, a Content
|
|
28
|
+
Security Policy, `force_ssl` and host checks, ERB-style templates with
|
|
29
|
+
components, application helpers, the number, form and Turbo helpers and
|
|
30
|
+
per-controller layouts, digested assets with an import map, SQLite models
|
|
31
|
+
(decimal, json, date and datetime columns; declared associations with
|
|
32
|
+
`includes`; Rails' validators; model concerns; encrypted columns; bulk
|
|
33
|
+
inserts and typed aggregates), migrations, i18n (Rails' `t` and `l`,
|
|
34
|
+
dictionaries compiled in, English and Japanese bundled), and the
|
|
35
|
+
authentication generator (sign-in, sign-up or invite-only, password reset,
|
|
36
|
+
confirmation, lockout, remember me, an account page, admin users) and the
|
|
37
|
+
file-upload generator all work end to end. So do background jobs (job classes with retry
|
|
38
|
+
policies, cron schedules, a jobs page), a cache store, Turbo Streams over
|
|
39
|
+
Server-Sent Events or WebSocket, and the outbound clients: HTTP (streamed,
|
|
40
|
+
gzip and Server-Sent Events responses), SMTP (plain or HTML mail, sent now
|
|
41
|
+
or from a job) and WebSocket feeds, and the tests: Rails' test classes
|
|
42
|
+
(`ActiveSupport::TestCase`, `ActionDispatch::IntegrationTest`,
|
|
43
|
+
`ViewComponent::TestCase`) with `setup` / `teardown`, fixtures,
|
|
44
|
+
`travel_to`, `assert_select`, broadcast, mail and job assertions, all run
|
|
45
|
+
by `cybertrain test` as one compiled program, and the time helpers
|
|
46
|
+
(`5.minutes.ago`, `beginning_of_day`) in a configured time zone. [examples/blog](examples/blog) is the
|
|
47
|
+
Rails "Getting Started" blog built on it, in English and Japanese, with an
|
|
48
|
+
admin user guarding its `/admin` pages. Since 2026-11-05 porting an
|
|
49
|
+
existing Rails application is a goal, in five phases
|
|
50
|
+
([design record](docs/design.md), D1); a SQLite database that a Rails
|
|
51
|
+
application created can be taken over as it is. Not built: mailer
|
|
52
|
+
classes (a mailer is a module whose methods return a `MailMessage` to
|
|
53
|
+
`deliver_later`), mail attachments, HTTP/2
|
|
54
|
+
([what differs from Rails](https://saeki-mototsune.github.io/CyberTrain/guide/how-it-works.html#differences-from-rails)).
|
|
55
|
+
|
|
56
|
+
## Quick start
|
|
57
|
+
|
|
58
|
+
Needs Ruby 3.2+, a C toolchain, `make`, `git`, `curl` and the SQLite, zlib and
|
|
59
|
+
OpenSSL headers ([details](https://saeki-mototsune.github.io/CyberTrain/guide/install.html#requirements); `cybertrain doctor`
|
|
60
|
+
checks them).
|
|
137
61
|
|
|
138
62
|
```sh
|
|
139
63
|
gem install cybertrain
|
|
@@ -142,1944 +66,53 @@ cd blog
|
|
|
142
66
|
cybertrain g scaffold article title:string body:text
|
|
143
67
|
cybertrain db migrate
|
|
144
68
|
cybertrain server # http://127.0.0.1:3000
|
|
145
|
-
cybertrain build # production: dist/
|
|
146
|
-
```
|
|
147
|
-
|
|
148
|
-
The `cybertrain` command is plain Ruby and the gem runs it under CRuby
|
|
149
|
-
(3.2 or newer); neither the framework nor Spinel is in the gem. Its
|
|
150
|
-
commands (`cybertrain help` prints them):
|
|
151
|
-
|
|
152
|
-
- `cybertrain new NAME [--path DIR | --version V | --git URL [--ref R]]
|
|
153
|
-
[--skip-spin]` creates an application in NAME, then runs `spin lock` and
|
|
154
|
-
`spin run gen` in it, installing Spinel first when it is needed.
|
|
155
|
-
- `cybertrain generate scaffold NAME field:type ...` (alias `g scaffold`)
|
|
156
|
-
adds a resource to the application in the current directory.
|
|
157
|
-
- `cybertrain generate authentication` adds email-and-password sign-in to
|
|
158
|
-
it: users and sessions tables, `User` and `Session` models, the sign-in
|
|
159
|
-
and sign-up controllers and views, the routes, a `before_action` in
|
|
160
|
-
`ApplicationController` and a test (see "Add authentication").
|
|
161
|
-
With `--invite-only` there is no sign-up page: users are created with
|
|
162
|
-
`./NAME user EMAIL`, and the first one also at a `/setup` page that
|
|
163
|
-
answers only to the server's own machine, with the one-time token its
|
|
164
|
-
boot warning prints, while no user exists.
|
|
165
|
-
- `cybertrain generate attachment MODEL FIELD [--any]` adds a file upload
|
|
166
|
-
to a scaffolded model: five metadata columns, model methods and
|
|
167
|
-
callbacks, a download action, the route, the form field and a test (see
|
|
168
|
-
"Add a file upload"). A scaffold field `cover:attachment` does the same.
|
|
169
|
-
- `cybertrain db migrate` generates, applies pending migrations, and
|
|
170
|
-
generates again (`spin run gen; spin run db -- migrate; spin run gen`);
|
|
171
|
-
`cybertrain migration` is kept as an alias.
|
|
172
|
-
- `cybertrain db COMMAND...` runs any other database command after `spin
|
|
173
|
-
run gen`: `status`, `rollback [N]` (which generates again afterwards),
|
|
174
|
-
`schema:dump`, `create`.
|
|
175
|
-
- `cybertrain server [PORT]` generates, then starts the development server
|
|
176
|
-
(`spin run gen; spin run NAME`, or `spin run NAME -- PORT`; default port
|
|
177
|
-
3000).
|
|
178
|
-
- `cybertrain build` builds NAME with `app/views/` embedded and assembles
|
|
179
|
-
`dist/` (the binary, `public/`, `storage/`, `tmp/`).
|
|
180
|
-
- `cybertrain spin ARGS...` runs `spin` with the pinned Spinel from any
|
|
181
|
-
directory (`cybertrain spin test`, `cybertrain spin run gen -- --check`).
|
|
182
|
-
- `cybertrain setup [--force]` installs Spinel `2026.10.27.7020` into
|
|
183
|
-
`~/.cybertrain` unless a `spinel` of that release is already on `PATH`;
|
|
184
|
-
`--force` rebuilds the copy under `~/.cybertrain`.
|
|
185
|
-
- `cybertrain doctor` checks the C toolchain, the SQLite headers and the
|
|
186
|
-
Spinel install.
|
|
187
|
-
- `cybertrain version` prints the version.
|
|
188
|
-
|
|
189
|
-
`db`, `server` and `build` run inside an application; NAME is the
|
|
190
|
-
`[package] name` in its `spin.toml`.
|
|
191
|
-
|
|
192
|
-
Every command that runs `spin` (`new`, `db`, `server`, `build`, `spin`) first makes
|
|
193
|
-
sure a Spinel of the release in `Cybertrain::SPINEL_TAG` is available. A
|
|
194
|
-
`spinel` on `PATH` (with a `spin`) whose `spinel --version` reports that
|
|
195
|
-
release is used as it is; otherwise the copy `cybertrain` keeps under
|
|
196
|
-
`~/.cybertrain/spinel/2026.10.27.7020/` is used; otherwise `cybertrain`
|
|
197
|
-
builds that copy from the release tag of `Cybertrain::SPINEL_REPOSITORY`
|
|
198
|
-
(`git clone`, `make deps`, `make -j`, `make install PREFIX=...`), once, in a
|
|
199
|
-
few minutes. The build log is
|
|
200
|
-
`~/.cybertrain/log/spinel-2026.10.27.7020-build.log` (its last lines are printed
|
|
201
|
-
if a step fails); if a tool the build needs is missing, `cybertrain` says
|
|
202
|
-
which and prints the install command for your platform before it starts. A
|
|
203
|
-
`spinel` of another release on `PATH` is never used or touched:
|
|
204
|
-
`cybertrain` keeps its own copy (and says so when it installs it).
|
|
205
|
-
|
|
206
|
-
`cybertrain setup` does the same on request and, whenever its own copy is
|
|
207
|
-
the one in use, prints the line that puts it on `PATH`, for running `spin`
|
|
208
|
-
yourself (`~/.cybertrain/bin` holds `spinel` and `spin` links to the release
|
|
209
|
-
in use, so the line survives a release bump; `cybertrain spin ...` needs no
|
|
210
|
-
`PATH` at all):
|
|
211
|
-
|
|
212
|
-
```sh
|
|
213
|
-
export PATH="$HOME/.cybertrain/bin:$PATH"
|
|
214
|
-
```
|
|
215
|
-
|
|
216
|
-
Two environment variables override all this: `CYBERTRAIN_HOME` replaces
|
|
217
|
-
`~/.cybertrain` (a path without spaces, quotes, `$` or `:`, which Spinel's
|
|
218
|
-
Makefile and `PATH` cannot take), and `CYBERTRAIN_SPINEL_HOME=PREFIX` forces
|
|
219
|
-
an existing install — `cybertrain` then uses the `spinel` and `spin` in `PREFIX/bin` (or
|
|
220
|
-
in `PREFIX` itself, if `spinel` is directly in it) and stops with an error
|
|
221
|
-
if they are not that release.
|
|
222
|
-
|
|
223
|
-
`cybertrain new` writes the application with its `spin.toml` pointing at
|
|
224
|
-
the release matching the CLI (`cybertrain = { git =
|
|
225
|
-
"https://github.com/saeki-mototsune/cybertrain", ref = "v0.3.1" }`), makes
|
|
226
|
-
sure Spinel is installed, then runs `spin lock` (spin fetches the framework
|
|
227
|
-
into its cache, `~/.cache/spin/packages/`, and pins the commit in
|
|
228
|
-
`spin.lock`) and `spin run gen`, much as `rails new` runs `bundle install`,
|
|
229
|
-
and prints the next steps. Nothing of the framework is copied into the app;
|
|
230
|
-
commit `spin.toml` and `spin.lock`. If Spinel or one of the two `spin`
|
|
231
|
-
steps fails, the files stay, `new` exits with 1 and prints what to run once
|
|
232
|
-
the cause is fixed (for Spinel: `cybertrain setup`, then `cybertrain spin
|
|
233
|
-
lock && cybertrain spin run gen` in the new directory).
|
|
234
|
-
|
|
235
|
-
`--skip-spin` only writes the files: no Spinel install, no `spin lock`, no
|
|
236
|
-
`spin run gen`. `--path DIR` depends on a local checkout instead,
|
|
237
|
-
`--git URL [--ref R]` on the framework at URL (at branch or tag R), and
|
|
238
|
-
`--version V` on an index version once cybertrain is on a `spin-index`.
|
|
239
|
-
Give only one of `--path`, `--version` and `--git`; `--ref` needs `--git`.
|
|
240
|
-
|
|
241
|
-
Working on cybertrain itself, run the CLI from the checkout (`spin install`
|
|
242
|
-
builds `bin/cybertrain.rb` into `~/.local/bin/cybertrain`; `ruby -I.
|
|
243
|
-
bin/cybertrain.rb` also works) and create apps against it with `cybertrain
|
|
244
|
-
new NAME --path ~/src/cybertrain` (wherever the checkout is).
|
|
245
|
-
|
|
246
|
-
## Walkthrough: building a blog
|
|
247
|
-
|
|
248
|
-
Mirrors Rails' own [Getting
|
|
249
|
-
Started](https://guides.rubyonrails.org/getting_started.html): a blog with
|
|
250
|
-
articles and comments.
|
|
251
|
-
|
|
252
|
-
### Create the application
|
|
253
|
-
|
|
254
|
-
```sh
|
|
255
|
-
cybertrain new blog
|
|
256
|
-
cd blog
|
|
257
|
-
cybertrain server # works right away: `new` already locked the framework and ran `spin run gen`
|
|
258
|
-
```
|
|
259
|
-
|
|
260
|
-
```
|
|
261
|
-
blog/
|
|
262
|
-
spin.toml config/app.rb config/routes.rb
|
|
263
|
-
db/schema.rb db/migrate/
|
|
264
|
-
app/controllers/application_controller.rb app/models/ app/views/ app/helpers/
|
|
265
|
-
public/ bin/blog.rb bin/gen.rb bin/db.rb
|
|
266
|
-
gen/ storage/ tmp/ test/
|
|
267
|
-
```
|
|
268
|
-
|
|
269
|
-
`bin/blog.rb` is the application's one entry point, the program `spin build
|
|
270
|
-
blog` compiles: `./blog` serves, `./blog migrate` and `./blog db status`
|
|
271
|
-
manage the database. `bin/gen.rb` runs the generator (`spin run gen`), and
|
|
272
|
-
`bin/db.rb` runs the migrator during development (`cybertrain db migrate`
|
|
273
|
-
uses it, since `bin/blog.rb` cannot compile until the first migration has
|
|
274
|
-
generated `gen/models/`). `gen/` already holds `views.rb`, the committed,
|
|
275
|
-
empty table of embedded views (see "How it works").
|
|
276
|
-
|
|
277
|
-
Operational commands of your own (create a user, prune old rows, take a
|
|
278
|
-
backup) go on the same binary: register them in `bin/blog.rb` before
|
|
279
|
-
`Main.run`.
|
|
280
|
-
|
|
281
|
-
```ruby
|
|
282
|
-
Cybertrain::Main.command("prune", "delete comments older than DAYS") do |argv|
|
|
283
|
-
days = argv.empty? ? 30 : argv[0].to_i
|
|
284
|
-
Cybertrain::DB.with do |c|
|
|
285
|
-
c.execute("DELETE FROM comments WHERE created_at < datetime('now', ?)", ["-#{days} days"])
|
|
286
|
-
end
|
|
287
|
-
0 # the exit status
|
|
288
|
-
end
|
|
289
|
-
```
|
|
290
|
-
|
|
291
|
-
`./blog prune 90` then runs the block with the arguments after the name
|
|
292
|
-
(`["90"]`), with `Cybertrain::DB` already connected to the configured
|
|
293
|
-
database (a pool of one connection), and exits with the block's value; a
|
|
294
|
-
`Cybertrain::DB::Error` is printed as `prune failed: ...` and exits 1, and
|
|
295
|
-
`./blog help` lists the command. The block takes `argv` explicitly, as a
|
|
296
|
-
`before_action` block takes the controller (Spinel has no `instance_eval`).
|
|
297
|
-
A built-in word (`server`, `migrate`, `db`, `help`), a number (the port
|
|
298
|
-
argument) or a name already registered raises `ArgumentError`. The server,
|
|
299
|
-
views and session secret are not set up for a command.
|
|
300
|
-
|
|
301
|
-
Apps created before `cybertrain build` existed have `bin/server.rb` and
|
|
302
|
-
no `bin/<name>.rb`: copy `bin/<name>.rb` from a fresh `cybertrain new` (with
|
|
303
|
-
the same NAME), delete `bin/server.rb`, add `/dist/` to `.gitignore` (the old
|
|
304
|
-
template did not ignore it) and run `cybertrain spin run gen`.
|
|
305
|
-
|
|
306
|
-
`ApplicationController` starts with the one thing every controller inherits:
|
|
307
|
-
|
|
308
|
-
```ruby
|
|
309
|
-
class ApplicationController < Cybertrain::Controller
|
|
310
|
-
rescue_from Cybertrain::RecordNotFound, with: :record_not_found
|
|
311
|
-
|
|
312
|
-
private
|
|
313
|
-
|
|
314
|
-
def record_not_found
|
|
315
|
-
render plain: "Not Found", status: :not_found
|
|
316
|
-
end
|
|
317
|
-
end
|
|
318
|
-
```
|
|
319
|
-
|
|
320
|
-
### Scaffold an article
|
|
321
|
-
|
|
322
|
-
```sh
|
|
323
|
-
cybertrain generate scaffold article title:string body:text
|
|
324
|
-
```
|
|
325
|
-
|
|
326
|
-
writes a migration, a model, a controller and five views, and inserts
|
|
327
|
-
`resources :articles` into `config/routes.rb`. Point root at it and, as
|
|
328
|
-
Rails' guide does, add a length validation by hand:
|
|
329
|
-
|
|
330
|
-
```ruby
|
|
331
|
-
# config/routes.rb
|
|
332
|
-
Cybertrain::Routes.draw do
|
|
333
|
-
root "articles#index"
|
|
334
|
-
resources :articles
|
|
335
|
-
end
|
|
336
|
-
```
|
|
337
|
-
|
|
338
|
-
```ruby
|
|
339
|
-
# app/models/article.rb -- only the first string field gets a default
|
|
340
|
-
# presence validation; add anything past that yourself.
|
|
341
|
-
class Article
|
|
342
|
-
validates :title, presence: true
|
|
343
|
-
validates :body, presence: true, length: { minimum: 10 }
|
|
344
|
-
end
|
|
345
|
-
```
|
|
346
|
-
|
|
347
|
-
The controller is full CRUD, Rails-shaped down to strong params and
|
|
348
|
-
flash-then-redirect, except for two things Spinel forces everywhere: `new` is
|
|
349
|
-
written `new_action` (a method literally named `new` would shadow
|
|
350
|
-
`ArticlesController.new(ctx)`), and every action/callback is called by its
|
|
351
|
-
literal name, never `send`:
|
|
352
|
-
|
|
353
|
-
```ruby
|
|
354
|
-
# app/controllers/articles_controller.rb (index/show/edit are as plain as
|
|
355
|
-
# Rails' own scaffold; update mirrors create)
|
|
356
|
-
class ArticlesController < ApplicationController
|
|
357
|
-
before_action :set_article, only: [:show, :edit, :update, :destroy]
|
|
358
|
-
|
|
359
|
-
def new_action # a `new` method would shadow ArticlesController.new
|
|
360
|
-
@article = Article.new
|
|
361
|
-
end
|
|
362
|
-
|
|
363
|
-
def create
|
|
364
|
-
@article = Article.new(article_params)
|
|
365
|
-
if @article.save
|
|
366
|
-
flash[:notice] = "Article was successfully created."
|
|
367
|
-
redirect_to article_path(@article), status: :see_other
|
|
368
|
-
else
|
|
369
|
-
render :new, status: :unprocessable_entity
|
|
370
|
-
end
|
|
371
|
-
end
|
|
372
|
-
|
|
373
|
-
private
|
|
374
|
-
|
|
375
|
-
def set_article
|
|
376
|
-
@article = Article.find(params[:id])
|
|
377
|
-
end
|
|
378
|
-
|
|
379
|
-
def article_params
|
|
380
|
-
params.require(:article).permit(:title, :body)
|
|
381
|
-
end
|
|
382
|
-
end
|
|
383
|
-
```
|
|
384
|
-
|
|
385
|
-
### Generate, migrate, run
|
|
386
|
-
|
|
387
|
-
```sh
|
|
388
|
-
cybertrain db migrate # spin run gen; spin run db -- migrate; spin run gen
|
|
389
|
-
cybertrain server # http://127.0.0.1:3000; rebuilds on Ruby edits, views reload without a rebuild
|
|
390
|
-
```
|
|
391
|
-
|
|
392
|
-
`spin run gen` reads `db/schema.rb` and `config/routes.rb`, scans `app/` and
|
|
393
|
-
writes `gen/`; `spin run db -- migrate` applies `db/migrate/*.rb` and
|
|
394
|
-
rewrites `db/schema.rb`.
|
|
395
|
-
|
|
396
|
-
Re-run `cybertrain spin run gen` (and commit `gen/`) after touching the
|
|
397
|
-
schema, routes, or a controller/model's callbacks and ivars — the dev server does this for
|
|
398
|
-
you automatically (see "How it works").
|
|
399
|
-
|
|
400
|
-
### Add comments, nested under articles
|
|
401
|
-
|
|
402
|
-
```sh
|
|
403
|
-
cybertrain generate scaffold comment commenter:string body:text article:references
|
|
404
|
-
cybertrain db migrate # the scaffold wrote a migration for the comments table
|
|
405
|
-
```
|
|
406
|
-
|
|
407
|
-
`article:references` adds an `article_id` column, index and foreign key —
|
|
408
|
-
enough for `Article`/`Comment` to get their association; there's no
|
|
409
|
-
`has_many`/`belongs_to` to write, `spin run gen` reads it off the foreign
|
|
410
|
-
key. This also inserts an independent `resources :comments` line; Rails'
|
|
411
|
-
guide nests comments under articles instead, so edit `config/routes.rb`:
|
|
412
|
-
|
|
413
|
-
```ruby
|
|
414
|
-
Cybertrain::Routes.draw do
|
|
415
|
-
root "articles#index"
|
|
416
|
-
|
|
417
|
-
# A nested block takes the mapper explicitly (Spinel's instance_eval
|
|
418
|
-
# trampoline only rewires `self` for the outermost block): `articles.resources`.
|
|
419
|
-
resources :articles do |articles|
|
|
420
|
-
articles.resources :comments, only: [:create, :destroy]
|
|
421
|
-
end
|
|
422
|
-
end
|
|
423
|
-
```
|
|
424
|
-
|
|
425
|
-
Trim `CommentsController` to the two actions the nested routes call,
|
|
426
|
-
resolving the parent from the URL instead of `:id`:
|
|
427
|
-
|
|
428
|
-
```ruby
|
|
429
|
-
class CommentsController < ApplicationController
|
|
430
|
-
before_action :set_article
|
|
431
|
-
|
|
432
|
-
def create
|
|
433
|
-
@comment = Comment.new(comment_params)
|
|
434
|
-
@comment.article_id = @article.id
|
|
435
|
-
if @comment.save
|
|
436
|
-
flash[:notice] = "Comment was successfully created."
|
|
437
|
-
else
|
|
438
|
-
flash[:alert] = "Comment could not be saved: #{@comment.errors.full_messages.join(", ")}"
|
|
439
|
-
end
|
|
440
|
-
redirect_to article_path(@article), status: :see_other
|
|
441
|
-
end
|
|
442
|
-
|
|
443
|
-
def destroy
|
|
444
|
-
@comment = Comment.where(article_id: @article.id).find(params[:id])
|
|
445
|
-
@comment.destroy
|
|
446
|
-
flash[:notice] = "Comment was successfully destroyed."
|
|
447
|
-
redirect_to article_path(@article), status: :see_other
|
|
448
|
-
end
|
|
449
|
-
|
|
450
|
-
private
|
|
451
|
-
|
|
452
|
-
def set_article
|
|
453
|
-
@article = Article.find(params[:article_id])
|
|
454
|
-
end
|
|
455
|
-
|
|
456
|
-
def comment_params
|
|
457
|
-
params.require(:comment).permit(:commenter, :body)
|
|
458
|
-
end
|
|
459
|
-
end
|
|
460
|
-
```
|
|
461
|
-
|
|
462
|
-
Delete the now-unreachable `index`/`show`/`new`/`edit` comment views, give
|
|
463
|
-
`ArticlesController#show` a blank `@comment` to bind against, and add a list
|
|
464
|
-
and a form to the article page:
|
|
465
|
-
|
|
466
|
-
```erb
|
|
467
|
-
<%# app/views/articles/show.html.erb %>
|
|
468
|
-
<% @article.comments.each do |comment| %>
|
|
469
|
-
<%= render "comments/comment", comment: comment %>
|
|
470
|
-
<% end %>
|
|
471
|
-
<%= render "comments/form" %>
|
|
472
|
-
|
|
473
|
-
<%# app/views/comments/_form.html.erb %>
|
|
474
|
-
<%= form_with(model: [@article, @comment]) do |f| %>
|
|
475
|
-
<%= f.label :commenter %> <%= f.text_field :commenter %>
|
|
476
|
-
<%= f.label :body %> <%= f.text_area :body %>
|
|
477
|
-
<%= f.submit %>
|
|
478
|
-
<% end %>
|
|
479
|
-
```
|
|
480
|
-
|
|
481
|
-
`comments/_comment.html.erb` destroys with `button_to "Destroy comment",
|
|
482
|
-
[@article, comment], method: :delete`. Both that and `form_with(model:
|
|
483
|
-
[@article, @comment])` resolve through the same `[parent, child]` rule: a new
|
|
484
|
-
child routes to the nested collection (`article_comments_path`), a persisted
|
|
485
|
-
one to the nested member (`article_comment_path`) — generated into
|
|
486
|
-
`gen/routes.rb` from the nested `resources` block above.
|
|
487
|
-
|
|
488
|
-
### Add authentication
|
|
489
|
-
|
|
490
|
-
```sh
|
|
491
|
-
cybertrain generate authentication
|
|
492
|
-
cybertrain db migrate
|
|
493
|
-
cybertrain server # then open http://127.0.0.1:3000/sign_up
|
|
494
|
-
```
|
|
495
|
-
|
|
496
|
-
Like Rails 8's `bin/rails generate authentication`, this writes code the
|
|
497
|
-
app owns rather than installing an engine: migrations for `users`
|
|
498
|
-
(`email_address`, `password_digest`, a unique index) and `sessions` (one
|
|
499
|
-
row per signed-in browser, with `ip_address` and `user_agent`), the `User`
|
|
500
|
-
and `Session` models, `SessionsController` (`/session/new`, `POST
|
|
501
|
-
/session`, `DELETE /session`), `RegistrationsController` (`/sign_up`,
|
|
502
|
-
because there is no console to create the first user in; delete it for an
|
|
503
|
-
invitation-only app), their views, a `sessions/_nav` partial rendered at
|
|
504
|
-
the top of the layout, and `test/authentication.rb` with its
|
|
505
|
-
`test/support/app_test.rb`. It also inserts a block after the class line of
|
|
506
|
-
`ApplicationController`:
|
|
507
|
-
|
|
508
|
-
```ruby
|
|
509
|
-
class ApplicationController < Cybertrain::Controller
|
|
510
|
-
before_action :resume_session # sets @current_user and @current_session
|
|
511
|
-
before_action :require_authentication # 303 to /session/new unless signed in
|
|
512
|
-
|
|
513
|
-
def self.allow_unauthenticated_access(only: [], except: [])
|
|
514
|
-
skip_before_action(:require_authentication, only: only, except: except)
|
|
515
|
-
end
|
|
516
|
-
# authenticated?, start_new_session_for(user), terminate_session,
|
|
517
|
-
# after_authentication_url ...
|
|
518
|
-
```
|
|
519
|
-
|
|
520
|
-
so every action needs a signed-in user unless its controller says
|
|
521
|
-
`allow_unauthenticated_access only: [:new, :create]`, as the two generated
|
|
522
|
-
controllers do. Templates read `@current_user` (`<% if @current_user %>`).
|
|
523
|
-
Passwords are hashed by `Cybertrain::Password` (PBKDF2-HMAC-SHA256, 100,000
|
|
524
|
-
iterations by default, about 0.2 s compiled; raise it with
|
|
525
|
-
`Cybertrain::Password.iterations = 600_000` in `config/app.rb`, and
|
|
526
|
-
`User#authenticate` rehashes each user as they sign in). The `User` model
|
|
527
|
-
spells out what `has_secure_password` would define (`password=`,
|
|
528
|
-
`authenticate`, `User.authenticate_by(email_address:, password:)`), since
|
|
529
|
-
Spinel cannot define methods at run time; edit it like any other model.
|
|
530
|
-
Password reset is not generated (no mailers). Running the generator again
|
|
531
|
-
writes nothing. The design, and why not a package: [docs/design.md](docs/design.md) D15.
|
|
532
|
-
|
|
533
|
-
Sign-in is protected from password guessing the way Rails 8's generator
|
|
534
|
-
does it: `SessionsController` declares `rate_limit to: 10, within: 3 * 60,
|
|
535
|
-
only: [:create], redirect_to: "/session/new", alert: "Try again later."`,
|
|
536
|
-
so an address that has tried ten times in three minutes goes back to the
|
|
537
|
-
form until the window closes. Behind a reverse proxy on the same machine,
|
|
538
|
-
set `CYBERTRAIN_TRUSTED_PROXY=1`, or every visitor shares the proxy's
|
|
539
|
-
address (see [Rate limiting](#rate-limiting-and-the-client-address)); behind
|
|
540
|
-
kamal-proxy, the generated `config/deploy.yml` already sets
|
|
541
|
-
`CYBERTRAIN_TRUSTED_PROXIES` to the Docker network.
|
|
542
|
-
`AppTest.sign_in` forgets the counts first, since every test client comes
|
|
543
|
-
from 127.0.0.1.
|
|
544
|
-
|
|
545
|
-
#### An invitation-only app: `--invite-only`
|
|
546
|
-
|
|
547
|
-
The sign-up page lets anyone who reaches the server register, which is the
|
|
548
|
-
wrong default for an internal tool: whoever gets there before the admin
|
|
549
|
-
does has an account. `--invite-only` writes no `RegistrationsController`
|
|
550
|
-
and no sign-up link; the first user comes from the command line or from a
|
|
551
|
-
page only the server's own machine can open, with a link the server prints
|
|
552
|
-
when it starts:
|
|
553
|
-
|
|
554
|
-
```sh
|
|
555
|
-
cybertrain generate authentication --invite-only
|
|
556
|
-
cybertrain db migrate
|
|
557
|
-
cybertrain server # then open the http://127.0.0.1:3000/setup?token=... link it prints, on the same machine
|
|
558
|
-
```
|
|
559
|
-
|
|
560
|
-
In production, from the directory the binary runs in:
|
|
561
|
-
|
|
562
|
-
```sh
|
|
563
|
-
./shop user ann@example.com # reads the password from stdin (one line)
|
|
564
|
-
printf '%s' "$PASSWORD" | ./shop user ann@example.com
|
|
565
|
-
./shop user ann@example.com 's3cr3t-s3cr3t' # or as an argument (visible in ps)
|
|
566
|
-
```
|
|
567
|
-
|
|
568
|
-
- `app/commands/user_command.rb` registers the `user` command with
|
|
569
|
-
`Cybertrain::Main.command` (so `./shop help` lists it) and is required
|
|
570
|
-
from `bin/shop.rb`. `EMAIL [PASSWORD]` creates the user, or sets the
|
|
571
|
-
password of the one with that address (which signs that user's browsers
|
|
572
|
-
out, as any password change does); it prints `created ADDRESS` or
|
|
573
|
-
`updated ADDRESS` and exits 0. Without `PASSWORD` it reads one line of
|
|
574
|
-
standard input (prompting when that is a terminal; the typing is echoed)
|
|
575
|
-
and drops the trailing newline, so the password stays out of `ps` and
|
|
576
|
-
the shell history. An empty password, a record that does not validate
|
|
577
|
-
(a password under 8 characters) or bad arguments print an error or the
|
|
578
|
-
usage line and exit 1.
|
|
579
|
-
- `SetupController` (`GET`/`POST /setup`, `app/views/setup/new.html.erb`)
|
|
580
|
-
creates the first user and signs them in. It answers only while no user
|
|
581
|
-
exists, only with the one-time token (below), and only to a connection
|
|
582
|
-
whose socket peer is the loopback address (`request.loopback?`:
|
|
583
|
-
`127.0.0.0/8` or `::1`) and that carries no `X-Forwarded-For` header (a
|
|
584
|
-
proxy on the same host makes every request a loopback one, and it adds
|
|
585
|
-
that header). A header can close the page, never open it. Everyone else,
|
|
586
|
-
and a missing or wrong token, gets a 404.
|
|
587
|
-
- The token: `SetupController::SETUP_TOKEN`, 32 random hex characters
|
|
588
|
-
(`Cybertrain::Crypto.random_token(16)`) made when the program starts and
|
|
589
|
-
kept until it exits, so a restart makes a new one. `GET /setup` takes it
|
|
590
|
-
as `?token=`, the form carries it back in a hidden field, and both
|
|
591
|
-
compare it in constant time (`Cybertrain::Crypto.secure_compare`). It is
|
|
592
|
-
what keeps the page closed when the address check cannot: a proxy on the
|
|
593
|
-
same host that adds no `X-Forwarded-For` (nginx without
|
|
594
|
-
`proxy_set_header X-Forwarded-For ...`) makes every client look like
|
|
595
|
-
loopback, and only someone who can read the server's log has the token.
|
|
596
|
-
The request log's `Started GET "/setup?token=..."` line writes it to the
|
|
597
|
-
same log; it opens nothing once the user exists.
|
|
598
|
-
- For a first run behind a proxy on another host, `SHOP_SETUP_OPEN=1`
|
|
599
|
-
(declared in `config/app.rb` with `c.setting_bool :setup_open`) lifts the
|
|
600
|
-
address check, still only while no user exists and still only with the
|
|
601
|
-
token. Unset it once the user is there.
|
|
602
|
-
- While no user exists, the server logs `[WARN] no user exists yet: run
|
|
603
|
-
./shop user EMAIL, or open /setup?token=<token> from this machine` before
|
|
604
|
-
it boots (`UserCommand.warn_if_no_user`, called from `bin/shop.rb` when
|
|
605
|
-
`Cybertrain::Main.server?(ARGV)`; it connects for that one question and
|
|
606
|
-
says nothing when the users table is not migrated yet). That line is the
|
|
607
|
-
only place the token is printed.
|
|
608
|
-
|
|
609
|
-
The generated `test/authentication.rb` covers both: `/setup` with and
|
|
610
|
-
without the token (`SetupController.setup_token`) or with a wrong one, from
|
|
611
|
-
another address (`client.peer_address = "203.0.113.7"`), through a proxy,
|
|
612
|
-
once a user exists and with the switch on, and the `user` command run in-process
|
|
613
|
-
through `Cybertrain::Main.run`. All of it is application code, yours to
|
|
614
|
-
change (docs/design.md D29); the framework only provides
|
|
615
|
-
`request.loopback?`, `Main.command`, `Main.server?`,
|
|
616
|
-
`Crypto.random_token` / `Crypto.secure_compare` and the test client's
|
|
617
|
-
`peer_address`.
|
|
618
|
-
|
|
619
|
-
In the blog, open the pages anyone may read again, as Rails' guide does, and
|
|
620
|
-
hide what a visitor cannot use (`resume_session` still runs on those pages,
|
|
621
|
-
so `@current_user` is set):
|
|
622
|
-
|
|
623
|
-
```ruby
|
|
624
|
-
# app/controllers/articles_controller.rb
|
|
625
|
-
class ArticlesController < ApplicationController
|
|
626
|
-
allow_unauthenticated_access only: [:index, :show]
|
|
627
|
-
before_action :set_article, only: [:show, :edit, :update, :destroy]
|
|
628
|
-
```
|
|
629
|
-
|
|
630
|
-
```erb
|
|
631
|
-
<%# app/views/articles/show.html.erb (excerpt) %>
|
|
632
|
-
<% if @current_user %>
|
|
633
|
-
<%= button_to "Destroy this article", article_path(@article), method: :delete %>
|
|
634
|
-
<% end %>
|
|
635
|
-
|
|
636
|
-
<% if @current_user %>
|
|
637
|
-
<h2>Add a comment:</h2>
|
|
638
|
-
<%= render "comments/form" %>
|
|
639
|
-
<% end %>
|
|
640
|
-
```
|
|
641
|
-
|
|
642
|
-
A test that expects a form, submits one, or reaches an action other than
|
|
643
|
-
`index` and `show` now meets the sign-in page: sign the client in first
|
|
644
|
-
with the helpers in `test/support/app_test.rb`.
|
|
645
|
-
|
|
646
|
-
```ruby
|
|
647
|
-
require_relative "support/app_test" # written by generate authentication: APP, AppTest
|
|
648
|
-
|
|
649
|
-
test "a signed-in user creates an article" do
|
|
650
|
-
client = Cybertrain::Test::Client.new(APP)
|
|
651
|
-
AppTest.user("ann@example.com", "s3cr3t-s3cr3t")
|
|
652
|
-
AppTest.sign_in(client, "ann@example.com", "s3cr3t-s3cr3t")
|
|
653
|
-
client.get("/articles/new")
|
|
654
|
-
res = client.post("/articles", { "authenticity_token" => AppTest.form_token(client),
|
|
655
|
-
"article[title]" => "Hello Rails",
|
|
656
|
-
"article[body]" => "I am on Rails! This is my first article." })
|
|
657
|
-
assert_redirected_to res, "/articles/#{Article.last.id}"
|
|
658
|
-
end
|
|
659
|
-
```
|
|
660
|
-
|
|
661
|
-
`examples/blog` itself stops before this step.
|
|
662
|
-
|
|
663
|
-
## Authentication: what the framework provides
|
|
664
|
-
|
|
665
|
-
The generator above writes the flow; the pieces it is built from are
|
|
666
|
-
framework API, usable without it (for an API token, an admin password, a
|
|
667
|
-
hand-written sign-in):
|
|
668
|
-
|
|
669
|
-
- **`Cybertrain::Password`**: `hash(plain)` makes a salted
|
|
670
|
-
PBKDF2-HMAC-SHA256 digest that records its own parameters
|
|
671
|
-
(`$pbkdf2-sha256$i=100000$salt$key`), `verify?(plain, digest)` checks one
|
|
672
|
-
in constant time and never raises on a malformed digest, and
|
|
673
|
-
`needs_rehash?(digest)` says when one is behind the current
|
|
674
|
-
`Password.iterations`. For a sign-in action:
|
|
675
|
-
- `verify_serialized(plain, digest)` is `verify?` one at a time per
|
|
676
|
-
process (a Mutex). A hash takes about 0.2 s of CPU; a burst of attempts
|
|
677
|
-
side by side would hold every worker, through this method they queue and
|
|
678
|
-
the other workers keep serving pages.
|
|
679
|
-
- `dummy_digest` is a digest of a random password, made once per process
|
|
680
|
-
at the current cost. Verify against it when the address is unknown, so
|
|
681
|
-
the response time does not tell which addresses have accounts:
|
|
682
|
-
`Cybertrain::Password.verify_serialized(password, user.nil? ? Cybertrain::Password.dummy_digest : user.password_digest)`.
|
|
683
|
-
- `session_key(digest)` is an HMAC of the password digest under
|
|
684
|
-
`secret_key_base`; keep it in the session at sign-in and check it with
|
|
685
|
-
`session_key_matches?(session[:password_key], user.password_digest)` on
|
|
686
|
-
each request. A password change makes a new digest, so every session
|
|
687
|
-
started before it is signed out.
|
|
688
|
-
- `iterations` is `Password::TEST_ITERATIONS` (1,000) when
|
|
689
|
-
`Cybertrain.config.test?`, so test suites do not pay the production cost
|
|
690
|
-
per user; digests record their count, so nothing else changes.
|
|
691
|
-
- **The session** (`session[:key]`, a signed cookie): `clear` empties it
|
|
692
|
-
(CSRF token included) on sign-out, and `config.session_absolute_max_age`
|
|
693
|
-
(two weeks; `0` turns it off) bounds a session's whole life however often
|
|
694
|
-
it is used: the cookie records when it was first written, a cookie older
|
|
695
|
-
than that loads as an empty session, so a copied cookie cannot be
|
|
696
|
-
replayed forever. `session.restart_lifetime!` starts the count again (at
|
|
697
|
-
sign-in).
|
|
698
|
-
- **`Controller#local_path?(text)` and `safe_redirect_path(text, fallback)`**:
|
|
699
|
-
a path that came from the request (a `return_to` parameter, a remembered
|
|
700
|
-
page) is redirected to only when it starts with `/` but not `//` or `/\`,
|
|
701
|
-
holds printable ASCII only and is at most 2,048 bytes:
|
|
702
|
-
`redirect_to safe_redirect_path(params[:return_to], "/"), status: :see_other`.
|
|
703
|
-
- **`skip_before_action(name, only:, except:)`** on a controller class drops
|
|
704
|
-
an inherited `before_action` for some actions; the generated
|
|
705
|
-
`allow_unauthenticated_access` is a one-line wrapper around it.
|
|
706
|
-
- **`cybertrain generate authentication`** writes, as code the app owns: the
|
|
707
|
-
`users` and `sessions` migrations and models, `SessionsController`,
|
|
708
|
-
`RegistrationsController`, their views and nav partial, the
|
|
709
|
-
`ApplicationController` block (`resume_session`, `require_authentication`,
|
|
710
|
-
`start_new_session_for`, `terminate_session`, `after_authentication_url`)
|
|
711
|
-
and `test/authentication.rb`. That code uses everything in this list:
|
|
712
|
-
`User.authenticate_by` verifies with `verify_serialized` and the dummy
|
|
713
|
-
digest, `start_new_session_for` stores the password key and restarts the
|
|
714
|
-
lifetime, `resume_session` ends a Session row whose key no longer matches
|
|
715
|
-
or that is older than `session_absolute_max_age`, and
|
|
716
|
-
`after_authentication_url` is `safe_redirect_path`. To keep the browser
|
|
717
|
-
that changed a password signed in, store the new key after saving:
|
|
718
|
-
`session[:password_key] = Cybertrain::Password.session_key(user.password_digest)`.
|
|
719
|
-
The design: [docs/design.md](docs/design.md) D25.
|
|
720
|
-
|
|
721
|
-
### Add a file upload
|
|
722
|
-
|
|
723
|
-
```sh
|
|
724
|
-
cybertrain generate attachment article cover # or: generate scaffold article title:string cover:attachment
|
|
725
|
-
cybertrain db migrate
|
|
726
|
-
```
|
|
727
|
-
|
|
728
|
-
The same split as authentication: the framework parses
|
|
729
|
-
`multipart/form-data` and stores bytes, and the generator writes the rest
|
|
730
|
-
as code the app owns. It writes a migration adding five columns to
|
|
731
|
-
`articles` (`cover_key`, `cover_filename`, `cover_content_type`,
|
|
732
|
-
`cover_byte_size`, `cover_checksum`; the file itself is not in SQLite) and
|
|
733
|
-
a block in `Article` (`cover_file=`, `cover_attached?`, `cover_url`, a
|
|
734
|
-
validation and save/destroy callbacks). In `ArticlesController` it adds a
|
|
735
|
-
`cover` action behind `before_action :set_article` and one line in
|
|
736
|
-
`create` and `update`:
|
|
737
|
-
|
|
738
|
-
```ruby
|
|
739
|
-
@article.cover_file = params.nested(:article).upload(:cover)
|
|
740
|
-
```
|
|
741
|
-
|
|
742
|
-
It also adds `get "/articles/:id/cover"` to `config/routes.rb`, and to the
|
|
743
|
-
form `form_with(model: article, multipart: true)` with `f.file_field :cover`.
|
|
744
|
-
The show page gets an `<img>`, and `test/attachment_article_cover.rb` tests
|
|
745
|
-
the whole flow.
|
|
746
|
-
|
|
747
|
-
What is accepted and how it is served:
|
|
748
|
-
|
|
749
|
-
- **Types.** By default only PNG, JPEG, GIF and WebP up to 2 MB are
|
|
750
|
-
accepted (`COVER_TYPES` / `COVER_MAX_BYTES` in the model; `--any` accepts
|
|
751
|
-
any type). The type is detected from the file's first bytes, never taken
|
|
752
|
-
from the browser.
|
|
753
|
-
- **Storage.** The bytes go to `storage/uploads/<2 hex>/<32 hex key>`
|
|
754
|
-
(`Cybertrain::BlobStore`, mode 0600). Replacing a file deletes the old
|
|
755
|
-
one, and destroying the record deletes its file.
|
|
756
|
-
- **Serving.** The `cover` action serves files with `send_blob`, so the
|
|
757
|
-
controller's `before_action`s (sign-in, scoping) apply to downloads too.
|
|
758
|
-
The four image types are served inline; anything else is a download
|
|
759
|
-
(`application/octet-stream`). Like every response, they carry
|
|
760
|
-
`X-Content-Type-Options: nosniff` and `X-Frame-Options: SAMEORIGIN` (the
|
|
761
|
-
framework-wide defaults, below), and `send_blob` adds
|
|
762
|
-
`Content-Security-Policy: default-src 'none'; sandbox`. Never put uploads
|
|
763
|
-
under `public/`. The download is exactly as public as show: in the blog
|
|
764
|
-
above, where `allow_unauthenticated_access only: [:index, :show]` lets
|
|
765
|
-
visitors read articles, the generator adds `:cover` to that list, so the
|
|
766
|
-
image on a public page loads (only when the download loads the record
|
|
767
|
-
through the same `before_action :set_article` as show, and show checks
|
|
768
|
-
nothing more). The generator checks inherited callbacks too. If a policy
|
|
769
|
-
applies only to show, show contains inline checks, or the policy cannot
|
|
770
|
-
be read safely, generation stops before writing any files. Put authorization
|
|
771
|
-
in an explicit callback shared by show and the download, for example
|
|
772
|
-
`before_action :authorize_article, only: [:show, :cover]`, and load both
|
|
773
|
-
through `before_action :set_article`, then rerun the generator. Keeping a
|
|
774
|
-
download behind sign-in alone does not preserve record or role permissions.
|
|
775
|
-
Where show needs a sign-in, so does the download.
|
|
776
|
-
- **Params and CSRF.** `params.nested(:article).upload(:cover)` is a
|
|
777
|
-
`Cybertrain::UploadedFile` (`original_filename`, `byte_size`,
|
|
778
|
-
`detected_content_type`, `body_bytes`, `sha256_hex`), or nil when no
|
|
779
|
-
file was chosen. `permit` never returns files. The CSRF token and
|
|
780
|
-
`_method` are read from multipart bodies like any form.
|
|
781
|
-
- **Limits.** Three config keys bound uploads: `max_upload_bytes`,
|
|
782
|
-
`max_multipart_parts` and `upload_root` (see "Configuration"). A broken
|
|
783
|
-
or oversize body is a 400.
|
|
784
|
-
- **Not built yet:** several files per field, S3 or other remote storage,
|
|
785
|
-
image variants, and streaming large bodies to disk (the whole body is
|
|
786
|
-
read into memory, up to `max_body_bytes`).
|
|
787
|
-
|
|
788
|
-
Back up `storage/uploads/` together with the SQLite file. The design, and
|
|
789
|
-
why it is not a separate library: [docs/design.md](docs/design.md) D16.
|
|
790
|
-
|
|
791
|
-
### Build for deployment
|
|
792
|
-
|
|
793
|
-
```sh
|
|
794
|
-
cybertrain build
|
|
795
|
-
```
|
|
796
|
-
|
|
797
|
-
writes `dist/`: `dist/blog`, the app compiled with every `app/views/`
|
|
798
|
-
template embedded (rendered from memory, error messages still say
|
|
799
|
-
`articles/show.html.erb:12`), `dist/public/`, and empty `dist/storage/` and
|
|
800
|
-
`dist/tmp/`. Copy `dist/` to a machine with the same OS and CPU (the binary
|
|
801
|
-
links the system SQLite) and run `./blog migrate && ./blog`; a built binary
|
|
802
|
-
is production by default (`CYBERTRAIN_ENV` overrides). Set
|
|
803
|
-
`CYBERTRAIN_SECRET_KEY_BASE`. Static files can also be served by the reverse
|
|
804
|
-
proxy from `dist/public/` — see [docs/deploy.en.md](docs/deploy.en.md).
|
|
805
|
-
|
|
806
|
-
A running `cybertrain server` skips its rebuilds while `cybertrain build` is
|
|
807
|
-
in progress (the build holds `tmp/cybertrain-build.lock`, since both write
|
|
808
|
-
`build/bin/blog`), logs the skip, and rebuilds as usual on the next save
|
|
809
|
-
once the build has finished. The lock records the build's PID; one left
|
|
810
|
-
behind by a killed build is ignored and removed.
|
|
811
|
-
|
|
812
|
-
## How it works
|
|
813
|
-
|
|
814
|
-
**Build-time code generation.** `spin run gen` (`bin/gen.rb`) executes
|
|
815
|
-
`db/schema.rb`'s `create_table` DSL and `config/routes.rb`'s `Routes.draw`
|
|
816
|
-
DSL to get table and route *data*, and separately does a **lexical scan**
|
|
817
|
-
(regexes, not a parser — Spinel exposes none at run time) of
|
|
818
|
-
`app/controllers/**/*.rb` and `app/models/**/*.rb` for `@ivar =`
|
|
819
|
-
assignments, `before_action`/`after_action`/`rescue_from ... with:` names and
|
|
820
|
-
zero-argument `def`s. From that it writes plain, readable Ruby under `gen/`:
|
|
821
|
-
`gen/models/<name>.rb` (accessors, casts, finders, FK-derived associations),
|
|
822
|
-
`gen/routes.rb` (route table, dispatcher, `*_path`/`*_url` helpers),
|
|
823
|
-
`gen/controllers.rb` (`view_assigns` and `run_callback` per controller),
|
|
824
|
-
`gen/migrations.rb`, `gen/views.rb` (the embedded views table, empty unless
|
|
825
|
-
building for deployment) and `gen/app.rb` (the `require_relative` manifest).
|
|
826
|
-
`gen/` is committed, not gitignored — `spin build`/`spin test` have no hook
|
|
827
|
-
to generate first, so the checked-in output has to already be what compiles.
|
|
828
|
-
CI re-runs `spin run gen` and fails on a diff, so stale generated code can't
|
|
829
|
-
merge.
|
|
830
|
-
|
|
831
|
-
**Views are interpreted, not compiled.** `app/views/**/*.html.erb` files look
|
|
832
|
-
like Rails ERB but are parsed into an AST at request time, then walked by a
|
|
833
|
-
tree-walking interpreter. Where the source comes from depends on the mode:
|
|
834
|
-
in development (and test) the engine reads `app/views/` from disk, and in
|
|
835
|
-
development re-parses a template when it changes, so view edits need no rebuild.
|
|
836
|
-
`cybertrain build` runs `spin run gen -- --embed-views`, which writes every
|
|
837
|
-
template's source into `gen/views.rb` as a string table for that one `spin
|
|
838
|
-
build`, then runs `spin run gen` again to restore the empty table that is
|
|
839
|
-
committed; the production binary renders only that table (parsing each
|
|
840
|
-
template once) and refuses to boot when it is empty (`error: views are not
|
|
841
|
-
embedded in this binary`). Spinel has no `eval`, so this is a closed grammar (literals, `@ivar`/local lookups, calls
|
|
842
|
-
through fixed per-type dispatch tables, `if`/`unless`,
|
|
843
|
-
`each`/`each_with_index`/one helper block), not real Ruby. Full grammar,
|
|
844
|
-
helpers and gaps: [docs/template-language.md](docs/template-language.md).
|
|
845
|
-
|
|
846
|
-
**The HTTP server.** `Cybertrain::Server` is plain-Ruby HTTP/1.1:
|
|
847
|
-
`TCPServer#accept` plus one Spinel green thread per connection (M:N
|
|
848
|
-
scheduled, no GVL), typed `Request`/`Response`, no Rack layer. It calls an `HttpHandler` (`call(request) -> Response`); `ContextHandler` joins it to the middleware stack, so the HTTP layer (`cybertrain/http/`, plus `Params` and `Logger`) does not depend on the rest of the framework; `script/check-http-layer` enforces it in CI. TLS/HTTP/2 are
|
|
849
|
-
left to a reverse proxy. `SPINEL_WORKERS` defaults to `1` — spikes found that
|
|
850
|
-
faster and stall-free for this I/O-bound shape at 100 concurrent connections.
|
|
851
|
-
|
|
852
|
-
**SQLite via FFI.** Models talk to SQLite through Spinel's
|
|
853
|
-
`ffi_func`/`ffi_lib` directly — no C extension, no gem. A pool (`SizedQueue`,
|
|
854
|
-
4 connections by default) hands out connections with WAL, a busy timeout of
|
|
855
|
-
about five seconds (a busy handler retrying every 100 µs, where SQLite's
|
|
856
|
-
`busy_timeout` sleeps a millisecond or more), up to 64 cached prepared
|
|
857
|
-
statements and foreign keys on; every query is bound, never interpolated. It's the only
|
|
858
|
-
adapter today (PostgreSQL via `libpq` FFI is noted as possible future work).
|
|
859
|
-
|
|
860
|
-
**Transactions.** `Cybertrain::DB.transaction { |c| ... }` (or
|
|
861
|
-
`Article.transaction`, the same thing under the Rails name) checks a
|
|
862
|
-
connection out, runs `BEGIN`, yields the connection, `COMMIT`s when the block
|
|
863
|
-
returns and rolls back and re-raises when it raises. `immediate: true` runs
|
|
864
|
-
`BEGIN IMMEDIATE`, which takes SQLite's write lock at the start: use it for a
|
|
865
|
-
transaction that reads and then writes, so a concurrent writer makes it wait
|
|
866
|
-
at `BEGIN` rather than fail with "database is locked" at its first write.
|
|
867
|
-
While the block runs, everything on that thread that goes through the pool
|
|
868
|
-
(`save`, `update`, `destroy`, Relation queries, `DB.with`) uses the
|
|
869
|
-
transaction's connection, so it is part of the transaction:
|
|
870
|
-
|
|
871
|
-
```ruby
|
|
872
|
-
Article.transaction(immediate: true) do |c|
|
|
873
|
-
article.save!
|
|
874
|
-
c.execute("UPDATE authors SET articles_count = articles_count + 1 WHERE id = ?", [article.author_id])
|
|
875
|
-
end
|
|
69
|
+
cybertrain build # production: one binary in dist/
|
|
876
70
|
```
|
|
877
71
|
|
|
878
|
-
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
72
|
+
The binary runs every role a deployment needs: `./blog server` for HTTP and
|
|
73
|
+
the realtime endpoint (`turbo_stream_from @article` on a page,
|
|
74
|
+
`Cybertrain::Stream.broadcast_append_to` from an action or a job),
|
|
75
|
+
`./blog worker` for jobs and scheduled tasks (several may share the
|
|
76
|
+
database; jobs are delivered at least once), `./blog feeds` for WebSocket
|
|
77
|
+
subscriptions, and `./blog console` for a prompt over the models.
|
|
884
78
|
|
|
885
|
-
|
|
886
|
-
|
|
887
|
-
|
|
888
|
-
and `gen/**/*.rb` every half second (views are excluded — the engine reloads
|
|
889
|
-
those itself — and so is `gen/views.rb`, empty in development). A change runs `spin run gen && spin build NAME -- --incremental` in the
|
|
890
|
-
background (Spinel's incremental build, which recompiles only the C units the
|
|
891
|
-
edit changed; a Spinel release without `--incremental` rejects the flag, and the
|
|
892
|
-
loop then builds without it); success requests a restart (the same flag an external `kill -HUP`
|
|
893
|
-
sets): a monitor thread stops the listener and drains open requests, then the
|
|
894
|
-
main thread `execv`s the new binary on the same port and PID, invisible to a
|
|
895
|
-
client mid-session. A failed build keeps serving the old binary and banners the
|
|
896
|
-
compiler output on every HTML response. None of this loads in production.
|
|
79
|
+
Or try it with nothing installed:
|
|
80
|
+
[open the playground in GitHub Codespaces](https://codespaces.new/saeki-mototsune/CyberTrain?quickstart=1)
|
|
81
|
+
([what the playground holds](https://saeki-mototsune.github.io/CyberTrain/playground.html)).
|
|
897
82
|
|
|
898
|
-
|
|
83
|
+
On one CPU core, with 16 connections, the blog serves 2.2 to 8.8 times the requests per second of
|
|
84
|
+
the same app in Rails 8.1, in about a tenth of the memory
|
|
85
|
+
([numbers](https://saeki-mototsune.github.io/CyberTrain/index.html#performance)).
|
|
899
86
|
|
|
900
|
-
|
|
901
|
-
subclass. Its `change` records schema operations (`create_table`,
|
|
902
|
-
`add_column`, `add_index`, `rename_column`, `add_reference`, `remove_column`,
|
|
903
|
-
`remove_index`, `drop_table`); `cybertrain db migrate` applies each pending
|
|
904
|
-
version in its own transaction and rewrites `db/schema.rb` from the database,
|
|
905
|
-
and `cybertrain db rollback [N]` runs the inverse of `change`. When `change`
|
|
906
|
-
holds an operation with no automatic inverse, write `down` (and `up`).
|
|
87
|
+
## Documentation
|
|
907
88
|
|
|
908
|
-
|
|
909
|
-
[])`. It runs at its place in the migration, on the migrator's own
|
|
910
|
-
connection and in the same transaction as the version's schema operations,
|
|
911
|
-
so the column added before it exists when it runs, and a failure undoes the
|
|
912
|
-
whole version:
|
|
913
|
-
|
|
914
|
-
```ruby
|
|
915
|
-
class AddRetentionDaysToBackups < Cybertrain::Migration::Base
|
|
916
|
-
def change
|
|
917
|
-
add_column "backups", "retention_days", :integer, null: false, default: "90"
|
|
918
|
-
execute "UPDATE backups SET retention_days = ?", [0] # existing rows keep "forever"
|
|
919
|
-
end
|
|
920
|
-
|
|
921
|
-
def down
|
|
922
|
-
remove_column "backups", "retention_days"
|
|
923
|
-
end
|
|
924
|
-
end
|
|
925
|
-
```
|
|
926
|
-
|
|
927
|
-
Without binds the SQL may be several `;`-separated statements; with binds
|
|
928
|
-
it is one statement with `?` placeholders. `execute` has no inverse: rolling
|
|
929
|
-
back a `change` that executes SQL, with no `down` of its own, stops with
|
|
930
|
-
`version N is irreversible: it executes SQL in change; write down` before
|
|
931
|
-
anything is reverted. A data statement never appears in `db/schema.rb`,
|
|
932
|
-
which is dumped from the database. Recording a migration runs nothing, so
|
|
933
|
-
the statement runs exactly once per `migrate` of its version.
|
|
934
|
-
|
|
935
|
-
## Querying beyond `where`
|
|
936
|
-
|
|
937
|
-
The generated relations cover `where` / `where_sql` / `order` / `limit` /
|
|
938
|
-
`offset` / `count` / `exists?` / `delete_all`. For the counters, statistics
|
|
939
|
-
and search boxes most applications also need, there are typed additions
|
|
940
|
-
(issue #74, `docs/design.md` D22) rather than a query builder; each takes
|
|
941
|
-
SQL column names, checked and quoted (anything else is an `ArgumentError`),
|
|
942
|
-
and binds every value:
|
|
943
|
-
|
|
944
|
-
```ruby
|
|
945
|
-
# INSERT ... ON CONFLICT (project_id, key) DO UPDATE SET ...; returns rows written
|
|
946
|
-
Stat.upsert({ project_id: project.id, key: path, calls: 1, max_ms: ms, seen_at: Time.now },
|
|
947
|
-
conflict: [:project_id, :key],
|
|
948
|
-
update: { calls: "calls + excluded.calls", max_ms: "MAX(max_ms, excluded.max_ms)",
|
|
949
|
-
seen_at: "excluded.seen_at" })
|
|
950
|
-
Article.where(id: article.id).increment(:views) # UPDATE ... SET views = views + 1
|
|
951
|
-
Comment.where(article_id: article.id).pluck_int(:id) # => [3, 7] (also pluck_str, pluck_float)
|
|
952
|
-
Comment.all.count_by(:article_id) # => {"1" => 2, "4" => 1}
|
|
953
|
-
Comment.all.count_by_sql("strftime('%Y-%m-%dT%H', created_at)") # hour buckets
|
|
954
|
-
Stat.where(project_id: 1).sum_int(:calls) # also sum_float
|
|
955
|
-
Article.where_like(:title, params[:q].to_s) # `title` LIKE '%...%' ESCAPE '\'
|
|
956
|
-
```
|
|
957
|
-
|
|
958
|
-
- `upsert`'s `update` values and `count_by_sql`'s expression are SQL text, like
|
|
959
|
-
`order_sql`: never build them from request data. An empty `update` is
|
|
960
|
-
`DO NOTHING`. `upsert` and `increment` load no model and run no
|
|
961
|
-
validation or callback, and fill in no timestamps.
|
|
962
|
-
- `count_by` keys are Strings (through `Cast.str`), ascending; NULL and `""`
|
|
963
|
-
share the key `""`. With a `limit` / `offset`, `count_by`, `sum_*` and
|
|
964
|
-
`increment` work on that window, as `count` and `delete_all` do.
|
|
965
|
-
- `where_like` is a substring match (case-insensitive for ASCII, as SQLite's
|
|
966
|
-
LIKE is). A prefix search writes its own pattern:
|
|
967
|
-
`Article.all.where_sql("title LIKE ? ESCAPE '\\'", [Cybertrain::Cast.like_escape(q) + "%"])`.
|
|
968
|
-
- For hand-written SQL, `Cybertrain::Cast.binds(a, b, c)` builds the bind
|
|
969
|
-
Array `execute` takes (a Time as UTC ISO 8601 text, booleans as 1/0):
|
|
970
|
-
`Cybertrain::DB.with { |c| c.execute(sql, Cybertrain::Cast.binds(id, since)) }`.
|
|
971
|
-
- `Cybertrain::Cast.hash(v)` / `Cast.array(v)` narrow a parsed JSON value to
|
|
972
|
-
a Hash / Array, or a fresh empty one when it is not one (a Time is checked
|
|
973
|
-
first, spikes/NOTES.md rule 8). They return a typed shallow copy, not `v`
|
|
974
|
-
itself; `Cast.empty_hash` / `Cast.empty_array` are the seeded empty
|
|
975
|
-
containers.
|
|
976
|
-
|
|
977
|
-
## Pagination
|
|
978
|
-
|
|
979
|
-
A list page reads the page number from `?page=N`, windows the relation, and
|
|
980
|
-
renders the links:
|
|
981
|
-
|
|
982
|
-
```ruby
|
|
983
|
-
# app/controllers/articles_controller.rb
|
|
984
|
-
def index
|
|
985
|
-
@articles = Article.all.order("created_at DESC")
|
|
986
|
-
@page = @articles.page(params.page_number, per: 20)
|
|
987
|
-
end
|
|
988
|
-
```
|
|
989
|
-
|
|
990
|
-
```erb
|
|
991
|
-
<%# app/views/articles/index.html.erb %>
|
|
992
|
-
<% @articles.each do |article| %>
|
|
993
|
-
<p><%= link_to article.title, article %></p>
|
|
994
|
-
<% end %>
|
|
995
|
-
<%= paginate @page, articles_path %>
|
|
996
|
-
```
|
|
997
|
-
|
|
998
|
-
`params.page_number` is the 1-based `page` parameter, 1 when it is absent or
|
|
999
|
-
not a number, clamped to `1..max_page` (`config.max_page`, 10000 by default,
|
|
1000
|
-
`CYBERTRAIN_MAX_PAGE`); it never hands unbounded text to `String#to_i`, which
|
|
1001
|
-
raises `RangeError` past 2**63-1 under Spinel. `params.int(:per, default: 20,
|
|
1002
|
-
min: 1, max: 100)` is the same bounded read for any key (an optional `-` and
|
|
1003
|
-
at most 18 digits, else the default; clamped to `min..max`).
|
|
1004
|
-
`relation.page(n, per: 50)` sets `LIMIT per OFFSET (n - 1) * per` on the
|
|
1005
|
-
relation (replacing an earlier `limit`/`offset`, so call it last), clamps `n`
|
|
1006
|
-
the same way, so the offset never passes `(max_page - 1) * per`, and returns a
|
|
1007
|
-
`Cybertrain::Page`: `number`, `per`, `row_offset`, `next?` (asked with one
|
|
1008
|
-
`SELECT 1 ... LIMIT per + 1` over the window, no `COUNT(*)`), `prev?`,
|
|
1009
|
-
`next_number`, `prev_number`. The rows stay on the relation, so
|
|
1010
|
-
`@articles.each` yields typed `Article`s; `per` outside 1..1000 is an
|
|
1011
|
-
`ArgumentError`. `paginate @page, path` renders `<nav class="pager">` with a
|
|
1012
|
-
"Newer" link (`rel="prev"`), "Page N" and an "Older" link (`rel="next"`),
|
|
1013
|
-
each left out when there is no such page; the links keep the current query
|
|
1014
|
-
string with `page` replaced (page 1 drops it), re-encoded and escaped, so a
|
|
1015
|
-
search or filter survives paging, and a path with its own query string
|
|
1016
|
-
(`"/search?q=x"`) uses that one instead. Offset pagination only: keyset
|
|
1017
|
-
pagination (`after: id`) for very large tables is not built yet. The scaffold's
|
|
1018
|
-
index view lists every record; add the three lines above to page it.
|
|
1019
|
-
|
|
1020
|
-
## Rate limiting and the client address
|
|
1021
|
-
|
|
1022
|
-
`rate_limit` in a controller caps how often one client may call some
|
|
1023
|
-
actions, counted before the action runs:
|
|
1024
|
-
|
|
1025
|
-
```ruby
|
|
1026
|
-
class SessionsController < ApplicationController
|
|
1027
|
-
# a 303 back to the form, with flash[:alert], once exhausted
|
|
1028
|
-
rate_limit to: 10, within: 3 * 60, only: [:create],
|
|
1029
|
-
redirect_to: "/session/new", alert: "Try again later."
|
|
1030
|
-
end
|
|
1031
|
-
|
|
1032
|
-
class ApiKeysController < ApplicationController
|
|
1033
|
-
rate_limit to: 100, within: 60, only: [:create, :update] # 429 Too Many Requests + Retry-After
|
|
1034
|
-
end
|
|
1035
|
-
```
|
|
1036
|
-
|
|
1037
|
-
Each declaration counts at most `to` requests per `within` seconds for each
|
|
1038
|
-
client address and controller (a fixed window that opens with the first
|
|
1039
|
-
request), then answers 429 with a `Retry-After` header, or redirects when
|
|
1040
|
-
given `redirect_to:`. `within:` is in seconds, `only:`/`except:` take
|
|
1041
|
-
Arrays, and there is no `with:` block. The counts live in the process, in a
|
|
1042
|
-
table of at most 10,000 addresses (when it is full, expired windows and
|
|
1043
|
-
then the oldest ones make room), so they are lost on restart and each
|
|
1044
|
-
process counts on its own. Every request to a limited action takes one lock:
|
|
1045
|
-
name the actions with `only:`. In a test, `Cybertrain::RateLimit.reset_all!`
|
|
1046
|
-
forgets every count and `Cybertrain::RateLimit.current_time = n` fixes the
|
|
1047
|
-
clock (`-1` restores it).
|
|
1048
|
-
|
|
1049
|
-
The address counted is `request.client_ip` (`by: :remote_addr` counts the
|
|
1050
|
-
connection's peer instead). It is `request.remote_addr` unless
|
|
1051
|
-
`config.trusted_proxy` is on (`CYBERTRAIN_TRUSTED_PROXY=1`) **and** the
|
|
1052
|
-
connection comes from loopback, that is, from a reverse proxy on the same
|
|
1053
|
-
host (nginx, Caddy, kamal-proxy); then it is the last address in
|
|
1054
|
-
`X-Forwarded-For`, the one the proxy appended. Earlier entries are whatever
|
|
1055
|
-
the client sent and are never used, and a last entry that is empty, longer
|
|
1056
|
-
than 45 bytes or not printable ASCII gives `remote_addr`. Without the
|
|
1057
|
-
setting, every visitor behind such a proxy shares its address, and ten bad
|
|
1058
|
-
passwords lock everyone out; with it, a client connecting directly from
|
|
1059
|
-
elsewhere still cannot choose its own address. `request.loopback?` tells
|
|
1060
|
-
whether the connection comes from this machine (127.0.0.0/8, `::1`,
|
|
1061
|
-
`::ffff:127.x.x.x`), for pages only an operator on the server should
|
|
1062
|
-
reach; behind a proxy on the same host every request is loopback, so check
|
|
1063
|
-
`client_ip` too there. Design: [docs/design.md](docs/design.md) D26.
|
|
1064
|
-
|
|
1065
|
-
A proxy that reaches the app over a network, not loopback, is listed by
|
|
1066
|
-
address instead: `config.trusted_proxies`, from `CYBERTRAIN_TRUSTED_PROXIES`,
|
|
1067
|
-
takes comma-separated IPv4 or IPv6 addresses or CIDRs
|
|
1068
|
-
(`CYBERTRAIN_TRUSTED_PROXIES=172.16.0.0/12,10.0.0.0/8`, or a single
|
|
1069
|
-
`172.18.0.2`). A connection from inside one of them gets the same treatment
|
|
1070
|
-
as a loopback one above: the last `X-Forwarded-For` address, validated the
|
|
1071
|
-
same way. An IPv4 entry also matches the IPv4-mapped form of its addresses
|
|
1072
|
-
(`::ffff:172.18.0.2`), and a peer's `%zone` is ignored. An entry that is not
|
|
1073
|
-
an address or CIDR, or has a prefix past `/32` (IPv4) or `/128` (IPv6),
|
|
1074
|
-
stops the server at boot. `CYBERTRAIN_TRUSTED_PROXY=1` is the shorthand for
|
|
1075
|
-
`127.0.0.0/8,::1`, and the two add up. This is the Kamal case: kamal-proxy
|
|
1076
|
-
runs in its own Docker container and connects from its address on the Docker
|
|
1077
|
-
network, so the `config/deploy.yml` that `cybertrain new` writes sets
|
|
1078
|
-
`CYBERTRAIN_TRUSTED_PROXIES: 172.16.0.0/12` (Docker's default range; change
|
|
1079
|
-
it if your networks use another, `docker network inspect kamal`). Leave
|
|
1080
|
-
Kamal's `proxy.forward_headers` unset: with `ssl: true`, kamal-proxy sets
|
|
1081
|
-
`X-Forwarded-For` itself rather than passing on the client's. List only
|
|
1082
|
-
networks that clients cannot connect from: anyone inside a listed network
|
|
1083
|
-
can choose the address a rate limit counts.
|
|
1084
|
-
|
|
1085
|
-
## Parameterised headers
|
|
1086
|
-
|
|
1087
|
-
API clients often send `Scheme k1=v1, k2="v2"` headers (`Authorization:
|
|
1088
|
-
Digest ...`, a vendor `X-Example-Auth`) or `k1=v1;k2=v2, k3` lists
|
|
1089
|
-
(`Forwarded`, `Prefer`). `request.header_params(name)` parses one into a
|
|
1090
|
-
`Hash` of lowercase names to values, so an API controller does not split it
|
|
1091
|
-
by hand:
|
|
1092
|
-
|
|
1093
|
-
```ruby
|
|
1094
|
-
# Authorization: Digest username="Mufasa", realm="api@example.com", nc=00000001, qop=auth
|
|
1095
|
-
auth = request.header_params("authorization")
|
|
1096
|
-
auth["username"] # => "Mufasa"
|
|
1097
|
-
request.header_scheme("authorization") # => "Digest"
|
|
1098
|
-
|
|
1099
|
-
# Forwarded: for=192.0.2.60;proto=https;by=203.0.113.43
|
|
1100
|
-
request.header_params("forwarded")["proto"] # => "https"
|
|
1101
|
-
|
|
1102
|
-
# Prefer: respond-async, wait=5
|
|
1103
|
-
request.header_params("prefer").key?("respond-async") # => true
|
|
1104
|
-
```
|
|
1105
|
-
|
|
1106
|
-
`,` and `;` both separate parameters and the first occurrence of a name
|
|
1107
|
-
wins; a quoted value is unquoted (`\"` and `\\` undone), an unquoted one
|
|
1108
|
-
has the spaces around it trimmed, and a name without `=` maps to `""`. A
|
|
1109
|
-
leading scheme word (`Digest`, `Bearer`) is not a parameter:
|
|
1110
|
-
`request.header_scheme(name)` returns it (`""` when there is none). A header
|
|
1111
|
-
that is missing or longer than 8 KiB gives `{}`. Values are the bytes as sent,
|
|
1112
|
-
never percent-decoded and possibly not valid UTF-8. A `Bearer <token>` or
|
|
1113
|
-
`Basic <base64>` credential is not a parameter: check the scheme, then read
|
|
1114
|
-
the credential from `request.header("authorization")`. Design:
|
|
1115
|
-
[docs/design.md](docs/design.md) D31.
|
|
1116
|
-
|
|
1117
|
-
## Background work
|
|
1118
|
-
|
|
1119
|
-
An application that has to do something outside a request (send a
|
|
1120
|
-
notification after the response went out, sweep old rows every hour, retry a
|
|
1121
|
-
call that failed) registers it with the framework, which runs it on one
|
|
1122
|
-
background thread of the served process:
|
|
1123
|
-
|
|
1124
|
-
```ruby
|
|
1125
|
-
# config/app.rb (or a file bin/<app>.rb requires before Cybertrain::Main.run)
|
|
1126
|
-
Cybertrain::Scheduler.every(3600, "prune sessions") do
|
|
1127
|
-
Cybertrain::DB.with { |c| c.execute("DELETE FROM sessions WHERE updated_at < datetime('now', '-30 days')") }
|
|
1128
|
-
end
|
|
1129
|
-
|
|
1130
|
-
Cybertrain::Job.handle("welcome_mail") do |payload|
|
|
1131
|
-
data = Cybertrain::Cast.hash(JSON.parse(payload))
|
|
1132
|
-
WelcomeMail.deliver(data["email"].to_s) # raise to have it retried
|
|
1133
|
-
end
|
|
1134
|
-
```
|
|
1135
|
-
|
|
1136
|
-
```ruby
|
|
1137
|
-
# in an action, after saving the user
|
|
1138
|
-
Cybertrain::Job.enqueue("welcome_mail", JSON.generate({ "email" => @user.email }))
|
|
1139
|
-
Cybertrain::Job.enqueue("reminder", @order.id.to_s, run_at: Time.now + 600)
|
|
1140
|
-
```
|
|
1141
|
-
|
|
1142
|
-
- **When it runs.** The thread starts when the binary serves (`./<app>`,
|
|
1143
|
-
`./<app> server`), after boot has connected the database, bound the port
|
|
1144
|
-
and set `SPINEL_WORKERS`; `migrate`, `db ...` and `Main.command` commands
|
|
1145
|
-
never start it, and neither does `Cybertrain::Test::Client`. A task runs
|
|
1146
|
-
first `seconds` after the start, then `seconds` after the start of each
|
|
1147
|
-
run. Each run is wrapped in a rescue that logs `[ERROR] Scheduled task
|
|
1148
|
-
"name" failed: ...` and the loop goes on. The block takes no arguments
|
|
1149
|
-
(Spinel has no `instance_eval`). `Cybertrain::Scheduler.last_tick` is the
|
|
1150
|
-
epoch second of the last tick, for a health check of your own.
|
|
1151
|
-
- **Jobs.** `Job.enqueue` adds a row to the `cybertrain_jobs` table, which
|
|
1152
|
-
the framework creates itself (`CREATE TABLE IF NOT EXISTS`, on first use)
|
|
1153
|
-
and leaves out of `db/schema.rb`. The payload is a String: JSON-encode
|
|
1154
|
-
structured data. Inside a `DB.transaction` the row commits or rolls back
|
|
1155
|
-
with the rest. The worker, a scheduler task polling every
|
|
1156
|
-
`job_poll_interval` seconds (5), claims the oldest due job in a short
|
|
1157
|
-
`BEGIN IMMEDIATE` transaction, runs its handler outside it, and marks it
|
|
1158
|
-
`done`. A handler that raises is retried 1, 5, 15 and then 60 minutes
|
|
1159
|
-
later, with `Class: message` (up to 1000 characters) in `last_error`, until
|
|
1160
|
-
`job_max_attempts` (5) attempts have failed: the row is then `failed` and
|
|
1161
|
-
stays for you to look at. A job whose kind has no handler is `failed` at
|
|
1162
|
-
once; `Job.enqueue` refuses an unregistered kind. Rows a crashed process
|
|
1163
|
-
left `running` are requeued when the worker starts. `Job.state_count("failed")`
|
|
1164
|
-
and `Job.prune_done(before_epoch)` help with a health page and housekeeping.
|
|
1165
|
-
- **One worker per database.** Every served process runs the thread unless
|
|
1166
|
-
`CYBERTRAIN_WORKER=0` (`config.worker = false`): when several processes
|
|
1167
|
-
share one SQLite file, turn it off in all but one, or each scheduled task
|
|
1168
|
-
runs once per process.
|
|
1169
|
-
- **Keep tasks short.** Tasks and handlers run one at a time on that one
|
|
1170
|
-
thread, and a SQLite call holds its OS worker until it returns: with
|
|
1171
|
-
`SPINEL_WORKERS=1` (the default) a long query stalls every request for as
|
|
1172
|
-
long. Batch big work (delete a thousand rows per run, enqueue one job per
|
|
1173
|
-
item) rather than doing it in one go.
|
|
1174
|
-
- **Testing.** `Cybertrain::Scheduler.run_due(now)` runs one tick and
|
|
1175
|
-
`Cybertrain::Job.work_one(now)` one job, with an epoch time you choose, so
|
|
1176
|
-
a test never sleeps.
|
|
1177
|
-
|
|
1178
|
-
There is no cron syntax, no priority or uniqueness for jobs and no mailer:
|
|
1179
|
-
this covers "send later" and "run every N seconds" (`docs/design.md` D28).
|
|
1180
|
-
|
|
1181
|
-
## Operations: backups, vacuum, health check
|
|
1182
|
-
|
|
1183
|
-
The data of a deployed application is one SQLite file, so the binary carries
|
|
1184
|
-
the three operational tools such an application needs.
|
|
1185
|
-
|
|
1186
|
-
**Backups.** `./<app> db backup PATH` writes a consistent copy of the
|
|
1187
|
-
configured database (`CYBERTRAIN_DATABASE`) while the server keeps running:
|
|
1188
|
-
SQLite's `VACUUM INTO` (3.27 or newer; the binary checks) reads one snapshot,
|
|
1189
|
-
and under WAL the server's writes go on meanwhile. The copy is written to a
|
|
1190
|
-
temporary file next to `PATH`, created exclusively with mode 0600 (the file
|
|
1191
|
-
holds password digests and session tokens), fsynced, and only then renamed
|
|
1192
|
-
to `PATH`, so a failed or interrupted copy never leaves a partial file under
|
|
1193
|
-
the real name (the temporary one is deleted). An existing `PATH` is refused
|
|
1194
|
-
unless you pass `--force`; the database itself never is a target, nor is a
|
|
1195
|
-
`:memory:` database a source.
|
|
1196
|
-
|
|
1197
|
-
```sh
|
|
1198
|
-
./blog db backup /srv/blog/backups/before-upgrade.sqlite3
|
|
1199
|
-
./blog db backup /srv/blog/backups/ --keep 14 # blog-20261025T031500Z.sqlite3, newest 14 kept
|
|
1200
|
-
```
|
|
1201
|
-
|
|
1202
|
-
A target ending in `/` (or an existing directory) gets a dated name,
|
|
1203
|
-
`<app>-<YYYYmmddTHHMMSSZ>.sqlite3` in UTC, in a directory created with mode
|
|
1204
|
-
0700 if it does not exist; `--keep N` then deletes the oldest files of
|
|
1205
|
-
exactly that shape beyond N and touches nothing else there. Restoring is
|
|
1206
|
-
copying a backup over the database file with the server stopped (and
|
|
1207
|
-
deleting that file's `-wal` and `-shm` beside it).
|
|
1208
|
-
|
|
1209
|
-
**Vacuum.** Deleting rows leaves free pages in the file. `./<app> db vacuum`
|
|
1210
|
-
runs `VACUUM` and then `PRAGMA wal_checkpoint(TRUNCATE)`, so the rebuilt
|
|
1211
|
-
pages reach the database file, the file is cut to its new size and the WAL
|
|
1212
|
-
is emptied; it prints the size on disk (database plus WAL) before and after.
|
|
1213
|
-
`VACUUM` rewrites the whole file and holds the write lock while it does, so
|
|
1214
|
-
run it when the application is quiet. If a reader keeps the checkpoint from
|
|
1215
|
-
finishing it says so, and the WAL shrinks at a later checkpoint.
|
|
1216
|
-
|
|
1217
|
-
**Run backups outside the server.** A SQLite call holds its OS worker until
|
|
1218
|
-
it returns, so a large `VACUUM INTO` on a `Scheduler` task would stall
|
|
1219
|
-
requests (with `SPINEL_WORKERS=1`, all of them) for as long as the copy
|
|
1220
|
-
takes. Run the command from cron or a systemd timer instead, as a separate
|
|
1221
|
-
process:
|
|
1222
|
-
|
|
1223
|
-
```sh
|
|
1224
|
-
# crontab of the app's user: every night at 03:15, keep two weeks
|
|
1225
|
-
15 3 * * * cd /srv/blog/current && ./blog db backup /srv/blog/backups/ --keep 14
|
|
1226
|
-
```
|
|
1227
|
-
|
|
1228
|
-
An application that wants to take backups itself can run its own binary as
|
|
1229
|
-
a child process from a scheduled task, rather than run the SQL on its own
|
|
1230
|
-
thread: `system` waits for the child on the scheduler's thread only, and
|
|
1231
|
-
requests go on being served meanwhile (Spinel has no `Process.wait`, and a
|
|
1232
|
-
`Process.spawn` without it leaves a zombie, so use `system` with an argument
|
|
1233
|
-
list, never a string with request data in it). A copy that takes longer than
|
|
1234
|
-
`health_worker_stale` makes the health check answer `worker stale` until it
|
|
1235
|
-
is done.
|
|
1236
|
-
|
|
1237
|
-
```ruby
|
|
1238
|
-
# config/app.rb, in a served binary started from its own directory (dist/)
|
|
1239
|
-
Cybertrain::Scheduler.every(86_400, "nightly backup") do
|
|
1240
|
-
system("./blog", "db", "backup", "storage/backups/", "--keep", "14")
|
|
1241
|
-
end
|
|
1242
|
-
```
|
|
1243
|
-
|
|
1244
|
-
**Health check.** While the health check is enabled (it is by default;
|
|
1245
|
-
an empty `config.health_path` turns it off), the served application
|
|
1246
|
-
answers `GET /up` (and `HEAD`) before the static files, the session, CSRF and the router: no controller
|
|
1247
|
-
is involved and no login `before_action` can redirect it, and it sets no
|
|
1248
|
-
cookie. It runs `SELECT count(*) FROM sqlite_master`, which has to read the
|
|
1249
|
-
schema page from the file (a bare `SELECT 1` never touches it), and answers
|
|
1250
|
-
in `text/plain` with `Cache-Control: no-store`:
|
|
1251
|
-
|
|
1252
|
-
| Status | Body | When |
|
|
1253
|
-
| --- | --- | --- |
|
|
1254
|
-
| 200 | `ok` | the query succeeded |
|
|
1255
|
-
| 503 | `database unavailable` | it raised `Cybertrain::DB::Error` (the file is missing or unreadable, the pool is closed) |
|
|
1256
|
-
| 503 | `worker stale` | the background `Scheduler` runs and its last tick is older than `health_worker_stale` seconds (300): a task is stuck, or runs far too long |
|
|
1257
|
-
|
|
1258
|
-
A poll is logged like any request (two lines). The path is
|
|
1259
|
-
`config.health_path` (`CYBERTRAIN_HEALTH_PATH`, default `/up`; an empty
|
|
1260
|
-
value turns the check off, and the path then goes on to `public/` and the
|
|
1261
|
-
router as before). A load balancer, kamal-proxy or a deploy script can poll
|
|
1262
|
-
it; `public/up.html`, which `cybertrain new` writes and its Kamal
|
|
1263
|
-
configuration still polls, only shows that the process answers HTTP.
|
|
1264
|
-
|
|
1265
|
-
## Differences from Rails
|
|
1266
|
-
|
|
1267
|
-
| Rails | cybertrain |
|
|
89
|
+
| | |
|
|
1268
90
|
| --- | --- |
|
|
1269
|
-
|
|
|
1270
|
-
|
|
|
1271
|
-
|
|
|
1272
|
-
|
|
|
1273
|
-
|
|
|
1274
|
-
|
|
|
1275
|
-
|
|
|
1276
|
-
|
|
|
1277
|
-
|
|
|
1278
|
-
|
|
|
1279
|
-
|
|
|
1280
|
-
|
|
|
1281
|
-
|
|
|
1282
|
-
|
|
|
1283
|
-
|
|
|
1284
|
-
|
|
|
1285
|
-
|
|
1286
|
-
|
|
1287
|
-
| `order("lower(title)")`, `order("posts.title")`, `order(params[:sort])` | `order` takes only `column [ASC\|DESC]` lists (each column quoted) and raises `ArgumentError` (a 500: the string is the developer's) on anything else; request data must go through an allowlist first, e.g. `SORTS = { "title" => "title", "newest" => "created_at DESC" }` and `Post.order(SORTS.fetch(params[:sort].to_s, "id"))`, because a well-formed term can still name a missing column (a 500 from SQLite) or a column the client must not sort by; raw ORDER BY text goes through `order_sql` (`Post.order_sql("lower(title)")`, also on a relation), which must never see request data. A `limit`/`offset` on `delete_all` is honoured (a subselect), and request parameters nest at most 32 levels / 4096 pairs (400 past that). `Query.parse`, `Query.value_of`, `Query.decode` and `request.form_value` / `query_value` raise `Cybertrain::QueryMalformed` (a `StandardError` under `Cybertrain::QueryInvalid`, not an `ArgumentError`) on a malformed percent-escape, on both runtimes (`Cookies.parse` still keeps the raw value). They raise it too on an invalid UTF-8 byte sequence, raw (whatever the String's tag) or percent-encoded (Rails likewise answers 400 for an invalid parameter encoding), and so does a raw byte that only an escape would complete (`a=\xC3%A9`); `Cookies.parse` keeps such a value raw, and a path with an invalid sequence (raw, or percent-decoded in a segment) is a 400 as well (`request.path_segments`; a malformed escape in a path stays literal, a 404 by non-match). App code that rescued `ArgumentError` around them should rescue `Cybertrain::QueryInvalid` instead (upgrade note). The request keeps no parameter tree of its own (the Router builds `params`, the middleware read one key); an app that wants the whole tree parses it itself, `Query.parse(request.utf8_query_string)`, and owns the result |
|
|
1288
|
-
| Full backtrace on an exception | Class, message, request line, template name/line — Spinel exposes no backtraces |
|
|
1289
|
-
| Rack, its middleware, and any gem in a `Gemfile` | No Rack compatibility; a small fixed middleware set; Spinel's own `spin-index`, limited to what compiles under its Ruby subset |
|
|
1290
|
-
| minitest / RSpec | `Cybertrain::Test` — reflection-based runners can't work ahead-of-time |
|
|
1291
|
-
| `bin/rails generate authentication`, `has_secure_password` + bcrypt | `cybertrain generate authentication` writes the same shape (User, Session, SessionsController, the before_action) plus a sign-up page, since there is no console to create a user in; `has_secure_password` is written out as methods on the User model over `Cybertrain::Password` (PBKDF2-HMAC-SHA256 in Ruby: bcrypt would need a C library the binary links, and `String#crypt` has no bcrypt on macOS). No password reset (no mailers); sign-in attempts are limited by `rate_limit` (counted per process; see [Rate limiting](#rate-limiting-and-the-client-address)) |
|
|
1292
|
-
| Active Storage, `has_one_attached :cover` | `cybertrain generate attachment post cover` writes five `cover_*` columns on the owner table and the model methods (`cover_file=`, `cover_url`, ...) into the app; files live on disk under `storage/uploads/` and are served by a generated action through `send_blob`. One file per field, local disk only, no variants or direct uploads |
|
|
1293
|
-
| `skip_before_action :x` | The same, for named before callbacks (a block callback has no name to skip); no `skip_after_action` |
|
|
1294
|
-
| `resource :session` | The same six routes to `SessionsController`; no nesting under it |
|
|
1295
|
-
| `get "/x"` then `post "/x"` | Both are fine: the second route's derived name is taken, so it is unnamed, as in Rails; a duplicate `as:` is still an error |
|
|
1296
|
-
| `Time.zone = "Asia/Tokyo"`, `time.in_time_zone`, ActiveSupport's bundled zone data | `Cybertrain::TimeZone.find("Asia/Tokyo")` reads the system's TZif files (`/usr/share/zoneinfo`, or `TZDIR`) and converts Integer epoch seconds (`utc_offset_at`, `utc_to_local`, `local_to_utc`); no per-request `Time.zone` and no `l(time, zone:)` yet ([Time zones](#time-zones)) |
|
|
1297
|
-
| `Net::HTTP`, Faraday | `Cybertrain::HttpClient` for webhooks and APIs: every wait bounded, non-public destinations refused by default, no redirects or keep-alive (see "Outbound HTTP"); Spinel's `Net::HTTP` blocks the whole server on a stalled peer |
|
|
1298
|
-
| Action Mailer (`deliver_later`, mailer classes, views) | `Cybertrain::Mailer.deliver(to:, subject:, body:)` sends plain text over SMTP (STARTTLS, implicit TLS or plain to loopback; AUTH PLAIN; every wait bounded); no templates, HTML, attachments or queue (see "Sending e-mail") |
|
|
1299
|
-
| Active Job, `perform_later`, `solid_queue`, recurring tasks | `Cybertrain::Job.handle(kind) { \|payload\| ... }` and `Job.enqueue(kind, payload, run_at:)` (a String payload in a `cybertrain_jobs` table, retried with backoff, then `failed`), `Cybertrain::Scheduler.every(seconds, name) { ... }`; one thread in the served process, `CYBERTRAIN_WORKER=0` elsewhere ([Background work](#background-work)) |
|
|
1300
|
-
| ActionCable client, faye-websocket, websocket-client-simple | `Cybertrain::WebSocket.subscribe(name, url).on_message { \|conn, data\| ... }` keeps a feed connected while the server runs (reconnects, pings, silence detection), `Cybertrain::WebSocket::Client` for one synchronous connection; outbound only, every wait bounded (see "Consuming WebSocket feeds") |
|
|
1301
|
-
| `config.filter_parameters += [:passw, /\Apin\z/, ->(k, v) { ... }]` | `config.filter_parameters` is an Array of Strings matched by substring, case-insensitively (no Regexps or lambdas); it applies to the request log and the development error page ([Filtered parameters](#configuration-and-environment-variables)) |
|
|
1302
|
-
|
|
1303
|
-
| `get "up" => "rails/health#show"` (a route to a controller) | `GET /up` is built in and answered before the static files, the session and the router, so no `before_action` runs; it queries the database and the background worker's last tick, `200 ok` or `503` ([Operations](#operations-backups-vacuum-health-check)) |
|
|
1304
|
-
| `bin/rails db:...` tasks for backups (none built in) | `./<app> db backup PATH` / `db backup DIR/ --keep N` (`VACUUM INTO`, mode 0600, renamed into place) and `./<app> db vacuum` |
|
|
1305
|
-
| ActionCable (the server), asset pipeline, i18n | Not built (MVP non-goals, `docs/design.md` §2.3) |
|
|
1306
|
-
|
|
1307
|
-
Not exhaustive — `docs/design.md` §8 ("捨てたもの") is the fuller record.
|
|
1308
|
-
|
|
1309
|
-
## Performance
|
|
1310
|
-
|
|
1311
|
-
[docs/benchmark.md](docs/benchmark.md) measures examples/blog against the
|
|
1312
|
-
same blog written in Rails 8.1, [bench/rails_blog](bench/rails_blog) (the same
|
|
1313
|
-
views, routes, models and schema), both in production mode, each server
|
|
1314
|
-
pinned to the same CPU cores of one 4-vCPU cloud VM and loaded by wrk from
|
|
1315
|
-
two others. Medians of three 10-second runs, recorded 2026-10-09:
|
|
1316
|
-
|
|
1317
|
-
| 1 CPU core, 16 connections | CyberTrain 0.2.1 | Rails 8.1.4 (YJIT, Puma) | |
|
|
1318
|
-
| --- | --: | --: | --: |
|
|
1319
|
-
| `GET /articles` (30 articles) | 2,641 req/s | 468 req/s | 5.6x |
|
|
1320
|
-
| `GET /articles/1` (10 comments, 12 forms) | 2,450 req/s | 201 req/s | 12.2x |
|
|
1321
|
-
| `POST /articles/2/comments` | 1,507 req/s | 390 req/s | 3.9x |
|
|
1322
|
-
| `GET /style.css` | 13,300 req/s | 5,127 req/s | 2.6x |
|
|
1323
|
-
| Memory after the load (RSS) | 10.6 MB | 126 MB | |
|
|
1324
|
-
| Spawn to first response | 0.02 s | 1.3 s | |
|
|
1325
|
-
| What the server needs | 2.5 MB (one binary) | 109 MB (Ruby, 69 gems, the app) | |
|
|
1326
|
-
|
|
1327
|
-
With jemalloc in both (`allocator = "jemalloc"` in spin.toml; Rails'
|
|
1328
|
-
production image preloads it) the ratios are 2.8 to 14.6. On two cores the lead
|
|
1329
|
-
narrows to 2.0 to 5.5: a second Puma worker doubles Rails, while
|
|
1330
|
-
`SPINEL_WORKERS=2` adds 9 to 58% to CyberTrain (set it to the core count on a
|
|
1331
|
-
multi-core server; it defaults to 1). The binary also has to be built
|
|
1332
|
-
(`cybertrain build` took about three minutes there). docs/benchmark.md has
|
|
1333
|
-
latency, every configuration, the method and what was optimized to get here;
|
|
1334
|
-
`bench/run` repeats the measurement and `bench/report` prints its tables.
|
|
1335
|
-
|
|
1336
|
-
## Configuration and environment variables
|
|
1337
|
-
|
|
1338
|
-
```ruby
|
|
1339
|
-
# config/app.rb
|
|
1340
|
-
Cybertrain.configure do |c|
|
|
1341
|
-
c.port = 3000
|
|
1342
|
-
c.workers = 1
|
|
1343
|
-
end
|
|
1344
|
-
```
|
|
1345
|
-
|
|
1346
|
-
`Cybertrain::Config` reads the environment first; `config/app.rb` overrides:
|
|
1347
|
-
|
|
1348
|
-
| Variable | Attribute | Default |
|
|
1349
|
-
| --- | --- | --- |
|
|
1350
|
-
| `CYBERTRAIN_ENV` | `env` | `"development"`; `"production"` in a binary from `cybertrain build` (also `"test"`) |
|
|
1351
|
-
| `PORT` | `port` | `3000` |
|
|
1352
|
-
| `CYBERTRAIN_DATABASE` | `database_path` | `storage/#{env}.sqlite3` |
|
|
1353
|
-
| `CYBERTRAIN_SECRET_KEY_BASE` | `secret_key_base` | required in production; dev/test auto-generate one into `tmp/secret_key` |
|
|
1354
|
-
| `SPINEL_WORKERS` | `workers` | `1` |
|
|
1355
|
-
| `CYBERTRAIN_HOST` | `host` | `"127.0.0.1"`; `0.0.0.0` listens on every interface (containers) |
|
|
1356
|
-
| `CYBERTRAIN_ALLOWED_HOSTS` | `allowed_hosts` | Development accepts `localhost`, `127.0.0.1`, `[::1]`; comma-separated exact hostnames extend this list (no scheme, port or wildcard). Other hosts get 403 before static files, sessions, routing or diagnostics. This is separate from the listening address; production and test do not apply this development guard |
|
|
1357
|
-
| `CYBERTRAIN_SESSION_SAME_SITE` | `session_same_site` | `"Lax"`; also `Strict` or `None` (`None` always adds `Secure`); anything else stops the server at boot |
|
|
1358
|
-
| `CYBERTRAIN_SESSION_PARTITIONED` | `session_partitioned` | `false`; `1` or `true` adds `Partitioned` (and `Secure`) |
|
|
1359
|
-
| `CYBERTRAIN_TRUSTED_PROXY` | `trusted_proxy` | `false`; `1` or `true` makes `request.client_ip` (what `rate_limit` counts) the last `X-Forwarded-For` address of a connection from loopback, for a reverse proxy on the same host (see [Rate limiting](#rate-limiting-and-the-client-address)); shorthand for `CYBERTRAIN_TRUSTED_PROXIES=127.0.0.0/8,::1` |
|
|
1360
|
-
| `CYBERTRAIN_TRUSTED_PROXIES` | `trusted_proxies` | `[]`; comma-separated IPv4/IPv6 addresses or CIDRs (`172.16.0.0/12,10.0.0.0/8`) whose connections have `request.client_ip` taken from the last `X-Forwarded-For` address, for a proxy reached over a network, such as kamal-proxy in its container; a bad entry stops the server at boot |
|
|
1361
|
-
| `CYBERTRAIN_MAX_PAGE` | `max_page` | `10000`: the largest page `params.page_number` gives and `Relation#page` accepts (see [Pagination](#pagination)); a value that is not a number of at most 18 digits, or below 1, is the default |
|
|
1362
|
-
| `CYBERTRAIN_FILTER_PARAMETERS` | `filter_parameters` | `["password", "password_confirmation", "token", "secret", "secret_key_base", "session_key", "api_key", "authorization"]`; comma-separated names replace the list. A parameter whose name contains one of them (ASCII case-insensitively: `password` also matches `user[password]`) has its value written as `[FILTERED]` in the request log and on the development error page; `[]` in `config/app.rb` turns this off |
|
|
1363
|
-
| `CYBERTRAIN_WORKER` | `worker` | `true`; `0` or `false` keeps this process from running scheduled tasks and jobs (see [Background work](#background-work)); `scheduler_interval` (1 s), `job_poll_interval` (5 s) and `job_max_attempts` (5) are set in `config/app.rb` |
|
|
1364
|
-
| `CYBERTRAIN_HEALTH_PATH` | `health_path` | `"/up"`; the built-in health check's path (see [Operations](#operations-backups-vacuum-health-check)); an empty value turns it off; one without a leading `/` stops the server at boot. `health_worker_stale` (300 s) is set in `config/app.rb` |
|
|
1365
|
-
| `CYBERTRAIN_HTTP_ALLOW_PRIVATE` | `http_allow_private` | `false`; `1` or `true` lets `Cybertrain::HttpClient` connect to loopback, private and other non-public addresses (see "Outbound HTTP") |
|
|
1366
|
-
| `CYBERTRAIN_SMTP_HOST` | `smtp_host` | `""`; the SMTP server `Cybertrain::Mailer` sends through (see "Sending e-mail") |
|
|
1367
|
-
| `CYBERTRAIN_SMTP_PORT` | `smtp_port` | `0`: the mode's port, 587 for `starttls`, 465 for `tls`, 25 for `none`; anything but 1 to 65535 is 0 |
|
|
1368
|
-
| `CYBERTRAIN_SMTP_USER` | `smtp_user` | `""`: no AUTH; set, the mailer authenticates with AUTH PLAIN |
|
|
1369
|
-
| `CYBERTRAIN_SMTP_PASSWORD` | `smtp_password` | `""`; a secret, never logged, never in the boot banner or an error message |
|
|
1370
|
-
| `CYBERTRAIN_SMTP_FROM` | `smtp_from` | `""`: the sender `Mailer.deliver` uses when it is not given `from:` |
|
|
1371
|
-
| `CYBERTRAIN_SMTP_TLS` | `smtp_tls` | `"starttls"`; also `tls` (implicit TLS) or `none` (plain text, credentials to loopback only); both TLS modes need `require "cybertrain/net/tls"` |
|
|
1372
|
-
|
|
1373
|
-
Use `None` (with `CYBERTRAIN_SESSION_PARTITIONED=1`) only for an app shown
|
|
1374
|
-
inside another site's frame; the browser must reach the app over HTTPS.
|
|
1375
|
-
|
|
1376
|
-
Other attributes with fixed, overridable defaults: `views_root`
|
|
1377
|
-
(`"app/views"`), `public_root` (`"public"`), `layout`
|
|
1378
|
-
(`"layouts/application"`), `log_level` (`:info`), `session_cookie_name`,
|
|
1379
|
-
`session_max_age` (2 weeks), `session_secure` (`true` in production, which
|
|
1380
|
-
marks the session cookie `Secure`; `false` elsewhere, though `SameSite=None`
|
|
1381
|
-
and `Partitioned` add `Secure` anyway), `pool_size` (4), `static_files`/`csrf`
|
|
1382
|
-
(`true`), `csrf_exempt_prefixes` (`[]`: path prefixes such as `"/api/"` whose
|
|
1383
|
-
non-GET requests skip the CSRF check, for an endpoint that programs call with
|
|
1384
|
-
a credential of their own; each must start and end with `/` and not be `/`
|
|
1385
|
-
alone, or boot fails; the prefix is matched on the path as sent, so a
|
|
1386
|
-
percent-encoded spelling is still checked; no other route may match a path
|
|
1387
|
-
under the prefix, and the exempt controller must not rely on the session
|
|
1388
|
-
for authentication, since the session is still loaded there), `max_body_bytes`
|
|
1389
|
-
(10 MiB: a longer request body, as sent, is answered 413 before it is read;
|
|
1390
|
-
raise it for large uploads; a gzip body's inflated size is bounded separately
|
|
1391
|
-
by `request.decoded_body(limit)`), `max_render_depth` (12 renders open at once: the page and its
|
|
1392
|
-
partials; raise it for partials that legitimately recurse deeper; it must be
|
|
1393
|
-
at least 1, or boot fails). Upgrade note: render nesting, unlimited before, is
|
|
1394
|
-
now capped at 12 by default (the page is depth 1, so partials can nest 11
|
|
1395
|
-
levels; the layout renders after the page and does not nest), and an app with
|
|
1396
|
-
a deeper tree (threaded comments, a category menu) gets a template error until
|
|
1397
|
-
it sets `max_render_depth`. File uploads: `max_upload_bytes` (5 MiB per
|
|
1398
|
-
uploaded file; it must not exceed `max_body_bytes`), `max_multipart_parts`
|
|
1399
|
-
(128 parts, text fields included, at most 4096), `upload_root`
|
|
1400
|
-
(`"storage/uploads"`, where `Cybertrain::BlobStore` writes; created on the
|
|
1401
|
-
first upload, not at boot).
|
|
1402
|
-
|
|
1403
|
-
**Security headers:** every response the server writes (actions, the
|
|
1404
|
-
static-file middleware, error pages and the development diagnostics page,
|
|
1405
|
-
the server's own 400/411/413/500, and HEAD) carries
|
|
1406
|
-
`X-Content-Type-Options: nosniff` and `X-Frame-Options: SAMEORIGIN`, unless
|
|
1407
|
-
the response already sets a header of that name (any case): `Response#to_http`
|
|
1408
|
-
adds them after the response's own headers and only when missing, so
|
|
1409
|
-
`response.set_header("X-Frame-Options", "DENY")` in an action sends exactly
|
|
1410
|
-
your value, once. There is deliberately no `Config`
|
|
1411
|
-
switch to turn them off app-wide; an embeddable widget overrides the header
|
|
1412
|
-
on its own responses (a `Content-Security-Policy: frame-ancestors ...` header
|
|
1413
|
-
takes precedence over `X-Frame-Options` in current browsers). Because of
|
|
1414
|
-
`nosniff`, browsers take `Content-Type` literally: the static-file middleware
|
|
1415
|
-
knows the usual `public/` extensions (html, css, js/mjs, images including
|
|
1416
|
-
webp and avif, fonts, wasm, pdf, audio and video, json, csv, xml), and
|
|
1417
|
-
anything else is served as `application/octet-stream`, a download; add the
|
|
1418
|
-
extension to `Cybertrain::Static::EXTENSIONS` if an app needs another one
|
|
1419
|
-
inline. A reverse proxy that serves `public/` itself bypasses these headers
|
|
1420
|
-
(see [docs/deploy.en.md](docs/deploy.en.md)). `Cybertrain::Test::Client` calls the
|
|
1421
|
-
app in-process and never serializes the response, so it does not show the two
|
|
1422
|
-
defaults; assert them on `response.to_http`.
|
|
1423
|
-
|
|
1424
|
-
**Filtered parameters:** the request log's `Started GET "/login?..."` line
|
|
1425
|
-
and the development error page (its request line, and its dump of the
|
|
1426
|
-
params and of a JSON body) show the value of every parameter named in
|
|
1427
|
-
`config.filter_parameters` as `[FILTERED]`, as Rails' `filter_parameters`
|
|
1428
|
-
does, so a `?token=` in a URL or a password posted to a form that fails
|
|
1429
|
-
does not appear in the framework's request log or on the development
|
|
1430
|
-
error page. Only the values are replaced: the rest of the
|
|
1431
|
-
query string is logged as sent (control characters escaped). The list
|
|
1432
|
-
matches by substring, case-insensitively; extend it in `config/app.rb`
|
|
1433
|
-
(`c.filter_parameters = c.filter_parameters + ["otp", "ssn"]`) or replace
|
|
1434
|
-
it with `CYBERTRAIN_FILTER_PARAMETERS`. Values an action logs itself are not
|
|
1435
|
-
filtered; ask `Cybertrain::ParameterFilter.filter?(name)` or
|
|
1436
|
-
`ParameterFilter.filter_query(text)` there.
|
|
1437
|
-
|
|
1438
|
-
**Deployment:** `cybertrain build` produces `dist/`: the binary `dist/NAME`
|
|
1439
|
-
with the views embedded, `dist/public/` (static assets, or let a reverse
|
|
1440
|
-
proxy serve it), and `dist/storage/` (the SQLite file by default) and
|
|
1441
|
-
`dist/tmp/`. Run it from `dist/` with `CYBERTRAIN_SECRET_KEY_BASE` set;
|
|
1442
|
-
it is production unless `CYBERTRAIN_ENV` says otherwise, and `./NAME
|
|
1443
|
-
migrate` applies the migrations compiled into it. The binary
|
|
1444
|
-
speaks plain HTTP/1.1 only; put nginx/Caddy in front for TLS and HTTP/2 (the
|
|
1445
|
-
session cookie is `Secure` in production, so serve it over HTTPS). In
|
|
1446
|
-
production an exception answers 500 and a bare error response (the router's
|
|
1447
|
-
plain-text 404, `head :not_found`, `render plain:`) is replaced by
|
|
1448
|
-
`public/<status>.html` when that file exists; errors an action renders as HTML
|
|
1449
|
-
or JSON pass through. `SIGTERM` stops accepting, lets requests already in
|
|
1450
|
-
flight finish (answered with `Connection: close`; idle keep-alive connections
|
|
1451
|
-
are closed at once) and exits when they are done, or after the server's
|
|
1452
|
-
`drain_timeout` (10 s). The watcher/rebuild loop never runs in production. [docs/deploy.en.md](docs/deploy.en.md)
|
|
1453
|
-
(in Japanese: [docs/deploy.md](docs/deploy.md)) walks through it end to end: `cybertrain new`, an Ubuntu server,
|
|
1454
|
-
systemd, Caddy with HTTPS, redeploys, rollbacks and backups.
|
|
1455
|
-
|
|
1456
|
-
**Kamal:** `cybertrain new` also writes what [Kamal 2](https://kamal-deploy.org)
|
|
1457
|
-
needs: a `Dockerfile` that runs `cybertrain setup` and `cybertrain build` in an
|
|
1458
|
-
Ubuntu build stage and copies `dist/` into a slim stage of the same release
|
|
1459
|
-
(migrate on boot, then serve on `0.0.0.0:3000`), a `.dockerignore`,
|
|
1460
|
-
`config/deploy.yml` (kamal-proxy with Let's Encrypt in front, the
|
|
1461
|
-
health check on `public/up.html`, the SQLite file on a volume at
|
|
1462
|
-
`/app/storage`) and `.kamal/secrets` (`KAMAL_REGISTRY_PASSWORD` and
|
|
1463
|
-
`CYBERTRAIN_SECRET_KEY_BASE` from the environment). Fill in the values marked
|
|
1464
|
-
`CHANGE ME`, commit, and `kamal setup`;
|
|
1465
|
-
[docs/deploy-kamal.md](docs/deploy-kamal.md) has the details (part 8 of
|
|
1466
|
-
the deployment tutorial).
|
|
1467
|
-
[examples/blog](examples/blog#deploying-with-kamal) carries the same setup,
|
|
1468
|
-
built against this checkout's framework.
|
|
1469
|
-
|
|
1470
|
-
### Application settings
|
|
1471
|
-
|
|
1472
|
-
An application's own settings (a public base URL, a feature switch, an
|
|
1473
|
-
interval, an SMTP host) are declared on the same `Config`, with a type, an
|
|
1474
|
-
environment variable and a default:
|
|
1475
|
-
|
|
1476
|
-
```ruby
|
|
1477
|
-
# config/app.rb
|
|
1478
|
-
Cybertrain.configure do |c|
|
|
1479
|
-
c.setting_string :base_url, env: "MYAPP_BASE_URL", default: ""
|
|
1480
|
-
c.setting_string :smtp_password, env: "MYAPP_SMTP_PASSWORD", default: "", secret: true
|
|
1481
|
-
c.setting_bool :worker, env: "MYAPP_WORKER", default: true
|
|
1482
|
-
c.setting_int :prune_interval, env: "MYAPP_PRUNE_INTERVAL", default: 3600, min: 1, max: 9_999_999
|
|
1483
|
-
end
|
|
1484
|
-
|
|
1485
|
-
Cybertrain.config.string_setting(:base_url)
|
|
1486
|
-
Cybertrain.config.bool_setting(:worker)
|
|
1487
|
-
Cybertrain.config.int_setting(:prune_interval)
|
|
1488
|
-
```
|
|
1489
|
-
|
|
1490
|
-
Each value is read once, when `config/app.rb` runs at startup. An empty or
|
|
1491
|
-
unset variable means the default. A boolean takes `1`/`true` or
|
|
1492
|
-
`0`/`false`; an integer takes decimal digits (an optional leading `-`) within
|
|
1493
|
-
`min..max` (both optional). Any other value falls back to the default and
|
|
1494
|
-
logs a `[WARN]` line saying so. Reading an undeclared name, or one declared
|
|
1495
|
-
with another type, raises `ArgumentError`, as does declaring a name twice.
|
|
1496
|
-
Tests change a value with `set_string_setting`, `set_int_setting` and
|
|
1497
|
-
`set_bool_setting`. `./NAME help` lists every declared setting with its
|
|
1498
|
-
variable and default, and the boot banner prints the current values
|
|
1499
|
-
(`=> Settings: base_url="" worker=true ...`); a `secret: true` setting shows
|
|
1500
|
-
up in both only by name, as `[FILTERED]`. There is no
|
|
1501
|
-
`Cybertrain.config.base_url` accessor: Spinel has no `define_method`, so the
|
|
1502
|
-
readers are typed methods that take the name.
|
|
1503
|
-
|
|
1504
|
-
## Time zones
|
|
1505
|
-
|
|
1506
|
-
Spinel ships no tz database, and setting `ENV["TZ"]` to convert a time
|
|
1507
|
-
changes it for every thread of the process. `Cybertrain::TimeZone` reads the
|
|
1508
|
-
system's zone files instead: `/usr/share/zoneinfo/<name>`, or `$TZDIR/<name>`
|
|
1509
|
-
when `TZDIR` is set (install the `tzdata` package in a slim container image).
|
|
1510
|
-
It parses TZif versions 1 to 4 (RFC 8536), including the POSIX TZ rule a
|
|
1511
|
-
"slim" file relies on after its last transition, and caches each zone for the
|
|
1512
|
-
life of the process (copy-on-write, so lookups from several workers take no
|
|
1513
|
-
lock).
|
|
1514
|
-
|
|
1515
|
-
Everything is Integer epoch seconds. A *local epoch* is a wall-clock reading
|
|
1516
|
-
counted as if it were UTC, so wall-clock arithmetic is plain addition and
|
|
1517
|
-
`TimeZone.civil_to_epoch` / `TimeZone.epoch_to_civil` move between it and
|
|
1518
|
-
`[year, month, day, hour, minute, second]`:
|
|
1519
|
-
|
|
1520
|
-
```ruby
|
|
1521
|
-
tokyo = Cybertrain::TimeZone.find("Asia/Tokyo")
|
|
1522
|
-
tokyo.found? # false for an unknown name: the zone then behaves as UTC
|
|
1523
|
-
tokyo.utc_offset_at(Time.now.to_i) # => 32400
|
|
1524
|
-
tokyo.abbreviation_at(Time.now.to_i) # => "JST"
|
|
1525
|
-
|
|
1526
|
-
# The next 09:00 in Tokyo:
|
|
1527
|
-
now = Time.now.to_i
|
|
1528
|
-
today = Cybertrain::TimeZone.epoch_to_civil(tokyo.utc_to_local(now))
|
|
1529
|
-
run = tokyo.local_to_utc(Cybertrain::TimeZone.civil_to_epoch(today[0], today[1], today[2], 9, 0, 0))
|
|
1530
|
-
run = tokyo.local_to_utc(Cybertrain::TimeZone.civil_to_epoch(today[0], today[1], today[2] + 1, 9, 0, 0)) if run <= now
|
|
1531
|
-
```
|
|
1532
|
-
|
|
1533
|
-
At a daylight-saving transition `local_to_utc` is deterministic: a wall time
|
|
1534
|
-
that a forward jump skips (02:30 on a US spring-forward day) is read with the
|
|
1535
|
-
offset before the jump, so it lands after it (03:30 EDT); a wall time that a
|
|
1536
|
-
backward jump repeats gives the earlier instant. `find` accepts only a
|
|
1537
|
-
relative name made of `A-Za-z0-9_+-./` with no empty, `.` or `..` segment, so
|
|
1538
|
-
a zone name taken from a user's profile cannot reach a file outside the zone
|
|
1539
|
-
directory; an unknown or malformed zone is not cached, and `"UTC"` needs no
|
|
1540
|
-
file. Formatting helpers for templates (`l(time, zone:)`) are not there yet.
|
|
1541
|
-
|
|
1542
|
-
## Outbound HTTP
|
|
1543
|
-
|
|
1544
|
-
An application that calls another server (a webhook, a Slack incoming
|
|
1545
|
-
webhook, a third-party API) uses `Cybertrain::HttpClient`. It is not in
|
|
1546
|
-
`require "cybertrain"`; require it where it is used, and the TLS layer too
|
|
1547
|
-
when any URL is https:
|
|
1548
|
-
|
|
1549
|
-
```ruby
|
|
1550
|
-
require "json"
|
|
1551
|
-
require "cybertrain/http_client"
|
|
1552
|
-
require "cybertrain/net/tls" # only for https://; links the system libssl
|
|
1553
|
-
|
|
1554
|
-
client = Cybertrain::HttpClient.new(timeout: 5, deadline: 15, max_body: 1 << 20)
|
|
1555
|
-
begin
|
|
1556
|
-
res = client.post(ENV["SLACK_WEBHOOK_URL"], JSON.generate({ "text" => "Deployed" }),
|
|
1557
|
-
content_type: "application/json")
|
|
1558
|
-
Cybertrain.logger.warn("slack answered #{res.status}") unless res.success?
|
|
1559
|
-
rescue Cybertrain::HttpClient::Error => e
|
|
1560
|
-
Cybertrain.logger.warn("slack: #{e.message}") # names host:port, never the path
|
|
1561
|
-
end
|
|
1562
|
-
|
|
1563
|
-
res = client.get("https://api.example.com/v1/status", headers: { "Authorization" => "Bearer #{token}" })
|
|
1564
|
-
res.status # 200
|
|
1565
|
-
res.header("ETag") # names in any case; res.headers has them lowercase
|
|
1566
|
-
res.body # the bytes, chunked coding removed
|
|
1567
|
-
```
|
|
1568
|
-
|
|
1569
|
-
Why it exists: Spinel's `Net::HTTP` does the TLS handshake and every read
|
|
1570
|
-
after the first byte as blocking C calls, and each one holds an OS worker,
|
|
1571
|
-
which with the default `SPINEL_WORKERS=1` is the whole web server, for as
|
|
1572
|
-
long as a stalled endpoint likes. `HttpClient` never blocks a worker on the
|
|
1573
|
-
network:
|
|
1574
|
-
|
|
1575
|
-
- **Every wait is bounded.** Connecting (a non-blocking connect; Spinel's
|
|
1576
|
-
`TCPSocket.new` ignores `connect_timeout:`), each TLS handshake step, each
|
|
1577
|
-
read and each write waits at most `timeout` seconds (default 5), and the
|
|
1578
|
-
whole request at most `deadline` seconds (default 15). Past either,
|
|
1579
|
-
`HttpClient::TimeoutError`. Reads go through a byte buffer fed by
|
|
1580
|
-
`read_nonblock`, so bytes already received are never waited for.
|
|
1581
|
-
- **Name resolution blocks.** `getaddrinfo` has no non-blocking form in the
|
|
1582
|
-
runtime: the thread making the request waits for DNS (usually
|
|
1583
|
-
milliseconds, but a dead resolver can take the system's timeout, about 5 s
|
|
1584
|
-
per attempt). The deadline counts that time but cannot interrupt it.
|
|
1585
|
-
- **Non-public destinations are refused** before anything is sent:
|
|
1586
|
-
loopback (`127.0.0.0/8`, `::1`), private (`10/8`, `172.16/12`,
|
|
1587
|
-
`192.168/16`, `fc00::/7`), link-local (`169.254/16`, which includes cloud
|
|
1588
|
-
metadata services, `fe80::/10`), unspecified (`0.0.0.0/8`, `::`),
|
|
1589
|
-
multicast, broadcast, CGNAT (`100.64/10`), the documentation and
|
|
1590
|
-
benchmarking ranges, and IPv4-mapped, NAT64 and 6to4 forms of those
|
|
1591
|
-
(`Cybertrain::Net::Address.blocked_reason`). The host name is resolved
|
|
1592
|
-
once and the addresses it resolved to are the ones checked and dialled
|
|
1593
|
-
(nothing is looked up again), so a name that resolves to a private
|
|
1594
|
-
address (an `/etc/hosts` entry, DNS rebinding) is refused like the
|
|
1595
|
-
literal; the TLS certificate is still checked against the name. The
|
|
1596
|
-
check is per address: a refused address is skipped, and only when every
|
|
1597
|
-
address is refused does the request fail.
|
|
1598
|
-
`HttpClient::BlockedAddressError` says why. To call a service on your
|
|
1599
|
-
own network, or a local server in a test, set `c.http_allow_private =
|
|
1600
|
-
true` in `config/app.rb` (or `CYBERTRAIN_HTTP_ALLOW_PRIVATE=1`), or pass
|
|
1601
|
-
`allow_private: true` to `HttpClient.new`; leave it off whenever a URL
|
|
1602
|
-
can come from a user.
|
|
1603
|
-
- **Every resolved address is tried**, in the order the resolver gives
|
|
1604
|
-
them, IPv6 and IPv4 alike: when one refuses, is unreachable, resets or
|
|
1605
|
-
does not answer within `timeout`, the next is dialled, all of them
|
|
1606
|
-
within `deadline`, and when every one fails the last one's error is
|
|
1607
|
-
raised. This is what makes `localhost` work on macOS, where it resolves
|
|
1608
|
-
to `::1` first and a server listening on `127.0.0.1` refuses that.
|
|
1609
|
-
`Cybertrain::Mailer` dials the same way.
|
|
1610
|
-
- **TLS** is the openssl package's: `VERIFY_PEER` against the operating
|
|
1611
|
-
system's trust store (`SSL_CERT_FILE` / `SSL_CERT_DIR` point it
|
|
1612
|
-
elsewhere; the package has no `X509::Store` for a custom one), host-name
|
|
1613
|
-
checking, SNI, TLS 1.2 or newer. An https URL needs a host name: an IP
|
|
1614
|
-
address does not match a certificate's name. Without `require
|
|
1615
|
-
"cybertrain/net/tls"`, an https request raises `HttpClient::TLSError`
|
|
1616
|
-
saying so; `cybertrain/http_client` alone never links libssl.
|
|
1617
|
-
- **A notification client, not a browser:** HTTP/1.1 with `Connection:
|
|
1618
|
-
close`, one connection per request, no keep-alive, no redirects (a 3xx is
|
|
1619
|
-
returned as it is), no proxy, no cookies, no compression. Responses may be
|
|
1620
|
-
`Content-Length`, chunked or read to the close; the headers are capped at
|
|
1621
|
-
64 KiB and 100 lines, the body at `max_body` (default 1 MiB), past which
|
|
1622
|
-
`HttpClient::TooLargeError`. A response that is not HTTP/1.x, or ends
|
|
1623
|
-
early, is `HttpClient::BadResponseError`; a refused or reset connection, or
|
|
1624
|
-
a name that does not resolve, `HttpClient::RefusedError`. A connection
|
|
1625
|
-
that is reset or closed during the TLS handshake (a server that does not
|
|
1626
|
-
speak TLS) is `HttpClient::TLSError` ("TLS handshake failed") on every
|
|
1627
|
-
platform, like a handshake OpenSSL rejects. All of them are
|
|
1628
|
-
`HttpClient::Error`s.
|
|
1629
|
-
- **Errors never contain the path or the query**, where a webhook URL
|
|
1630
|
-
carries its secret: messages name the host and port only. A malformed URL
|
|
1631
|
-
or a header with CR/LF is an `ArgumentError` that does not repeat it
|
|
1632
|
-
either. `Host`, `Connection`, `Content-Length` and `Transfer-Encoding` are
|
|
1633
|
-
the client's; `User-Agent` defaults to `cybertrain/<version>`.
|
|
1634
|
-
|
|
1635
|
-
The request runs on the calling thread, a green thread like the request
|
|
1636
|
-
being served, so other connections keep being served while it waits. The
|
|
1637
|
-
socket layer (`cybertrain/net/socket.rb`: bounded connect, buffered reads,
|
|
1638
|
-
bounded writes) is separate from the HTTP layer so that other protocols can
|
|
1639
|
-
use it. The design: [docs/design.md](docs/design.md) D24.
|
|
1640
|
-
|
|
1641
|
-
## Consuming WebSocket feeds
|
|
1642
|
-
|
|
1643
|
-
An application that listens to an outside feed (market ticks, device
|
|
1644
|
-
telemetry, an event stream) while the server runs subscribes to it with
|
|
1645
|
-
`Cybertrain::WebSocket.subscribe`. Like the HTTP client it is not in
|
|
1646
|
-
`require "cybertrain"`; require it where the subscription is written, and the
|
|
1647
|
-
TLS layer too for `wss://`:
|
|
1648
|
-
|
|
1649
|
-
```ruby
|
|
1650
|
-
# config/app.rb (or a file bin/<app>.rb requires before Cybertrain::Main.run)
|
|
1651
|
-
require "json"
|
|
1652
|
-
require "cybertrain/websocket"
|
|
1653
|
-
require "cybertrain/net/tls" # only for wss://; links the system libssl
|
|
1654
|
-
|
|
1655
|
-
Cybertrain::WebSocket.subscribe("trades", ENV["FEED_URL"],
|
|
1656
|
-
headers: { "Authorization" => "Bearer #{ENV["FEED_TOKEN"]}" },
|
|
1657
|
-
stale_after: 30)
|
|
1658
|
-
.on_open { |conn| conn.send_text(JSON.generate({ "op" => "subscribe", "channel" => "trades" })) }
|
|
1659
|
-
.on_message { |conn, text| Cybertrain::DB.with { |c| c.execute("INSERT INTO trades(raw) VALUES (?)", [text]) } }
|
|
1660
|
-
```
|
|
1661
|
-
|
|
1662
|
-
Requiring `cybertrain/websocket` registers the feeds with
|
|
1663
|
-
`Cybertrain::Background`, which `Application#serve` starts after the
|
|
1664
|
-
scheduler and stops at shutdown, so there is nothing to call. Each
|
|
1665
|
-
subscription gets one thread. It connects, runs `on_open` (send your
|
|
1666
|
-
subscribe message there: it runs again after every reconnect), then hands
|
|
1667
|
-
each message to `on_message` with the `Connection` (`send_text`,
|
|
1668
|
-
`send_binary`, `close`, and `last_opcode`, `:text` or `:binary`). The
|
|
1669
|
-
blocks take their arguments and cannot use `self` (Spinel has no
|
|
1670
|
-
`instance_eval`).
|
|
1671
|
-
|
|
1672
|
-
- **When it runs.** The feeds start when the binary serves, like the
|
|
1673
|
-
scheduler ([Background work](#background-work)); `migrate`, `db ...`,
|
|
1674
|
-
`Main.command` commands and `Cybertrain::Test::Client` never start them.
|
|
1675
|
-
With `CYBERTRAIN_WORKER=0` (`config.worker = false`) they do not run in
|
|
1676
|
-
that process (they simply do not start; the only notice is the
|
|
1677
|
-
scheduler's "run elsewhere" line, and `WebSocket.start!` called directly
|
|
1678
|
-
logs its own): turn it off in all but one when
|
|
1679
|
-
several processes serve, or each one receives the whole feed.
|
|
1680
|
-
`subscribe` must be called before the server starts; there is no adding or
|
|
1681
|
-
removing a feed while it runs. A name is unique (`ArgumentError`), and a
|
|
1682
|
-
feed without `on_message` fails the boot before the port is bound
|
|
1683
|
-
(`error: WebSocket subscription "trades" has no on_message`, exit 1).
|
|
1684
|
-
- **Options.** `headers:` (extra handshake headers), `timeout:` (5 seconds
|
|
1685
|
-
per wait while connecting and per write), `max_message:` (1 MiB; a larger
|
|
1686
|
-
message closes the connection with 1009), `ping_interval:` (15) and
|
|
1687
|
-
`stale_after:` (45), and `allow_private:` (as `HttpClient`, below).
|
|
1688
|
-
- **Waiting and reconnecting.** After any failure the thread waits
|
|
1689
|
-
`min(60, 2 ** (attempt - 1))` seconds, with 20 % of jitter either way: 1, 2,
|
|
1690
|
-
4, ... 60. A connection that lasted 60 seconds starts again from 1. Every
|
|
1691
|
-
wait is bounded. `WebSocket.stop!` returns within about 2 seconds
|
|
1692
|
-
whatever the servers do: it waits at most 2 seconds for all the threads
|
|
1693
|
-
together, a connected feed sends a close (1000) and waits at most 1 second
|
|
1694
|
-
for the server's answer, and every feed's `connected?` is false once it
|
|
1695
|
-
returns (a thread still closing finishes on its own).
|
|
1696
|
-
- **Silence.** A feed can go quiet without ever closing the connection, so
|
|
1697
|
-
the thread pings after `ping_interval` seconds without any frame (and then
|
|
1698
|
-
once per interval), and when no frame at all, a pong included, has arrived
|
|
1699
|
-
for `stale_after` seconds it closes the connection and reconnects.
|
|
1700
|
-
`stale_after` must be more than `ping_interval`, which must be at least 1
|
|
1701
|
-
(`ArgumentError` from `subscribe`).
|
|
1702
|
-
- **Errors in your blocks.** An exception in `on_message` is logged and the
|
|
1703
|
-
connection stays. An exception in `on_open` is logged and the connection is
|
|
1704
|
-
closed and retried, since a feed that did not subscribe is of no use.
|
|
1705
|
-
- **What is logged.**
|
|
1706
|
-
|
|
1707
|
-
| When | Level | Line |
|
|
1708
|
-
| --- | --- | --- |
|
|
1709
|
-
| start | INFO | `WebSocket started: trades (wss://feed.example.com:443)` |
|
|
1710
|
-
| connected | INFO | `WebSocket "trades" connected to feed.example.com:443` |
|
|
1711
|
-
| disconnected | WARN | `WebSocket "trades" disconnected: Cybertrain::WebSocket::ClosedError: feed.example.com:443: closed by the server (1001) (retry in 2s)` |
|
|
1712
|
-
| silent | WARN | `WebSocket "trades" silent for 45s; reconnecting` |
|
|
1713
|
-
| a block raised | ERROR | `WebSocket "trades" on_message failed: KeyError: ...` (or `on_open`) |
|
|
1714
|
-
|
|
1715
|
-
The path and the query are never logged or put in an error message: they
|
|
1716
|
-
often carry a token.
|
|
1717
|
-
- **Errors.** All are `Cybertrain::WebSocket::Error`s, with host and port
|
|
1718
|
-
only in the message. Once the handshake is done, a lost connection or a
|
|
1719
|
-
close by the server is a `ClosedError` (`close_code`, `close_reason`; a
|
|
1720
|
-
reset or an end without a close frame is 1006); a write that times out or
|
|
1721
|
-
fails in TLS is still a `TimeoutError` or `TLSError`. While connecting it
|
|
1722
|
-
is `RefusedError`, `TimeoutError`, `TLSError`, `BlockedAddressError` (a non-public address,
|
|
1723
|
-
refused by default exactly as in [Outbound HTTP](#outbound-http), with
|
|
1724
|
-
the same `allow_private:` and `http_allow_private` setting) or
|
|
1725
|
-
`HandshakeError` (not a 101, a wrong `Sec-WebSocket-Accept`, or the
|
|
1726
|
-
server answering with an extension or subprotocol). `ProtocolError`
|
|
1727
|
-
(the server broke the framing; the client sends 1002) and
|
|
1728
|
-
`TooLargeError` can come at any time.
|
|
1729
|
-
- **Health.** `Cybertrain::WebSocket.feed_status("trades")` answers a
|
|
1730
|
-
`Status`: `connected?`, `last_message_at` (epoch seconds, 0 before the
|
|
1731
|
-
first message) and `reconnects`. It is not part of `GET /up`; use it in
|
|
1732
|
-
your own check. The three are read one after the other, not as one
|
|
1733
|
-
snapshot, and `connected?` turns true when the connection is made, before
|
|
1734
|
-
`on_open` has run.
|
|
1735
|
-
- **Keep `on_message` short.** The thread waits with `wait_readable` or
|
|
1736
|
-
`sleep`, so a stalled feed does not hold an OS worker while it waits. Two
|
|
1737
|
-
things do hold one: each connect attempt resolves the host name with a
|
|
1738
|
-
blocking `getaddrinfo` (see "Name resolution blocks" in [Outbound
|
|
1739
|
-
HTTP](#outbound-http); with `SPINEL_WORKERS=1` a dead resolver stalls
|
|
1740
|
-
every request for the resolver's timeout, once per attempt per feed, and
|
|
1741
|
-
an IP address in the URL avoids the lookup), and a SQLite call inside
|
|
1742
|
-
`on_message` until it returns, as in a scheduled task ([Background
|
|
1743
|
-
work](#background-work)): with `SPINEL_WORKERS=1` a long write stalls every
|
|
1744
|
-
request. For a busy feed insert one small row, or collect messages in
|
|
1745
|
-
memory and write them in batches from a `Scheduler.every` task.
|
|
1746
|
-
- **Not built.** Extensions (`permessage-deflate`) and subprotocols are not
|
|
1747
|
-
negotiated. There is no proxy, redirect or cookie support, no automatic
|
|
1748
|
-
saving of messages, and no WebSocket server (ActionCable). Text frames are
|
|
1749
|
-
not checked for valid UTF-8 (a departure from RFC 6455 section 8.1): the
|
|
1750
|
-
bytes are passed on as they came. The `Connection` is valid only during
|
|
1751
|
-
the callback; sending from a request or a task is not supported.
|
|
1752
|
-
|
|
1753
|
-
For one connection without the reconnect loop (a script, a one-shot read)
|
|
1754
|
-
use the synchronous client:
|
|
1755
|
-
|
|
1756
|
-
```ruby
|
|
1757
|
-
require "cybertrain/websocket"
|
|
1758
|
-
require "cybertrain/net/tls" # only for wss://
|
|
1759
|
-
|
|
1760
|
-
client = Cybertrain::WebSocket::Client.connect("wss://feed.example.com/v1", timeout: 5)
|
|
1761
|
-
client.send_text("hello")
|
|
1762
|
-
msg = client.receive(10) # waits at most 10 s; nil when nothing came
|
|
1763
|
-
msg.ws_opcode # :text or :binary
|
|
1764
|
-
msg.ws_data # the bytes
|
|
1765
|
-
client.close
|
|
1766
|
-
```
|
|
1767
|
-
|
|
1768
|
-
The design: [docs/design.md](docs/design.md) D33.
|
|
1769
|
-
|
|
1770
|
-
## Sending e-mail
|
|
1771
|
-
|
|
1772
|
-
`Cybertrain::Mailer` sends plain-text e-mail through an SMTP server (your
|
|
1773
|
-
provider's submission server, or a local relay). Like the HTTP client it is
|
|
1774
|
-
not in `require "cybertrain"`, and TLS is a separate require:
|
|
1775
|
-
|
|
1776
|
-
```ruby
|
|
1777
|
-
require "cybertrain/mailer"
|
|
1778
|
-
require "cybertrain/net/tls" # for starttls (the default) and tls; links the system libssl
|
|
1779
|
-
|
|
1780
|
-
# Settings from CYBERTRAIN_SMTP_HOST, _PORT, _USER, _PASSWORD, _FROM, _TLS:
|
|
1781
|
-
begin
|
|
1782
|
-
Cybertrain::Mailer.deliver(to: user.email, subject: "Welcome", body: "Hello #{user.name},\n...")
|
|
1783
|
-
rescue Cybertrain::Mailer::Error => e
|
|
1784
|
-
Cybertrain.logger.warn("mail: #{e.message}") # never contains the password
|
|
1785
|
-
end
|
|
1786
|
-
|
|
1787
|
-
# Or explicit settings, bypassing Config:
|
|
1788
|
-
mailer = Cybertrain::Mailer.new(host: "smtp.example.com", port: 587, tls: "starttls",
|
|
1789
|
-
user: "app", password: ENV["SMTP_PASSWORD"].to_s,
|
|
1790
|
-
from: "noreply@example.com", timeout: 10, deadline: 60)
|
|
1791
|
-
mailer.deliver(to: "a@example.com, b@example.com", subject: "Report", body: report)
|
|
1792
|
-
```
|
|
1793
|
-
|
|
1794
|
-
| Variable | Attribute | Default |
|
|
1795
|
-
| --- | --- | --- |
|
|
1796
|
-
| `CYBERTRAIN_SMTP_HOST` | `smtp_host` | `""` (`Mailer.new` raises `ArgumentError` until it is set) |
|
|
1797
|
-
| `CYBERTRAIN_SMTP_PORT` | `smtp_port` | `0`: 587 for `starttls`, 465 for `tls`, 25 for `none` |
|
|
1798
|
-
| `CYBERTRAIN_SMTP_USER` | `smtp_user` | `""`: no AUTH |
|
|
1799
|
-
| `CYBERTRAIN_SMTP_PASSWORD` | `smtp_password` | `""` (secret) |
|
|
1800
|
-
| `CYBERTRAIN_SMTP_FROM` | `smtp_from` | `""` |
|
|
1801
|
-
| `CYBERTRAIN_SMTP_TLS` | `smtp_tls` | `"starttls"`; `tls` or `none` |
|
|
1802
|
-
|
|
1803
|
-
- **TLS modes.** `starttls` connects in plain text, requires the server to
|
|
1804
|
-
offer STARTTLS (else `Mailer::TLSError`) and upgrades before anything
|
|
1805
|
-
else is said; bytes the server sends after its STARTTLS reply are
|
|
1806
|
-
refused. `tls` speaks TLS from the first byte (port 465). Both check the
|
|
1807
|
-
certificate against `host` as the HTTP client does (give a host name, not
|
|
1808
|
-
an IP address), and both need `require "cybertrain/net/tls"`: without
|
|
1809
|
-
it, `deliver` raises `Mailer::TLSError` saying so, before connecting.
|
|
1810
|
-
`none` is plain text, for a relay on the same machine or a test server:
|
|
1811
|
-
with a user configured it refuses (`Mailer::InsecureError`, before
|
|
1812
|
-
connecting) unless the host resolves to a loopback address, and then
|
|
1813
|
-
dials only its loopback addresses.
|
|
1814
|
-
- **The password is never logged or shown.** It is not in the boot banner,
|
|
1815
|
-
`help`, the log or any error message; it is sent only inside the AUTH
|
|
1816
|
-
PLAIN command, over TLS (or to loopback). Do not `p` or `inspect` a
|
|
1817
|
-
`Mailer` or the `Config`, which hold it.
|
|
1818
|
-
- **Every wait is bounded**, as in "Outbound HTTP": connecting, each TLS
|
|
1819
|
-
step and each reply waits at most `timeout` seconds (default 10), the
|
|
1820
|
-
whole delivery at most `deadline` seconds (default 60), else
|
|
1821
|
-
`Mailer::TimeoutError`. Name resolution blocks the calling thread.
|
|
1822
|
-
- **The dialogue:** the 220 greeting, `EHLO` (the name is `helo:`, default
|
|
1823
|
-
`localhost`), `STARTTLS` and `EHLO` again, `AUTH PLAIN` when a user is
|
|
1824
|
-
set (the server must offer PLAIN), `MAIL FROM`, one `RCPT TO` per
|
|
1825
|
-
recipient, `DATA`, `QUIT`. A reply with another code is a
|
|
1826
|
-
`Mailer::RejectedError` whose `smtp_code` is the server's (421, 535,
|
|
1827
|
-
550, ...) and whose message has the step and the server's text. The
|
|
1828
|
-
return value is the server's reply to the message (`"250 2.0.0 Ok: queued
|
|
1829
|
-
as ..."`).
|
|
1830
|
-
- **The message:** `Date`, `From`, `To`, `Subject`, `MIME-Version: 1.0`,
|
|
1831
|
-
`Content-Type: text/plain; charset=UTF-8`. A non-ASCII subject is RFC 2047
|
|
1832
|
-
encoded (`=?UTF-8?B?...?=`, split between characters). The body's line
|
|
1833
|
-
endings become CRLF and lines starting with `.` are dot-stuffed. Its
|
|
1834
|
-
`Content-Transfer-Encoding` is `7bit` for ASCII, `8bit` (with `MAIL FROM
|
|
1835
|
-
... BODY=8BITMIME`) for UTF-8 when the server advertises 8BITMIME, and
|
|
1836
|
-
`base64` otherwise or when a line is longer than 998 bytes. No
|
|
1837
|
-
`Message-ID` is added (submission servers add one).
|
|
1838
|
-
- **What is refused** with an `ArgumentError`, before connecting: an
|
|
1839
|
-
address that is not a bare ASCII `local@domain` (no display name, no
|
|
1840
|
-
`<>`, spaces, commas, quotes, CR or LF; at most 254 bytes), a subject with
|
|
1841
|
-
CR, LF or NUL (header injection) or over 998 bytes, a body that is not
|
|
1842
|
-
valid UTF-8 or holds a NUL, more than 100 recipients. `to:` takes one
|
|
1843
|
-
address or several separated by commas.
|
|
1844
|
-
- **Not yet:** HTML and multipart messages, attachments, templates, display
|
|
1845
|
-
names, `Cc`/`Bcc` headers (send to several recipients instead), AUTH
|
|
1846
|
-
LOGIN/CRAM-MD5, SMTPUTF8 addresses, pipelining, and a delivery queue (a
|
|
1847
|
-
delivery runs on the calling thread, a green thread like a request).
|
|
1848
|
-
`cybertrain generate authentication` still writes no password reset.
|
|
1849
|
-
|
|
1850
|
-
The design: [docs/design.md](docs/design.md) D27.
|
|
1851
|
-
|
|
1852
|
-
## Testing an app
|
|
1853
|
-
|
|
1854
|
-
App tests are plain Spinel programs under `test/`, like the framework's own:
|
|
1855
|
-
|
|
1856
|
-
```ruby
|
|
1857
|
-
require_relative "support/blog_test" # the app, BLOG and Cybertrain::Test
|
|
1858
|
-
|
|
1859
|
-
test "title must be present" do
|
|
1860
|
-
article = Article.new(body: "a body long enough to pass length")
|
|
1861
|
-
refute article.save
|
|
1862
|
-
assert_includes article.errors.full_messages, "Title can't be blank"
|
|
1863
|
-
end
|
|
1864
|
-
```
|
|
1865
|
-
|
|
1866
|
-
Assertions: `assert`, `refute`, `assert_equal`, `assert_nil`,
|
|
1867
|
-
`assert_includes`, `assert_raises("Class") { }`, `flunk`. For controllers and
|
|
1868
|
-
full HTTP flows, `Cybertrain::Test::Client` drives an app in-process (no
|
|
1869
|
-
socket) with a cookie jar, so sessions and CSRF behave as behind a browser:
|
|
1870
|
-
|
|
1871
|
-
```ruby
|
|
1872
|
-
require_relative "support/blog_test" # the app, BLOG and Cybertrain::Test
|
|
1873
|
-
|
|
1874
|
-
test "POST /articles creates an article and redirects to it" do
|
|
1875
|
-
client = Cybertrain::Test::Client.new(BLOG)
|
|
1876
|
-
client.get("/articles/new")
|
|
1877
|
-
res = client.post("/articles", { "authenticity_token" => BlogTest.form_token(client),
|
|
1878
|
-
"article[title]" => "Hello Rails",
|
|
1879
|
-
"article[body]" => "I am on Rails! This is my first article." })
|
|
1880
|
-
assert_redirected_to res, BlogTest.article_path(Article.last)
|
|
1881
|
-
assert_response res, :see_other
|
|
1882
|
-
end
|
|
1883
|
-
```
|
|
1884
|
-
|
|
1885
|
-
Both snippets start from the example's helper,
|
|
1886
|
-
[`examples/blog/test/support/blog_test.rb`](examples/blog/test/support/blog_test.rb):
|
|
1887
|
-
copy it into an app's `test/support/`, where files are required, not run.
|
|
1888
|
-
It migrates a fresh `storage/test.sqlite3`, boots the app as `BLOG`, loads
|
|
1889
|
-
`Cybertrain::Test` and its client, and reads the CSRF token out of the last
|
|
1890
|
-
rendered form (`BlogTest.form_token`). End each test file with
|
|
1891
|
-
`Cybertrain::Test.run!`; `examples/blog/test/articles.rb` and `comments.rb`
|
|
1892
|
-
show the full pattern.
|
|
1893
|
-
|
|
1894
|
-
Every request a test client sends comes from `127.0.0.1` (`request.remote_addr`,
|
|
1895
|
-
so `request.loopback?` is true). To see what a page does for a visitor on
|
|
1896
|
-
another machine, set `client.peer_address = "203.0.113.7"` before the
|
|
1897
|
-
request.
|
|
1898
|
-
|
|
1899
|
-
Run with `cybertrain spin test`: it compiles each `test/*.rb` into its own
|
|
1900
|
-
program and diffs its output (stdout and stderr together) against
|
|
1901
|
-
`test/<name>.rb.expected`. Regenerate with `cybertrain spin test --regen
|
|
1902
|
-
test/<name>.rb`; FFI/database tests can't run under CRuby, so their snapshot
|
|
1903
|
-
comes from the compiled binary instead: `rm -f build/test/<name>; cybertrain
|
|
1904
|
-
spin test test/<name>.rb; [ -x build/test/<name> ] && ./build/test/<name> >
|
|
1905
|
-
test/<name>.rb.expected 2>&1` (`cybertrain spin test` exits non-zero while the
|
|
1906
|
-
snapshot is stale, so it is followed by `;`, not `&&`; the `[ -x ]` check skips
|
|
1907
|
-
the capture when the compile failed, so the old snapshot is kept; in the
|
|
1908
|
-
framework checkout, `script/regen-snapshot test/<name>.rb` does exactly this and
|
|
1909
|
-
prints the diff stat).
|
|
1910
|
-
|
|
1911
|
-
### Testing outbound calls
|
|
1912
|
-
|
|
1913
|
-
When the code under test sends a webhook ([Outbound HTTP](#outbound-http)) or
|
|
1914
|
-
an e-mail ([Sending e-mail](#sending-e-mail)), the test receives it
|
|
1915
|
-
in-process: `Cybertrain::Test::HttpReceiver` and `Cybertrain::Test::SmtpReceiver`
|
|
1916
|
-
listen on `127.0.0.1`, record what arrives and answer with a canned reply.
|
|
1917
|
-
|
|
1918
|
-
```ruby
|
|
1919
|
-
require_relative "support/app_test"
|
|
1920
|
-
require "cybertrain/test/http_receiver"
|
|
1921
|
-
require "cybertrain/test/smtp_receiver"
|
|
1922
|
-
|
|
1923
|
-
test "a sign-up posts a webhook" do
|
|
1924
|
-
hook = Cybertrain::Test::HttpReceiver.new(status: 202, body: "queued")
|
|
1925
|
-
hook.start # binds a free port, returns it
|
|
1926
|
-
Cybertrain.config.http_allow_private = true # HttpClient refuses loopback otherwise
|
|
1927
|
-
SignupHook.post(hook.url("/hooks/signup"), "ann@example.com") # the app's code
|
|
1928
|
-
req = hook.received_request # waits at most 5 s; nil if nothing came
|
|
1929
|
-
hook.stop
|
|
1930
|
-
assert_equal "POST", req.http_method
|
|
1931
|
-
assert_equal "/hooks/signup", req.http_target
|
|
1932
|
-
assert_equal "application/json", req.http_header("Content-Type")
|
|
1933
|
-
assert_includes req.http_body, "ann@example.com"
|
|
1934
|
-
end
|
|
1935
|
-
|
|
1936
|
-
test "a refused recipient is reported" do
|
|
1937
|
-
smtp = Cybertrain::Test::SmtpReceiver.new(reject: { rcpt: "550 5.1.1 no such user" })
|
|
1938
|
-
port = smtp.start
|
|
1939
|
-
mailer = Cybertrain::Mailer.new(host: "127.0.0.1", port: port, tls: "none", from: "app@example.com")
|
|
1940
|
-
assert_raises("RejectedError") { mailer.deliver(to: "nobody@example.org", subject: "Hi", body: "Hello\n") }
|
|
1941
|
-
mail = smtp.received_message
|
|
1942
|
-
smtp.stop
|
|
1943
|
-
assert_equal ["EHLO localhost", "MAIL FROM:<app@example.com>", "RCPT TO:<nobody@example.org>"],
|
|
1944
|
-
mail.transcript.first(3)
|
|
1945
|
-
refute mail.delivered?
|
|
1946
|
-
end
|
|
1947
|
-
```
|
|
1948
|
-
|
|
1949
|
-
`HttpReceiver.new` takes the canned `status:`, `headers:`, `body:` and
|
|
1950
|
-
`chunked:` (`respond_with` changes them between requests); the request comes
|
|
1951
|
-
back as a `ReceivedRequest` (`http_method`, `http_target`, `http_path`,
|
|
1952
|
-
`http_headers` with lowercased names, `http_header(name)`, `http_body` with a
|
|
1953
|
-
chunked body decoded). `SmtpReceiver` plays 220, EHLO (with 8BITMIME unless
|
|
1954
|
-
`eight_bit_mime: false`, AUTH PLAIN with `auth: true`), MAIL, RCPT, DATA and
|
|
1955
|
-
QUIT; `reject:` fails any of `:greeting`, `:ehlo`, `:mail`, `:rcpt`, `:data`
|
|
1956
|
-
or `:message` with the reply given. A `ReceivedMessage` has the
|
|
1957
|
-
`envelope_from`, `envelope_recipients`, the `message_data` (dot-stuffing
|
|
1958
|
-
undone), `message_header(name)`, `message_body` and the `transcript` of
|
|
1959
|
-
commands. Both accept one connection unless `max:` says more
|
|
1960
|
-
(`received_requests` / `received_messages` return them all). Every wait is
|
|
1961
|
-
bounded: a receiver gives up after `accept_window:` (10 s) without a
|
|
1962
|
-
connection, a silent client after `read_timeout:` (5 s), and `stop` (safe
|
|
1963
|
-
to call twice) ends and joins the receiver's thread and closes its port, so
|
|
1964
|
-
a receiver never takes a later test's connection. They need only Ruby's
|
|
1965
|
-
socket library, not openssl.
|
|
1966
|
-
|
|
1967
|
-
A feed is tested the same way with `Cybertrain::Test::WebSocketReceiver`, a
|
|
1968
|
-
scripted WebSocket server on `127.0.0.1`: each connection plays a list of
|
|
1969
|
-
`Cybertrain::Test::WsStep`s (`"accept"`, `"send_text"`, `"expect"`,
|
|
1970
|
-
`"silent"`, `"close"`, `"drop"`, `"echo_until_close"`, ...), and the frames
|
|
1971
|
-
the client sent come back unmasked from `ws_received`.
|
|
1972
|
-
|
|
1973
|
-
```ruby
|
|
1974
|
-
require "cybertrain/websocket"
|
|
1975
|
-
require "cybertrain/test/websocket_receiver"
|
|
1976
|
-
|
|
1977
|
-
TICKS = Array.new(0) { "" } # a constant: the blocks cannot see locals
|
|
1978
|
-
|
|
1979
|
-
# The blocks are written in a method, where their parameters are used,
|
|
1980
|
-
# not inside the test block.
|
|
1981
|
-
def subscribe_trades(url)
|
|
1982
|
-
Cybertrain::WebSocket.subscribe("trades", url, allow_private: true)
|
|
1983
|
-
.on_open { |conn| conn.send_text("subscribe") }
|
|
1984
|
-
.on_message { |conn, text| TICKS << text }
|
|
1985
|
-
end
|
|
1986
|
-
|
|
1987
|
-
test "a trade message is stored" do
|
|
1988
|
-
rx = Cybertrain::Test::WebSocketReceiver.new
|
|
1989
|
-
rx.ws_script([Cybertrain::Test::WsStep.new("accept"), Cybertrain::Test::WsStep.new("expect"),
|
|
1990
|
-
Cybertrain::Test::WsStep.new("send_text", "tick"), Cybertrain::Test::WsStep.new("echo_until_close")])
|
|
1991
|
-
rx.start
|
|
1992
|
-
subscribe_trades(rx.url("/trades"))
|
|
1993
|
-
Cybertrain::WebSocket.start! # Test::Client never starts feeds
|
|
1994
|
-
# ... poll TICKS for at most a few seconds ...
|
|
1995
|
-
Cybertrain::WebSocket.stop!
|
|
1996
|
-
assert_equal ["tick"], TICKS
|
|
1997
|
-
assert_equal "subscribe", rx.ws_received[0].rec_data
|
|
1998
|
-
rx.stop
|
|
1999
|
-
end
|
|
2000
|
-
```
|
|
2001
|
-
|
|
2002
|
-
`Cybertrain::WebSocket.random_source = -> { 0.5 }` makes the retry delays exact.
|
|
2003
|
-
`test/websocket_integration.rb` in this repository shows a complete test.
|
|
2004
|
-
|
|
2005
|
-
### Continuous integration
|
|
2006
|
-
|
|
2007
|
-
`cybertrain new` writes `.github/workflows/ci.yml` for the app. It calls the
|
|
2008
|
-
`setup-cybertrain` action of this repository
|
|
2009
|
-
([.github/actions/setup-cybertrain](.github/actions/setup-cybertrain/action.yml)),
|
|
2010
|
-
pinned to the CLI's release tag, which installs the SQLite, zlib and OpenSSL
|
|
2011
|
-
headers and Ruby, reads the framework commit the app is locked to out of
|
|
2012
|
-
`spin.lock`, fetches the framework at that commit and runs its `cybertrain
|
|
2013
|
-
doctor` and `cybertrain setup` (with `~/.cybertrain/spinel` cached per Spinel
|
|
2014
|
-
release and runner image). The workflow then checks that `spin.lock` is
|
|
2015
|
-
current (`spin lock && git diff --exit-code spin.lock`) and `gen/` is fresh
|
|
2016
|
-
(`spin run gen -- --check`), and runs `spin test` once `test/` has a test
|
|
2017
|
-
program. When the toolchain steps change, the action changes, not every app's
|
|
2018
|
-
copy. The commit is read from `spin.lock`, so the workflow assumes the
|
|
2019
|
-
default git dependency; an app made with `--path` passes the framework
|
|
2020
|
-
checkout to the action as `framework-path`.
|
|
2021
|
-
|
|
2022
|
-
## Learn more
|
|
2023
|
-
|
|
2024
|
-
- [Homepage](https://saeki-mototsune.github.io/CyberTrain/),
|
|
2025
|
-
[tutorial](https://saeki-mototsune.github.io/CyberTrain/tutorial.html) and
|
|
2026
|
-
[playground](https://saeki-mototsune.github.io/CyberTrain/playground.html) —
|
|
2027
|
-
the source lives in [site/](site/), published by
|
|
2028
|
-
[.github/workflows/pages.yml](.github/workflows/pages.yml); the logo files and
|
|
2029
|
-
brand notes are in [site/assets/brand/](site/assets/brand/BRAND.md).
|
|
2030
|
-
- [API reference](https://saeki-mototsune.github.io/CyberTrain/api/) — every
|
|
2031
|
-
class and method an application uses (controllers, params, session and
|
|
2032
|
-
flash, models and relations, routes, migrations, configuration, tests),
|
|
2033
|
-
including what `spin run gen` generates. It is built by YARD
|
|
2034
|
-
(`script/api-docs`, into `site/api/`) from the doc comments marked
|
|
2035
|
-
`@api public` in `cybertrain/` and from [docs/api/](docs/api/); a method
|
|
2036
|
-
without that tag is internal and stays out. CI fails on any YARD warning.
|
|
2037
|
-
- [docs/design.md](docs/design.md) — the design record (Japanese): every
|
|
2038
|
-
decision, what was rejected and why, and the Spinel constraints behind it.
|
|
2039
|
-
- [docs/template-language.md](docs/template-language.md) — the full template
|
|
2040
|
-
grammar and helpers, and where it differs from Rails' ERB.
|
|
2041
|
-
- [docs/benchmark.md](docs/benchmark.md) — examples/blog measured against the
|
|
2042
|
-
same blog in Rails 8.1 ([bench/](bench/)), and how to run it again.
|
|
2043
|
-
- [spikes/NOTES.md](spikes/NOTES.md) — the spikes that answered
|
|
2044
|
-
`docs/design.md`'s open questions, and the compiler constraints they found.
|
|
2045
|
-
- [Spinel](https://github.com/matz/spinel) — the AOT Ruby compiler cybertrain targets;
|
|
2046
|
-
cybertrain pins a release of the
|
|
2047
|
-
[saeki-mototsune/spinel](https://github.com/saeki-mototsune/spinel) fork.
|
|
2048
|
-
- [Roundhouse](https://github.com/rubys/roundhouse) (Sam Ruby) — transpiles
|
|
2049
|
-
*existing* Rails apps to Spinel. cybertrain is not that: it's a native
|
|
2050
|
-
framework you write directly against, not a compatibility layer.
|
|
91
|
+
| [Tutorial](https://saeki-mototsune.github.io/CyberTrain/tutorial.html) | Build the blog step by step: scaffold, comments, authentication, uploads, deployment |
|
|
92
|
+
| [Installing](https://saeki-mototsune.github.io/CyberTrain/guide/install.html) | Requirements, the `cybertrain` command, how Spinel and the framework are installed |
|
|
93
|
+
| [How it works](https://saeki-mototsune.github.io/CyberTrain/guide/how-it-works.html) | Build-time code generation, interpreted views, the server, the development loop, and a table of the differences from Rails |
|
|
94
|
+
| [Models and the database](https://saeki-mototsune.github.io/CyberTrain/guide/models.html) | Migrations, column types, queries, associations, validations, encrypted columns, transactions, taking over a Rails database |
|
|
95
|
+
| [Requests](https://saeki-mototsune.github.io/CyberTrain/guide/requests.html) | Routing, formats and `respond_to`, concerns, strong parameters, HTTP caching, cookies and the session, rate limiting, CSP, `force_ssl` and hosts, headers, query strings and request bodies |
|
|
96
|
+
| [Authentication](https://saeki-mototsune.github.io/CyberTrain/guide/authentication.html) | `generate authentication`: sign-in, sign-up or invite-only, password reset, confirmation, lockout, remember me, the account page, admin users, upgrading from 0.4 |
|
|
97
|
+
| [Assets](https://saeki-mototsune.github.io/CyberTrain/guide/assets.html) | Digested stylesheets, images and JavaScript, CSS `url()`, the helpers, the import map, serving |
|
|
98
|
+
| [Internationalization](https://saeki-mototsune.github.io/CyberTrain/guide/i18n.html) | Dictionaries in `config/locales`, `t` and `l`, lazy lookup, plural forms, the locale per request, translated validation messages |
|
|
99
|
+
| [Configuration](https://saeki-mototsune.github.io/CyberTrain/guide/configuration.html) | `config/app.rb`, environment variables, time zones |
|
|
100
|
+
| [Background work and operations](https://saeki-mototsune.github.io/CyberTrain/guide/background-and-operations.html) | Job classes, retries and concurrency keys (delivered at least once, so make jobs idempotent), `every` and `cron` tasks, the jobs page, the `worker` and `feeds` process roles, the cache store, the console, backups, the health check |
|
|
101
|
+
| [Realtime](https://saeki-mototsune.github.io/CyberTrain/guide/realtime.html) | Turbo Streams over Server-Sent Events or WebSocket, broadcasts from actions, jobs and feeds, signed stream names, feed health |
|
|
102
|
+
| [Outbound HTTP, WebSocket feeds and e-mail](https://saeki-mototsune.github.io/CyberTrain/guide/external-services.html) | `HttpClient` (streaming, gzip, redirects, Server-Sent Events), `WebSocket.subscribe` (timers, restarts), `Mailer` (HTML mail, `deliver_later`) |
|
|
103
|
+
| [Testing](https://saeki-mototsune.github.io/CyberTrain/guide/testing.html) | Test classes, fixtures, `cybertrain test`, time travel, components and streams, mail, jobs, outbound calls, CI |
|
|
104
|
+
| [Deploying](https://saeki-mototsune.github.io/CyberTrain/deploy.html) | To a server with systemd, or with Kamal |
|
|
105
|
+
| [API reference](https://saeki-mototsune.github.io/CyberTrain/api/) | Every class and method an application uses; also [the template language](https://saeki-mototsune.github.io/CyberTrain/guide/template-language.html) |
|
|
106
|
+
| [Design record](docs/design.md) · [benchmark](docs/benchmark.md) · [spikes](spikes/NOTES.md) | Why it is built this way, and the Spinel constraints behind it |
|
|
107
|
+
|
|
108
|
+
The guides' Markdown sources are in [docs/guide/](docs/guide/).
|
|
2051
109
|
|
|
2052
110
|
## Contributing
|
|
2053
111
|
|
|
2054
|
-
Bug reports, questions and pull requests are welcome
|
|
2055
|
-
|
|
2056
|
-
|
|
2057
|
-
does), and what a pull request needs. Security issues go through
|
|
2058
|
-
[SECURITY.md](SECURITY.md), not an issue.
|
|
2059
|
-
|
|
2060
|
-
## Releasing
|
|
2061
|
-
|
|
2062
|
-
The gem and the framework are released from the same tag, and `cybertrain
|
|
2063
|
-
new` depends on the tag `v` + `Cybertrain::VERSION`, so the tag must exist
|
|
2064
|
-
before the gem is pushed:
|
|
2065
|
-
|
|
2066
|
-
1. Bump `Cybertrain::VERSION` (`cybertrain/version.rb`) and `version` in
|
|
2067
|
-
`spin.toml` together (CI checks they match); spin caches a git
|
|
2068
|
-
dependency by that version.
|
|
2069
|
-
2. Merge to `main`, then tag and push: `git tag vX.Y.Z && git push origin vX.Y.Z`.
|
|
2070
|
-
3. `gem build cybertrain.gemspec && gem push cybertrain-X.Y.Z.gem`.
|
|
2071
|
-
4. The tag also publishes the playground image
|
|
2072
|
-
`ghcr.io/saeki-mototsune/cybertrain-playground:X.Y.Z` (and `latest`) through
|
|
2073
|
-
[.github/workflows/playground-image.yml](.github/workflows/playground-image.yml);
|
|
2074
|
-
check that run.
|
|
2075
|
-
|
|
2076
|
-
`Cybertrain::SPINEL_TAG` (also in `cybertrain/version.rb`) is the Spinel
|
|
2077
|
-
release `cybertrain setup` installs. It must equal `SPINEL_TAG` in
|
|
2078
|
-
[.github/workflows/ci.yml](.github/workflows/ci.yml) and in
|
|
2079
|
-
[script/claude-cloud-setup](script/claude-cloud-setup), which `script/gate`
|
|
2080
|
-
checks, so change the three together when the framework moves to another
|
|
2081
|
-
Spinel release.
|
|
112
|
+
Bug reports, questions and pull requests are welcome; see
|
|
113
|
+
[CONTRIBUTING.md](CONTRIBUTING.md) (setup, tests, the commit gate, releasing).
|
|
114
|
+
Security issues go through [SECURITY.md](SECURITY.md).
|
|
2082
115
|
|
|
2083
116
|
## License
|
|
2084
117
|
|
|
2085
|
-
MIT; see [LICENSE](LICENSE).
|
|
118
|
+
MIT; see [LICENSE](LICENSE).
|