monkrb 0.22.2 → 0.23.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c4e809c375ae56d6ec8e0831623b9fb588550bc329e68a3c722cc81910e6bcc8
4
- data.tar.gz: 542338ca1dd6750276d91fedce501c02e06d68506f6bbf6ffaafea64a9485611
3
+ metadata.gz: ef86c4755d55dba598019b3964d0c9abd29176f4a6bf2807a699e1ac883202e5
4
+ data.tar.gz: 6b7082937eb3b58f03ebe309df0ad6a2eed1722347d38ffce2e4d24cfdd4a92a
5
5
  SHA512:
6
- metadata.gz: aa576c261a4aaa57a117b163202a9c2b0b4865eb20fec828bebf39c9ef8e76bb868e809913c216773843cc69264b04471457b54c0124005db85cfc87d49b585b
7
- data.tar.gz: 3844bc6db9d0b2edf4cc4cdeed5ac5dbe8e3035ac6634c7f30e525cbf6705e140ca0dcc2e30993e92ece55cb8aecb259d0dfdd5cca73232886875f8ab01e859b
6
+ metadata.gz: 65464f1edfc7ec5edcb77a4f2fed00b0a94ba4e51f534b7e3f089b9e90b578a352d58d32b58f77e9657aa1329ca26d2446beb118bce311652fbe25c0ff220d9c
7
+ data.tar.gz: 9801a314ff3b7a299626fd4085752a87079bfd6b1a4110021f117906a5e4c4c5040fb82c0ea3dad74510dd34491ae71211124cd6ba9b0db5bdea35285b3622db
data/CHANGELOG.md CHANGED
@@ -4,6 +4,84 @@ All notable changes to this project are documented here. Format is loosely
4
4
  [Keep a Changelog](https://keepachangelog.com/); versions are as released
5
5
  in `lib/monk/version.rb`.
6
6
 
7
+ ## 0.23.1 - 2026-11-10
8
+
9
+ ### Changed
10
+
11
+ - **Fewer allocations per request in models and views.** On Ruby 4.0
12
+ every GC stops every worker Ractor, so allocations are what limit a
13
+ process's scaling. A board page (50 messages through a partial) goes
14
+ from 2,157 objects to 1,063, and a JSON read of the same messages
15
+ from 998 to 748, with byte-identical responses; GC time per request
16
+ falls about 30%.
17
+ - `Pg::Model` asks pg for Symbol-keyed rows (`field_name_type`)
18
+ instead of transforming String keys: one Hash per row, not two.
19
+ - An Array condition in `where` (and `find_all`) is one array
20
+ parameter, `col = ANY($1)`, instead of `col IN ($1, ..., $n)`;
21
+ a list of Integers is encoded without a String per id. Same rows,
22
+ nil items included.
23
+ - `<%= %>` escapes with `ERB::Escape.html_escape`, which returns
24
+ text with nothing to escape as is, and appends the result without
25
+ wrapping it in `Raw`; a template's buffer is a `Raw` from the start
26
+ rather than copied into one. `h` uses `ERB::Escape` too.
27
+ - `Context#render` no longer allocates its header key per template.
28
+ - **Route blocks run as methods.** `freeze!` defines each route and
29
+ `error` block as a private method of `Monk::Context`, once, and a
30
+ request calls it instead of `instance_exec`-ing the block. Same `self`,
31
+ ivars and return value; a block taking the context (`|c|`) still gets
32
+ it, and one needing two or more arguments keeps `instance_exec`. On
33
+ Ruby 4.1.0-preview1, `instance_exec` in a worker Ractor stops that
34
+ Ractor's own GCs: the heap grew to 1–3.8 GB between global,
35
+ stop-the-world GCs, and 4 workers served 64K req/s. As methods: 148K
36
+ req/s at 225 MB, against 120K at 86 MB on 4.0.7 (`GET /health`, 4
37
+ cores, locally). No change on 4.0.
38
+
39
+ ## 0.23.0 - 2026-10-26
40
+
41
+ ### Added
42
+
43
+ - **File storage, `Monk::Storage`** (`require "monk/storage"`, ADR 0018,
44
+ `docs/guides/storage.md`). Files are stored by key in any S3-compatible
45
+ bucket (Hetzner, Cloudflare R2, Backblaze B2, AWS, MinIO), chosen by
46
+ one `STORAGE_URL`, and on the local disk in development and test.
47
+ `put`, `get` (with `range:`), `head`, `delete`, `copy`, signed download
48
+ links (`url`, the same URL within an hour so browsers cache it) and
49
+ browser uploads (`presigned_put`, with the type and length inside the
50
+ signature), all from any worker Ractor. SigV4 is Monk's own, with no
51
+ gem. Uploads go from the browser straight to the bucket and downloads
52
+ redirect to it, so no worker Ractor waits on the bytes. The file
53
+ backend behaves like a bucket, signed links and refusals included, at
54
+ `/_monk/storage`. `Monk::Storage::Sniff` tells PNG, JPEG, GIF, WebP,
55
+ Ogg/Opus and PDF from a file's first 64 bytes, and refuses SVG.
56
+ Checked on Hetzner Object Storage: the signed upload limits, CORS for
57
+ browser uploads, the lifecycle rule that expires `tmp/`, and `copy`
58
+ with both addressing styles (`bin/storage_check`).
59
+ - **`monk add storage`**: `config/storage.rb`, the upload, confirm and
60
+ download flow as commented examples in `app/routes/storage.rb` (driven
61
+ with `curl`), and a test. No migration and no JavaScript: the guide
62
+ has an uploads table and the browser code to copy. An existing app
63
+ gets it the same way (`docs/guides/upgrading.md`, 0.23).
64
+ - **`bin/storage_check`**, in the monk repository: checks a real bucket's
65
+ CORS rule for browser uploads and its `tmp/` lifecycle rule, then
66
+ removes both.
67
+
68
+ ### Changed
69
+
70
+ - **A new app runs as production in its Docker image.** The scaffolded
71
+ `Dockerfile` sets `MONK_ENV=production`, which a platform's env can
72
+ override (`staging`). Unset, `MONK_ENV` means development, and a deploy
73
+ that forgot it ran that way without an error: mail printed instead of
74
+ sent, magic links logged, auth cookies without `Secure`, demo routes on,
75
+ storage links signed with a public value. `bin/server` now warns when it
76
+ listens beyond the machine (`--bind 0.0.0.0`) with `MONK_ENV` unset,
77
+ `monk new` lists `MONK_ENV` among the values to set before production,
78
+ and `docs/guides/deploying.md` sets it in every platform's example
79
+ (Render's start command is now `bin/server`, so the warning applies
80
+ there too). The deploy guide also gains storage's row.
81
+ - **A new app's `Gemfile` asks for `monkrb ~> 0.23`.**
82
+ - `docs/framework-comparison.md` is recomputed for 0.23.0, and
83
+ `docs/roadmap.md` moves storage up to 0.23 (it was planned as 0.26).
84
+
7
85
  ## 0.22.2 - 2026-10-22
8
86
 
9
87
  ### Fixed
data/README.md CHANGED
@@ -21,7 +21,7 @@ bin/server # -> http://localhost:9292/hello
21
21
  monk add --list # the modules you can add: postgres, auth, jobs, live...
22
22
  ```
23
23
 
24
- `monk new` scaffolds the app's own `Gemfile` with `gem "monkrb", "~> 0.22", require: "monk"`.
24
+ `monk new` scaffolds the app's own `Gemfile` with `gem "monkrb", "~> 0.23", require: "monk"`.
25
25
 
26
26
  `monk new` writes a working skeleton (an HTML home page, a `/hello` route, a `/api/hello` JSON route, `public/`, its tests, a `SETUP.md` and an `AGENTS.md` for coding agents), with the app's own code under `app/`: `app/app.rb` and `app/routes/` for routes, `app/views/`, and one directory per role (`models/`, `presenters/`, `helpers/`, `mailers/`, `broadcasts/`, `jobs/`). `monk add` adds Postgres, Redis, email, auth, background jobs, WebSocket and live updates, to a new app or an old one, each with its wiring, commented examples of how to use it, and a test (`monk new my_app --with auth,jobs` does both at once): see [`docs/guides/scaffolding.md`](docs/guides/scaffolding.md), which also says where each kind of code goes.
27
27
 
@@ -65,6 +65,7 @@ Everything beyond the core is opt-in (`require "monk"` alone loads none of it).
65
65
  | Background jobs | `Monk::Jobs`: a queue in Postgres, run by `bin/jobs` on worker Ractors; retries with backoff, scheduled jobs, recurring jobs on a cron schedule (UTC), enqueue inside the app's own transaction, `drain!` for tests | `require "monk/jobs"` (+ `require "monk/jobs/runtime"` in the job process); needs the `pg` gem and a registered Postgres connection | [`jobs.md`](docs/guides/jobs.md) |
66
66
  | WebSocket | `Monk::WebSocket`: RFC 6455 server as its own process, Redis or Postgres fan-out | `require "monk/websocket"`; fan-out: `require "monk/websocket/redis_fanout"` (+ the `redis` gem) or `require "monk/websocket/pg_fanout"` (+ the `pg` gem) | [`websocket.md`](docs/guides/websocket.md) |
67
67
  | Live updates | `Monk::Live`: server-rendered HTML patches pushed to open tabs | `require "monk/live"`; needs the WebSocket server and, across processes, Redis or Postgres | [`live.md`](docs/guides/live.md) |
68
+ | File storage | `Monk::Storage`: files by key in any S3-compatible bucket (Hetzner, R2, B2, AWS), local files in development and test; browser uploads and downloads through signed links, never through a worker Ractor | `require "monk/storage"`; no gem needed | [`storage.md`](docs/guides/storage.md) |
68
69
  | Scaffolding | `monk new` and `monk add`: one generator per module, for a new app or an existing one; text or `--json` output | — (the `monk` command: `monk new NAME --with ...`, `monk add MODULE`, `monk add --list`) | [`scaffolding.md`](docs/guides/scaffolding.md) |
69
70
 
70
71
  ## More documentation
@@ -88,6 +89,6 @@ More (server modes, Docker) in [`docs/development.md`](docs/development.md).
88
89
 
89
90
  ## Status
90
91
 
91
- Monk is **pre-1.0** (see `lib/monk/version.rb` and the [changelog](CHANGELOG.md)): the API may still change between minor versions. The core is described in [`docs/history/core-plan.md`](docs/history/core-plan.md). Features to add are listed in [`docs/roadmap.md`](docs/roadmap.md). Done so far, each with its own design doc and plan: persistence, migrations, HTML templating and static assets, auth and sessions, WebSocket with Redis fan-out, log levels, live updates (`Monk::Live`, as of 2026-09-19), deployment support (Dockerfile scaffolding, `docs/guides/deploying.md`), the RubyGems release as `monkrb` (both 2026-09-22), and `Monk::WebSocket::PgFanout`/`monk new --live --postgres` — the same cross-process fan-out over Postgres `LISTEN`/`NOTIFY` instead of Redis, for an app that doesn't want a second piece of infrastructure (`docs/design/live-pg-fanout.md`, 2026-09-24), the built-in mailer `Monk::Mail` (0.17, 2026-09-26, ADR 0012), and background jobs on Postgres, `Monk::Jobs`, with mail sent from jobs (0.18, 2026-09-29, ADRs 0013 and 0014). In 0.19 (2026-10-01): `monk new` lays the app out under `app/`, one directory per role (ADR 0015); a WebSocket fanout listens only in the process that holds the sockets (`#listen!`); `authenticate: :optional` lets visitors' sockets in anonymously, for Live's rules to decide; and the WebSocket server stops cleanly on `TERM` (`docs/design/websocket.md`). In 0.20 (2026-10-05): Monk recovers from a dropped database connection (a Postgres restart, a failover) without a process restart. `checkout` reconnects a dead connection before running its block, never retrying the block itself; both fanouts' listeners reconnect and then make every open page resync; and a route that raises logs why (`docs/history/plan-pg-reconnect.md`). Also new: connection pools for Ractors that shouldn't each hold a connection (`Monk::Persistence::Pg.pool`). The WebSocket server's session checks, Live's subscribe rules and `PgFanout`'s publisher can run through one, so open pages share a few connections instead of holding one each: 1,000 open sockets on 4 connections (`docs/guides/persistence.md`, "Pools"; `docs/history/plan-pg-pool.md`). In 0.21 (2026-10-09): scaffolding by module. `monk new` makes the core app, and `monk add` adds a module (Postgres, Redis, mail, auth, jobs, WebSocket, Live) to any app, new or existing, with its wiring, commented examples, a test, and its sections of `SETUP.md` and an `AGENTS.md` for coding agents; `--json` for agents and scripts. Redis and WebSocket become modules of their own, the WebSocket transport is chosen once per app, and `monk new`'s module flags are gone (ADR 0017, `docs/guides/scaffolding.md`, `docs/history/plan-scaffold.md`). In 0.22 (2026-10-13): recurring jobs on a cron schedule, `Monk::Jobs.recurring` in `config/recurring.rb`, enqueued once per occurrence however many `bin/jobs` run (ADR 0016, `docs/guides/jobs.md`); auth housekeeping, with a login-email rate limit in the database, an index on `sessions (subject)` and a nightly cleanup job (`docs/guides/auth.md`); and an upgrade guide for apps made with an earlier Monk (`docs/guides/upgrading.md`). In 0.22.1 (2026-10-20): the WebSocket server queues a connect burst instead of collapsing under it, refuses a handshake it can't check with a `503`, and takes more connects a second with thousands of sockets open: fewer allocations a connect, GC settings in `bin/websocket_server`, and `WS_PROCESSES` for several server processes on one port (`docs/guides/websocket.md`, `docs/plan-ws-connect-ceiling.md`). In 0.22.2 (2026-10-22): a fix for a handshake that could die when two connections were the first after boot (`docs/plan-digest-preload.md`).
92
+ Monk is **pre-1.0** (see `lib/monk/version.rb` and the [changelog](CHANGELOG.md)): the API may still change between minor versions. The core is described in [`docs/history/core-plan.md`](docs/history/core-plan.md). Features to add are listed in [`docs/roadmap.md`](docs/roadmap.md). Done so far, each with its own design doc and plan: persistence, migrations, HTML templating and static assets, auth and sessions, WebSocket with Redis fan-out, log levels, live updates (`Monk::Live`, as of 2026-09-19), deployment support (Dockerfile scaffolding, `docs/guides/deploying.md`), the RubyGems release as `monkrb` (both 2026-09-22), and `Monk::WebSocket::PgFanout`/`monk new --live --postgres` — the same cross-process fan-out over Postgres `LISTEN`/`NOTIFY` instead of Redis, for an app that doesn't want a second piece of infrastructure (`docs/design/live-pg-fanout.md`, 2026-09-24), the built-in mailer `Monk::Mail` (0.17, 2026-09-26, ADR 0012), and background jobs on Postgres, `Monk::Jobs`, with mail sent from jobs (0.18, 2026-09-29, ADRs 0013 and 0014). In 0.19 (2026-10-01): `monk new` lays the app out under `app/`, one directory per role (ADR 0015); a WebSocket fanout listens only in the process that holds the sockets (`#listen!`); `authenticate: :optional` lets visitors' sockets in anonymously, for Live's rules to decide; and the WebSocket server stops cleanly on `TERM` (`docs/design/websocket.md`). In 0.20 (2026-10-05): Monk recovers from a dropped database connection (a Postgres restart, a failover) without a process restart. `checkout` reconnects a dead connection before running its block, never retrying the block itself; both fanouts' listeners reconnect and then make every open page resync; and a route that raises logs why (`docs/history/plan-pg-reconnect.md`). Also new: connection pools for Ractors that shouldn't each hold a connection (`Monk::Persistence::Pg.pool`). The WebSocket server's session checks, Live's subscribe rules and `PgFanout`'s publisher can run through one, so open pages share a few connections instead of holding one each: 1,000 open sockets on 4 connections (`docs/guides/persistence.md`, "Pools"; `docs/history/plan-pg-pool.md`). In 0.21 (2026-10-09): scaffolding by module. `monk new` makes the core app, and `monk add` adds a module (Postgres, Redis, mail, auth, jobs, WebSocket, Live) to any app, new or existing, with its wiring, commented examples, a test, and its sections of `SETUP.md` and an `AGENTS.md` for coding agents; `--json` for agents and scripts. Redis and WebSocket become modules of their own, the WebSocket transport is chosen once per app, and `monk new`'s module flags are gone (ADR 0017, `docs/guides/scaffolding.md`, `docs/history/plan-scaffold.md`). In 0.22 (2026-10-13): recurring jobs on a cron schedule, `Monk::Jobs.recurring` in `config/recurring.rb`, enqueued once per occurrence however many `bin/jobs` run (ADR 0016, `docs/guides/jobs.md`); auth housekeeping, with a login-email rate limit in the database, an index on `sessions (subject)` and a nightly cleanup job (`docs/guides/auth.md`); and an upgrade guide for apps made with an earlier Monk (`docs/guides/upgrading.md`). In 0.22.1 (2026-10-20): the WebSocket server queues a connect burst instead of collapsing under it, refuses a handshake it can't check with a `503`, and takes more connects a second with thousands of sockets open: fewer allocations a connect, GC settings in `bin/websocket_server`, and `WS_PROCESSES` for several server processes on one port (`docs/guides/websocket.md`, `docs/plan-ws-connect-ceiling.md`). In 0.22.2 (2026-10-22): a fix for a handshake that could die when two connections were the first after boot (`docs/plan-digest-preload.md`). In 0.23 (2026-10-26): file storage, `Monk::Storage`, in any S3-compatible bucket (Hetzner, R2, B2, AWS) through Monk's own SigV4 signer, with local files in development and test that behave like a bucket. Browsers upload straight to the bucket and download through a redirect to a signed link, so no worker Ractor waits on the bytes, and `monk add storage` scaffolds the flow (ADR 0018, `docs/guides/storage.md`, `docs/history/plan-storage.md`). In 0.23.1 (2026-11-10): fewer allocations per request in models and views, about half as many objects for a page of 50 rows through a partial, so less GC on Ruby 4.0, where every GC stops every worker; responses are byte-identical. Route blocks also run as methods of `Monk::Context` instead of through `instance_exec`, the same on 4.0 and what lets Ruby 4.1's per-Ractor GC work.
92
93
 
93
94
  [Made with Love ❤️, Ruby 💎 and AI 🤖](docs/ai_usage_disclaimer.md)
data/lib/monk/base.rb CHANGED
@@ -111,6 +111,8 @@ module Monk
111
111
  end
112
112
  end
113
113
 
114
+ define_block_methods!
115
+
114
116
  Monk.freeze!
115
117
 
116
118
  # Settled once here, not read from ENV per request in
@@ -126,6 +128,7 @@ module Monk
126
128
  Ractor.make_shareable(@static_routes)
127
129
  Ractor.make_shareable(@dynamic_routes)
128
130
  Ractor.make_shareable(error_handlers)
131
+ Ractor.make_shareable(@block_methods)
129
132
  self
130
133
  end
131
134
 
@@ -156,6 +159,11 @@ module Monk
156
159
  asset = Monk::Assets.response(env)
157
160
  return asset if asset
158
161
 
162
+ # The file storage backend's signed links (docs/history/plan-storage.md
163
+ # decision 7), only when the app requires monk/storage.
164
+ stored = Monk::Storage.response(env) if defined?(Monk::Storage)
165
+ return stored if stored
166
+
159
167
  route, path_params = find_route(env["REQUEST_METHOD"], env["PATH_INFO"])
160
168
  return not_found_response(env) unless route
161
169
 
@@ -163,14 +171,14 @@ module Monk
163
171
  context = Context.new(params, env)
164
172
  catch(:monk_halt) do
165
173
  begin
166
- body = context.instance_exec(context, &route[:block])
174
+ body = run_block(context, route[:block])
167
175
  [200, context.headers, [body]]
168
176
  rescue StandardError => e
169
177
  handler = error_handlers.find { |matcher, _| matcher.is_a?(Class) && e.is_a?(matcher) }
170
178
  log_exception(env, e, handled: !handler.nil?)
171
179
  if handler
172
180
  context.status = 500
173
- body = context.instance_exec(context, &handler.last)
181
+ body = run_block(context, handler.last)
174
182
  [500, context.headers, [body]]
175
183
  else
176
184
  [500, { "content-type" => "application/json" }, ['{"error":"Internal Server Error"}']]
@@ -179,6 +187,51 @@ module Monk
179
187
  end
180
188
  end
181
189
 
190
+ # Each route and error-handler block becomes a private method of
191
+ # Context, once, here, rather than running through instance_exec on
192
+ # every request. Same self, same ivars, same return value. On Ruby
193
+ # 4.1 (preview1) an instance_exec in a worker Ractor stops that
194
+ # Ractor's own, cheap GCs: its heap grows to GBs between global,
195
+ # stop-the-world ones, and adding workers makes the app slower. A
196
+ # method call doesn't, and on 4.0 the two cost the same. Blocks are
197
+ # made shareable before this, so the methods run in any Ractor.
198
+ #
199
+ # A method's arity is strict, as a lambda's: a block with no
200
+ # positional parameter is called with none, one that can take the
201
+ # context (|c|, |c = nil|, |*args|) with it. A block that needs two
202
+ # or more (instance_exec passes nil for the rest) keeps
203
+ # instance_exec.
204
+ def define_block_methods!
205
+ @block_methods = {}.compare_by_identity
206
+ blocks = routes.map { |route| route[:block] } + error_handlers.map(&:last)
207
+ blocks.each_with_index do |block, i|
208
+ name = :"__monk_block_#{object_id}_#{i}"
209
+ Context.define_method(name, &block)
210
+ Context.send(:private, name)
211
+
212
+ takes_context = block.parameters.any? { |type, _| %i[req opt rest].include?(type) }
213
+ arity = Context.instance_method(name).arity
214
+ if !takes_context
215
+ @block_methods[block] = [name, false].freeze
216
+ elsif arity == 1 || arity.between?(-2, -1)
217
+ @block_methods[block] = [name, true].freeze
218
+ else
219
+ Context.send(:remove_method, name)
220
+ end
221
+ end
222
+ end
223
+
224
+ def run_block(context, block)
225
+ name, with_context = @block_methods[block]
226
+ if name.nil?
227
+ context.instance_exec(context, &block)
228
+ elsif with_context
229
+ context.__send__(name, context)
230
+ else
231
+ context.__send__(name)
232
+ end
233
+ end
234
+
182
235
  # The request line alone ("GET /x -> 500") says nothing about why.
183
236
  # Unhandled: an error, with the first backtrace lines. Handled by an
184
237
  # `error` block: the app chose the response, so one info line --
@@ -213,7 +266,7 @@ module Monk
213
266
 
214
267
  context = Context.new({}, env, status: 404)
215
268
  catch(:monk_halt) do
216
- body = context.instance_exec(context, &handler.last)
269
+ body = run_block(context, handler.last)
217
270
  [404, context.headers, [body]]
218
271
  end
219
272
  end
data/lib/monk/context.rb CHANGED
@@ -17,6 +17,11 @@ module Monk
17
17
  include Monk::Views::Compiled
18
18
  include Monk::HeadTags
19
19
 
20
+ # Frozen once here: as literals in #render, the key was a new String
21
+ # for every template rendered, partials included.
22
+ CONTENT_TYPE = "content-type".freeze
23
+ HTML_CONTENT_TYPE = "text/html; charset=utf-8".freeze
24
+
20
25
  attr_reader :params, :env, :headers
21
26
  attr_accessor :status
22
27
 
@@ -60,7 +65,7 @@ module Monk
60
65
  # `self` bound to this same Context), and the `locals` hash passed
61
66
  # here, which is what a partial rendered inside a loop wants.
62
67
  def render(name, layout: :default, **locals)
63
- headers["content-type"] ||= "text/html; charset=utf-8"
68
+ headers[CONTENT_TYPE] ||= HTML_CONTENT_TYPE
64
69
  Monk::Views.render(self, name, locals, layout: layout)
65
70
  end
66
71
 
@@ -15,5 +15,6 @@ Monk::Generator.define(:base) do
15
15
 
16
16
  setup_section "base/setup.md"
17
17
  agents_section "base/agents.md"
18
+ set_before_production "MONK_ENV", "production or staging; unset = development (the Dockerfile sets production)"
18
19
  next_step "bin/server", note: "http://localhost:9292"
19
20
  end
@@ -0,0 +1,28 @@
1
+ # Files users send (Monk::Storage): local files in development and test,
2
+ # an S3-compatible bucket in production. No service to start, no migration
3
+ # and no JS: the upload flow comes as commented routes driven with curl,
4
+ # and the guide carries the uploads table and browser code
5
+ # (docs/history/plan-storage.md Phase 8).
6
+ Monk::Generator.define(:storage) do
7
+ summary "File storage (local files in development, an S3 bucket in production)"
8
+ installed_if "config/storage.rb"
9
+
10
+ copy "config/storage.rb", role: :wiring
11
+ copy "app/routes/storage.rb", role: :example
12
+ copy "test/storage_test.rb", role: :test
13
+
14
+ # Unset STORAGE_URL is file://storage/<env> in development and test, so
15
+ # neither .env file needs a line, and tests never touch development's
16
+ # files.
17
+ env :example, { "STORAGE_URL" => "s3://KEY:SECRET@bucket?endpoint=fsn1.your-objectstorage.com&region=fsn1",
18
+ "STORAGE_SECRET" => "only-with-file-urls-a-random-hex-value", }, commented: true
19
+ ignore "/storage/"
20
+
21
+ setup_section "storage/setup.md"
22
+ agents_section "storage/agents.md"
23
+ set_before_production "STORAGE_URL", "an S3 bucket; file:// only on one server with a persistent volume"
24
+ set_before_production "STORAGE_SECRET", "only with file://: signs its links"
25
+ example "storage-upload", todo: "uncomment, and set the types and sizes this app accepts"
26
+ example "storage-confirm", todo: "uncomment, and save a record of each file in the app's own table"
27
+ example "storage-download", todo: "uncomment, then replace with a record lookup and an access check"
28
+ end
@@ -2,6 +2,6 @@ require_relative "generator"
2
2
 
3
3
  # Monk's own modules, one generator each (docs/adr/0017), in dependency
4
4
  # order only for readability: the registry resolves dependencies itself.
5
- %w[base postgres redis mail auth jobs websocket live].each do |name|
5
+ %w[base postgres redis mail auth jobs websocket live storage].each do |name|
6
6
  require_relative "generators/#{name}"
7
7
  end
@@ -19,6 +19,13 @@ module Monk
19
19
  ne: "<>",
20
20
  }.freeze
21
21
 
22
+ # An Array condition's values as one array parameter (#in_clause).
23
+ # The Integer one, for a list of ids, writes each number straight
24
+ # into the literal; the generic one makes a String of each item
25
+ # first. Shareable: worker Ractors encode with them.
26
+ ARRAY_ENCODER = Ractor.make_shareable(PG::TextEncoder::Array.new)
27
+ INTEGER_ARRAY_ENCODER = Ractor.make_shareable(PG::TextEncoder::Array.new(elements_type: PG::TextEncoder::Integer.new))
28
+
22
29
  class << self
23
30
  def create(data)
24
31
  Monk::Persistence::Pg.checkout(db_name) do |conn|
@@ -196,14 +203,17 @@ module Monk
196
203
  "#{ident} #{operator} $#{values.size}"
197
204
  end
198
205
 
206
+ # `= ANY($n)` with the whole list as one Postgres array literal,
207
+ # not `IN ($1, ..., $n)`: a placeholder String and an encoded
208
+ # parameter per item made a 40-id lookup ~160 objects. Postgres
209
+ # types the parameter from the column, as it typed each `$n`,
210
+ # and a nil item matches nothing in either form.
199
211
  def in_clause(ident, list, values)
200
212
  return "1 = 0" if list.empty?
201
213
 
202
- placeholders = list.map do |item|
203
- values << item
204
- "$#{values.size}"
205
- end
206
- "#{ident} IN (#{placeholders.join(", ")})"
214
+ integers = list.all? { |item| item.nil? || item.is_a?(Integer) }
215
+ values << (integers ? INTEGER_ARRAY_ENCODER : ARRAY_ENCODER).encode(list)
216
+ "#{ident} = ANY($#{values.size})"
207
217
  end
208
218
 
209
219
  def order_clause(conn, order)
@@ -219,8 +229,11 @@ module Monk
219
229
  " ORDER BY #{clause}"
220
230
  end
221
231
 
232
+ # Symbol keys straight from pg, not String keys transformed: one
233
+ # Hash per row instead of two.
222
234
  def to_rows(result)
223
- result.map { |row| row.transform_keys(&:to_sym) }
235
+ result.field_name_type = :symbol
236
+ result.to_a
224
237
  end
225
238
 
226
239
  def to_row(result)
@@ -0,0 +1,242 @@
1
+ require "digest"
2
+ # Loaded here, at boot: left to Digest.const_missing it loads on the first
3
+ # put, and a worker Ractor storing another in parallel can catch it
4
+ # half-defined (test/digest_preload_test.rb).
5
+ require "digest/md5"
6
+ require "fileutils"
7
+ require "json"
8
+ require "securerandom"
9
+
10
+ module Monk
11
+ module Storage
12
+ module Backends
13
+ # Local disk, the development and test default (file://storage/<env>)
14
+ # and an explicit choice for one server with a persistent volume.
15
+ # root is absolute, expanded at configure, so a worker Ractor never
16
+ # depends on the current directory. secret signs the links of the
17
+ # /_monk/storage routes (docs/adr/0018-storage-s3-compatible-direct-upload.md).
18
+ #
19
+ # It behaves like a bucket: an object's bytes are in
20
+ # <root>/objects/<key>~ and its metadata (content type, size, MD5) in
21
+ # <root>/meta/<key>~.json. The "~", which no key contains, keeps a
22
+ # key's file apart from the directory of a longer key ("a" and
23
+ # "a/b" both exist in a bucket). Writes go to a temporary file then
24
+ # File.rename, so a reader never sees half an object. An object
25
+ # exists once its metadata does.
26
+ #
27
+ # Inside this block a bare File is this class: the disk is ::File.
28
+ File = Data.define(:root, :secret) do
29
+ # Returns the key. body is a String, or an IO that responds to size,
30
+ # read and rewind (File, Tempfile, StringIO).
31
+ def put(key, body, content_type: Input::DEFAULT_CONTENT_TYPE)
32
+ Input.content_type!(content_type)
33
+ Input.body!(body)
34
+ object = path_for("objects", key, "~")
35
+ meta = path_for("meta", key, "~.json")
36
+
37
+ md5, size = write_atomically(object) { |out| copy_body(body, out) }
38
+ write_atomically(meta) do |out|
39
+ out.write(JSON.generate(content_type: content_type, byte_size: size, etag: md5))
40
+ end
41
+ key
42
+ rescue SystemCallError, IOError => e
43
+ raise RequestError, "file storage put #{key} failed: #{e.class}: #{e.message}"
44
+ end
45
+
46
+ # The bytes, as a frozen binary String. range: is a Range of byte
47
+ # offsets (0..15 for the first 16 bytes); a range past the end gives
48
+ # what's left, possibly "". A missing key raises NotFoundError.
49
+ def get(key, range: nil)
50
+ span = Input.byte_span(range) if range
51
+ path = path_for("objects", key, "~")
52
+ meta = read_meta(key) or raise NotFoundError, "no stored object #{key.inspect}"
53
+ bytes = span ? read_span(path, span, meta[:byte_size]) : ::File.binread(path)
54
+ bytes.freeze
55
+ rescue Errno::ENOENT
56
+ raise NotFoundError, "no stored object #{key.inspect}"
57
+ rescue SystemCallError, IOError => e
58
+ raise RequestError, "file storage get #{key} failed: #{e.class}: #{e.message}"
59
+ end
60
+
61
+ # A Storage::Object, or nil when the key is missing.
62
+ def head(key)
63
+ meta = read_meta(key) or return nil
64
+ Storage::Object.new(key: key, byte_size: meta[:byte_size], content_type: meta[:content_type],
65
+ etag: meta[:etag],)
66
+ rescue SystemCallError, IOError => e
67
+ raise RequestError, "file storage head #{key} failed: #{e.class}: #{e.message}"
68
+ end
69
+
70
+ # Idempotent, like S3: deleting a missing key is not an error.
71
+ def delete(key)
72
+ meta = path_for("meta", key, "~.json")
73
+ object = path_for("objects", key, "~")
74
+ ::FileUtils.rm_f(meta)
75
+ ::FileUtils.rm_f(object)
76
+ nil
77
+ rescue SystemCallError, IOError => e
78
+ raise RequestError, "file storage delete #{key} failed: #{e.class}: #{e.message}"
79
+ end
80
+
81
+ # Returns to. A missing from raises NotFoundError.
82
+ def copy(from, to)
83
+ source = path_for("objects", from, "~")
84
+ target = path_for("objects", to, "~")
85
+ target_meta = path_for("meta", to, "~.json")
86
+ meta = read_meta(from) or raise NotFoundError, "no stored object #{from.inspect}"
87
+ return to if source == target
88
+
89
+ write_atomically(target) { |out| ::File.open(source, "rb") { |input| ::IO.copy_stream(input, out) } }
90
+ write_atomically(target_meta) { |out| out.write(JSON.generate(meta)) }
91
+ to
92
+ rescue Errno::ENOENT
93
+ raise NotFoundError, "no stored object #{from.inspect}"
94
+ rescue SystemCallError, IOError => e
95
+ raise RequestError, "file storage copy #{from} -> #{to} failed: #{e.class}: #{e.message}"
96
+ end
97
+
98
+ # A signed download link, the same for every call within one
99
+ # cache_window, so the browser's cache works: valid until the end
100
+ # of the window plus expires_in (docs/history/plan-storage.md decision 9).
101
+ def url(key, expires_in:, cache_window:, now: Time.now.to_i)
102
+ Key.validate!(key)
103
+ Input.link_times!(expires_in: expires_in, cache_window: cache_window)
104
+ window_start = now - (now % cache_window)
105
+ link({ m: "GET", k: key, e: window_start + cache_window + expires_in })
106
+ end
107
+
108
+ # A signed upload link for a browser, with the length and content
109
+ # type inside it.
110
+ def presigned_put(key, content_type:, byte_size:, expires_in:, now: Time.now.to_i)
111
+ Key.validate!(key)
112
+ Input.content_type!(content_type)
113
+ Input.byte_size!(byte_size)
114
+ Input.link_times!(expires_in: expires_in, cache_window: 1)
115
+ url = link({ m: "PUT", k: key, e: now + expires_in, l: byte_size, t: content_type })
116
+ Upload.new(url: url, headers: { "Content-Type" => content_type, "Content-Length" => byte_size.to_s }.freeze)
117
+ end
118
+
119
+ # The /_monk/storage PUT: stores input (rack.input, read once) as
120
+ # key, if it holds exactly byte_size bytes. Returns the MD5, or nil
121
+ # when the length is wrong, and then stores nothing.
122
+ def receive(key, input, content_type:, byte_size:)
123
+ object = path_for("objects", key, "~")
124
+ meta = path_for("meta", key, "~.json")
125
+
126
+ md5 = write_atomically(object) do |out|
127
+ digest, size = copy_at_most(input, out, byte_size)
128
+ return nil unless size == byte_size
129
+
130
+ digest
131
+ end
132
+ write_atomically(meta) do |out|
133
+ out.write(JSON.generate(content_type: content_type, byte_size: byte_size, etag: md5))
134
+ end
135
+ md5
136
+ rescue SystemCallError, IOError => e
137
+ raise RequestError, "file storage upload #{key} failed: #{e.class}: #{e.message}"
138
+ end
139
+
140
+ # Never print the secret -- a backend ends up in logs and exception
141
+ # messages more easily than anyone intends (same as Mail's SMTP).
142
+ def inspect
143
+ "#<#{self.class.name} file://#{root}>"
144
+ end
145
+ alias_method :to_s, :inspect
146
+
147
+ private
148
+
149
+ # The key is validated first, then the path is checked to stay under
150
+ # root -- the Assets#disk_entry containment check -- and to go
151
+ # through no symbolic link below root, which could point anywhere.
152
+ def path_for(area, key, suffix)
153
+ Key.validate!(key)
154
+ base = ::File.join(root, area)
155
+ full = ::File.expand_path("#{key}#{suffix}", base)
156
+ unless full.start_with?("#{base}/")
157
+ raise InvalidKeyError,
158
+ "storage key #{key.inspect} resolves outside #{base}"
159
+ end
160
+
161
+ refuse_symlinks!(key, base, full)
162
+ full
163
+ end
164
+
165
+ def refuse_symlinks!(key, base, full)
166
+ path = base
167
+ [nil, *full.delete_prefix("#{base}/").split("/")].each do |segment|
168
+ path = ::File.join(path, segment) if segment
169
+ next unless ::File.symlink?(path)
170
+
171
+ raise InvalidKeyError, "storage key #{key.inspect} goes through a symbolic link (#{path})"
172
+ end
173
+ end
174
+
175
+ def read_meta(key)
176
+ JSON.parse(::File.read(path_for("meta", key, "~.json")), symbolize_names: true)
177
+ rescue Errno::ENOENT
178
+ nil
179
+ end
180
+
181
+ def read_span(path, span, size)
182
+ first, last = span
183
+ last = size - 1 if last.nil? || last >= size
184
+ return "".b if first >= size
185
+
186
+ ::File.open(path, "rb") do |file|
187
+ file.seek(first)
188
+ file.read(last - first + 1) || "".b
189
+ end
190
+ end
191
+
192
+ # Yields a binary IO on a temporary file next to path, then renames
193
+ # it into place. Returns the block's result.
194
+ def write_atomically(path, &)
195
+ ::FileUtils.mkdir_p(::File.dirname(path))
196
+ temporary = "#{path}#{SecureRandom.hex(8)}.tmp"
197
+ result = ::File.open(temporary, "wb", &)
198
+ ::File.rename(temporary, path)
199
+ result
200
+ ensure
201
+ ::FileUtils.rm_f(temporary) if temporary && ::File.exist?(temporary)
202
+ end
203
+
204
+ def link(payload)
205
+ "#{ROUTE_PREFIX}#{Token.sign(payload, secret)}"
206
+ end
207
+
208
+ # Reads at most limit + 1 bytes, enough to tell a body that's too
209
+ # long. Returns [md5 hex, bytes read].
210
+ def copy_at_most(input, out, limit)
211
+ md5 = Digest::MD5.new
212
+ size = 0
213
+ while size <= limit && (chunk = input.read([Input::CHUNK_BYTES, limit + 1 - size].min))
214
+ out.write(chunk)
215
+ md5 << chunk
216
+ size += chunk.bytesize
217
+ end
218
+ [md5.hexdigest, size]
219
+ end
220
+
221
+ # Returns [md5 hex, bytes written].
222
+ def copy_body(body, out)
223
+ md5 = Digest::MD5.new
224
+ if body.is_a?(String)
225
+ out.write(body)
226
+ md5 << body
227
+ return [md5.hexdigest, body.bytesize]
228
+ end
229
+
230
+ body.rewind
231
+ size = 0
232
+ while (chunk = body.read(Input::CHUNK_BYTES))
233
+ out.write(chunk)
234
+ md5 << chunk
235
+ size += chunk.bytesize
236
+ end
237
+ [md5.hexdigest, size]
238
+ end
239
+ end
240
+ end
241
+ end
242
+ end