ractor-rails-shim 0.2.1 → 0.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +66 -1
- data/lib/ractor_rails_shim/patches/activerecord.rb +52 -3
- data/lib/ractor_rails_shim/patches/class_attribute.rb +44 -41
- data/lib/ractor_rails_shim/patches/core.rb +16 -4
- data/lib/ractor_rails_shim/patches/kaminari.rb +1 -1
- data/lib/ractor_rails_shim/patches/route_helpers.rb +21 -0
- data/lib/ractor_rails_shim/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f309da49240393f31079209d41e422a4f5fabde1428d344d6300ace7ffc5a94d
|
|
4
|
+
data.tar.gz: d6e330ac46078ff31534d594d5c90f7ae1c739bf765223fda27f03d546350fcb
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 147c5795a7bf89b07dc1ebc19503cc12fd37c556d727a05a2c4c776c4bf3822def6c37e289343345fa6e861ecdf9a2ef331b13f69dee377cde97f5364dcd42d5
|
|
7
|
+
data.tar.gz: 3ef1bdce56de6ee989797f780c451f4a56d4ebd6427979a120cc25e90a16d3779e97fe83e2ef9fa48f2e011a7de21bd2df81e2731e77914c2b6d01dd7743460d
|
data/CHANGELOG.md
CHANGED
|
@@ -11,6 +11,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
11
11
|
- Improved the published gem summary and description (gemspec). Metadata-only
|
|
12
12
|
release — no code changes.
|
|
13
13
|
|
|
14
|
+
## [0.2.4]
|
|
15
|
+
|
|
16
|
+
### Performance
|
|
17
|
+
- **class_attribute reader (ractor mode) is now allocation-free.** The
|
|
18
|
+
generated `__class_attr_*` accessors previously walked `self.ancestors` and
|
|
19
|
+
built a fresh `Symbol` via string interpolation for *every* ancestor on
|
|
20
|
+
*every* read — the dominant allocation source for GET requests (a Rails class
|
|
21
|
+
has 20–40 ancestors, and class_attribute is read constantly: controller
|
|
22
|
+
filters, view partial paths, AR `strict_loading`, form builder, logger, …).
|
|
23
|
+
In ractor mode the writer already collapses every write to the defining
|
|
24
|
+
owner's single key, so the ancestor walk was dead code. The reader now does a
|
|
25
|
+
single literal-symbol lookup against `IsolatedExecutionState[key]` then
|
|
26
|
+
`SHAREABLE_FALLBACK[key]`, eliminating the per-read `Array` + `Symbol`
|
|
27
|
+
churn. This cuts request allocations substantially and, with them, the
|
|
28
|
+
garbage-collection share of CPU time (was ~33% of CPU on `GET /posts`).
|
|
29
|
+
|
|
30
|
+
## [0.2.3]
|
|
31
|
+
|
|
32
|
+
### Fixed
|
|
33
|
+
- **Cold `GET /up` `SystemStackError` in worker Ractors (kino `:ractor`).**
|
|
34
|
+
Rails' `ActionDispatch::Routing::RouteSet#generate_url_helpers` builds a
|
|
35
|
+
module whose `self.included(base)` hook re-dups the module and re-includes
|
|
36
|
+
it while `!base._routes.equal?(@_proxy._routes)`. Under the frozen,
|
|
37
|
+
Ractor-shareable app graph a worker Ractor's controller reports
|
|
38
|
+
`base._routes` as `nil`, so the equality never holds and the hook
|
|
39
|
+
re-includes forever (empty Ruby backtrace, first request only; respawns and
|
|
40
|
+
later requests are fine). The shim's `route_helpers.rb` `generate_url_helpers`
|
|
41
|
+
override now bounds the reinclude to once per base, preserving the
|
|
42
|
+
route-alignment intent without the infinite loop. Verified: cold `GET /up`
|
|
43
|
+
returns 200 in `kino -m ractor`; both `:ractor` and `:threaded` modes clean.
|
|
44
|
+
|
|
45
|
+
## [0.2.2]
|
|
46
|
+
|
|
47
|
+
### Fixed
|
|
48
|
+
- **Multi-threaded worker race in `WorkerApp#setup_once!` (kino `:ractor`,
|
|
49
|
+
`-wN -tM` with M > 1).** All threads inside a worker Ractor share
|
|
50
|
+
`Ractor.current`, so the previous `Ractor.current[:rrs_worker_ready]` guard
|
|
51
|
+
let multiple threads race through `rebind_constants` +
|
|
52
|
+
`init_worker_ar_connections!`, producing
|
|
53
|
+
`ActiveRecord::ConnectionNotEstablished` on the very first request. Setup is
|
|
54
|
+
now serialized with a per-Ractor `Thread::Mutex`, and `rebind_constants`
|
|
55
|
+
re-fetches each namespace parent so concurrent setup cannot clobber a module
|
|
56
|
+
out from under another thread.
|
|
57
|
+
- **Per-thread ActiveRecord connection handler (intermittent
|
|
58
|
+
`ConnectionNotEstablished` on `kino :ractor -w5 -t5`).** `init_worker_ar_connections!`
|
|
59
|
+
stored the per-Ractor `ConnectionHandler` in `ActiveSupport::IsolatedExecutionState`,
|
|
60
|
+
which is **per-thread** (`Thread.attr_accessor`). The init thread set it, but
|
|
61
|
+
the other worker threads in the same Ractor saw `nil` →
|
|
62
|
+
`ConnectionNotEstablished: No connection handler for Ractor X` on a fraction of
|
|
63
|
+
requests. The handler now lives in `Ractor.current` (per-Ractor, shared by all
|
|
64
|
+
of the worker's threads), and `ActiveRecord::Base.connection_handler` reads it
|
|
65
|
+
there first.
|
|
66
|
+
- **Write-path `Ractor::IsolationError` on `redirect_to @post`
|
|
67
|
+
(`kino :ractor -w5 -t5`, `POST /posts`).** `ActiveModel::AttributeMethods::
|
|
68
|
+
ClassMethods#attribute_method_patterns_cache` stored a mutable `Concurrent::Map`
|
|
69
|
+
in a class instance variable (unshareable). Hit via `redirect_to @post` →
|
|
70
|
+
`respond_to?` → `matched_attribute_method`, raising from worker Ractors. The
|
|
71
|
+
cache is now routed through `Ractor.current` (per-Ractor, keyed per class) and
|
|
72
|
+
populated lazily per Ractor — content is deterministic, so per-Ractor
|
|
73
|
+
recomputation is correct.
|
|
74
|
+
|
|
75
|
+
With these three fixes, `kino :ractor (-w5 -t5)` serves `/up`, `GET /posts`,
|
|
76
|
+
and `POST /posts` (authenticated write + 302) with 0 transport failures and 0
|
|
77
|
+
server errors under sustained load.
|
|
78
|
+
|
|
14
79
|
### Added — ActiveRecord query-path ractor-safety (Blocker 1 deep work)
|
|
15
80
|
- `RactorRailsShim.worker_ar_init(app)` — a shareable Rack middleware that
|
|
16
81
|
calls `init_worker_ar_connections!` on each worker's first request. Kino's
|
|
@@ -47,7 +112,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
47
112
|
`TYPE_MAP` (Procs) and `QUOTED_*` `Concurrent::Map` quoting caches.
|
|
48
113
|
These are intrinsically unshareable and require upstream Rails changes
|
|
49
114
|
(shareable callables instead of Procs; per-Ractor quoting caches). See
|
|
50
|
-
|
|
115
|
+
the "deep AR ractor-unsafety" wall in the project notes.
|
|
51
116
|
|
|
52
117
|
## [0.2.0] - 2026-07-09
|
|
53
118
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
# Patch ActiveRecord's connection handler to work in per-Ractor mode.
|
|
4
4
|
#
|
|
5
|
-
# Blocker 1
|
|
5
|
+
# Blocker 1:
|
|
6
6
|
# ActiveRecord::Base.connection_pool calls
|
|
7
7
|
# default_connection_handler.retrieve_connection_pool(...)
|
|
8
8
|
# at connection_handling.rb:346. The default_connection_handler
|
|
@@ -229,7 +229,14 @@ module RactorRailsShim
|
|
|
229
229
|
return unless defined?(::ActiveRecord::Base)
|
|
230
230
|
|
|
231
231
|
key = :active_record_connection_handler
|
|
232
|
-
|
|
232
|
+
# Store the handler in Ractor-local storage (Ractor.current), NOT in
|
|
233
|
+
# ActiveSupport::IsolatedExecutionState. IES is per-THREAD (it is a
|
|
234
|
+
# Thread.attr_accessor), so a value set on the init thread is invisible
|
|
235
|
+
# to the other worker threads in the same worker Ractor ->
|
|
236
|
+
# ConnectionNotEstablished ("No connection handler for Ractor X").
|
|
237
|
+
# Ractor.current storage is per-Ractor and shared by every thread of the
|
|
238
|
+
# worker, so connection_handler resolves the same handler for all threads.
|
|
239
|
+
existing = Ractor.current[key]
|
|
233
240
|
return if existing
|
|
234
241
|
|
|
235
242
|
# Establish a fresh, per-Ractor connection handler + pool from the
|
|
@@ -256,7 +263,7 @@ module RactorRailsShim
|
|
|
256
263
|
end
|
|
257
264
|
end
|
|
258
265
|
|
|
259
|
-
|
|
266
|
+
Ractor.current[key] = handler
|
|
260
267
|
end
|
|
261
268
|
end
|
|
262
269
|
|
|
@@ -320,6 +327,35 @@ module RactorRailsShim
|
|
|
320
327
|
_share_relation_delegate_caches! if Ractor.main?
|
|
321
328
|
end
|
|
322
329
|
|
|
330
|
+
# ActiveModel::AttributeMethods::ClassMethods#attribute_method_patterns_cache
|
|
331
|
+
# stores a mutable Concurrent::Map in a CLASS instance variable
|
|
332
|
+
# (@attribute_method_patterns_cache). That ivar is unshareable, so reading
|
|
333
|
+
# it from a worker Ractor raises Ractor::IsolationError ("can not get
|
|
334
|
+
# unshareable values from instance variables of classes/modules from
|
|
335
|
+
# non-main Ractors") — hit on the write path via redirect_to @post ->
|
|
336
|
+
# respond_to? -> matched_attribute_method -> attribute_method_patterns_cache.
|
|
337
|
+
#
|
|
338
|
+
# Unlike @relation_delegate_cache (populated once, freezable), this map is
|
|
339
|
+
# mutated lazily per method_name (compute_if_absent) during request handling,
|
|
340
|
+
# so it cannot be frozen. Instead route it through Ractor-local storage:
|
|
341
|
+
# each Ractor gets its own Concurrent::Map, shared by all of its threads.
|
|
342
|
+
# The cache content is deterministic (a pure function of the class's
|
|
343
|
+
# attribute_method_patterns), so per-Ractor recomputation is correct.
|
|
344
|
+
def _install_active_model_attribute_method_patterns_patch
|
|
345
|
+
return if @am_amp_patched
|
|
346
|
+
@am_amp_patched = true
|
|
347
|
+
_register_patch :active_model_attribute_method_patterns, "8.1"
|
|
348
|
+
return unless defined?(::ActiveModel::AttributeMethods)
|
|
349
|
+
|
|
350
|
+
mod = ::ActiveModel::AttributeMethods::ClassMethods
|
|
351
|
+
mod.module_eval do
|
|
352
|
+
def attribute_method_patterns_cache
|
|
353
|
+
store = Ractor.current[:__am_attribute_method_patterns_cache__] ||= {}
|
|
354
|
+
store[object_id] ||= Concurrent::Map.new(initial_capacity: 4)
|
|
355
|
+
end
|
|
356
|
+
end
|
|
357
|
+
end
|
|
358
|
+
|
|
323
359
|
# Make every loaded AR model class's @relation_delegate_cache shareable.
|
|
324
360
|
# Idempotent; must run in the main Ractor after eager_load so that all
|
|
325
361
|
# model classes (and their caches) exist.
|
|
@@ -1310,6 +1346,19 @@ module RactorRailsShim
|
|
|
1310
1346
|
def default_connection_handler=(val)
|
|
1311
1347
|
ActiveSupport::IsolatedExecutionState[#{dch_key_str}] = val
|
|
1312
1348
|
end
|
|
1349
|
+
# Route the per-Ractor handler through Ractor-local storage. IES is
|
|
1350
|
+
# per-thread, so a handler stored on the init thread is invisible to the
|
|
1351
|
+
# worker's other threads; Ractor.current is per-Ractor and shared by all
|
|
1352
|
+
# threads of the worker. Falls back to IES (legacy) then
|
|
1353
|
+
# default_connection_handler (main Ractor only).
|
|
1354
|
+
def connection_handler
|
|
1355
|
+
v = Ractor.current[:active_record_connection_handler]
|
|
1356
|
+
return v unless v.nil?
|
|
1357
|
+
ActiveSupport::IsolatedExecutionState[:active_record_connection_handler] || default_connection_handler
|
|
1358
|
+
end
|
|
1359
|
+
def connection_handler=(handler)
|
|
1360
|
+
Ractor.current[:active_record_connection_handler] = handler
|
|
1361
|
+
end
|
|
1313
1362
|
RUBY
|
|
1314
1363
|
|
|
1315
1364
|
# Also ensure connection_handler (which Rails already routes through IES
|
|
@@ -135,50 +135,53 @@ module RactorRailsShim
|
|
|
135
135
|
end
|
|
136
136
|
RUBY
|
|
137
137
|
end
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
138
|
+
else
|
|
139
|
+
# Ractor mode: the writer (below) ALWAYS targets this owner's single
|
|
140
|
+
# `key` (#{key_str}), so there is no per-subclass copy-on-write to
|
|
141
|
+
# resolve by walking ancestors. The original `self.ancestors.each`
|
|
142
|
+
# + per-ancestor Symbol interpolation allocated a fresh Array AND a
|
|
143
|
+
# Symbol per ancestor on EVERY read — the dominant allocation source
|
|
144
|
+
# for GET requests. Replace it with a direct literal-key lookup
|
|
145
|
+
# (zero per-read allocation). The per-Ractor value already lives in
|
|
146
|
+
# ActiveSupport::IsolatedExecutionState[key]; SHAREABLE_FALLBACK[key]
|
|
147
|
+
# is the frozen, shared default built at prepare_for_ractors! time.
|
|
148
|
+
target.module_eval <<-RUBY, __FILE__, __LINE__ + 1
|
|
149
|
+
def #{namespaced_name}
|
|
150
|
+
v = ActiveSupport::IsolatedExecutionState[#{key_str}]
|
|
151
|
+
return v unless v.nil?
|
|
152
|
+
fb = RactorRailsShim::SHAREABLE_FALLBACK[#{key_str}]
|
|
153
|
+
return fb unless fb.nil?
|
|
154
|
+
RactorRailsShim::CLASS_ATTR_VALUES[#{key_str}] if Ractor.main?
|
|
155
|
+
end
|
|
150
156
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
+
def #{namespaced_name}=(new_value)
|
|
158
|
+
ActiveSupport::IsolatedExecutionState[#{key_str}] = new_value
|
|
159
|
+
RactorRailsShim::CLASS_ATTR_VALUES[#{key_str}] = new_value if Ractor.main?
|
|
160
|
+
new_value
|
|
161
|
+
end
|
|
162
|
+
RUBY
|
|
157
163
|
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
end
|
|
171
|
-
RactorRailsShim::CLASS_ATTR_VALUES[#{key_str}] if Ractor.main?
|
|
172
|
-
end
|
|
164
|
+
# When owner is a module's singleton class, the original also
|
|
165
|
+
# defines a public reader `def #{name} { value }` on owner directly
|
|
166
|
+
# (block-based). Override it with the IES-routed version + fallback.
|
|
167
|
+
if owner.singleton_class? && owner.attached_object.is_a?(Module)
|
|
168
|
+
owner.module_eval <<-RUBY, __FILE__, __LINE__ + 1
|
|
169
|
+
def #{name}
|
|
170
|
+
v = ActiveSupport::IsolatedExecutionState[#{key_str}]
|
|
171
|
+
return v unless v.nil?
|
|
172
|
+
fb = RactorRailsShim::SHAREABLE_FALLBACK[#{key_str}]
|
|
173
|
+
return fb unless fb.nil?
|
|
174
|
+
RactorRailsShim::CLASS_ATTR_VALUES[#{key_str}] if Ractor.main?
|
|
175
|
+
end
|
|
173
176
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
177
|
+
def #{name}=(new_value)
|
|
178
|
+
ActiveSupport::IsolatedExecutionState[#{key_str}] = new_value
|
|
179
|
+
RactorRailsShim::CLASS_ATTR_VALUES[#{key_str}] = new_value if Ractor.main?
|
|
180
|
+
new_value
|
|
181
|
+
end
|
|
182
|
+
RUBY
|
|
183
|
+
end
|
|
184
|
+
end
|
|
182
185
|
end
|
|
183
186
|
|
|
184
187
|
# redefine_method is used by `redefine` internally and by other call
|
|
@@ -377,6 +377,7 @@ module RactorRailsShim
|
|
|
377
377
|
_install_activerecord_primary_key_patch
|
|
378
378
|
_install_activerecord_query_constraints_patch
|
|
379
379
|
_install_activerecord_relation_delegate_cache_patch
|
|
380
|
+
_install_active_model_attribute_method_patterns_patch
|
|
380
381
|
_install_activerecord_model_classes_patch
|
|
381
382
|
_install_active_model_naming_patch
|
|
382
383
|
_install_active_record_core_patch
|
|
@@ -781,10 +782,18 @@ module RactorRailsShim
|
|
|
781
782
|
private
|
|
782
783
|
|
|
783
784
|
def setup_once!
|
|
784
|
-
|
|
785
|
-
|
|
786
|
-
|
|
787
|
-
|
|
785
|
+
# All threads inside a worker Ractor share Ractor.current, so a single
|
|
786
|
+
# per-Ractor mutex serializes the one-time init across the worker's
|
|
787
|
+
# threads. (The mutex is created under ||= which is racy under extreme
|
|
788
|
+
# contention, but both init steps below are themselves idempotent, so a
|
|
789
|
+
# rare double-init is harmless.)
|
|
790
|
+
m = Ractor.current[:rrs_worker_mutex] ||= Thread::Mutex.new
|
|
791
|
+
m.synchronize do
|
|
792
|
+
return if Ractor.current[:rrs_worker_ready]
|
|
793
|
+
rebind_constants
|
|
794
|
+
RactorRailsShim.init_worker_ar_connections! if defined?(RactorRailsShim)
|
|
795
|
+
Ractor.current[:rrs_worker_ready] = true
|
|
796
|
+
end
|
|
788
797
|
end
|
|
789
798
|
|
|
790
799
|
def rebind_constants
|
|
@@ -792,6 +801,9 @@ module RactorRailsShim
|
|
|
792
801
|
parent = Object
|
|
793
802
|
parts = cpath.split("::")
|
|
794
803
|
parts[0...-1].each do |p|
|
|
804
|
+
# Re-fetch the parent each iteration so concurrent setup (multiple
|
|
805
|
+
# threads racing through setup_once! on the same worker) cannot
|
|
806
|
+
# clobber a namespace module out from under us.
|
|
795
807
|
parent = if parent.const_defined?(p, false)
|
|
796
808
|
parent.const_get(p, false)
|
|
797
809
|
else
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
# Patch Kaminari::config to not read the @_config class ivar from a worker
|
|
4
4
|
# Ractor.
|
|
5
5
|
#
|
|
6
|
-
# Blocker 2
|
|
6
|
+
# Blocker 2:
|
|
7
7
|
# Kaminari.config reads @_config directly at kaminari/config.rb:14:
|
|
8
8
|
# def self.config; @_config ||= Kaminari::Configuration.new end
|
|
9
9
|
# Not mattr_accessor-backed, so the shim's mattr rewrite doesn't catch it.
|
|
@@ -93,6 +93,27 @@ module RactorRailsShim
|
|
|
93
93
|
mod = _rrs_orig_generate_url_helpers(supports_path)
|
|
94
94
|
mod.module_eval("def _routes\n @_routes || ::Rails.application.routes\nend", __FILE__, __LINE__ + 1)
|
|
95
95
|
mod.module_eval("def _generate_paths_by_default\n " + supports_path.inspect + "\nend", __FILE__, __LINE__ + 1)
|
|
96
|
+
|
|
97
|
+
# Bound the stock `self.included` reinclude (actionpack
|
|
98
|
+
# action_dispatch/routing/route_set.rb). That hook re-dups the module
|
|
99
|
+
# and re-includes it while `!base._routes.equal?(@_proxy._routes)`.
|
|
100
|
+
# Under the frozen, Ractor-shareable app graph a worker Ractor's
|
|
101
|
+
# controller can report `_routes` as nil, so the equality never holds
|
|
102
|
+
# and the reinclude loops forever (SystemStackError on the very first
|
|
103
|
+
# request, e.g. GET /up -> Rails::HealthController). The intent is to
|
|
104
|
+
# re-align `_routes` per base exactly once; bound it to one reinclude
|
|
105
|
+
# per base so the loop can never happen while preserving that intent.
|
|
106
|
+
seen = ::Set.new
|
|
107
|
+
mod.instance_variable_set(:@_rrs_reinclude_seen, seen)
|
|
108
|
+
mod.singleton_class.class_eval do
|
|
109
|
+
alias_method :_rrs_orig_self_included, :included
|
|
110
|
+
def included(base)
|
|
111
|
+
seen = instance_variable_get(:@_rrs_reinclude_seen)
|
|
112
|
+
return if seen.include?(base.object_id)
|
|
113
|
+
seen << base.object_id
|
|
114
|
+
_rrs_orig_self_included(base)
|
|
115
|
+
end
|
|
116
|
+
end
|
|
96
117
|
mod
|
|
97
118
|
end
|
|
98
119
|
RUBY
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ractor-rails-shim
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.4
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- dev
|
|
@@ -106,7 +106,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
106
106
|
- !ruby/object:Gem::Version
|
|
107
107
|
version: '0'
|
|
108
108
|
requirements: []
|
|
109
|
-
rubygems_version: 4.0.
|
|
109
|
+
rubygems_version: 4.0.16
|
|
110
110
|
specification_version: 4
|
|
111
111
|
summary: Make Rails apps Ractor-safe so they can run in Ractor mode (e.g. under the
|
|
112
112
|
kino web server).
|