hotcell-server 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/lib/hot_cell/server/version.rb +1 -1
- data/lib/hot_cell/slot.rb +87 -22
- data/lib/hot_cell/supervisor.rb +34 -34
- data/lib/hot_cell/test_operations.rb +40 -0
- data/lib/hot_cell/worker.rb +69 -13
- metadata +8 -12
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 50d90e8b9f7ecb2a139d9c678a3cfdb228288d590f33cdf19390b6cbd92a924e
|
|
4
|
+
data.tar.gz: 66c3bcb911c378507bc9623cb1badc8c3372b6a58c3cd0312eae81c72b003877
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c489f02a8b5670abda9df8c52f62fc4657a725abf01c6e49a39ad1065671535f96942e9ed5168fed5062d9b1ce3f55cef5aba0c2dfab2ec2f1c9c35b14980490
|
|
7
|
+
data.tar.gz: 5b633520243d9f70ab9adfaa1f2427516012ca477e3acd4d5917cbfe5a02ca01dc830e6d6855faeda5e11e8d3b8978634cadfa39712938aeeaeb6560e9f8032c
|
data/lib/hot_cell/slot.rb
CHANGED
|
@@ -11,8 +11,7 @@ module HotCell
|
|
|
11
11
|
#
|
|
12
12
|
# **A slot has one directory per request and it is that request's `$HOME`.** It is created when the
|
|
13
13
|
# request starts and removed when the request ends, so nothing a tool writes under `$HOME` reaches the
|
|
14
|
-
# next request on this slot.
|
|
15
|
-
# reused worker reports the same path twice.
|
|
14
|
+
# next request on this slot.
|
|
16
15
|
#
|
|
17
16
|
# This directory used to survive, to give a tool with an expensive per-user profile a warm one. That is
|
|
18
17
|
# withdrawn. What a tool reads from `$HOME` is configuration, and for the toolchains a cell carries
|
|
@@ -22,23 +21,48 @@ module HotCell
|
|
|
22
21
|
# `max_requests_per_worker: 1` is supposed to hold. adr/0003 records the reversal and adr/0002 the
|
|
23
22
|
# reasoning it supersedes.
|
|
24
23
|
#
|
|
25
|
-
#
|
|
26
|
-
#
|
|
27
|
-
#
|
|
28
|
-
#
|
|
29
|
-
#
|
|
24
|
+
# **The name is fresh for every request, and that is what makes the removal above a guarantee rather than
|
|
25
|
+
# an intention.** A stable name only holds while the deletion behind it works, and the tool that filled
|
|
26
|
+
# the directory runs as the user that owns it: `chmod 0500` on a directory it has written makes its own
|
|
27
|
+
# configuration unremovable, and the same mode on the slot directory makes it unrenameable too. Both
|
|
28
|
+
# cleanups answer false and both callers log — and a stable name then handed the next request the tree
|
|
29
|
+
# that had just refused to go. A name no earlier request has held is not a name an earlier request could
|
|
30
|
+
# have prepared, so what a failed cleanup now costs is disk rather than the isolation the cell is for.
|
|
31
|
+
#
|
|
32
|
+
# The mode of the slot directory is reasserted for the same reason. It is the one name here that is
|
|
33
|
+
# predictable, so it is the one an earlier request can lock, and a mode on a directory this uid owns is
|
|
34
|
+
# ours to put back.
|
|
35
|
+
#
|
|
36
|
+
# **This bounds what an earlier request left behind, and not what a live one is doing.** Workers share a
|
|
37
|
+
# uid and `0700` is the owner's own mode, so a concurrent sibling — or a `setsid` descendant of a finished
|
|
38
|
+
# request, which process groups do not contain — can still write into a home the moment it exists. The
|
|
39
|
+
# slot directory itself can be renamed aside and replaced, and `chmod` follows what it finds. Both are the
|
|
40
|
+
# residuals `docs/DESIGN.md` records under worker isolation, and neither is closed here.
|
|
41
|
+
#
|
|
42
|
+
# There is one directory per request and not two. A request's staged inputs and outputs are named inside
|
|
43
|
+
# `$HOME` rather than in a scratch directory of their own, because the two had the same lifetime and the
|
|
44
|
+
# same owner once the home stopped surviving. Staging used to create its directory on demand, which is
|
|
45
|
+
# what kept a descriptor-only operation from paying for one; `$HOME` has to exist for every request
|
|
46
|
+
# either way, so that laziness bought nothing and is gone with it.
|
|
30
47
|
#
|
|
31
48
|
# The filesystem behaviour belongs here rather than in the two processes that call it. The directory is
|
|
32
49
|
# removed from both — the worker before it answers, the supervisor at finish and at reap — so the guard
|
|
33
50
|
# and the swallowed SystemCallError are a rule that has to hold on both sides of a fork, and it had a
|
|
34
51
|
# copy on each.
|
|
35
|
-
Slot = Struct.new(:number, :home) do
|
|
52
|
+
Slot = Struct.new(:number, :directory, :home) do
|
|
36
53
|
def self.build(workspace, number)
|
|
37
|
-
new number, File.join(workspace, number.to_s
|
|
54
|
+
new number, File.join(workspace, number.to_s)
|
|
38
55
|
end
|
|
39
56
|
|
|
57
|
+
# A name no request has held before, so nothing an earlier one did to the tree it was given reaches this
|
|
58
|
+
# one. The suffix is random rather than a counter for the reason the discarded name's is: the previous
|
|
59
|
+
# request could write to this directory, and a predictable name is one it can pre-create.
|
|
40
60
|
def make_home
|
|
41
|
-
FileUtils.mkdir_p
|
|
61
|
+
FileUtils.mkdir_p directory, mode: 0o700
|
|
62
|
+
FileUtils.chmod 0o700, directory
|
|
63
|
+
|
|
64
|
+
self.home = File.join(directory, "home-#{SecureRandom.hex(8)}")
|
|
65
|
+
Dir.mkdir home, 0o700
|
|
42
66
|
home
|
|
43
67
|
end
|
|
44
68
|
|
|
@@ -55,10 +79,11 @@ module HotCell
|
|
|
55
79
|
# response with a crash, and the supervisor calls discard_home from finish and reap, where nothing above
|
|
56
80
|
# rescues anything and a raise stops the cell with every request it holds.
|
|
57
81
|
def remove_home
|
|
58
|
-
|
|
82
|
+
return true if home.nil?
|
|
83
|
+
return false unless remove_tree(home)
|
|
84
|
+
|
|
85
|
+
self.home = nil
|
|
59
86
|
true
|
|
60
|
-
rescue SystemCallError
|
|
61
|
-
false
|
|
62
87
|
end
|
|
63
88
|
|
|
64
89
|
# **The supervisor renames rather than deletes, and that is a scheduling decision.**
|
|
@@ -73,34 +98,74 @@ module HotCell
|
|
|
73
98
|
# where the unlinking costs nobody's latency.
|
|
74
99
|
#
|
|
75
100
|
# The destination carries a random suffix rather than a counter, because the tool that filled the
|
|
76
|
-
# directory runs as this user and can write to the slot's
|
|
101
|
+
# directory runs as this user and can write to the slot's directory. A predictable name lets it
|
|
77
102
|
# pre-create a colliding entry, fail the rename, and send the supervisor into the recursive delete the
|
|
78
103
|
# rename exists to avoid. On any rename failure the tree is left where it is, for a later worker's own
|
|
79
104
|
# cleanup to remove off the hot path. The supervisor never deletes a tree inline, whatever goes wrong.
|
|
105
|
+
# It renames what is there rather than the name it is holding, because the supervisor is the caller that
|
|
106
|
+
# matters and it does not know the name. `home` is assigned in the worker after the fork, so the
|
|
107
|
+
# supervisor's copy of the slot is still nil when it discards a killed worker's tree. At most one worker
|
|
108
|
+
# holds a slot at a time, so everything matching `home-*` under it is that worker's and nobody else's.
|
|
80
109
|
def discard_home
|
|
81
|
-
|
|
110
|
+
FileUtils.chmod 0o700, directory if Dir.exist?(directory)
|
|
82
111
|
|
|
83
|
-
File.
|
|
112
|
+
Dir.glob(File.join(directory, "home-*")).each do |path|
|
|
113
|
+
File.rename path, File.join(directory, "discarded-#{Process.pid}-#{SecureRandom.hex(8)}")
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
self.home = nil
|
|
84
117
|
true
|
|
85
118
|
rescue SystemCallError
|
|
86
119
|
false
|
|
87
120
|
end
|
|
88
121
|
|
|
89
122
|
# Nothing here is created at boot, because nothing survives a request. This only clears what an earlier
|
|
90
|
-
# boot left behind
|
|
123
|
+
# boot left behind — the whole slot directory, since the names inside it are an earlier boot's and not
|
|
124
|
+
# this one's to reconstruct.
|
|
125
|
+
#
|
|
126
|
+
# `Dir.exist?` is not the guard, because it follows symlinks and answers false for a dangling one. An
|
|
127
|
+
# entry a tool left in the slot's place is exactly what this has to remove, and it used to survive the
|
|
128
|
+
# sweep and raise from every later `make_home`.
|
|
91
129
|
def prepare
|
|
92
|
-
|
|
93
|
-
cleared = remove_home
|
|
94
|
-
sweep && cleared
|
|
130
|
+
remove_tree directory
|
|
95
131
|
end
|
|
96
132
|
|
|
97
133
|
# Unlinks whatever discard_home renamed out of the way. Partial progress is fine: a sweep killed
|
|
98
134
|
# part-way leaves fewer entries for the next one, so this converges rather than repeating.
|
|
99
135
|
def sweep
|
|
100
|
-
Dir.glob("
|
|
101
|
-
true
|
|
136
|
+
Dir.glob(File.join(directory, "discarded-*")).map { |path| remove_tree(path) }.all?
|
|
102
137
|
rescue SystemCallError
|
|
138
|
+
# The glob itself can fail, because the slot directory is a name a tool can replace — a symlink loop
|
|
139
|
+
# in its place answers ELOOP here rather than for any one entry. This runs from the worker's ensure,
|
|
140
|
+
# where a raise would replace the caller's response with a crash.
|
|
103
141
|
false
|
|
104
142
|
end
|
|
143
|
+
|
|
144
|
+
private
|
|
145
|
+
# **A mode is the only thing a tool needs to make its own tree unremovable, and a mode on a tree this
|
|
146
|
+
# uid owns is ours to put back.** `chmod 0500` on a directory a conversion wrote is enough to fail the
|
|
147
|
+
# recursive delete underneath it, and the delete failing used to be the end of it: one tree per
|
|
148
|
+
# request stayed on the tmpfs until the container ended. The repair is not a permission the process
|
|
149
|
+
# gains, it is one it never lost.
|
|
150
|
+
#
|
|
151
|
+
# Repair only after a failure, never before, so the common request pays for a walk of its own tree
|
|
152
|
+
# once rather than twice. `force:` on the chmod because a partial repair that removes most of the tree
|
|
153
|
+
# is better than none, and the remove that follows is what reports the outcome either way.
|
|
154
|
+
def remove_tree(path)
|
|
155
|
+
return true unless File.exist?(path) || File.symlink?(path)
|
|
156
|
+
|
|
157
|
+
FileUtils.remove_entry path
|
|
158
|
+
true
|
|
159
|
+
rescue SystemCallError
|
|
160
|
+
repair_and_remove path
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
def repair_and_remove(path)
|
|
164
|
+
FileUtils.chmod_R 0o700, path, force: true
|
|
165
|
+
FileUtils.remove_entry path
|
|
166
|
+
true
|
|
167
|
+
rescue SystemCallError
|
|
168
|
+
false
|
|
169
|
+
end
|
|
105
170
|
end
|
|
106
171
|
end
|
data/lib/hot_cell/supervisor.rb
CHANGED
|
@@ -90,31 +90,6 @@ module HotCell
|
|
|
90
90
|
# bytes than Linux, and control.sock is the longer of the two names, so it overflows first.
|
|
91
91
|
SUN_PATH_MAX = RUBY_PLATFORM.include?("darwin") ? 104 : 108
|
|
92
92
|
|
|
93
|
-
# The signals this cell can attribute to the request the worker was holding. XFSZ is that worker passing
|
|
94
|
-
# RLIMIT_FSIZE, and SEGV, ABRT and TRAP are how libvips and GLib die on their own allocation failures —
|
|
95
|
-
# libvips dereferences null after printing the correct diagnostic, and g_malloc aborts.
|
|
96
|
-
#
|
|
97
|
-
# These three are the worker hitting its own per-worker RLIMIT_DATA, which is a property of the input
|
|
98
|
-
# this worker held, so the same bytes do it again and the verdict is permanent. `Codes` says a signal
|
|
99
|
-
# tells how a process died and never why, so a signal is transient by default — and that is not in
|
|
100
|
-
# conflict with a permanent verdict here, because aggregate pressure the worker did not cause arrives as
|
|
101
|
-
# SIGKILL, not as these. The two are different signals, and SIGKILL is excluded below.
|
|
102
|
-
#
|
|
103
|
-
# SIGKILL is deliberately absent, and its absence is the point. The supervisor's own deadline kill is
|
|
104
|
-
# already named by `killed_for`, so a SIGKILL reaching this table came from somewhere this process
|
|
105
|
-
# cannot see: a cgroup OOM chosen on aggregate pressure, or a sibling worker, which shares a uid and is
|
|
106
|
-
# not prevented from signalling. Reading it as this request's memory condemned an input for someone
|
|
107
|
-
# else's pressure.
|
|
108
|
-
#
|
|
109
|
-
# Anything not here is `crashed`, which is also where a worker that exited without a signal lands. They
|
|
110
|
-
# were two names, `signal` and `crashed`, for one amount of knowledge: the worker died and nothing says
|
|
111
|
-
# why. One name is honest about that.
|
|
112
|
-
SIGNAL_CAUSES = {
|
|
113
|
-
"XFSZ" => Codes::FSIZE,
|
|
114
|
-
"SEGV" => Codes::MEMORY,
|
|
115
|
-
"ABRT" => Codes::MEMORY,
|
|
116
|
-
"TRAP" => Codes::MEMORY,
|
|
117
|
-
}.freeze
|
|
118
93
|
|
|
119
94
|
SOCKETS = [ "work.sock", "control.sock" ].freeze
|
|
120
95
|
|
|
@@ -557,7 +532,7 @@ module HotCell
|
|
|
557
532
|
elsif message[:idle]
|
|
558
533
|
return unreadable_report child, "idle report from a worker with no request" unless child.busy?
|
|
559
534
|
|
|
560
|
-
finish child, message[:code]
|
|
535
|
+
finish child, message[:code], message[:cause]
|
|
561
536
|
end
|
|
562
537
|
rescue MessageError => error
|
|
563
538
|
unreadable_report child, Failure.sanitize(error.message)
|
|
@@ -568,10 +543,13 @@ module HotCell
|
|
|
568
543
|
# off the hot path, because the supervisor must never delete one inline. It is not tolerated silently:
|
|
569
544
|
# the random suffix exists because a tool running as this user can pre-create a colliding name, so a
|
|
570
545
|
# rename that fails is the shape of that attempt as well as of an ordinary error.
|
|
546
|
+
# The slot directory rather than the request's home, because the supervisor does not know the home. It
|
|
547
|
+
# is named in the worker after the fork, so this copy of the slot holds nil for the whole life of the
|
|
548
|
+
# child — and logging it said `null` on every failure, which is worse than saying nothing.
|
|
571
549
|
def discard(child)
|
|
572
550
|
return if child.slot.discard_home
|
|
573
551
|
|
|
574
|
-
log.write "slot.undiscarded", pid: child.pid, slot: child.slot.number, home: child.slot.
|
|
552
|
+
log.write "slot.undiscarded", pid: child.pid, slot: child.slot.number, home: child.slot.directory
|
|
575
553
|
end
|
|
576
554
|
|
|
577
555
|
def unreadable_report(child, message)
|
|
@@ -579,8 +557,9 @@ module HotCell
|
|
|
579
557
|
nil
|
|
580
558
|
end
|
|
581
559
|
|
|
582
|
-
def finish(child, code)
|
|
560
|
+
def finish(child, code, cause = nil)
|
|
583
561
|
counters.record outcome_code(code)
|
|
562
|
+
counters.record_kill cause if reported_cause?(code, cause)
|
|
584
563
|
child.finished
|
|
585
564
|
discard child
|
|
586
565
|
|
|
@@ -592,6 +571,14 @@ module HotCell
|
|
|
592
571
|
# NoMethodError on `to_sym` past `apply_report`'s rescue and took the cell down, and a stream of unique
|
|
593
572
|
# strings grew the counters without bound. An unknown code is recorded, so a misreporting worker is
|
|
594
573
|
# visible rather than silent, but under one fixed bucket.
|
|
574
|
+
# A worker reports its own kill cause now, so this is a value from a process that may be compromised.
|
|
575
|
+
# It is checked against the known causes rather than passed through, because `record_kill` interns it
|
|
576
|
+
# as a symbol and an unchecked one is an unbounded symbol table keyed by whatever a tool decides.
|
|
577
|
+
def reported_cause?(code, cause)
|
|
578
|
+
outcome_code(code) == Codes::KILLED && cause.is_a?(String) &&
|
|
579
|
+
Codes::PERMANENT_BY_CAUSE.key?(cause)
|
|
580
|
+
end
|
|
581
|
+
|
|
595
582
|
def outcome_code(reported)
|
|
596
583
|
return reported if reported.is_a?(String) && (reported == "ok" || Codes.known?(reported))
|
|
597
584
|
|
|
@@ -725,18 +712,31 @@ module HotCell
|
|
|
725
712
|
nil
|
|
726
713
|
end
|
|
727
714
|
|
|
728
|
-
# A killed worker cannot report its own death, because
|
|
729
|
-
#
|
|
730
|
-
#
|
|
715
|
+
# A killed worker cannot report its own death, because the deadline KILL is enforced by a signal. So
|
|
716
|
+
# the supervisor holds its copy of every dispatched connection and writes the verdict itself. Without
|
|
717
|
+
# this the cold side sees a bare end of stream and cannot tell a limit
|
|
731
718
|
# breach from a crash.
|
|
732
719
|
# A worker still holding a connection at reap time never answered: it reports itself idle after writing,
|
|
733
720
|
# and that report is drained above. So this is the only thing that can answer, and whether it says the
|
|
734
721
|
# input did this or the cell did turns on how the worker died.
|
|
722
|
+
# **The only cause this can authenticate is its own.** `killed_for` is the deadline kill this
|
|
723
|
+
# supervisor sent, and it is the one thing here that knows why a worker died. Everything else is a
|
|
724
|
+
# wait status, which says how — and workers share a uid, so any signal in one may have come from a
|
|
725
|
+
# sibling rather than from the kernel. Reading XFSZ, SEGV, ABRT or TRAP as this request's file size or
|
|
726
|
+
# memory let one compromised worker write a permanent verdict against another request's unrelated
|
|
727
|
+
# input, which Active Storage then kept. `Codes` already stated the rule this broke.
|
|
728
|
+
#
|
|
729
|
+
# The verdicts themselves are not gone, they moved to where they can be earned: the worker answers
|
|
730
|
+
# `memory` when it catches NoMemoryError and `fsize` when a write of its own returns EFBIG, on the
|
|
731
|
+
# connection it is holding. See Worker#disarm_file_size_signal.
|
|
732
|
+
#
|
|
733
|
+
# What that buys is bounded, and it is worth being exact. The supervisor is no longer an instrument
|
|
734
|
+
# for one worker to condemn another's input. A cell compromised outright still answers whatever it
|
|
735
|
+
# likes on a connection it holds, which is the socket-theft residual under "Worker isolation".
|
|
735
736
|
def answer_for(child, status)
|
|
736
737
|
return child.connection&.close unless child.busy?
|
|
737
738
|
|
|
738
|
-
cause = child.killed_for ||
|
|
739
|
-
SIGNAL_CAUSES.fetch(signal_name(status), Codes::CRASHED)
|
|
739
|
+
cause = child.killed_for || Codes::CRASHED
|
|
740
740
|
counters.record Codes::KILLED
|
|
741
741
|
counters.record_kill cause
|
|
742
742
|
|
|
@@ -860,7 +860,7 @@ module HotCell
|
|
|
860
860
|
slot = Slot.build(workspace, number)
|
|
861
861
|
next if slot.prepare
|
|
862
862
|
|
|
863
|
-
log.write "slot.uncleaned", slot: number, home: slot.
|
|
863
|
+
log.write "slot.uncleaned", slot: number, home: slot.directory,
|
|
864
864
|
message: "an earlier boot's files are still here"
|
|
865
865
|
end
|
|
866
866
|
end
|
|
@@ -164,6 +164,46 @@ module HotCell
|
|
|
164
164
|
end
|
|
165
165
|
end
|
|
166
166
|
|
|
167
|
+
class SignalsSibling < HotCell::Operation
|
|
168
|
+
operation "test.signals_sibling"
|
|
169
|
+
|
|
170
|
+
# Workers share a uid and a pid namespace, so one finds another by looking for a process the
|
|
171
|
+
# supervisor also fathered. This is the reproducer for the forged verdict: nothing here touches the
|
|
172
|
+
# victim's input, and the victim is holding an unrelated one.
|
|
173
|
+
def perform(_inputs, _outputs, signal:)
|
|
174
|
+
sibling = siblings.first
|
|
175
|
+
Process.kill signal, sibling if sibling
|
|
176
|
+
|
|
177
|
+
{ signalled: sibling }
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
private
|
|
181
|
+
# Two ways to ask the same question, because the suite runs on macOS as well and only one of them
|
|
182
|
+
# has /proc. An attacker inside a cell has whichever the image gives it; the point of the reproducer
|
|
183
|
+
# is that the answer is obtainable at all.
|
|
184
|
+
def siblings
|
|
185
|
+
pids = Dir.exist?("/proc") ? procfs_children : ps_children
|
|
186
|
+
|
|
187
|
+
pids.reject { |pid| pid == Process.pid }
|
|
188
|
+
end
|
|
189
|
+
|
|
190
|
+
def procfs_children
|
|
191
|
+
Dir.glob("/proc/[0-9]*").filter_map do |path|
|
|
192
|
+
status = File.read(File.join(path, "status"))
|
|
193
|
+
File.basename(path).to_i if status[/^PPid:\s+(\d+)/, 1].to_i == Process.ppid
|
|
194
|
+
rescue SystemCallError
|
|
195
|
+
nil
|
|
196
|
+
end
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
def ps_children
|
|
200
|
+
`ps -A -o pid=,ppid=`.lines.filter_map do |line|
|
|
201
|
+
pid, ppid = line.split.map(&:to_i)
|
|
202
|
+
pid if ppid == Process.ppid
|
|
203
|
+
end
|
|
204
|
+
end
|
|
205
|
+
end
|
|
206
|
+
|
|
167
207
|
# Spawns a process and does not wait for it, the way a worker that crashed mid-request would leave a
|
|
168
208
|
# tool behind. The spawned process inherits the worker's process group, so the supervisor's group sweep
|
|
169
209
|
# at reap is what must kill it — nothing else is watching it, and it has no deadline. Returns the pid so
|
data/lib/hot_cell/worker.rb
CHANGED
|
@@ -40,7 +40,7 @@ module HotCell
|
|
|
40
40
|
# It swallows nothing: `exit! 1` runs whatever was caught.
|
|
41
41
|
def run
|
|
42
42
|
configuration.limits.apply
|
|
43
|
-
|
|
43
|
+
disarm_file_size_signal
|
|
44
44
|
|
|
45
45
|
while (dispatch = await_dispatch)
|
|
46
46
|
serve(*dispatch)
|
|
@@ -56,6 +56,33 @@ module HotCell
|
|
|
56
56
|
private
|
|
57
57
|
attr_reader :slot, :configuration, :control, :log
|
|
58
58
|
|
|
59
|
+
# **The handler carries nothing, and that is the whole point.**
|
|
60
|
+
#
|
|
61
|
+
# RLIMIT_FSIZE is enforced by SIGXFSZ, and the supervisor used to read that signal off a wait status
|
|
62
|
+
# and answer `fsize`, permanently, against whatever input the worker was holding. Workers share a uid,
|
|
63
|
+
# so a sibling sends SIGXFSZ as easily as the kernel does and a wait status cannot tell them apart —
|
|
64
|
+
# nor can a handler, because Ruby hands one only the signal number and no siginfo, so SI_KERNEL and
|
|
65
|
+
# SI_USER are not reachable from here.
|
|
66
|
+
#
|
|
67
|
+
# Catching it makes the kernel fail the offending write with EFBIG rather than killing the process,
|
|
68
|
+
# and that error return is what a signal is not: it is raised by a write this process made, and no
|
|
69
|
+
# signal any sibling sends produces one. So the verdict below keys on Errno::EFBIG and this handler
|
|
70
|
+
# does nothing at all. A handler that set so much as a flag the verdict consulted would hand the
|
|
71
|
+
# forgery straight back.
|
|
72
|
+
#
|
|
73
|
+
# EFBIG is evidence of this request's own write and not proof of which limit stopped it. A filesystem
|
|
74
|
+
# maximum, or the caller's own file, answers the same errno — so this says the bytes did not go, by
|
|
75
|
+
# something this request did, rather than naming RLIMIT_FSIZE. Narrowing it further means checking the
|
|
76
|
+
# written file against the effective limit at each write site, which is not what this changes.
|
|
77
|
+
#
|
|
78
|
+
# A block rather than "IGNORE", because an ignored disposition survives execve and a handled one does
|
|
79
|
+
# not. A tool must keep dying on its own RLIMIT_FSIZE rather than writing past it, and a tool that
|
|
80
|
+
# died by signal is `crashed` and transient — its wait status is no more trustworthy than a worker's,
|
|
81
|
+
# since a sibling can signal a tool too.
|
|
82
|
+
def disarm_file_size_signal
|
|
83
|
+
Signal.trap("XFSZ") { nil }
|
|
84
|
+
end
|
|
85
|
+
|
|
59
86
|
# Returns [connection, queued_ms], or nil once the supervisor has retired this worker by closing the
|
|
60
87
|
# control socket. The connection arrives as a descriptor: the supervisor accepted it and never
|
|
61
88
|
# called recvmsg, so the caller's own descriptors are still queued on it and this worker's recvmsg
|
|
@@ -75,12 +102,17 @@ module HotCell
|
|
|
75
102
|
received = []
|
|
76
103
|
response = nil
|
|
77
104
|
|
|
78
|
-
# Before the request rather than at boot, and one directory rather than two. A tool reads its
|
|
79
|
-
# configuration from $HOME and that configuration is executable, so a home that outlived the request
|
|
80
|
-
# let one compromised conversion reconfigure every later one on this slot. adr/0003.
|
|
81
|
-
slot.make_home
|
|
82
|
-
|
|
83
105
|
begin
|
|
106
|
+
# Before the request rather than at boot, and one directory rather than two. A tool reads its
|
|
107
|
+
# configuration from $HOME and that configuration is executable, so a home that outlived the
|
|
108
|
+
# request let one compromised conversion reconfigure every later one on this slot. adr/0003.
|
|
109
|
+
#
|
|
110
|
+
# Inside the begin, because a home that cannot be created is a broken deployment and answers
|
|
111
|
+
# `failed`, which is transient. Outside it the raise skipped every response path and reached
|
|
112
|
+
# `run`, which exits the worker — after this ensure had already reported idle `"ok"`, counting a
|
|
113
|
+
# success for a request that never ran.
|
|
114
|
+
ENV["HOME"] = slot.make_home
|
|
115
|
+
|
|
84
116
|
line, received = connection.receive_message
|
|
85
117
|
response = if line.nil?
|
|
86
118
|
# The caller closed before sending a request. Transient, so it is never written against a blob,
|
|
@@ -98,6 +130,10 @@ module HotCell
|
|
|
98
130
|
# transient. Adding it back would condemn a blob for the cell's own bad moment.
|
|
99
131
|
rescue NoMemoryError, MemoryExhausted => error
|
|
100
132
|
response = refuse(Codes::KILLED, error, timing, cause: Codes::MEMORY)
|
|
133
|
+
# The one place a file-size verdict can be earned. EFBIG comes back from a write this worker made
|
|
134
|
+
# past its own RLIMIT_FSIZE, so unlike the signal it cannot arrive from anywhere else.
|
|
135
|
+
rescue Errno::EFBIG => error
|
|
136
|
+
response = refuse(Codes::KILLED, error, timing, cause: Codes::FSIZE)
|
|
101
137
|
rescue StandardError => error
|
|
102
138
|
response = refuse("failed", error, timing)
|
|
103
139
|
end
|
|
@@ -107,6 +143,7 @@ module HotCell
|
|
|
107
143
|
ensure
|
|
108
144
|
received.each(&:close)
|
|
109
145
|
connection.close
|
|
146
|
+
home = slot.home
|
|
110
147
|
swept = slot.remove_home
|
|
111
148
|
|
|
112
149
|
# After the answer and before reporting idle, which is the only window where this costs nobody. How
|
|
@@ -115,17 +152,24 @@ module HotCell
|
|
|
115
152
|
# staging, where it would spend the next request's deadline on the previous request's mess. Here the
|
|
116
153
|
# caller already has its response, and `report_idle` is what makes this worker available — so the
|
|
117
154
|
# supervisor will not dispatch into a worker that is still sweeping.
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
report_idle response&.failure
|
|
155
|
+
report_uncleaned home unless swept
|
|
156
|
+
report_unswept unless slot.sweep
|
|
157
|
+
report_idle response&.failure
|
|
121
158
|
end
|
|
122
159
|
|
|
123
160
|
# A removal that failed is the one thing here nobody else can see. The bytes stay on the shared tmpfs
|
|
124
161
|
# after the caller has been told the request is over, and a sibling worker can cause it by writing into
|
|
125
162
|
# the tree while remove_entry walks it. It cannot raise from an ensure, so it says so instead. One line
|
|
126
163
|
# per request, from the ensure, because that is the attempt that knows the final state.
|
|
127
|
-
def report_uncleaned
|
|
128
|
-
log.write "slot.uncleaned", pid: Process.pid, slot: slot.number, home:
|
|
164
|
+
def report_uncleaned(home)
|
|
165
|
+
log.write "slot.uncleaned", pid: Process.pid, slot: slot.number, home: home
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
# The other half of the same fact. A sweep removes what the supervisor renamed out of the way after a
|
|
169
|
+
# killed request, and its failure was the one cleanup outcome nobody said anything about — so a slot
|
|
170
|
+
# accumulating one tree per request looked exactly like a slot that was clean.
|
|
171
|
+
def report_unswept
|
|
172
|
+
log.write "slot.unswept", pid: Process.pid, slot: slot.number, home: slot.directory
|
|
129
173
|
end
|
|
130
174
|
|
|
131
175
|
def handle(line, received, timing)
|
|
@@ -224,8 +268,20 @@ module HotCell
|
|
|
224
268
|
tell deadline: effective(operation).deadline
|
|
225
269
|
end
|
|
226
270
|
|
|
227
|
-
|
|
228
|
-
|
|
271
|
+
# The cause travels with the code so the supervisor can still count a kill by cause. It used to read
|
|
272
|
+
# that off a wait status; it reads the worker's own report now, and only when there is a cause to send,
|
|
273
|
+
# so an ordinary idle report is the two keys it always was.
|
|
274
|
+
#
|
|
275
|
+
# **This is a metric and not a verdict.** The verdict went to the caller on the work connection before
|
|
276
|
+
# this line runs. A compromised worker can report a cause its request never had, or withhold one it
|
|
277
|
+
# did, so `killed_by` is what workers said rather than what happened — which is why the supervisor
|
|
278
|
+
# checks the value against the known causes before interning it, and why nothing downstream may treat
|
|
279
|
+
# it as evidence about a blob.
|
|
280
|
+
def report_idle(failure)
|
|
281
|
+
return tell(idle: true, code: "ok") if failure.nil?
|
|
282
|
+
return tell(idle: true, code: failure.code) if failure.cause.nil?
|
|
283
|
+
|
|
284
|
+
tell idle: true, code: failure.code, cause: failure.cause
|
|
229
285
|
end
|
|
230
286
|
|
|
231
287
|
def tell(**message)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: hotcell-server
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Mike Dalessio
|
|
@@ -15,22 +15,18 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - '='
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: 0.
|
|
18
|
+
version: 0.2.0
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
23
|
- - '='
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: 0.
|
|
25
|
+
version: 0.2.0
|
|
26
26
|
description: |
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
An operation subclasses HotCell::Operation, declares its limits, and implements perform. This gem
|
|
32
|
-
deliberately depends on no application framework: nothing about it loads ActiveSupport, because a
|
|
33
|
-
sandbox should carry only what the conversion needs.
|
|
27
|
+
Runs a HotCell container. A supervisor listens on two Unix sockets, forks a worker for each request,
|
|
28
|
+
and enforces a wall clock deadline and resource limits on it. Write the work as a subclass of
|
|
29
|
+
HotCell::Operation.
|
|
34
30
|
email:
|
|
35
31
|
- mike@37signals.com
|
|
36
32
|
executables:
|
|
@@ -65,8 +61,8 @@ licenses:
|
|
|
65
61
|
- MIT
|
|
66
62
|
metadata:
|
|
67
63
|
homepage_uri: https://github.com/basecamp/hotcell
|
|
68
|
-
source_code_uri: https://github.com/basecamp/hotcell/tree/v0.
|
|
69
|
-
changelog_uri: https://github.com/basecamp/hotcell/blob/v0.
|
|
64
|
+
source_code_uri: https://github.com/basecamp/hotcell/tree/v0.2.0/hotcell-server
|
|
65
|
+
changelog_uri: https://github.com/basecamp/hotcell/blob/v0.2.0/CHANGELOG.md
|
|
70
66
|
bug_tracker_uri: https://github.com/basecamp/hotcell/issues
|
|
71
67
|
rubygems_mfa_required: 'true'
|
|
72
68
|
rdoc_options: []
|