hotcell-server 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: db6e168c58fc1e84ea4411d8935574649f5464059f1e785d43d2388440bcef21
4
- data.tar.gz: 9c2e43635ad2233707d59aa48d76f4b20823ae64bf67d7fc651f8e6d8178a88b
3
+ metadata.gz: 50d90e8b9f7ecb2a139d9c678a3cfdb228288d590f33cdf19390b6cbd92a924e
4
+ data.tar.gz: 66c3bcb911c378507bc9623cb1badc8c3372b6a58c3cd0312eae81c72b003877
5
5
  SHA512:
6
- metadata.gz: 81c839e7cd18af7add06ba34c0e3953787734eaf7906f9c0c6eaf7144e039d4d5dc7caef119fb9d98c006a7c09ce93b99576fa615cddff65c1868bb102a1567a
7
- data.tar.gz: 0744a62b685fd7e96b39f9169869813e7ccc11e9331052742cc3a017be1464090aad6ee6cc0a36de4ec52e0e2802ced1d04215230a2d186e23c6ad2cfd5d41d1
6
+ metadata.gz: c489f02a8b5670abda9df8c52f62fc4657a725abf01c6e49a39ad1065671535f96942e9ed5168fed5062d9b1ce3f55cef5aba0c2dfab2ec2f1c9c35b14980490
7
+ data.tar.gz: 5b633520243d9f70ab9adfaa1f2427516012ca477e3acd4d5917cbfe5a02ca01dc830e6d6855faeda5e11e8d3b8978634cadfa39712938aeeaeb6560e9f8032c
@@ -2,6 +2,6 @@
2
2
 
3
3
  module HotCell
4
4
  module Server
5
- VERSION = "0.1.0"
5
+ VERSION = "0.2.0"
6
6
  end
7
7
  end
data/lib/hot_cell/slot.rb CHANGED
@@ -11,8 +11,7 @@ module HotCell
11
11
  #
12
12
  # **A slot has one directory per request and it is that request's `$HOME`.** It is created when the
13
13
  # request starts and removed when the request ends, so nothing a tool writes under `$HOME` reaches the
14
- # next request on this slot. The name is stable and the directory behind it is not, which is why a
15
- # reused worker reports the same path twice.
14
+ # next request on this slot.
16
15
  #
17
16
  # This directory used to survive, to give a tool with an expensive per-user profile a warm one. That is
18
17
  # withdrawn. What a tool reads from `$HOME` is configuration, and for the toolchains a cell carries
@@ -22,23 +21,48 @@ module HotCell
22
21
  # `max_requests_per_worker: 1` is supposed to hold. adr/0003 records the reversal and adr/0002 the
23
22
  # reasoning it supersedes.
24
23
  #
25
- # There is one directory and not two. A request's staged inputs and outputs are named inside `$HOME`
26
- # rather than in a scratch directory of their own, because the two had the same lifetime and the same
27
- # owner once the home stopped surviving. Staging used to create its directory on demand, which is what
28
- # kept a descriptor-only operation from paying for one; `$HOME` has to exist for every request either
29
- # way, so that laziness bought nothing and is gone with it.
24
+ # **The name is fresh for every request, and that is what makes the removal above a guarantee rather than
25
+ # an intention.** A stable name only holds while the deletion behind it works, and the tool that filled
26
+ # the directory runs as the user that owns it: `chmod 0500` on a directory it has written makes its own
27
+ # configuration unremovable, and the same mode on the slot directory makes it unrenameable too. Both
28
+ # cleanups answer false and both callers log — and a stable name then handed the next request the tree
29
+ # that had just refused to go. A name no earlier request has held is not a name an earlier request could
30
+ # have prepared, so what a failed cleanup now costs is disk rather than the isolation the cell is for.
31
+ #
32
+ # The mode of the slot directory is reasserted for the same reason. It is the one name here that is
33
+ # predictable, so it is the one an earlier request can lock, and a mode on a directory this uid owns is
34
+ # ours to put back.
35
+ #
36
+ # **This bounds what an earlier request left behind, and not what a live one is doing.** Workers share a
37
+ # uid and `0700` is the owner's own mode, so a concurrent sibling — or a `setsid` descendant of a finished
38
+ # request, which process groups do not contain — can still write into a home the moment it exists. The
39
+ # slot directory itself can be renamed aside and replaced, and `chmod` follows what it finds. Both are the
40
+ # residuals `docs/DESIGN.md` records under worker isolation, and neither is closed here.
41
+ #
42
+ # There is one directory per request and not two. A request's staged inputs and outputs are named inside
43
+ # `$HOME` rather than in a scratch directory of their own, because the two had the same lifetime and the
44
+ # same owner once the home stopped surviving. Staging used to create its directory on demand, which is
45
+ # what kept a descriptor-only operation from paying for one; `$HOME` has to exist for every request
46
+ # either way, so that laziness bought nothing and is gone with it.
30
47
  #
31
48
  # The filesystem behaviour belongs here rather than in the two processes that call it. The directory is
32
49
  # removed from both — the worker before it answers, the supervisor at finish and at reap — so the guard
33
50
  # and the swallowed SystemCallError are a rule that has to hold on both sides of a fork, and it had a
34
51
  # copy on each.
35
- Slot = Struct.new(:number, :home) do
52
+ Slot = Struct.new(:number, :directory, :home) do
36
53
  def self.build(workspace, number)
37
- new number, File.join(workspace, number.to_s, "home")
54
+ new number, File.join(workspace, number.to_s)
38
55
  end
39
56
 
57
+ # A name no request has held before, so nothing an earlier one did to the tree it was given reaches this
58
+ # one. The suffix is random rather than a counter for the reason the discarded name's is: the previous
59
+ # request could write to this directory, and a predictable name is one it can pre-create.
40
60
  def make_home
41
- FileUtils.mkdir_p home, mode: 0o700
61
+ FileUtils.mkdir_p directory, mode: 0o700
62
+ FileUtils.chmod 0o700, directory
63
+
64
+ self.home = File.join(directory, "home-#{SecureRandom.hex(8)}")
65
+ Dir.mkdir home, 0o700
42
66
  home
43
67
  end
44
68
 
@@ -55,10 +79,11 @@ module HotCell
55
79
  # response with a crash, and the supervisor calls discard_home from finish and reap, where nothing above
56
80
  # rescues anything and a raise stops the cell with every request it holds.
57
81
  def remove_home
58
- FileUtils.remove_entry home if Dir.exist?(home)
82
+ return true if home.nil?
83
+ return false unless remove_tree(home)
84
+
85
+ self.home = nil
59
86
  true
60
- rescue SystemCallError
61
- false
62
87
  end
63
88
 
64
89
  # **The supervisor renames rather than deletes, and that is a scheduling decision.**
@@ -73,34 +98,74 @@ module HotCell
73
98
  # where the unlinking costs nobody's latency.
74
99
  #
75
100
  # The destination carries a random suffix rather than a counter, because the tool that filled the
76
- # directory runs as this user and can write to the slot's workspace. A predictable name lets it
101
+ # directory runs as this user and can write to the slot's directory. A predictable name lets it
77
102
  # pre-create a colliding entry, fail the rename, and send the supervisor into the recursive delete the
78
103
  # rename exists to avoid. On any rename failure the tree is left where it is, for a later worker's own
79
104
  # cleanup to remove off the hot path. The supervisor never deletes a tree inline, whatever goes wrong.
105
+ # It renames what is there rather than the name it is holding, because the supervisor is the caller that
106
+ # matters and it does not know the name. `home` is assigned in the worker after the fork, so the
107
+ # supervisor's copy of the slot is still nil when it discards a killed worker's tree. At most one worker
108
+ # holds a slot at a time, so everything matching `home-*` under it is that worker's and nobody else's.
80
109
  def discard_home
81
- return true unless Dir.exist?(home)
110
+ FileUtils.chmod 0o700, directory if Dir.exist?(directory)
82
111
 
83
- File.rename home, "#{home}.discarded-#{Process.pid}-#{SecureRandom.hex(8)}"
112
+ Dir.glob(File.join(directory, "home-*")).each do |path|
113
+ File.rename path, File.join(directory, "discarded-#{Process.pid}-#{SecureRandom.hex(8)}")
114
+ end
115
+
116
+ self.home = nil
84
117
  true
85
118
  rescue SystemCallError
86
119
  false
87
120
  end
88
121
 
89
122
  # Nothing here is created at boot, because nothing survives a request. This only clears what an earlier
90
- # boot left behind.
123
+ # boot left behind — the whole slot directory, since the names inside it are an earlier boot's and not
124
+ # this one's to reconstruct.
125
+ #
126
+ # `Dir.exist?` is not the guard, because it follows symlinks and answers false for a dangling one. An
127
+ # entry a tool left in the slot's place is exactly what this has to remove, and it used to survive the
128
+ # sweep and raise from every later `make_home`.
91
129
  def prepare
92
- # Both, rather than short-circuiting: a home that could not be removed must not stop the sweep.
93
- cleared = remove_home
94
- sweep && cleared
130
+ remove_tree directory
95
131
  end
96
132
 
97
133
  # Unlinks whatever discard_home renamed out of the way. Partial progress is fine: a sweep killed
98
134
  # part-way leaves fewer entries for the next one, so this converges rather than repeating.
99
135
  def sweep
100
- Dir.glob("#{home}.discarded-*").each { |path| FileUtils.remove_entry path }
101
- true
136
+ Dir.glob(File.join(directory, "discarded-*")).map { |path| remove_tree(path) }.all?
102
137
  rescue SystemCallError
138
+ # The glob itself can fail, because the slot directory is a name a tool can replace — a symlink loop
139
+ # in its place answers ELOOP here rather than for any one entry. This runs from the worker's ensure,
140
+ # where a raise would replace the caller's response with a crash.
103
141
  false
104
142
  end
143
+
144
+ private
145
+ # **A mode is the only thing a tool needs to make its own tree unremovable, and a mode on a tree this
146
+ # uid owns is ours to put back.** `chmod 0500` on a directory a conversion wrote is enough to fail the
147
+ # recursive delete underneath it, and the delete failing used to be the end of it: one tree per
148
+ # request stayed on the tmpfs until the container ended. The repair is not a permission the process
149
+ # gains, it is one it never lost.
150
+ #
151
+ # Repair only after a failure, never before, so the common request pays for a walk of its own tree
152
+ # once rather than twice. `force:` on the chmod because a partial repair that removes most of the tree
153
+ # is better than none, and the remove that follows is what reports the outcome either way.
154
+ def remove_tree(path)
155
+ return true unless File.exist?(path) || File.symlink?(path)
156
+
157
+ FileUtils.remove_entry path
158
+ true
159
+ rescue SystemCallError
160
+ repair_and_remove path
161
+ end
162
+
163
+ def repair_and_remove(path)
164
+ FileUtils.chmod_R 0o700, path, force: true
165
+ FileUtils.remove_entry path
166
+ true
167
+ rescue SystemCallError
168
+ false
169
+ end
105
170
  end
106
171
  end
@@ -90,31 +90,6 @@ module HotCell
90
90
  # bytes than Linux, and control.sock is the longer of the two names, so it overflows first.
91
91
  SUN_PATH_MAX = RUBY_PLATFORM.include?("darwin") ? 104 : 108
92
92
 
93
- # The signals this cell can attribute to the request the worker was holding. XFSZ is that worker passing
94
- # RLIMIT_FSIZE, and SEGV, ABRT and TRAP are how libvips and GLib die on their own allocation failures —
95
- # libvips dereferences null after printing the correct diagnostic, and g_malloc aborts.
96
- #
97
- # These three are the worker hitting its own per-worker RLIMIT_DATA, which is a property of the input
98
- # this worker held, so the same bytes do it again and the verdict is permanent. `Codes` says a signal
99
- # tells how a process died and never why, so a signal is transient by default — and that is not in
100
- # conflict with a permanent verdict here, because aggregate pressure the worker did not cause arrives as
101
- # SIGKILL, not as these. The two are different signals, and SIGKILL is excluded below.
102
- #
103
- # SIGKILL is deliberately absent, and its absence is the point. The supervisor's own deadline kill is
104
- # already named by `killed_for`, so a SIGKILL reaching this table came from somewhere this process
105
- # cannot see: a cgroup OOM chosen on aggregate pressure, or a sibling worker, which shares a uid and is
106
- # not prevented from signalling. Reading it as this request's memory condemned an input for someone
107
- # else's pressure.
108
- #
109
- # Anything not here is `crashed`, which is also where a worker that exited without a signal lands. They
110
- # were two names, `signal` and `crashed`, for one amount of knowledge: the worker died and nothing says
111
- # why. One name is honest about that.
112
- SIGNAL_CAUSES = {
113
- "XFSZ" => Codes::FSIZE,
114
- "SEGV" => Codes::MEMORY,
115
- "ABRT" => Codes::MEMORY,
116
- "TRAP" => Codes::MEMORY,
117
- }.freeze
118
93
 
119
94
  SOCKETS = [ "work.sock", "control.sock" ].freeze
120
95
 
@@ -557,7 +532,7 @@ module HotCell
557
532
  elsif message[:idle]
558
533
  return unreadable_report child, "idle report from a worker with no request" unless child.busy?
559
534
 
560
- finish child, message[:code]
535
+ finish child, message[:code], message[:cause]
561
536
  end
562
537
  rescue MessageError => error
563
538
  unreadable_report child, Failure.sanitize(error.message)
@@ -568,10 +543,13 @@ module HotCell
568
543
  # off the hot path, because the supervisor must never delete one inline. It is not tolerated silently:
569
544
  # the random suffix exists because a tool running as this user can pre-create a colliding name, so a
570
545
  # rename that fails is the shape of that attempt as well as of an ordinary error.
546
+ # The slot directory rather than the request's home, because the supervisor does not know the home. It
547
+ # is named in the worker after the fork, so this copy of the slot holds nil for the whole life of the
548
+ # child — and logging it said `null` on every failure, which is worse than saying nothing.
571
549
  def discard(child)
572
550
  return if child.slot.discard_home
573
551
 
574
- log.write "slot.undiscarded", pid: child.pid, slot: child.slot.number, home: child.slot.home
552
+ log.write "slot.undiscarded", pid: child.pid, slot: child.slot.number, home: child.slot.directory
575
553
  end
576
554
 
577
555
  def unreadable_report(child, message)
@@ -579,8 +557,9 @@ module HotCell
579
557
  nil
580
558
  end
581
559
 
582
- def finish(child, code)
560
+ def finish(child, code, cause = nil)
583
561
  counters.record outcome_code(code)
562
+ counters.record_kill cause if reported_cause?(code, cause)
584
563
  child.finished
585
564
  discard child
586
565
 
@@ -592,6 +571,14 @@ module HotCell
592
571
  # NoMethodError on `to_sym` past `apply_report`'s rescue and took the cell down, and a stream of unique
593
572
  # strings grew the counters without bound. An unknown code is recorded, so a misreporting worker is
594
573
  # visible rather than silent, but under one fixed bucket.
574
+ # A worker reports its own kill cause now, so this is a value from a process that may be compromised.
575
+ # It is checked against the known causes rather than passed through, because `record_kill` interns it
576
+ # as a symbol and an unchecked one is an unbounded symbol table keyed by whatever a tool decides.
577
+ def reported_cause?(code, cause)
578
+ outcome_code(code) == Codes::KILLED && cause.is_a?(String) &&
579
+ Codes::PERMANENT_BY_CAUSE.key?(cause)
580
+ end
581
+
595
582
  def outcome_code(reported)
596
583
  return reported if reported.is_a?(String) && (reported == "ok" || Codes.known?(reported))
597
584
 
@@ -725,18 +712,31 @@ module HotCell
725
712
  nil
726
713
  end
727
714
 
728
- # A killed worker cannot report its own death, because RLIMIT_FSIZE and the deadline KILL are
729
- # enforced by a signal. So the supervisor holds its copy of every dispatched connection and writes
730
- # the verdict itself. Without this the cold side sees a bare end of stream and cannot tell a limit
715
+ # A killed worker cannot report its own death, because the deadline KILL is enforced by a signal. So
716
+ # the supervisor holds its copy of every dispatched connection and writes the verdict itself. Without
717
+ # this the cold side sees a bare end of stream and cannot tell a limit
731
718
  # breach from a crash.
732
719
  # A worker still holding a connection at reap time never answered: it reports itself idle after writing,
733
720
  # and that report is drained above. So this is the only thing that can answer, and whether it says the
734
721
  # input did this or the cell did turns on how the worker died.
722
+ # **The only cause this can authenticate is its own.** `killed_for` is the deadline kill this
723
+ # supervisor sent, and it is the one thing here that knows why a worker died. Everything else is a
724
+ # wait status, which says how — and workers share a uid, so any signal in one may have come from a
725
+ # sibling rather than from the kernel. Reading XFSZ, SEGV, ABRT or TRAP as this request's file size or
726
+ # memory let one compromised worker write a permanent verdict against another request's unrelated
727
+ # input, which Active Storage then kept. `Codes` already stated the rule this broke.
728
+ #
729
+ # The verdicts themselves are not gone, they moved to where they can be earned: the worker answers
730
+ # `memory` when it catches NoMemoryError and `fsize` when a write of its own returns EFBIG, on the
731
+ # connection it is holding. See Worker#disarm_file_size_signal.
732
+ #
733
+ # What that buys is bounded, and it is worth being exact. The supervisor is no longer an instrument
734
+ # for one worker to condemn another's input. A cell compromised outright still answers whatever it
735
+ # likes on a connection it holds, which is the socket-theft residual under "Worker isolation".
735
736
  def answer_for(child, status)
736
737
  return child.connection&.close unless child.busy?
737
738
 
738
- cause = child.killed_for ||
739
- SIGNAL_CAUSES.fetch(signal_name(status), Codes::CRASHED)
739
+ cause = child.killed_for || Codes::CRASHED
740
740
  counters.record Codes::KILLED
741
741
  counters.record_kill cause
742
742
 
@@ -860,7 +860,7 @@ module HotCell
860
860
  slot = Slot.build(workspace, number)
861
861
  next if slot.prepare
862
862
 
863
- log.write "slot.uncleaned", slot: number, home: slot.home,
863
+ log.write "slot.uncleaned", slot: number, home: slot.directory,
864
864
  message: "an earlier boot's files are still here"
865
865
  end
866
866
  end
@@ -164,6 +164,46 @@ module HotCell
164
164
  end
165
165
  end
166
166
 
167
+ class SignalsSibling < HotCell::Operation
168
+ operation "test.signals_sibling"
169
+
170
+ # Workers share a uid and a pid namespace, so one finds another by looking for a process the
171
+ # supervisor also fathered. This is the reproducer for the forged verdict: nothing here touches the
172
+ # victim's input, and the victim is holding an unrelated one.
173
+ def perform(_inputs, _outputs, signal:)
174
+ sibling = siblings.first
175
+ Process.kill signal, sibling if sibling
176
+
177
+ { signalled: sibling }
178
+ end
179
+
180
+ private
181
+ # Two ways to ask the same question, because the suite runs on macOS as well and only one of them
182
+ # has /proc. An attacker inside a cell has whichever the image gives it; the point of the reproducer
183
+ # is that the answer is obtainable at all.
184
+ def siblings
185
+ pids = Dir.exist?("/proc") ? procfs_children : ps_children
186
+
187
+ pids.reject { |pid| pid == Process.pid }
188
+ end
189
+
190
+ def procfs_children
191
+ Dir.glob("/proc/[0-9]*").filter_map do |path|
192
+ status = File.read(File.join(path, "status"))
193
+ File.basename(path).to_i if status[/^PPid:\s+(\d+)/, 1].to_i == Process.ppid
194
+ rescue SystemCallError
195
+ nil
196
+ end
197
+ end
198
+
199
+ def ps_children
200
+ `ps -A -o pid=,ppid=`.lines.filter_map do |line|
201
+ pid, ppid = line.split.map(&:to_i)
202
+ pid if ppid == Process.ppid
203
+ end
204
+ end
205
+ end
206
+
167
207
  # Spawns a process and does not wait for it, the way a worker that crashed mid-request would leave a
168
208
  # tool behind. The spawned process inherits the worker's process group, so the supervisor's group sweep
169
209
  # at reap is what must kill it — nothing else is watching it, and it has no deadline. Returns the pid so
@@ -40,7 +40,7 @@ module HotCell
40
40
  # It swallows nothing: `exit! 1` runs whatever was caught.
41
41
  def run
42
42
  configuration.limits.apply
43
- ENV["HOME"] = slot.home
43
+ disarm_file_size_signal
44
44
 
45
45
  while (dispatch = await_dispatch)
46
46
  serve(*dispatch)
@@ -56,6 +56,33 @@ module HotCell
56
56
  private
57
57
  attr_reader :slot, :configuration, :control, :log
58
58
 
59
+ # **The handler carries nothing, and that is the whole point.**
60
+ #
61
+ # RLIMIT_FSIZE is enforced by SIGXFSZ, and the supervisor used to read that signal off a wait status
62
+ # and answer `fsize`, permanently, against whatever input the worker was holding. Workers share a uid,
63
+ # so a sibling sends SIGXFSZ as easily as the kernel does and a wait status cannot tell them apart —
64
+ # nor can a handler, because Ruby hands one only the signal number and no siginfo, so SI_KERNEL and
65
+ # SI_USER are not reachable from here.
66
+ #
67
+ # Catching it makes the kernel fail the offending write with EFBIG rather than killing the process,
68
+ # and that error return is what a signal is not: it is raised by a write this process made, and no
69
+ # signal any sibling sends produces one. So the verdict below keys on Errno::EFBIG and this handler
70
+ # does nothing at all. A handler that set so much as a flag the verdict consulted would hand the
71
+ # forgery straight back.
72
+ #
73
+ # EFBIG is evidence of this request's own write and not proof of which limit stopped it. A filesystem
74
+ # maximum, or the caller's own file, answers the same errno — so this says the bytes did not go, by
75
+ # something this request did, rather than naming RLIMIT_FSIZE. Narrowing it further means checking the
76
+ # written file against the effective limit at each write site, which is not what this changes.
77
+ #
78
+ # A block rather than "IGNORE", because an ignored disposition survives execve and a handled one does
79
+ # not. A tool must keep dying on its own RLIMIT_FSIZE rather than writing past it, and a tool that
80
+ # died by signal is `crashed` and transient — its wait status is no more trustworthy than a worker's,
81
+ # since a sibling can signal a tool too.
82
+ def disarm_file_size_signal
83
+ Signal.trap("XFSZ") { nil }
84
+ end
85
+
59
86
  # Returns [connection, queued_ms], or nil once the supervisor has retired this worker by closing the
60
87
  # control socket. The connection arrives as a descriptor: the supervisor accepted it and never
61
88
  # called recvmsg, so the caller's own descriptors are still queued on it and this worker's recvmsg
@@ -75,12 +102,17 @@ module HotCell
75
102
  received = []
76
103
  response = nil
77
104
 
78
- # Before the request rather than at boot, and one directory rather than two. A tool reads its
79
- # configuration from $HOME and that configuration is executable, so a home that outlived the request
80
- # let one compromised conversion reconfigure every later one on this slot. adr/0003.
81
- slot.make_home
82
-
83
105
  begin
106
+ # Before the request rather than at boot, and one directory rather than two. A tool reads its
107
+ # configuration from $HOME and that configuration is executable, so a home that outlived the
108
+ # request let one compromised conversion reconfigure every later one on this slot. adr/0003.
109
+ #
110
+ # Inside the begin, because a home that cannot be created is a broken deployment and answers
111
+ # `failed`, which is transient. Outside it the raise skipped every response path and reached
112
+ # `run`, which exits the worker — after this ensure had already reported idle `"ok"`, counting a
113
+ # success for a request that never ran.
114
+ ENV["HOME"] = slot.make_home
115
+
84
116
  line, received = connection.receive_message
85
117
  response = if line.nil?
86
118
  # The caller closed before sending a request. Transient, so it is never written against a blob,
@@ -98,6 +130,10 @@ module HotCell
98
130
  # transient. Adding it back would condemn a blob for the cell's own bad moment.
99
131
  rescue NoMemoryError, MemoryExhausted => error
100
132
  response = refuse(Codes::KILLED, error, timing, cause: Codes::MEMORY)
133
+ # The one place a file-size verdict can be earned. EFBIG comes back from a write this worker made
134
+ # past its own RLIMIT_FSIZE, so unlike the signal it cannot arrive from anywhere else.
135
+ rescue Errno::EFBIG => error
136
+ response = refuse(Codes::KILLED, error, timing, cause: Codes::FSIZE)
101
137
  rescue StandardError => error
102
138
  response = refuse("failed", error, timing)
103
139
  end
@@ -107,6 +143,7 @@ module HotCell
107
143
  ensure
108
144
  received.each(&:close)
109
145
  connection.close
146
+ home = slot.home
110
147
  swept = slot.remove_home
111
148
 
112
149
  # After the answer and before reporting idle, which is the only window where this costs nobody. How
@@ -115,17 +152,24 @@ module HotCell
115
152
  # staging, where it would spend the next request's deadline on the previous request's mess. Here the
116
153
  # caller already has its response, and `report_idle` is what makes this worker available — so the
117
154
  # supervisor will not dispatch into a worker that is still sweeping.
118
- slot.sweep
119
- report_uncleaned unless swept
120
- report_idle response&.failure&.code
155
+ report_uncleaned home unless swept
156
+ report_unswept unless slot.sweep
157
+ report_idle response&.failure
121
158
  end
122
159
 
123
160
  # A removal that failed is the one thing here nobody else can see. The bytes stay on the shared tmpfs
124
161
  # after the caller has been told the request is over, and a sibling worker can cause it by writing into
125
162
  # the tree while remove_entry walks it. It cannot raise from an ensure, so it says so instead. One line
126
163
  # per request, from the ensure, because that is the attempt that knows the final state.
127
- def report_uncleaned
128
- log.write "slot.uncleaned", pid: Process.pid, slot: slot.number, home: slot.home
164
+ def report_uncleaned(home)
165
+ log.write "slot.uncleaned", pid: Process.pid, slot: slot.number, home: home
166
+ end
167
+
168
+ # The other half of the same fact. A sweep removes what the supervisor renamed out of the way after a
169
+ # killed request, and its failure was the one cleanup outcome nobody said anything about — so a slot
170
+ # accumulating one tree per request looked exactly like a slot that was clean.
171
+ def report_unswept
172
+ log.write "slot.unswept", pid: Process.pid, slot: slot.number, home: slot.directory
129
173
  end
130
174
 
131
175
  def handle(line, received, timing)
@@ -224,8 +268,20 @@ module HotCell
224
268
  tell deadline: effective(operation).deadline
225
269
  end
226
270
 
227
- def report_idle(code)
228
- tell idle: true, code: code || "ok"
271
+ # The cause travels with the code so the supervisor can still count a kill by cause. It used to read
272
+ # that off a wait status; it reads the worker's own report now, and only when there is a cause to send,
273
+ # so an ordinary idle report is the two keys it always was.
274
+ #
275
+ # **This is a metric and not a verdict.** The verdict went to the caller on the work connection before
276
+ # this line runs. A compromised worker can report a cause its request never had, or withhold one it
277
+ # did, so `killed_by` is what workers said rather than what happened — which is why the supervisor
278
+ # checks the value against the known causes before interning it, and why nothing downstream may treat
279
+ # it as evidence about a blob.
280
+ def report_idle(failure)
281
+ return tell(idle: true, code: "ok") if failure.nil?
282
+ return tell(idle: true, code: failure.code) if failure.cause.nil?
283
+
284
+ tell idle: true, code: failure.code, cause: failure.cause
229
285
  end
230
286
 
231
287
  def tell(**message)
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: hotcell-server
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Mike Dalessio
@@ -15,22 +15,18 @@ dependencies:
15
15
  requirements:
16
16
  - - '='
17
17
  - !ruby/object:Gem::Version
18
- version: 0.1.0
18
+ version: 0.2.0
19
19
  type: :runtime
20
20
  prerelease: false
21
21
  version_requirements: !ruby/object:Gem::Requirement
22
22
  requirements:
23
23
  - - '='
24
24
  - !ruby/object:Gem::Version
25
- version: 0.1.0
25
+ version: 0.2.0
26
26
  description: |
27
- The cell side of HotCell. A supervisor listens on two Unix sockets, forks a worker per request,
28
- holds each worker to a wall-clock deadline and a set of resource limits, and answers for one that
29
- dies without reporting.
30
-
31
- An operation subclasses HotCell::Operation, declares its limits, and implements perform. This gem
32
- deliberately depends on no application framework: nothing about it loads ActiveSupport, because a
33
- sandbox should carry only what the conversion needs.
27
+ Runs a HotCell container. A supervisor listens on two Unix sockets, forks a worker for each request,
28
+ and enforces a wall clock deadline and resource limits on it. Write the work as a subclass of
29
+ HotCell::Operation.
34
30
  email:
35
31
  - mike@37signals.com
36
32
  executables:
@@ -65,8 +61,8 @@ licenses:
65
61
  - MIT
66
62
  metadata:
67
63
  homepage_uri: https://github.com/basecamp/hotcell
68
- source_code_uri: https://github.com/basecamp/hotcell/tree/v0.1.0/hotcell-server
69
- changelog_uri: https://github.com/basecamp/hotcell/blob/v0.1.0/CHANGELOG.md
64
+ source_code_uri: https://github.com/basecamp/hotcell/tree/v0.2.0/hotcell-server
65
+ changelog_uri: https://github.com/basecamp/hotcell/blob/v0.2.0/CHANGELOG.md
70
66
  bug_tracker_uri: https://github.com/basecamp/hotcell/issues
71
67
  rubygems_mfa_required: 'true'
72
68
  rdoc_options: []