@quenty/datastore 13.51.0 → 13.51.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +10 -0
- package/docs/shutdown-and-session-locks.md +183 -0
- package/package.json +2 -2
- package/src/Server/DataStore.GracefulClose.spec.lua +12 -10
- package/src/Server/DataStoreTestUtils.lua +79 -3
- package/src/Server/PlayerDataStoreManager.RemovalCallbacks.spec.lua +5 -0
- package/src/Server/PlayerDataStoreManager.lua +24 -34
- package/src/Server/PlayerDataStoreManager.spec.lua +69 -44
- package/src/Server/PlayerDataStoreService.spec.lua +28 -23
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,13 @@
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
|
|
5
5
|
|
|
6
|
+
## [13.51.1](https://github.com/Quenty/NevermoreEngine/compare/@quenty/datastore@13.51.0...@quenty/datastore@13.51.1) (2026-07-30)
|
|
7
|
+
|
|
8
|
+
### Bug Fixes
|
|
9
|
+
|
|
10
|
+
- **datastore:** stop manager teardown from destroying stores before PlayerRemoving saves them ([44d786e](https://github.com/Quenty/NevermoreEngine/commit/44d786edb1501c37d540d6a3226084f4d0859f8b))
|
|
11
|
+
- **datastore:** stop the spec shutdown helper stalling on a bag that was never started ([0696638](https://github.com/Quenty/NevermoreEngine/commit/0696638700a33cbcedc85f4c75099f54cb952a6c))
|
|
12
|
+
|
|
6
13
|
# [13.51.0](https://github.com/Quenty/NevermoreEngine/compare/@quenty/datastore@13.50.2...@quenty/datastore@13.51.0) (2026-07-28)
|
|
7
14
|
|
|
8
15
|
**Note:** Version bump only for package @quenty/datastore
|
package/README.md
CHANGED
|
@@ -18,6 +18,16 @@ This system is a reliable datastore system designed with promises and asyncronio
|
|
|
18
18
|
## Executive overiew
|
|
19
19
|
This datastore prevents data loss by being explicit about what we're writing to, and only modifying the data that exists there instead of modifying the whole structure.
|
|
20
20
|
|
|
21
|
+
## Working on this package
|
|
22
|
+
|
|
23
|
+
Read this before changing how player data is saved, and before adding any cleanup, teardown, or
|
|
24
|
+
flush to that path. It records design intent that cannot live in the code, because the shape is
|
|
25
|
+
driven by Roblox shutdown and cross-server session-lock behavior that is not visible from reading it:
|
|
26
|
+
|
|
27
|
+
- [`docs/shutdown-and-session-locks.md`](docs/shutdown-and-session-locks.md) — why every save routes
|
|
28
|
+
through one function, what `PromiseAllSaves` has to wait on, and why flushing stores on manager
|
|
29
|
+
teardown was tried twice and removed.
|
|
30
|
+
|
|
21
31
|
## Comparison to other solutions
|
|
22
32
|
|
|
23
33
|
* Not specifically locked to players
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
# Shutdown and session locks
|
|
2
|
+
|
|
3
|
+
Why [PlayerDataStoreManager] saves the way it does. Written for whoever changes this package next.
|
|
4
|
+
Consumers do not need any of it — using the manager correctly requires nothing from this file.
|
|
5
|
+
|
|
6
|
+
Read it before adding any cleanup, teardown, or flush to the save path. Two attempts at exactly that
|
|
7
|
+
regressed player data in production, in opposite directions, and both looked obviously correct.
|
|
8
|
+
|
|
9
|
+
## The engine behavior this rests on
|
|
10
|
+
|
|
11
|
+
Two facts about Roblox, neither of them discoverable from this package's code:
|
|
12
|
+
|
|
13
|
+
**A closing server fires `PlayerRemoving` for every player still in it**, and holds the shutdown open
|
|
14
|
+
for those handlers the same way it does for `BindToClose`. A restart is not a case where players
|
|
15
|
+
"never leave" — they all leave, then the server closes. So `PlayerRemoving` is the save path during a
|
|
16
|
+
shutdown, not a peacetime-only path that something else has to stand in for.
|
|
17
|
+
|
|
18
|
+
**A session lock can only be released by the server holding it.** Nothing gives one server the
|
|
19
|
+
authority to unlock another's key. If a server dies still holding a lock, the next server that loads
|
|
20
|
+
that key sees a lock that still looks live and takes the graceful route: it messages the holder's
|
|
21
|
+
JobId asking it to close, and that JobId is gone, so nothing answers.
|
|
22
|
+
|
|
23
|
+
The cost of that is worth knowing precisely, because the obvious knob is the wrong one. Each attempt
|
|
24
|
+
dies on the **hardcoded 5s timeout in `DataStoreMessageHelper.PromiseCloseSessionGraceful`**, not on
|
|
25
|
+
`SetSessionMessagingCloseDelaySeconds` — that delay sits in the *fulfilled* branch of
|
|
26
|
+
`_promiseGetAsyncNoCache` and is never reached when the holder is dead. Six attempts of that is 30s,
|
|
27
|
+
and `PromiseRetryUtils` adds five jittered inter-attempt waits totalling ~49s, so a player waits
|
|
28
|
+
roughly **80 seconds** before the lock is finally stolen. Turning `SetSessionMessagingCloseDelaySeconds`
|
|
29
|
+
down does not shorten it; the lever is that hardcoded 5.
|
|
30
|
+
|
|
31
|
+
## The shape
|
|
32
|
+
|
|
33
|
+
Everything converges on one function:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
Players.PlayerRemoving ─┐
|
|
37
|
+
SessionStolen ──────────┤
|
|
38
|
+
SessionCloseRequested ──┼──► _removePlayerDataStore(userId)
|
|
39
|
+
PromiseSessionLockingFailed ─┤ removing callbacks
|
|
40
|
+
RemovePlayerDataStore ──┘ └─► SaveAndCloseSession() -- writes data AND releases the lock
|
|
41
|
+
└─► Destroy() -- only after that write settles
|
|
42
|
+
|
|
43
|
+
BindToCloseService ────────► PromiseAllSaves()
|
|
44
|
+
removes anything left, then WAITS
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
`_removePlayerDataStore` is idempotent and order-independent: it returns early if the store is already
|
|
48
|
+
gone from `_datastores` or already `_removing`. So on a shutdown, whichever entry point reaches a given
|
|
49
|
+
player first performs the complete sequence and the others no-op. There is deliberately **no path that
|
|
50
|
+
destroys a store before its save-and-close has been attempted and settled** — note "attempted": the
|
|
51
|
+
`Destroy()` sits in a `Finally`, so a save that rejects still tears the store down.
|
|
52
|
+
|
|
53
|
+
That `Finally` has a second consequence worth knowing. `Promise.Finally` is `Then(f, f)` and `f`
|
|
54
|
+
returns nothing, so the derived promise *fulfills* even when the save rejected. `removalPromise`
|
|
55
|
+
therefore never rejects, and `PromiseAllSaves` resolving means every removal **settled**, not that
|
|
56
|
+
every write succeeded. It is the right shape for a shutdown — one player's failed write must not
|
|
57
|
+
abandon everyone else's — but do not read a resolved close as proof of a successful save.
|
|
58
|
+
|
|
59
|
+
`PromiseAllSaves` is a *waiter*, not a saver. `BindToCloseService` yields on it, which is the only
|
|
60
|
+
thing keeping the server alive long enough for the writes to land.
|
|
61
|
+
|
|
62
|
+
## What `PromiseAllSaves` has to wait on, and why it isn't obvious
|
|
63
|
+
|
|
64
|
+
It waits on the in-flight removal chains in `_removingPromises`, not only on `_pendingSaves`. This is
|
|
65
|
+
the part that actually loses live-server data, so it is worth following exactly.
|
|
66
|
+
|
|
67
|
+
`_pendingSaves` is fed by a `datastore.Saving:Connect` that lives on the per-user maid at
|
|
68
|
+
`self._maid._savingConns[userId]`. The last thing `_removePlayerDataStore` does is
|
|
69
|
+
`self._maid._savingConns[userId] = nil`, which cleans that maid and **disconnects `Saving`**. And
|
|
70
|
+
`DataStore` fires `Saving` at the very *end* of `_doDataSync` — after `PromiseViewUpToDate()` and after
|
|
71
|
+
every saving callback has resolved.
|
|
72
|
+
|
|
73
|
+
So the two only line up when the whole removal chain runs synchronously. The moment anything in it
|
|
74
|
+
yields — an async removing callback, an async saving callback, or a session-locked load still in flight
|
|
75
|
+
because the player left seconds after joining — `SaveAndCloseSession` reaches `Saving` *after* the
|
|
76
|
+
connection is already gone, and that save can **never** enter `_pendingSaves` at all. It is not a race
|
|
77
|
+
that usually goes the right way; it is a permanent miss.
|
|
78
|
+
|
|
79
|
+
The old `PromiseAllSaves` then had literally nothing to wait on: `PromiseUtils.all({})` returns an
|
|
80
|
+
already-fulfilled promise, `BindToCloseService` stops yielding, and Roblox kills the server mid-write —
|
|
81
|
+
leaving the session locked, with nobody able to release it, and the next server paying the ~80 seconds.
|
|
82
|
+
Any consumer with an async removing or saving callback hits this, which is most of them.
|
|
83
|
+
|
|
84
|
+
Waiting on `_removingPromises` fixes that, because the entry is inserted synchronously and cleared in an
|
|
85
|
+
identity-guarded `Finally`, so a fast leave/rejoin cannot clobber the entry the close is waiting on.
|
|
86
|
+
|
|
87
|
+
It does not close the hole completely, and the remaining sliver is worth knowing rather than
|
|
88
|
+
rediscovering. `_removePlayerDataStore` latches `_removing[userId] = true` *before* it invokes the
|
|
89
|
+
removing callbacks, and only inserts into `_removingPromises` *after* that loop returns. A callback that
|
|
90
|
+
yields inside the loop — or throws out of it — leaves a window where the removal is latched but untracked,
|
|
91
|
+
and a close landing in that window early-returns on the `_removing` guard and again finds nothing to wait
|
|
92
|
+
on. A throwing callback makes it permanent: `_removing` stays true, so that store can never be removed.
|
|
93
|
+
|
|
94
|
+
This is pre-existing, not introduced by the change that added the wait, and the window is far smaller
|
|
95
|
+
than what it replaced: previously the miss lasted the whole removal for *any* non-synchronous chain,
|
|
96
|
+
where now it lasts only while a consumer callback is on the stack inside one loop.
|
|
97
|
+
`RemovalCallbacks.spec.lua` characterizes both callback behaviours, though neither test drives a close
|
|
98
|
+
across the window, so the sliver itself is uncovered.
|
|
99
|
+
|
|
100
|
+
Closing it means publishing the tracking entry before any consumer code runs — a pending promise
|
|
101
|
+
inserted at latch time and resolved to the real chain afterwards. That is a change to
|
|
102
|
+
`_removePlayerDataStore` itself, so it wants its own review, and note it only fixes the yielding case:
|
|
103
|
+
after a *throw* the placeholder is never resolved, so the close would hang on it instead of resolving
|
|
104
|
+
early. Arguably the better failure, but it needs the callback loop isolated to actually be closed.
|
|
105
|
+
|
|
106
|
+
## Rejected: flushing the stores on manager teardown
|
|
107
|
+
|
|
108
|
+
`_flushAndDestroyAll` existed for twelve days (added 2026-07-17, removed 2026-07-29) and was removed. It
|
|
109
|
+
ran from the manager's Maid and, for every store still in `_datastores`, saved and then destroyed it
|
|
110
|
+
synchronously.
|
|
111
|
+
|
|
112
|
+
It was added for a real reason — a `DataStore` starts a `task.spawn` auto-save loop once loaded and
|
|
113
|
+
only cancels it on `Destroy()`, so a manager torn down without destroying its stores leaks those loops
|
|
114
|
+
(which matters in the shared test place, where a leaked loop fires inside a later package's window).
|
|
115
|
+
|
|
116
|
+
It was wrong anyway, because **destroying the manager is not a shutdown.** No path in `ServiceBag`,
|
|
117
|
+
`BindToCloseService`, or any game in this repo destroys it on close; only a hot reload, a Studio stop, or
|
|
118
|
+
a test does. But wherever something did, the teardown got there before `PlayerRemoving`, cleared
|
|
119
|
+
`_datastores`, and destroyed the stores — so the removal that would have saved and closed the session
|
|
120
|
+
found nothing and returned early. Both flavors failed:
|
|
121
|
+
|
|
122
|
+
- With `Save()`, the data was flushed but the lock was never released, handing the next server the
|
|
123
|
+
~80-second ladder.
|
|
124
|
+
- With `SaveAndCloseSession()`, the lock was released, but the store was destroyed synchronously
|
|
125
|
+
against its own in-flight write, and the session was closed while players were still in the server
|
|
126
|
+
and still writing. Everything still holding the store wrote into a destroyed object:
|
|
127
|
+
`attempt to call missing method 'GetSubStore'`, and consumers' own "datastore already cleaned up"
|
|
128
|
+
guards firing on a loop for the rest of the shutdown window.
|
|
129
|
+
|
|
130
|
+
The lesson is narrow and worth stating plainly: **the leak was a test-harness problem and belonged in
|
|
131
|
+
the test harness.** `DataStoreTestUtils` now shuts a manager down the way Roblox does before tearing it
|
|
132
|
+
down (`promiseSimulatedShutdown`), which removes the stores through the real path and cancels their
|
|
133
|
+
loops as a side effect. Production code did not need a second save path, and could not safely have one.
|
|
134
|
+
|
|
135
|
+
## Deliberate: a removing callback that never resolves blocks its removal forever
|
|
136
|
+
|
|
137
|
+
There is no timeout on the removing callbacks, and that is the intended behavior, not an oversight. A
|
|
138
|
+
callback that never settles holds its removal open, so the save never happens and the lock stays held,
|
|
139
|
+
and Roblox tears the thread down at the shutdown cap. A timeout here would convert that into a silent
|
|
140
|
+
partial save on a cadence nobody chose — better to let the consumer's broken callback take the blame.
|
|
141
|
+
|
|
142
|
+
Be honest about the diagnostic, though: nothing here names the callback that hung. By the time the
|
|
143
|
+
removal is stuck, the callback has already *returned* (it handed back a promise that never settles), so
|
|
144
|
+
its frame is gone; the thread Roblox kills is `BindToCloseService`'s `:Yield()`. If chasing one of these
|
|
145
|
+
gets painful, the missing piece is a `task.delay` warn naming the still-pending userIds — not a timeout.
|
|
146
|
+
|
|
147
|
+
`PlayerDataStoreManager.RemovalCallbacks.spec.lua` characterizes it under "failure modes" so the
|
|
148
|
+
behavior is pinned rather than accidental.
|
|
149
|
+
|
|
150
|
+
Since `PromiseAllSaves` now awaits the removal chains, this is also a new source of shutdown latency,
|
|
151
|
+
and it is bounded and benign: `PromiseUtils.all` never short-circuits, so one stuck member delays the
|
|
152
|
+
close but cannot cancel or skip the others, and every other player's removal was already dispatched and
|
|
153
|
+
completes concurrently. Roblox's ~30s cap ends it either way. The likely trigger in practice is not a
|
|
154
|
+
hung callback but a contended session-locked load — a player who joined seconds before the shutdown can
|
|
155
|
+
hold the close for as long as the acquire ladder runs.
|
|
156
|
+
|
|
157
|
+
## Testing this
|
|
158
|
+
|
|
159
|
+
One consequence of dropping the flush: the `DataStore` instances are not owned by any maid (only
|
|
160
|
+
`_removePlayerDataStore`'s `Finally` destroys them), so **`manager:Destroy()` now destroys no stores.**
|
|
161
|
+
A harness that tears down by destroying a ServiceBag therefore leaves every loaded store alive with its
|
|
162
|
+
auto-save loop running — and a `PlayerMock` never fires the real `Players.PlayerRemoving`, so nothing
|
|
163
|
+
else removes them either. Any spec built that way has to shut down explicitly before it tears down.
|
|
164
|
+
|
|
165
|
+
Resist the urge to fix that by re-adding a destroy on the manager's maid. A destroy-only teardown is
|
|
166
|
+
worse than the leak: it would leave `_datastores` populated with destroyed stores, and clearing
|
|
167
|
+
`_datastores` too would silently drop the save instead.
|
|
168
|
+
|
|
169
|
+
Not via a later `PlayerRemoving`, though — `Maid.DoCleaning` disconnects every `RBXScriptConnection` in a
|
|
170
|
+
dedicated first pass before it runs any function task, so the manager's `PlayerRemoving` handler is
|
|
171
|
+
already gone by then. The reachable case is a removal *already in flight*: a removing callback that
|
|
172
|
+
yielded resumes to find the store destroyed under it, and `SaveAndCloseSession()` hits a nil metatable.
|
|
173
|
+
The leak is a harness problem; keep the fix in the harness.
|
|
174
|
+
|
|
175
|
+
`manager:Destroy()` is not a shutdown and specs must not use it as one — several did, which is how the
|
|
176
|
+
teardown flush looked well covered while being wrong. Drive
|
|
177
|
+
`DataStoreTestUtils.promiseSimulatedShutdown(manager, userIds)` instead: it fires the removals, then
|
|
178
|
+
returns the promise `BindToCloseService` would yield on.
|
|
179
|
+
|
|
180
|
+
The assertion that matters is not "the data was saved" but "the data was saved *by the time the close
|
|
181
|
+
resolved*" — that is the difference between a server that shuts down cleanly and one that dies holding
|
|
182
|
+
a lock. `PlayerDataStoreManager.spec.lua`'s "does not resolve the close while a PlayerRemoving save is
|
|
183
|
+
still in flight" pins it, using an async removing callback to open the window.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@quenty/datastore",
|
|
3
|
-
"version": "13.51.
|
|
3
|
+
"version": "13.51.1",
|
|
4
4
|
"description": "Quenty's Datastore implementation for Roblox",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"Roblox",
|
|
@@ -51,5 +51,5 @@
|
|
|
51
51
|
"publishConfig": {
|
|
52
52
|
"access": "public"
|
|
53
53
|
},
|
|
54
|
-
"gitHead": "
|
|
54
|
+
"gitHead": "03307dd337fa003fce541cac45106b0df6fbf7d9"
|
|
55
55
|
}
|
|
@@ -22,12 +22,11 @@ local describe = Jest.Globals.describe
|
|
|
22
22
|
local expect = Jest.Globals.expect
|
|
23
23
|
local it = Jest.Globals.it
|
|
24
24
|
|
|
25
|
-
describe("clean session end releases the lock (server-shutdown
|
|
26
|
-
it("
|
|
25
|
+
describe("clean session end releases the lock (server-shutdown path)", function()
|
|
26
|
+
it("a closing server releases the session so the next server loads immediately", function()
|
|
27
27
|
local mock = DataStoreMock.new()
|
|
28
28
|
|
|
29
|
-
-- Server A: the
|
|
30
|
-
-- first), which is exactly what a ServiceBag destroy after a clean session looks like.
|
|
29
|
+
-- Server A: a player in the server with a live session, about to be shut down.
|
|
31
30
|
local maidA = Maid.new()
|
|
32
31
|
local serviceBagA = DataStoreTestUtils.newServiceBag(maidA, MessagingServiceMock.new())
|
|
33
32
|
local managerA = maidA:Add(PlayerDataStoreManager.new(serviceBagA, mock :: any, function(userId)
|
|
@@ -47,14 +46,17 @@ describe("clean session end releases the lock (server-shutdown teardown path)",
|
|
|
47
46
|
return
|
|
48
47
|
end
|
|
49
48
|
|
|
50
|
-
-- Clean shutdown:
|
|
49
|
+
-- Clean shutdown: Roblox fires PlayerRemoving for the player still in the server, and holds the
|
|
50
|
+
-- close open until that removal has flushed.
|
|
51
|
+
if not PromiseTestUtils.awaitSettled(DataStoreTestUtils.promiseSimulatedShutdown(managerA, { 1 }), 10) then
|
|
52
|
+
expect("A's shutdown never flushed").toEqual("A's shutdown flushed")
|
|
53
|
+
maidA:DoCleaning()
|
|
54
|
+
return
|
|
55
|
+
end
|
|
51
56
|
maidA:DoCleaning()
|
|
52
57
|
|
|
53
|
-
-- The
|
|
54
|
-
expect(
|
|
55
|
-
local raw = mock:GetRaw("user_1")
|
|
56
|
-
return raw ~= nil and raw.lock == nil
|
|
57
|
-
end, 5)).toEqual(true)
|
|
58
|
+
-- The close must have written the graceful release: data saved, lock gone.
|
|
59
|
+
expect(mock:GetRaw("user_1").lock).toEqual(nil)
|
|
58
60
|
expect(mock:GetRaw("user_1").coins).toEqual(42)
|
|
59
61
|
|
|
60
62
|
-- Server B: an immediate clean takeover -- bounded well under the 5s graceful-close
|
|
@@ -102,14 +102,83 @@ function DataStoreTestUtils.setup()
|
|
|
102
102
|
}
|
|
103
103
|
end
|
|
104
104
|
|
|
105
|
+
--[=[
|
|
106
|
+
Simulates what a real Roblox server does when it shuts down, which is the only accurate model for
|
|
107
|
+
the save path: Roblox fires PlayerRemoving for every player still in the server, giving those
|
|
108
|
+
handlers the same "hold the shutdown open for me" treatment BindToClose gets. So the removals are
|
|
109
|
+
what save and close each session; the close callback's job is only to wait for them to flush.
|
|
110
|
+
|
|
111
|
+
Destroying the manager is NOT a shutdown and never has been -- nothing in a live server destroys it.
|
|
112
|
+
|
|
113
|
+
@param manager PlayerDataStoreManager
|
|
114
|
+
@param userIds { PlayerUserId }? -- players still in the server when it began closing
|
|
115
|
+
@return Promise -- what BindToCloseService yields on, so the server cannot die until it settles
|
|
116
|
+
]=]
|
|
117
|
+
function DataStoreTestUtils.promiseSimulatedShutdown(manager, userIds)
|
|
118
|
+
for _, userId in userIds or {} do
|
|
119
|
+
manager:RemovePlayerDataStore(userId)
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
return manager:PromiseAllSaves()
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
--[=[
|
|
126
|
+
Shuts down the manager a [PlayerDataStoreService] owns, the way Roblox would, and waits for it.
|
|
127
|
+
|
|
128
|
+
Call this from the `destroy()` of any spec that injects a datastore into the service and tears down
|
|
129
|
+
by destroying its ServiceBag. `manager:Destroy()` destroys no stores -- they are only ever destroyed
|
|
130
|
+
by a removal -- and a [PlayerMock] never fires the real `Players.PlayerRemoving`, so without this
|
|
131
|
+
every store the spec loaded outlives it with its `task.spawn` auto-save loop running. In the shared
|
|
132
|
+
test place that loop later fires inside another package's window and fails it.
|
|
133
|
+
|
|
134
|
+
`userIds` is only needed to model "PlayerRemoving landed first" for an ordering assertion. For
|
|
135
|
+
cleanup, omit it: the close removes every store the manager still owns.
|
|
136
|
+
|
|
137
|
+
@param playerDataStoreService PlayerDataStoreService
|
|
138
|
+
@param userIds { PlayerUserId }?
|
|
139
|
+
@param timeout number? -- defaults to 5
|
|
140
|
+
@return boolean -- false only if a shutdown was started and did not settle in time
|
|
141
|
+
]=]
|
|
142
|
+
function DataStoreTestUtils.awaitServiceShutdown(playerDataStoreService, userIds, timeout)
|
|
143
|
+
local managerPromise = playerDataStoreService:PromiseManager()
|
|
144
|
+
|
|
145
|
+
-- Nothing to shut down yet, so return rather than sit out the timeout. The manager is only
|
|
146
|
+
-- constructed inside the last link of PromiseManager's chain, so a promise still pending here means
|
|
147
|
+
-- that link never ran: no manager exists, so _createDataStore was never called and no store exists to
|
|
148
|
+
-- leak. That holds whatever the promise is waiting on -- a ServiceBag that was never started, or a
|
|
149
|
+
-- real datastore still resolving. Specs that start the bag per-test would otherwise pay the full
|
|
150
|
+
-- timeout on every teardown.
|
|
151
|
+
if managerPromise:IsPending() then
|
|
152
|
+
return true
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
return PromiseTestUtils.awaitSettled(
|
|
156
|
+
managerPromise:Then(function(manager)
|
|
157
|
+
-- Promise runs handlers without a pcall, so calling a destroyed manager would throw straight
|
|
158
|
+
-- out of the spec's teardown rather than fail it.
|
|
159
|
+
if not manager.Destroy then
|
|
160
|
+
return nil
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
return DataStoreTestUtils.promiseSimulatedShutdown(manager, userIds)
|
|
164
|
+
end),
|
|
165
|
+
timeout or 5
|
|
166
|
+
)
|
|
167
|
+
end
|
|
168
|
+
|
|
105
169
|
--[=[
|
|
106
170
|
Builds the controller the [PlayerDataStoreManager] specs share: a session-locked manager wired to
|
|
107
|
-
a fresh [DataStoreMock] (keyed `user_<userId>`), all owned by a Maid.
|
|
108
|
-
|
|
171
|
+
a fresh [DataStoreMock] (keyed `user_<userId>`), all owned by a Maid.
|
|
172
|
+
|
|
173
|
+
`destroy()` shuts the server down the way Roblox would (see
|
|
174
|
+
[DataStoreTestUtils.promiseSimulatedShutdown]) and then tears the objects down. The shutdown is not
|
|
175
|
+
optional bookkeeping: a store the spec loaded keeps its auto-save loop running until something
|
|
176
|
+
removes it, and in the shared test place that loop outlives the spec and fires inside a later
|
|
177
|
+
package's window.
|
|
109
178
|
|
|
110
179
|
Fields: `manager`, `mock`, `serviceBag`.
|
|
111
180
|
Helpers: `storeAndAwaitLock()` -> boolean -- stores a value on user 1's store and waits for the
|
|
112
|
-
session-locked load to write the lock envelope.
|
|
181
|
+
session-locked load to write the lock envelope. `promiseShutdown(userIds?)` -> Promise.
|
|
113
182
|
|
|
114
183
|
@return { manager: PlayerDataStoreManager, mock: DataStoreMock, ... }
|
|
115
184
|
]=]
|
|
@@ -123,6 +192,7 @@ function DataStoreTestUtils.setupDataStoreManager()
|
|
|
123
192
|
return "user_" .. tostring(userId)
|
|
124
193
|
end, true))
|
|
125
194
|
|
|
195
|
+
-- Returns exactly one value: specs call this straight through expect(), which rejects a second arg.
|
|
126
196
|
local function storeAndAwaitLock()
|
|
127
197
|
local dataStore = manager:GetDataStore(1)
|
|
128
198
|
dataStore:Store("coins", 5)
|
|
@@ -133,12 +203,18 @@ function DataStoreTestUtils.setupDataStoreManager()
|
|
|
133
203
|
end, 10)
|
|
134
204
|
end
|
|
135
205
|
|
|
206
|
+
local function promiseShutdown(userIds)
|
|
207
|
+
return DataStoreTestUtils.promiseSimulatedShutdown(manager, userIds)
|
|
208
|
+
end
|
|
209
|
+
|
|
136
210
|
return {
|
|
137
211
|
manager = manager,
|
|
138
212
|
mock = mock,
|
|
139
213
|
serviceBag = serviceBag,
|
|
140
214
|
storeAndAwaitLock = storeAndAwaitLock,
|
|
215
|
+
promiseShutdown = promiseShutdown,
|
|
141
216
|
destroy = function()
|
|
217
|
+
PromiseTestUtils.awaitSettled(promiseShutdown(), 5)
|
|
142
218
|
maid:DoCleaning()
|
|
143
219
|
end,
|
|
144
220
|
}
|
|
@@ -57,6 +57,7 @@ describe("PlayerDataStoreManager removal matrix (misbehaving removing callbacks)
|
|
|
57
57
|
error("removing callback boom")
|
|
58
58
|
end)
|
|
59
59
|
|
|
60
|
+
local dataStore = controller.manager:GetDataStore(1)
|
|
60
61
|
expect(controller.storeAndAwaitLock()).toEqual(true)
|
|
61
62
|
|
|
62
63
|
expect(function()
|
|
@@ -64,6 +65,10 @@ describe("PlayerDataStoreManager removal matrix (misbehaving removing callbacks)
|
|
|
64
65
|
end).toThrow("removing callback boom")
|
|
65
66
|
expect(controller.mock:GetRaw("user_1").lock ~= nil).toEqual(true)
|
|
66
67
|
|
|
68
|
+
-- The throw latched _removing before the store left _datastores, so no later removal can
|
|
69
|
+
-- reach it and nothing else destroys it. Kill its auto-save loop by hand, or it outlives this
|
|
70
|
+
-- spec and fires inside a later package's window.
|
|
71
|
+
dataStore:Destroy()
|
|
67
72
|
controller:destroy()
|
|
68
73
|
end)
|
|
69
74
|
|
|
@@ -130,12 +130,6 @@ function PlayerDataStoreManager.new(
|
|
|
130
130
|
self:_removePlayerDataStore(player.UserId)
|
|
131
131
|
end))
|
|
132
132
|
|
|
133
|
-
-- On teardown (e.g. a hot-reloaded ServiceBag, or unit tests) flush and destroy any datastores we
|
|
134
|
-
-- still own. See _flushAndDestroyAll.
|
|
135
|
-
self._maid:GiveTask(function()
|
|
136
|
-
self:_flushAndDestroyAll()
|
|
137
|
-
end)
|
|
138
|
-
|
|
139
133
|
if skipBindingToClose ~= true then
|
|
140
134
|
-- Route through BindToCloseService so the callback is unregistered on :Destroy()
|
|
141
135
|
-- (unlike a raw game:BindToClose, which can never be unbound and would leak on hot reload).
|
|
@@ -152,33 +146,6 @@ function PlayerDataStoreManager.new(
|
|
|
152
146
|
return self
|
|
153
147
|
end
|
|
154
148
|
|
|
155
|
-
--[=[
|
|
156
|
-
Flushes and tears down every datastore we still own. Runs on manager teardown (a hot-reloaded
|
|
157
|
-
ServiceBag, or a unit test). SaveAndCloseSession() is a best-effort synchronous write: the
|
|
158
|
-
underlying UpdateAsync request is dispatched before Destroy() cancels the promise, so a live
|
|
159
|
-
server usually honors it, but it is not guaranteed. A store whose load failed rejects, so the
|
|
160
|
-
rejection is swallowed. Stores handed off gracefully via _removePlayerDataStore have already been
|
|
161
|
-
pulled out of _datastores, so this only covers the ones nothing else cleaned up.
|
|
162
|
-
]=]
|
|
163
|
-
function PlayerDataStoreManager._flushAndDestroyAll(self: PlayerDataStoreManager): ()
|
|
164
|
-
for userId, datastore in self._datastores do
|
|
165
|
-
-- Cast past the DataStore intersection type: the solver otherwise blows up ("code too complex")
|
|
166
|
-
-- resolving :SaveAndCloseSession()/:Destroy() through it.
|
|
167
|
-
local store = datastore :: any
|
|
168
|
-
-- A failed load makes the save reject unconditionally; skip it so teardown does not
|
|
169
|
-
-- manufacture a guaranteed rejection.
|
|
170
|
-
if not store:DidLoadFail() then
|
|
171
|
-
-- Close the session, don't just save: this teardown ends the session, so it must also
|
|
172
|
-
-- release the session lock. A lock left held here reads as a live foreign session to the
|
|
173
|
-
-- next server that loads the key, which then grinds through the whole graceful-close/steal
|
|
174
|
-
-- retry ladder against a holder that no longer exists before it can load.
|
|
175
|
-
store:SaveAndCloseSession()
|
|
176
|
-
end
|
|
177
|
-
store:Destroy()
|
|
178
|
-
self._datastores[userId] = nil
|
|
179
|
-
end
|
|
180
|
-
end
|
|
181
|
-
|
|
182
149
|
--[=[
|
|
183
150
|
For if you want to disable saving in studio for faster close time!
|
|
184
151
|
]=]
|
|
@@ -315,13 +282,36 @@ end
|
|
|
315
282
|
--[=[
|
|
316
283
|
Removes all player data stores, and returns a promise that
|
|
317
284
|
resolves when all pending saves are saved.
|
|
285
|
+
|
|
286
|
+
On a closing server Roblox fires PlayerRemoving for every player, so a removal is usually already
|
|
287
|
+
in flight by the time this runs. Those removals do the real save-and-close themselves; this waits
|
|
288
|
+
for them rather than starting anything of its own.
|
|
289
|
+
|
|
318
290
|
@return Promise
|
|
319
291
|
]=]
|
|
320
292
|
function PlayerDataStoreManager.PromiseAllSaves(self: PlayerDataStoreManager): Promise.Promise<()>
|
|
321
293
|
for userId, _ in self._datastores do
|
|
322
294
|
self:_removePlayerDataStore(userId)
|
|
323
295
|
end
|
|
324
|
-
|
|
296
|
+
|
|
297
|
+
local promises: { Promise.Promise<any> } = {}
|
|
298
|
+
|
|
299
|
+
-- Wait on the removals still in flight, not just on _pendingSaves. A removal only reaches its write
|
|
300
|
+
-- after the removing callbacks and then DataStore's own saving callbacks resolve, and Saving fires at
|
|
301
|
+
-- the very end of that sync -- so _pendingSaves can be empty while a PlayerRemoving triggered moments
|
|
302
|
+
-- earlier is still working toward its UpdateAsync. Resolving on that empty set lets BindToClose
|
|
303
|
+
-- return and the server die mid-write, leaving the session locked with nobody able to release it: a
|
|
304
|
+
-- lock belongs to the server that holds it, so the next server can only recover by grinding the
|
|
305
|
+
-- graceful-close handshake against a dead JobId and then stealing it.
|
|
306
|
+
for _, removalPromise in self._removingPromises do
|
|
307
|
+
table.insert(promises, removalPromise :: any)
|
|
308
|
+
end
|
|
309
|
+
|
|
310
|
+
for _, savePromise in self._pendingSaves:GetAll() do
|
|
311
|
+
table.insert(promises, savePromise :: any)
|
|
312
|
+
end
|
|
313
|
+
|
|
314
|
+
return self._maid:GivePromise(PromiseUtils.all(promises))
|
|
325
315
|
end
|
|
326
316
|
|
|
327
317
|
function PlayerDataStoreManager._createDataStore(
|
|
@@ -8,6 +8,7 @@ local DataStoreTestUtils = require("DataStoreTestUtils")
|
|
|
8
8
|
local Jest = require("Jest")
|
|
9
9
|
local PlayerMock = require("PlayerMock")
|
|
10
10
|
local PromiseTestUtils = require("PromiseTestUtils")
|
|
11
|
+
local PromiseUtils = require("PromiseUtils")
|
|
11
12
|
|
|
12
13
|
local describe = Jest.Globals.describe
|
|
13
14
|
local expect = Jest.Globals.expect
|
|
@@ -189,94 +190,115 @@ describe("PlayerDataStoreManager.PromiseAllSaves", function()
|
|
|
189
190
|
end)
|
|
190
191
|
end)
|
|
191
192
|
|
|
192
|
-
|
|
193
|
-
|
|
193
|
+
-- Models a closing server the way Roblox actually behaves: PlayerRemoving fires for everyone still in
|
|
194
|
+
-- the server and does the save-and-close, and the close is held open until those removals flush.
|
|
195
|
+
describe("PlayerDataStoreManager server shutdown", function()
|
|
196
|
+
it("saves the staged data and releases the lock for the leaving player", function()
|
|
194
197
|
local controller = DataStoreTestUtils.setupDataStoreManager()
|
|
195
198
|
|
|
196
|
-
|
|
197
|
-
|
|
199
|
+
if not controller.storeAndAwaitLock() then
|
|
200
|
+
expect("lock was never acquired").toEqual("lock was acquired")
|
|
198
201
|
controller:destroy()
|
|
199
202
|
return
|
|
200
203
|
end
|
|
201
204
|
|
|
202
|
-
controller.
|
|
205
|
+
if not expectSettled(controller.promiseShutdown({ 1 }), 10) then
|
|
206
|
+
controller:destroy()
|
|
207
|
+
return
|
|
208
|
+
end
|
|
203
209
|
|
|
204
|
-
|
|
210
|
+
local raw = controller.mock:GetRaw("user_1")
|
|
211
|
+
expect(raw.coins).toEqual(5)
|
|
212
|
+
expect(raw.lock).toEqual(nil)
|
|
205
213
|
|
|
206
214
|
controller:destroy()
|
|
207
215
|
end)
|
|
208
216
|
|
|
209
|
-
it("
|
|
217
|
+
it("releases the lock for every player still in the server", function()
|
|
210
218
|
local controller = DataStoreTestUtils.setupDataStoreManager()
|
|
211
219
|
|
|
212
|
-
|
|
213
|
-
|
|
220
|
+
controller.manager:GetDataStore(1):Store("coins", 1)
|
|
221
|
+
controller.manager:GetDataStore(2):Store("coins", 2)
|
|
222
|
+
|
|
223
|
+
local locked = PromiseTestUtils.awaitValue(function()
|
|
224
|
+
local rawOne = controller.mock:GetRaw("user_1")
|
|
225
|
+
local rawTwo = controller.mock:GetRaw("user_2")
|
|
226
|
+
return rawOne ~= nil and rawOne.lock ~= nil and rawTwo ~= nil and rawTwo.lock ~= nil
|
|
227
|
+
end, 10)
|
|
228
|
+
if not locked then
|
|
229
|
+
expect("both locks were never acquired").toEqual("both locks were acquired")
|
|
214
230
|
controller:destroy()
|
|
215
231
|
return
|
|
216
232
|
end
|
|
217
233
|
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
234
|
+
if not expectSettled(controller.promiseShutdown({ 1, 2 }), 10) then
|
|
235
|
+
controller:destroy()
|
|
236
|
+
return
|
|
237
|
+
end
|
|
221
238
|
|
|
222
|
-
|
|
223
|
-
expect(
|
|
224
|
-
expect(
|
|
239
|
+
expect(controller.mock:GetRaw("user_1").lock).toEqual(nil)
|
|
240
|
+
expect(controller.mock:GetRaw("user_2").lock).toEqual(nil)
|
|
241
|
+
expect(controller.mock:GetRaw("user_1").coins).toEqual(1)
|
|
242
|
+
expect(controller.mock:GetRaw("user_2").coins).toEqual(2)
|
|
225
243
|
|
|
226
244
|
controller:destroy()
|
|
227
245
|
end)
|
|
228
246
|
|
|
229
|
-
it("
|
|
247
|
+
it("destroys each store once its removal has flushed", function()
|
|
230
248
|
local controller = DataStoreTestUtils.setupDataStoreManager()
|
|
231
249
|
|
|
232
|
-
|
|
233
|
-
|
|
250
|
+
local dataStore = controller.manager:GetDataStore(1)
|
|
251
|
+
if not expectSettled(dataStore:PromiseLoadSuccessful(), 10) then
|
|
234
252
|
controller:destroy()
|
|
235
253
|
return
|
|
236
254
|
end
|
|
237
255
|
|
|
238
|
-
controller.
|
|
256
|
+
if not expectSettled(controller.promiseShutdown({ 1 }), 10) then
|
|
257
|
+
controller:destroy()
|
|
258
|
+
return
|
|
259
|
+
end
|
|
239
260
|
|
|
240
|
-
expect(
|
|
241
|
-
local raw = controller.mock:GetRaw("user_1")
|
|
242
|
-
return raw ~= nil and raw.lock == nil
|
|
243
|
-
end, 5)).toEqual(true)
|
|
244
|
-
expect(controller.mock:GetRaw("user_1").coins).toEqual(5)
|
|
261
|
+
expect(getmetatable(dataStore)).toBeNil()
|
|
245
262
|
|
|
246
263
|
controller:destroy()
|
|
247
264
|
end)
|
|
248
265
|
|
|
249
|
-
it("
|
|
266
|
+
it("does not resolve the close while a PlayerRemoving save is still in flight", function()
|
|
250
267
|
local controller = DataStoreTestUtils.setupDataStoreManager()
|
|
251
268
|
|
|
252
|
-
|
|
253
|
-
|
|
269
|
+
-- An async removing callback pushes the write past the moment the close is requested, which is
|
|
270
|
+
-- exactly the window where waiting on pending saves alone finds nothing to wait for.
|
|
271
|
+
controller.manager:AddRemovingCallback(function()
|
|
272
|
+
return PromiseUtils.delayed(0.5)
|
|
273
|
+
end)
|
|
254
274
|
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
local rawTwo = controller.mock:GetRaw("user_2")
|
|
258
|
-
return rawOne ~= nil and rawOne.lock ~= nil and rawTwo ~= nil and rawTwo.lock ~= nil
|
|
259
|
-
end, 10)
|
|
260
|
-
if not locked then
|
|
261
|
-
expect("both locks were never acquired").toEqual("both locks were acquired")
|
|
275
|
+
if not controller.storeAndAwaitLock() then
|
|
276
|
+
expect("lock was never acquired").toEqual("lock was acquired")
|
|
262
277
|
controller:destroy()
|
|
263
278
|
return
|
|
264
279
|
end
|
|
265
280
|
|
|
266
|
-
|
|
281
|
+
-- PlayerRemoving lands first, then the server begins closing.
|
|
282
|
+
controller.manager:RemovePlayerDataStore(1)
|
|
267
283
|
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
284
|
+
local closePromise = controller.manager:PromiseAllSaves()
|
|
285
|
+
expect(closePromise:IsPending()).toEqual(true)
|
|
286
|
+
|
|
287
|
+
if not expectSettled(closePromise, 10) then
|
|
288
|
+
controller:destroy()
|
|
289
|
+
return
|
|
290
|
+
end
|
|
291
|
+
|
|
292
|
+
-- The close resolving has to mean the write landed. If it can resolve first, the real server
|
|
293
|
+
-- dies here with the session still locked and no other server able to release it.
|
|
294
|
+
local raw = controller.mock:GetRaw("user_1")
|
|
295
|
+
expect(raw.coins).toEqual(5)
|
|
296
|
+
expect(raw.lock).toEqual(nil)
|
|
275
297
|
|
|
276
298
|
controller:destroy()
|
|
277
299
|
end)
|
|
278
300
|
|
|
279
|
-
it("
|
|
301
|
+
it("closes cleanly when a store's load failed, leaking no rejection and writing no lock", function()
|
|
280
302
|
local controller = DataStoreTestUtils.setupDataStoreManager()
|
|
281
303
|
|
|
282
304
|
controller.mock:FailAllRequests()
|
|
@@ -290,7 +312,10 @@ describe("PlayerDataStoreManager teardown", function()
|
|
|
290
312
|
end
|
|
291
313
|
expect((loaded:Wait())).toEqual(false)
|
|
292
314
|
|
|
293
|
-
controller.
|
|
315
|
+
if not expectSettled(controller.promiseShutdown({ 1 }), 10) then
|
|
316
|
+
controller:destroy()
|
|
317
|
+
return
|
|
318
|
+
end
|
|
294
319
|
|
|
295
320
|
expect(controller.mock:GetRaw("user_1")).toBeNil()
|
|
296
321
|
|
|
@@ -5,8 +5,10 @@
|
|
|
5
5
|
local require = require(script.Parent.loader).load(script)
|
|
6
6
|
|
|
7
7
|
local DataStoreMock = require("DataStoreMock")
|
|
8
|
+
local DataStoreTestUtils = require("DataStoreTestUtils")
|
|
8
9
|
local Jest = require("Jest")
|
|
9
10
|
local Maid = require("Maid")
|
|
11
|
+
local Promise = require("Promise")
|
|
10
12
|
local PromiseTestUtils = require("PromiseTestUtils")
|
|
11
13
|
local ServiceBag = require("ServiceBag")
|
|
12
14
|
|
|
@@ -26,10 +28,26 @@ local function setup(mock)
|
|
|
26
28
|
serviceBag:Start()
|
|
27
29
|
end
|
|
28
30
|
|
|
31
|
+
-- Drives the real close path: the manager the service owns, shut down the way Roblox does. Without a
|
|
32
|
+
-- mock the bag was never started, so there is no manager to reach and nothing to shut down.
|
|
33
|
+
local function promiseShutdown(userIds)
|
|
34
|
+
if not mock then
|
|
35
|
+
return Promise.resolved()
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
return service:PromiseManager():Then(function(manager)
|
|
39
|
+
return DataStoreTestUtils.promiseSimulatedShutdown(manager, userIds)
|
|
40
|
+
end)
|
|
41
|
+
end
|
|
42
|
+
|
|
29
43
|
return {
|
|
30
44
|
service = service,
|
|
31
45
|
mock = mock,
|
|
46
|
+
promiseShutdown = promiseShutdown,
|
|
32
47
|
destroy = function()
|
|
48
|
+
-- Otherwise a store the spec loaded outlives it with its auto-save loop running, and fires
|
|
49
|
+
-- inside a later package's window in the shared test place.
|
|
50
|
+
PromiseTestUtils.awaitSettled(promiseShutdown(), 5)
|
|
33
51
|
maid:DoCleaning()
|
|
34
52
|
end,
|
|
35
53
|
}
|
|
@@ -157,8 +175,10 @@ describe("PlayerDataStoreService failure handling", function()
|
|
|
157
175
|
end)
|
|
158
176
|
end)
|
|
159
177
|
|
|
160
|
-
|
|
161
|
-
|
|
178
|
+
-- The service registers manager:PromiseAllSaves() as its BindToClose callback, so a closing server is
|
|
179
|
+
-- PlayerRemoving doing the save-and-close with that callback held open until it flushes.
|
|
180
|
+
describe("PlayerDataStoreService server shutdown", function()
|
|
181
|
+
it("saves the staged data and destroys the store when the server closes", function()
|
|
162
182
|
local controller = setup(DataStoreMock.new())
|
|
163
183
|
|
|
164
184
|
local promise = controller.service:PromiseDataStore(1)
|
|
@@ -175,34 +195,19 @@ describe("PlayerDataStoreService teardown", function()
|
|
|
175
195
|
return
|
|
176
196
|
end
|
|
177
197
|
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
expect(getmetatable(dataStore)).toBeNil()
|
|
181
|
-
end)
|
|
182
|
-
|
|
183
|
-
it("flushes staged data synchronously to the underlying store when destroyed", function()
|
|
184
|
-
local controller = setup(DataStoreMock.new())
|
|
185
|
-
|
|
186
|
-
local promise = controller.service:PromiseDataStore(1)
|
|
187
|
-
if not PromiseTestUtils.awaitSettled(promise, 10) then
|
|
188
|
-
expect("hung").toEqual("settled")
|
|
189
|
-
controller:destroy()
|
|
190
|
-
return
|
|
191
|
-
end
|
|
192
|
-
local _ok, dataStore = promise:Yield()
|
|
198
|
+
dataStore:Store("coins", 7)
|
|
193
199
|
|
|
194
|
-
if not PromiseTestUtils.awaitSettled(
|
|
195
|
-
expect("
|
|
200
|
+
if not PromiseTestUtils.awaitSettled(controller.promiseShutdown({ 1 }), 10) then
|
|
201
|
+
expect("shutdown never flushed").toEqual("shutdown flushed")
|
|
196
202
|
controller:destroy()
|
|
197
203
|
return
|
|
198
204
|
end
|
|
199
205
|
|
|
200
|
-
dataStore:Store("coins", 7)
|
|
201
|
-
|
|
202
|
-
controller:destroy()
|
|
203
|
-
|
|
204
206
|
local raw = controller.mock:GetRaw("1")
|
|
205
207
|
expect(raw).never.toBeNil()
|
|
206
208
|
expect(raw.coins).toEqual(7)
|
|
209
|
+
expect(getmetatable(dataStore)).toBeNil()
|
|
210
|
+
|
|
211
|
+
controller:destroy()
|
|
207
212
|
end)
|
|
208
213
|
end)
|