noblog 0.0.2 → 0.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/dist/cmds/generate.js +3 -3
  2. package/dist/cmds/generate.js.map +1 -1
  3. package/dist/index.js +24 -10
  4. package/dist/index.js.map +1 -1
  5. package/dist/lib/nomd/nomd.d.ts.map +1 -1
  6. package/dist/lib/nomd/nomd.js +16 -1
  7. package/dist/lib/nomd/nomd.js.map +1 -1
  8. package/dist/lib/notion/client.d.ts +1 -0
  9. package/dist/lib/notion/client.d.ts.map +1 -1
  10. package/dist/lib/notion/client.js +16 -0
  11. package/dist/lib/notion/client.js.map +1 -1
  12. package/package.json +44 -28
  13. package/output/00097769-2aac-4b6f-8f37-d9b658d46b2e +0 -435
  14. package/output/1707063b-7c0a-4572-9526-f7db2f7bbc49 +0 -162
  15. package/output/1ed588e1-2604-4e33-9685-722b03f66374 +0 -89
  16. package/output/2b1c0d09-40f7-4151-a053-11391f775ece +0 -33
  17. package/output/336b6a5a-f383-4d6c-8fe5-54c9a918c4c0 +0 -264
  18. package/output/37b77a0a-e233-405a-93f5-a60a808eee35 +0 -72
  19. package/output/3ac660cb-9503-4e88-8939-5aee30c98351 +0 -24
  20. package/output/5bc1b19a-8350-48b3-aa19-8073561ad45b +0 -109
  21. package/output/765976b5-7790-497b-8634-4be1d4770e16 +0 -111
  22. package/output/90f2cb44-90ab-40b7-a736-02bfb83a314c +0 -1167
  23. package/output/99a7b0c5-f9de-403c-95ec-eba3d700a195 +0 -49
  24. package/output/c844b66d-0bf3-4f89-bbe1-5974f2de454b +0 -193
  25. package/output/cd6299f7-54bb-4160-8735-16e629422a57 +0 -193
  26. package/output/d462cfca-0a39-4b65-8430-4eae35edbb76 +0 -27
  27. package/output/daafaad2-2570-43da-8e9e-f95a1b29ca17 +0 -12
  28. package/output/e45ef6bf-6c30-4fb2-9efc-a1984e5dc530 +0 -84
  29. package/output/sub/05009ca9-cb39-4356-8f24-c0db39cb4661 +0 -64
  30. package/output/sub/0b903359-2337-4ad9-adef-e77b69142581 +0 -328
  31. package/output/sub/18f0fae1-b7ca-40ea-b725-4e69404429dc +0 -109
  32. package/output/sub/232ae8ce-5ec3-4737-a18e-077774b84b1e +0 -940
  33. package/output/sub/34f4d4af-812e-4e56-a6dd-deb96e7f605c +0 -3
  34. package/output/sub/44331753-a264-40a4-b86f-eafd676b3da0 +0 -509
  35. package/output/sub/4bac355b-24f0-49e7-a317-e4f1cad63c79 +0 -65
  36. package/output/sub/4d6663a2-e93a-402e-89f8-a47cccc0c8cf +0 -76
  37. package/output/sub/5771c956-25bf-4714-bc2e-65959c9156d9 +0 -52
  38. package/output/sub/6ff91a18-494e-4ffc-a33b-b9fd2c36f43f +0 -42
  39. package/output/sub/81e4a2b0-05e2-4013-bd41-14b144979d91 +0 -0
  40. package/output/sub/99ef1497-0b94-490f-b9a0-f54b0e64194d +0 -6
  41. package/output/sub/9a33b8cc-101a-4f6a-9852-115595cd166b +0 -172
  42. package/output/sub/9c060d29-39a2-432b-b06b-b0bd91989a52 +0 -1159
  43. package/output/sub/a45b383c-d894-4480-8322-9c5d028722e3 +0 -9
  44. package/output/sub/b875bfe4-3c1e-4a8f-a56c-27a48ab874ca +0 -438
  45. package/output/sub/b8cbb8f9-f702-4a43-943c-02a6e39683d5 +0 -222
  46. package/output/sub/e339a686-bbb1-41d9-b6b5-18111e72a68c +0 -1392
  47. package/output/sub/e7931a8e-45ca-444a-8d6f-e6fed9415c26 +0 -110
  48. package/output/sub/e859552b-c597-49ed-9ede-fab320fafd87 +0 -150
  49. package/output/sub/f4237e72-fdce-44a4-b62c-659dce24b1ee +0 -107
  50. package/output/sub/ff7ea97c-55ac-471c-902f-b7a7bd101ee6 +0 -0
  51. package/src/cmds/generate.ts +0 -53
  52. package/src/index.ts +0 -22
  53. package/src/lib/nomd/cvter.ts +0 -143
  54. package/src/lib/nomd/nomd.ts +0 -88
  55. package/src/lib/nosm/default.ts +0 -0
  56. package/src/lib/nosm/nosm.ts +0 -3
  57. package/src/lib/notion/client.ts +0 -59
  58. package/src/lib/notion/object.ts +0 -176
  59. package/src/lib/notion/request.ts +0 -42
  60. package/tsconfig.json +0 -35
package/output/90f2cb44-90ab-40b7-a736-02bfb83a314c DELETED
@@ -1,1167 +0,0 @@
1
-
2
- > 😇 All I need to know about 6.5840
3
- > Full code on:
4
- > [https://github.com/humbornjo/MIT-6.5840](https://github.com/humbornjo/MIT-6.5840)
5
-
6
-
7
- # Lab1
8
-
9
-
10
- > 💡 虽然本次实验没有触及，但是GFS才是Map Reduce的灵魂。这也是很多MR方法局限在单机运行的根本原因。在分布式的场景中，intermediate的交换才是事实上的重点。
11
-
12
-
13
- ## Difficulties & solution
14
-
15
- <details>
16
- <summary>coordinator 作为调度者，就算其元素都是线程安全的，依然是需要加一个整体的锁的。（仅针对我的设计与实现）</summary>
17
-
18
- ```go
19
- type Coordinator struct {
20
- // Your definitions here.
21
- nMap int
22
- nReduce int
23
- stage int
24
- currTask chan Task
25
- runnTask map[Task]chan int
26
- lock sync.RWMutex
27
- }
28
- ```
29
-
30
-
31
- 在我的实现中，当currTask和runnTask全为空时方可触发stage的转换（比如从MAP转移到REDUCE）。如果分别加锁，就可能导致 →
32
-
33
- 1. 检查currTask时为空
34
- 2. runnTask超时，删除任务，未完成的任务重新入队currTask
35
- 3. 检查runnTask，也为空
36
- 4. 状态转换了，MR失败
37
-
38
- </details>
39
-
40
- <details>
41
- <summary>rpc的设计与细节</summary>
42
-
43
- ```go
44
- /*
45
- type Task struct {
46
- Tid int
47
- Fname string
48
- }
49
- */
50
-
51
- type AskForTaskArgs struct {
52
- Wid int
53
- }
54
-
55
- type AskForTaskReply struct {
56
- Stage int
57
- Task Task
58
- Nout int
59
- }
60
-
61
- type ReportForTaskArgs struct {
62
- Task Task
63
- }
64
-
65
- type ReportForTaskReply struct {
66
- Foo bool
67
- }
68
- ```
69
-
70
-
71
- 总体分为Ask和Report两种，主要的问题在于Report RPC前后临时文件的保存和状态的转变，下面进行分类分析。
72
-
73
- 1. MAP阶段的Report,是在RPC获得true的回复后进行保存临时文件，还是先保存临时文件，再call RPC。
74
- 1. 如果在获得RPC回复后再保存。如果RPC结果为true（在我的实现中，是`reply.Foo == true`）。会出现一个恐怖的事，那就是coordinator已经转换状态了。开始分发REDUCE的任务了，但是MAP阶段的intermediate文件还没有保存完。这就会导致，如果有一个Worker此时恰好接了任务，并竞争运行了，然而目标文件甚至不存在，最终REDUCE的任务结果出错。如果RPC结果为false，影响不大，就算保存了也会被另一个Worker的文件写入替换。
75
- 2. 如果在获得RPC回复前再保存。如果RPC结果为true，不会有任何问题，解决了上方的问题。如果RPC结果为false，影响不大，就算保存了也会被另一个Worker的文件写入替换。
76
- 3. 综上，选择在RPC回复前保存。
77
- 2. Reduce阶段的Report，没啥好说的，保存必然是在RPC之前的，因为实现borrow了mrsequential的代码，目前来看，换成在RPC之后阻塞地保存，唯一的风险就是Worker的临时宕机。（最后不要忘了删除临时文件）
78
-
79
- ```go
80
- if !ReportForTask(
81
- &ReportForTaskArgs{Task: reply.Task},
82
- &ReportForTaskReply{}) {
83
- os.Remove(oname)
84
- } else {
85
- for _, filename := range filenames {
86
- os.Remove(filename)
87
- }
88
- }
89
- ```
90
-
91
-
92
- </details>
93
-
94
- - [x] PASS LOG
95
-
96
- ```bash
97
- MIT-6.5840/src/main via 🐹 v1.20.6
98
- > bash test-mr.sh
99
- *** Starting wc test.
100
- --- wc test: PASS
101
- *** Starting indexer test.
102
- --- indexer test: PASS
103
- *** Starting map parallelism test.
104
- --- map parallelism test: PASS
105
- *** Starting reduce parallelism test.
106
- --- reduce parallelism test: PASS
107
- *** Starting job count test.
108
- --- job count test: PASS
109
- *** Starting early exit test.
110
- --- early exit test: PASS
111
- *** Starting crash test.
112
- --- crash test: PASS
113
- *** PASSED ALL TESTS
114
- ```
115
-
116
-
117
- # Lab2
118
-
119
-
120
- > 💡 What u deserve to understand is that, even raft can make mistake. When the delay is high and command flood in.
121
-
122
-
123
- ## Core Idea
124
-
125
- 1. Raft RPCS is idempotent
126
- 2. Randomization causes simplicity
127
- 3. Leader takes it all
128
- 4. Logical timing instead of physical timing
129
-
130
- $$
131
- broadcastTime ≪ electionTimeout ≪ MTBF
132
- $$
133
-
134
-
135
- ## Mind Trail
136
-
137
-
138
- [lab2 Raft](e859552b-c597-49ed-9ede-fab320fafd87.md)
139
-
140
-
141
- ## Difficulties & solution
142
-
143
-
144
- ### 2A
145
-
146
- - [x] Term严格小的才能给Term大的server投票，但是容易出现以下情况 (忽略第三列)，即server S1和S2几乎以同一时间开始选举 (即使存在随机timeout)，导致多轮都无法选出leader。
147
- <details>
148
- <summary>Debug log</summary>
149
-
150
- ```go
151
- S0 C0 ask for vote, T: 8
152
- S0 not leader, election timeout...
153
- S0 convert to candidate, calling election T: 7
154
- S2 C0 ask for vote, T: 8
155
- S1 C0 ask for vote, T: 8
156
- S0 not leader, election timeout...
157
- S0 convert to candidate, calling election T: 8
158
- S2 C0 ask for vote, T: 9
159
- S1 not leader, election timeout...
160
- S1 convert to candidate, calling election T: 8
161
- S2 C0 ask for vote, T: 9
162
- S1 C0 ask for vote, T: 9
163
- S0 C0 ask for vote, T: 9
164
- S0 not leader, election timeout...
165
- S0 convert to candidate, calling election T: 9
166
- S2 C0 ask for vote, T: 10
167
- S1 C0 ask for vote, T: 10
168
- S1 not leader, election timeout...
169
- S1 convert to candidate, calling election T: 9
170
- S2 C0 ask for vote, T: 10
171
- S0 C0 ask for vote, T: 10
172
- S0 -> S1 vote false, VT: 0
173
- S0 not leader, election timeout...
174
- S0 convert to candidate, calling election T: 10
175
- ```
176
-
177
-
178
- </details>
179
-
180
- <details>
181
- <summary>排查出的问题</summary>
182
- 1. 在RequestVote RPCs中就更新了自己的Term，使得落后的server一下子就追上了因random而领先的server
183
- 2. timeout区间为 [150, 300] ms
184
-
185
- </details>
186
-
187
- <details>
188
- <summary>可能的解决办法</summary>
189
- 1. ~~不在RequestVote RPCs中更新Term。~~
190
- 在更新以后把isTimeout置为false，跳过一轮选举自增。
191
- 2. 增大timeout的区间
192
-
193
- </details>
194
-
195
- <details>
196
- <summary>方案</summary>
197
-
198
- 采用了方法1，并将timeout区间修改为 [200, 500] ms，暂时解决问题。
199
-
200
-
201
- </details>
202
-
203
-
204
- ---
205
-
206
- - [x] 观察第二列和最后一列，可以发现这里 S1 和 S6 出现了脑裂现象，但是并未得到解决。原因是两方此时的Term相同，两方互相会拒绝对方的heartbeat log而不会进行状态转换。
207
- <details>
208
- <summary>Debug log</summary>
209
-
210
- ```go
211
- S1 multi leaders(2)
212
- T: 7
213
- S6 multi leaders(2) T:
214
- 7
215
- S6 -> S5 T: 7, send
216
- {PLI: 0, PLT: 0}
217
- S1 -> S6 T: 7, send
218
- {PLI: 0, PLT: 0}
219
- S2 convert to
220
- candidate, calling
221
- election T: 7
222
- S6 C0 ask for vote, T:
223
- 8
224
- S6 -> S0 T: 7, send
225
- {PLI: 0, PLT: 0}
226
- S6 -> S1 T: 7, send
227
- {PLI: 0, PLT: 0}
228
- S6 -> S2 T: 7, send
229
- {PLI: 0, PLT: 0}
230
- S6 -> S3 T: 7, send
231
- {PLI: 0, PLT: 0}
232
- S6 -> S4 T: 7, send
233
- {PLI: 0, PLT: 0}
234
- S1 -> S0 T: 7, send
235
- {PLI: 0, PLT: 0}
236
- S1 -> S2 T: 7, send
237
- {PLI: 0, PLT: 0}
238
- S1 -> S3 T: 7, send
239
- {PLI: 0, PLT: 0}
240
- S1 -> S4 T: 7, send
241
- {PLI: 0, PLT: 0}
242
- S1 -> S5 T: 7, send
243
- {PLI: 0, PLT: 0}
244
- ```
245
-
246
-
247
- </details>
248
-
249
- <details>
250
- <summary>排查出的问题</summary>
251
- 1. Term相同，在身为leader的同时对峙，由于拒绝了heartbeat log导致的状态无法转换
252
-
253
- </details>
254
-
255
- <details>
256
- <summary>可能的解决方法</summary>
257
- 1. 本质上是在disconnect的时候没有检查自己的killed()，及时转换为follower。论文中没有提到Term相同的情况下Entry (Including heartbeat)的处理情况。
258
-
259
- </details>
260
-
261
- <details>
262
- <summary>方案</summary>
263
- 1. 在每个heartbeat loop中检查killed()，及时转换状态。
264
- 2. 运行时又发现了一个小问题，下面分析出问题的代码。AppendEntry分为三个情况，currTerm < args.Term, currTerm = args.Term 和 currTerm > args.Term。**第一种情况**没啥好说的，**第二种情况下**，若server还未投票 (voteFor == -1) 则server一定不是Leader；投了票的情况下拒绝可以保证一个Term只投一次票的原则。第三种情况下，需要注意，server的状态可能是三个状态中的任意一种，所以一定要重置server的状态为Follower来保证Term大的优先的原则。
265
-
266
- ```go
267
- if rf.currTerm > args.Term {
268
- reply.VoteGranted = false
269
- DebugLog(dVote, "S%d -> S%d vote false, T: %d, CT: %d\n", rf.me, args.CandidateId, rf.currTerm, args.Term)
270
- return
271
- } else if rf.currTerm == args.Term {
272
- if rf.votedFor == -1 {
273
- rf.votedFor = args.CandidateId
274
- reply.VoteGranted = true
275
- DebugLog(dVote, "S%d -> S%d vote true, same term\n", rf.me, args.CandidateId)
276
- return
277
- } else {
278
- reply.VoteGranted = false
279
- DebugLog(dVote, "S%d -> S%d vote false, VT: %d\n", rf.me, args.CandidateId, rf.votedFor)
280
- return
281
- }
282
- } else {
283
- rf.currTerm = args.Term
284
- rf.isTimeout = false
285
- rf.state = Follower // !!!!!!!!!!!!!! IMPORTANT !!!!!!!!!!!!!!
286
- rf.votedFor = args.CandidateId
287
- reply.VoteGranted = true
288
- DebugLog(dVote, "S%d -> S%d vote true, from T: %d to CT: %d\n", rf.me, args.CandidateId, reply.Term, rf.currTerm)
289
- return
290
- }
291
- ```
292
-
293
-
294
- </details>
295
-
296
-
297
- ---
298
-
299
- - [x] PASS LOG
300
-
301
- ```go
302
- humborn@shin MIT-6.5840/src/raft on master [!] [!] 󱑆 09:36:06
303
- ➜ go test -run 2A
304
- Test (2A): initial election ...
305
- ... Passed -- 3.0 3 58 15440 0
306
- Test (2A): election after network failure ...
307
- ... Passed -- 4.4 3 140 27904 0
308
- Test (2A): multiple elections ...
309
- ... Passed -- 6.2 7 686 129664 0
310
- PASS
311
- ok 6.824/raft 13.599s
312
- ```
313
-
314
-
315
- ---
316
-
317
-
318
-
319
-
320
-
321
- ### 2B
322
-
323
- - [x] Server在成功append以后，竟然没有让Leader更新commitIndex
324
- <details>
325
- <summary>Debug log</summary>
326
-
327
- ```go
328
- S0 not leader, election timeout...
329
- S1 -> S2 T: 1, send {PLI: 0, PLT: 0,
330
- CI: 0}
331
- S1 -> S0 T: 1, send {PLI: 0, PLT: 0,
332
- CI: 0}
333
- S0 -> S1 append ok T: 1, LENLOG: 2
334
- S2 -> S1 append ok T: 1, LENLOG: 2
335
- S2 not leader, election timeout...
336
- S1 -> S2 T: 1, send {PLI: 1, PLT: 1,
337
- CI: 0}
338
- S1 -> S0 T: 1, send {PLI: 1, PLT: 1,
339
- CI: 0}
340
- S0 -> S1 append ok T: 1, LENLOG: 2
341
- S2 -> S1 append ok T: 1, LENLOG: 2
342
- S1 -> S2 T: 1, send {PLI: 0, PLT: 0,
343
- CI: 0}
344
- S1 -> S0 T: 1, send {PLI: 0, PLT: 0,
345
- CI: 0}
346
- S0 -> S1 append ok T: 1, LENLOG: 2
347
- S2 -> S1 append ok T: 1, LENLOG: 2
348
- FAIL
349
- FAIL 6.824/raft 2.313s
350
- ```
351
-
352
-
353
- </details>
354
-
355
- <details>
356
- <summary>排查出的问题</summary>
357
- 1. sort中索引有问题，原来是从小到大排序后，取len(rf.peers)/2-1，当#Server为3时，结果为0,很显然不对。
358
- 2. 搞错索引了，AppendEntries时可能会传入空Entries。这时想普适地满足AppendEntries RPCs的最后一条实现规则，就需要取server自身logEntries的最后一条entry的index
359
- 3. 创建新logEntry时index和term位置反了，cao!
360
-
361
- </details>
362
-
363
- <details>
364
- <summary>方案</summary>
365
- 1. 改为(len(rf.peers)-1)/2
366
- 2. 由args.Entries→rf.logEntries取最后一个entry的index
367
- 3. cao！
368
-
369
- </details>
370
-
371
- <details>
372
- <summary>Task log</summary>
373
-
374
- ```go
375
- ➜ go test -run BasicAgree
376
- Test (2B): basic agreement ...
377
- ... Passed -- 0.5 3 16 4258 3
378
- PASS
379
- ok 6.824/raft 0.494s
380
- ```
381
-
382
-
383
- </details>
384
-
385
- - [x] 如果一个server掉线了，会不断`StartElection()`自增，当它重联时，它的Term将会很大。就会导致一个情况，当前 leader 的 AppendEntries RPCs 将会被拒绝，reply中的Term巨大，会将Leader打回Follower并更新Term,最好给Leader特权，让Leader尽快timeout,再次成为Leader；由它召开的Election也会被其他server拒绝(LastLogIndex很小)并使其他server更新Term，不用给特权通道。但是在下方log中
386
- _S0 apply msg: {true 101 1 false [] 0 0}_
387
- 重复出现，且重联后，并未一直尝试Append(在第一次被拒绝后)导致没有及时更新。
388
- <details>
389
- <summary>Debug log</summary>
390
-
391
- ```go
392
- S0 reconnect...
393
- S2 recv command, LENLOG: 6, T: 1
394
- S2 send to S1, ENTRIES: [{106 1 6}]
395
- S2 send to S0, ENTRIES: [{102 1 2} {103 1 3} {104 1
396
- 4} {105 1 5} {106 1 6}]
397
- S1 recv append from S2, T: 1, ENTRIES: [{106 1 6}]
398
- S1 -> S2 append ok T: 1, LENLOG: 7
399
- S0 recv append from S2, T: 1, ENTRIES: [{102 1 2}
400
- {103 1 3} {104 1 4} {105 1 5} {106 1 6}]
401
- S0 append fail, T: 6 large term
402
- S2 -> S1 T: 1, send {PLI: 5, PLT: 1, CI: 5,
403
- BEGINLOGIDX: 6, ENDLOGIDX: 6, LOG: [{106 1 6}]}
404
- S2 update commitIndex CI: 6
405
- S2 apply msg: {true 106 6 false [] 0 0}
406
- S2 -> S0 T: 1, send {PLI: 1, PLT: 1, CI: 5,
407
- BEGINLOGIDX: 2, ENDLOGIDX: 6, LOG: [{102 1 2} {103 1
408
- 3} {104 1 4} {105 1 5} {106 1 6}]}
409
- S2 convert to candidate, calling election T: 6
410
- S1 C6 ask for vote, T: 7
411
- S0 C6 ask for vote, T: 7
412
- S1 -> S2 vote true, from T: 1 to CT: 7
413
- S0 -> S2 vote true, from T: 6 to CT: 7
414
- S2 convert to leader at T: 7
415
- S1 recv append from S2, T: 7, ENTRIES: []
416
- S1 -> S2 recv heartbeat
417
- S1 apply msg: {true 106 6 false [] 0 0}
418
- S2 -> S1 T: 7, send heartbeat
419
- S0 recv append from S2, T: 7, ENTRIES: []
420
- S0 append fail, diff term
421
- S2 -> S0 T: 7, send heartbeat
422
- S2 -> S0 update failed, NEXT: 6
423
- S1 recv append from S2, T: 7, ENTRIES: []
424
- S1 -> S2 recv heartbeat
425
- S0 recv append from S2, T: 7, ENTRIES: []
426
- S0 append fail, diff term
427
- S2 -> S1 T: 7, send heartbeat
428
- S2 -> S0 T: 7, send heartbeat
429
- S2 -> S0 update failed, NEXT: 5
430
- S0 recv append from S2, T: 7, ENTRIES: []
431
- S0 append fail, diff term
432
- S1 recv append from S2, T: 7, ENTRIES: []
433
- S1 -> S2 recv heartbeat
434
- S2 -> S0 T: 7, send heartbeat
435
- S2 -> S0 update failed, NEXT: 4
436
- S2 -> S1 T: 7, send heartbeat
437
- S2 -> S0 T: 7, fail send {PLI: 1, PLT: 1, CI: 4}
438
- S0 recv append from S2, T: 7, ENTRIES: []
439
- S0 append fail, diff term
440
- S1 recv append from S2, T: 7, ENTRIES: []
441
- S1 -> S2 recv heartbeat
442
- S2 -> S1 T: 7, send heartbeat
443
- S2 -> S0 T: 7, send heartbeat
444
- S2 -> S0 update failed, NEXT: 3
445
- S1 recv append from S2, T: 7, ENTRIES: []
446
- S1 -> S2 recv heartbeat
447
- S0 recv append from S2, T: 7, ENTRIES: []
448
- S0 append fail, diff term
449
- S2 -> S1 T: 7, send heartbeat
450
- S2 -> S0 T: 7, send heartbeat
451
- S2 -> S0 update failed, NEXT: 2
452
- S1 recv append from S2, T: 7, ENTRIES: []
453
- S1 -> S2 recv heartbeat
454
- S0 recv append from S2, T: 7, ENTRIES: []
455
- S0 -> S2 recv heartbeat
456
- S0 apply msg: {true 101 1 false [] 0 0}
457
- S2 -> S1 T: 7, send heartbeat
458
- S2 -> S0 T: 7, send heartbeat
459
- S2 -> S0 T: 7, fail send {PLI: 1, PLT: 1, CI: 3}
460
- S1 recv append from S2, T: 7, ENTRIES: []
461
- S1 -> S2 recv heartbeat
462
- S2 -> S1 T: 7, send heartbeat
463
- S0 recv append from S2, T: 7, ENTRIES: []
464
- S0 -> S2 recv heartbeat
465
- S2 -> S0 T: 7, send heartbeat
466
- ```
467
-
468
-
469
- </details>
470
-
471
- <details>
472
- <summary>排查出的问题</summary>
473
- 1. 只要leaderCommit比自己的commitId要大，就会信号提示applyMsg。
474
- 2. 重联后会重新选举Leader，也就是说重新选出的Leader会重新初始化自己的nextIndex。自然的，第一次尝试对重联的Server进行append会因为没有匹配的index而被拒绝，于是寄了。
475
-
476
- </details>
477
-
478
- <details>
479
- <summary>方案</summary>
480
- 1. 不再对heartbeat特殊对待，也就是说heartbeat和broadcast争抢对Server的更新
481
-
482
-
483
-
484
-
485
- </details>
486
-
487
- - [x] Apply massage的时候出现了顺序出错的问题
488
- <details>
489
- <summary>Debug log</summary>
490
-
491
- ```go
492
- // log
493
- S0 commitIndex: 6
494
- S0 -> S2 append ok T: 9, LENLOG: 7
495
- S2 -> S1 T: 9, send heartbeat
496
- S2 -> S1 update success, NEXT: [1 7 7], MATCH: [0 6 0]
497
- S2 get N: 6
498
- S0 apply msg: {true 106 6 false [] 0 0}
499
- S0 ######## CI: 6, cfg.log: [map[0:init server 1:101]
500
- map[0:init server 1:101 2:102 3:103 4:104 5:105 6:106]
501
- map[0:init server 1:101 2:102 3:103 4:104 5:105 6:106]] S2 -> S0 T: 7, send heartbeat
502
-
503
-
504
- // related code
505
- func (cfg *config) checkLogs(i int, m ApplyMsg) (string, bool) {
506
- err_msg := ""
507
- v := m.Command
508
- for j := 0; j < len(cfg.logs); j++ {
509
- if old, oldok := cfg.logs[j][m.CommandIndex]; oldok && old != v {
510
- log.Printf("%v: log %v; server %v\n", i, cfg.logs[i], cfg.logs[j])
511
- // some server has already committed a different value for this entry!
512
- err_msg = fmt.Sprintf("commit index=%v server=%v %v != server=%v %v",
513
- m.CommandIndex, i, m.Command, j, old)
514
- }
515
- }
516
-
517
- DebugLog(dDrop, "S%d ######## CI: %d, cfg.log: %v\n", i, m.CommandIndex, cfg.logs)
518
- _, prevok := cfg.logs[i][m.CommandIndex-1]
519
- cfg.logs[i][m.CommandIndex] = v
520
- if m.CommandIndex > cfg.maxIndex {
521
- cfg.maxIndex = m.CommandIndex
522
- }
523
- return err_msg, prevok
524
- }
525
- ```
526
-
527
-
528
- </details>
529
-
530
- <details>
531
- <summary>排查出的问题</summary>
532
- 1. 首先，chan是一个fifo。这里会根据fifo中吐出的ApplyMsg进行判断，将ApplyMsg中的信息提取并储存在cfg.log中。Server重联后，仅放入最新的ApplyMsg会导致index out of range的问题。
533
-
534
- </details>
535
-
536
- <details>
537
- <summary>方案</summary>
538
- 1. 也就是说，需要按照顺序往chan里一个一个放入ApplyMsg而不是仅放入最新的ApplyMsg。
539
-
540
- </details>
541
-
542
- - [x] PASS LOG
543
-
544
- ```go
545
- ➜ go test -run 2B
546
- Test (2B): basic agreement ...
547
- ... Passed -- 0.5 3 16 4290 3
548
- Test (2B): RPC byte count ...
549
- ... Passed -- 1.3 3 48 113614 11
550
- Test (2B): agreement after follower reconnects ...
551
- ... Passed -- 5.3 3 192 50800 8
552
- Test (2B): no agreement if too many followers disconnect ...
553
- ... Passed -- 3.4 5 264 51340 3
554
- Test (2B): concurrent Start()s ...
555
- ... Passed -- 0.6 3 24 6790 6
556
- Test (2B): rejoin of partitioned leader ...
557
- ... Passed -- 3.9 3 146 35480 4
558
- Test (2B): leader backs up quickly over incorrect follower logs ...
559
- ... Passed -- 14.0 5 2244 1728755 102
560
- Test (2B): RPC counts aren't too high ...
561
- ... Passed -- 2.0 3 62 18665 12
562
- PASS
563
- ok 6.824/raft 31.039s
564
- ```
565
-
566
-
567
- ### 2C
568
-
569
- - [x] 一运行到unrealiable的test就出现out of index的错误，现在想来好像在**2B**中偶尔也会出现这个错误，还债了属于是。
570
- <details>
571
- <summary>Debug log</summary>
572
-
573
- ```go
574
- ➜ go test -run UnreliableChurn
575
- Test (2C): unreliable churn ...
576
- panic: runtime error: index out of range [8] with length 8
577
-
578
- goroutine 22 [running]:
579
- 6.824/raft.(*Raft).Apply(0xc0001b2000)
580
- /home/humborn/Code/go/MIT-6.5840/src/raft/raft.go:596 +0x214
581
- created by 6.824/raft.Make
582
- /home/humborn/Code/go/MIT-6.5840/src/raft/raft.go:757 +0x3d6
583
- exit status 2
584
- FAIL 6.824/raft 0.300s
585
- ```
586
-
587
-
588
- </details>
589
-
590
- <details>
591
- <summary>排查出的问题</summary>
592
- 1. 很奇怪，有时可以通过，有时不行，2C中的Unreliable测试放大了系统的弱点，可以确定的是，心跳间隔的设置和这个错误是有关的。后来发现原来是因为延迟，Follower接受了Index更古老的消息，并覆盖了新的消息。
593
-
594
- ```go
595
- S0 reconnect...
596
- S1 reconnect...
597
- S2 reconnect...
598
- S3 reconnect...
599
- S3 -> S4 append ok T: 26,
600
- LENLOG: 207
601
- S4 -> S2 LogInfo {IDX: 192,
602
- LENLOG: 207, LOG:
603
- {426083054347800940 26 206}}
604
- S4 LogInfo {IDX: 192,
605
- LENLOG: 207, LOG:
606
- {426083054347800940 26 206}}
607
- S4 send to S2, ENTRIES:
608
- {426083054347800940 26 206}
609
- S4 -> S3 update success,
610
- NEXT: [193 205 193 207 193],
611
- MATCH: [0 204 0 206 0]
612
- S4 get N: 204
613
- S4, LENLOG: 207, LA: 204
614
- apply msg {CI: 205}
615
- S4 -> S3 LogInfo {IDX: 206,
616
- LENLOG: 207, LOG:
617
- {426083054347800940 26 206}}
618
- S4 LogInfo {IDX: 206,
619
- LENLOG: 207, LOG:
620
- {426083054347800940 26 206}}
621
- S4 send to S3, ENTRIES:
622
- {426083054347800940 26 206}
623
- S1 -> S4 append ok T: 26,
624
- LENLOG: 207
625
- S2 append fail, T: 27 large
626
- term
627
- S4 -> S1 update success,
628
- NEXT: [193 207 193 207 193],
629
- MATCH: [0 206 0 206 0]
630
- S4 get N: 206
631
- S4 update commitIndex CI:
632
- 206
633
- S4, LENLOG: 207, LA: 205
634
- apply msg {CI: 206}
635
- S4 -> S2 LogInfo {IDX: 192,
636
- LENLOG: 207, LOG:
637
- {426083054347800940 26 206}}
638
- S4 LogInfo {IDX: 192,
639
- LENLOG: 207, LOG:
640
- {426083054347800940 26 206}}
641
- S4 send to S2, ENTRIES:
642
- {426083054347800940 26 206}
643
- S3 commit {CI: 205, LENLOG:
644
- 207}
645
- S2 append fail, T: 27 large
646
- term
647
- S4 reconnect...
648
- S1 -> S4 append ok T: 26,
649
- LENLOG: 206
650
- S4 -> S1 LogInfo {IDX: 206,
651
- LENLOG: 207, LOG:
652
- {426083054347800940 26 206}}
653
- S4 LogInfo {IDX: 206,
654
- LENLOG: 207, LOG:
655
- {426083054347800940 26 206}}
656
- S4 send to S1, ENTRIES:
657
- {426083054347800940 26 206}
658
- S3 -> S4 append ok T: 26,
659
- LENLOG: 205
660
- ```
661
-
662
- 2. 有时候会出现索引取到-1的情况，排查出是append时条件设置有误。此外，当一个Server detach以后，会持续向别的server发送信息，不断收到fail update，自然会不断减小nextIndex的值。直到nextIndex变为0,自减后变为-1,索引越界。
663
-
664
- </details>
665
-
666
- <details>
667
- <summary>方案</summary>
668
- 1. 修改条目更新的规则，遵循VoteRequest RPCs中的up-to-date原则。
669
-
670
- ```go
671
- if len(args.Entries) == 0 {
672
- DebugLog(dLog, "S%d -> S%d recv heartbeat\n", rf.me, args.LeaderId, rf.currTerm, len(rf.logEntries))
673
- } else {
674
- rfLastLogIndex, rfLastLogTerm := rf.LogInfoByIndex(len(rf.logEntries) - 1)
675
- argLastLogIndex, argLastLogTerm := args.Entries[len(args.Entries)-1].Index, args.Entries[len(args.Entries)-1].Term
676
- if rfLastLogTerm < argLastLogTerm || (rfLastLogTerm == argLastLogTerm && rfLastLogIndex < argLastLogIndex) {
677
- rf.logEntries = rf.logEntries[:args.PrevLogIndex+1]
678
- rf.logEntries = append(rf.logEntries, args.Entries...) // persist
679
- rf.persist()
680
- }
681
-
682
- DebugLog(dLog2, "S%d -> S%d append ok T: %d, LENLOG: %d\n", rf.me, args.LeaderId, rf.currTerm, len(rf.logEntries))
683
- }
684
- ```
685
-
686
-
687
- 同时要注意修改reply处理部分的代码，不然会将nextIndex和matchIndex更新成旧的索引。
688
-
689
-
690
- ```go
691
- if reply.Success {
692
- if rf.nextIndex[id] < endOfIndex {
693
- rf.nextIndex[id] = endOfIndex
694
- rf.matchIndex[id] = endOfIndex - 1
695
- DebugLog(dError, "S%d -> S%d update success, NEXT: %v, MATCH: %v\n", rf.me, id, rf.nextIndex, rf.matchIndex)
696
- go rf.updateCommitIndex()
697
- }
698
- // 讲道理，更新成功就要试试能不能update commitIndex
699
- } else {
700
- rf.nextIndex[id] /= 2
701
- DebugLog(dError, "S%d -> S%d update failed, NEXT: %d\n", rf.me, id, rf.nextIndex[id])
702
- }
703
- ```
704
-
705
- 2. 贴出错误代码
706
-
707
- ```go
708
- ~~if args.PrevLogIndex >= len(rf.logEntries) || rf.logEntries[args.PrevLogIndex].Term != args.PrevLogTerm {~~
709
- if args.PrevLogIndex > len(rf.logEntries)-1 || rf.logEntries[args.PrevLogIndex].Term != args.PrevLogTerm {
710
- reply.Success = false
711
- DebugLog(dVote, "S%d append fail, diff term\n", rf.me)
712
- return
713
- }
714
- ```
715
-
716
-
717
- 此外，打了个补丁，让nextIndex等于max(1, nextIndex/2)，不知道这个补丁可不可以美化。
718
-
719
-
720
- </details>
721
-
722
- - [x] 有极小的概率出现问题：ApplyMsg顺序错位。
723
- <details>
724
- <summary>Debug log</summary>
725
-
726
- ```go
727
- Test (2C): Figure 8 (unreliable) ...
728
- 3: log map[1:2296 2:812 3:2449 4:9494 5:4661 6:4352 7:549 8:7084 9:6810 10:2131 11:5797 12:3999 13:6910 14:5209 15:4842 16:8485 17:8166 18:6914 19:6957 20:2262 21:4452 22:1043 23:1784 24:9148 25:759 26:2552 27:7916 28:6243 29:3173 30:749 31:3392 32:2467 33:1591 34:2306 35:2107 36:9868 37:9973 38:1755 39:8718 40:6107 41:7839 42:2187 43:1553 44:629 45:6596 46:4295 47:1687 48:9196 49:2304 50:54 51:6167]; server map[1:2296 2:812 3:2449 4:9494 5:4661 6:4352 7:549 8:7084 9:6810 10:2131 11:5797 12:3999 13:6910 14:5209 15:4842 16:8485 17:8166 18:6914 19:6957 20:2262 21:4452 22:1043 23:1784 24:9148 25:759 26:2552 27:7916 28:6243 29:3173 30:749 31:3392 32:2467 33:1591 34:2306 35:2107 36:9868 37:9973 38:1755 39:8718 40:6107 41:7839 42:2187 43:1553 44:629 45:6596 46:4295 47:1687 48:9196 49:2304 50:54 51:6167 52:9070 53:9855 54:4432 55:1244 56:1773 57:5898 58:2104 59:6905 60:3111 61:6309 62:8721 63:1023 64:4093 65:7848 66:7256 67:6673 68:4167 69:7717 70:9556]
729
- 3: log map[1:2296 2:812 3:2449 4:9494 5:4661 6:4352 7:549 8:7084 9:6810 10:2131 11:5797 12:3999 13:6910 14:5209 15:4842 16:8485 17:8166 18:6914 19:6957 20:2262 21:4452 22:1043 23:1784 24:9148 25:759 26:2552 27:7916 28:6243 29:3173 30:749 31:3392 32:2467 33:1591 34:2306 35:2107 36:9868 37:9973 38:1755 39:8718 40:6107 41:7839 42:2187 43:1553 44:629 45:6596 46:4295 47:1687 48:9196 49:2304 50:54 51:6167]; server map[1:2296 2:812 3:2449 4:9494 5:4661 6:4352 7:549 8:7084 9:6810 10:2131 11:5797 12:3999 13:6910 14:5209 15:4842 16:8485 17:8166 18:6914 19:6957 20:2262 21:4452 22:1043 23:1784 24:9148 25:759 26:2552 27:7916 28:6243 29:3173 30:749 31:3392 32:2467 33:1591 34:2306 35:2107 36:9868 37:9973 38:1755 39:8718 40:6107 41:7839 42:2187 43:1553 44:629 45:6596 46:4295 47:1687 48:9196 49:2304 50:54 51:6167 52:9070 53:9855 54:4432 55:1244 56:1773 57:5898 58:2104 59:6905 60:3111 61:6309 62:8721 63:1023 64:4093 65:7848 66:7256 67:6673 68:4167 69:7717 70:9556]
730
- 3: log map[1:2296 2:812 3:2449 4:9494 5:4661 6:4352 7:549 8:7084 9:6810 10:2131 11:5797 12:3999 13:6910 14:5209 15:4842 16:8485 17:8166 18:6914 19:6957 20:2262 21:4452 22:1043 23:1784 24:9148 25:759 26:2552 27:7916 28:6243 29:3173 30:749 31:3392 32:2467 33:1591 34:2306 35:2107 36:9868 37:9973 38:1755 39:8718 40:6107 41:7839 42:2187 43:1553 44:629 45:6596 46:4295 47:1687 48:9196 49:2304 50:54 51:6167]; server map[1:2296 2:812 3:2449 4:9494 5:4661 6:4352 7:549 8:7084 9:6810 10:2131 11:5797 12:3999 13:6910 14:5209 15:4842 16:8485 17:8166 18:6914 19:6957 20:2262 21:4452 22:1043 23:1784 24:9148 25:759 26:2552 27:7916 28:6243 29:3173 30:749 31:3392 32:2467 33:1591 34:2306 35:2107 36:9868 37:9973 38:1755 39:8718 40:6107 41:7839 42:2187 43:1553 44:629 45:6596 46:4295 47:1687 48:9196 49:2304 50:54 51:6167 52:9070 53:9855 54:4432 55:1244 56:1773 57:5898 58:2104 59:6905 60:3111 61:6309 62:8721 63:1023 64:4093 65:7848 66:7256 67:6673 68:4167 69:7717 70:9556]
731
- apply error: commit index=52 server=3 2540 != server=4 9070
732
- ```
733
-
734
-
735
- </details>
736
-
737
- <details>
738
- <summary>排查出的问题</summary>
739
- 1. 没排查出来
740
-
741
- </details>
742
-
743
- <details>
744
- <summary>方案</summary>
745
- 1. 将AppendEntries中的 `defer rf.applyCh.Signal()` 变成了 `rf.applyCh.Signal()`
746
-
747
- </details>
748
-
749
- - [x] PASS LOG
750
-
751
- ```go
752
- ➜ go test -run 2C
753
- Test (2C): basic persistence ...
754
- ... Passed -- 3.1 3 78 18954 6
755
- Test (2C): more persistence ...
756
- ... Passed -- 14.4 5 1036 200240 16
757
- Test (2C): partitioned leader and one follower crash, leader restarts ...
758
- ... Passed -- 1.3 3 36 8539 4
759
- Test (2C): Figure 8 ...
760
- ... Passed -- 27.0 5 1180 256006 58
761
- Test (2C): unreliable agreement ...
762
- ... Passed -- 1.7 5 1048 345471 246
763
- Test (2C): Figure 8 (unreliable) ...
764
- ... Passed -- 33.6 5 19228 18476272 138
765
- Test (2C): churn ...
766
- ... Passed -- 16.4 5 6992 9763562 1432
767
- Test (2C): unreliable churn ...
768
- ... Passed -- 16.1 5 4236 5664994 384
769
- PASS
770
- ok 6.824/raft 113.623s
771
- ```
772
-
773
-
774
- ### 2D
775
-
776
-
777
- > 在2D中，很重要的一点是，切片底层是数组实现，如果想要让GC reach到被snapshot掉的无用内存，就需要重新创建切片，原切片指针置空。
778
-
779
- - [x] 有概率出现问题：在更新commitIndex的时候出现out of Index问题。
780
- <details>
781
- <summary>Debug log</summary>
782
-
783
- ```go
784
- Test (2C): unreliable churn ...
785
- panic: runtime error: index out of range [1656] with length 1656
786
-
787
- goroutine 106315 [running]:
788
- 6.824/raft.(*Raft).updateCommitIndex(0xc000136200)
789
- /home/humborn/Code/go/MIT-6.5840/src/raft/raft.go:647 +0x345
790
- created by 6.824/raft.(*Raft).broadcastAppendEntries.func1
791
- /home/humborn/Code/go/MIT-6.5840/src/raft/raft.go:622 +0x796
792
- exit status 2
793
- ```
794
-
795
-
796
- </details>
797
-
798
- <details>
799
- <summary>排查出的问题</summary>
800
- 1. updateCommitIndex函数原本是使用协程实现的，所以有极小的概率，在updateCommitIndex之前，完成了Leader转换、重新广播、更新新的CommitIndex这一系列操作。使得一开始的更新协程出现越界。
801
-
802
- </details>
803
-
804
- <details>
805
- <summary>方案</summary>
806
- 1. 线性执行updateCommitIndex，去除函数中的加锁操作，统一由主协程管理锁
807
-
808
- ```go
809
- func (rf *Raft) updateCommitIndex() {
810
- // rf.mu.Lock()
811
- // defer rf.mu.Unlock()
812
-
813
- var dupMatchIndex []int
814
- dupMatchIndex = append(dupMatchIndex, rf.matchIndex...)
815
- dupMatchIndex[rf.me] = rf.logEntries[len(rf.logEntries)-1].Index
816
- sort.Ints(dupMatchIndex)
817
- DebugLog(dError, "S%d get N: %d\n", rf.me, dupMatchIndex[(len(rf.peers)-1)/2])
818
-
819
- var N int = dupMatchIndex[(len(rf.peers)-1)/2]
820
- if N > rf.commitIndex && rf.logEntries[N].Term == rf.currTerm {
821
- rf.commitIndex = N
822
- defer rf.applyCond.Signal()
823
- DebugLog(dError, "S%d update commitIndex CI: %d\n", rf.me, rf.commitIndex)
824
- }
825
- }
826
- ```
827
-
828
-
829
- </details>
830
-
831
- - [x] 死锁问题：一开始执行Snapshot函数就卡在获得锁的操作这里。
832
- <details>
833
- <summary>Debug log</summary>
834
-
835
- ```go
836
- S2 LogInfo {IDX: 13,
837
- LENLOG: 14, LOG:
838
- {8815208668614808923 1
839
- 13}}
840
- S2 -> S1 append ok T: 1,
841
- LENLOG: 14
842
- S1 start create snapshot
843
- LII: 9, T: 1
844
- S1 start create snapshot,
845
- before lock
846
- S1, LENLOG: 14, LA: 1
847
- apply msg {CI: 10}
848
- S0 convert to candidate,
849
- calling election T: 1
850
- S0 LogInfo {IDX: 1, LENLOG:
851
- 2, LOG:
852
- {7859782014890577642 1 1}}
853
- S2 C1 ask for vote, T: 2
854
- S1 C1 ask for vote, T: 2
855
- S2 LogInfo {IDX: 13,
856
- LENLOG: 14, LOG:
857
- {8815208668614808923 1
858
- 13}}
859
- S2 -> S0 vote false, old
860
- log
861
- S2 convert to candidate,
862
- calling election T: 2
863
- S2 LogInfo {IDX: 13,
864
- LENLOG: 14, LOG:
865
- {8815208668614808923 1
866
- 13}}
867
- S1 C1 ask for vote, T: 3
868
- S0 C1 ask for vote, T: 3
869
- S0 LogInfo {IDX: 1, LENLOG:
870
- 2, LOG:
871
- {7859782014890577642 1 1}}
872
- S0 -> S2 vote true, from T:
873
- 2 to CT: 3
874
- S2 convert to leader at
875
- T: 3
876
- S2 -> S1 LogInfo {IDX:
877
- 13, LENLOG: 14, LOG:
878
- {8815208668614808923 1
879
- 13}}
880
- S2 LogInfo {IDX: 13,
881
- LENLOG: 14, LOG:
882
- {8815208668614808923 1
883
- 13}}
884
- ```
885
-
886
-
887
- </details>
888
-
889
- <details>
890
- <summary>排查出的问题</summary>
891
- 1. 快照的间隔是每10条command进行一次快照，因此节点在进行将已经提交了的指令发送到applyCh进行执行的时候不能获取有rf.mu这个互斥锁，因为在你提交指令并将该指令发送到applyCh执行的同时，测试脚本会调用Snapshot函数进行快照，但是我设计的这个函数也需要获取rf.mu互斥锁，那么这个节点就会进入死锁状态：无法获取rf.mu互斥锁进行快照，另一边是需要等快照结束才能继续提交指令并执行，以及后续动作。
892
-
893
- </details>
894
-
895
- <details>
896
- <summary>方案</summary>
897
- 1. 在Apply函数中进行如下修改
898
-
899
- ```go
900
- ++ rf.mu.Unlock()
901
- rf.applyCh <- msg
902
- ++ rf.mu.Lock()
903
- ```
904
-
905
-
906
- </details>
907
-
908
- - [x] AppendEntries中出现负数的Log索引
909
- <details>
910
- <summary>排查出的问题</summary>
911
- 1. 原来只需要检查args.PrevLogIndex小于等于Log最后一项的索引值。但是在引入snapshot后还需要检查args.PrevLogIndex大于等于lastIncludedIndex，因为历史的AppendEntries RPC可以延迟抵达
912
-
913
- ```go
914
- if args.PrevLogIndex > rf.logEntries[len(rf.logEntries)-1].Index ||
915
- rf.logEntries[args.PrevLogIndex-rf.snapshot.LastIncludedIndex].Term != args.PrevLogTerm { // ATTENTION 这里可能会越界，要减去lastincludedindex
916
- reply.Success = false
917
- DebugLog(dVote, "S%d append fail, diff term\n", rf.me)
918
- return
919
- }
920
- ```
921
-
922
-
923
- </details>
924
-
925
- <details>
926
- <summary>方案</summary>
927
- 1. 在这种情况下，可以直接将reply的success置true并返回，因为对于成功时返回值的处理已经考虑到了延迟消息（2C）
928
-
929
- ```go
930
- if rf.snapshot.LastIncludedIndex > args.PrevLogIndex {
931
- reply.Success = true
932
- DebugLog(dVote, "S%d append success, snapshot interrupt...\n", rf.me)
933
- return
934
- }
935
- ```
936
-
937
-
938
- </details>
939
-
940
- - [x] broadcast中出现负数的Log索引
941
- <details>
942
- <summary>Debug log</summary>
943
-
944
- nextIndex[0] == snapshot.lastIncludedIndex
945
-
946
-
947
- ```go
948
- S1 -> S0 LogInfo {IDX: 108,
949
- LENLOG: 8, LOG:
950
- {5178380306894727311 1 116}},
951
- NEXT: [109 1 117], SS: {109 1}
952
- S2, LENLOG: 18, LA: 108 apply
953
- msg {CI: 109}
954
- S2, LENLOG: 18, LA: 108 apply
955
- msg {CI: 110}
956
- S2 -> S1 recv heartbeat
957
-
958
- #################################################
959
- panic: runtime error: index out of range [-1]
960
-
961
- goroutine 890 [running]:
962
- 6.824/raft.(*Raft).LogInfoByIndex(0xc000134100, 0xffffffffffffffff)
963
- /home/humborn/Code/go/MIT-6.5840/src/raft/util.go:101 +0xbd
964
- 6.824/raft.(*Raft).broadcastAppendEntries.func1(0x0)
965
- /home/humborn/Code/go/MIT-6.5840/src/raft/raft.go:716 +0x84c
966
- created by 6.824/raft.(*Raft).broadcastAppendEntries
967
- /home/humborn/Code/go/MIT-6.5840/src/raft/raft.go:669 +0x15f
968
- exit status 2
969
- ```
970
-
971
-
972
- </details>
973
-
974
- <details>
975
- <summary>排查出的问题</summary>
976
- 1. 同样的，广播发布了协程后并未立即执行，而snapshot在协程运行之前更新了lastIncludedIndex。仔细观察可以发现出错的Log的索引都是mod 10余9的，对应了snapshot的生成规律，验证了猜想。
977
-
978
- ```go
979
- if args.PrevLogIndex > rf.logEntries[len(rf.logEntries)-1].Index ||
980
- rf.logEntries[args.PrevLogIndex-rf.snapshot.LastIncludedIndex].Term != args.PrevLogTerm { // ATTENTION 这里可能会越界，要减去lastincludedindex
981
- reply.Success = false
982
- DebugLog(dVote, "S%d append fail, diff term\n", rf.me)
983
- return
984
- }
985
- ```
986
-
987
-
988
- </details>
989
-
990
- <details>
991
- <summary>方案</summary>
992
- 1. 将广播条件中触发快照加载的条件由 < 改为 ≤ 。
993
-
994
- ```go
995
- go func(id int) {
996
- rf.mu.Lock()
997
- // 需要注意的是，当rf.nextIndex[id] == rf.logEntries[len(rf.logEntries)-1].Index+1，对应的是发送heartbeat的情况
998
- if rf.nextIndex[id] > rf.logEntries[len(rf.logEntries)-1].Index+1 {
999
- rf.mu.Unlock()
1000
- return
1001
- }
1002
-
1003
- ~~if rf.nextIndex[id] < rf.snapshot.LastIncludedIndex {~~
1004
- if rf.nextIndex[id] <= rf.snapshot.LastIncludedIndex {
1005
- ```
1006
-
1007
-
1008
- </details>
1009
-
1010
- - [x] ApplyMsg顺序错位第二弹，详循2C中的第二个问题
1011
- <details>
1012
- <summary>Debug log</summary>
1013
-
1014
- ```go
1015
- Test (2D): snapshots basic ...
1016
- apply error: server 0 apply out of order, expected index 100, got 101
1017
- exit status 1
1018
- FAIL 6.824/raft 1.269s
1019
- ```
1020
-
1021
-
1022
- </details>
1023
-
1024
- <details>
1025
- <summary>排查出的问题</summary>
1026
- 1. 在原来的Apply实现中，循环中的cid只被初始化了一次，但是在这里，每次循环中会随插入记录的数量增加生成snapshot，改变原来的初始值。
1027
-
1028
- ```go
1029
- for cid := max(rf.lastApplied, rf.snapshot.LastIncludedIndex) + 1; cid <= rf.commitIndex; cid++ {
1030
- DebugLog(dError, "S%d, LENLOG: %d, LA: %d apply msg {CI: %d} \n", rf.me, len(rf.logEntries), rf.lastApplied, cid)
1031
- msg := ApplyMsg{
1032
- CommandValid: true,
1033
- Command: rf.logEntries[cid-rf.snapshot.LastIncludedIndex].Command,
1034
- CommandIndex: cid,
1035
- }
1036
- rf.mu.Unlock()
1037
- rf.applyCh <- msg
1038
- rf.mu.Lock()
1039
- }
1040
- ```
1041
-
1042
-
1043
- </details>
1044
-
1045
- <details>
1046
- <summary>方案</summary>
1047
- 1. 多次初始化。
1048
-
1049
- ```go
1050
- cid := max(rf.lastApplied, rf.snapshot.LastIncludedIndex) + 1
1051
- for cid <= rf.commitIndex {
1052
- DebugLog(dError, "S%d, LENLOG: %d, LA: %d apply msg {CI: %d} \n",
1053
- rf.me, len(rf.logEntries), rf.lastApplied, cid)
1054
- msg := ApplyMsg{
1055
- CommandValid: true,
1056
- Command: rf.logEntries[cid-rf.snapshot.LastIncludedIndex].Command,
1057
- CommandIndex: cid,
1058
- }
1059
- rf.mu.Unlock()
1060
- rf.applyCh <- msg
1061
- rf.mu.Lock()
1062
- rf.lastApplied = cid
1063
- cid = max(rf.lastApplied, rf.snapshot.LastIncludedIndex) + 1
1064
- }
1065
- ```
1066
-
1067
-
1068
- </details>
1069
-
1070
- - [x] 首先，可以知道，apply out of order有两种形态，一种是接收到了有延迟的信息，还有一种是接收到了未来的信息。以InstallSnapshot RPC为例，可以通过下面这段代码对延迟信息进行过滤。但是raft的Term、commitIndex和lastApplied等索引都是完备的，讲道理不会出现“未来的”信息。
1071
-
1072
- ```go
1073
- if rf.commitIndex < args.LastIncludedIndex {
1074
- rf.commitIndex = args.LastIncludedIndex
1075
- }
1076
- if rf.lastApplied < args.LastIncludedIndex {
1077
- rf.lastApplied = args.LastIncludedIndex
1078
- }
1079
- ```
1080
-
1081
- <details>
1082
- <summary>Debug log</summary>
1083
-
1084
- ```go
1085
- apply error: server 0 apply out of order, expected index 230, got 231
1086
- exit status 1
1087
- FAIL 6.824/raft 1.852s
1088
- ```
1089
-
1090
-
1091
- </details>
1092
-
1093
- <details>
1094
- <summary>排查出的问题</summary>
1095
- 1. 在测试代码中，snapshot在ApplyCh中被接收后，会以自身的lastIncludedIndex刷新service server的对应参数，造成“版本回退”。
1096
-
1097
- </details>
1098
-
1099
- <details>
1100
- <summary>方案</summary>
1101
- 1. 在InstallSnapshot中与lastApplied进行比较判断过滤“版本回退”
1102
-
1103
- ```go
1104
- if args.LastIncludedIndex <= rf.snapshot.LastIncludedIndex ||
1105
- args.LastIncludedIndex <= rf.lastApplied {
1106
- rf.mu.Unlock()
1107
- return
1108
- }
1109
- ```
1110
-
1111
-
1112
- </details>
1113
-
1114
- - [x] 会报莫名其妙的错误
1115
- <details>
1116
- <summary>Debug log</summary>
1117
-
1118
- 第一列的Index和LII不对应造成的
1119
-
1120
-
1121
- ```go
1122
- S0 ingestSnap: snapshot doesn't match
1123
- m.SnapshotIndex, INDEX: 39, LII: 29
1124
- S0 C46 install snapshot, IDX: 39 T: 1,
1125
- start from 39 to 46
1126
- #############################################################################
1127
-
1128
- apply error: server 0 snapshot doesn't match m.SnapshotIndex
1129
- exit status 1
1130
- FAIL 6.824/raft 1.041s
1131
- ```
1132
-
1133
-
1134
- </details>
1135
-
1136
- <details>
1137
- <summary>排查出的问题</summary>
1138
- 1. 由于在前面的问题排查中已经基本解决了out of order的问题，推测出现这种情况的原因是SaveStateAndSnapshot函数存在先后执行顺序，协程有延迟，使得保存的时候没有起到拦截out of order的作用
1139
-
1140
- </details>
1141
-
1142
- <details>
1143
- <summary>方案</summary>
1144
- 1. 把所有的rf.mu.Lock()统统移到SaveStateAndSnapshot函数后面保证线性顺序。
1145
-
1146
- </details>
1147
-
1148
- - [x] PASS LOG
1149
-
1150
- ```go
1151
- ➜ go test -run 2D
1152
- Test (2D): snapshots basic ...
1153
- ... Passed -- 1.8 3 506 164008 235
1154
- Test (2D): install snapshots (disconnect) ...
1155
- ... Passed -- 32.9 3 2444 1245742 316
1156
- Test (2D): install snapshots (disconnect+unreliable) ...
1157
- ... Passed -- 38.7 3 3604 1300893 327
1158
- Test (2D): install snapshots (crash) ...
1159
- ... Passed -- 24.5 3 1148 879888 334
1160
- Test (2D): install snapshots (unreliable+crash) ...
1161
- ... Passed -- 33.0 3 1432 692990 322
1162
- Test (2D): crash and restart all servers ...
1163
- ... Passed -- 6.7 3 244 64312 54
1164
- PASS
1165
- ok 6.824/raft 137.622s
1166
- ```
1167
-