meeseeker 1.0.0 → 2.0.0.pre.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +116 -98
- data/Rakefile +121 -44
- data/lib/meeseeker.rb +178 -6
- data/lib/meeseeker/block_follower_job.rb +33 -17
- data/lib/meeseeker/hive_engine.rb +20 -0
- data/lib/meeseeker/steem_engine/agent.rb +33 -19
- data/lib/meeseeker/steem_engine/follower_job.rb +31 -14
- data/lib/meeseeker/version.rb +1 -1
- data/lib/meeseeker/witness_schedule_job.rb +6 -2
- data/meeseeker.gemspec +1 -0
- data/test/meeseeker/meeseeker_test.rb +247 -20
- data/test/test_helper.rb +10 -0
- metadata +29 -9
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9bb58923eb232294d7a41e43e963543cfbf09f90e4fa5b20d8dfa5eb62ed05ab
|
4
|
+
data.tar.gz: 04b201a7c6e1b8c4f4b7e19e58f7bc5eaa09c5a184c43a21add8d411fb57dc2e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1e0ce806be69a0b9af06df823b443a498a8789bd93fc323367c5de3e556ce8d9bae569ef54dcf880a555d7e937491eb003ef4ff8c2a01388fb295f65887d19bc
|
7
|
+
data.tar.gz: e27cdfe179c60ab9faf17855ab5c6a4522dfb2b8ae892326ee256095513b4a8a70e8228d7eba331a52eb6ddbda94e3c9334f5b7f58284940bf4c5cf0d92e8af8
|
data/README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
# meeseeker
|
2
2
|
|
3
|
-
Redis based block follower is an efficient way for multiple apps to stream the
|
3
|
+
Redis based block follower is an efficient way for multiple apps to stream the Hive Blockchain.
|
4
4
|
|
5
5
|
[![Build Status](https://travis-ci.org/inertia186/meeseeker.svg?branch=master)](https://travis-ci.org/inertia186/meeseeker)
|
6
6
|
|
@@ -64,10 +64,22 @@ To specify an alternative redis source:
|
|
64
64
|
MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync
|
65
65
|
```
|
66
66
|
|
67
|
-
You can also specify
|
67
|
+
You can also specify am alternative Hive node:
|
68
68
|
|
69
69
|
```bash
|
70
|
-
MEESEEKER_NODE_URL=
|
70
|
+
MEESEEKER_NODE_URL=http://anyx.io meeseeker sync
|
71
|
+
```
|
72
|
+
|
73
|
+
You can also specify a Steem node instead of Hive (if that's your thing):
|
74
|
+
|
75
|
+
```bash
|
76
|
+
MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync[steem]
|
77
|
+
```
|
78
|
+
|
79
|
+
Or, you can have meeseeker automatically use random Hive nodes:
|
80
|
+
|
81
|
+
```bash
|
82
|
+
MEESEEKER_NODE_URL=shuffle meeseeker sync
|
71
83
|
```
|
72
84
|
|
73
85
|
To sync from the head block instead of the last irreversible block:
|
@@ -82,7 +94,7 @@ To ignore virtual operations (useful if the node doesn't enable `get_ops_in_bloc
|
|
82
94
|
MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
|
83
95
|
```
|
84
96
|
|
85
|
-
Normally, block headers are added to the `
|
97
|
+
Normally, block headers are added to the `hive:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `hive:block` channel to only publish with the `block_num`:
|
86
98
|
|
87
99
|
```bash
|
88
100
|
MEESEEKER_INCLUDE_BLOCK_HEADER=false meeseeker sync
|
@@ -101,6 +113,12 @@ If you never want the keys to expire (not recommended), set
|
|
101
113
|
MEESEEKER_EXPIRE_KEYS=-1 meeseeker sync
|
102
114
|
```
|
103
115
|
|
116
|
+
Normally, sync will create keys until it uses up all available memory. If you would like to only sync a certain number of keys, then sleep until those keys expire so it can pick up where it left off, set `MEESEEKER_MAX_KEYS` to a positive value:
|
117
|
+
|
118
|
+
```bash
|
119
|
+
MEESEEKER_MAX_KEYS=99 meeseeker sync
|
120
|
+
```
|
121
|
+
|
104
122
|
### Usage
|
105
123
|
|
106
124
|
When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
|
@@ -113,35 +131,35 @@ For `redis-cli`, please see: https://redis.io/topics/pubsub
|
|
113
131
|
|
114
132
|
When running `meeseeker sync`, the following channels are available:
|
115
133
|
|
116
|
-
* `
|
117
|
-
* `
|
118
|
-
* `
|
119
|
-
* `
|
120
|
-
* `
|
121
|
-
* `
|
122
|
-
* `
|
134
|
+
* `hive:block`
|
135
|
+
* `hive:transaction`
|
136
|
+
* `hive:op:vote`
|
137
|
+
* `hive:op:comment`
|
138
|
+
* `hive:op:comment_options`
|
139
|
+
* `hive:op:whatever` (replace "whatever" with the op you want)
|
140
|
+
* `hive:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
|
123
141
|
|
124
|
-
As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.
|
142
|
+
As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.hive.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
|
125
143
|
|
126
|
-
In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `
|
144
|
+
In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `hive:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `hive:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
|
127
145
|
|
128
146
|
For example, from `redis-cli`, if we wanted to stream block numbers:
|
129
147
|
|
130
148
|
```bash
|
131
149
|
$ redis-cli
|
132
|
-
127.0.0.1:6379> subscribe
|
150
|
+
127.0.0.1:6379> subscribe hive:block
|
133
151
|
Reading messages... (press Ctrl-C to quit)
|
134
152
|
1) "subscribe"
|
135
|
-
2) "
|
153
|
+
2) "hive:block"
|
136
154
|
3) (integer) 1
|
137
155
|
1) "message"
|
138
|
-
2) "
|
156
|
+
2) "hive:block"
|
139
157
|
3) "{\"block_num\":29861068,\"previous\":\"01c7a4cb4424b4dc0cb0cc72fd36b1644f8aeba5\",\"timestamp\":\"2019-01-28T20:55:03\",\"witness\":\"ausbitbank\",\"transaction_merkle_root\":\"a318bb82625bd78af8d8b506ccd4f53116372c8e\",\"extensions\":[]}"
|
140
158
|
1) "message"
|
141
|
-
2) "
|
159
|
+
2) "hive:block"
|
142
160
|
3) "{\"block_num\":29861069,\"previous\":\"01c7a4cc1bed060876cab57476846a91568a9f8a\",\"timestamp\":\"2019-01-28T20:55:06\",\"witness\":\"followbtcnews\",\"transaction_merkle_root\":\"834e05d40b9666e5ef50deb9f368c63070c0105b\",\"extensions\":[]}"
|
143
161
|
1) "message"
|
144
|
-
2) "
|
162
|
+
2) "hive:block"
|
145
163
|
3) "{\"block_num\":29861070,\"previous\":\"01c7a4cd3bbf872895654765faa4409a8e770e91\",\"timestamp\":\"2019-01-28T20:55:09\",\"witness\":\"timcliff\",\"transaction_merkle_root\":\"b2366ce9134d627e00423b28d33cc57f1e6e453f\",\"extensions\":[]}"
|
146
164
|
```
|
147
165
|
|
@@ -155,26 +173,26 @@ Which allows subscription to specific `id` patterns:
|
|
155
173
|
|
156
174
|
```
|
157
175
|
$ redis-cli
|
158
|
-
127.0.0.1:6379> subscribe
|
176
|
+
127.0.0.1:6379> subscribe hive:op:custom_json:sm_team_reveal
|
159
177
|
Reading messages... (press Ctrl-C to quit)
|
160
178
|
1) "subscribe"
|
161
|
-
2) "
|
179
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
162
180
|
3) (integer) 1
|
163
181
|
1) "message"
|
164
|
-
2) "
|
165
|
-
3) "{\"key\":\"
|
182
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
183
|
+
3) "{\"key\":\"hive:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
|
166
184
|
1) "message"
|
167
|
-
2) "
|
168
|
-
3) "{\"key\":\"
|
185
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
186
|
+
3) "{\"key\":\"hive:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
|
169
187
|
1) "message"
|
170
|
-
2) "
|
171
|
-
3) "{\"key\":\"
|
188
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
189
|
+
3) "{\"key\":\"hive:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
|
172
190
|
1) "message"
|
173
|
-
2) "
|
174
|
-
3) "{\"key\":\"
|
191
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
192
|
+
3) "{\"key\":\"hive:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
|
175
193
|
1) "message"
|
176
|
-
2) "
|
177
|
-
3) "{\"key\":\"
|
194
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
195
|
+
3) "{\"key\":\"hive:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
|
178
196
|
```
|
179
197
|
|
180
198
|
A `ruby` application can subscribe to a channel as well, using the `redis` gem:
|
@@ -185,7 +203,7 @@ require 'redis'
|
|
185
203
|
url = 'redis://127.0.0.1:6379/0'
|
186
204
|
ctx = Redis.new(url: url)
|
187
205
|
|
188
|
-
Redis.new(url: url).subscribe('
|
206
|
+
Redis.new(url: url).subscribe('hive:op:comment') do |on|
|
189
207
|
on.message do |channel, message|
|
190
208
|
payload = JSON[message]
|
191
209
|
comment = JSON[ctx.get(payload['key'])]
|
@@ -199,19 +217,19 @@ Many other clients are supported: https://redis.io/clients
|
|
199
217
|
|
200
218
|
##### Witness Schedule
|
201
219
|
|
202
|
-
When running `meeseeker witness:schedule`, the `
|
220
|
+
When running `meeseeker witness:schedule`, the `hive:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
|
203
221
|
|
204
222
|
For example, from `redis-cli`, if we wanted to subscribe to the witness schedule:
|
205
223
|
|
206
224
|
```
|
207
225
|
$ redis-cli
|
208
|
-
127.0.0.1:6379> subscribe
|
226
|
+
127.0.0.1:6379> subscribe hive:witness:schedule
|
209
227
|
Reading messages... (press Ctrl-C to quit)
|
210
228
|
1) "subscribe"
|
211
|
-
2) "
|
229
|
+
2) "hive:witness:schedule"
|
212
230
|
3) (integer) 1
|
213
231
|
1) "message"
|
214
|
-
2) "
|
232
|
+
2) "hive:witness:schedule"
|
215
233
|
3) "{\"id\":0,\"current_virtual_time\":\"415293532210075480213212125\",\"next_shuffle_block_num\":30035208,\"current_shuffled_witnesses\":[\"thecryptodrive\",\"timcliff\",\"utopian-io\",\"themarkymark\",\"aggroed\",\"smooth.witness\",\"someguy123\",\"gtg\",\"followbtcnews\",\"yabapmatt\",\"therealwolf\",\"ausbitbank\",\"curie\",\"clayop\",\"drakos\",\"blocktrades\",\"good-karma\",\"roelandp\",\"lukestokes.mhth\",\"liondani\",\"anyx\"],\"num_scheduled_witnesses\":21,\"elected_weight\":1,\"timeshare_weight\":5,\"miner_weight\":1,\"witness_pay_normalization_factor\":25,\"median_props\":{\"account_creation_fee\":{\"amount\":\"3000\",\"precision\":3,\"nai\":\"@@000000021\"},\"maximum_block_size\":65536,\"sbd_interest_rate\":0,\"account_subsidy_budget\":797,\"account_subsidy_decay\":347321},\"majority_version\":\"0.20.8\",\"max_voted_witnesses\":20,\"max_miner_witnesses\":0,\"max_runner_witnesses\":1,\"hardfork_required_witnesses\":17,\"account_subsidy_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":797,\"pool_eq\":157691079,\"max_pool_size\":157691079,\"decay_params\":{\"decay_per_time_unit\":347321,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":0},\"account_subsidy_witness_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":996,\"pool_eq\":9384019,\"max_pool_size\":9384019,\"decay_params\":{\"decay_per_time_unit\":7293741,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":257},\"min_witness_account_subsidy_decay\":0}"
|
216
234
|
```
|
217
235
|
|
@@ -232,120 +250,120 @@ See: https://redis.io/commands/scan#scan-basic-usage
|
|
232
250
|
Once your sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
|
233
251
|
|
234
252
|
```bash
|
235
|
-
redis-cli --scan --pattern '
|
253
|
+
redis-cli --scan --pattern 'hive:*:vote'
|
236
254
|
```
|
237
255
|
|
238
256
|
This returns the keys, for example:
|
239
257
|
|
240
258
|
```
|
241
|
-
|
242
|
-
|
243
|
-
|
244
|
-
|
245
|
-
|
246
|
-
|
247
|
-
|
248
|
-
|
249
|
-
|
259
|
+
hive:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
|
260
|
+
hive:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
|
261
|
+
hive:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
|
262
|
+
hive:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
|
263
|
+
hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
|
264
|
+
hive:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
|
265
|
+
hive:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
|
266
|
+
hive:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
|
267
|
+
hive:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
|
250
268
|
```
|
251
269
|
|
252
270
|
To get the actual vote operation for a particular key, use:
|
253
271
|
|
254
272
|
```bash
|
255
|
-
redis-cli get
|
273
|
+
redis-cli get hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
|
256
274
|
```
|
257
275
|
|
258
276
|
If, on the other hand, you want `custom_json` only:
|
259
277
|
|
260
278
|
```bash
|
261
|
-
redis-cli --scan --pattern '
|
279
|
+
redis-cli --scan --pattern 'hive:*:custom_json'
|
262
280
|
```
|
263
281
|
|
264
282
|
This only returns the related keys, for example:
|
265
283
|
|
266
284
|
```
|
267
|
-
|
268
|
-
|
269
|
-
|
270
|
-
|
271
|
-
|
272
|
-
|
285
|
+
hive:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
|
286
|
+
hive:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
|
287
|
+
hive:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
|
288
|
+
hive:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
|
289
|
+
hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
|
290
|
+
hive:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
|
273
291
|
```
|
274
292
|
|
275
293
|
To get the actual custom json operation for a particular key, use:
|
276
294
|
|
277
295
|
```bash
|
278
|
-
redis-cli get
|
296
|
+
redis-cli get hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
|
279
297
|
```
|
280
298
|
|
281
299
|
To get all transactions for a particular block number:
|
282
300
|
|
283
301
|
```bash
|
284
|
-
redis-cli --scan --pattern '
|
302
|
+
redis-cli --scan --pattern 'hive:29811085:*'
|
285
303
|
```
|
286
304
|
|
287
305
|
Or to get all ops for a particular transaction:
|
288
306
|
|
289
307
|
```bash
|
290
|
-
redis-cli --scan --pattern '
|
308
|
+
redis-cli --scan --pattern 'hive:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
|
291
309
|
```
|
292
310
|
|
293
|
-
###
|
311
|
+
### Hive Engine Support
|
294
312
|
|
295
|
-
As of `v0.0.6`, meeseeker can also follow the
|
313
|
+
As of `v0.0.6`, meeseeker can also follow the Hive Engine side-chain. This is optional and requires a separate process.
|
296
314
|
|
297
|
-
To sync
|
315
|
+
To sync Hive Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
|
298
316
|
|
299
317
|
```bash
|
300
|
-
meeseeker sync
|
301
|
-
```
|
302
|
-
|
303
|
-
When running `meeseeker sync
|
304
|
-
|
305
|
-
* `
|
306
|
-
* `
|
307
|
-
* `
|
308
|
-
* `
|
309
|
-
* `
|
310
|
-
* `
|
311
|
-
* `
|
312
|
-
* `
|
313
|
-
* `
|
314
|
-
* `
|
315
|
-
* `
|
316
|
-
* `
|
317
|
-
* `
|
318
|
-
* `
|
319
|
-
* `
|
320
|
-
* `
|
321
|
-
* `
|
322
|
-
* `
|
323
|
-
* `
|
324
|
-
* `
|
325
|
-
* `
|
326
|
-
* `
|
327
|
-
* `
|
328
|
-
* `
|
329
|
-
* `
|
330
|
-
* `
|
331
|
-
* `
|
318
|
+
meeseeker sync hive_engine
|
319
|
+
```
|
320
|
+
|
321
|
+
When running `meeseeker sync hive_engine`, the following channels are available:
|
322
|
+
|
323
|
+
* `hive_engine:block`
|
324
|
+
* `hive_engine:transaction`
|
325
|
+
* `hive_engine:virtual_transaction`
|
326
|
+
* `hive_engine:contract`
|
327
|
+
* `hive_engine:contract:deploy`
|
328
|
+
* `hive_engine:contract:update`
|
329
|
+
* `hive_engine:market`
|
330
|
+
* `hive_engine:market:buy`
|
331
|
+
* `hive_engine:market:cancel`
|
332
|
+
* `hive_engine:market:sell`
|
333
|
+
* `hive_engine:sscstore`
|
334
|
+
* `hive_engine:sscstore:buy`
|
335
|
+
* `hive_engine:steempegged`
|
336
|
+
* `hive_engine:steempegged:buy`
|
337
|
+
* `hive_engine:steempegged:removeWithdrawal`
|
338
|
+
* `hive_engine:steempegged:withdraw`
|
339
|
+
* `hive_engine:tokens`
|
340
|
+
* `hive_engine:tokens:checkPendingUnstake`
|
341
|
+
* `hive_engine:tokens:create`
|
342
|
+
* `hive_engine:tokens:enableStaking`
|
343
|
+
* `hive_engine:tokens:issue`
|
344
|
+
* `hive_engine:tokens:transfer`
|
345
|
+
* `hive_engine:tokens:transferOwnership`
|
346
|
+
* `hive_engine:tokens:unstake`
|
347
|
+
* `hive_engine:tokens:updateMetadata`
|
348
|
+
* `hive_engine:tokens:updateParams`
|
349
|
+
* `hive_engine:tokens:updateUrl`
|
332
350
|
|
333
351
|
The above "channel/action" patterns are the ones that are known that the time of writing. In addition, if a new contract is added or updated, meeseeker will automatically publish to these corresponding channels as they appear, without needing to update or even restart meeseeker.
|
334
352
|
|
335
353
|
See main section on [Using `SUBSCRIBE`](#using-subscribe).
|
336
354
|
|
337
|
-
Once your
|
355
|
+
Once your HiveEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
|
338
356
|
|
339
357
|
```bash
|
340
|
-
redis-cli --scan --pattern '
|
358
|
+
redis-cli --scan --pattern 'hive_engine:*:tokens:transfer'
|
341
359
|
```
|
342
360
|
|
343
361
|
This returns the keys, for example:
|
344
362
|
|
345
363
|
```
|
346
|
-
|
347
|
-
|
348
|
-
|
364
|
+
hive_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
|
365
|
+
hive_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
|
366
|
+
hive_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
|
349
367
|
```
|
350
368
|
|
351
369
|
See main section on [Using `SCAN`](#using-scan).
|
@@ -376,11 +394,11 @@ Also see: https://hub.docker.com/r/inertia/meeseeker/
|
|
376
394
|
<img src="https://i.imgur.com/Y3Sa2GW.jpg" />
|
377
395
|
</center>
|
378
396
|
|
379
|
-
See some of my previous Ruby How To posts in: [#radiator](https://
|
397
|
+
See some of my previous Ruby How To posts in: [#radiator](https://hive.blog/created/radiator) [#ruby](https://hive.blog/created/ruby)
|
380
398
|
|
381
399
|
## Get in touch!
|
382
400
|
|
383
|
-
If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on
|
401
|
+
If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on Hive.
|
384
402
|
|
385
403
|
## License
|
386
404
|
|
data/Rakefile
CHANGED
@@ -52,29 +52,35 @@ task :check_schema do
|
|
52
52
|
end
|
53
53
|
|
54
54
|
task(:sync, [:chain, :at_block_num] => [:check_schema]) do |t, args|
|
55
|
-
chain =
|
55
|
+
chain = args[:chain] if args[:chain]
|
56
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
56
57
|
|
57
|
-
job = case chain
|
58
|
-
when :steem
|
59
|
-
Meeseeker::BlockFollowerJob.new
|
58
|
+
job = case chain.to_sym
|
60
59
|
when :steem_engine
|
61
60
|
Meeseeker::SteemEngine::FollowerJob.new
|
62
|
-
|
61
|
+
when :hive_engine
|
62
|
+
Meeseeker::HiveEngine::FollowerJob.new
|
63
|
+
else
|
64
|
+
Meeseeker::BlockFollowerJob.new
|
63
65
|
end
|
64
66
|
|
65
|
-
job.perform(at_block_num: args[:at_block_num])
|
67
|
+
job.perform(chain: chain, at_block_num: args[:at_block_num])
|
66
68
|
end
|
67
69
|
|
68
70
|
namespace :witness do
|
69
|
-
desc 'Publish the witness schedule every minute or so (
|
70
|
-
task :schedule do
|
71
|
+
desc 'Publish the witness schedule every minute or so (e.g.: hive:witness:schedule).'
|
72
|
+
task :schedule, [:chain] do |t, args|
|
73
|
+
chain = args[:chain] if args[:chain]
|
74
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
75
|
+
|
71
76
|
job = Meeseeker::WitnessScheduleJob.new
|
72
|
-
job.perform
|
77
|
+
job.perform(chain: chain)
|
73
78
|
end
|
74
79
|
end
|
75
80
|
|
76
81
|
task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
|
77
|
-
chain =
|
82
|
+
chain = args[:chain] if args[:chain]
|
83
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
78
84
|
redis = Meeseeker.redis
|
79
85
|
|
80
86
|
match = case args[:what].downcase.to_sym
|
@@ -99,11 +105,15 @@ task :reset, [:chain] => [:check_schema] do |t, args|
|
|
99
105
|
print 'Dropping keys for set: %s ...' % chain.to_s
|
100
106
|
|
101
107
|
case chain
|
102
|
-
when :steem then keys += Meeseeker.redis.keys('steem:*')
|
103
108
|
when :steem_engine then keys += Meeseeker.redis.keys('steem_engine:*')
|
109
|
+
when :hive_engine then keys += Meeseeker.redis.keys('hive_engine:*')
|
104
110
|
when :all
|
105
111
|
keys += Meeseeker.redis.keys('steem:*')
|
112
|
+
keys += Meeseeker.redis.keys('hive:*')
|
106
113
|
keys += Meeseeker.redis.keys('steem_engine:*')
|
114
|
+
keys += Meeseeker.redis.keys('hive_engine:*')
|
115
|
+
else
|
116
|
+
keys += Meeseeker.redis.keys("#{chain}:*")
|
107
117
|
end
|
108
118
|
|
109
119
|
if keys.any?
|
@@ -117,10 +127,13 @@ end
|
|
117
127
|
|
118
128
|
namespace :verify do
|
119
129
|
desc 'Verifies transactions land where they should.'
|
120
|
-
task :block_org, [:max_blocks] do |t, args|
|
130
|
+
task :block_org, [:chain, :max_blocks] do |t, args|
|
131
|
+
chain = args[:chain] if args[:chain]
|
132
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
133
|
+
chain_key_prefix = chain.to_s
|
121
134
|
max_blocks = args[:max_blocks]
|
122
|
-
node_url =
|
123
|
-
database_api =
|
135
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
136
|
+
database_api = Meeseeker.database_api_class(chain).new(url: node_url)
|
124
137
|
mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
|
125
138
|
until_block_num = if !!max_blocks
|
126
139
|
database_api.get_dynamic_global_properties do |dgpo|
|
@@ -139,7 +152,7 @@ namespace :verify do
|
|
139
152
|
|
140
153
|
loop do
|
141
154
|
begin
|
142
|
-
job.perform(mode: mode, until_block_num: until_block_num)
|
155
|
+
job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
|
143
156
|
rescue => e
|
144
157
|
puts e.inspect
|
145
158
|
sleep 5
|
@@ -152,8 +165,8 @@ namespace :verify do
|
|
152
165
|
end
|
153
166
|
|
154
167
|
begin
|
155
|
-
block_api =
|
156
|
-
block_channel =
|
168
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
169
|
+
block_channel = "#{chain_key_prefix}:block"
|
157
170
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
158
171
|
subscription = Redis.new(url: redis_url)
|
159
172
|
ctx = Redis.new(url: redis_url)
|
@@ -186,14 +199,21 @@ namespace :verify do
|
|
186
199
|
end
|
187
200
|
end
|
188
201
|
|
189
|
-
|
202
|
+
5.times do
|
203
|
+
break unless ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
204
|
+
|
190
205
|
# This ensures at least the next block has been indexed before
|
191
206
|
# proceeding.
|
192
207
|
|
193
|
-
puts "Waiting for block: #{next_block_num} ..."
|
208
|
+
puts "Waiting for block (verify:block_org): #{next_block_num} ..."
|
209
|
+
|
194
210
|
sleep 6
|
195
211
|
end
|
196
212
|
|
213
|
+
if ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
214
|
+
puts "Gave up waiting for block (check current_aslot slippage): #{next_block_num}"
|
215
|
+
end
|
216
|
+
|
197
217
|
database_api.get_dynamic_global_properties do |dgpo|
|
198
218
|
raise 'Got empty dynamic_global_properties result.' if dgpo.nil?
|
199
219
|
|
@@ -207,7 +227,7 @@ namespace :verify do
|
|
207
227
|
end
|
208
228
|
|
209
229
|
# In theory, we should have all the keys using this pattern.
|
210
|
-
keys = ctx.keys("
|
230
|
+
keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
|
211
231
|
|
212
232
|
# If we have all the keys, we should also have all transaction ids.
|
213
233
|
expected_ids = keys.map { |k| k.split(':')[2] }.uniq
|
@@ -252,18 +272,35 @@ namespace :verify do
|
|
252
272
|
end
|
253
273
|
end
|
254
274
|
|
275
|
+
desc 'Verifies Hive Engine transactions land where they should.'
|
276
|
+
task :hive_engine_block_org, [:max_blocks] do |t, args|
|
277
|
+
Rake::Task['verify:engine_block_org'].invoke('hive_engine', args[:max_blocks])
|
278
|
+
end
|
279
|
+
|
255
280
|
desc 'Verifies Steem Engine transactions land where they should.'
|
256
281
|
task :steem_engine_block_org, [:max_blocks] do |t, args|
|
282
|
+
Rake::Task['verify:engine_block_org'].invoke('steem_engine', args[:max_blocks])
|
283
|
+
end
|
284
|
+
|
285
|
+
desc 'Verifies Steem/Hive Engine transactions land where they should.'
|
286
|
+
task :engine_block_org, [:chain_key_prefix, :max_blocks] do |t, args|
|
287
|
+
chain_key_prefix = args[:chain_key_prefix]
|
257
288
|
max_blocks = args[:max_blocks]
|
258
|
-
|
259
|
-
|
289
|
+
case chain_key_prefix.to_sym
|
290
|
+
when :steem_engine
|
291
|
+
node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
|
292
|
+
agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
|
293
|
+
job = Meeseeker::SteemEngine::FollowerJob.new
|
294
|
+
when :hive_engine
|
295
|
+
node_url = ENV.fetch('MEESEEKER_HIVE_ENGINE_NODE_URL', 'https://api.hive-engine.com/rpc')
|
296
|
+
agent = Meeseeker::HiveEngine::Agent.new(url: node_url)
|
297
|
+
job = Meeseeker::HiveEngine::FollowerJob.new
|
298
|
+
end
|
260
299
|
until_block_num = if !!max_blocks
|
261
300
|
agent.latest_block_info['blockNumber']
|
262
301
|
end
|
263
302
|
|
264
303
|
Thread.new do
|
265
|
-
job = Meeseeker::SteemEngine::FollowerJob.new
|
266
|
-
|
267
304
|
loop do
|
268
305
|
begin
|
269
306
|
at_block_num = agent.latest_block_info["blockNumber"] - max_blocks.to_i
|
@@ -281,7 +318,7 @@ namespace :verify do
|
|
281
318
|
end
|
282
319
|
|
283
320
|
begin
|
284
|
-
block_channel =
|
321
|
+
block_channel = "#{chain_key_prefix}:block"
|
285
322
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
286
323
|
subscription = Redis.new(url: redis_url)
|
287
324
|
ctx = Redis.new(url: redis_url)
|
@@ -313,16 +350,16 @@ namespace :verify do
|
|
313
350
|
end
|
314
351
|
end
|
315
352
|
|
316
|
-
while ctx.keys("
|
353
|
+
while ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
317
354
|
# This ensures at least the next block has been indexed before
|
318
355
|
# proceeding.
|
319
356
|
|
320
|
-
puts "Waiting for block: #{next_block_num} ..."
|
357
|
+
puts "Waiting for block (verify:#{chain_key_prefix}_engine_block_org): #{next_block_num} ..."
|
321
358
|
sleep 6
|
322
359
|
end
|
323
360
|
|
324
361
|
# In theory, we should have all the keys using this pattern.
|
325
|
-
keys = ctx.keys("
|
362
|
+
keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
|
326
363
|
|
327
364
|
# If we have all the keys, we should also have all transaction ids.
|
328
365
|
expected_ids = keys.map { |k| k.split(':')[2] }.uniq
|
@@ -368,19 +405,40 @@ namespace :verify do
|
|
368
405
|
agent.shutdown
|
369
406
|
end
|
370
407
|
|
408
|
+
desc 'Verifies Hive Engine sidechain against the mainnet.'
|
409
|
+
task :hive_engine_ref_blocks do |t|
|
410
|
+
Rake::Task['verify:engine_ref_blocks'].invoke('hive_engine')
|
411
|
+
end
|
412
|
+
|
371
413
|
desc 'Verifies Steem Engine sidechain against the mainnet.'
|
372
414
|
task :steem_engine_ref_blocks do |t|
|
415
|
+
Rake::Task['verify:engine_ref_blocks'].invoke('steem_engine')
|
416
|
+
end
|
417
|
+
|
418
|
+
desc 'Verifies Steem/Hive Engine sidechain against the mainnet.'
|
419
|
+
task :engine_ref_blocks, [:chain_key_prefix] do |t, args|
|
420
|
+
chain_key_prefix = args[:chain_key_prefix]
|
373
421
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
374
422
|
ctx = ctx = Redis.new(url: redis_url)
|
375
|
-
keys = ctx.keys(
|
376
|
-
|
423
|
+
keys = ctx.keys("#{chain_key_prefix}:*:*")
|
424
|
+
mainchain, mainchain_url = case chain_key_prefix
|
425
|
+
when 'steem_engine' then ['steem', 'https://api.steemit.com']
|
426
|
+
when 'hive_engine' then ['hive', 'https://api.openhive.network']
|
427
|
+
end
|
428
|
+
block_api = Meeseeker.block_api_class(mainchain).new(url: mainchain_url)
|
377
429
|
block_trxs = {}
|
378
430
|
|
379
|
-
puts "Checking
|
431
|
+
puts "Checking #{chain_key_prefix} keys: #{keys.size}"
|
380
432
|
|
381
433
|
keys.each do |key|
|
382
434
|
transaction = JSON[ctx.get(key)]
|
383
|
-
|
435
|
+
|
436
|
+
next if transaction.class == Integer
|
437
|
+
|
438
|
+
block_num = case chain_key_prefix
|
439
|
+
when 'steem_engine' then transaction.fetch('refSteemBlockNumber')
|
440
|
+
when 'hive_engine' then transaction.fetch('refHiveBlockNumber')
|
441
|
+
end.to_i
|
384
442
|
|
385
443
|
block_trxs[block_num] ||= []
|
386
444
|
block_trxs[block_num] << transaction['transactionId'].to_s.split('-').first
|
@@ -390,6 +448,8 @@ namespace :verify do
|
|
390
448
|
|
391
449
|
skipped_blocks = []
|
392
450
|
|
451
|
+
next if block_trxs.empty?
|
452
|
+
|
393
453
|
block_api.get_blocks(block_range: block_trxs.keys) do |block, block_num|
|
394
454
|
if block.nil? || block[:transaction_ids].nil?
|
395
455
|
print 'S'
|
@@ -399,24 +459,34 @@ namespace :verify do
|
|
399
459
|
else
|
400
460
|
print '.'
|
401
461
|
end
|
402
|
-
|
403
|
-
|
462
|
+
|
463
|
+
trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
|
464
|
+
|
465
|
+
if trx_ids.any? && (block.transaction_ids & trx_ids).none?
|
404
466
|
puts "\nNo intersection in #{block_num}!"
|
405
|
-
puts "Expected the following sidechain trx_ids: #{
|
467
|
+
puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
|
406
468
|
end
|
407
469
|
end
|
408
470
|
|
409
471
|
puts "\nBlocks to retry: #{skipped_blocks.size}"
|
410
472
|
|
411
473
|
skipped_blocks.each do |block_num|
|
474
|
+
block_found = false
|
475
|
+
|
412
476
|
block_api.get_block(block_num: block_num) do |result|
|
477
|
+
break unless !!result.block
|
478
|
+
|
413
479
|
block = result.block
|
480
|
+
block_found = true
|
481
|
+
trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
|
414
482
|
|
415
|
-
if (block.transaction_ids &
|
483
|
+
if trx_ids.any? && (block.transaction_ids & trx_ids).none?
|
416
484
|
puts "No intersection in #{block_num}!"
|
417
|
-
puts "Expected the following sidechain trx_ids: #{
|
485
|
+
puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
|
418
486
|
end
|
419
487
|
end
|
488
|
+
|
489
|
+
redo unless block_found
|
420
490
|
end
|
421
491
|
|
422
492
|
puts "Done."
|
@@ -424,10 +494,12 @@ namespace :verify do
|
|
424
494
|
|
425
495
|
namespace :witness do
|
426
496
|
desc 'Verifies witnessses in the schedule produced a block.'
|
427
|
-
task :schedule, [:max_blocks] do |t, args|
|
497
|
+
task :schedule, [:chain, :max_blocks] do |t, args|
|
498
|
+
chain = args[:chain] if !!args[:chain]
|
499
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
428
500
|
max_blocks = args[:max_blocks]
|
429
|
-
node_url =
|
430
|
-
database_api =
|
501
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
502
|
+
database_api = Meeseeker.database_api_class(chain).new(url: node_url)
|
431
503
|
mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
|
432
504
|
until_block_num = if !!max_blocks
|
433
505
|
database_api.get_dynamic_global_properties do |dgpo|
|
@@ -446,7 +518,7 @@ namespace :verify do
|
|
446
518
|
|
447
519
|
loop do
|
448
520
|
begin
|
449
|
-
job.perform(mode: mode, until_block_num: until_block_num)
|
521
|
+
job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
|
450
522
|
rescue => e
|
451
523
|
puts e.inspect
|
452
524
|
sleep 5
|
@@ -459,8 +531,10 @@ namespace :verify do
|
|
459
531
|
end
|
460
532
|
|
461
533
|
begin
|
462
|
-
block_api =
|
463
|
-
|
534
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
535
|
+
chain_key_prefix = chain.to_s if args[:chain]
|
536
|
+
chain_key_prefix ||= Meeseeker.default_chain_key_prefix
|
537
|
+
schedule_channel = "#{chain_key_prefix}:witness:schedule"
|
464
538
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
465
539
|
subscription = Redis.new(url: redis_url)
|
466
540
|
timeout = (max_blocks).to_i * 3
|
@@ -519,7 +593,10 @@ namespace :verify do
|
|
519
593
|
unless !!header
|
520
594
|
# Can happen when there's excess p2p latency and/or jussi
|
521
595
|
# cache is under load.
|
522
|
-
puts "Waiting for block header: #{block_num}"
|
596
|
+
puts "Waiting for block header (witness:schedule): #{block_num}"
|
597
|
+
|
598
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
599
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
523
600
|
|
524
601
|
next
|
525
602
|
end
|