meeseeker 0.0.7 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +118 -89
- data/Rakefile +131 -49
- data/lib/meeseeker.rb +178 -6
- data/lib/meeseeker/block_follower_job.rb +111 -65
- data/lib/meeseeker/hive_engine.rb +20 -0
- data/lib/meeseeker/steem_engine/agent.rb +33 -19
- data/lib/meeseeker/steem_engine/follower_job.rb +88 -18
- data/lib/meeseeker/version.rb +1 -1
- data/lib/meeseeker/witness_schedule_job.rb +6 -2
- data/meeseeker.gemspec +4 -1
- data/test/meeseeker/meeseeker_test.rb +253 -16
- data/test/test_helper.rb +11 -0
- metadata +84 -24
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3198bed0cd41758cb9b2c5669b6f63cb6e610c35289a60dd3efd631c417117c3
|
4
|
+
data.tar.gz: 9c032a2d5711fae827a1325ec782c818e07f3c7dae99b4985ea2cf94d3d6a4d1
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e75b91ca8d0290919dce1a424297a8ae74add9fb8936ea28ae16a249b08c0a3f882953129e17605a88f41196cd7530d63aee0807f5a6dad01ac7f23ea1b64efb
|
7
|
+
data.tar.gz: e0c794e530599986f9800cb8f8bf8f2c305e46ab60ba45078bb5ae9ddf2f4c878b66a142043a32d00a174488fc909de889e6b7354c238474990b791434bcf86f
|
data/README.md
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
# meeseeker
|
2
2
|
|
3
|
-
Redis based block follower is an efficient way for multiple apps to stream the
|
3
|
+
Redis based block follower is an efficient way for multiple apps to stream the Hive Blockchain.
|
4
|
+
|
5
|
+
[](https://travis-ci.org/inertia186/meeseeker)
|
4
6
|
|
5
7
|
If you have multiple applications that need to perform actions as operations occur, `meeseeker` will allow your apps to each perform actions for specific operations without each app having to stream the entire blockchain.
|
6
8
|
|
@@ -62,10 +64,22 @@ To specify an alternative redis source:
|
|
62
64
|
MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync
|
63
65
|
```
|
64
66
|
|
65
|
-
You can also specify
|
67
|
+
You can also specify am alternative Hive node:
|
66
68
|
|
67
69
|
```bash
|
68
|
-
MEESEEKER_NODE_URL=
|
70
|
+
MEESEEKER_NODE_URL=http://anyx.io meeseeker sync
|
71
|
+
```
|
72
|
+
|
73
|
+
You can also specify a Steem node instead of Hive (if that's your thing):
|
74
|
+
|
75
|
+
```bash
|
76
|
+
MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync[steem]
|
77
|
+
```
|
78
|
+
|
79
|
+
Or, you can have meeseeker automatically use random Hive nodes:
|
80
|
+
|
81
|
+
```bash
|
82
|
+
MEESEEKER_NODE_URL=shuffle meeseeker sync
|
69
83
|
```
|
70
84
|
|
71
85
|
To sync from the head block instead of the last irreversible block:
|
@@ -80,7 +94,7 @@ To ignore virtual operations (useful if the node doesn't enable `get_ops_in_bloc
|
|
80
94
|
MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
|
81
95
|
```
|
82
96
|
|
83
|
-
Normally, block headers are added to the `
|
97
|
+
Normally, block headers are added to the `hive:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `hive:block` channel to only publish with the `block_num`:
|
84
98
|
|
85
99
|
```bash
|
86
100
|
MEESEEKER_INCLUDE_BLOCK_HEADER=false meeseeker sync
|
@@ -99,6 +113,12 @@ If you never want the keys to expire (not recommended), set
|
|
99
113
|
MEESEEKER_EXPIRE_KEYS=-1 meeseeker sync
|
100
114
|
```
|
101
115
|
|
116
|
+
Normally, sync will create keys until it uses up all available memory. If you would like to only sync a certain number of keys, then sleep until those keys expire so it can pick up where it left off, set `MEESEEKER_MAX_KEYS` to a positive value:
|
117
|
+
|
118
|
+
```bash
|
119
|
+
MEESEEKER_MAX_KEYS=99 meeseeker sync
|
120
|
+
```
|
121
|
+
|
102
122
|
### Usage
|
103
123
|
|
104
124
|
When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
|
@@ -111,35 +131,35 @@ For `redis-cli`, please see: https://redis.io/topics/pubsub
|
|
111
131
|
|
112
132
|
When running `meeseeker sync`, the following channels are available:
|
113
133
|
|
114
|
-
* `
|
115
|
-
* `
|
116
|
-
* `
|
117
|
-
* `
|
118
|
-
* `
|
119
|
-
* `
|
120
|
-
* `
|
134
|
+
* `hive:block`
|
135
|
+
* `hive:transaction`
|
136
|
+
* `hive:op:vote`
|
137
|
+
* `hive:op:comment`
|
138
|
+
* `hive:op:comment_options`
|
139
|
+
* `hive:op:whatever` (replace "whatever" with the op you want)
|
140
|
+
* `hive:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
|
121
141
|
|
122
|
-
As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.
|
142
|
+
As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.hive.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
|
123
143
|
|
124
|
-
In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `
|
144
|
+
In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `hive:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `hive:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
|
125
145
|
|
126
146
|
For example, from `redis-cli`, if we wanted to stream block numbers:
|
127
147
|
|
128
148
|
```bash
|
129
149
|
$ redis-cli
|
130
|
-
127.0.0.1:6379> subscribe
|
150
|
+
127.0.0.1:6379> subscribe hive:block
|
131
151
|
Reading messages... (press Ctrl-C to quit)
|
132
152
|
1) "subscribe"
|
133
|
-
2) "
|
153
|
+
2) "hive:block"
|
134
154
|
3) (integer) 1
|
135
155
|
1) "message"
|
136
|
-
2) "
|
156
|
+
2) "hive:block"
|
137
157
|
3) "{\"block_num\":29861068,\"previous\":\"01c7a4cb4424b4dc0cb0cc72fd36b1644f8aeba5\",\"timestamp\":\"2019-01-28T20:55:03\",\"witness\":\"ausbitbank\",\"transaction_merkle_root\":\"a318bb82625bd78af8d8b506ccd4f53116372c8e\",\"extensions\":[]}"
|
138
158
|
1) "message"
|
139
|
-
2) "
|
159
|
+
2) "hive:block"
|
140
160
|
3) "{\"block_num\":29861069,\"previous\":\"01c7a4cc1bed060876cab57476846a91568a9f8a\",\"timestamp\":\"2019-01-28T20:55:06\",\"witness\":\"followbtcnews\",\"transaction_merkle_root\":\"834e05d40b9666e5ef50deb9f368c63070c0105b\",\"extensions\":[]}"
|
141
161
|
1) "message"
|
142
|
-
2) "
|
162
|
+
2) "hive:block"
|
143
163
|
3) "{\"block_num\":29861070,\"previous\":\"01c7a4cd3bbf872895654765faa4409a8e770e91\",\"timestamp\":\"2019-01-28T20:55:09\",\"witness\":\"timcliff\",\"transaction_merkle_root\":\"b2366ce9134d627e00423b28d33cc57f1e6e453f\",\"extensions\":[]}"
|
144
164
|
```
|
145
165
|
|
@@ -153,26 +173,26 @@ Which allows subscription to specific `id` patterns:
|
|
153
173
|
|
154
174
|
```
|
155
175
|
$ redis-cli
|
156
|
-
127.0.0.1:6379> subscribe
|
176
|
+
127.0.0.1:6379> subscribe hive:op:custom_json:sm_team_reveal
|
157
177
|
Reading messages... (press Ctrl-C to quit)
|
158
178
|
1) "subscribe"
|
159
|
-
2) "
|
179
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
160
180
|
3) (integer) 1
|
161
181
|
1) "message"
|
162
|
-
2) "
|
163
|
-
3) "{\"key\":\"
|
182
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
183
|
+
3) "{\"key\":\"hive:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
|
164
184
|
1) "message"
|
165
|
-
2) "
|
166
|
-
3) "{\"key\":\"
|
185
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
186
|
+
3) "{\"key\":\"hive:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
|
167
187
|
1) "message"
|
168
|
-
2) "
|
169
|
-
3) "{\"key\":\"
|
188
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
189
|
+
3) "{\"key\":\"hive:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
|
170
190
|
1) "message"
|
171
|
-
2) "
|
172
|
-
3) "{\"key\":\"
|
191
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
192
|
+
3) "{\"key\":\"hive:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
|
173
193
|
1) "message"
|
174
|
-
2) "
|
175
|
-
3) "{\"key\":\"
|
194
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
195
|
+
3) "{\"key\":\"hive:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
|
176
196
|
```
|
177
197
|
|
178
198
|
A `ruby` application can subscribe to a channel as well, using the `redis` gem:
|
@@ -183,7 +203,7 @@ require 'redis'
|
|
183
203
|
url = 'redis://127.0.0.1:6379/0'
|
184
204
|
ctx = Redis.new(url: url)
|
185
205
|
|
186
|
-
Redis.new(url: url).subscribe('
|
206
|
+
Redis.new(url: url).subscribe('hive:op:comment') do |on|
|
187
207
|
on.message do |channel, message|
|
188
208
|
payload = JSON[message]
|
189
209
|
comment = JSON[ctx.get(payload['key'])]
|
@@ -197,19 +217,19 @@ Many other clients are supported: https://redis.io/clients
|
|
197
217
|
|
198
218
|
##### Witness Schedule
|
199
219
|
|
200
|
-
When running `meeseeker witness:schedule`, the `
|
220
|
+
When running `meeseeker witness:schedule`, the `hive:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
|
201
221
|
|
202
222
|
For example, from `redis-cli`, if we wanted to subscribe to the witness schedule:
|
203
223
|
|
204
224
|
```
|
205
225
|
$ redis-cli
|
206
|
-
127.0.0.1:6379> subscribe
|
226
|
+
127.0.0.1:6379> subscribe hive:witness:schedule
|
207
227
|
Reading messages... (press Ctrl-C to quit)
|
208
228
|
1) "subscribe"
|
209
|
-
2) "
|
229
|
+
2) "hive:witness:schedule"
|
210
230
|
3) (integer) 1
|
211
231
|
1) "message"
|
212
|
-
2) "
|
232
|
+
2) "hive:witness:schedule"
|
213
233
|
3) "{\"id\":0,\"current_virtual_time\":\"415293532210075480213212125\",\"next_shuffle_block_num\":30035208,\"current_shuffled_witnesses\":[\"thecryptodrive\",\"timcliff\",\"utopian-io\",\"themarkymark\",\"aggroed\",\"smooth.witness\",\"someguy123\",\"gtg\",\"followbtcnews\",\"yabapmatt\",\"therealwolf\",\"ausbitbank\",\"curie\",\"clayop\",\"drakos\",\"blocktrades\",\"good-karma\",\"roelandp\",\"lukestokes.mhth\",\"liondani\",\"anyx\"],\"num_scheduled_witnesses\":21,\"elected_weight\":1,\"timeshare_weight\":5,\"miner_weight\":1,\"witness_pay_normalization_factor\":25,\"median_props\":{\"account_creation_fee\":{\"amount\":\"3000\",\"precision\":3,\"nai\":\"@@000000021\"},\"maximum_block_size\":65536,\"sbd_interest_rate\":0,\"account_subsidy_budget\":797,\"account_subsidy_decay\":347321},\"majority_version\":\"0.20.8\",\"max_voted_witnesses\":20,\"max_miner_witnesses\":0,\"max_runner_witnesses\":1,\"hardfork_required_witnesses\":17,\"account_subsidy_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":797,\"pool_eq\":157691079,\"max_pool_size\":157691079,\"decay_params\":{\"decay_per_time_unit\":347321,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":0},\"account_subsidy_witness_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":996,\"pool_eq\":9384019,\"max_pool_size\":9384019,\"decay_params\":{\"decay_per_time_unit\":7293741,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":257},\"min_witness_account_subsidy_decay\":0}"
|
214
234
|
```
|
215
235
|
|
@@ -230,111 +250,120 @@ See: https://redis.io/commands/scan#scan-basic-usage
|
|
230
250
|
Once your sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
|
231
251
|
|
232
252
|
```bash
|
233
|
-
redis-cli --scan --pattern '
|
253
|
+
redis-cli --scan --pattern 'hive:*:vote'
|
234
254
|
```
|
235
255
|
|
236
256
|
This returns the keys, for example:
|
237
257
|
|
238
258
|
```
|
239
|
-
|
240
|
-
|
241
|
-
|
242
|
-
|
243
|
-
|
244
|
-
|
245
|
-
|
246
|
-
|
247
|
-
|
259
|
+
hive:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
|
260
|
+
hive:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
|
261
|
+
hive:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
|
262
|
+
hive:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
|
263
|
+
hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
|
264
|
+
hive:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
|
265
|
+
hive:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
|
266
|
+
hive:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
|
267
|
+
hive:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
|
248
268
|
```
|
249
269
|
|
250
270
|
To get the actual vote operation for a particular key, use:
|
251
271
|
|
252
272
|
```bash
|
253
|
-
redis-cli get
|
273
|
+
redis-cli get hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
|
254
274
|
```
|
255
275
|
|
256
276
|
If, on the other hand, you want `custom_json` only:
|
257
277
|
|
258
278
|
```bash
|
259
|
-
redis-cli --scan --pattern '
|
279
|
+
redis-cli --scan --pattern 'hive:*:custom_json'
|
260
280
|
```
|
261
281
|
|
262
282
|
This only returns the related keys, for example:
|
263
283
|
|
264
284
|
```
|
265
|
-
|
266
|
-
|
267
|
-
|
268
|
-
|
269
|
-
|
270
|
-
|
285
|
+
hive:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
|
286
|
+
hive:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
|
287
|
+
hive:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
|
288
|
+
hive:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
|
289
|
+
hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
|
290
|
+
hive:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
|
271
291
|
```
|
272
292
|
|
273
293
|
To get the actual custom json operation for a particular key, use:
|
274
294
|
|
275
295
|
```bash
|
276
|
-
redis-cli get
|
296
|
+
redis-cli get hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
|
277
297
|
```
|
278
298
|
|
279
299
|
To get all transactions for a particular block number:
|
280
300
|
|
281
301
|
```bash
|
282
|
-
redis-cli --scan --pattern '
|
302
|
+
redis-cli --scan --pattern 'hive:29811085:*'
|
283
303
|
```
|
284
304
|
|
285
305
|
Or to get all ops for a particular transaction:
|
286
306
|
|
287
307
|
```bash
|
288
|
-
redis-cli --scan --pattern '
|
308
|
+
redis-cli --scan --pattern 'hive:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
|
289
309
|
```
|
290
310
|
|
291
|
-
###
|
311
|
+
### Hive Engine Support
|
292
312
|
|
293
|
-
As of `v0.0.6`, meeseeker can also follow the
|
313
|
+
As of `v0.0.6`, meeseeker can also follow the Hive Engine side-chain. This is optional and requires a separate process.
|
294
314
|
|
295
|
-
To sync
|
315
|
+
To sync Hive Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
|
296
316
|
|
297
317
|
```bash
|
298
|
-
meeseeker sync
|
299
|
-
```
|
300
|
-
|
301
|
-
When running `meeseeker sync
|
302
|
-
|
303
|
-
* `
|
304
|
-
* `
|
305
|
-
* `
|
306
|
-
* `
|
307
|
-
* `
|
308
|
-
* `
|
309
|
-
* `
|
310
|
-
* `
|
311
|
-
* `
|
312
|
-
* `
|
313
|
-
* `
|
314
|
-
* `
|
315
|
-
* `
|
316
|
-
* `
|
317
|
-
* `
|
318
|
-
* `
|
319
|
-
* `
|
320
|
-
* `
|
318
|
+
meeseeker sync hive_engine
|
319
|
+
```
|
320
|
+
|
321
|
+
When running `meeseeker sync hive_engine`, the following channels are available:
|
322
|
+
|
323
|
+
* `hive_engine:block`
|
324
|
+
* `hive_engine:transaction`
|
325
|
+
* `hive_engine:virtual_transaction`
|
326
|
+
* `hive_engine:contract`
|
327
|
+
* `hive_engine:contract:deploy`
|
328
|
+
* `hive_engine:contract:update`
|
329
|
+
* `hive_engine:market`
|
330
|
+
* `hive_engine:market:buy`
|
331
|
+
* `hive_engine:market:cancel`
|
332
|
+
* `hive_engine:market:sell`
|
333
|
+
* `hive_engine:sscstore`
|
334
|
+
* `hive_engine:sscstore:buy`
|
335
|
+
* `hive_engine:steempegged`
|
336
|
+
* `hive_engine:steempegged:buy`
|
337
|
+
* `hive_engine:steempegged:removeWithdrawal`
|
338
|
+
* `hive_engine:steempegged:withdraw`
|
339
|
+
* `hive_engine:tokens`
|
340
|
+
* `hive_engine:tokens:checkPendingUnstake`
|
341
|
+
* `hive_engine:tokens:create`
|
342
|
+
* `hive_engine:tokens:enableStaking`
|
343
|
+
* `hive_engine:tokens:issue`
|
344
|
+
* `hive_engine:tokens:transfer`
|
345
|
+
* `hive_engine:tokens:transferOwnership`
|
346
|
+
* `hive_engine:tokens:unstake`
|
347
|
+
* `hive_engine:tokens:updateMetadata`
|
348
|
+
* `hive_engine:tokens:updateParams`
|
349
|
+
* `hive_engine:tokens:updateUrl`
|
321
350
|
|
322
351
|
The above "channel/action" patterns are the ones that are known that the time of writing. In addition, if a new contract is added or updated, meeseeker will automatically publish to these corresponding channels as they appear, without needing to update or even restart meeseeker.
|
323
352
|
|
324
353
|
See main section on [Using `SUBSCRIBE`](#using-subscribe).
|
325
354
|
|
326
|
-
Once your
|
355
|
+
Once your HiveEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
|
327
356
|
|
328
357
|
```bash
|
329
|
-
redis-cli --scan --pattern '
|
358
|
+
redis-cli --scan --pattern 'hive_engine:*:tokens:transfer'
|
330
359
|
```
|
331
360
|
|
332
361
|
This returns the keys, for example:
|
333
362
|
|
334
363
|
```
|
335
|
-
|
336
|
-
|
337
|
-
|
364
|
+
hive_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
|
365
|
+
hive_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
|
366
|
+
hive_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
|
338
367
|
```
|
339
368
|
|
340
369
|
See main section on [Using `SCAN`](#using-scan).
|
@@ -365,11 +394,11 @@ Also see: https://hub.docker.com/r/inertia/meeseeker/
|
|
365
394
|
<img src="https://i.imgur.com/Y3Sa2GW.jpg" />
|
366
395
|
</center>
|
367
396
|
|
368
|
-
See some of my previous Ruby How To posts in: [#radiator](https://
|
397
|
+
See some of my previous Ruby How To posts in: [#radiator](https://hive.blog/created/radiator) [#ruby](https://hive.blog/created/ruby)
|
369
398
|
|
370
399
|
## Get in touch!
|
371
400
|
|
372
|
-
If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on
|
401
|
+
If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on Hive.
|
373
402
|
|
374
403
|
## License
|
375
404
|
|
data/Rakefile
CHANGED
@@ -52,29 +52,35 @@ task :check_schema do
|
|
52
52
|
end
|
53
53
|
|
54
54
|
task(:sync, [:chain, :at_block_num] => [:check_schema]) do |t, args|
|
55
|
-
chain =
|
55
|
+
chain = args[:chain] if args[:chain]
|
56
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
56
57
|
|
57
|
-
job = case chain
|
58
|
-
when :steem
|
59
|
-
Meeseeker::BlockFollowerJob.new
|
58
|
+
job = case chain.to_sym
|
60
59
|
when :steem_engine
|
61
60
|
Meeseeker::SteemEngine::FollowerJob.new
|
62
|
-
|
61
|
+
when :hive_engine
|
62
|
+
Meeseeker::HiveEngine::FollowerJob.new
|
63
|
+
else
|
64
|
+
Meeseeker::BlockFollowerJob.new
|
63
65
|
end
|
64
66
|
|
65
|
-
job.perform(at_block_num: args[:at_block_num])
|
67
|
+
job.perform(chain: chain, at_block_num: args[:at_block_num])
|
66
68
|
end
|
67
69
|
|
68
70
|
namespace :witness do
|
69
|
-
desc 'Publish the witness schedule every minute or so (
|
70
|
-
task :schedule do
|
71
|
+
desc 'Publish the witness schedule every minute or so (e.g.: hive:witness:schedule).'
|
72
|
+
task :schedule, [:chain] do |t, args|
|
73
|
+
chain = args[:chain] if args[:chain]
|
74
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
75
|
+
|
71
76
|
job = Meeseeker::WitnessScheduleJob.new
|
72
|
-
job.perform
|
77
|
+
job.perform(chain: chain)
|
73
78
|
end
|
74
79
|
end
|
75
80
|
|
76
81
|
task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
|
77
|
-
chain =
|
82
|
+
chain = args[:chain] if args[:chain]
|
83
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
78
84
|
redis = Meeseeker.redis
|
79
85
|
|
80
86
|
match = case args[:what].downcase.to_sym
|
@@ -99,8 +105,15 @@ task :reset, [:chain] => [:check_schema] do |t, args|
|
|
99
105
|
print 'Dropping keys for set: %s ...' % chain.to_s
|
100
106
|
|
101
107
|
case chain
|
102
|
-
when :
|
103
|
-
when :
|
108
|
+
when :steem_engine then keys += Meeseeker.redis.keys('steem_engine:*')
|
109
|
+
when :hive_engine then keys += Meeseeker.redis.keys('hive_engine:*')
|
110
|
+
when :all
|
111
|
+
keys += Meeseeker.redis.keys('steem:*')
|
112
|
+
keys += Meeseeker.redis.keys('hive:*')
|
113
|
+
keys += Meeseeker.redis.keys('steem_engine:*')
|
114
|
+
keys += Meeseeker.redis.keys('hive_engine:*')
|
115
|
+
else
|
116
|
+
keys += Meeseeker.redis.keys("#{chain}:*")
|
104
117
|
end
|
105
118
|
|
106
119
|
if keys.any?
|
@@ -114,10 +127,13 @@ end
|
|
114
127
|
|
115
128
|
namespace :verify do
|
116
129
|
desc 'Verifies transactions land where they should.'
|
117
|
-
task :block_org, [:max_blocks] do |t, args|
|
130
|
+
task :block_org, [:chain, :max_blocks] do |t, args|
|
131
|
+
chain = args[:chain] if args[:chain]
|
132
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
133
|
+
chain_key_prefix = chain.to_s
|
118
134
|
max_blocks = args[:max_blocks]
|
119
|
-
node_url =
|
120
|
-
database_api =
|
135
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
136
|
+
database_api = Meeseeker.database_api_class(chain).new(url: node_url)
|
121
137
|
mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
|
122
138
|
until_block_num = if !!max_blocks
|
123
139
|
database_api.get_dynamic_global_properties do |dgpo|
|
@@ -128,7 +144,7 @@ namespace :verify do
|
|
128
144
|
when :irreversible then dgpo.last_irreversible_block_num
|
129
145
|
else; abort "Unknown block mode: #{mode}"
|
130
146
|
end
|
131
|
-
end + max_blocks.to_i
|
147
|
+
end + max_blocks.to_i + 1
|
132
148
|
end
|
133
149
|
|
134
150
|
Thread.new do
|
@@ -136,7 +152,7 @@ namespace :verify do
|
|
136
152
|
|
137
153
|
loop do
|
138
154
|
begin
|
139
|
-
job.perform(mode: mode, until_block_num: until_block_num)
|
155
|
+
job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
|
140
156
|
rescue => e
|
141
157
|
puts e.inspect
|
142
158
|
sleep 5
|
@@ -149,8 +165,8 @@ namespace :verify do
|
|
149
165
|
end
|
150
166
|
|
151
167
|
begin
|
152
|
-
block_api =
|
153
|
-
block_channel =
|
168
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
169
|
+
block_channel = "#{chain_key_prefix}:block"
|
154
170
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
155
171
|
subscription = Redis.new(url: redis_url)
|
156
172
|
ctx = Redis.new(url: redis_url)
|
@@ -183,14 +199,21 @@ namespace :verify do
|
|
183
199
|
end
|
184
200
|
end
|
185
201
|
|
186
|
-
|
202
|
+
5.times do
|
203
|
+
break unless ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
204
|
+
|
187
205
|
# This ensures at least the next block has been indexed before
|
188
206
|
# proceeding.
|
189
207
|
|
190
|
-
puts "Waiting for block: #{next_block_num} ..."
|
208
|
+
puts "Waiting for block (verify:block_org): #{next_block_num} ..."
|
209
|
+
|
191
210
|
sleep 6
|
192
211
|
end
|
193
212
|
|
213
|
+
if ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
214
|
+
puts "Gave up waiting for block (check current_aslot slippage): #{next_block_num}"
|
215
|
+
end
|
216
|
+
|
194
217
|
database_api.get_dynamic_global_properties do |dgpo|
|
195
218
|
raise 'Got empty dynamic_global_properties result.' if dgpo.nil?
|
196
219
|
|
@@ -204,7 +227,7 @@ namespace :verify do
|
|
204
227
|
end
|
205
228
|
|
206
229
|
# In theory, we should have all the keys using this pattern.
|
207
|
-
keys = ctx.keys("
|
230
|
+
keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
|
208
231
|
|
209
232
|
# If we have all the keys, we should also have all transaction ids.
|
210
233
|
expected_ids = keys.map { |k| k.split(':')[2] }.uniq
|
@@ -249,18 +272,35 @@ namespace :verify do
|
|
249
272
|
end
|
250
273
|
end
|
251
274
|
|
275
|
+
desc 'Verifies Hive Engine transactions land where they should.'
|
276
|
+
task :hive_engine_block_org, [:max_blocks] do |t, args|
|
277
|
+
Rake::Task['verify:engine_block_org'].invoke('hive_engine', args[:max_blocks])
|
278
|
+
end
|
279
|
+
|
252
280
|
desc 'Verifies Steem Engine transactions land where they should.'
|
253
281
|
task :steem_engine_block_org, [:max_blocks] do |t, args|
|
282
|
+
Rake::Task['verify:engine_block_org'].invoke('steem_engine', args[:max_blocks])
|
283
|
+
end
|
284
|
+
|
285
|
+
desc 'Verifies Steem/Hive Engine transactions land where they should.'
|
286
|
+
task :engine_block_org, [:chain_key_prefix, :max_blocks] do |t, args|
|
287
|
+
chain_key_prefix = args[:chain_key_prefix]
|
254
288
|
max_blocks = args[:max_blocks]
|
255
|
-
|
256
|
-
|
289
|
+
case chain_key_prefix.to_sym
|
290
|
+
when :steem_engine
|
291
|
+
node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
|
292
|
+
agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
|
293
|
+
job = Meeseeker::SteemEngine::FollowerJob.new
|
294
|
+
when :hive_engine
|
295
|
+
node_url = ENV.fetch('MEESEEKER_HIVE_ENGINE_NODE_URL', 'https://api.hive-engine.com/rpc')
|
296
|
+
agent = Meeseeker::HiveEngine::Agent.new(url: node_url)
|
297
|
+
job = Meeseeker::HiveEngine::FollowerJob.new
|
298
|
+
end
|
257
299
|
until_block_num = if !!max_blocks
|
258
300
|
agent.latest_block_info['blockNumber']
|
259
301
|
end
|
260
302
|
|
261
303
|
Thread.new do
|
262
|
-
job = Meeseeker::SteemEngine::FollowerJob.new
|
263
|
-
|
264
304
|
loop do
|
265
305
|
begin
|
266
306
|
at_block_num = agent.latest_block_info["blockNumber"] - max_blocks.to_i
|
@@ -278,7 +318,7 @@ namespace :verify do
|
|
278
318
|
end
|
279
319
|
|
280
320
|
begin
|
281
|
-
block_channel =
|
321
|
+
block_channel = "#{chain_key_prefix}:block"
|
282
322
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
283
323
|
subscription = Redis.new(url: redis_url)
|
284
324
|
ctx = Redis.new(url: redis_url)
|
@@ -310,25 +350,26 @@ namespace :verify do
|
|
310
350
|
end
|
311
351
|
end
|
312
352
|
|
313
|
-
while ctx.keys("
|
353
|
+
while ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
314
354
|
# This ensures at least the next block has been indexed before
|
315
355
|
# proceeding.
|
316
356
|
|
317
|
-
puts "Waiting for block: #{next_block_num} ..."
|
357
|
+
puts "Waiting for block (verify:#{chain_key_prefix}_engine_block_org): #{next_block_num} ..."
|
318
358
|
sleep 6
|
319
359
|
end
|
320
360
|
|
321
361
|
# In theory, we should have all the keys using this pattern.
|
322
|
-
keys = ctx.keys("
|
362
|
+
keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
|
323
363
|
|
324
364
|
# If we have all the keys, we should also have all transaction ids.
|
325
365
|
expected_ids = keys.map { |k| k.split(':')[2] }.uniq
|
366
|
+
expected_ids -= [Meeseeker::VIRTUAL_TRX_ID]
|
326
367
|
actual_ids = nil
|
327
368
|
|
328
369
|
agent.block(block_num).tap do |block|
|
329
370
|
raise 'Got empty block result.' if block.nil?
|
330
371
|
|
331
|
-
actual_ids = block['transactions'].map{|trx| trx['transactionId'].split('-').first}.uniq
|
372
|
+
actual_ids = block['transactions'].map{|trx| trx['transactionId'].to_s.split('-').first}.uniq
|
332
373
|
end
|
333
374
|
|
334
375
|
# We do an intersection to make sure there's no difference between
|
@@ -360,30 +401,55 @@ namespace :verify do
|
|
360
401
|
end
|
361
402
|
end
|
362
403
|
end
|
404
|
+
|
405
|
+
agent.shutdown
|
406
|
+
end
|
407
|
+
|
408
|
+
desc 'Verifies Hive Engine sidechain against the mainnet.'
|
409
|
+
task :hive_engine_ref_blocks do |t|
|
410
|
+
Rake::Task['verify:engine_ref_blocks'].invoke('hive_engine')
|
363
411
|
end
|
364
412
|
|
365
413
|
desc 'Verifies Steem Engine sidechain against the mainnet.'
|
366
414
|
task :steem_engine_ref_blocks do |t|
|
415
|
+
Rake::Task['verify:engine_ref_blocks'].invoke('steem_engine')
|
416
|
+
end
|
417
|
+
|
418
|
+
desc 'Verifies Steem/Hive Engine sidechain against the mainnet.'
|
419
|
+
task :engine_ref_blocks, [:chain_key_prefix] do |t, args|
|
420
|
+
chain_key_prefix = args[:chain_key_prefix]
|
367
421
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
368
422
|
ctx = ctx = Redis.new(url: redis_url)
|
369
|
-
keys = ctx.keys(
|
370
|
-
|
423
|
+
keys = ctx.keys("#{chain_key_prefix}:*:*")
|
424
|
+
mainchain, mainchain_url = case chain_key_prefix
|
425
|
+
when 'steem_engine' then ['steem', 'https://api.steemit.com']
|
426
|
+
when 'hive_engine' then ['hive', 'https://api.openhive.network']
|
427
|
+
end
|
428
|
+
block_api = Meeseeker.block_api_class(mainchain).new(url: mainchain_url)
|
371
429
|
block_trxs = {}
|
372
430
|
|
373
|
-
puts "Checking
|
431
|
+
puts "Checking #{chain_key_prefix} keys: #{keys.size}"
|
374
432
|
|
375
433
|
keys.each do |key|
|
376
434
|
transaction = JSON[ctx.get(key)]
|
377
|
-
|
435
|
+
|
436
|
+
next if transaction.class == Integer
|
437
|
+
|
438
|
+
block_num = case chain_key_prefix
|
439
|
+
when 'steem_engine' then transaction.fetch('refSteemBlockNumber')
|
440
|
+
when 'hive_engine' then transaction.fetch('refHiveBlockNumber')
|
441
|
+
end.to_i
|
378
442
|
|
379
443
|
block_trxs[block_num] ||= []
|
380
|
-
block_trxs[block_num] << transaction['transactionId'].split('-').first
|
444
|
+
block_trxs[block_num] << transaction['transactionId'].to_s.split('-').first
|
381
445
|
end
|
382
446
|
|
383
447
|
puts "Related mainnet blocks: #{block_trxs.keys.size}"
|
384
448
|
|
385
449
|
skipped_blocks = []
|
386
450
|
|
451
|
+
next if block_trxs.empty?
|
452
|
+
|
387
453
|
block_api.get_blocks(block_range: block_trxs.keys) do |block, block_num|
|
388
454
|
if block.nil? || block[:transaction_ids].nil?
|
389
455
|
print 'S'
|
@@ -393,24 +459,34 @@ namespace :verify do
|
|
393
459
|
else
|
394
460
|
print '.'
|
395
461
|
end
|
396
|
-
|
397
|
-
|
462
|
+
|
463
|
+
trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
|
464
|
+
|
465
|
+
if trx_ids.any? && (block.transaction_ids & trx_ids).none?
|
398
466
|
puts "\nNo intersection in #{block_num}!"
|
399
|
-
puts "Expected the following sidechain trx_ids: #{
|
467
|
+
puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
|
400
468
|
end
|
401
469
|
end
|
402
470
|
|
403
471
|
puts "\nBlocks to retry: #{skipped_blocks.size}"
|
404
472
|
|
405
473
|
skipped_blocks.each do |block_num|
|
474
|
+
block_found = false
|
475
|
+
|
406
476
|
block_api.get_block(block_num: block_num) do |result|
|
477
|
+
break unless !!result.block
|
478
|
+
|
407
479
|
block = result.block
|
480
|
+
block_found = true
|
481
|
+
trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
|
408
482
|
|
409
|
-
if (block.transaction_ids &
|
483
|
+
if trx_ids.any? && (block.transaction_ids & trx_ids).none?
|
410
484
|
puts "No intersection in #{block_num}!"
|
411
|
-
puts "Expected the following sidechain trx_ids: #{
|
485
|
+
puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
|
412
486
|
end
|
413
487
|
end
|
488
|
+
|
489
|
+
redo unless block_found
|
414
490
|
end
|
415
491
|
|
416
492
|
puts "Done."
|
@@ -418,10 +494,12 @@ namespace :verify do
|
|
418
494
|
|
419
495
|
namespace :witness do
|
420
496
|
desc 'Verifies witnessses in the schedule produced a block.'
|
421
|
-
task :schedule, [:max_blocks] do |t, args|
|
497
|
+
task :schedule, [:chain, :max_blocks] do |t, args|
|
498
|
+
chain = args[:chain] if !!args[:chain]
|
499
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
422
500
|
max_blocks = args[:max_blocks]
|
423
|
-
node_url =
|
424
|
-
database_api =
|
501
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
502
|
+
database_api = Meeseeker.database_api_class(chain).new(url: node_url)
|
425
503
|
mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
|
426
504
|
until_block_num = if !!max_blocks
|
427
505
|
database_api.get_dynamic_global_properties do |dgpo|
|
@@ -440,7 +518,7 @@ namespace :verify do
|
|
440
518
|
|
441
519
|
loop do
|
442
520
|
begin
|
443
|
-
job.perform(mode: mode, until_block_num: until_block_num)
|
521
|
+
job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
|
444
522
|
rescue => e
|
445
523
|
puts e.inspect
|
446
524
|
sleep 5
|
@@ -453,11 +531,12 @@ namespace :verify do
|
|
453
531
|
end
|
454
532
|
|
455
533
|
begin
|
456
|
-
block_api =
|
457
|
-
|
534
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
535
|
+
chain_key_prefix = chain.to_s if args[:chain]
|
536
|
+
chain_key_prefix ||= Meeseeker.default_chain_key_prefix
|
537
|
+
schedule_channel = "#{chain_key_prefix}:witness:schedule"
|
458
538
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
459
539
|
subscription = Redis.new(url: redis_url)
|
460
|
-
ctx = Redis.new(url: redis_url)
|
461
540
|
timeout = (max_blocks).to_i * 3
|
462
541
|
|
463
542
|
subscribe_mode, subscribe_args = if timeout > 0
|
@@ -514,7 +593,10 @@ namespace :verify do
|
|
514
593
|
unless !!header
|
515
594
|
# Can happen when there's excess p2p latency and/or jussi
|
516
595
|
# cache is under load.
|
517
|
-
puts "Waiting for block header: #{block_num}"
|
596
|
+
puts "Waiting for block header (witness:schedule): #{block_num}"
|
597
|
+
|
598
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
599
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
518
600
|
|
519
601
|
next
|
520
602
|
end
|