meeseeker 0.0.8 → 2.0.1.pre.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/README.md +118 -89
- data/Rakefile +125 -45
- data/bin/meeseeker +2 -2
- data/lib/meeseeker.rb +178 -6
- data/lib/meeseeker/block_follower_job.rb +113 -65
- data/lib/meeseeker/hive_engine.rb +20 -0
- data/lib/meeseeker/steem_engine/agent.rb +33 -19
- data/lib/meeseeker/steem_engine/follower_job.rb +31 -14
- data/lib/meeseeker/version.rb +1 -1
- data/lib/meeseeker/witness_schedule_job.rb +6 -2
- data/meeseeker.gemspec +3 -1
- data/test/meeseeker/meeseeker_test.rb +261 -16
- data/test/test_helper.rb +11 -0
- metadata +66 -26
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: e6750a9b00a2ab979ca2944add9cdee2b31c16466821445c32e1ef9e7de638a0
|
4
|
+
data.tar.gz: 4489e40ea24703752ad9c359e1440fe4db66c20e3fe9e710f93f8657a49e50eb
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: fd4df8812f0252737c0a43fc1a0db029c2361277dc3d6074e6fd3e642c7fdbabbcbc3b2b30cd56f9ea0a103540ef1146601873c303c106516a8fb2a815ee1bb3
|
7
|
+
data.tar.gz: 94895f366e1c9ebc3e1c38e731130d19f4f39b231d1593c877a21085254171afdb7bf2f59ebafbed32f447af98eb188ccce3efdbeef95465c1e66079ffb09c6b
|
data/README.md
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
# meeseeker
|
2
2
|
|
3
|
-
Redis based block follower is an efficient way for multiple apps to stream the
|
3
|
+
Redis based block follower is an efficient way for multiple apps to stream the Hive Blockchain.
|
4
|
+
|
5
|
+
[](https://travis-ci.org/inertia186/meeseeker)
|
4
6
|
|
5
7
|
If you have multiple applications that need to perform actions as operations occur, `meeseeker` will allow your apps to each perform actions for specific operations without each app having to stream the entire blockchain.
|
6
8
|
|
@@ -62,10 +64,22 @@ To specify an alternative redis source:
|
|
62
64
|
MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync
|
63
65
|
```
|
64
66
|
|
65
|
-
You can also specify
|
67
|
+
You can also specify am alternative Hive node:
|
66
68
|
|
67
69
|
```bash
|
68
|
-
MEESEEKER_NODE_URL=
|
70
|
+
MEESEEKER_NODE_URL=http://anyx.io meeseeker sync
|
71
|
+
```
|
72
|
+
|
73
|
+
You can also specify a Steem node instead of Hive (if that's your thing):
|
74
|
+
|
75
|
+
```bash
|
76
|
+
MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync[steem]
|
77
|
+
```
|
78
|
+
|
79
|
+
Or, you can have meeseeker automatically use random Hive nodes:
|
80
|
+
|
81
|
+
```bash
|
82
|
+
MEESEEKER_NODE_URL=shuffle meeseeker sync
|
69
83
|
```
|
70
84
|
|
71
85
|
To sync from the head block instead of the last irreversible block:
|
@@ -80,7 +94,7 @@ To ignore virtual operations (useful if the node doesn't enable `get_ops_in_bloc
|
|
80
94
|
MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
|
81
95
|
```
|
82
96
|
|
83
|
-
Normally, block headers are added to the `
|
97
|
+
Normally, block headers are added to the `hive:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `hive:block` channel to only publish with the `block_num`:
|
84
98
|
|
85
99
|
```bash
|
86
100
|
MEESEEKER_INCLUDE_BLOCK_HEADER=false meeseeker sync
|
@@ -99,6 +113,12 @@ If you never want the keys to expire (not recommended), set
|
|
99
113
|
MEESEEKER_EXPIRE_KEYS=-1 meeseeker sync
|
100
114
|
```
|
101
115
|
|
116
|
+
Normally, sync will create keys until it uses up all available memory. If you would like to only sync a certain number of keys, then sleep until those keys expire so it can pick up where it left off, set `MEESEEKER_MAX_KEYS` to a positive value:
|
117
|
+
|
118
|
+
```bash
|
119
|
+
MEESEEKER_MAX_KEYS=99 meeseeker sync
|
120
|
+
```
|
121
|
+
|
102
122
|
### Usage
|
103
123
|
|
104
124
|
When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
|
@@ -111,35 +131,35 @@ For `redis-cli`, please see: https://redis.io/topics/pubsub
|
|
111
131
|
|
112
132
|
When running `meeseeker sync`, the following channels are available:
|
113
133
|
|
114
|
-
* `
|
115
|
-
* `
|
116
|
-
* `
|
117
|
-
* `
|
118
|
-
* `
|
119
|
-
* `
|
120
|
-
* `
|
134
|
+
* `hive:block`
|
135
|
+
* `hive:transaction`
|
136
|
+
* `hive:op:vote`
|
137
|
+
* `hive:op:comment`
|
138
|
+
* `hive:op:comment_options`
|
139
|
+
* `hive:op:whatever` (replace "whatever" with the op you want)
|
140
|
+
* `hive:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
|
121
141
|
|
122
|
-
As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.
|
142
|
+
As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.hive.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
|
123
143
|
|
124
|
-
In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `
|
144
|
+
In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `hive:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `hive:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
|
125
145
|
|
126
146
|
For example, from `redis-cli`, if we wanted to stream block numbers:
|
127
147
|
|
128
148
|
```bash
|
129
149
|
$ redis-cli
|
130
|
-
127.0.0.1:6379> subscribe
|
150
|
+
127.0.0.1:6379> subscribe hive:block
|
131
151
|
Reading messages... (press Ctrl-C to quit)
|
132
152
|
1) "subscribe"
|
133
|
-
2) "
|
153
|
+
2) "hive:block"
|
134
154
|
3) (integer) 1
|
135
155
|
1) "message"
|
136
|
-
2) "
|
156
|
+
2) "hive:block"
|
137
157
|
3) "{\"block_num\":29861068,\"previous\":\"01c7a4cb4424b4dc0cb0cc72fd36b1644f8aeba5\",\"timestamp\":\"2019-01-28T20:55:03\",\"witness\":\"ausbitbank\",\"transaction_merkle_root\":\"a318bb82625bd78af8d8b506ccd4f53116372c8e\",\"extensions\":[]}"
|
138
158
|
1) "message"
|
139
|
-
2) "
|
159
|
+
2) "hive:block"
|
140
160
|
3) "{\"block_num\":29861069,\"previous\":\"01c7a4cc1bed060876cab57476846a91568a9f8a\",\"timestamp\":\"2019-01-28T20:55:06\",\"witness\":\"followbtcnews\",\"transaction_merkle_root\":\"834e05d40b9666e5ef50deb9f368c63070c0105b\",\"extensions\":[]}"
|
141
161
|
1) "message"
|
142
|
-
2) "
|
162
|
+
2) "hive:block"
|
143
163
|
3) "{\"block_num\":29861070,\"previous\":\"01c7a4cd3bbf872895654765faa4409a8e770e91\",\"timestamp\":\"2019-01-28T20:55:09\",\"witness\":\"timcliff\",\"transaction_merkle_root\":\"b2366ce9134d627e00423b28d33cc57f1e6e453f\",\"extensions\":[]}"
|
144
164
|
```
|
145
165
|
|
@@ -153,26 +173,26 @@ Which allows subscription to specific `id` patterns:
|
|
153
173
|
|
154
174
|
```
|
155
175
|
$ redis-cli
|
156
|
-
127.0.0.1:6379> subscribe
|
176
|
+
127.0.0.1:6379> subscribe hive:op:custom_json:sm_team_reveal
|
157
177
|
Reading messages... (press Ctrl-C to quit)
|
158
178
|
1) "subscribe"
|
159
|
-
2) "
|
179
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
160
180
|
3) (integer) 1
|
161
181
|
1) "message"
|
162
|
-
2) "
|
163
|
-
3) "{\"key\":\"
|
182
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
183
|
+
3) "{\"key\":\"hive:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
|
164
184
|
1) "message"
|
165
|
-
2) "
|
166
|
-
3) "{\"key\":\"
|
185
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
186
|
+
3) "{\"key\":\"hive:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
|
167
187
|
1) "message"
|
168
|
-
2) "
|
169
|
-
3) "{\"key\":\"
|
188
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
189
|
+
3) "{\"key\":\"hive:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
|
170
190
|
1) "message"
|
171
|
-
2) "
|
172
|
-
3) "{\"key\":\"
|
191
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
192
|
+
3) "{\"key\":\"hive:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
|
173
193
|
1) "message"
|
174
|
-
2) "
|
175
|
-
3) "{\"key\":\"
|
194
|
+
2) "hive:op:custom_json:sm_team_reveal"
|
195
|
+
3) "{\"key\":\"hive:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
|
176
196
|
```
|
177
197
|
|
178
198
|
A `ruby` application can subscribe to a channel as well, using the `redis` gem:
|
@@ -183,7 +203,7 @@ require 'redis'
|
|
183
203
|
url = 'redis://127.0.0.1:6379/0'
|
184
204
|
ctx = Redis.new(url: url)
|
185
205
|
|
186
|
-
Redis.new(url: url).subscribe('
|
206
|
+
Redis.new(url: url).subscribe('hive:op:comment') do |on|
|
187
207
|
on.message do |channel, message|
|
188
208
|
payload = JSON[message]
|
189
209
|
comment = JSON[ctx.get(payload['key'])]
|
@@ -197,19 +217,19 @@ Many other clients are supported: https://redis.io/clients
|
|
197
217
|
|
198
218
|
##### Witness Schedule
|
199
219
|
|
200
|
-
When running `meeseeker witness:schedule`, the `
|
220
|
+
When running `meeseeker witness:schedule`, the `hive:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
|
201
221
|
|
202
222
|
For example, from `redis-cli`, if we wanted to subscribe to the witness schedule:
|
203
223
|
|
204
224
|
```
|
205
225
|
$ redis-cli
|
206
|
-
127.0.0.1:6379> subscribe
|
226
|
+
127.0.0.1:6379> subscribe hive:witness:schedule
|
207
227
|
Reading messages... (press Ctrl-C to quit)
|
208
228
|
1) "subscribe"
|
209
|
-
2) "
|
229
|
+
2) "hive:witness:schedule"
|
210
230
|
3) (integer) 1
|
211
231
|
1) "message"
|
212
|
-
2) "
|
232
|
+
2) "hive:witness:schedule"
|
213
233
|
3) "{\"id\":0,\"current_virtual_time\":\"415293532210075480213212125\",\"next_shuffle_block_num\":30035208,\"current_shuffled_witnesses\":[\"thecryptodrive\",\"timcliff\",\"utopian-io\",\"themarkymark\",\"aggroed\",\"smooth.witness\",\"someguy123\",\"gtg\",\"followbtcnews\",\"yabapmatt\",\"therealwolf\",\"ausbitbank\",\"curie\",\"clayop\",\"drakos\",\"blocktrades\",\"good-karma\",\"roelandp\",\"lukestokes.mhth\",\"liondani\",\"anyx\"],\"num_scheduled_witnesses\":21,\"elected_weight\":1,\"timeshare_weight\":5,\"miner_weight\":1,\"witness_pay_normalization_factor\":25,\"median_props\":{\"account_creation_fee\":{\"amount\":\"3000\",\"precision\":3,\"nai\":\"@@000000021\"},\"maximum_block_size\":65536,\"sbd_interest_rate\":0,\"account_subsidy_budget\":797,\"account_subsidy_decay\":347321},\"majority_version\":\"0.20.8\",\"max_voted_witnesses\":20,\"max_miner_witnesses\":0,\"max_runner_witnesses\":1,\"hardfork_required_witnesses\":17,\"account_subsidy_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":797,\"pool_eq\":157691079,\"max_pool_size\":157691079,\"decay_params\":{\"decay_per_time_unit\":347321,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":0},\"account_subsidy_witness_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":996,\"pool_eq\":9384019,\"max_pool_size\":9384019,\"decay_params\":{\"decay_per_time_unit\":7293741,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":257},\"min_witness_account_subsidy_decay\":0}"
|
214
234
|
```
|
215
235
|
|
@@ -230,111 +250,120 @@ See: https://redis.io/commands/scan#scan-basic-usage
|
|
230
250
|
Once your sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
|
231
251
|
|
232
252
|
```bash
|
233
|
-
redis-cli --scan --pattern '
|
253
|
+
redis-cli --scan --pattern 'hive:*:vote'
|
234
254
|
```
|
235
255
|
|
236
256
|
This returns the keys, for example:
|
237
257
|
|
238
258
|
```
|
239
|
-
|
240
|
-
|
241
|
-
|
242
|
-
|
243
|
-
|
244
|
-
|
245
|
-
|
246
|
-
|
247
|
-
|
259
|
+
hive:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
|
260
|
+
hive:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
|
261
|
+
hive:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
|
262
|
+
hive:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
|
263
|
+
hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
|
264
|
+
hive:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
|
265
|
+
hive:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
|
266
|
+
hive:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
|
267
|
+
hive:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
|
248
268
|
```
|
249
269
|
|
250
270
|
To get the actual vote operation for a particular key, use:
|
251
271
|
|
252
272
|
```bash
|
253
|
-
redis-cli get
|
273
|
+
redis-cli get hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
|
254
274
|
```
|
255
275
|
|
256
276
|
If, on the other hand, you want `custom_json` only:
|
257
277
|
|
258
278
|
```bash
|
259
|
-
redis-cli --scan --pattern '
|
279
|
+
redis-cli --scan --pattern 'hive:*:custom_json'
|
260
280
|
```
|
261
281
|
|
262
282
|
This only returns the related keys, for example:
|
263
283
|
|
264
284
|
```
|
265
|
-
|
266
|
-
|
267
|
-
|
268
|
-
|
269
|
-
|
270
|
-
|
285
|
+
hive:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
|
286
|
+
hive:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
|
287
|
+
hive:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
|
288
|
+
hive:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
|
289
|
+
hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
|
290
|
+
hive:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
|
271
291
|
```
|
272
292
|
|
273
293
|
To get the actual custom json operation for a particular key, use:
|
274
294
|
|
275
295
|
```bash
|
276
|
-
redis-cli get
|
296
|
+
redis-cli get hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
|
277
297
|
```
|
278
298
|
|
279
299
|
To get all transactions for a particular block number:
|
280
300
|
|
281
301
|
```bash
|
282
|
-
redis-cli --scan --pattern '
|
302
|
+
redis-cli --scan --pattern 'hive:29811085:*'
|
283
303
|
```
|
284
304
|
|
285
305
|
Or to get all ops for a particular transaction:
|
286
306
|
|
287
307
|
```bash
|
288
|
-
redis-cli --scan --pattern '
|
308
|
+
redis-cli --scan --pattern 'hive:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
|
289
309
|
```
|
290
310
|
|
291
|
-
###
|
311
|
+
### Hive Engine Support
|
292
312
|
|
293
|
-
As of `v0.0.6`, meeseeker can also follow the
|
313
|
+
As of `v0.0.6`, meeseeker can also follow the Hive Engine side-chain. This is optional and requires a separate process.
|
294
314
|
|
295
|
-
To sync
|
315
|
+
To sync Hive Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
|
296
316
|
|
297
317
|
```bash
|
298
|
-
meeseeker sync
|
299
|
-
```
|
300
|
-
|
301
|
-
When running `meeseeker sync
|
302
|
-
|
303
|
-
* `
|
304
|
-
* `
|
305
|
-
* `
|
306
|
-
* `
|
307
|
-
* `
|
308
|
-
* `
|
309
|
-
* `
|
310
|
-
* `
|
311
|
-
* `
|
312
|
-
* `
|
313
|
-
* `
|
314
|
-
* `
|
315
|
-
* `
|
316
|
-
* `
|
317
|
-
* `
|
318
|
-
* `
|
319
|
-
* `
|
320
|
-
* `
|
318
|
+
meeseeker sync hive_engine
|
319
|
+
```
|
320
|
+
|
321
|
+
When running `meeseeker sync hive_engine`, the following channels are available:
|
322
|
+
|
323
|
+
* `hive_engine:block`
|
324
|
+
* `hive_engine:transaction`
|
325
|
+
* `hive_engine:virtual_transaction`
|
326
|
+
* `hive_engine:contract`
|
327
|
+
* `hive_engine:contract:deploy`
|
328
|
+
* `hive_engine:contract:update`
|
329
|
+
* `hive_engine:market`
|
330
|
+
* `hive_engine:market:buy`
|
331
|
+
* `hive_engine:market:cancel`
|
332
|
+
* `hive_engine:market:sell`
|
333
|
+
* `hive_engine:sscstore`
|
334
|
+
* `hive_engine:sscstore:buy`
|
335
|
+
* `hive_engine:steempegged`
|
336
|
+
* `hive_engine:steempegged:buy`
|
337
|
+
* `hive_engine:steempegged:removeWithdrawal`
|
338
|
+
* `hive_engine:steempegged:withdraw`
|
339
|
+
* `hive_engine:tokens`
|
340
|
+
* `hive_engine:tokens:checkPendingUnstake`
|
341
|
+
* `hive_engine:tokens:create`
|
342
|
+
* `hive_engine:tokens:enableStaking`
|
343
|
+
* `hive_engine:tokens:issue`
|
344
|
+
* `hive_engine:tokens:transfer`
|
345
|
+
* `hive_engine:tokens:transferOwnership`
|
346
|
+
* `hive_engine:tokens:unstake`
|
347
|
+
* `hive_engine:tokens:updateMetadata`
|
348
|
+
* `hive_engine:tokens:updateParams`
|
349
|
+
* `hive_engine:tokens:updateUrl`
|
321
350
|
|
322
351
|
The above "channel/action" patterns are the ones that are known that the time of writing. In addition, if a new contract is added or updated, meeseeker will automatically publish to these corresponding channels as they appear, without needing to update or even restart meeseeker.
|
323
352
|
|
324
353
|
See main section on [Using `SUBSCRIBE`](#using-subscribe).
|
325
354
|
|
326
|
-
Once your
|
355
|
+
Once your HiveEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
|
327
356
|
|
328
357
|
```bash
|
329
|
-
redis-cli --scan --pattern '
|
358
|
+
redis-cli --scan --pattern 'hive_engine:*:tokens:transfer'
|
330
359
|
```
|
331
360
|
|
332
361
|
This returns the keys, for example:
|
333
362
|
|
334
363
|
```
|
335
|
-
|
336
|
-
|
337
|
-
|
364
|
+
hive_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
|
365
|
+
hive_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
|
366
|
+
hive_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
|
338
367
|
```
|
339
368
|
|
340
369
|
See main section on [Using `SCAN`](#using-scan).
|
@@ -365,11 +394,11 @@ Also see: https://hub.docker.com/r/inertia/meeseeker/
|
|
365
394
|
<img src="https://i.imgur.com/Y3Sa2GW.jpg" />
|
366
395
|
</center>
|
367
396
|
|
368
|
-
See some of my previous Ruby How To posts in: [#radiator](https://
|
397
|
+
See some of my previous Ruby How To posts in: [#radiator](https://hive.blog/created/radiator) [#ruby](https://hive.blog/created/ruby)
|
369
398
|
|
370
399
|
## Get in touch!
|
371
400
|
|
372
|
-
If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on
|
401
|
+
If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on Hive.
|
373
402
|
|
374
403
|
## License
|
375
404
|
|
data/Rakefile
CHANGED
@@ -51,30 +51,38 @@ task :check_schema do
|
|
51
51
|
end
|
52
52
|
end
|
53
53
|
|
54
|
+
desc 'Begin/resume sync.'
|
54
55
|
task(:sync, [:chain, :at_block_num] => [:check_schema]) do |t, args|
|
55
|
-
chain =
|
56
|
+
chain = args[:chain] if args[:chain]
|
57
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
56
58
|
|
57
|
-
job = case chain
|
58
|
-
when :steem
|
59
|
-
Meeseeker::BlockFollowerJob.new
|
59
|
+
job = case chain.to_sym
|
60
60
|
when :steem_engine
|
61
61
|
Meeseeker::SteemEngine::FollowerJob.new
|
62
|
-
|
62
|
+
when :hive_engine
|
63
|
+
Meeseeker::HiveEngine::FollowerJob.new
|
64
|
+
else
|
65
|
+
Meeseeker::BlockFollowerJob.new
|
63
66
|
end
|
64
67
|
|
65
|
-
job.perform(at_block_num: args[:at_block_num])
|
68
|
+
job.perform(chain: chain, at_block_num: args[:at_block_num])
|
66
69
|
end
|
67
70
|
|
68
71
|
namespace :witness do
|
69
|
-
desc 'Publish the witness schedule every minute or so (
|
70
|
-
task :schedule do
|
72
|
+
desc 'Publish the witness schedule every minute or so (e.g.: hive:witness:schedule).'
|
73
|
+
task :schedule, [:chain] do |t, args|
|
74
|
+
chain = args[:chain] if args[:chain]
|
75
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
76
|
+
|
71
77
|
job = Meeseeker::WitnessScheduleJob.new
|
72
|
-
job.perform
|
78
|
+
job.perform(chain: chain)
|
73
79
|
end
|
74
80
|
end
|
75
81
|
|
82
|
+
desc 'Find block or transaction.'
|
76
83
|
task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
|
77
|
-
chain =
|
84
|
+
chain = args[:chain] if args[:chain]
|
85
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
78
86
|
redis = Meeseeker.redis
|
79
87
|
|
80
88
|
match = case args[:what].downcase.to_sym
|
@@ -92,6 +100,7 @@ task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
|
|
92
100
|
end
|
93
101
|
end
|
94
102
|
|
103
|
+
desc 'Clear all keys.'
|
95
104
|
task :reset, [:chain] => [:check_schema] do |t, args|
|
96
105
|
chain = (args[:chain] || 'all').to_sym
|
97
106
|
keys = []
|
@@ -99,11 +108,15 @@ task :reset, [:chain] => [:check_schema] do |t, args|
|
|
99
108
|
print 'Dropping keys for set: %s ...' % chain.to_s
|
100
109
|
|
101
110
|
case chain
|
102
|
-
when :steem then keys += Meeseeker.redis.keys('steem:*')
|
103
111
|
when :steem_engine then keys += Meeseeker.redis.keys('steem_engine:*')
|
112
|
+
when :hive_engine then keys += Meeseeker.redis.keys('hive_engine:*')
|
104
113
|
when :all
|
105
114
|
keys += Meeseeker.redis.keys('steem:*')
|
115
|
+
keys += Meeseeker.redis.keys('hive:*')
|
106
116
|
keys += Meeseeker.redis.keys('steem_engine:*')
|
117
|
+
keys += Meeseeker.redis.keys('hive_engine:*')
|
118
|
+
else
|
119
|
+
keys += Meeseeker.redis.keys("#{chain}:*")
|
107
120
|
end
|
108
121
|
|
109
122
|
if keys.any?
|
@@ -117,10 +130,13 @@ end
|
|
117
130
|
|
118
131
|
namespace :verify do
|
119
132
|
desc 'Verifies transactions land where they should.'
|
120
|
-
task :block_org, [:max_blocks] do |t, args|
|
133
|
+
task :block_org, [:chain, :max_blocks] do |t, args|
|
134
|
+
chain = args[:chain] if args[:chain]
|
135
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
136
|
+
chain_key_prefix = chain.to_s
|
121
137
|
max_blocks = args[:max_blocks]
|
122
|
-
node_url =
|
123
|
-
database_api =
|
138
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
139
|
+
database_api = Meeseeker.database_api_class(chain).new(url: node_url)
|
124
140
|
mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
|
125
141
|
until_block_num = if !!max_blocks
|
126
142
|
database_api.get_dynamic_global_properties do |dgpo|
|
@@ -131,7 +147,7 @@ namespace :verify do
|
|
131
147
|
when :irreversible then dgpo.last_irreversible_block_num
|
132
148
|
else; abort "Unknown block mode: #{mode}"
|
133
149
|
end
|
134
|
-
end + max_blocks.to_i
|
150
|
+
end + max_blocks.to_i + 1
|
135
151
|
end
|
136
152
|
|
137
153
|
Thread.new do
|
@@ -139,7 +155,7 @@ namespace :verify do
|
|
139
155
|
|
140
156
|
loop do
|
141
157
|
begin
|
142
|
-
job.perform(mode: mode, until_block_num: until_block_num)
|
158
|
+
job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
|
143
159
|
rescue => e
|
144
160
|
puts e.inspect
|
145
161
|
sleep 5
|
@@ -152,8 +168,8 @@ namespace :verify do
|
|
152
168
|
end
|
153
169
|
|
154
170
|
begin
|
155
|
-
block_api =
|
156
|
-
block_channel =
|
171
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
172
|
+
block_channel = "#{chain_key_prefix}:block"
|
157
173
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
158
174
|
subscription = Redis.new(url: redis_url)
|
159
175
|
ctx = Redis.new(url: redis_url)
|
@@ -186,14 +202,21 @@ namespace :verify do
|
|
186
202
|
end
|
187
203
|
end
|
188
204
|
|
189
|
-
|
205
|
+
5.times do
|
206
|
+
break unless ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
207
|
+
|
190
208
|
# This ensures at least the next block has been indexed before
|
191
209
|
# proceeding.
|
192
210
|
|
193
|
-
puts "Waiting for block: #{next_block_num} ..."
|
211
|
+
puts "Waiting for block (verify:block_org): #{next_block_num} ..."
|
212
|
+
|
194
213
|
sleep 6
|
195
214
|
end
|
196
215
|
|
216
|
+
if ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
217
|
+
puts "Gave up waiting for block (check current_aslot slippage): #{next_block_num}"
|
218
|
+
end
|
219
|
+
|
197
220
|
database_api.get_dynamic_global_properties do |dgpo|
|
198
221
|
raise 'Got empty dynamic_global_properties result.' if dgpo.nil?
|
199
222
|
|
@@ -207,7 +230,7 @@ namespace :verify do
|
|
207
230
|
end
|
208
231
|
|
209
232
|
# In theory, we should have all the keys using this pattern.
|
210
|
-
keys = ctx.keys("
|
233
|
+
keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
|
211
234
|
|
212
235
|
# If we have all the keys, we should also have all transaction ids.
|
213
236
|
expected_ids = keys.map { |k| k.split(':')[2] }.uniq
|
@@ -252,18 +275,35 @@ namespace :verify do
|
|
252
275
|
end
|
253
276
|
end
|
254
277
|
|
278
|
+
desc 'Verifies Hive Engine transactions land where they should.'
|
279
|
+
task :hive_engine_block_org, [:max_blocks] do |t, args|
|
280
|
+
Rake::Task['verify:engine_block_org'].invoke('hive_engine', args[:max_blocks])
|
281
|
+
end
|
282
|
+
|
255
283
|
desc 'Verifies Steem Engine transactions land where they should.'
|
256
284
|
task :steem_engine_block_org, [:max_blocks] do |t, args|
|
285
|
+
Rake::Task['verify:engine_block_org'].invoke('steem_engine', args[:max_blocks])
|
286
|
+
end
|
287
|
+
|
288
|
+
desc 'Verifies Steem/Hive Engine transactions land where they should.'
|
289
|
+
task :engine_block_org, [:chain_key_prefix, :max_blocks] do |t, args|
|
290
|
+
chain_key_prefix = args[:chain_key_prefix]
|
257
291
|
max_blocks = args[:max_blocks]
|
258
|
-
|
259
|
-
|
292
|
+
case chain_key_prefix.to_sym
|
293
|
+
when :steem_engine
|
294
|
+
node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
|
295
|
+
agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
|
296
|
+
job = Meeseeker::SteemEngine::FollowerJob.new
|
297
|
+
when :hive_engine
|
298
|
+
node_url = ENV.fetch('MEESEEKER_HIVE_ENGINE_NODE_URL', 'https://api.hive-engine.com/rpc')
|
299
|
+
agent = Meeseeker::HiveEngine::Agent.new(url: node_url)
|
300
|
+
job = Meeseeker::HiveEngine::FollowerJob.new
|
301
|
+
end
|
260
302
|
until_block_num = if !!max_blocks
|
261
303
|
agent.latest_block_info['blockNumber']
|
262
304
|
end
|
263
305
|
|
264
306
|
Thread.new do
|
265
|
-
job = Meeseeker::SteemEngine::FollowerJob.new
|
266
|
-
|
267
307
|
loop do
|
268
308
|
begin
|
269
309
|
at_block_num = agent.latest_block_info["blockNumber"] - max_blocks.to_i
|
@@ -281,7 +321,7 @@ namespace :verify do
|
|
281
321
|
end
|
282
322
|
|
283
323
|
begin
|
284
|
-
block_channel =
|
324
|
+
block_channel = "#{chain_key_prefix}:block"
|
285
325
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
286
326
|
subscription = Redis.new(url: redis_url)
|
287
327
|
ctx = Redis.new(url: redis_url)
|
@@ -313,16 +353,16 @@ namespace :verify do
|
|
313
353
|
end
|
314
354
|
end
|
315
355
|
|
316
|
-
while ctx.keys("
|
356
|
+
while ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
|
317
357
|
# This ensures at least the next block has been indexed before
|
318
358
|
# proceeding.
|
319
359
|
|
320
|
-
puts "Waiting for block: #{next_block_num} ..."
|
360
|
+
puts "Waiting for block (verify:#{chain_key_prefix}_engine_block_org): #{next_block_num} ..."
|
321
361
|
sleep 6
|
322
362
|
end
|
323
363
|
|
324
364
|
# In theory, we should have all the keys using this pattern.
|
325
|
-
keys = ctx.keys("
|
365
|
+
keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
|
326
366
|
|
327
367
|
# If we have all the keys, we should also have all transaction ids.
|
328
368
|
expected_ids = keys.map { |k| k.split(':')[2] }.uniq
|
@@ -368,19 +408,40 @@ namespace :verify do
|
|
368
408
|
agent.shutdown
|
369
409
|
end
|
370
410
|
|
411
|
+
desc 'Verifies Hive Engine sidechain against the mainnet.'
|
412
|
+
task :hive_engine_ref_blocks do |t|
|
413
|
+
Rake::Task['verify:engine_ref_blocks'].invoke('hive_engine')
|
414
|
+
end
|
415
|
+
|
371
416
|
desc 'Verifies Steem Engine sidechain against the mainnet.'
|
372
417
|
task :steem_engine_ref_blocks do |t|
|
418
|
+
Rake::Task['verify:engine_ref_blocks'].invoke('steem_engine')
|
419
|
+
end
|
420
|
+
|
421
|
+
desc 'Verifies Steem/Hive Engine sidechain against the mainnet.'
|
422
|
+
task :engine_ref_blocks, [:chain_key_prefix] do |t, args|
|
423
|
+
chain_key_prefix = args[:chain_key_prefix]
|
373
424
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
374
425
|
ctx = ctx = Redis.new(url: redis_url)
|
375
|
-
keys = ctx.keys(
|
376
|
-
|
426
|
+
keys = ctx.keys("#{chain_key_prefix}:*:*")
|
427
|
+
mainchain, mainchain_url = case chain_key_prefix
|
428
|
+
when 'steem_engine' then ['steem', 'https://api.steemit.com']
|
429
|
+
when 'hive_engine' then ['hive', 'https://api.openhive.network']
|
430
|
+
end
|
431
|
+
block_api = Meeseeker.block_api_class(mainchain).new(url: mainchain_url)
|
377
432
|
block_trxs = {}
|
378
433
|
|
379
|
-
puts "Checking
|
434
|
+
puts "Checking #{chain_key_prefix} keys: #{keys.size}"
|
380
435
|
|
381
436
|
keys.each do |key|
|
382
437
|
transaction = JSON[ctx.get(key)]
|
383
|
-
|
438
|
+
|
439
|
+
next if transaction.class == Integer
|
440
|
+
|
441
|
+
block_num = case chain_key_prefix
|
442
|
+
when 'steem_engine' then transaction.fetch('refSteemBlockNumber')
|
443
|
+
when 'hive_engine' then transaction.fetch('refHiveBlockNumber')
|
444
|
+
end.to_i
|
384
445
|
|
385
446
|
block_trxs[block_num] ||= []
|
386
447
|
block_trxs[block_num] << transaction['transactionId'].to_s.split('-').first
|
@@ -390,6 +451,8 @@ namespace :verify do
|
|
390
451
|
|
391
452
|
skipped_blocks = []
|
392
453
|
|
454
|
+
next if block_trxs.empty?
|
455
|
+
|
393
456
|
block_api.get_blocks(block_range: block_trxs.keys) do |block, block_num|
|
394
457
|
if block.nil? || block[:transaction_ids].nil?
|
395
458
|
print 'S'
|
@@ -399,24 +462,34 @@ namespace :verify do
|
|
399
462
|
else
|
400
463
|
print '.'
|
401
464
|
end
|
402
|
-
|
403
|
-
|
465
|
+
|
466
|
+
trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
|
467
|
+
|
468
|
+
if trx_ids.any? && (block.transaction_ids & trx_ids).none?
|
404
469
|
puts "\nNo intersection in #{block_num}!"
|
405
|
-
puts "Expected the following sidechain trx_ids: #{
|
470
|
+
puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
|
406
471
|
end
|
407
472
|
end
|
408
473
|
|
409
474
|
puts "\nBlocks to retry: #{skipped_blocks.size}"
|
410
475
|
|
411
476
|
skipped_blocks.each do |block_num|
|
477
|
+
block_found = false
|
478
|
+
|
412
479
|
block_api.get_block(block_num: block_num) do |result|
|
480
|
+
break unless !!result.block
|
481
|
+
|
413
482
|
block = result.block
|
483
|
+
block_found = true
|
484
|
+
trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
|
414
485
|
|
415
|
-
if (block.transaction_ids &
|
486
|
+
if trx_ids.any? && (block.transaction_ids & trx_ids).none?
|
416
487
|
puts "No intersection in #{block_num}!"
|
417
|
-
puts "Expected the following sidechain trx_ids: #{
|
488
|
+
puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
|
418
489
|
end
|
419
490
|
end
|
491
|
+
|
492
|
+
redo unless block_found
|
420
493
|
end
|
421
494
|
|
422
495
|
puts "Done."
|
@@ -424,10 +497,12 @@ namespace :verify do
|
|
424
497
|
|
425
498
|
namespace :witness do
|
426
499
|
desc 'Verifies witnessses in the schedule produced a block.'
|
427
|
-
task :schedule, [:max_blocks] do |t, args|
|
500
|
+
task :schedule, [:chain, :max_blocks] do |t, args|
|
501
|
+
chain = args[:chain] if !!args[:chain]
|
502
|
+
chain ||= Meeseeker.default_chain_key_prefix
|
428
503
|
max_blocks = args[:max_blocks]
|
429
|
-
node_url =
|
430
|
-
database_api =
|
504
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
505
|
+
database_api = Meeseeker.database_api_class(chain).new(url: node_url)
|
431
506
|
mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
|
432
507
|
until_block_num = if !!max_blocks
|
433
508
|
database_api.get_dynamic_global_properties do |dgpo|
|
@@ -446,7 +521,7 @@ namespace :verify do
|
|
446
521
|
|
447
522
|
loop do
|
448
523
|
begin
|
449
|
-
job.perform(mode: mode, until_block_num: until_block_num)
|
524
|
+
job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
|
450
525
|
rescue => e
|
451
526
|
puts e.inspect
|
452
527
|
sleep 5
|
@@ -459,8 +534,10 @@ namespace :verify do
|
|
459
534
|
end
|
460
535
|
|
461
536
|
begin
|
462
|
-
block_api =
|
463
|
-
|
537
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
538
|
+
chain_key_prefix = chain.to_s if args[:chain]
|
539
|
+
chain_key_prefix ||= Meeseeker.default_chain_key_prefix
|
540
|
+
schedule_channel = "#{chain_key_prefix}:witness:schedule"
|
464
541
|
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
465
542
|
subscription = Redis.new(url: redis_url)
|
466
543
|
timeout = (max_blocks).to_i * 3
|
@@ -519,7 +596,10 @@ namespace :verify do
|
|
519
596
|
unless !!header
|
520
597
|
# Can happen when there's excess p2p latency and/or jussi
|
521
598
|
# cache is under load.
|
522
|
-
puts "Waiting for block header: #{block_num}"
|
599
|
+
puts "Waiting for block header (witness:schedule): #{block_num}"
|
600
|
+
|
601
|
+
node_url = Meeseeker.shuffle_node_url(chain)
|
602
|
+
block_api = Meeseeker.block_api_class(chain).new(url: node_url)
|
523
603
|
|
524
604
|
next
|
525
605
|
end
|