meeseeker 0.0.7 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 857cc4efa38d7428ba13457a6ffc3db8cb01aab92a26a3cf052b15698d91ca56
4
- data.tar.gz: 3f6398e6bbad560624fe0b5e331f0cba2b3b207edfa15074627b7f964edd262d
3
+ metadata.gz: 3198bed0cd41758cb9b2c5669b6f63cb6e610c35289a60dd3efd631c417117c3
4
+ data.tar.gz: 9c032a2d5711fae827a1325ec782c818e07f3c7dae99b4985ea2cf94d3d6a4d1
5
5
  SHA512:
6
- metadata.gz: 6ae2f1bdea39419820d35ce2fedb63ceac5c0c59d61c63ef9f7a01470baea9cc0d784a630996997750ece65fdee78ec5b8deb7b339b2e8343e82db3c38682527
7
- data.tar.gz: 635057da03b642a258436e6d78c0bb2c830ddb475a595c16cfdc80b33b4b92cf58e5782bd4a4cd0b6dd3e85c06f7d137170f2ae6395543086e9070cf595a2cfd
6
+ metadata.gz: e75b91ca8d0290919dce1a424297a8ae74add9fb8936ea28ae16a249b08c0a3f882953129e17605a88f41196cd7530d63aee0807f5a6dad01ac7f23ea1b64efb
7
+ data.tar.gz: e0c794e530599986f9800cb8f8bf8f2c305e46ab60ba45078bb5ae9ddf2f4c878b66a142043a32d00a174488fc909de889e6b7354c238474990b791434bcf86f
data/README.md CHANGED
@@ -1,6 +1,8 @@
1
1
  # meeseeker
2
2
 
3
- Redis based block follower is an efficient way for multiple apps to stream the Steem Blockchain.
3
+ Redis based block follower is an efficient way for multiple apps to stream the Hive Blockchain.
4
+
5
+ [![Build Status](https://travis-ci.org/inertia186/meeseeker.svg?branch=master)](https://travis-ci.org/inertia186/meeseeker)
4
6
 
5
7
  If you have multiple applications that need to perform actions as operations occur, `meeseeker` will allow your apps to each perform actions for specific operations without each app having to stream the entire blockchain.
6
8
 
@@ -62,10 +64,22 @@ To specify an alternative redis source:
62
64
  MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync
63
65
  ```
64
66
 
65
- You can also specify an alternative Steem node:
67
+ You can also specify am alternative Hive node:
66
68
 
67
69
  ```bash
68
- MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync
70
+ MEESEEKER_NODE_URL=http://anyx.io meeseeker sync
71
+ ```
72
+
73
+ You can also specify a Steem node instead of Hive (if that's your thing):
74
+
75
+ ```bash
76
+ MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync[steem]
77
+ ```
78
+
79
+ Or, you can have meeseeker automatically use random Hive nodes:
80
+
81
+ ```bash
82
+ MEESEEKER_NODE_URL=shuffle meeseeker sync
69
83
  ```
70
84
 
71
85
  To sync from the head block instead of the last irreversible block:
@@ -80,7 +94,7 @@ To ignore virtual operations (useful if the node doesn't enable `get_ops_in_bloc
80
94
  MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
81
95
  ```
82
96
 
83
- Normally, block headers are added to the `steem:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `steem:block` channel to only publish with the `block_num`:
97
+ Normally, block headers are added to the `hive:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `hive:block` channel to only publish with the `block_num`:
84
98
 
85
99
  ```bash
86
100
  MEESEEKER_INCLUDE_BLOCK_HEADER=false meeseeker sync
@@ -99,6 +113,12 @@ If you never want the keys to expire (not recommended), set
99
113
  MEESEEKER_EXPIRE_KEYS=-1 meeseeker sync
100
114
  ```
101
115
 
116
+ Normally, sync will create keys until it uses up all available memory. If you would like to only sync a certain number of keys, then sleep until those keys expire so it can pick up where it left off, set `MEESEEKER_MAX_KEYS` to a positive value:
117
+
118
+ ```bash
119
+ MEESEEKER_MAX_KEYS=99 meeseeker sync
120
+ ```
121
+
102
122
  ### Usage
103
123
 
104
124
  When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
@@ -111,35 +131,35 @@ For `redis-cli`, please see: https://redis.io/topics/pubsub
111
131
 
112
132
  When running `meeseeker sync`, the following channels are available:
113
133
 
114
- * `steem:block`
115
- * `steem:transaction`
116
- * `steem:op:vote`
117
- * `steem:op:comment`
118
- * `steem:op:comment_options`
119
- * `steem:op:whatever` (replace "whatever" with the op you want)
120
- * `steem:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
134
+ * `hive:block`
135
+ * `hive:transaction`
136
+ * `hive:op:vote`
137
+ * `hive:op:comment`
138
+ * `hive:op:comment_options`
139
+ * `hive:op:whatever` (replace "whatever" with the op you want)
140
+ * `hive:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
121
141
 
122
- As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.steem.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
142
+ As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.hive.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
123
143
 
124
- In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `steem:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `steem:op:custom_json:follow`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
144
+ In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `hive:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `hive:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
125
145
 
126
146
  For example, from `redis-cli`, if we wanted to stream block numbers:
127
147
 
128
148
  ```bash
129
149
  $ redis-cli
130
- 127.0.0.1:6379> subscribe steem:block
150
+ 127.0.0.1:6379> subscribe hive:block
131
151
  Reading messages... (press Ctrl-C to quit)
132
152
  1) "subscribe"
133
- 2) "steem:block"
153
+ 2) "hive:block"
134
154
  3) (integer) 1
135
155
  1) "message"
136
- 2) "steem:block"
156
+ 2) "hive:block"
137
157
  3) "{\"block_num\":29861068,\"previous\":\"01c7a4cb4424b4dc0cb0cc72fd36b1644f8aeba5\",\"timestamp\":\"2019-01-28T20:55:03\",\"witness\":\"ausbitbank\",\"transaction_merkle_root\":\"a318bb82625bd78af8d8b506ccd4f53116372c8e\",\"extensions\":[]}"
138
158
  1) "message"
139
- 2) "steem:block"
159
+ 2) "hive:block"
140
160
  3) "{\"block_num\":29861069,\"previous\":\"01c7a4cc1bed060876cab57476846a91568a9f8a\",\"timestamp\":\"2019-01-28T20:55:06\",\"witness\":\"followbtcnews\",\"transaction_merkle_root\":\"834e05d40b9666e5ef50deb9f368c63070c0105b\",\"extensions\":[]}"
141
161
  1) "message"
142
- 2) "steem:block"
162
+ 2) "hive:block"
143
163
  3) "{\"block_num\":29861070,\"previous\":\"01c7a4cd3bbf872895654765faa4409a8e770e91\",\"timestamp\":\"2019-01-28T20:55:09\",\"witness\":\"timcliff\",\"transaction_merkle_root\":\"b2366ce9134d627e00423b28d33cc57f1e6e453f\",\"extensions\":[]}"
144
164
  ```
145
165
 
@@ -153,26 +173,26 @@ Which allows subscription to specific `id` patterns:
153
173
 
154
174
  ```
155
175
  $ redis-cli
156
- 127.0.0.1:6379> subscribe steem:op:custom_json:sm_team_reveal
176
+ 127.0.0.1:6379> subscribe hive:op:custom_json:sm_team_reveal
157
177
  Reading messages... (press Ctrl-C to quit)
158
178
  1) "subscribe"
159
- 2) "steem:op:custom_json:sm_team_reveal"
179
+ 2) "hive:op:custom_json:sm_team_reveal"
160
180
  3) (integer) 1
161
181
  1) "message"
162
- 2) "steem:op:custom_json:sm_team_reveal"
163
- 3) "{\"key\":\"steem:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
182
+ 2) "hive:op:custom_json:sm_team_reveal"
183
+ 3) "{\"key\":\"hive:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
164
184
  1) "message"
165
- 2) "steem:op:custom_json:sm_team_reveal"
166
- 3) "{\"key\":\"steem:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
185
+ 2) "hive:op:custom_json:sm_team_reveal"
186
+ 3) "{\"key\":\"hive:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
167
187
  1) "message"
168
- 2) "steem:op:custom_json:sm_team_reveal"
169
- 3) "{\"key\":\"steem:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
188
+ 2) "hive:op:custom_json:sm_team_reveal"
189
+ 3) "{\"key\":\"hive:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
170
190
  1) "message"
171
- 2) "steem:op:custom_json:sm_team_reveal"
172
- 3) "{\"key\":\"steem:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
191
+ 2) "hive:op:custom_json:sm_team_reveal"
192
+ 3) "{\"key\":\"hive:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
173
193
  1) "message"
174
- 2) "steem:op:custom_json:sm_team_reveal"
175
- 3) "{\"key\":\"steem:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
194
+ 2) "hive:op:custom_json:sm_team_reveal"
195
+ 3) "{\"key\":\"hive:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
176
196
  ```
177
197
 
178
198
  A `ruby` application can subscribe to a channel as well, using the `redis` gem:
@@ -183,7 +203,7 @@ require 'redis'
183
203
  url = 'redis://127.0.0.1:6379/0'
184
204
  ctx = Redis.new(url: url)
185
205
 
186
- Redis.new(url: url).subscribe('steem:op:comment') do |on|
206
+ Redis.new(url: url).subscribe('hive:op:comment') do |on|
187
207
  on.message do |channel, message|
188
208
  payload = JSON[message]
189
209
  comment = JSON[ctx.get(payload['key'])]
@@ -197,19 +217,19 @@ Many other clients are supported: https://redis.io/clients
197
217
 
198
218
  ##### Witness Schedule
199
219
 
200
- When running `meeseeker witness:schedule`, the `steem:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
220
+ When running `meeseeker witness:schedule`, the `hive:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
201
221
 
202
222
  For example, from `redis-cli`, if we wanted to subscribe to the witness schedule:
203
223
 
204
224
  ```
205
225
  $ redis-cli
206
- 127.0.0.1:6379> subscribe steem:witness:schedule
226
+ 127.0.0.1:6379> subscribe hive:witness:schedule
207
227
  Reading messages... (press Ctrl-C to quit)
208
228
  1) "subscribe"
209
- 2) "steem:witness:schedule"
229
+ 2) "hive:witness:schedule"
210
230
  3) (integer) 1
211
231
  1) "message"
212
- 2) "steem:witness:schedule"
232
+ 2) "hive:witness:schedule"
213
233
  3) "{\"id\":0,\"current_virtual_time\":\"415293532210075480213212125\",\"next_shuffle_block_num\":30035208,\"current_shuffled_witnesses\":[\"thecryptodrive\",\"timcliff\",\"utopian-io\",\"themarkymark\",\"aggroed\",\"smooth.witness\",\"someguy123\",\"gtg\",\"followbtcnews\",\"yabapmatt\",\"therealwolf\",\"ausbitbank\",\"curie\",\"clayop\",\"drakos\",\"blocktrades\",\"good-karma\",\"roelandp\",\"lukestokes.mhth\",\"liondani\",\"anyx\"],\"num_scheduled_witnesses\":21,\"elected_weight\":1,\"timeshare_weight\":5,\"miner_weight\":1,\"witness_pay_normalization_factor\":25,\"median_props\":{\"account_creation_fee\":{\"amount\":\"3000\",\"precision\":3,\"nai\":\"@@000000021\"},\"maximum_block_size\":65536,\"sbd_interest_rate\":0,\"account_subsidy_budget\":797,\"account_subsidy_decay\":347321},\"majority_version\":\"0.20.8\",\"max_voted_witnesses\":20,\"max_miner_witnesses\":0,\"max_runner_witnesses\":1,\"hardfork_required_witnesses\":17,\"account_subsidy_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":797,\"pool_eq\":157691079,\"max_pool_size\":157691079,\"decay_params\":{\"decay_per_time_unit\":347321,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":0},\"account_subsidy_witness_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":996,\"pool_eq\":9384019,\"max_pool_size\":9384019,\"decay_params\":{\"decay_per_time_unit\":7293741,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":257},\"min_witness_account_subsidy_decay\":0}"
214
234
  ```
215
235
 
@@ -230,111 +250,120 @@ See: https://redis.io/commands/scan#scan-basic-usage
230
250
  Once your sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
231
251
 
232
252
  ```bash
233
- redis-cli --scan --pattern 'steem:*:vote'
253
+ redis-cli --scan --pattern 'hive:*:vote'
234
254
  ```
235
255
 
236
256
  This returns the keys, for example:
237
257
 
238
258
  ```
239
- steem:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
240
- steem:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
241
- steem:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
242
- steem:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
243
- steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
244
- steem:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
245
- steem:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
246
- steem:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
247
- steem:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
259
+ hive:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
260
+ hive:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
261
+ hive:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
262
+ hive:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
263
+ hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
264
+ hive:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
265
+ hive:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
266
+ hive:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
267
+ hive:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
248
268
  ```
249
269
 
250
270
  To get the actual vote operation for a particular key, use:
251
271
 
252
272
  ```bash
253
- redis-cli get steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
273
+ redis-cli get hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
254
274
  ```
255
275
 
256
276
  If, on the other hand, you want `custom_json` only:
257
277
 
258
278
  ```bash
259
- redis-cli --scan --pattern 'steem:*:custom_json'
279
+ redis-cli --scan --pattern 'hive:*:custom_json'
260
280
  ```
261
281
 
262
282
  This only returns the related keys, for example:
263
283
 
264
284
  ```
265
- steem:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
266
- steem:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
267
- steem:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
268
- steem:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
269
- steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
270
- steem:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
285
+ hive:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
286
+ hive:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
287
+ hive:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
288
+ hive:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
289
+ hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
290
+ hive:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
271
291
  ```
272
292
 
273
293
  To get the actual custom json operation for a particular key, use:
274
294
 
275
295
  ```bash
276
- redis-cli get steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
296
+ redis-cli get hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
277
297
  ```
278
298
 
279
299
  To get all transactions for a particular block number:
280
300
 
281
301
  ```bash
282
- redis-cli --scan --pattern 'steem:29811085:*'
302
+ redis-cli --scan --pattern 'hive:29811085:*'
283
303
  ```
284
304
 
285
305
  Or to get all ops for a particular transaction:
286
306
 
287
307
  ```bash
288
- redis-cli --scan --pattern 'steem:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
308
+ redis-cli --scan --pattern 'hive:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
289
309
  ```
290
310
 
291
- ### Steem Engine Support
311
+ ### Hive Engine Support
292
312
 
293
- As of `v0.0.6`, meeseeker can also follow the Steem Engine side-chain. This is optional and requires a separate process.
313
+ As of `v0.0.6`, meeseeker can also follow the Hive Engine side-chain. This is optional and requires a separate process.
294
314
 
295
- To sync Steem Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
315
+ To sync Hive Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
296
316
 
297
317
  ```bash
298
- meeseeker sync steem_engine
299
- ```
300
-
301
- When running `meeseeker sync steem_engine`, the following channels are available:
302
-
303
- * `steem_engine:block`
304
- * `steem_engine:transaction`
305
- * `steem_engine:market`
306
- * `steem_engine:market:buy`
307
- * `steem_engine:market:cancel`
308
- * `steem_engine:market:sell`
309
- * `steem_engine:sscstore`
310
- * `steem_engine:sscstore:buy`
311
- * `steem_engine:steempegged`
312
- * `steem_engine:steempegged:buy`
313
- * `steem_engine:steempegged:removeWithdrawal`
314
- * `steem_engine:steempegged:withdraw`
315
- * `steem_engine:tokens`
316
- * `steem_engine:tokens:create`
317
- * `steem_engine:tokens:issue`
318
- * `steem_engine:tokens:transfer`
319
- * `steem_engine:tokens:updateMetadata`
320
- * `steem_engine:tokens:updateUrl`
318
+ meeseeker sync hive_engine
319
+ ```
320
+
321
+ When running `meeseeker sync hive_engine`, the following channels are available:
322
+
323
+ * `hive_engine:block`
324
+ * `hive_engine:transaction`
325
+ * `hive_engine:virtual_transaction`
326
+ * `hive_engine:contract`
327
+ * `hive_engine:contract:deploy`
328
+ * `hive_engine:contract:update`
329
+ * `hive_engine:market`
330
+ * `hive_engine:market:buy`
331
+ * `hive_engine:market:cancel`
332
+ * `hive_engine:market:sell`
333
+ * `hive_engine:sscstore`
334
+ * `hive_engine:sscstore:buy`
335
+ * `hive_engine:steempegged`
336
+ * `hive_engine:steempegged:buy`
337
+ * `hive_engine:steempegged:removeWithdrawal`
338
+ * `hive_engine:steempegged:withdraw`
339
+ * `hive_engine:tokens`
340
+ * `hive_engine:tokens:checkPendingUnstake`
341
+ * `hive_engine:tokens:create`
342
+ * `hive_engine:tokens:enableStaking`
343
+ * `hive_engine:tokens:issue`
344
+ * `hive_engine:tokens:transfer`
345
+ * `hive_engine:tokens:transferOwnership`
346
+ * `hive_engine:tokens:unstake`
347
+ * `hive_engine:tokens:updateMetadata`
348
+ * `hive_engine:tokens:updateParams`
349
+ * `hive_engine:tokens:updateUrl`
321
350
 
322
351
  The above "channel/action" patterns are the ones that are known that the time of writing. In addition, if a new contract is added or updated, meeseeker will automatically publish to these corresponding channels as they appear, without needing to update or even restart meeseeker.
323
352
 
324
353
  See main section on [Using `SUBSCRIBE`](#using-subscribe).
325
354
 
326
- Once your SteemEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
355
+ Once your HiveEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
327
356
 
328
357
  ```bash
329
- redis-cli --scan --pattern 'steem_engine:*:tokens:transfer'
358
+ redis-cli --scan --pattern 'hive_engine:*:tokens:transfer'
330
359
  ```
331
360
 
332
361
  This returns the keys, for example:
333
362
 
334
363
  ```
335
- steem_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
336
- steem_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
337
- steem_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
364
+ hive_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
365
+ hive_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
366
+ hive_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
338
367
  ```
339
368
 
340
369
  See main section on [Using `SCAN`](#using-scan).
@@ -365,11 +394,11 @@ Also see: https://hub.docker.com/r/inertia/meeseeker/
365
394
  <img src="https://i.imgur.com/Y3Sa2GW.jpg" />
366
395
  </center>
367
396
 
368
- See some of my previous Ruby How To posts in: [#radiator](https://steemit.com/created/radiator) [#ruby](https://steemit.com/created/ruby)
397
+ See some of my previous Ruby How To posts in: [#radiator](https://hive.blog/created/radiator) [#ruby](https://hive.blog/created/ruby)
369
398
 
370
399
  ## Get in touch!
371
400
 
372
- If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on STEEM.
401
+ If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on Hive.
373
402
 
374
403
  ## License
375
404
 
data/Rakefile CHANGED
@@ -52,29 +52,35 @@ task :check_schema do
52
52
  end
53
53
 
54
54
  task(:sync, [:chain, :at_block_num] => [:check_schema]) do |t, args|
55
- chain = (args[:chain] || 'steem').to_sym
55
+ chain = args[:chain] if args[:chain]
56
+ chain ||= Meeseeker.default_chain_key_prefix
56
57
 
57
- job = case chain
58
- when :steem
59
- Meeseeker::BlockFollowerJob.new
58
+ job = case chain.to_sym
60
59
  when :steem_engine
61
60
  Meeseeker::SteemEngine::FollowerJob.new
62
- else; abort("Unknown chain: #{chain}")
61
+ when :hive_engine
62
+ Meeseeker::HiveEngine::FollowerJob.new
63
+ else
64
+ Meeseeker::BlockFollowerJob.new
63
65
  end
64
66
 
65
- job.perform(at_block_num: args[:at_block_num])
67
+ job.perform(chain: chain, at_block_num: args[:at_block_num])
66
68
  end
67
69
 
68
70
  namespace :witness do
69
- desc 'Publish the witness schedule every minute or so (steem:witness:schedule).'
70
- task :schedule do
71
+ desc 'Publish the witness schedule every minute or so (e.g.: hive:witness:schedule).'
72
+ task :schedule, [:chain] do |t, args|
73
+ chain = args[:chain] if args[:chain]
74
+ chain ||= Meeseeker.default_chain_key_prefix
75
+
71
76
  job = Meeseeker::WitnessScheduleJob.new
72
- job.perform
77
+ job.perform(chain: chain)
73
78
  end
74
79
  end
75
80
 
76
81
  task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
77
- chain = (args[:chain] || 'steem').downcase.to_sym
82
+ chain = args[:chain] if args[:chain]
83
+ chain ||= Meeseeker.default_chain_key_prefix
78
84
  redis = Meeseeker.redis
79
85
 
80
86
  match = case args[:what].downcase.to_sym
@@ -99,8 +105,15 @@ task :reset, [:chain] => [:check_schema] do |t, args|
99
105
  print 'Dropping keys for set: %s ...' % chain.to_s
100
106
 
101
107
  case chain
102
- when :steem, :all then keys += Meeseeker.redis.keys('steem:*')
103
- when :steem_engine, :all then keys += Meeseeker.redis.keys('steem_engine:*')
108
+ when :steem_engine then keys += Meeseeker.redis.keys('steem_engine:*')
109
+ when :hive_engine then keys += Meeseeker.redis.keys('hive_engine:*')
110
+ when :all
111
+ keys += Meeseeker.redis.keys('steem:*')
112
+ keys += Meeseeker.redis.keys('hive:*')
113
+ keys += Meeseeker.redis.keys('steem_engine:*')
114
+ keys += Meeseeker.redis.keys('hive_engine:*')
115
+ else
116
+ keys += Meeseeker.redis.keys("#{chain}:*")
104
117
  end
105
118
 
106
119
  if keys.any?
@@ -114,10 +127,13 @@ end
114
127
 
115
128
  namespace :verify do
116
129
  desc 'Verifies transactions land where they should.'
117
- task :block_org, [:max_blocks] do |t, args|
130
+ task :block_org, [:chain, :max_blocks] do |t, args|
131
+ chain = args[:chain] if args[:chain]
132
+ chain ||= Meeseeker.default_chain_key_prefix
133
+ chain_key_prefix = chain.to_s
118
134
  max_blocks = args[:max_blocks]
119
- node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
120
- database_api = Steem::DatabaseApi.new(url: node_url)
135
+ node_url = Meeseeker.shuffle_node_url(chain)
136
+ database_api = Meeseeker.database_api_class(chain).new(url: node_url)
121
137
  mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
122
138
  until_block_num = if !!max_blocks
123
139
  database_api.get_dynamic_global_properties do |dgpo|
@@ -128,7 +144,7 @@ namespace :verify do
128
144
  when :irreversible then dgpo.last_irreversible_block_num
129
145
  else; abort "Unknown block mode: #{mode}"
130
146
  end
131
- end + max_blocks.to_i
147
+ end + max_blocks.to_i + 1
132
148
  end
133
149
 
134
150
  Thread.new do
@@ -136,7 +152,7 @@ namespace :verify do
136
152
 
137
153
  loop do
138
154
  begin
139
- job.perform(mode: mode, until_block_num: until_block_num)
155
+ job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
140
156
  rescue => e
141
157
  puts e.inspect
142
158
  sleep 5
@@ -149,8 +165,8 @@ namespace :verify do
149
165
  end
150
166
 
151
167
  begin
152
- block_api = Steem::BlockApi.new(url: node_url)
153
- block_channel = 'steem:block'
168
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
169
+ block_channel = "#{chain_key_prefix}:block"
154
170
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
155
171
  subscription = Redis.new(url: redis_url)
156
172
  ctx = Redis.new(url: redis_url)
@@ -183,14 +199,21 @@ namespace :verify do
183
199
  end
184
200
  end
185
201
 
186
- while ctx.keys("steem:#{next_block_num}:*").size == 0
202
+ 5.times do
203
+ break unless ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
204
+
187
205
  # This ensures at least the next block has been indexed before
188
206
  # proceeding.
189
207
 
190
- puts "Waiting for block: #{next_block_num} ..."
208
+ puts "Waiting for block (verify:block_org): #{next_block_num} ..."
209
+
191
210
  sleep 6
192
211
  end
193
212
 
213
+ if ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
214
+ puts "Gave up waiting for block (check current_aslot slippage): #{next_block_num}"
215
+ end
216
+
194
217
  database_api.get_dynamic_global_properties do |dgpo|
195
218
  raise 'Got empty dynamic_global_properties result.' if dgpo.nil?
196
219
 
@@ -204,7 +227,7 @@ namespace :verify do
204
227
  end
205
228
 
206
229
  # In theory, we should have all the keys using this pattern.
207
- keys = ctx.keys("steem:#{block_num}:*")
230
+ keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
208
231
 
209
232
  # If we have all the keys, we should also have all transaction ids.
210
233
  expected_ids = keys.map { |k| k.split(':')[2] }.uniq
@@ -249,18 +272,35 @@ namespace :verify do
249
272
  end
250
273
  end
251
274
 
275
+ desc 'Verifies Hive Engine transactions land where they should.'
276
+ task :hive_engine_block_org, [:max_blocks] do |t, args|
277
+ Rake::Task['verify:engine_block_org'].invoke('hive_engine', args[:max_blocks])
278
+ end
279
+
252
280
  desc 'Verifies Steem Engine transactions land where they should.'
253
281
  task :steem_engine_block_org, [:max_blocks] do |t, args|
282
+ Rake::Task['verify:engine_block_org'].invoke('steem_engine', args[:max_blocks])
283
+ end
284
+
285
+ desc 'Verifies Steem/Hive Engine transactions land where they should.'
286
+ task :engine_block_org, [:chain_key_prefix, :max_blocks] do |t, args|
287
+ chain_key_prefix = args[:chain_key_prefix]
254
288
  max_blocks = args[:max_blocks]
255
- node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
256
- agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
289
+ case chain_key_prefix.to_sym
290
+ when :steem_engine
291
+ node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
292
+ agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
293
+ job = Meeseeker::SteemEngine::FollowerJob.new
294
+ when :hive_engine
295
+ node_url = ENV.fetch('MEESEEKER_HIVE_ENGINE_NODE_URL', 'https://api.hive-engine.com/rpc')
296
+ agent = Meeseeker::HiveEngine::Agent.new(url: node_url)
297
+ job = Meeseeker::HiveEngine::FollowerJob.new
298
+ end
257
299
  until_block_num = if !!max_blocks
258
300
  agent.latest_block_info['blockNumber']
259
301
  end
260
302
 
261
303
  Thread.new do
262
- job = Meeseeker::SteemEngine::FollowerJob.new
263
-
264
304
  loop do
265
305
  begin
266
306
  at_block_num = agent.latest_block_info["blockNumber"] - max_blocks.to_i
@@ -278,7 +318,7 @@ namespace :verify do
278
318
  end
279
319
 
280
320
  begin
281
- block_channel = 'steem_engine:block'
321
+ block_channel = "#{chain_key_prefix}:block"
282
322
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
283
323
  subscription = Redis.new(url: redis_url)
284
324
  ctx = Redis.new(url: redis_url)
@@ -310,25 +350,26 @@ namespace :verify do
310
350
  end
311
351
  end
312
352
 
313
- while ctx.keys("steem_engine:#{next_block_num}:*").size == 0
353
+ while ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
314
354
  # This ensures at least the next block has been indexed before
315
355
  # proceeding.
316
356
 
317
- puts "Waiting for block: #{next_block_num} ..."
357
+ puts "Waiting for block (verify:#{chain_key_prefix}_engine_block_org): #{next_block_num} ..."
318
358
  sleep 6
319
359
  end
320
360
 
321
361
  # In theory, we should have all the keys using this pattern.
322
- keys = ctx.keys("steem_engine:#{block_num}:*")
362
+ keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
323
363
 
324
364
  # If we have all the keys, we should also have all transaction ids.
325
365
  expected_ids = keys.map { |k| k.split(':')[2] }.uniq
366
+ expected_ids -= [Meeseeker::VIRTUAL_TRX_ID]
326
367
  actual_ids = nil
327
368
 
328
369
  agent.block(block_num).tap do |block|
329
370
  raise 'Got empty block result.' if block.nil?
330
371
 
331
- actual_ids = block['transactions'].map{|trx| trx['transactionId'].split('-').first}.uniq
372
+ actual_ids = block['transactions'].map{|trx| trx['transactionId'].to_s.split('-').first}.uniq
332
373
  end
333
374
 
334
375
  # We do an intersection to make sure there's no difference between
@@ -360,30 +401,55 @@ namespace :verify do
360
401
  end
361
402
  end
362
403
  end
404
+
405
+ agent.shutdown
406
+ end
407
+
408
+ desc 'Verifies Hive Engine sidechain against the mainnet.'
409
+ task :hive_engine_ref_blocks do |t|
410
+ Rake::Task['verify:engine_ref_blocks'].invoke('hive_engine')
363
411
  end
364
412
 
365
413
  desc 'Verifies Steem Engine sidechain against the mainnet.'
366
414
  task :steem_engine_ref_blocks do |t|
415
+ Rake::Task['verify:engine_ref_blocks'].invoke('steem_engine')
416
+ end
417
+
418
+ desc 'Verifies Steem/Hive Engine sidechain against the mainnet.'
419
+ task :engine_ref_blocks, [:chain_key_prefix] do |t, args|
420
+ chain_key_prefix = args[:chain_key_prefix]
367
421
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
368
422
  ctx = ctx = Redis.new(url: redis_url)
369
- keys = ctx.keys('steem_engine:*')
370
- block_api = Steem::BlockApi.new
423
+ keys = ctx.keys("#{chain_key_prefix}:*:*")
424
+ mainchain, mainchain_url = case chain_key_prefix
425
+ when 'steem_engine' then ['steem', 'https://api.steemit.com']
426
+ when 'hive_engine' then ['hive', 'https://api.openhive.network']
427
+ end
428
+ block_api = Meeseeker.block_api_class(mainchain).new(url: mainchain_url)
371
429
  block_trxs = {}
372
430
 
373
- puts "Checking Steem Engine keys: #{keys.size}"
431
+ puts "Checking #{chain_key_prefix} keys: #{keys.size}"
374
432
 
375
433
  keys.each do |key|
376
434
  transaction = JSON[ctx.get(key)]
377
- block_num = transaction['refSteemBlockNumber']
435
+
436
+ next if transaction.class == Integer
437
+
438
+ block_num = case chain_key_prefix
439
+ when 'steem_engine' then transaction.fetch('refSteemBlockNumber')
440
+ when 'hive_engine' then transaction.fetch('refHiveBlockNumber')
441
+ end.to_i
378
442
 
379
443
  block_trxs[block_num] ||= []
380
- block_trxs[block_num] << transaction['transactionId'].split('-').first
444
+ block_trxs[block_num] << transaction['transactionId'].to_s.split('-').first
381
445
  end
382
446
 
383
447
  puts "Related mainnet blocks: #{block_trxs.keys.size}"
384
448
 
385
449
  skipped_blocks = []
386
450
 
451
+ next if block_trxs.empty?
452
+
387
453
  block_api.get_blocks(block_range: block_trxs.keys) do |block, block_num|
388
454
  if block.nil? || block[:transaction_ids].nil?
389
455
  print 'S'
@@ -393,24 +459,34 @@ namespace :verify do
393
459
  else
394
460
  print '.'
395
461
  end
396
-
397
- if (block.transaction_ids & block_trxs[block_num]).none?
462
+
463
+ trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
464
+
465
+ if trx_ids.any? && (block.transaction_ids & trx_ids).none?
398
466
  puts "\nNo intersection in #{block_num}!"
399
- puts "Expected the following sidechain trx_ids: #{block_trxs[block_num].join(', ')}"
467
+ puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
400
468
  end
401
469
  end
402
470
 
403
471
  puts "\nBlocks to retry: #{skipped_blocks.size}"
404
472
 
405
473
  skipped_blocks.each do |block_num|
474
+ block_found = false
475
+
406
476
  block_api.get_block(block_num: block_num) do |result|
477
+ break unless !!result.block
478
+
407
479
  block = result.block
480
+ block_found = true
481
+ trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
408
482
 
409
- if (block.transaction_ids & block_trxs[block_num]).none?
483
+ if trx_ids.any? && (block.transaction_ids & trx_ids).none?
410
484
  puts "No intersection in #{block_num}!"
411
- puts "Expected the following sidechain trx_ids: #{block_trxs[block_num].join(', ')}"
485
+ puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
412
486
  end
413
487
  end
488
+
489
+ redo unless block_found
414
490
  end
415
491
 
416
492
  puts "Done."
@@ -418,10 +494,12 @@ namespace :verify do
418
494
 
419
495
  namespace :witness do
420
496
  desc 'Verifies witnessses in the schedule produced a block.'
421
- task :schedule, [:max_blocks] do |t, args|
497
+ task :schedule, [:chain, :max_blocks] do |t, args|
498
+ chain = args[:chain] if !!args[:chain]
499
+ chain ||= Meeseeker.default_chain_key_prefix
422
500
  max_blocks = args[:max_blocks]
423
- node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
424
- database_api = Steem::DatabaseApi.new(url: node_url)
501
+ node_url = Meeseeker.shuffle_node_url(chain)
502
+ database_api = Meeseeker.database_api_class(chain).new(url: node_url)
425
503
  mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
426
504
  until_block_num = if !!max_blocks
427
505
  database_api.get_dynamic_global_properties do |dgpo|
@@ -440,7 +518,7 @@ namespace :verify do
440
518
 
441
519
  loop do
442
520
  begin
443
- job.perform(mode: mode, until_block_num: until_block_num)
521
+ job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
444
522
  rescue => e
445
523
  puts e.inspect
446
524
  sleep 5
@@ -453,11 +531,12 @@ namespace :verify do
453
531
  end
454
532
 
455
533
  begin
456
- block_api = Steem::BlockApi.new(url: node_url)
457
- schedule_channel = 'steem:witness:schedule'
534
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
535
+ chain_key_prefix = chain.to_s if args[:chain]
536
+ chain_key_prefix ||= Meeseeker.default_chain_key_prefix
537
+ schedule_channel = "#{chain_key_prefix}:witness:schedule"
458
538
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
459
539
  subscription = Redis.new(url: redis_url)
460
- ctx = Redis.new(url: redis_url)
461
540
  timeout = (max_blocks).to_i * 3
462
541
 
463
542
  subscribe_mode, subscribe_args = if timeout > 0
@@ -514,7 +593,10 @@ namespace :verify do
514
593
  unless !!header
515
594
  # Can happen when there's excess p2p latency and/or jussi
516
595
  # cache is under load.
517
- puts "Waiting for block header: #{block_num}"
596
+ puts "Waiting for block header (witness:schedule): #{block_num}"
597
+
598
+ node_url = Meeseeker.shuffle_node_url(chain)
599
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
518
600
 
519
601
  next
520
602
  end