meeseeker 1.0.0 → 2.0.0.pre.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d0ed288dbda6b6cac7cd3e77e520d443a3c908cd9abd5933d5e867e836f11139
4
- data.tar.gz: c0fad654f2f6afc353b02c8404ee7a3ceac2096ed797d32a1078b8ae28cced9e
3
+ metadata.gz: 9bb58923eb232294d7a41e43e963543cfbf09f90e4fa5b20d8dfa5eb62ed05ab
4
+ data.tar.gz: 04b201a7c6e1b8c4f4b7e19e58f7bc5eaa09c5a184c43a21add8d411fb57dc2e
5
5
  SHA512:
6
- metadata.gz: 210246f8ab9fa1bab680be81a1b8577ec09b0d252227186a83f2a9e68445722a3dbd9a61a4bb9eda990ca3b3da0c5a377d3c70f245c330c5cc5a74ffdba2cb83
7
- data.tar.gz: 26dc21a2cddb435bf480a00ec46ea23c2afd94d0c068967266a1120a6dab6050093e0d9969e81c6fe0caaae37c8854327da8a039c1a269b5c9fe047dea3df550
6
+ metadata.gz: 1e0ce806be69a0b9af06df823b443a498a8789bd93fc323367c5de3e556ce8d9bae569ef54dcf880a555d7e937491eb003ef4ff8c2a01388fb295f65887d19bc
7
+ data.tar.gz: e27cdfe179c60ab9faf17855ab5c6a4522dfb2b8ae892326ee256095513b4a8a70e8228d7eba331a52eb6ddbda94e3c9334f5b7f58284940bf4c5cf0d92e8af8
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # meeseeker
2
2
 
3
- Redis based block follower is an efficient way for multiple apps to stream the Steem Blockchain.
3
+ Redis based block follower is an efficient way for multiple apps to stream the Hive Blockchain.
4
4
 
5
5
  [![Build Status](https://travis-ci.org/inertia186/meeseeker.svg?branch=master)](https://travis-ci.org/inertia186/meeseeker)
6
6
 
@@ -64,10 +64,22 @@ To specify an alternative redis source:
64
64
  MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync
65
65
  ```
66
66
 
67
- You can also specify an alternative Steem node:
67
+ You can also specify am alternative Hive node:
68
68
 
69
69
  ```bash
70
- MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync
70
+ MEESEEKER_NODE_URL=http://anyx.io meeseeker sync
71
+ ```
72
+
73
+ You can also specify a Steem node instead of Hive (if that's your thing):
74
+
75
+ ```bash
76
+ MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync[steem]
77
+ ```
78
+
79
+ Or, you can have meeseeker automatically use random Hive nodes:
80
+
81
+ ```bash
82
+ MEESEEKER_NODE_URL=shuffle meeseeker sync
71
83
  ```
72
84
 
73
85
  To sync from the head block instead of the last irreversible block:
@@ -82,7 +94,7 @@ To ignore virtual operations (useful if the node doesn't enable `get_ops_in_bloc
82
94
  MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
83
95
  ```
84
96
 
85
- Normally, block headers are added to the `steem:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `steem:block` channel to only publish with the `block_num`:
97
+ Normally, block headers are added to the `hive:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `hive:block` channel to only publish with the `block_num`:
86
98
 
87
99
  ```bash
88
100
  MEESEEKER_INCLUDE_BLOCK_HEADER=false meeseeker sync
@@ -101,6 +113,12 @@ If you never want the keys to expire (not recommended), set
101
113
  MEESEEKER_EXPIRE_KEYS=-1 meeseeker sync
102
114
  ```
103
115
 
116
+ Normally, sync will create keys until it uses up all available memory. If you would like to only sync a certain number of keys, then sleep until those keys expire so it can pick up where it left off, set `MEESEEKER_MAX_KEYS` to a positive value:
117
+
118
+ ```bash
119
+ MEESEEKER_MAX_KEYS=99 meeseeker sync
120
+ ```
121
+
104
122
  ### Usage
105
123
 
106
124
  When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
@@ -113,35 +131,35 @@ For `redis-cli`, please see: https://redis.io/topics/pubsub
113
131
 
114
132
  When running `meeseeker sync`, the following channels are available:
115
133
 
116
- * `steem:block`
117
- * `steem:transaction`
118
- * `steem:op:vote`
119
- * `steem:op:comment`
120
- * `steem:op:comment_options`
121
- * `steem:op:whatever` (replace "whatever" with the op you want)
122
- * `steem:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
134
+ * `hive:block`
135
+ * `hive:transaction`
136
+ * `hive:op:vote`
137
+ * `hive:op:comment`
138
+ * `hive:op:comment_options`
139
+ * `hive:op:whatever` (replace "whatever" with the op you want)
140
+ * `hive:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
123
141
 
124
- As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.steem.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
142
+ As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.hive.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
125
143
 
126
- In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `steem:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `steem:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
144
+ In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `hive:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `hive:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
127
145
 
128
146
  For example, from `redis-cli`, if we wanted to stream block numbers:
129
147
 
130
148
  ```bash
131
149
  $ redis-cli
132
- 127.0.0.1:6379> subscribe steem:block
150
+ 127.0.0.1:6379> subscribe hive:block
133
151
  Reading messages... (press Ctrl-C to quit)
134
152
  1) "subscribe"
135
- 2) "steem:block"
153
+ 2) "hive:block"
136
154
  3) (integer) 1
137
155
  1) "message"
138
- 2) "steem:block"
156
+ 2) "hive:block"
139
157
  3) "{\"block_num\":29861068,\"previous\":\"01c7a4cb4424b4dc0cb0cc72fd36b1644f8aeba5\",\"timestamp\":\"2019-01-28T20:55:03\",\"witness\":\"ausbitbank\",\"transaction_merkle_root\":\"a318bb82625bd78af8d8b506ccd4f53116372c8e\",\"extensions\":[]}"
140
158
  1) "message"
141
- 2) "steem:block"
159
+ 2) "hive:block"
142
160
  3) "{\"block_num\":29861069,\"previous\":\"01c7a4cc1bed060876cab57476846a91568a9f8a\",\"timestamp\":\"2019-01-28T20:55:06\",\"witness\":\"followbtcnews\",\"transaction_merkle_root\":\"834e05d40b9666e5ef50deb9f368c63070c0105b\",\"extensions\":[]}"
143
161
  1) "message"
144
- 2) "steem:block"
162
+ 2) "hive:block"
145
163
  3) "{\"block_num\":29861070,\"previous\":\"01c7a4cd3bbf872895654765faa4409a8e770e91\",\"timestamp\":\"2019-01-28T20:55:09\",\"witness\":\"timcliff\",\"transaction_merkle_root\":\"b2366ce9134d627e00423b28d33cc57f1e6e453f\",\"extensions\":[]}"
146
164
  ```
147
165
 
@@ -155,26 +173,26 @@ Which allows subscription to specific `id` patterns:
155
173
 
156
174
  ```
157
175
  $ redis-cli
158
- 127.0.0.1:6379> subscribe steem:op:custom_json:sm_team_reveal
176
+ 127.0.0.1:6379> subscribe hive:op:custom_json:sm_team_reveal
159
177
  Reading messages... (press Ctrl-C to quit)
160
178
  1) "subscribe"
161
- 2) "steem:op:custom_json:sm_team_reveal"
179
+ 2) "hive:op:custom_json:sm_team_reveal"
162
180
  3) (integer) 1
163
181
  1) "message"
164
- 2) "steem:op:custom_json:sm_team_reveal"
165
- 3) "{\"key\":\"steem:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
182
+ 2) "hive:op:custom_json:sm_team_reveal"
183
+ 3) "{\"key\":\"hive:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
166
184
  1) "message"
167
- 2) "steem:op:custom_json:sm_team_reveal"
168
- 3) "{\"key\":\"steem:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
185
+ 2) "hive:op:custom_json:sm_team_reveal"
186
+ 3) "{\"key\":\"hive:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
169
187
  1) "message"
170
- 2) "steem:op:custom_json:sm_team_reveal"
171
- 3) "{\"key\":\"steem:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
188
+ 2) "hive:op:custom_json:sm_team_reveal"
189
+ 3) "{\"key\":\"hive:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
172
190
  1) "message"
173
- 2) "steem:op:custom_json:sm_team_reveal"
174
- 3) "{\"key\":\"steem:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
191
+ 2) "hive:op:custom_json:sm_team_reveal"
192
+ 3) "{\"key\":\"hive:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
175
193
  1) "message"
176
- 2) "steem:op:custom_json:sm_team_reveal"
177
- 3) "{\"key\":\"steem:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
194
+ 2) "hive:op:custom_json:sm_team_reveal"
195
+ 3) "{\"key\":\"hive:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
178
196
  ```
179
197
 
180
198
  A `ruby` application can subscribe to a channel as well, using the `redis` gem:
@@ -185,7 +203,7 @@ require 'redis'
185
203
  url = 'redis://127.0.0.1:6379/0'
186
204
  ctx = Redis.new(url: url)
187
205
 
188
- Redis.new(url: url).subscribe('steem:op:comment') do |on|
206
+ Redis.new(url: url).subscribe('hive:op:comment') do |on|
189
207
  on.message do |channel, message|
190
208
  payload = JSON[message]
191
209
  comment = JSON[ctx.get(payload['key'])]
@@ -199,19 +217,19 @@ Many other clients are supported: https://redis.io/clients
199
217
 
200
218
  ##### Witness Schedule
201
219
 
202
- When running `meeseeker witness:schedule`, the `steem:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
220
+ When running `meeseeker witness:schedule`, the `hive:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
203
221
 
204
222
  For example, from `redis-cli`, if we wanted to subscribe to the witness schedule:
205
223
 
206
224
  ```
207
225
  $ redis-cli
208
- 127.0.0.1:6379> subscribe steem:witness:schedule
226
+ 127.0.0.1:6379> subscribe hive:witness:schedule
209
227
  Reading messages... (press Ctrl-C to quit)
210
228
  1) "subscribe"
211
- 2) "steem:witness:schedule"
229
+ 2) "hive:witness:schedule"
212
230
  3) (integer) 1
213
231
  1) "message"
214
- 2) "steem:witness:schedule"
232
+ 2) "hive:witness:schedule"
215
233
  3) "{\"id\":0,\"current_virtual_time\":\"415293532210075480213212125\",\"next_shuffle_block_num\":30035208,\"current_shuffled_witnesses\":[\"thecryptodrive\",\"timcliff\",\"utopian-io\",\"themarkymark\",\"aggroed\",\"smooth.witness\",\"someguy123\",\"gtg\",\"followbtcnews\",\"yabapmatt\",\"therealwolf\",\"ausbitbank\",\"curie\",\"clayop\",\"drakos\",\"blocktrades\",\"good-karma\",\"roelandp\",\"lukestokes.mhth\",\"liondani\",\"anyx\"],\"num_scheduled_witnesses\":21,\"elected_weight\":1,\"timeshare_weight\":5,\"miner_weight\":1,\"witness_pay_normalization_factor\":25,\"median_props\":{\"account_creation_fee\":{\"amount\":\"3000\",\"precision\":3,\"nai\":\"@@000000021\"},\"maximum_block_size\":65536,\"sbd_interest_rate\":0,\"account_subsidy_budget\":797,\"account_subsidy_decay\":347321},\"majority_version\":\"0.20.8\",\"max_voted_witnesses\":20,\"max_miner_witnesses\":0,\"max_runner_witnesses\":1,\"hardfork_required_witnesses\":17,\"account_subsidy_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":797,\"pool_eq\":157691079,\"max_pool_size\":157691079,\"decay_params\":{\"decay_per_time_unit\":347321,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":0},\"account_subsidy_witness_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":996,\"pool_eq\":9384019,\"max_pool_size\":9384019,\"decay_params\":{\"decay_per_time_unit\":7293741,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":257},\"min_witness_account_subsidy_decay\":0}"
216
234
  ```
217
235
 
@@ -232,120 +250,120 @@ See: https://redis.io/commands/scan#scan-basic-usage
232
250
  Once your sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
233
251
 
234
252
  ```bash
235
- redis-cli --scan --pattern 'steem:*:vote'
253
+ redis-cli --scan --pattern 'hive:*:vote'
236
254
  ```
237
255
 
238
256
  This returns the keys, for example:
239
257
 
240
258
  ```
241
- steem:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
242
- steem:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
243
- steem:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
244
- steem:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
245
- steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
246
- steem:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
247
- steem:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
248
- steem:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
249
- steem:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
259
+ hive:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
260
+ hive:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
261
+ hive:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
262
+ hive:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
263
+ hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
264
+ hive:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
265
+ hive:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
266
+ hive:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
267
+ hive:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
250
268
  ```
251
269
 
252
270
  To get the actual vote operation for a particular key, use:
253
271
 
254
272
  ```bash
255
- redis-cli get steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
273
+ redis-cli get hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
256
274
  ```
257
275
 
258
276
  If, on the other hand, you want `custom_json` only:
259
277
 
260
278
  ```bash
261
- redis-cli --scan --pattern 'steem:*:custom_json'
279
+ redis-cli --scan --pattern 'hive:*:custom_json'
262
280
  ```
263
281
 
264
282
  This only returns the related keys, for example:
265
283
 
266
284
  ```
267
- steem:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
268
- steem:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
269
- steem:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
270
- steem:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
271
- steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
272
- steem:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
285
+ hive:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
286
+ hive:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
287
+ hive:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
288
+ hive:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
289
+ hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
290
+ hive:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
273
291
  ```
274
292
 
275
293
  To get the actual custom json operation for a particular key, use:
276
294
 
277
295
  ```bash
278
- redis-cli get steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
296
+ redis-cli get hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
279
297
  ```
280
298
 
281
299
  To get all transactions for a particular block number:
282
300
 
283
301
  ```bash
284
- redis-cli --scan --pattern 'steem:29811085:*'
302
+ redis-cli --scan --pattern 'hive:29811085:*'
285
303
  ```
286
304
 
287
305
  Or to get all ops for a particular transaction:
288
306
 
289
307
  ```bash
290
- redis-cli --scan --pattern 'steem:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
308
+ redis-cli --scan --pattern 'hive:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
291
309
  ```
292
310
 
293
- ### Steem Engine Support
311
+ ### Hive Engine Support
294
312
 
295
- As of `v0.0.6`, meeseeker can also follow the Steem Engine side-chain. This is optional and requires a separate process.
313
+ As of `v0.0.6`, meeseeker can also follow the Hive Engine side-chain. This is optional and requires a separate process.
296
314
 
297
- To sync Steem Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
315
+ To sync Hive Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
298
316
 
299
317
  ```bash
300
- meeseeker sync steem_engine
301
- ```
302
-
303
- When running `meeseeker sync steem_engine`, the following channels are available:
304
-
305
- * `steem_engine:block`
306
- * `steem_engine:transaction`
307
- * `steem_engine:virtual_transaction`
308
- * `steem_engine:contract`
309
- * `steem_engine:contract:deploy`
310
- * `steem_engine:contract:update`
311
- * `steem_engine:market`
312
- * `steem_engine:market:buy`
313
- * `steem_engine:market:cancel`
314
- * `steem_engine:market:sell`
315
- * `steem_engine:sscstore`
316
- * `steem_engine:sscstore:buy`
317
- * `steem_engine:steempegged`
318
- * `steem_engine:steempegged:buy`
319
- * `steem_engine:steempegged:removeWithdrawal`
320
- * `steem_engine:steempegged:withdraw`
321
- * `steem_engine:tokens`
322
- * `steem_engine:tokens:checkPendingUnstake`
323
- * `steem_engine:tokens:create`
324
- * `steem_engine:tokens:enableStaking`
325
- * `steem_engine:tokens:issue`
326
- * `steem_engine:tokens:transfer`
327
- * `steem_engine:tokens:transferOwnership`
328
- * `steem_engine:tokens:unstake`
329
- * `steem_engine:tokens:updateMetadata`
330
- * `steem_engine:tokens:updateParams`
331
- * `steem_engine:tokens:updateUrl`
318
+ meeseeker sync hive_engine
319
+ ```
320
+
321
+ When running `meeseeker sync hive_engine`, the following channels are available:
322
+
323
+ * `hive_engine:block`
324
+ * `hive_engine:transaction`
325
+ * `hive_engine:virtual_transaction`
326
+ * `hive_engine:contract`
327
+ * `hive_engine:contract:deploy`
328
+ * `hive_engine:contract:update`
329
+ * `hive_engine:market`
330
+ * `hive_engine:market:buy`
331
+ * `hive_engine:market:cancel`
332
+ * `hive_engine:market:sell`
333
+ * `hive_engine:sscstore`
334
+ * `hive_engine:sscstore:buy`
335
+ * `hive_engine:steempegged`
336
+ * `hive_engine:steempegged:buy`
337
+ * `hive_engine:steempegged:removeWithdrawal`
338
+ * `hive_engine:steempegged:withdraw`
339
+ * `hive_engine:tokens`
340
+ * `hive_engine:tokens:checkPendingUnstake`
341
+ * `hive_engine:tokens:create`
342
+ * `hive_engine:tokens:enableStaking`
343
+ * `hive_engine:tokens:issue`
344
+ * `hive_engine:tokens:transfer`
345
+ * `hive_engine:tokens:transferOwnership`
346
+ * `hive_engine:tokens:unstake`
347
+ * `hive_engine:tokens:updateMetadata`
348
+ * `hive_engine:tokens:updateParams`
349
+ * `hive_engine:tokens:updateUrl`
332
350
 
333
351
  The above "channel/action" patterns are the ones that are known that the time of writing. In addition, if a new contract is added or updated, meeseeker will automatically publish to these corresponding channels as they appear, without needing to update or even restart meeseeker.
334
352
 
335
353
  See main section on [Using `SUBSCRIBE`](#using-subscribe).
336
354
 
337
- Once your SteemEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
355
+ Once your HiveEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
338
356
 
339
357
  ```bash
340
- redis-cli --scan --pattern 'steem_engine:*:tokens:transfer'
358
+ redis-cli --scan --pattern 'hive_engine:*:tokens:transfer'
341
359
  ```
342
360
 
343
361
  This returns the keys, for example:
344
362
 
345
363
  ```
346
- steem_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
347
- steem_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
348
- steem_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
364
+ hive_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
365
+ hive_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
366
+ hive_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
349
367
  ```
350
368
 
351
369
  See main section on [Using `SCAN`](#using-scan).
@@ -376,11 +394,11 @@ Also see: https://hub.docker.com/r/inertia/meeseeker/
376
394
  <img src="https://i.imgur.com/Y3Sa2GW.jpg" />
377
395
  </center>
378
396
 
379
- See some of my previous Ruby How To posts in: [#radiator](https://steemit.com/created/radiator) [#ruby](https://steemit.com/created/ruby)
397
+ See some of my previous Ruby How To posts in: [#radiator](https://hive.blog/created/radiator) [#ruby](https://hive.blog/created/ruby)
380
398
 
381
399
  ## Get in touch!
382
400
 
383
- If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on STEEM.
401
+ If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on Hive.
384
402
 
385
403
  ## License
386
404
 
data/Rakefile CHANGED
@@ -52,29 +52,35 @@ task :check_schema do
52
52
  end
53
53
 
54
54
  task(:sync, [:chain, :at_block_num] => [:check_schema]) do |t, args|
55
- chain = (args[:chain] || 'steem').to_sym
55
+ chain = args[:chain] if args[:chain]
56
+ chain ||= Meeseeker.default_chain_key_prefix
56
57
 
57
- job = case chain
58
- when :steem
59
- Meeseeker::BlockFollowerJob.new
58
+ job = case chain.to_sym
60
59
  when :steem_engine
61
60
  Meeseeker::SteemEngine::FollowerJob.new
62
- else; abort("Unknown chain: #{chain}")
61
+ when :hive_engine
62
+ Meeseeker::HiveEngine::FollowerJob.new
63
+ else
64
+ Meeseeker::BlockFollowerJob.new
63
65
  end
64
66
 
65
- job.perform(at_block_num: args[:at_block_num])
67
+ job.perform(chain: chain, at_block_num: args[:at_block_num])
66
68
  end
67
69
 
68
70
  namespace :witness do
69
- desc 'Publish the witness schedule every minute or so (steem:witness:schedule).'
70
- task :schedule do
71
+ desc 'Publish the witness schedule every minute or so (e.g.: hive:witness:schedule).'
72
+ task :schedule, [:chain] do |t, args|
73
+ chain = args[:chain] if args[:chain]
74
+ chain ||= Meeseeker.default_chain_key_prefix
75
+
71
76
  job = Meeseeker::WitnessScheduleJob.new
72
- job.perform
77
+ job.perform(chain: chain)
73
78
  end
74
79
  end
75
80
 
76
81
  task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
77
- chain = (args[:chain] || 'steem').downcase.to_sym
82
+ chain = args[:chain] if args[:chain]
83
+ chain ||= Meeseeker.default_chain_key_prefix
78
84
  redis = Meeseeker.redis
79
85
 
80
86
  match = case args[:what].downcase.to_sym
@@ -99,11 +105,15 @@ task :reset, [:chain] => [:check_schema] do |t, args|
99
105
  print 'Dropping keys for set: %s ...' % chain.to_s
100
106
 
101
107
  case chain
102
- when :steem then keys += Meeseeker.redis.keys('steem:*')
103
108
  when :steem_engine then keys += Meeseeker.redis.keys('steem_engine:*')
109
+ when :hive_engine then keys += Meeseeker.redis.keys('hive_engine:*')
104
110
  when :all
105
111
  keys += Meeseeker.redis.keys('steem:*')
112
+ keys += Meeseeker.redis.keys('hive:*')
106
113
  keys += Meeseeker.redis.keys('steem_engine:*')
114
+ keys += Meeseeker.redis.keys('hive_engine:*')
115
+ else
116
+ keys += Meeseeker.redis.keys("#{chain}:*")
107
117
  end
108
118
 
109
119
  if keys.any?
@@ -117,10 +127,13 @@ end
117
127
 
118
128
  namespace :verify do
119
129
  desc 'Verifies transactions land where they should.'
120
- task :block_org, [:max_blocks] do |t, args|
130
+ task :block_org, [:chain, :max_blocks] do |t, args|
131
+ chain = args[:chain] if args[:chain]
132
+ chain ||= Meeseeker.default_chain_key_prefix
133
+ chain_key_prefix = chain.to_s
121
134
  max_blocks = args[:max_blocks]
122
- node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
123
- database_api = Steem::DatabaseApi.new(url: node_url)
135
+ node_url = Meeseeker.shuffle_node_url(chain)
136
+ database_api = Meeseeker.database_api_class(chain).new(url: node_url)
124
137
  mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
125
138
  until_block_num = if !!max_blocks
126
139
  database_api.get_dynamic_global_properties do |dgpo|
@@ -139,7 +152,7 @@ namespace :verify do
139
152
 
140
153
  loop do
141
154
  begin
142
- job.perform(mode: mode, until_block_num: until_block_num)
155
+ job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
143
156
  rescue => e
144
157
  puts e.inspect
145
158
  sleep 5
@@ -152,8 +165,8 @@ namespace :verify do
152
165
  end
153
166
 
154
167
  begin
155
- block_api = Steem::BlockApi.new(url: node_url)
156
- block_channel = 'steem:block'
168
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
169
+ block_channel = "#{chain_key_prefix}:block"
157
170
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
158
171
  subscription = Redis.new(url: redis_url)
159
172
  ctx = Redis.new(url: redis_url)
@@ -186,14 +199,21 @@ namespace :verify do
186
199
  end
187
200
  end
188
201
 
189
- while ctx.keys("steem:#{next_block_num}:*").size == 0
202
+ 5.times do
203
+ break unless ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
204
+
190
205
  # This ensures at least the next block has been indexed before
191
206
  # proceeding.
192
207
 
193
- puts "Waiting for block: #{next_block_num} ..."
208
+ puts "Waiting for block (verify:block_org): #{next_block_num} ..."
209
+
194
210
  sleep 6
195
211
  end
196
212
 
213
+ if ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
214
+ puts "Gave up waiting for block (check current_aslot slippage): #{next_block_num}"
215
+ end
216
+
197
217
  database_api.get_dynamic_global_properties do |dgpo|
198
218
  raise 'Got empty dynamic_global_properties result.' if dgpo.nil?
199
219
 
@@ -207,7 +227,7 @@ namespace :verify do
207
227
  end
208
228
 
209
229
  # In theory, we should have all the keys using this pattern.
210
- keys = ctx.keys("steem:#{block_num}:*")
230
+ keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
211
231
 
212
232
  # If we have all the keys, we should also have all transaction ids.
213
233
  expected_ids = keys.map { |k| k.split(':')[2] }.uniq
@@ -252,18 +272,35 @@ namespace :verify do
252
272
  end
253
273
  end
254
274
 
275
+ desc 'Verifies Hive Engine transactions land where they should.'
276
+ task :hive_engine_block_org, [:max_blocks] do |t, args|
277
+ Rake::Task['verify:engine_block_org'].invoke('hive_engine', args[:max_blocks])
278
+ end
279
+
255
280
  desc 'Verifies Steem Engine transactions land where they should.'
256
281
  task :steem_engine_block_org, [:max_blocks] do |t, args|
282
+ Rake::Task['verify:engine_block_org'].invoke('steem_engine', args[:max_blocks])
283
+ end
284
+
285
+ desc 'Verifies Steem/Hive Engine transactions land where they should.'
286
+ task :engine_block_org, [:chain_key_prefix, :max_blocks] do |t, args|
287
+ chain_key_prefix = args[:chain_key_prefix]
257
288
  max_blocks = args[:max_blocks]
258
- node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
259
- agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
289
+ case chain_key_prefix.to_sym
290
+ when :steem_engine
291
+ node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
292
+ agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
293
+ job = Meeseeker::SteemEngine::FollowerJob.new
294
+ when :hive_engine
295
+ node_url = ENV.fetch('MEESEEKER_HIVE_ENGINE_NODE_URL', 'https://api.hive-engine.com/rpc')
296
+ agent = Meeseeker::HiveEngine::Agent.new(url: node_url)
297
+ job = Meeseeker::HiveEngine::FollowerJob.new
298
+ end
260
299
  until_block_num = if !!max_blocks
261
300
  agent.latest_block_info['blockNumber']
262
301
  end
263
302
 
264
303
  Thread.new do
265
- job = Meeseeker::SteemEngine::FollowerJob.new
266
-
267
304
  loop do
268
305
  begin
269
306
  at_block_num = agent.latest_block_info["blockNumber"] - max_blocks.to_i
@@ -281,7 +318,7 @@ namespace :verify do
281
318
  end
282
319
 
283
320
  begin
284
- block_channel = 'steem_engine:block'
321
+ block_channel = "#{chain_key_prefix}:block"
285
322
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
286
323
  subscription = Redis.new(url: redis_url)
287
324
  ctx = Redis.new(url: redis_url)
@@ -313,16 +350,16 @@ namespace :verify do
313
350
  end
314
351
  end
315
352
 
316
- while ctx.keys("steem_engine:#{next_block_num}:*").size == 0
353
+ while ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
317
354
  # This ensures at least the next block has been indexed before
318
355
  # proceeding.
319
356
 
320
- puts "Waiting for block: #{next_block_num} ..."
357
+ puts "Waiting for block (verify:#{chain_key_prefix}_engine_block_org): #{next_block_num} ..."
321
358
  sleep 6
322
359
  end
323
360
 
324
361
  # In theory, we should have all the keys using this pattern.
325
- keys = ctx.keys("steem_engine:#{block_num}:*")
362
+ keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
326
363
 
327
364
  # If we have all the keys, we should also have all transaction ids.
328
365
  expected_ids = keys.map { |k| k.split(':')[2] }.uniq
@@ -368,19 +405,40 @@ namespace :verify do
368
405
  agent.shutdown
369
406
  end
370
407
 
408
+ desc 'Verifies Hive Engine sidechain against the mainnet.'
409
+ task :hive_engine_ref_blocks do |t|
410
+ Rake::Task['verify:engine_ref_blocks'].invoke('hive_engine')
411
+ end
412
+
371
413
  desc 'Verifies Steem Engine sidechain against the mainnet.'
372
414
  task :steem_engine_ref_blocks do |t|
415
+ Rake::Task['verify:engine_ref_blocks'].invoke('steem_engine')
416
+ end
417
+
418
+ desc 'Verifies Steem/Hive Engine sidechain against the mainnet.'
419
+ task :engine_ref_blocks, [:chain_key_prefix] do |t, args|
420
+ chain_key_prefix = args[:chain_key_prefix]
373
421
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
374
422
  ctx = ctx = Redis.new(url: redis_url)
375
- keys = ctx.keys('steem_engine:*')
376
- block_api = Steem::BlockApi.new
423
+ keys = ctx.keys("#{chain_key_prefix}:*:*")
424
+ mainchain, mainchain_url = case chain_key_prefix
425
+ when 'steem_engine' then ['steem', 'https://api.steemit.com']
426
+ when 'hive_engine' then ['hive', 'https://api.openhive.network']
427
+ end
428
+ block_api = Meeseeker.block_api_class(mainchain).new(url: mainchain_url)
377
429
  block_trxs = {}
378
430
 
379
- puts "Checking Steem Engine keys: #{keys.size}"
431
+ puts "Checking #{chain_key_prefix} keys: #{keys.size}"
380
432
 
381
433
  keys.each do |key|
382
434
  transaction = JSON[ctx.get(key)]
383
- block_num = transaction['refSteemBlockNumber']
435
+
436
+ next if transaction.class == Integer
437
+
438
+ block_num = case chain_key_prefix
439
+ when 'steem_engine' then transaction.fetch('refSteemBlockNumber')
440
+ when 'hive_engine' then transaction.fetch('refHiveBlockNumber')
441
+ end.to_i
384
442
 
385
443
  block_trxs[block_num] ||= []
386
444
  block_trxs[block_num] << transaction['transactionId'].to_s.split('-').first
@@ -390,6 +448,8 @@ namespace :verify do
390
448
 
391
449
  skipped_blocks = []
392
450
 
451
+ next if block_trxs.empty?
452
+
393
453
  block_api.get_blocks(block_range: block_trxs.keys) do |block, block_num|
394
454
  if block.nil? || block[:transaction_ids].nil?
395
455
  print 'S'
@@ -399,24 +459,34 @@ namespace :verify do
399
459
  else
400
460
  print '.'
401
461
  end
402
-
403
- if (block.transaction_ids & block_trxs[block_num]).none?
462
+
463
+ trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
464
+
465
+ if trx_ids.any? && (block.transaction_ids & trx_ids).none?
404
466
  puts "\nNo intersection in #{block_num}!"
405
- puts "Expected the following sidechain trx_ids: #{block_trxs[block_num].join(', ')}"
467
+ puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
406
468
  end
407
469
  end
408
470
 
409
471
  puts "\nBlocks to retry: #{skipped_blocks.size}"
410
472
 
411
473
  skipped_blocks.each do |block_num|
474
+ block_found = false
475
+
412
476
  block_api.get_block(block_num: block_num) do |result|
477
+ break unless !!result.block
478
+
413
479
  block = result.block
480
+ block_found = true
481
+ trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
414
482
 
415
- if (block.transaction_ids & block_trxs[block_num]).none?
483
+ if trx_ids.any? && (block.transaction_ids & trx_ids).none?
416
484
  puts "No intersection in #{block_num}!"
417
- puts "Expected the following sidechain trx_ids: #{block_trxs[block_num].join(', ')}"
485
+ puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
418
486
  end
419
487
  end
488
+
489
+ redo unless block_found
420
490
  end
421
491
 
422
492
  puts "Done."
@@ -424,10 +494,12 @@ namespace :verify do
424
494
 
425
495
  namespace :witness do
426
496
  desc 'Verifies witnessses in the schedule produced a block.'
427
- task :schedule, [:max_blocks] do |t, args|
497
+ task :schedule, [:chain, :max_blocks] do |t, args|
498
+ chain = args[:chain] if !!args[:chain]
499
+ chain ||= Meeseeker.default_chain_key_prefix
428
500
  max_blocks = args[:max_blocks]
429
- node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
430
- database_api = Steem::DatabaseApi.new(url: node_url)
501
+ node_url = Meeseeker.shuffle_node_url(chain)
502
+ database_api = Meeseeker.database_api_class(chain).new(url: node_url)
431
503
  mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
432
504
  until_block_num = if !!max_blocks
433
505
  database_api.get_dynamic_global_properties do |dgpo|
@@ -446,7 +518,7 @@ namespace :verify do
446
518
 
447
519
  loop do
448
520
  begin
449
- job.perform(mode: mode, until_block_num: until_block_num)
521
+ job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
450
522
  rescue => e
451
523
  puts e.inspect
452
524
  sleep 5
@@ -459,8 +531,10 @@ namespace :verify do
459
531
  end
460
532
 
461
533
  begin
462
- block_api = Steem::BlockApi.new(url: node_url)
463
- schedule_channel = 'steem:witness:schedule'
534
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
535
+ chain_key_prefix = chain.to_s if args[:chain]
536
+ chain_key_prefix ||= Meeseeker.default_chain_key_prefix
537
+ schedule_channel = "#{chain_key_prefix}:witness:schedule"
464
538
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
465
539
  subscription = Redis.new(url: redis_url)
466
540
  timeout = (max_blocks).to_i * 3
@@ -519,7 +593,10 @@ namespace :verify do
519
593
  unless !!header
520
594
  # Can happen when there's excess p2p latency and/or jussi
521
595
  # cache is under load.
522
- puts "Waiting for block header: #{block_num}"
596
+ puts "Waiting for block header (witness:schedule): #{block_num}"
597
+
598
+ node_url = Meeseeker.shuffle_node_url(chain)
599
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
523
600
 
524
601
  next
525
602
  end