meeseeker 0.0.8 → 2.0.1.pre.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 3204b511fca2e18ee8612eb32cd520edaf8bc72e
4
- data.tar.gz: 803ab700225d7781a0de824ed1f3221f1dce8033
2
+ SHA256:
3
+ metadata.gz: e6750a9b00a2ab979ca2944add9cdee2b31c16466821445c32e1ef9e7de638a0
4
+ data.tar.gz: 4489e40ea24703752ad9c359e1440fe4db66c20e3fe9e710f93f8657a49e50eb
5
5
  SHA512:
6
- metadata.gz: '079b763077af4e430b84183e96b2bc683668a89677aa6d90bf4238f8a1533200c1c0c64ffe3e00a2f838ecb1e99e6c12a93f3be9ea4e188b24e501de8cd596cd'
7
- data.tar.gz: 13cc852acec9b743017e20dc58c6566f9acb0f513a0feb30187536540abe8cd6aacb09707f981adf0f69022b5bdb699211fb76096c6a8ee84d310ea1869ef109
6
+ metadata.gz: fd4df8812f0252737c0a43fc1a0db029c2361277dc3d6074e6fd3e642c7fdbabbcbc3b2b30cd56f9ea0a103540ef1146601873c303c106516a8fb2a815ee1bb3
7
+ data.tar.gz: 94895f366e1c9ebc3e1c38e731130d19f4f39b231d1593c877a21085254171afdb7bf2f59ebafbed32f447af98eb188ccce3efdbeef95465c1e66079ffb09c6b
data/README.md CHANGED
@@ -1,6 +1,8 @@
1
1
  # meeseeker
2
2
 
3
- Redis based block follower is an efficient way for multiple apps to stream the Steem Blockchain.
3
+ Redis based block follower is an efficient way for multiple apps to stream the Hive Blockchain.
4
+
5
+ [![Build Status](https://travis-ci.org/inertia186/meeseeker.svg?branch=master)](https://travis-ci.org/inertia186/meeseeker)
4
6
 
5
7
  If you have multiple applications that need to perform actions as operations occur, `meeseeker` will allow your apps to each perform actions for specific operations without each app having to stream the entire blockchain.
6
8
 
@@ -62,10 +64,22 @@ To specify an alternative redis source:
62
64
  MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync
63
65
  ```
64
66
 
65
- You can also specify an alternative Steem node:
67
+ You can also specify am alternative Hive node:
66
68
 
67
69
  ```bash
68
- MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync
70
+ MEESEEKER_NODE_URL=http://anyx.io meeseeker sync
71
+ ```
72
+
73
+ You can also specify a Steem node instead of Hive (if that's your thing):
74
+
75
+ ```bash
76
+ MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync[steem]
77
+ ```
78
+
79
+ Or, you can have meeseeker automatically use random Hive nodes:
80
+
81
+ ```bash
82
+ MEESEEKER_NODE_URL=shuffle meeseeker sync
69
83
  ```
70
84
 
71
85
  To sync from the head block instead of the last irreversible block:
@@ -80,7 +94,7 @@ To ignore virtual operations (useful if the node doesn't enable `get_ops_in_bloc
80
94
  MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
81
95
  ```
82
96
 
83
- Normally, block headers are added to the `steem:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `steem:block` channel to only publish with the `block_num`:
97
+ Normally, block headers are added to the `hive:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `hive:block` channel to only publish with the `block_num`:
84
98
 
85
99
  ```bash
86
100
  MEESEEKER_INCLUDE_BLOCK_HEADER=false meeseeker sync
@@ -99,6 +113,12 @@ If you never want the keys to expire (not recommended), set
99
113
  MEESEEKER_EXPIRE_KEYS=-1 meeseeker sync
100
114
  ```
101
115
 
116
+ Normally, sync will create keys until it uses up all available memory. If you would like to only sync a certain number of keys, then sleep until those keys expire so it can pick up where it left off, set `MEESEEKER_MAX_KEYS` to a positive value:
117
+
118
+ ```bash
119
+ MEESEEKER_MAX_KEYS=99 meeseeker sync
120
+ ```
121
+
102
122
  ### Usage
103
123
 
104
124
  When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
@@ -111,35 +131,35 @@ For `redis-cli`, please see: https://redis.io/topics/pubsub
111
131
 
112
132
  When running `meeseeker sync`, the following channels are available:
113
133
 
114
- * `steem:block`
115
- * `steem:transaction`
116
- * `steem:op:vote`
117
- * `steem:op:comment`
118
- * `steem:op:comment_options`
119
- * `steem:op:whatever` (replace "whatever" with the op you want)
120
- * `steem:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
134
+ * `hive:block`
135
+ * `hive:transaction`
136
+ * `hive:op:vote`
137
+ * `hive:op:comment`
138
+ * `hive:op:comment_options`
139
+ * `hive:op:whatever` (replace "whatever" with the op you want)
140
+ * `hive:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
121
141
 
122
- As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.steem.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
142
+ As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.hive.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
123
143
 
124
- In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `steem:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `steem:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
144
+ In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `hive:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `hive:op:custom_json:sm_team_reveal`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
125
145
 
126
146
  For example, from `redis-cli`, if we wanted to stream block numbers:
127
147
 
128
148
  ```bash
129
149
  $ redis-cli
130
- 127.0.0.1:6379> subscribe steem:block
150
+ 127.0.0.1:6379> subscribe hive:block
131
151
  Reading messages... (press Ctrl-C to quit)
132
152
  1) "subscribe"
133
- 2) "steem:block"
153
+ 2) "hive:block"
134
154
  3) (integer) 1
135
155
  1) "message"
136
- 2) "steem:block"
156
+ 2) "hive:block"
137
157
  3) "{\"block_num\":29861068,\"previous\":\"01c7a4cb4424b4dc0cb0cc72fd36b1644f8aeba5\",\"timestamp\":\"2019-01-28T20:55:03\",\"witness\":\"ausbitbank\",\"transaction_merkle_root\":\"a318bb82625bd78af8d8b506ccd4f53116372c8e\",\"extensions\":[]}"
138
158
  1) "message"
139
- 2) "steem:block"
159
+ 2) "hive:block"
140
160
  3) "{\"block_num\":29861069,\"previous\":\"01c7a4cc1bed060876cab57476846a91568a9f8a\",\"timestamp\":\"2019-01-28T20:55:06\",\"witness\":\"followbtcnews\",\"transaction_merkle_root\":\"834e05d40b9666e5ef50deb9f368c63070c0105b\",\"extensions\":[]}"
141
161
  1) "message"
142
- 2) "steem:block"
162
+ 2) "hive:block"
143
163
  3) "{\"block_num\":29861070,\"previous\":\"01c7a4cd3bbf872895654765faa4409a8e770e91\",\"timestamp\":\"2019-01-28T20:55:09\",\"witness\":\"timcliff\",\"transaction_merkle_root\":\"b2366ce9134d627e00423b28d33cc57f1e6e453f\",\"extensions\":[]}"
144
164
  ```
145
165
 
@@ -153,26 +173,26 @@ Which allows subscription to specific `id` patterns:
153
173
 
154
174
  ```
155
175
  $ redis-cli
156
- 127.0.0.1:6379> subscribe steem:op:custom_json:sm_team_reveal
176
+ 127.0.0.1:6379> subscribe hive:op:custom_json:sm_team_reveal
157
177
  Reading messages... (press Ctrl-C to quit)
158
178
  1) "subscribe"
159
- 2) "steem:op:custom_json:sm_team_reveal"
179
+ 2) "hive:op:custom_json:sm_team_reveal"
160
180
  3) (integer) 1
161
181
  1) "message"
162
- 2) "steem:op:custom_json:sm_team_reveal"
163
- 3) "{\"key\":\"steem:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
182
+ 2) "hive:op:custom_json:sm_team_reveal"
183
+ 3) "{\"key\":\"hive:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
164
184
  1) "message"
165
- 2) "steem:op:custom_json:sm_team_reveal"
166
- 3) "{\"key\":\"steem:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
185
+ 2) "hive:op:custom_json:sm_team_reveal"
186
+ 3) "{\"key\":\"hive:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
167
187
  1) "message"
168
- 2) "steem:op:custom_json:sm_team_reveal"
169
- 3) "{\"key\":\"steem:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
188
+ 2) "hive:op:custom_json:sm_team_reveal"
189
+ 3) "{\"key\":\"hive:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
170
190
  1) "message"
171
- 2) "steem:op:custom_json:sm_team_reveal"
172
- 3) "{\"key\":\"steem:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
191
+ 2) "hive:op:custom_json:sm_team_reveal"
192
+ 3) "{\"key\":\"hive:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
173
193
  1) "message"
174
- 2) "steem:op:custom_json:sm_team_reveal"
175
- 3) "{\"key\":\"steem:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
194
+ 2) "hive:op:custom_json:sm_team_reveal"
195
+ 3) "{\"key\":\"hive:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
176
196
  ```
177
197
 
178
198
  A `ruby` application can subscribe to a channel as well, using the `redis` gem:
@@ -183,7 +203,7 @@ require 'redis'
183
203
  url = 'redis://127.0.0.1:6379/0'
184
204
  ctx = Redis.new(url: url)
185
205
 
186
- Redis.new(url: url).subscribe('steem:op:comment') do |on|
206
+ Redis.new(url: url).subscribe('hive:op:comment') do |on|
187
207
  on.message do |channel, message|
188
208
  payload = JSON[message]
189
209
  comment = JSON[ctx.get(payload['key'])]
@@ -197,19 +217,19 @@ Many other clients are supported: https://redis.io/clients
197
217
 
198
218
  ##### Witness Schedule
199
219
 
200
- When running `meeseeker witness:schedule`, the `steem:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
220
+ When running `meeseeker witness:schedule`, the `hive:witness:schedule` channel is available. This is offered as a separate command because most applications don't need to worry about this level of blockchain logistics.
201
221
 
202
222
  For example, from `redis-cli`, if we wanted to subscribe to the witness schedule:
203
223
 
204
224
  ```
205
225
  $ redis-cli
206
- 127.0.0.1:6379> subscribe steem:witness:schedule
226
+ 127.0.0.1:6379> subscribe hive:witness:schedule
207
227
  Reading messages... (press Ctrl-C to quit)
208
228
  1) "subscribe"
209
- 2) "steem:witness:schedule"
229
+ 2) "hive:witness:schedule"
210
230
  3) (integer) 1
211
231
  1) "message"
212
- 2) "steem:witness:schedule"
232
+ 2) "hive:witness:schedule"
213
233
  3) "{\"id\":0,\"current_virtual_time\":\"415293532210075480213212125\",\"next_shuffle_block_num\":30035208,\"current_shuffled_witnesses\":[\"thecryptodrive\",\"timcliff\",\"utopian-io\",\"themarkymark\",\"aggroed\",\"smooth.witness\",\"someguy123\",\"gtg\",\"followbtcnews\",\"yabapmatt\",\"therealwolf\",\"ausbitbank\",\"curie\",\"clayop\",\"drakos\",\"blocktrades\",\"good-karma\",\"roelandp\",\"lukestokes.mhth\",\"liondani\",\"anyx\"],\"num_scheduled_witnesses\":21,\"elected_weight\":1,\"timeshare_weight\":5,\"miner_weight\":1,\"witness_pay_normalization_factor\":25,\"median_props\":{\"account_creation_fee\":{\"amount\":\"3000\",\"precision\":3,\"nai\":\"@@000000021\"},\"maximum_block_size\":65536,\"sbd_interest_rate\":0,\"account_subsidy_budget\":797,\"account_subsidy_decay\":347321},\"majority_version\":\"0.20.8\",\"max_voted_witnesses\":20,\"max_miner_witnesses\":0,\"max_runner_witnesses\":1,\"hardfork_required_witnesses\":17,\"account_subsidy_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":797,\"pool_eq\":157691079,\"max_pool_size\":157691079,\"decay_params\":{\"decay_per_time_unit\":347321,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":0},\"account_subsidy_witness_rd\":{\"resource_unit\":10000,\"budget_per_time_unit\":996,\"pool_eq\":9384019,\"max_pool_size\":9384019,\"decay_params\":{\"decay_per_time_unit\":7293741,\"decay_per_time_unit_denom_shift\":36},\"min_decay\":257},\"min_witness_account_subsidy_decay\":0}"
214
234
  ```
215
235
 
@@ -230,111 +250,120 @@ See: https://redis.io/commands/scan#scan-basic-usage
230
250
  Once your sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
231
251
 
232
252
  ```bash
233
- redis-cli --scan --pattern 'steem:*:vote'
253
+ redis-cli --scan --pattern 'hive:*:vote'
234
254
  ```
235
255
 
236
256
  This returns the keys, for example:
237
257
 
238
258
  ```
239
- steem:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
240
- steem:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
241
- steem:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
242
- steem:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
243
- steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
244
- steem:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
245
- steem:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
246
- steem:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
247
- steem:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
259
+ hive:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
260
+ hive:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
261
+ hive:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
262
+ hive:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
263
+ hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
264
+ hive:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
265
+ hive:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
266
+ hive:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
267
+ hive:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
248
268
  ```
249
269
 
250
270
  To get the actual vote operation for a particular key, use:
251
271
 
252
272
  ```bash
253
- redis-cli get steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
273
+ redis-cli get hive:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
254
274
  ```
255
275
 
256
276
  If, on the other hand, you want `custom_json` only:
257
277
 
258
278
  ```bash
259
- redis-cli --scan --pattern 'steem:*:custom_json'
279
+ redis-cli --scan --pattern 'hive:*:custom_json'
260
280
  ```
261
281
 
262
282
  This only returns the related keys, for example:
263
283
 
264
284
  ```
265
- steem:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
266
- steem:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
267
- steem:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
268
- steem:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
269
- steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
270
- steem:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
285
+ hive:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
286
+ hive:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
287
+ hive:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
288
+ hive:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
289
+ hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
290
+ hive:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
271
291
  ```
272
292
 
273
293
  To get the actual custom json operation for a particular key, use:
274
294
 
275
295
  ```bash
276
- redis-cli get steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
296
+ redis-cli get hive:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
277
297
  ```
278
298
 
279
299
  To get all transactions for a particular block number:
280
300
 
281
301
  ```bash
282
- redis-cli --scan --pattern 'steem:29811085:*'
302
+ redis-cli --scan --pattern 'hive:29811085:*'
283
303
  ```
284
304
 
285
305
  Or to get all ops for a particular transaction:
286
306
 
287
307
  ```bash
288
- redis-cli --scan --pattern 'steem:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
308
+ redis-cli --scan --pattern 'hive:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
289
309
  ```
290
310
 
291
- ### Steem Engine Support
311
+ ### Hive Engine Support
292
312
 
293
- As of `v0.0.6`, meeseeker can also follow the Steem Engine side-chain. This is optional and requires a separate process.
313
+ As of `v0.0.6`, meeseeker can also follow the Hive Engine side-chain. This is optional and requires a separate process.
294
314
 
295
- To sync Steem Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
315
+ To sync Hive Engine to your local redis source (also defaults to `redis://127.0.0.1:6379/0`):
296
316
 
297
317
  ```bash
298
- meeseeker sync steem_engine
299
- ```
300
-
301
- When running `meeseeker sync steem_engine`, the following channels are available:
302
-
303
- * `steem_engine:block`
304
- * `steem_engine:transaction`
305
- * `steem_engine:market`
306
- * `steem_engine:market:buy`
307
- * `steem_engine:market:cancel`
308
- * `steem_engine:market:sell`
309
- * `steem_engine:sscstore`
310
- * `steem_engine:sscstore:buy`
311
- * `steem_engine:steempegged`
312
- * `steem_engine:steempegged:buy`
313
- * `steem_engine:steempegged:removeWithdrawal`
314
- * `steem_engine:steempegged:withdraw`
315
- * `steem_engine:tokens`
316
- * `steem_engine:tokens:create`
317
- * `steem_engine:tokens:issue`
318
- * `steem_engine:tokens:transfer`
319
- * `steem_engine:tokens:updateMetadata`
320
- * `steem_engine:tokens:updateUrl`
318
+ meeseeker sync hive_engine
319
+ ```
320
+
321
+ When running `meeseeker sync hive_engine`, the following channels are available:
322
+
323
+ * `hive_engine:block`
324
+ * `hive_engine:transaction`
325
+ * `hive_engine:virtual_transaction`
326
+ * `hive_engine:contract`
327
+ * `hive_engine:contract:deploy`
328
+ * `hive_engine:contract:update`
329
+ * `hive_engine:market`
330
+ * `hive_engine:market:buy`
331
+ * `hive_engine:market:cancel`
332
+ * `hive_engine:market:sell`
333
+ * `hive_engine:sscstore`
334
+ * `hive_engine:sscstore:buy`
335
+ * `hive_engine:steempegged`
336
+ * `hive_engine:steempegged:buy`
337
+ * `hive_engine:steempegged:removeWithdrawal`
338
+ * `hive_engine:steempegged:withdraw`
339
+ * `hive_engine:tokens`
340
+ * `hive_engine:tokens:checkPendingUnstake`
341
+ * `hive_engine:tokens:create`
342
+ * `hive_engine:tokens:enableStaking`
343
+ * `hive_engine:tokens:issue`
344
+ * `hive_engine:tokens:transfer`
345
+ * `hive_engine:tokens:transferOwnership`
346
+ * `hive_engine:tokens:unstake`
347
+ * `hive_engine:tokens:updateMetadata`
348
+ * `hive_engine:tokens:updateParams`
349
+ * `hive_engine:tokens:updateUrl`
321
350
 
322
351
  The above "channel/action" patterns are the ones that are known that the time of writing. In addition, if a new contract is added or updated, meeseeker will automatically publish to these corresponding channels as they appear, without needing to update or even restart meeseeker.
323
352
 
324
353
  See main section on [Using `SUBSCRIBE`](#using-subscribe).
325
354
 
326
- Once your SteemEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
355
+ Once your HiveEngine sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
327
356
 
328
357
  ```bash
329
- redis-cli --scan --pattern 'steem_engine:*:tokens:transfer'
358
+ redis-cli --scan --pattern 'hive_engine:*:tokens:transfer'
330
359
  ```
331
360
 
332
361
  This returns the keys, for example:
333
362
 
334
363
  ```
335
- steem_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
336
- steem_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
337
- steem_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
364
+ hive_engine:18000:d414373db84e6a642f289641ea1433fda22b8a4d:0:tokens:transfer
365
+ hive_engine:18004:c9e06c8449d2d04b4a0a31ec7b80d2f62009a5f0:0:tokens:transfer
366
+ hive_engine:17994:faf097391760ad896b19d5854e2822f62dee284b:0:tokens:transfer
338
367
  ```
339
368
 
340
369
  See main section on [Using `SCAN`](#using-scan).
@@ -365,11 +394,11 @@ Also see: https://hub.docker.com/r/inertia/meeseeker/
365
394
  <img src="https://i.imgur.com/Y3Sa2GW.jpg" />
366
395
  </center>
367
396
 
368
- See some of my previous Ruby How To posts in: [#radiator](https://steemit.com/created/radiator) [#ruby](https://steemit.com/created/ruby)
397
+ See some of my previous Ruby How To posts in: [#radiator](https://hive.blog/created/radiator) [#ruby](https://hive.blog/created/ruby)
369
398
 
370
399
  ## Get in touch!
371
400
 
372
- If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on STEEM.
401
+ If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on Hive.
373
402
 
374
403
  ## License
375
404
 
data/Rakefile CHANGED
@@ -51,30 +51,38 @@ task :check_schema do
51
51
  end
52
52
  end
53
53
 
54
+ desc 'Begin/resume sync.'
54
55
  task(:sync, [:chain, :at_block_num] => [:check_schema]) do |t, args|
55
- chain = (args[:chain] || 'steem').to_sym
56
+ chain = args[:chain] if args[:chain]
57
+ chain ||= Meeseeker.default_chain_key_prefix
56
58
 
57
- job = case chain
58
- when :steem
59
- Meeseeker::BlockFollowerJob.new
59
+ job = case chain.to_sym
60
60
  when :steem_engine
61
61
  Meeseeker::SteemEngine::FollowerJob.new
62
- else; abort("Unknown chain: #{chain}")
62
+ when :hive_engine
63
+ Meeseeker::HiveEngine::FollowerJob.new
64
+ else
65
+ Meeseeker::BlockFollowerJob.new
63
66
  end
64
67
 
65
- job.perform(at_block_num: args[:at_block_num])
68
+ job.perform(chain: chain, at_block_num: args[:at_block_num])
66
69
  end
67
70
 
68
71
  namespace :witness do
69
- desc 'Publish the witness schedule every minute or so (steem:witness:schedule).'
70
- task :schedule do
72
+ desc 'Publish the witness schedule every minute or so (e.g.: hive:witness:schedule).'
73
+ task :schedule, [:chain] do |t, args|
74
+ chain = args[:chain] if args[:chain]
75
+ chain ||= Meeseeker.default_chain_key_prefix
76
+
71
77
  job = Meeseeker::WitnessScheduleJob.new
72
- job.perform
78
+ job.perform(chain: chain)
73
79
  end
74
80
  end
75
81
 
82
+ desc 'Find block or transaction.'
76
83
  task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
77
- chain = (args[:chain] || 'steem').downcase.to_sym
84
+ chain = args[:chain] if args[:chain]
85
+ chain ||= Meeseeker.default_chain_key_prefix
78
86
  redis = Meeseeker.redis
79
87
 
80
88
  match = case args[:what].downcase.to_sym
@@ -92,6 +100,7 @@ task(:find, [:what, :key, :chain] => [:check_schema]) do |t, args|
92
100
  end
93
101
  end
94
102
 
103
+ desc 'Clear all keys.'
95
104
  task :reset, [:chain] => [:check_schema] do |t, args|
96
105
  chain = (args[:chain] || 'all').to_sym
97
106
  keys = []
@@ -99,11 +108,15 @@ task :reset, [:chain] => [:check_schema] do |t, args|
99
108
  print 'Dropping keys for set: %s ...' % chain.to_s
100
109
 
101
110
  case chain
102
- when :steem then keys += Meeseeker.redis.keys('steem:*')
103
111
  when :steem_engine then keys += Meeseeker.redis.keys('steem_engine:*')
112
+ when :hive_engine then keys += Meeseeker.redis.keys('hive_engine:*')
104
113
  when :all
105
114
  keys += Meeseeker.redis.keys('steem:*')
115
+ keys += Meeseeker.redis.keys('hive:*')
106
116
  keys += Meeseeker.redis.keys('steem_engine:*')
117
+ keys += Meeseeker.redis.keys('hive_engine:*')
118
+ else
119
+ keys += Meeseeker.redis.keys("#{chain}:*")
107
120
  end
108
121
 
109
122
  if keys.any?
@@ -117,10 +130,13 @@ end
117
130
 
118
131
  namespace :verify do
119
132
  desc 'Verifies transactions land where they should.'
120
- task :block_org, [:max_blocks] do |t, args|
133
+ task :block_org, [:chain, :max_blocks] do |t, args|
134
+ chain = args[:chain] if args[:chain]
135
+ chain ||= Meeseeker.default_chain_key_prefix
136
+ chain_key_prefix = chain.to_s
121
137
  max_blocks = args[:max_blocks]
122
- node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
123
- database_api = Steem::DatabaseApi.new(url: node_url)
138
+ node_url = Meeseeker.shuffle_node_url(chain)
139
+ database_api = Meeseeker.database_api_class(chain).new(url: node_url)
124
140
  mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
125
141
  until_block_num = if !!max_blocks
126
142
  database_api.get_dynamic_global_properties do |dgpo|
@@ -131,7 +147,7 @@ namespace :verify do
131
147
  when :irreversible then dgpo.last_irreversible_block_num
132
148
  else; abort "Unknown block mode: #{mode}"
133
149
  end
134
- end + max_blocks.to_i
150
+ end + max_blocks.to_i + 1
135
151
  end
136
152
 
137
153
  Thread.new do
@@ -139,7 +155,7 @@ namespace :verify do
139
155
 
140
156
  loop do
141
157
  begin
142
- job.perform(mode: mode, until_block_num: until_block_num)
158
+ job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
143
159
  rescue => e
144
160
  puts e.inspect
145
161
  sleep 5
@@ -152,8 +168,8 @@ namespace :verify do
152
168
  end
153
169
 
154
170
  begin
155
- block_api = Steem::BlockApi.new(url: node_url)
156
- block_channel = 'steem:block'
171
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
172
+ block_channel = "#{chain_key_prefix}:block"
157
173
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
158
174
  subscription = Redis.new(url: redis_url)
159
175
  ctx = Redis.new(url: redis_url)
@@ -186,14 +202,21 @@ namespace :verify do
186
202
  end
187
203
  end
188
204
 
189
- while ctx.keys("steem:#{next_block_num}:*").size == 0
205
+ 5.times do
206
+ break unless ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
207
+
190
208
  # This ensures at least the next block has been indexed before
191
209
  # proceeding.
192
210
 
193
- puts "Waiting for block: #{next_block_num} ..."
211
+ puts "Waiting for block (verify:block_org): #{next_block_num} ..."
212
+
194
213
  sleep 6
195
214
  end
196
215
 
216
+ if ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
217
+ puts "Gave up waiting for block (check current_aslot slippage): #{next_block_num}"
218
+ end
219
+
197
220
  database_api.get_dynamic_global_properties do |dgpo|
198
221
  raise 'Got empty dynamic_global_properties result.' if dgpo.nil?
199
222
 
@@ -207,7 +230,7 @@ namespace :verify do
207
230
  end
208
231
 
209
232
  # In theory, we should have all the keys using this pattern.
210
- keys = ctx.keys("steem:#{block_num}:*")
233
+ keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
211
234
 
212
235
  # If we have all the keys, we should also have all transaction ids.
213
236
  expected_ids = keys.map { |k| k.split(':')[2] }.uniq
@@ -252,18 +275,35 @@ namespace :verify do
252
275
  end
253
276
  end
254
277
 
278
+ desc 'Verifies Hive Engine transactions land where they should.'
279
+ task :hive_engine_block_org, [:max_blocks] do |t, args|
280
+ Rake::Task['verify:engine_block_org'].invoke('hive_engine', args[:max_blocks])
281
+ end
282
+
255
283
  desc 'Verifies Steem Engine transactions land where they should.'
256
284
  task :steem_engine_block_org, [:max_blocks] do |t, args|
285
+ Rake::Task['verify:engine_block_org'].invoke('steem_engine', args[:max_blocks])
286
+ end
287
+
288
+ desc 'Verifies Steem/Hive Engine transactions land where they should.'
289
+ task :engine_block_org, [:chain_key_prefix, :max_blocks] do |t, args|
290
+ chain_key_prefix = args[:chain_key_prefix]
257
291
  max_blocks = args[:max_blocks]
258
- node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
259
- agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
292
+ case chain_key_prefix.to_sym
293
+ when :steem_engine
294
+ node_url = ENV.fetch('MEESEEKER_STEEM_ENGINE_NODE_URL', 'https://api.steem-engine.com/rpc')
295
+ agent = Meeseeker::SteemEngine::Agent.new(url: node_url)
296
+ job = Meeseeker::SteemEngine::FollowerJob.new
297
+ when :hive_engine
298
+ node_url = ENV.fetch('MEESEEKER_HIVE_ENGINE_NODE_URL', 'https://api.hive-engine.com/rpc')
299
+ agent = Meeseeker::HiveEngine::Agent.new(url: node_url)
300
+ job = Meeseeker::HiveEngine::FollowerJob.new
301
+ end
260
302
  until_block_num = if !!max_blocks
261
303
  agent.latest_block_info['blockNumber']
262
304
  end
263
305
 
264
306
  Thread.new do
265
- job = Meeseeker::SteemEngine::FollowerJob.new
266
-
267
307
  loop do
268
308
  begin
269
309
  at_block_num = agent.latest_block_info["blockNumber"] - max_blocks.to_i
@@ -281,7 +321,7 @@ namespace :verify do
281
321
  end
282
322
 
283
323
  begin
284
- block_channel = 'steem_engine:block'
324
+ block_channel = "#{chain_key_prefix}:block"
285
325
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
286
326
  subscription = Redis.new(url: redis_url)
287
327
  ctx = Redis.new(url: redis_url)
@@ -313,16 +353,16 @@ namespace :verify do
313
353
  end
314
354
  end
315
355
 
316
- while ctx.keys("steem_engine:#{next_block_num}:*").size == 0
356
+ while ctx.keys("#{chain_key_prefix}:#{next_block_num}:*").size == 0
317
357
  # This ensures at least the next block has been indexed before
318
358
  # proceeding.
319
359
 
320
- puts "Waiting for block: #{next_block_num} ..."
360
+ puts "Waiting for block (verify:#{chain_key_prefix}_engine_block_org): #{next_block_num} ..."
321
361
  sleep 6
322
362
  end
323
363
 
324
364
  # In theory, we should have all the keys using this pattern.
325
- keys = ctx.keys("steem_engine:#{block_num}:*")
365
+ keys = ctx.keys("#{chain_key_prefix}:#{block_num}:*")
326
366
 
327
367
  # If we have all the keys, we should also have all transaction ids.
328
368
  expected_ids = keys.map { |k| k.split(':')[2] }.uniq
@@ -368,19 +408,40 @@ namespace :verify do
368
408
  agent.shutdown
369
409
  end
370
410
 
411
+ desc 'Verifies Hive Engine sidechain against the mainnet.'
412
+ task :hive_engine_ref_blocks do |t|
413
+ Rake::Task['verify:engine_ref_blocks'].invoke('hive_engine')
414
+ end
415
+
371
416
  desc 'Verifies Steem Engine sidechain against the mainnet.'
372
417
  task :steem_engine_ref_blocks do |t|
418
+ Rake::Task['verify:engine_ref_blocks'].invoke('steem_engine')
419
+ end
420
+
421
+ desc 'Verifies Steem/Hive Engine sidechain against the mainnet.'
422
+ task :engine_ref_blocks, [:chain_key_prefix] do |t, args|
423
+ chain_key_prefix = args[:chain_key_prefix]
373
424
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
374
425
  ctx = ctx = Redis.new(url: redis_url)
375
- keys = ctx.keys('steem_engine:*')
376
- block_api = Steem::BlockApi.new
426
+ keys = ctx.keys("#{chain_key_prefix}:*:*")
427
+ mainchain, mainchain_url = case chain_key_prefix
428
+ when 'steem_engine' then ['steem', 'https://api.steemit.com']
429
+ when 'hive_engine' then ['hive', 'https://api.openhive.network']
430
+ end
431
+ block_api = Meeseeker.block_api_class(mainchain).new(url: mainchain_url)
377
432
  block_trxs = {}
378
433
 
379
- puts "Checking Steem Engine keys: #{keys.size}"
434
+ puts "Checking #{chain_key_prefix} keys: #{keys.size}"
380
435
 
381
436
  keys.each do |key|
382
437
  transaction = JSON[ctx.get(key)]
383
- block_num = transaction['refSteemBlockNumber']
438
+
439
+ next if transaction.class == Integer
440
+
441
+ block_num = case chain_key_prefix
442
+ when 'steem_engine' then transaction.fetch('refSteemBlockNumber')
443
+ when 'hive_engine' then transaction.fetch('refHiveBlockNumber')
444
+ end.to_i
384
445
 
385
446
  block_trxs[block_num] ||= []
386
447
  block_trxs[block_num] << transaction['transactionId'].to_s.split('-').first
@@ -390,6 +451,8 @@ namespace :verify do
390
451
 
391
452
  skipped_blocks = []
392
453
 
454
+ next if block_trxs.empty?
455
+
393
456
  block_api.get_blocks(block_range: block_trxs.keys) do |block, block_num|
394
457
  if block.nil? || block[:transaction_ids].nil?
395
458
  print 'S'
@@ -399,24 +462,34 @@ namespace :verify do
399
462
  else
400
463
  print '.'
401
464
  end
402
-
403
- if (block.transaction_ids & block_trxs[block_num]).none?
465
+
466
+ trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
467
+
468
+ if trx_ids.any? && (block.transaction_ids & trx_ids).none?
404
469
  puts "\nNo intersection in #{block_num}!"
405
- puts "Expected the following sidechain trx_ids: #{block_trxs[block_num].join(', ')}"
470
+ puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
406
471
  end
407
472
  end
408
473
 
409
474
  puts "\nBlocks to retry: #{skipped_blocks.size}"
410
475
 
411
476
  skipped_blocks.each do |block_num|
477
+ block_found = false
478
+
412
479
  block_api.get_block(block_num: block_num) do |result|
480
+ break unless !!result.block
481
+
413
482
  block = result.block
483
+ block_found = true
484
+ trx_ids = block_trxs[block_num] - [Meeseeker::VIRTUAL_TRX_ID]
414
485
 
415
- if (block.transaction_ids & block_trxs[block_num]).none?
486
+ if trx_ids.any? && (block.transaction_ids & trx_ids).none?
416
487
  puts "No intersection in #{block_num}!"
417
- puts "Expected the following sidechain trx_ids: #{block_trxs[block_num].join(', ')}"
488
+ puts "Expected the following sidechain trx_ids: #{trx_ids.join(', ')}"
418
489
  end
419
490
  end
491
+
492
+ redo unless block_found
420
493
  end
421
494
 
422
495
  puts "Done."
@@ -424,10 +497,12 @@ namespace :verify do
424
497
 
425
498
  namespace :witness do
426
499
  desc 'Verifies witnessses in the schedule produced a block.'
427
- task :schedule, [:max_blocks] do |t, args|
500
+ task :schedule, [:chain, :max_blocks] do |t, args|
501
+ chain = args[:chain] if !!args[:chain]
502
+ chain ||= Meeseeker.default_chain_key_prefix
428
503
  max_blocks = args[:max_blocks]
429
- node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
430
- database_api = Steem::DatabaseApi.new(url: node_url)
504
+ node_url = Meeseeker.shuffle_node_url(chain)
505
+ database_api = Meeseeker.database_api_class(chain).new(url: node_url)
431
506
  mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
432
507
  until_block_num = if !!max_blocks
433
508
  database_api.get_dynamic_global_properties do |dgpo|
@@ -446,7 +521,7 @@ namespace :verify do
446
521
 
447
522
  loop do
448
523
  begin
449
- job.perform(mode: mode, until_block_num: until_block_num)
524
+ job.perform(chain: chain, mode: mode, until_block_num: until_block_num)
450
525
  rescue => e
451
526
  puts e.inspect
452
527
  sleep 5
@@ -459,8 +534,10 @@ namespace :verify do
459
534
  end
460
535
 
461
536
  begin
462
- block_api = Steem::BlockApi.new(url: node_url)
463
- schedule_channel = 'steem:witness:schedule'
537
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
538
+ chain_key_prefix = chain.to_s if args[:chain]
539
+ chain_key_prefix ||= Meeseeker.default_chain_key_prefix
540
+ schedule_channel = "#{chain_key_prefix}:witness:schedule"
464
541
  redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
465
542
  subscription = Redis.new(url: redis_url)
466
543
  timeout = (max_blocks).to_i * 3
@@ -519,7 +596,10 @@ namespace :verify do
519
596
  unless !!header
520
597
  # Can happen when there's excess p2p latency and/or jussi
521
598
  # cache is under load.
522
- puts "Waiting for block header: #{block_num}"
599
+ puts "Waiting for block header (witness:schedule): #{block_num}"
600
+
601
+ node_url = Meeseeker.shuffle_node_url(chain)
602
+ block_api = Meeseeker.block_api_class(chain).new(url: node_url)
523
603
 
524
604
  next
525
605
  end