meeseeker 0.0.2 → 0.0.3pre1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +107 -0
- data/Rakefile +132 -0
- data/bin/meeseeker +1 -1
- data/lib/meeseeker/block_follower_job.rb +88 -42
- data/lib/meeseeker/version.rb +1 -1
- data/lib/meeseeker.rb +5 -1
- data/meeseeker.gemspec +6 -1
- data/test/meeseeker/meeseeker_test.rb +27 -0
- data/test/test_helper.rb +21 -0
- metadata +109 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: '058271f9eb089b43a706275a96a06f4b87ac8bbf96b2404a2bbd7c239c2882fc'
|
4
|
+
data.tar.gz: f5f73c53bc34c485b46d94e887249ce65dc3114ee0cf60189849a28786656a27
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: dfb048376f1a0e17bed91a44116d2718a99127eeba0d770ae50d535eead02d298b1105401385b76e06124e94a3e777b7ecf8bd188a4a4c7c6bc3cff635774835
|
7
|
+
data.tar.gz: 74ac56d7d083c6f1e2d5da9496079bf670f4bd12d15122044ac295fe6b75b05c50402cc0d969ab902f54dc264ffa574ee6a78f89a60fb6c9f2597c6db484e417
|
data/README.md
CHANGED
@@ -80,16 +80,119 @@ To ignore virtual operations (useful if the node doesn't enable `get_ops_in_bloc
|
|
80
80
|
MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
|
81
81
|
```
|
82
82
|
|
83
|
+
Normally, block headers are added to the `steem:block` channel. This requires one additional API call for each block. If you don't need block headers, you can configure the `steem:block` channel to only publish with the `block_num`:
|
84
|
+
|
85
|
+
```bash
|
86
|
+
MEESEEKER_INCLUDE_BLOCK_HEADER=false meeseeker sync
|
87
|
+
```
|
88
|
+
|
83
89
|
Normally, keys stay on redis for 24 hours. If you want to change this behavior, use `MEESEEKER_EXPIRE_KEYS` and specify the new value in seconds, for example:
|
84
90
|
|
85
91
|
```bash
|
86
92
|
MEESEEKER_EXPIRE_KEYS=10 meeseeker sync
|
87
93
|
```
|
88
94
|
|
95
|
+
If you never want the keys to expire (not recommended), set
|
96
|
+
`MEESEEKER_EXPIRE_KEYS` to -1:
|
97
|
+
|
98
|
+
```bash
|
99
|
+
MEESEEKER_EXPIRE_KEYS=-1 meeseeker sync
|
100
|
+
```
|
101
|
+
|
89
102
|
### Usage
|
90
103
|
|
91
104
|
When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
|
92
105
|
|
106
|
+
#### Using `SUBSCRIBE`
|
107
|
+
|
108
|
+
For `redis-cli`, please see: https://redis.io/topics/pubsub
|
109
|
+
|
110
|
+
Channels available for `meeseeker`:
|
111
|
+
|
112
|
+
* `steem:block`
|
113
|
+
* `steem:transaction`
|
114
|
+
* `steem:op:vote`
|
115
|
+
* `steem:op:comment`
|
116
|
+
* `steem:op:comment_options`
|
117
|
+
* `steem:op:whatever` (replace "whatever" with the op you want)
|
118
|
+
* `steem:op:custom_json:whatever` (if enabled, replace "whatever" with the `custom_json.id` you want)
|
119
|
+
|
120
|
+
As mentioned in the first `whatever` example, for ops, [all operation types](https://developers.steem.io/apidefinitions/broadcast-ops) can be subscribed to as channels, including virtual operations, if enabled.
|
121
|
+
|
122
|
+
In the second `whatever` example, for `custom_json.id`, if you want to subscribe to the `follow` channel, use `steem:op:custom_json:follow`. Or if you want to subscribe to the `sm_team_reveal` channel, use `steem:op:custom_json:follow`. The `custom_json.id` channels are not enabled by default. To enable it, set the `MEESEEKER_PUBLISH_OP_CUSTOM_ID` to `true` (see example below).
|
123
|
+
|
124
|
+
For example, from `redis-cli`, if we wanted to stream block numbers:
|
125
|
+
|
126
|
+
```bash
|
127
|
+
$ redis-cli
|
128
|
+
127.0.0.1:6379> subscribe steem:block
|
129
|
+
Reading messages... (press Ctrl-C to quit)
|
130
|
+
1) "subscribe"
|
131
|
+
2) "steem:block"
|
132
|
+
3) (integer) 1
|
133
|
+
1) "message"
|
134
|
+
2) "steem:block"
|
135
|
+
3) "{\"block_num\":29861068,\"previous\":\"01c7a4cb4424b4dc0cb0cc72fd36b1644f8aeba5\",\"timestamp\":\"2019-01-28T20:55:03\",\"witness\":\"ausbitbank\",\"transaction_merkle_root\":\"a318bb82625bd78af8d8b506ccd4f53116372c8e\",\"extensions\":[]}"
|
136
|
+
1) "message"
|
137
|
+
2) "steem:block"
|
138
|
+
3) "{\"block_num\":29861069,\"previous\":\"01c7a4cc1bed060876cab57476846a91568a9f8a\",\"timestamp\":\"2019-01-28T20:55:06\",\"witness\":\"followbtcnews\",\"transaction_merkle_root\":\"834e05d40b9666e5ef50deb9f368c63070c0105b\",\"extensions\":[]}"
|
139
|
+
1) "message"
|
140
|
+
2) "steem:block"
|
141
|
+
3) "{\"block_num\":29861070,\"previous\":\"01c7a4cd3bbf872895654765faa4409a8e770e91\",\"timestamp\":\"2019-01-28T20:55:09\",\"witness\":\"timcliff\",\"transaction_merkle_root\":\"b2366ce9134d627e00423b28d33cc57f1e6e453f\",\"extensions\":[]}"
|
142
|
+
```
|
143
|
+
|
144
|
+
In addition to general op channels, there's an additional channel for `custom_json.id`. This option must be enabled:
|
145
|
+
|
146
|
+
```bash
|
147
|
+
MEESEEKER_PUBLISH_OP_CUSTOM_ID=true meeseeker sync
|
148
|
+
```
|
149
|
+
|
150
|
+
Which allows subscription to specific `id` patterns:
|
151
|
+
|
152
|
+
```
|
153
|
+
$ redis-cli
|
154
|
+
127.0.0.1:6379> subscribe steem:op:custom_json:sm_team_reveal
|
155
|
+
Reading messages... (press Ctrl-C to quit)
|
156
|
+
1) "subscribe"
|
157
|
+
2) "steem:op:custom_json:sm_team_reveal"
|
158
|
+
3) (integer) 1
|
159
|
+
1) "message"
|
160
|
+
2) "steem:op:custom_json:sm_team_reveal"
|
161
|
+
3) "{\"key\":\"steem:29890790:bcfa68d9be10b3587d81039b85fd0536ddeddffb:0:custom_json\"}"
|
162
|
+
1) "message"
|
163
|
+
2) "steem:op:custom_json:sm_team_reveal"
|
164
|
+
3) "{\"key\":\"steem:29890792:3f3b921ec6706bcd259f5cc6ac922dc59bbe2de5:0:custom_json\"}"
|
165
|
+
1) "message"
|
166
|
+
2) "steem:op:custom_json:sm_team_reveal"
|
167
|
+
3) "{\"key\":\"steem:29890792:4ceca16dd114b1851140086a82a5fb3a6eb6ec42:0:custom_json\"}"
|
168
|
+
1) "message"
|
169
|
+
2) "steem:op:custom_json:sm_team_reveal"
|
170
|
+
3) "{\"key\":\"steem:29890792:00930eff76b3f0af8ed7215e88cf351cc671490b:0:custom_json\"}"
|
171
|
+
1) "message"
|
172
|
+
2) "steem:op:custom_json:sm_team_reveal"
|
173
|
+
3) "{\"key\":\"steem:29890799:01483bd252ccadb05f546051bb20a4ba9afea243:0:custom_json\"}"
|
174
|
+
```
|
175
|
+
|
176
|
+
A `ruby` application can subscribe to a channel as well, using the `redis` gem:
|
177
|
+
|
178
|
+
```ruby
|
179
|
+
require 'redis'
|
180
|
+
|
181
|
+
url = 'redis://127.0.0.1:6379/0'
|
182
|
+
ctx = Redis.new(url: url)
|
183
|
+
|
184
|
+
Redis.new(url: url).subscribe('steem:op:comment') do |on|
|
185
|
+
on.message do |channel, message|
|
186
|
+
payload = JSON[message]
|
187
|
+
comment = JSON[ctx.get(payload['key'])]
|
188
|
+
|
189
|
+
puts comment['value']
|
190
|
+
end
|
191
|
+
end
|
192
|
+
```
|
193
|
+
|
194
|
+
Many other clients are supported: https://redis.io/clients
|
195
|
+
|
93
196
|
#### Using `SCAN`
|
94
197
|
|
95
198
|
From the redis manual:
|
@@ -100,6 +203,10 @@ From the redis manual:
|
|
100
203
|
|
101
204
|
See: https://redis.io/commands/scan
|
102
205
|
|
206
|
+
Keep in mind that `SCAN` requires pagination to get a complete result. Redis implements pagination using a cursor based iterator.
|
207
|
+
|
208
|
+
See: https://redis.io/commands/scan#scan-basic-usage
|
209
|
+
|
103
210
|
Once your sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
|
104
211
|
|
105
212
|
```bash
|
data/Rakefile
CHANGED
@@ -73,3 +73,135 @@ task reset: [:check_schema] do
|
|
73
73
|
puts ' nothing to drop.'
|
74
74
|
end
|
75
75
|
end
|
76
|
+
|
77
|
+
namespace :verify do
|
78
|
+
desc 'Verifies transactions land where they should.'
|
79
|
+
task :block_org, [:max_blocks] do |t, args|
|
80
|
+
defined? Thread.report_on_exception and Thread.report_on_exception = true
|
81
|
+
|
82
|
+
max_blocks = args[:max_blocks]
|
83
|
+
node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
|
84
|
+
database_api = Steem::DatabaseApi.new(url: node_url)
|
85
|
+
until_block_num = nil
|
86
|
+
|
87
|
+
Thread.new do
|
88
|
+
job = Meeseeker::BlockFollowerJob.new
|
89
|
+
mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').to_sym
|
90
|
+
until_block_num = if !!max_blocks
|
91
|
+
database_api.get_dynamic_global_properties do |dgpo|
|
92
|
+
case mode
|
93
|
+
when :head then dgpo.head_block_number
|
94
|
+
when :irreversible then dgpo.last_irreversible_block_num
|
95
|
+
else; abort "Unknown block mode: #{mode}"
|
96
|
+
end
|
97
|
+
end + max_blocks.to_i
|
98
|
+
end
|
99
|
+
|
100
|
+
loop do
|
101
|
+
begin
|
102
|
+
job.perform(mode: mode, until_block_num: until_block_num)
|
103
|
+
rescue => e
|
104
|
+
puts e.inspect
|
105
|
+
sleep 5
|
106
|
+
end
|
107
|
+
|
108
|
+
break # success
|
109
|
+
end
|
110
|
+
|
111
|
+
puts 'Background sync finished ...'
|
112
|
+
end
|
113
|
+
|
114
|
+
begin
|
115
|
+
block_api = Steem::BlockApi.new(url: node_url)
|
116
|
+
block_channel = 'steem:block'
|
117
|
+
redis_url = ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0')
|
118
|
+
subscription = Redis.new(url: redis_url)
|
119
|
+
ctx = Redis.new(url: redis_url)
|
120
|
+
timeout = (max_blocks).to_i * 3
|
121
|
+
|
122
|
+
subscribe_mode, subscribe_args = if timeout > 0
|
123
|
+
[:subscribe_with_timeout, [timeout, [block_channel]]]
|
124
|
+
else
|
125
|
+
[:subscribe, [[block_channel]]]
|
126
|
+
end
|
127
|
+
|
128
|
+
subscription.send(subscribe_mode, *subscribe_args) do |on|
|
129
|
+
on.subscribe do |channel, subscriptions|
|
130
|
+
puts "Subscribed to ##{channel} (subscriptions: #{subscriptions})"
|
131
|
+
end
|
132
|
+
|
133
|
+
on.message do |channel, message|
|
134
|
+
payload = JSON[message]
|
135
|
+
block_num = payload['block_num']
|
136
|
+
expected_witness = payload['witness']
|
137
|
+
next_block_num = block_num + 1
|
138
|
+
|
139
|
+
if !!max_blocks
|
140
|
+
if block_num >= until_block_num
|
141
|
+
subscription.unsubscribe
|
142
|
+
next
|
143
|
+
end
|
144
|
+
end
|
145
|
+
|
146
|
+
while ctx.keys("steem:#{next_block_num}:*").size == 0
|
147
|
+
# This ensures at least the next block has been indexed before
|
148
|
+
# proceeding.
|
149
|
+
|
150
|
+
puts "Waiting for block: #{next_block_num} ..."
|
151
|
+
sleep 6
|
152
|
+
end
|
153
|
+
|
154
|
+
database_api.get_dynamic_global_properties do |dgpo|
|
155
|
+
(block_num - dgpo.last_irreversible_block_num).tap do |offset|
|
156
|
+
# This will block all channel callbacks until the first known block
|
157
|
+
# is irreversible. After that, the offsets should mostly go
|
158
|
+
# negative.
|
159
|
+
|
160
|
+
sleep offset * 3 if offset > 0
|
161
|
+
end
|
162
|
+
end
|
163
|
+
|
164
|
+
# In theory, we should have all the keys using this pattern.
|
165
|
+
keys = ctx.keys("steem:#{block_num}:*")
|
166
|
+
|
167
|
+
# If we have all the keys, we should also have all transaction ids.
|
168
|
+
expected_ids = keys.map { |k| k.split(':')[2] }.uniq
|
169
|
+
expected_ids -= [Meeseeker::VIRTUAL_TRX_ID]
|
170
|
+
|
171
|
+
actual_ids, actual_witness = block_api.get_block(block_num: block_num) do |result|
|
172
|
+
block = result.block
|
173
|
+
[block.transaction_ids, block.witness]
|
174
|
+
end
|
175
|
+
|
176
|
+
# We do an intersection to make sure there's no difference between
|
177
|
+
# the two copies, regardless of order, as opposed to just checking that
|
178
|
+
# the lengths match.
|
179
|
+
|
180
|
+
(actual_ids & expected_ids).tap do |intersection|
|
181
|
+
all_sizes = [intersection.size, expected_ids.size, actual_ids.size]
|
182
|
+
puts 'intersection: %d; expected: %d; actual: %d' % all_sizes
|
183
|
+
|
184
|
+
if all_sizes.min != all_sizes.max
|
185
|
+
puts "Expected witness: #{expected_witness}; actual witness: #{actual_witness}"
|
186
|
+
puts "Expected transaction ids:"
|
187
|
+
puts expected_ids
|
188
|
+
puts "Actual transaction ids:"
|
189
|
+
puts actual_ids
|
190
|
+
|
191
|
+
puts "actual_ids minus expected:"
|
192
|
+
puts actual_ids - expected_ids
|
193
|
+
puts "expected_ids minus actual:"
|
194
|
+
puts expected_ids - actual_ids
|
195
|
+
|
196
|
+
exit(-1)
|
197
|
+
end
|
198
|
+
end
|
199
|
+
end
|
200
|
+
|
201
|
+
on.unsubscribe do |channel, subscriptions|
|
202
|
+
puts "Unsubscribed from ##{channel} (subscriptions: #{subscriptions})"
|
203
|
+
end
|
204
|
+
end
|
205
|
+
end
|
206
|
+
end
|
207
|
+
end
|
data/bin/meeseeker
CHANGED
@@ -1,23 +1,98 @@
|
|
1
1
|
module Meeseeker
|
2
2
|
class BlockFollowerJob
|
3
3
|
def perform(options = {})
|
4
|
-
|
5
|
-
database_api = Steem::DatabaseApi.new(url: Meeseeker.node_url)
|
4
|
+
block_api = Steem::BlockApi.new(url: Meeseeker.node_url)
|
6
5
|
redis = Meeseeker.redis
|
6
|
+
last_key_prefix = nil
|
7
|
+
trx_index = 0
|
8
|
+
current_block_num = nil
|
9
|
+
block_transactions = []
|
10
|
+
|
11
|
+
stream_operations(options) do |op, trx_id, block_num|
|
12
|
+
begin
|
13
|
+
current_key_prefix = "steem:#{block_num}:#{trx_id}"
|
14
|
+
|
15
|
+
if current_key_prefix == last_key_prefix
|
16
|
+
trx_index += 1
|
17
|
+
else
|
18
|
+
if !!last_key_prefix
|
19
|
+
n, b, t = last_key_prefix.split(':')
|
20
|
+
transaction_payload = {
|
21
|
+
block_num: b.to_i,
|
22
|
+
transaction_id: t,
|
23
|
+
transaction_num: block_transactions.size
|
24
|
+
}
|
25
|
+
|
26
|
+
block_transactions << trx_id unless trx_id == VIRTUAL_TRX_ID
|
27
|
+
redis.publish('steem:transaction', transaction_payload.to_json)
|
28
|
+
end
|
29
|
+
last_key_prefix = "steem:#{block_num}:#{trx_id}"
|
30
|
+
trx_index = 0
|
31
|
+
end
|
32
|
+
|
33
|
+
op_type = op.type.split('_')[0..-2].join('_')
|
34
|
+
key = "#{current_key_prefix}:#{trx_index}:#{op_type}"
|
35
|
+
puts key
|
36
|
+
end
|
37
|
+
|
38
|
+
redis.set(key, op.to_json)
|
39
|
+
redis.expire(key, Meeseeker.expire_keys) unless Meeseeker.expire_keys == -1
|
40
|
+
|
41
|
+
if current_block_num != block_num
|
42
|
+
block_transactions = []
|
43
|
+
block_payload = {
|
44
|
+
block_num: block_num
|
45
|
+
}
|
46
|
+
|
47
|
+
if Meeseeker.include_block_header
|
48
|
+
block_api.get_block_header(block_num: block_num) do |result|
|
49
|
+
block_payload = block_payload.merge(result.header.to_h)
|
50
|
+
end
|
51
|
+
end
|
52
|
+
|
53
|
+
redis.set(LAST_BLOCK_NUM_KEY, block_num)
|
54
|
+
redis.publish('steem:block', block_payload.to_json)
|
55
|
+
current_block_num = block_num
|
56
|
+
end
|
57
|
+
|
58
|
+
redis.publish("steem:op:#{op_type}", {key: key}.to_json)
|
59
|
+
|
60
|
+
if Meeseeker.publish_op_custom_id
|
61
|
+
if %w(custom custom_binary custom_json).include? op_type
|
62
|
+
id = (op["value"]["id"] rescue nil).to_s
|
63
|
+
|
64
|
+
if id.size > 0
|
65
|
+
redis.publish("steem:op:#{op_type}:#{id}", {key: key}.to_json)
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
71
|
+
private
|
72
|
+
def stream_operations(options = {}, &block)
|
73
|
+
redis = Meeseeker.redis
|
74
|
+
last_block_num = nil
|
75
|
+
mode = options.delete(:mode) || Meeseeker.stream_mode
|
76
|
+
options[:include_virtual] ||= Meeseeker.include_virtual
|
7
77
|
|
8
78
|
if !!options[:at_block_num]
|
9
79
|
last_block_num = options[:at_block_num].to_i
|
10
80
|
else
|
81
|
+
database_api = Steem::DatabaseApi.new(url: Meeseeker.node_url)
|
11
82
|
last_block_num = redis.get(LAST_BLOCK_NUM_KEY).to_i + 1
|
12
83
|
|
13
84
|
database_api.get_dynamic_global_properties do |dgpo|
|
14
|
-
block_num = case
|
85
|
+
block_num = case mode
|
15
86
|
when :head then dgpo.head_block_number
|
16
87
|
when :irreversible then dgpo.last_irreversible_block_num
|
17
|
-
else; abort "Unknown stream mode: #{
|
88
|
+
else; abort "Unknown stream mode: #{mode}"
|
18
89
|
end
|
19
90
|
|
20
|
-
if
|
91
|
+
if Meeseeker.expire_keys == -1
|
92
|
+
last_block_num = [last_block_num, block_num].max
|
93
|
+
|
94
|
+
puts "Sync from: #{last_block_num}"
|
95
|
+
elsif block_num - last_block_num > Meeseeker.expire_keys / 3
|
21
96
|
last_block_num = block_num
|
22
97
|
|
23
98
|
puts 'Starting new sync.'
|
@@ -30,46 +105,17 @@ module Meeseeker
|
|
30
105
|
end
|
31
106
|
end
|
32
107
|
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
}
|
37
|
-
|
38
|
-
if !!last_block_num
|
39
|
-
puts options.to_json
|
40
|
-
end
|
41
|
-
|
42
|
-
last_key_prefix = nil
|
43
|
-
trx_index = 0
|
44
|
-
current_block_num = nil
|
45
|
-
|
46
|
-
stream.operations(options) do |op, trx_id, block_num|
|
47
|
-
current_key_prefix = "steem:#{block_num}:#{trx_id}"
|
108
|
+
begin
|
109
|
+
stream_options = {url: Meeseeker.node_url, mode: mode}
|
110
|
+
options = options.merge(at_block_num: last_block_num)
|
48
111
|
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
redis.publish('steem:transaction', {block_num: b.to_i, trx_id: t}.to_json)
|
112
|
+
Steem::Stream.new(stream_options).tap do |stream|
|
113
|
+
puts "Stream begin: #{stream_options.to_json}; #{options.to_json}"
|
114
|
+
|
115
|
+
stream.operations(options) do |op, trx_id, block_num|
|
116
|
+
yield op, trx_id, block_num
|
55
117
|
end
|
56
|
-
last_key_prefix = "steem:#{block_num}:#{trx_id}"
|
57
|
-
trx_index = 0
|
58
118
|
end
|
59
|
-
|
60
|
-
op_type = op.type.split('_')[0..-2].join('_')
|
61
|
-
key = "#{current_key_prefix}:#{trx_index}:#{op_type}"
|
62
|
-
puts key
|
63
|
-
redis.set(key, op.to_json)
|
64
|
-
redis.expire(key, Meeseeker.expire_keys)
|
65
|
-
|
66
|
-
if current_block_num != block_num
|
67
|
-
redis.set(LAST_BLOCK_NUM_KEY, block_num)
|
68
|
-
redis.publish('steem:block', {block_num: block_num}.to_json)
|
69
|
-
current_block_num = block_num
|
70
|
-
end
|
71
|
-
|
72
|
-
redis.publish("steem:op:#{op_type}", {key: key}.to_json)
|
73
119
|
end
|
74
120
|
end
|
75
121
|
end
|
data/lib/meeseeker/version.rb
CHANGED
data/lib/meeseeker.rb
CHANGED
@@ -7,13 +7,17 @@ require 'meeseeker/block_follower_job'
|
|
7
7
|
module Meeseeker
|
8
8
|
LAST_BLOCK_NUM_KEY = 'steem:meeseeker:last_block_num'
|
9
9
|
BLOCKS_PER_DAY = 28800
|
10
|
+
VIRTUAL_TRX_ID = '0000000000000000000000000000000000000000'
|
10
11
|
@redis = Redis.new(url: ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0'))
|
11
12
|
@node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
|
12
13
|
@stream_mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').downcase.to_sym
|
13
14
|
@include_virtual = ENV.fetch('MEESEEKER_INCLUDE_VIRTUAL', 'true').downcase == 'true'
|
15
|
+
@include_block_header = ENV.fetch('MEESEEKER_INCLUDE_BLOCK_HEADER', 'true').downcase == 'true'
|
16
|
+
@publish_op_custom_id = ENV.fetch('MEESEEKER_PUBLISH_OP_CUSTOM_ID', 'false').downcase == 'true'
|
14
17
|
@expire_keys = ENV.fetch('MEESEEKER_EXPIRE_KEYS', BLOCKS_PER_DAY * 3).to_i
|
15
18
|
|
16
19
|
extend self
|
17
20
|
|
18
|
-
attr_accessor :redis, :node_url, :expire_keys, :stream_mode, :include_virtual
|
21
|
+
attr_accessor :redis, :node_url, :expire_keys, :stream_mode, :include_virtual,
|
22
|
+
:include_block_header, :publish_op_custom_id
|
19
23
|
end
|
data/meeseeker.gemspec
CHANGED
@@ -11,7 +11,7 @@ Gem::Specification.new do |s|
|
|
11
11
|
s.description = 'If you have multiple applications that need to perform actions as operations occur, `meeseeker` will allow your apps to each perform actions for specific operations without each app having to streaming the entire blockchain.'
|
12
12
|
s.authors = ['Anthony Martin']
|
13
13
|
s.email = ['meeseeker@martin-studio.com,']
|
14
|
-
s.files = Dir['bin/**/*', 'lib/**/*', 'Gemfile', 'LICENSE', 'Rakefile', 'README.md', 'meeseeker.gemspec']
|
14
|
+
s.files = Dir['bin/**/*', 'lib/**/*', 'test/**/*', 'Gemfile', 'LICENSE', 'Rakefile', 'README.md', 'meeseeker.gemspec']
|
15
15
|
s.test_files = Dir['test/**/*']
|
16
16
|
s.executables = Dir['bin/*'].map{ |f| File.basename(f) }
|
17
17
|
s.homepage = 'https://rubygems.org/gems/meeseeker'
|
@@ -21,6 +21,11 @@ Gem::Specification.new do |s|
|
|
21
21
|
|
22
22
|
# Ruby Make (interprets the Rakefile DSL).
|
23
23
|
s.add_development_dependency 'rake', '~> 12.3', '>= 12.3.1'
|
24
|
+
s.add_development_dependency 'minitest', '~> 5.10', '>= 5.10.3'
|
25
|
+
s.add_development_dependency 'minitest-line', '~> 0.6', '>= 0.6.4'
|
26
|
+
s.add_development_dependency 'minitest-proveit', '~> 1.0', '>= 1.0.0'
|
27
|
+
s.add_development_dependency 'simplecov', '~> 0.15', '>= 0.15.1'
|
28
|
+
s.add_development_dependency 'pry', '~> 0.11', '>= 0.11.3'
|
24
29
|
|
25
30
|
s.add_dependency 'redis', '~> 4.1', '>= 4.1.0'
|
26
31
|
s.add_dependency 'steem-mechanize', '~> 0.0', '>= 0.0.5'
|
@@ -0,0 +1,27 @@
|
|
1
|
+
require 'test_helper'
|
2
|
+
require 'rake'
|
3
|
+
|
4
|
+
module Meeseeker
|
5
|
+
class MeeseekerTest < Meeseeker::Test
|
6
|
+
def setup
|
7
|
+
gem_dir = File.expand_path("..", File.dirname(__FILE__))
|
8
|
+
$LOAD_PATH.unshift gem_dir
|
9
|
+
|
10
|
+
pwd = Dir.pwd
|
11
|
+
Dir.chdir(gem_dir)
|
12
|
+
Rake.application.init
|
13
|
+
Rake.application.load_rakefile
|
14
|
+
Dir.chdir(pwd)
|
15
|
+
end
|
16
|
+
|
17
|
+
def test_verify_block_org
|
18
|
+
max_blocks = 30 # must be at least 15 to get past irreversible
|
19
|
+
if !!Meeseeker.redis.get(Meeseeker::LAST_BLOCK_NUM_KEY)
|
20
|
+
fail "Found existing keys. Please use 'rake reset' to enable this test."
|
21
|
+
end
|
22
|
+
|
23
|
+
assert Rake::Task['verify:block_org'].invoke(max_blocks)
|
24
|
+
assert Rake::Task['reset'].invoke
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
data/test/test_helper.rb
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
|
2
|
+
|
3
|
+
require 'simplecov'
|
4
|
+
|
5
|
+
SimpleCov.start
|
6
|
+
SimpleCov.merge_timeout 3600
|
7
|
+
|
8
|
+
require 'meeseeker'
|
9
|
+
require 'minitest/autorun'
|
10
|
+
require 'minitest/line/describe_track'
|
11
|
+
require 'minitest/hell'
|
12
|
+
require 'minitest/proveit'
|
13
|
+
require 'pry'
|
14
|
+
|
15
|
+
class Minitest::Test
|
16
|
+
parallelize_me!
|
17
|
+
end
|
18
|
+
|
19
|
+
class Meeseeker::Test < MiniTest::Test
|
20
|
+
defined? prove_it! and prove_it!
|
21
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: meeseeker
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.3pre1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Anthony Martin
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2019-01
|
11
|
+
date: 2019-02-01 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rake
|
@@ -30,6 +30,106 @@ dependencies:
|
|
30
30
|
- - ">="
|
31
31
|
- !ruby/object:Gem::Version
|
32
32
|
version: 12.3.1
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
name: minitest
|
35
|
+
requirement: !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - "~>"
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
version: '5.10'
|
40
|
+
- - ">="
|
41
|
+
- !ruby/object:Gem::Version
|
42
|
+
version: 5.10.3
|
43
|
+
type: :development
|
44
|
+
prerelease: false
|
45
|
+
version_requirements: !ruby/object:Gem::Requirement
|
46
|
+
requirements:
|
47
|
+
- - "~>"
|
48
|
+
- !ruby/object:Gem::Version
|
49
|
+
version: '5.10'
|
50
|
+
- - ">="
|
51
|
+
- !ruby/object:Gem::Version
|
52
|
+
version: 5.10.3
|
53
|
+
- !ruby/object:Gem::Dependency
|
54
|
+
name: minitest-line
|
55
|
+
requirement: !ruby/object:Gem::Requirement
|
56
|
+
requirements:
|
57
|
+
- - "~>"
|
58
|
+
- !ruby/object:Gem::Version
|
59
|
+
version: '0.6'
|
60
|
+
- - ">="
|
61
|
+
- !ruby/object:Gem::Version
|
62
|
+
version: 0.6.4
|
63
|
+
type: :development
|
64
|
+
prerelease: false
|
65
|
+
version_requirements: !ruby/object:Gem::Requirement
|
66
|
+
requirements:
|
67
|
+
- - "~>"
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: '0.6'
|
70
|
+
- - ">="
|
71
|
+
- !ruby/object:Gem::Version
|
72
|
+
version: 0.6.4
|
73
|
+
- !ruby/object:Gem::Dependency
|
74
|
+
name: minitest-proveit
|
75
|
+
requirement: !ruby/object:Gem::Requirement
|
76
|
+
requirements:
|
77
|
+
- - "~>"
|
78
|
+
- !ruby/object:Gem::Version
|
79
|
+
version: '1.0'
|
80
|
+
- - ">="
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: 1.0.0
|
83
|
+
type: :development
|
84
|
+
prerelease: false
|
85
|
+
version_requirements: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - "~>"
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '1.0'
|
90
|
+
- - ">="
|
91
|
+
- !ruby/object:Gem::Version
|
92
|
+
version: 1.0.0
|
93
|
+
- !ruby/object:Gem::Dependency
|
94
|
+
name: simplecov
|
95
|
+
requirement: !ruby/object:Gem::Requirement
|
96
|
+
requirements:
|
97
|
+
- - "~>"
|
98
|
+
- !ruby/object:Gem::Version
|
99
|
+
version: '0.15'
|
100
|
+
- - ">="
|
101
|
+
- !ruby/object:Gem::Version
|
102
|
+
version: 0.15.1
|
103
|
+
type: :development
|
104
|
+
prerelease: false
|
105
|
+
version_requirements: !ruby/object:Gem::Requirement
|
106
|
+
requirements:
|
107
|
+
- - "~>"
|
108
|
+
- !ruby/object:Gem::Version
|
109
|
+
version: '0.15'
|
110
|
+
- - ">="
|
111
|
+
- !ruby/object:Gem::Version
|
112
|
+
version: 0.15.1
|
113
|
+
- !ruby/object:Gem::Dependency
|
114
|
+
name: pry
|
115
|
+
requirement: !ruby/object:Gem::Requirement
|
116
|
+
requirements:
|
117
|
+
- - "~>"
|
118
|
+
- !ruby/object:Gem::Version
|
119
|
+
version: '0.11'
|
120
|
+
- - ">="
|
121
|
+
- !ruby/object:Gem::Version
|
122
|
+
version: 0.11.3
|
123
|
+
type: :development
|
124
|
+
prerelease: false
|
125
|
+
version_requirements: !ruby/object:Gem::Requirement
|
126
|
+
requirements:
|
127
|
+
- - "~>"
|
128
|
+
- !ruby/object:Gem::Version
|
129
|
+
version: '0.11'
|
130
|
+
- - ">="
|
131
|
+
- !ruby/object:Gem::Version
|
132
|
+
version: 0.11.3
|
33
133
|
- !ruby/object:Gem::Dependency
|
34
134
|
name: redis
|
35
135
|
requirement: !ruby/object:Gem::Requirement
|
@@ -109,6 +209,8 @@ files:
|
|
109
209
|
- lib/meeseeker/block_follower_job.rb
|
110
210
|
- lib/meeseeker/version.rb
|
111
211
|
- meeseeker.gemspec
|
212
|
+
- test/meeseeker/meeseeker_test.rb
|
213
|
+
- test/test_helper.rb
|
112
214
|
homepage: https://rubygems.org/gems/meeseeker
|
113
215
|
licenses:
|
114
216
|
- CC0-1.0
|
@@ -125,9 +227,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
125
227
|
version: '0'
|
126
228
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
127
229
|
requirements:
|
128
|
-
- - "
|
230
|
+
- - ">"
|
129
231
|
- !ruby/object:Gem::Version
|
130
|
-
version:
|
232
|
+
version: 1.3.1
|
131
233
|
requirements: []
|
132
234
|
rubyforge_project:
|
133
235
|
rubygems_version: 2.7.7
|
@@ -135,4 +237,6 @@ signing_key:
|
|
135
237
|
specification_version: 4
|
136
238
|
summary: Redis based block follower is an efficient way for multiple apps to stream
|
137
239
|
the Steem Blockchain.
|
138
|
-
test_files:
|
240
|
+
test_files:
|
241
|
+
- test/test_helper.rb
|
242
|
+
- test/meeseeker/meeseeker_test.rb
|