meeseeker 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c77b50a00bf28514314ae08d98371ebdca3215549843665d82d111080ed466c3
+  data.tar.gz: 0ae8f242244db5535800836cede789ab459e3b6091e610cb1b0fb1bf7fab9019
+SHA512:
+  metadata.gz: c1a9b6e6280f27661b7a2447b34caf8fe2757b8bd87a52c7dc7112aab6f0ce2fac8307cdb631e728bdf503ac9f13d57748fc8d5f481acb104a8c54abaf5ec6d3
+  data.tar.gz: 37c5f658c328e6436f977305b9ecb7b2ea7d3343bf6a873aeb6baf6edcd3860d3fac1fa729b5178f6af18affe8557d38e9a7bd06a889d4bb6fb0aea88cf71a89

data/.gitignore

@@ -0,0 +1,50 @@
+*.gem
+*.rbc
+/.config
+/coverage/
+/InstalledFiles
+/pkg/
+/spec/reports/
+/spec/examples.txt
+/test/tmp/
+/test/version_tmp/
+/tmp/
+
+# Used by dotenv library to load environment variables.
+# .env
+
+## Specific to RubyMotion:
+.dat*
+.repl_history
+build/
+*.bridgesupport
+build-iPhoneOS/
+build-iPhoneSimulator/
+
+## Specific to RubyMotion (use of CocoaPods):
+#
+# We recommend against adding the Pods directory to your .gitignore. However
+# you should judge for yourself, the pros and cons are mentioned at:
+# https://guides.cocoapods.org/using/using-cocoapods.html#should-i-check-the-pods-directory-into-source-control
+#
+# vendor/Pods/
+
+## Documentation cache and generated files:
+/.yardoc/
+/_yardoc/
+/doc/
+/rdoc/
+
+## Environment normalization:
+/.bundle/
+/vendor/bundle
+/lib/bundler/man/
+
+# for a library or gem, you might want to ignore these files since the code is
+# intended to run in multiple environments; otherwise, check them in:
+Gemfile.lock
+# .ruby-version
+# .ruby-gemset
+
+# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
+.rvmrc

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in steem_api.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,41 @@
+CC0 1.0 Universal (CC0 1.0)
+Public Domain Dedication
+https://creativecommons.org/publicdomain/zero/1.0/
+
+This is a human-readable summary of the Legal Code:
+https://creativecommons.org/publicdomain/zero/1.0/legalcode
+
+Disclaimer
+
+The Commons Deed is not a legal instrument. It is simply a handy reference for
+understanding the CC0 Legal Code, a human-readable expression of some of its key
+terms. Think of it as the user-friendly interface to the CC0 Legal Code beneath.
+This Deed itself has no legal value, and its contents do not appear in CC0.
+Creative Commons is not a law firm and does not provide legal services.
+Distributing, displaying, or linking to this Commons Deed does not create an
+attorney-client relationship.
+
+Creative Commons has not verified the copyright status of any work to which CC0
+has been applied. CC makes no warranties about any work or its copyright status
+in any jurisdiction, and disclaims all liability for all uses of any work.
+
+No Copyright
+
+The person who associated a work with this deed has dedicated the work to the
+public domain by waiving all of his or her rights to the work worldwide under
+copyright law, including all related and neighboring rights, to the extent
+allowed by law.
+
+You can copy, modify, distribute and perform the work, even for commercial
+purposes, all without asking permission. See Other Information below.
+
+Other Information
+
+* In no way are the patent or trademark rights of any person affected by CC0,
+  nor are the rights that other persons may have in the work or in how the work
+  is used, such as publicity or privacy rights.
+* Unless expressly stated otherwise, the person who associated a work with this
+  deed makes no warranties about the work, and disclaims liability for all uses
+  of the work, to the fullest extent permitted by applicable law.
+* When using or citing the work, you should not imply endorsement by the author
+  or the affirmer.

data/README.md

@@ -0,0 +1,178 @@
+ # meeseeker
+
+Redis based block follower is an efficient way for multiple apps to stream the Steem Blockchain.
+
+If you have multiple applications that need to perform actions as operations occur, `meeseeker` will allow your apps to each perform actions for specific operations without each app having to streaming the entire blockchain.
+
+*In a nutshell:* The overarching intent here is to provide a "live view" of the blockchain, *not* store the entire blockchain.  Apps can attach to your redis source and ask, "What *just* happened?"
+
+## Purpose
+
+Although Meeseeker tracks all operations, it is only intended to provide other applications signals that those operations have happened.  It is not intended to provide cryptographically verifiable events.
+
+Possible uses:
+
+* Notifications of events, suitable for push to mobile devices or web browsers.
+* Invoke periodic updates on a threshold.
+* Light-weight bots that only care about a limit set of operations, reducing the number of API calls.
+
+## Why Redis?
+
+Redis is a persistent key-value database, with built-in net interface.  See: https://redis.io/
+
+It allows for quick storage and lookup of operations by key as well as the ability to automatically expire keys that are no longer needed.
+
+### Installation
+
+First, install redis:
+
+On linux:
+
+```bash
+sudo apt install redis-server
+```
+
+On macOS:
+
+```bash
+brew install redis
+```
+
+Next, install ruby.  One way to do this is install [rvm](https://rvm.io/install).  Once ruby is installed, install `meeseeker` with the `gem` command:
+
+```bash
+gem install meeseeker
+```
+
+This installs meeseeker as a command available to the OS, e.g.:
+
+```bash
+meeseeker help
+```
+
+To do the actual sync to your local redis source (defaults assume `redis://127.0.0.1:6379/0`):
+
+```bash
+meeseeker sync
+```
+
+To specify an alternative redis source:
+
+```bash
+MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync
+```
+
+You can also specify an alternative Steem node:
+
+```bash
+MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync
+```
+
+To sync from the head block instead of the last irreversible block:
+
+```bash
+MEESEEKER_STREAM_MODE=head meeseeker sync
+```
+
+To ignore virtual operations (useful if the node doesn't enable `get_ops_in_blocks` or if you want to sync from the head block):
+
+```bash
+MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync
+```
+
+Normally, keys stay on redis for 24 hours.  If you want to change this behavior, use `MEESEEKER_EXPIRE_KEYS` and specify the new value in seconds, for example:
+
+```bash
+MEESEEKER_EXPIRE_KEYS=10 meeseeker sync
+```
+
+### Usage
+
+When `meeseeker sync` starts for the first time, it initializes from the last irreversible block number.  If the sync is interrupted, it will resume from the last block sync'd unless that block is older than `MEESEEKER_EXPIRE_KEYS` in which case it will skip to the last irreversible block number.
+
+#### Using `SCAN`
+
+From the redis manual:
+
+> Since these commands allow for incremental iteration, returning only a small number of elements per call, they can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements.
+> 
+> However while blocking commands like SMEMBERS are able to provide all the elements that are part of a Set in a given moment, The SCAN family of commands only offer limited guarantees about the returned elements since the collection that we incrementally iterate can change during the iteration process.
+
+See: https://redis.io/commands/scan
+
+Once you're sync has started, you can begin doing queries against redis, for example, in the `redis-cli`:
+
+```bash
+redis-cli --scan --pattern 'steem:*:vote'
+```
+
+This returns the keys, for example:
+
+```
+steem:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
+steem:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
+steem:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
+steem:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
+steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
+steem:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
+steem:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
+steem:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
+steem:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote
+```
+
+To get the actual vote operation for a particular key, use:
+
+```bash
+redis-cli get steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
+```
+
+If, on the other hand, you want `custom_json` only:
+
+```bash
+redis-cli --scan --pattern 'steem:*:custom_json'
+```
+
+This only returns the related keys, for example:
+
+```
+steem:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
+steem:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
+steem:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
+steem:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
+steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
+steem:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json
+```
+
+To get the actual custom json operation for a particular key, use:
+
+```bash
+redis-cli get steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
+```
+
+To get all transactions for a particular block number:
+
+```bash
+redis-cli --scan --pattern 'steem:29811085:*'
+```
+
+Or to get all ops for a particular transaction:
+
+```bash
+redis-cli --scan --pattern 'steem:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'
+```
+
+---
+
+<center>
+  <img src="https://i.imgur.com/Y3Sa2GW.jpg" />
+</center>
+
+See some of my previous Ruby How To posts in: [#radiator](https://steemit.com/created/radiator) [#ruby](https://steemit.com/created/ruby)
+
+## Get in touch!
+
+If you're using Radiator, I'd love to hear from you.  Drop me a line and tell me what you think!  I'm @inertia on STEEM.
+  
+## License
+
+I don't believe in intellectual "property".  If you do, consider Radiator as licensed under a Creative Commons [![CC0](http://i.creativecommons.org/p/zero/1.0/80x15.png)](http://creativecommons.org/publicdomain/zero/1.0/) License.

data/Rakefile

@@ -0,0 +1,75 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+require 'meeseeker'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'test'
+  t.libs << 'lib'
+  t.test_files = FileList['test/**/*_test.rb']
+  t.ruby_opts << if ENV['HELL_ENABLED']
+    '-W2'
+  else
+    '-W1'
+  end
+end
+
+task :default => :test
+
+task :console do
+  exec "irb -r meeseeker -I ./lib"
+end
+
+desc 'Build a new version of the meeseeker gem.'
+task :build do
+  exec 'gem build meeseeker.gemspec'
+end
+
+desc 'Publish the current version of the meeseeker gem.'
+task :push do
+  exec "gem push meeseeker-#{Meeseeker::VERSION}.gem"
+end
+
+task :check_schema do
+  begin
+    abort 'Unable to ping redis source.' unless Meeseeker.redis.ping == 'PONG'
+  rescue Redis::CommandError => e
+    puts e
+  rescue Redis::CannotConnectError => e
+    puts e
+  end
+end
+
+task(:sync, [:at_block_num] => [:check_schema]) do |t, args|
+  job = Meeseeker::BlockFollowerJob.new
+  job.perform(at_block_num: args[:at_block_num])
+end
+
+task(:find, [:what, :key] => [:check_schema]) do |t, args|
+  redis = Meeseeker.redis
+  match = case args[:what].downcase.to_sym
+  when :block then "steem:#{args[:key]}:*"
+  when :trx then "steem:*:#{args[:key]}:*"
+  else; abort "Unknown lookup using #{args}"
+  end
+
+  puts "Looking for match on: #{match}"
+  keys = redis.keys(match)
+  
+  keys.each do |key|
+    puts key
+    puts redis.get(key)
+  end
+end
+
+task reset: [:check_schema] do
+  print 'Dropping keys ...'
+  keys = Meeseeker.redis.keys('steem:*')
+  
+  if keys.any?
+    print " found #{keys.size} keys ..."
+    dropped = Meeseeker.redis.del(*keys)
+    puts " dropped #{dropped} keys."
+  else
+    puts ' nothing to drop.'
+  end
+end

data/bin/meeseeker

@@ -0,0 +1,48 @@
+#!/usr/bin/env ruby
+
+require 'rake'
+require 'pp'
+require 'meeseeker'
+
+gem_dir = File.expand_path("..", File.dirname(__FILE__))
+$LOAD_PATH.unshift gem_dir
+
+pwd = Dir.pwd
+Dir.chdir(gem_dir)
+Rake.application.init
+Rake.application.load_rakefile
+Dir.chdir(pwd)
+
+puts "meeseeker-#{Meeseeker::VERSION}"
+filename = __FILE__.split('/').last
+
+case ARGV[0]
+when 'console' then Rake::Task['console'].invoke
+when 'sync'
+  backoff = 0.01
+  max_backoff = 30
+  
+  loop do; begin
+    Rake::Task['sync'].invoke(ARGV[1])
+  rescue Steem::BaseError => e
+    puts "Error: #{e.inspect}"
+    backoff = [backoff, max_backoff].min
+    sleep backoff *= 2
+    puts "Retrying ..."
+    Rake::Task['sync'].reenable
+  end; end
+when 'find' then Rake::Task['find'].invoke(*ARGV[1..-1])
+when 'reset' then Rake::Task['reset'].invoke
+else
+  puts "\nBegin/resume sync:"
+  puts "\t#{filename} sync\n\n"
+  puts "Start in the ruby console:"
+  puts "\t#{filename} console\n\n"
+  puts 'Find block or transaction:'
+  puts "\t#{filename} find block 3044538"
+  puts "\t#{filename} find trx 983c5e5c6aef52f1647d952a18771f76b885e6de\n\n"
+  puts 'Clear all keys:'
+  puts "\t#{filename} reset\n\n"
+  puts "Note, using find and reset is not intended for routine operations.  It is intended for analysis only."
+  puts "See: https://github.com/inertia186/meeseeker#usage"
+end

data/lib/meeseeker/block_follower_job.rb

@@ -0,0 +1,64 @@
+module Meeseeker
+  class BlockFollowerJob
+    def perform(options = {})
+      stream = Steem::Stream.new(url: Meeseeker.node_url, mode: Meeseeker.stream_mode)
+      database_api = Steem::DatabaseApi.new(url: Meeseeker.node_url)
+      redis = Meeseeker.redis
+      
+      if !!options[:at_block_num]
+        last_block_num = options[:at_block_num].to_i
+      else
+        last_block_num = redis.get(LAST_BLOCK_NUM_KEY).to_i + 1
+        
+        database_api.get_dynamic_global_properties do |dgpo|
+          block_num = case Meeseeker.stream_mode
+          when :head then dgpo.last_irreversible_block_num
+          when :irreversible then dgpo.last_irreversible_block_num
+          else; abort "Unknown stream mode: #{Meeseeker.stream_mode}"
+          end
+          
+          if block_num - last_block_num > Meeseeker.expire_keys / 3
+            last_block_num = block_num
+            
+            puts 'Starting new sync.'
+          else
+            behind_sec = block_num - last_block_num
+            behind_sec *= 3.0
+            
+            puts "Resuming from #{behind_sec / 60} minutes ago ..."
+          end
+        end
+      end
+      
+      options = {
+        at_block_num: last_block_num,
+        include_virtual: Meeseeker.include_virtual
+      }
+      
+      if !!last_block_num
+        puts options.to_json
+      end
+      
+      last_key_prefix = nil
+      trx_index = 0
+      
+      stream.operations(options) do |op, trx_id, block_num|
+        current_key_prefix = "steem:#{block_num}:#{trx_id}"
+        
+        if current_key_prefix == last_key_prefix
+          trx_index += 1
+        else
+          last_key_prefix = "steem:#{block_num}:#{trx_id}"
+          trx_index = 0
+        end
+        
+        op_type = op.type.split('_')[0..-2].join('_')
+        key = "#{current_key_prefix}:#{trx_index}:#{op_type}"
+        puts key
+        redis.set(key, op.to_json)
+        redis.expire(key, Meeseeker.expire_keys)
+        redis.set(LAST_BLOCK_NUM_KEY, block_num)
+      end
+    end
+  end
+end

data/lib/meeseeker/version.rb

@@ -0,0 +1,3 @@
+module Meeseeker
+  VERSION = '0.0.1'
+end

data/lib/meeseeker.rb

@@ -0,0 +1,19 @@
+require 'redis'
+require 'steem'
+
+require 'meeseeker/version'
+require 'meeseeker/block_follower_job'
+
+module Meeseeker
+  LAST_BLOCK_NUM_KEY = 'steem:meeseeker:last_block_num'
+  BLOCKS_PER_DAY = 28800
+  @redis = Redis.new(url: ENV.fetch('MEESEEKER_REDIS_URL', 'redis://127.0.0.1:6379/0'))
+  @node_url = ENV.fetch('MEESEEKER_NODE_URL', 'https://api.steemit.com')
+  @stream_mode = ENV.fetch('MEESEEKER_STREAM_MODE', 'head').downcase.to_sym
+  @include_virtual = ENV.fetch('MEESEEKER_INCLUDE_VIRTUAL', 'true').downcase == 'true'
+  @expire_keys = ENV.fetch('MEESEEKER_EXPIRE_KEYS', BLOCKS_PER_DAY * 3).to_i
+  
+  extend self
+
+  attr_accessor :redis, :node_url, :expire_keys, :stream_mode, :include_virtual
+end

data/meeseeker.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'meeseeker/version'
+
+Gem::Specification.new do |s|
+  s.name = 'meeseeker'
+  s.version = Meeseeker::VERSION
+  s.licenses = 'CC0-1.0'
+  s.summary = 'Redis based block follower is an efficient way for multiple apps to stream the Steem Blockchain.'
+  s.description = 'If you have multiple applications that need to perform actions as operations occur, `meeseeker` will allow your apps to each perform actions for specific operations without each app having to streaming the entire blockchain.'
+  s.authors = ['Anthony Martin']
+  s.email = ['meeseeker@martin-studio.com,']
+  s.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test)/}) }
+  s.homepage = 'https://rubygems.org/gems/meeseeker'
+  s.metadata = { 'source_code_uri' => 'https://github.com/inertia186/meeseeker' }
+  s.bindir = 'bin'
+  s.executables = 'meeseeker'
+
+  # Ruby Make (interprets the Rakefile DSL).
+  s.add_development_dependency 'rake', '~> 12.3', '>= 12.3.1'
+
+  s.add_dependency 'redis', '~> 4.1', '>= 4.1.0'
+  s.add_dependency 'steem-mechanize', '~> 0.0', '>= 0.0.5'
+  s.add_dependency 'rb-readline', '~> 0.5', '>= 0.5.5'
+end

metadata

@@ -0,0 +1,139 @@
+--- !ruby/object:Gem::Specification
+name: meeseeker
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Anthony Martin
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2019-01-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '12.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 12.3.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '12.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 12.3.1
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.1'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.1.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.1'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.1.0
+- !ruby/object:Gem::Dependency
+  name: steem-mechanize
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.5
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.5
+- !ruby/object:Gem::Dependency
+  name: rb-readline
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.5
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.5
+description: If you have multiple applications that need to perform actions as operations
+  occur, `meeseeker` will allow your apps to each perform actions for specific operations
+  without each app having to streaming the entire blockchain.
+email:
+- meeseeker@martin-studio.com,
+executables:
+- meeseeker
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- bin/meeseeker
+- lib/meeseeker.rb
+- lib/meeseeker/block_follower_job.rb
+- lib/meeseeker/version.rb
+- meeseeker.gemspec
+homepage: https://rubygems.org/gems/meeseeker
+licenses:
+- CC0-1.0
+metadata:
+  source_code_uri: https://github.com/inertia186/meeseeker
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.7
+signing_key: 
+specification_version: 4
+summary: Redis based block follower is an efficient way for multiple apps to stream
+  the Steem Blockchain.
+test_files: []