turnstile-rb 2.0.1 → 3.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 64573d2b1f394251482822e130a7d9ca777554d3
-  data.tar.gz: ef96805bb0b4dc31541894c950b102e389a6cc05
+SHA256:
+  metadata.gz: 0da02f1388bebec96709fa3da4b237960a3bc83a9ef8320921836933b9e1e540
+  data.tar.gz: 31193b223254facb6ee7863dfe21811da38dc0874cccfc7d624d1b1aea38d968
 SHA512:
-  metadata.gz: 3a66e638ee0a591f9ea4acc1c72f9b20e59a35f682bb147e0bf688f57b7d4e2ee36b0e97e540168218351fc875f22f253a1bad00c4fcc414b689d4e238be6c09
-  data.tar.gz: 139bf132ff6ab3f18f31f9f4196484889974a1eb7137556f1963e044c1cbffb3c61cfc731d5b4b204b7a4ae8ad51a77a2067279155feab5974f37355b57aac44
+  metadata.gz: d0ea0957e804292a8cab5c788afefb7a8eb5299448fcc71894383dc3d6e8c03ae01af1b4f0cde1010a57e2b04a0a8c7930d2c4c4a311f89af01e389598be005f
+  data.tar.gz: 6ec94cb380a8860f2b952adc95d87abbd716deea8823c2e4d7fa2299f82975de90ceb6eb86ce3e130df3b5d468b532a90867d7a73b57caefc0a173c35c3a4a7e

data/.gitignore

@@ -20,3 +20,6 @@ coverate
 Gemfile.lock
 .ruby-version
 .rake_tasks*
+*.log
+**/log
+

data/.travis.yml

@@ -1,7 +1,8 @@
 rvm:
-- 2.2.7
+- 2.2.8
 - 2.3.4
 - 2.4.1
+- 2.5.0
 services:
 - redis-server
 env:
@@ -19,4 +20,3 @@ script:
   - bundle exec rspec
 after_script:
   - ./cc-test-reporter after-build --exit-code $TRAVIS_TEST_RESULT
-

data/LICENSE.txt

@@ -1,4 +1,5 @@
-Copyright (c) 2012 Wanelo, Inc
+Copyright © 2012 Wanelo, Inc
+Copyright © 2018 Konstantin Gredeskoul
 
 MIT License
 

data/README.md

@@ -5,30 +5,175 @@
 
 # Turnstile
 
-The goal of this gem is to provide near real time tracking and reporting on the number of users currently online and accessing given application.  It requires that the reporting layer is able to uniquely identify each user and provide a unique identifier.  It may also optionally assign another dimension to the users accessing, such as, for example, _platform_ -- which in our case denotes how the user is accessing our application: from desktop browser, iOS app, Android app, mobile web, etc.  But any other partitioning schemee can be used, or none at all.
+The goal of this gem is to provide near real time tracking and reporting on the number of users currently online and accessing given application.  It requires that the reporting layer is able to uniquely identify each user. You may also add one another dimension to the tracking, such as i.e. a _platform_ — a coded device or device type the user is using. 
 
-The gem uses (and depends on) a [Redis](http://redis.io/) instance in order to keep track of _unique_ users, and can operate in **online* mode (tracking users from a Rack Middleware) or **offline**, by taling a file log.
+For example, you might support platforms: `ios`, `android`, `macos`, `windows`, etc.
+
+The gem uses [Redis](http://redis.io/) in order to keep track of the data, and can operate in either the **online* mode** (tracking users from a Rack Middleware) or **offline mode**, by taling an application file log file and searching for a particular pattern.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
     $ gem install turnstile-rb
+    $ turnstile --help
 
 ## Usage
 
-### Tracking 
+The gem provides command line interface shown below:
+
+ 
+### Description
+
+Turnstile is a Redis-based library that can accurately track total number
+of concurrent users accessing a web/API based server application. It can
+break it down by "platform" or a device type, and returns data in JSON,
+CSV of NAD formats. While user tracking may happen synchronously using a
+Rack middleware, another method is provided that is based on log file
+analysis, and can therefore be performed outside web server process.
+
+### Usage
+
+```
+   # Tail the log file as a proper daemon, and optionally HTTP end point
+   turnstile -f <file> [ --daemon ] [ --web PORT ] [ options ]
+
+   # Add a single item and exit
+   turnstile -a 'platform:ip:user'   [ options ]
+
+   # Print the summary stats and exit
+   turnstile -s [ json | csv | nad ] [ options ]
+```
+
+### Details
+
+Turnstile can be run as a daemon, in which case it watches a given log
+file. Or, you can run turnstile to print the current aggregated stats in
+several supported formats, such as JSON.
+   
+When Turnstile is used to tail the log files, ideally you should start
+turnstile daemon on each app sever that's generating log file, or be
+content with the effects of sampling.
+
+Note that the IP address is not required to track uniqueness. Only
+platform and UID are used. Also note that custom formatter can be
+specified in a config file to parse arbitrary complex log lines.
+
+   
+For tailing a log files, Turnstile must first match a log line expected to
+contain the tokens, and then extract is using one of the matchers. You can
+specify which matcher to use depending on whether you can add Turnstile's 
+tokens to your log or not. If you can, great! If not, implement your own 
+custom matcher and great again.
+
+The following matchers are available, and can be selected with -F:
+   
+1. Format named `delimited`, which expects the following token in the
+   log file:
+
+       x-turnstile:platform:ip:user-identifier
+
+   Where ':' (the delimiter) can be set via -l option, OR you can use one
+   of the following formats: `json_formatted`, `pipe_formatted`,
+   `comma_formatted`, `colon_formatted` (the default). The match is 
+   performed on a string `x-turnstile`, other log lines are skipped.
+
+2. Format `json_delimited`, which expects to find a single-line JSON
+   Hash, containing keys `platform`, `ip_address`, and `user_id`. The match
+   is performed on any log line containing string '`ip_address`', other
+   lines are skipped.
+ 
+3.  Format `custom` requires passing an additional flag `-c/--config
+   file.rb`, which will be required, and which can define a matcher and
+   assign it to the `Turnstile.config.custom_matcher` config variable.
+   
+Custom matcher is any object that responds to two methods:
+
+ * `matches?(line)` — must return true if the line is to be used in token extraction, and
+ * `tokenlize(line)` — must return a string of the form `platform:ip:user-id` as a result of parsing the `line`.
+
+### Command Line Options
+
+The following flags are available:
+
+```
+  Mode of Operation:
+    -f, --file FILE                  Starts Turnstile in a file-tailing mode
+                                      
+    -s, --show [FORMAT]              Print current stats and exit. Optional 
+                                     format can be "json" (default), "nad",
+                                     "yaml", or "csv"
+                                      
+    -w, --web [PORT]                 Starts a Sinatra app on a given port
+                                     offering /turnstile/<json|yaml> end point.
+                                     Can be used with file tailing mode, or
+                                     standalone. Default port is 9090
+                                      
+    -a, --add TOKEN                  Registers an event from the token, such as 
+                                     "ios:123.4.4.4:32442". Use -l to customize
+                                     the delimiter
+                                      
+    -p, --print-keys                 Prints all Turnstile keys in Redis
+                                      
+        --flushdb                    Wipes Redis database, and exit
+                                      
+  Tailing log file:
+    -d, --daemonize                  Daemonize to watch the logs
+                                      
+    -b, --read-backwards [LINES]     Like tail, read last LINES lines
+                                     instead of tailing from the end
+                                      
+    -F, --format FORMAT              Log file format (see above)
+                                      
+    -l, --delimiter CHAR             Forces "delimited" file type, and 
+                                     uses CHAR as the delimiter
+                                      
+    -c, --config FILE                Ruby config file that can define the
+                                     custom matcher, supporting arbitrary 
+                                     complex logs
+
+  Redis Server:
+    -r, --redis-url URL              Redis server URL
+        --redis-host HOST            Redis server host
+        --redis-port PORT            Redis server port
+        --redis-db DB                Redis server db
+        --hiredis                    Use hiredis high performance library
+
+  Miscellaneous:
+    -i, --idle-sleep SECONDS         When no work was detected, pause the 
+                                     threads for several seconds.
+    -v, --verbose                    Print status to stdout
+    -t, --trace                      Enable trace mode
+    -h, --help                       Show this message
+
+```
+
+To summarize, you can run `turnstile` in order to:
+
+ * start a daemon to tail a log file
+ * optionally start a Sinatra web server on port 9090, that returns aggregated results to `/turnstile/json`
+ * to print results
+ * to print all redis keys
+ * to reset all data
+ * to add new data
+
+### Data Collection and Reporting 
 
 Turnstile contains two primary parts: data collection and reporting.  
 
 Data collection may happen in two way:
 
-1. Synchronously — in real time — i.e. from a web request
-2. Or asynchronously — by "tailing" the logs on your servers
+  1. Synchronously — in real time — i.e. from a web request
+  2. Or asynchronously — by "tailing" the logs on your servers
+
+*Synchronous tracking* is accurate, supports sampling, but introduces a run-time dependency into your application middleware stack that might not be desirable. This is how `Rack::Attack` and other similar middleware operates. Luckily, there is a better way. 
 
-Synchronous tracking is more accurate, supports sampling, but introduces a run-time dependency into your application middleware stack that might not be desirable.
+*Asynchronous tracking* has a slight initial setup overhead and it requires a consistently formatted log file that includes lines with the platform, IP and user ID for ALL users at least some of the time. This method introduces zero run-time overhead, as the data collection happens outside of the web request in a standalone daemon, that simply tails your log files. 
+
+On high performance web applications it is highly recommended to employ the *asynchronous tracking* via the log file, and possibly — a custom matcher.
+
+> NOTE: as of Turnstile version 3, you can define a [custom matcher](#custom-matcher) in a ruby-syntax configuration file. Coupled with `-F custom` you are then able to parse arbitrary complex log files and extract the three tokens needed by Turnstile: the `platform` enum type, `IP addresss`, and `User ID`. 
 
-Asynchronous tracking has a slight initial setup overhead, but has zero run-time overhead, as the data collection happens outside of the web request.
 
 #### Real Time Tracking API
 
@@ -90,8 +235,8 @@ Turnstile supports two primary formats:
       { "user_id"     :17344742,
         "platform"    :"iphone",
         "session_id"  :"4eKMZJ4nggzvkix29zpS", 
-        "ip_address"  :"70.210.128.241",
-        .... }
+        "ip_address"  :"70.210.128.241"
+      }
       ```
  
  2. Plain text format, where lines are space delimited, and the token is one of the fields of your log line, itself delimited using a configurable character.
@@ -113,6 +258,51 @@ You can also pass the token delimiter on the command line, `-D | --delimiter ","
 
 > NOTE: Default format is **`pipe_delimited`**.
 
+<a name="custom-matcher"></a>
+
+### Custom Log Matchers
+
+> This feature is only available in Turnstile version 3.0 or later.
+
+To be able to tail a structured log file in any format, create a ruby config file, and pass it with `-c <file>`.
+
+For example, below we'll define a custom matcher that extracts our token from a CSV log file.
+
+```
+# config/custom_csv_matcher.rb
+
+# This matcher extracts platform, UID and IP from the following CSV string:
+# 2018-05-02 21:51:44.031,25928,3997,th-M4wDQM4w0,web,j5v-dzg0J,69.181.72.240,e2b1be795372c385c92a7df420752992
+
+class CSVMatcher
+  def tokenize(line)
+    words = line.split(',')
+    platform, ip, uid = [ words[4], web[6], web[7] ]
+    [platform, ip, uid].join(':')
+  end
+
+  def matches?(line)
+    line =~ /x-rqst/
+  end
+end
+
+Turnstile.config.custom_matcher = CSVMatcher.new
+```
+
+The above matcher defines a very simple block that receives a string as input, and returns a string which is a combined value from:
+
+```ruby
+"#{platform}:#{ip}:#{user_id}"
+``` 
+
+For example, `ios:1.4.54.25:fg7988798779` is a valid token. Note that IP addresses are not necessary for the purposes of tracking, and can be omitted.
+
+With the above file defined, we would start turnstile's collector process as follows, tailing the CSV log file:
+
+```bash
+turnstile -f log/production_log.csv -F custom -c config/custom_csv_matcher.rb
+```
+
 ### Examples
 
 For example:
@@ -137,8 +327,7 @@ For example:
 ^Ctrl-C
 ```
 
-Note that ideally you should run ```turnstile``` on all app servers, for completeness, and because
-this does not incur any additional cost for the application (as user tracking is happening outside web request).
+Note that ideally you should run ```turnstile``` on all app servers, for completeness, and because this does not incur any additional cost for the application (as user tracking is happening outside web request).
 
 ### Reporting
 

data/Rakefile

@@ -8,6 +8,7 @@ end
 
 task :permissions do
   shell('rm -rf pkg/')
+  shell('rm -f Gemfile.lock')
   shell("chmod -v o+r,g+r * */* */*/* */*/*/* */*/*/*/* */*/*/*/*/*")
   shell("find . -type d -exec chmod o+x,g+x {} \\;")
 end

data/bin/turnstile

@@ -1,11 +1,9 @@
 #!/usr/bin/env ruby
-ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
 
-require 'rubygems'
-require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])
-require 'turnstile'
+lib_path = File.expand_path(File.dirname(__FILE__) + '/../lib')
+$LOAD_PATH << lib_path if File.exist?(lib_path) && !$LOAD_PATH.include?(lib_path)
 
-require 'turnstile/runner'
+require 'turnstile'
 
-Turnstile::Runner.new(ARGV).execute!
+Turnstile::CLI::Runner.new(ARGV).execute!
 

data/example/custom_csv_matcher.rb

@@ -0,0 +1,22 @@
+# For a log file of the following format:
+#
+# I│2018-05-02 07:47:58.770 │ 25887 ⤶  3997 │ th-M4wDQDbBB  ❯❯❯ time:     73.8ms  ✔  │ x-rqst │ src=android │ uid=0xFFFFD978998789 │ ip=47.23.6.197  │ sess=BGHSsdCsX5VvsN1jLsAR             │ REQ GET    │ 200 │ /api/v3/dashboard
+# I│2018-05-02 21:51:43.925 │ 29122 ⤶  3997 │ th-7lxrgeJZA  ❯❯❯ time:     80.6ms  ✔  │ x-rqst │ src=android │ uid=0xF9909808f90809 │ ip=47.21.6.197  │ sess=BGHSsdCsX5VvsN1jLsAR             │ REQ GET    │ 200 │ /api/v4/stuff
+#
+module Application
+  class CSVLogFormat
+    def tokenize(line)
+      platform = line.scan(/src=(\w*)/).flatten.first
+      ip       = line.scan(/ip=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/).flatten.first
+      uid      = line.scan(/sess=(\w*)/).flatten.first
+
+      [platform, ip, uid].join(':')
+    end
+
+    def matches?(line)
+      line =~ /x-rqst/
+    end
+  end
+end
+
+Turnstile.config.custom_matcher = Application::CSVLogFormat.new

data/lib/turnstile.rb

@@ -1,19 +1,48 @@
 require 'turnstile/version'
 require 'turnstile/configuration'
+require 'turnstile/commands'
+require 'turnstile/logger'
+require 'turnstile/dependencies'
 require 'turnstile/sampler'
-require 'turnstile/adapter'
 require 'turnstile/tracker'
 require 'turnstile/observer'
-require 'turnstile/logger'
+require 'turnstile/redis/adapter'
 require 'turnstile/collector'
-require 'turnstile/summary'
+
+require 'turnstile/cli/runner'
 
 module Turnstile
-  def self.configure(&block)
-    @configuration = Turnstile::Configuration.new.configure(&block)
-  end
+  class CommandNotFoundError < StandardError; end
+  class ConfigFileError < StandardError; end
+  class HiredisDriverNotFound < StandardError; end
+
+  class << self
+    attr_accessor :debug
+
+    def debug?
+      self.debug
+    end
+
+    def configure(&block)
+      @configuration = create_config.configure(&block)
+    end
 
-  def self.config
-    @configuration ||= Turnstile::Configuration.new
+    def config
+      @configuration ||= create_config
+    end
+
+    private
+
+    def create_config
+      ::Turnstile::Configuration.new
+    end
   end
 end
+
+Kernel.send(:define_method, :tdb) do |msg, io = STDOUT|
+   io.puts ''.green + ' debug '.black.on.green+ ''.green + ' —— ' + msg
+end
+
+Kernel.send(:define_method, :terr) do |msg, io = STDERR|
+   io.puts ''.bold.red + ' error '.bold.white.on.red + ''.red + ' —— ' + msg
+end

data/lib/turnstile/cli/launcher.rb

@@ -0,0 +1,83 @@
+require 'turnstile/dependencies'
+
+module Turnstile
+  module CLI
+    class Launcher
+      include Dependencies
+
+      attr_reader :stdin, :stdout, :stderr, :sinatra_thread
+      attr_accessor :options
+
+      def initialize(options, stdin = STDIN, stdout = STDOUT, stderr = STDERR)
+        self.options            = options
+        @stdin, @stdout, @stderr= stdin, stdout, stderr
+      end
+
+      def launch
+        launch_sinatra_app if options[:web]
+        launch_signal_handler
+
+        tdb "config: #{config.to_h}" if Turnstile.config.trace
+        result = if options[:show]
+                   command(:show).execute(options[:show_format] || :json, options[:delimiter])
+
+                 elsif options[:token]
+                   tracker.track_token(options[:token], options[:delimiter])
+
+                 elsif options[:flushdb]
+                   command(:flushdb).execute
+
+                 elsif options[:print_keys]
+                   command(:print_keys).execute
+
+                 elsif options[:file]
+                   controller.start
+                 end
+
+
+        puts result if result && !result.empty?
+      rescue SystemExit, SignalException
+        exit 6
+      rescue Exception => e
+        handle_error('Error', e)
+      ensure
+        sinatra_thread.join if sinatra_thread
+      end
+
+      def launch_signal_handler
+        Signal.trap('INT') { sleep 1; Kernel.exit(5) }
+      end
+
+      def launch_sinatra_app
+        @sinatra_thread = Thread.new do
+          require_relative '../web_app'
+          Kernel.exit(0)
+        end
+      end
+
+      def command(name)
+        ::Turnstile::Commands.command(name).new(options)
+      end
+
+      def handle_error(title, e)
+        if options[:trace]
+          trace = e.backtrace.reverse
+          last  = trace.pop
+          stderr.puts trace.join("\n")
+          stderr.puts last.bold.red
+        end
+        stderr.puts
+        stderr.puts title.bold.yellow
+        stderr.puts "\t" + e.message.red
+        stderr.puts
+      end
+
+      private
+
+      def controller
+        @controller ||= Turnstile::Collector::Controller.new(options)
+      end
+
+    end
+  end
+end