reqsample 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f40922b479c65c01c72272a9dffe7d596b9d91db
+  data.tar.gz: cabd856beb4100130e670a90f55aa93d2f2b461d
+SHA512:
+  metadata.gz: 8e060ba69839a2abdd2ef117210c11fe6469ec32a6d4b7710f070747208929cf9e31cd1af109b4be563d4cce306b2cb03f69b5dc0300691ae87f6511c9f1868a
+  data.tar.gz: db8c94f6ad928c5926a8463eca20bc31813d6e592e0fa2896c6d8a0658db1f321e55c955f9d8459882022dc2672d2516f1a156d07ead9249ad6625b4646a69ca

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+README.md
+ChangeLog.md
+
+LICENSE.txt

data/.gitignore

@@ -0,0 +1,6 @@
+/.bundle
+/Gemfile.lock
+/html/
+/pkg/
+/vendor/cache/*.gem
+*.gem

data/.rdoc_options

@@ -0,0 +1,16 @@
+--- !ruby/object:RDoc::Options
+encoding: UTF-8
+static_path: []
+rdoc_include:
+  - .
+charset: UTF-8
+exclude: 
+hyperlink_all: false
+line_numbers: false
+main_page: README.md
+markup: markdown
+show_hash: false
+tab_width: 8
+title: reqsample Documentation
+visibility: :protected
+webcvs:

data/.rspec

@@ -0,0 +1 @@
+--colour --format documentation

data/ChangeLog.md

@@ -0,0 +1,12 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
+and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.0.1] (Sep 12, 2017)
+
+- Initial release

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/Guardfile

@@ -0,0 +1,15 @@
+notification ENV['INSIDE_EMACS'].nil? ? :tmux : :emacs,
+             display_message: true
+
+guard :rspec, cmd: 'bundle exec rspec' do
+  require 'guard/rspec/dsl'
+  dsl = Guard::RSpec::Dsl.new(self)
+
+  # RSpec files
+  rspec = dsl.rspec
+  watch(rspec.spec_helper) { rspec.spec_dir }
+  watch(rspec.spec_support) { rspec.spec_dir }
+  watch(rspec.spec_files)
+
+  watch(%r{^lib/(.+)\.rb}) { |m| "spec/lib/#{m[1]}_spec.rb" }
+end

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2017 Tyler Langlois
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,93 @@
+# reqsample
+
+* [Homepage](https://rubygems.org/gems/reqsample)
+* [Documentation](http://rubydoc.info/gems/reqsample/frames)
+
+## Description
+
+`reqsample` is a utility to generate somewhat-realistic public HTTP traffic. If you've ever needed a large corpus of Apache or nginx logs to test geoip processing, a Logstash pipeline, or as the source for a demo; this utility is for you.
+
+Data is sampled from publicly available data (sources noted in the [credits](#credits) section) and, whenever possible, the frequency of various datasets is observed and reflected in the random data. For example, Chrome will appear frequently in the `User-Agent` string since it is a common browser, and the most common source IPs originate from China due to the high amount of traffic observed from the country.
+
+Note that fine-tuning the generation scheme requires munging with the normal distribution curve and a few other tricky parameters, but usable defaults are used out-of-the-box.
+
+### Quickstart
+
+Generate 1,000 combined Apache log-formatted log entries, spanning the last 24 hours which peak 12 hours ago, and print them all to stdout:
+
+```shell
+$ gem install reqsample
+$ reqsample
+```
+
+See `reqsample help` for a list of commands, flags, and options.
+
+## Features
+
+- Weighted sampling for country of origin, user agents, and response codes to simulate real traffic.
+- Usable in standalone command form or as a Ruby library.
+- Ability to generate all traffic at once in bulk or streamed over time.
+- Frequency and count of request events following a statistically normal distribution.
+
+There are several different parameters that can be changed to modify how data is generated. In general:
+
+- A number of logs to be generated over a given period needs to be chosen, which by default is 1,000.
+- These many log events are generated over a normal distribution curve, with a configurable peak, standard deviation, and time cutoff - defaults are chosen with the assumption that you want to generate 1,000 logs over the previous 24 hours.
+  - The peak is 12 hours ago by default.
+  - The standard deviation is set to 4 by default, which translates to 4 hours in the logic of the random generation.
+  - The normal distribution of log data is truncated at 12 hours by default, which means all logs will fall within some timestamp within the past 24 hours.
+
+## Examples
+
+There are two methods to use `reqsample`, either through the installed executable or as a library.
+
+### Command-Line Utility
+
+Stream 5,000 log events to stdout with a tighter standard deviation:
+
+```
+reqsample stream --count 5000 --stdev 1
+```
+
+### Ruby Library
+
+The `ReqSample::Generator` class needs to be instantiated first, which parses and sets up several enumerables from which values will be sampled.
+
+```ruby
+gen = ReqSample::Generator.new
+```
+
+The `produce` method is the central way to generate log values:
+
+```ruby
+gen.produce
+```
+
+Will return an array of logs with the previously mentioned parameters. If a block is given to the `produce` method, the results will instead be streamed to the block by yielding each log event, simulating live incoming traffic.
+
+## Install
+
+```shell
+$ gem install reqsample
+```
+
+## Development
+
+Standard bundler practices are used, setup your environment with `bundle install` and use `bundle exec rake test` to run the still-incomplete test suite.
+
+Note that all of the source data is retrieved with rake tasks and vendored into the final library to avoid continually retrieving and parsing sources. See `rake -T` for what the tasks are and potentially re-run them if needed.
+
+## Credits
+
+- Country IP Addres Ranges
+  - http://www.nirsoft.net/countryip/
+- Country internet connectivity stats
+  - https://en.wikipedia.org/wiki/List_of_countries_by_number_of_Internet_users
+- User-Agents
+  - https://techblog.willshouse.com/2012/01/03/most-common-user-agents/
+
+## Copyright
+
+Copyright (c) 2017 Tyler Langlois
+
+See LICENSE.txt for details.

data/Rakefile

@@ -0,0 +1,104 @@
+# encoding: utf-8
+
+require 'rubygems'
+
+begin
+  require 'bundler/setup'
+rescue LoadError => e
+  abort e.message
+end
+
+require 'json'
+require 'iso_country_codes'
+require 'mechanize'
+require 'open-uri'
+require 'rake'
+
+COUNTRY_CONNECTIVITY = 'https://en.wikipedia.org/wiki/List_of_countries_by_number_of_Internet_users'.freeze
+CONNECTIVITY_XPATH = '//h2[span[contains(text(), "List")]]/following-sibling::table/tr[not(descendant::th)]'.freeze
+USER_AGENTS = 'https://techblog.willshouse.com/2012/01/03/most-common-user-agents/'
+
+require 'rubygems/tasks'
+Gem::Tasks.new
+
+require 'rdoc/task'
+RDoc::Task.new
+task :doc => :rdoc
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new
+
+task :test    => :spec
+task :default => :spec
+
+task :pry do
+  require 'pry'
+  require 'reqsample'
+  subject = ReqSample::Generator.new
+  ARGV.clear
+  binding.pry
+end
+
+desc 'Load in country IP ranges into a unified JSON dump.'
+task :load_country_networks do
+  agent = Mechanize.new
+  agent.get(URI('http://www.nirsoft.net/countryip/')) do |page|
+    page.links_with(:href => /^[a-z]{2}[.]html$/).reduce({}) do |h, country|
+      h[country.href.split('.').first] = country.click
+        .link_with(:href => /[.]csv$/).click.body
+        .strip.split("\n").map(&:strip).map do |ips|
+          ips.split(',')[0..1]
+        end
+      h
+    end.tap do |network_hash|
+      File.open('vendor/country_networks.json', 'w') do |fh|
+        fh.write JSON.dump(network_hash)
+      end
+    end
+  end
+end
+
+desc 'Retrieve list of internet-connected users by Country.'
+task :load_country_connectivity do
+  Nokogiri::HTML(open(COUNTRY_CONNECTIVITY)).tap do |page|
+    page.xpath(CONNECTIVITY_XPATH).map do |row|
+      [
+        IsoCountryCodes.search_by_name(
+          case (c = row.xpath('td')[0].xpath('a').text.strip.downcase)
+          when 'vietnam' then 'viet nam'
+          when 'south korea' then 'korea (republic'
+          when 'czech republic' then 'czech'
+          when 'ivory coast' then 'côte'
+          when 'laos' then 'lao'
+          when /congo/ then 'congo'
+          when /gambia/ then 'gambia'
+          when /bahama/ then 'bahama'
+          when /são/ then 'sao'
+          else c
+          end
+        ).first.alpha2.downcase,
+        row.xpath('td')[1].text.delete(',').to_i
+      ]
+    end.to_h.tap do |statistics|
+      File.open('vendor/country_connectivity.json', 'w') do |fh|
+        fh.write JSON.dump(statistics)
+      end
+    end
+  end
+end
+
+desc 'Retrieve list of common User-Agents.'
+task :load_user_agents do
+  Nokogiri::HTML(open(USER_AGENTS)).tap do |page|
+    page.at_css('.most-common-user-agents').xpath('tbody/tr').map do |row|
+      [
+        row.at_css('.useragent').text.strip,
+        row.at_css('.percent').text.strip.chomp('%').to_f
+      ]
+    end.to_h.tap do |list|
+      File.open('vendor/user_agents.json', 'w') do |fh|
+        fh.write JSON.dump(list)
+      end
+    end
+  end
+end

data/bin/reqsample

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+libdir = File.expand_path('../lib', File.dirname(__FILE__))
+$LOAD_PATH << libdir if File.exist?(File.join(libdir, 'reqsample', 'cli.rb'))
+require 'reqsample/cli'
+ReqSample::CLI.start

data/lib/reqsample.rb

@@ -0,0 +1,2 @@
+require 'reqsample/version'
+require 'reqsample/generator'

data/lib/reqsample/cli.rb

@@ -0,0 +1,45 @@
+require 'chronic'
+require 'thor'
+require 'reqsample'
+
+module ReqSample
+  # Command-line interface to the library
+  class CLI < Thor
+    class_option :count,
+                 default: 1000,
+                 type: :numeric
+    class_option :format,
+                 default: :apache,
+                 desc: 'Output format of generated logs'
+    class_option :stdev,
+                 default: 4,
+                 desc: 'Standard deviation to use for timespan normal distribution',
+                 type: :numeric
+    class_option :truncate,
+                 default: 12,
+                 desc: 'Cutoff (in hours) that logs should remain +/- within',
+                 type: :numeric
+
+    option :peak,
+           default: '12 hours ago',
+           desc: 'Time at which logs should peak (Chronic-style strings)'
+    desc 'generate', 'Generate a sample of webserver logs'
+    def generate
+      opts = options.dup
+      opts[:peak] = Chronic.parse options[:peak]
+      puts ReqSample::Generator.new(options[:stdev]).produce(opts).join("\n")
+    end
+
+    option :peak,
+           default: 'in 12 hours',
+           desc: 'Time at which logs should peak (Chronic-style strings)'
+    desc 'stream', 'Gradually stream generated logs over given time'
+    def stream
+      opts = options.dup
+      opts[:peak] = Chronic.parse options[:peak]
+      ReqSample::Generator.new(options[:stdev]).produce(opts) do |log|
+        puts log
+      end
+    end
+  end
+end

data/lib/reqsample/generator.rb

@@ -0,0 +1,132 @@
+require 'json'
+require 'ipaddr'
+require 'rubystats'
+
+require 'reqsample/hash'
+require 'reqsample/response_codes'
+require 'reqsample/request_paths'
+require 'reqsample/request_verbs'
+require 'reqsample/time'
+
+# Top-level module for ReqSample constants and classes.
+module ReqSample
+  # Main class for creating randomized data.
+  class Generator
+    attr_accessor :agents,
+                  :codes,
+                  :connectivity,
+                  :dist,
+                  :max_bytes,
+                  :networks,
+                  :verbs
+
+    DEFAULT_COUNT = 1000
+    DEFAULT_DOMAIN = 'http://example.com'.freeze
+    DEFAULT_FORMAT = :apache
+    DEFAULT_MAX_BYTES = 512
+
+    # @param peak_sd [Float] standard deviation in the normal distribution
+    def initialize(peak_sd = 4.0)
+      @agents = ReqSample::Hash.weighted(vendor('user_agents.json'))
+      @codes = ReqSample::Hash.weighted(ReqSample::RESPONSE_CODES)
+      # Peak at zero (will be summed with the Time object)
+      @connectivity = ReqSample::Hash.weighted(
+        vendor('country_connectivity.json')
+      )
+      @dist = Rubystats::NormalDistribution.new(0, peak_sd)
+      @max_bytes = DEFAULT_MAX_BYTES
+      @networks = vendor('country_networks.json')
+      @verbs = ReqSample::Hash.weighted(ReqSample::REQUEST_VERBS)
+    end
+
+    # @option opts [Integer] :count how many logs to generate
+    # @option opts [String] :format form to return logs, :apache or :hash
+    # @option opts [Time] :peak normal distribution peak for log timestamps
+    # @option opts [Integer] :truncate hard limit to keep log range within
+    #
+    # @return [Array<String, Hash>] the collection of generated log events
+    def produce(opts = {})
+      opts[:count] ||= DEFAULT_COUNT
+      opts[:format] ||= DEFAULT_FORMAT
+      opts[:peak] ||= Time.now - (12 * 60 * 60)
+      opts[:truncate] ||= 12
+
+      1.upto(opts[:count]).map do |_|
+        sample_time opts[:peak], opts[:truncate]
+      end.sort.map do |time|
+        if block_given?
+          if (delay = time - Time.now) > 0 then sleep delay end
+          yield sample time, opts[:format]
+        else
+          sample time, opts[:format]
+        end
+      end
+    end
+
+    def sample(time = nil, fmt = nil)
+      # Pull a random country, but ensure it's a valid country code for the
+      # list of networks that we have available.
+      country = connectivity.weighted_sample do |ccodes|
+        ccodes.detect do |ccode|
+          networks.keys.include? ccode
+        end
+      end
+
+      sample = {
+        address: sample_address(country),
+        agent: agents.weighted_sample,
+        bytes: rand(max_bytes),
+        code: codes.weighted_sample,
+        domain: DEFAULT_DOMAIN,
+        path: ReqSample::REQUEST_PATHS.sample,
+        time: time || sample_time(opts),
+        verb: verbs.weighted_sample
+      }
+
+      format fmt, country, sample
+    end
+
+    def format(style, country, sample)
+      case style.to_s
+      when 'apache'
+        [
+          "#{sample[:address]} - -",
+          "[#{sample[:time].strftime('%d/%b/%Y:%H:%M:%S %z')}]",
+          %("#{sample[:verb]} #{sample[:path]} HTTP/1.1"),
+          sample[:code],
+          sample[:bytes],
+          %("#{sample[:domain]}"),
+          %("#{sample[:agent]}")
+        ].join ' '
+      else
+        { country => sample }
+      end
+    end
+
+    def sample_address(country = nil)
+      country ||= networks.keys.sample
+
+      head, tail = networks[country].sample
+      IPAddr.new(
+        rand(IPAddr.new(head).to_i..IPAddr.new(tail).to_i),
+        Socket::AF_INET
+      )
+    end
+
+    # Limit the normal distribution to +/- 12 hours (assume we want to stay
+    # within a 24-hour period).
+    def sample_time(peak, truncate)
+      loop do
+        sample = ReqSample::Time.at((peak + (dist.rng * 60 * 60)).to_i)
+        break sample if sample.within peak, truncate
+      end
+    end
+  end
+
+  private
+
+  def vendor(file)
+    v = File.expand_path('../../../vendor', __FILE__)
+    JSON.parse(File.read(File.join(v, file)))
+  end
+end