keen-cli 0.1.5 → 0.1.6

data/README.md

@@ -18,24 +18,25 @@ Verify the `keen` command is in your path by running it:
 
 ``` shell
 Commands:
-  keen average              # Alias for queries:run -c average
-  keen count                # Alias for queries:run -c count
-  keen count-unique         # Alias for queries:run -c count_unique
-  keen events:add           # Add one or more events and print the result
-  keen extraction           # Alias for queries:run -c extraction
-  keen help [COMMAND]       # Describe available commands or one specific command
-  keen maximum              # Alias for queries:run -c maximum
-  keen median               # Alias for queries:run -c median
-  keen minimum              # Alias for queries:run -c minimum
-  keen percentile           # Alias for queries:run -c percentile
-  keen project:collections  # Print information about a project's collections
-  keen project:describe     # Print information about a project
-  keen project:open         # Open a project's overview page in a browser
-  keen project:workbench    # Open a project's workbench page in a browser
-  keen queries:run          # Run a query and print the result
-  keen select-unique        # Alias for queries:run -c select_unique
-  keen sum                  # Alias for queries:run -c sum
-  keen version              # Print the keen-cli version
+  keen average               # Alias for queries:run -a average
+  keen count                 # Alias for queries:run -a count
+  keen count-unique          # Alias for queries:run -a count_unique
+  keen events:add            # Add one or more events and print the result
+  keen extraction            # Alias for queries:run -a extraction
+  keen help [COMMAND]        # Describe available commands or one specific command
+  keen maximum               # Alias for queries:run -a maximum
+  keen median                # Alias for queries:run -a median
+  keen minimum               # Alias for queries:run -a minimum
+  keen percentile            # Alias for queries:run -a percentile
+  keen projects:collections  # Print information about a project's collections
+  keen projects:describe     # Print information about a project
+  keen projects:open         # Open a project's overview page in a browser
+  keen projects:workbench    # Open a project's workbench page in a browser
+  keen queries:run           # Run a query and print the result
+  keen queries:url           # Print the URL for a query
+  keen select-unique         # Alias for queries:run -a select_unique
+  keen sum                   # Alias for queries:run -a sum
+  keen version               # Print the keen-cli version
 ```
 
 You should see information about available commands.
@@ -60,7 +61,7 @@ If you run `keen` from a directory with this .env file, it will assume the proje
 To override the project context use the `--project` option:
 
 ``` shell
-$ keen project:describe --project XXXXXXXXXXXXXXX
+$ keen projects:describe --project XXXXXXXXXXXXXXX
 ```
 
 Similar overrides are available for specifiying API keys: `--master-key`, `--read-key` and `--write-key`.
@@ -68,13 +69,13 @@ Similar overrides are available for specifiying API keys: `--master-key`, `--rea
 For example:
 
 ``` shell
-$ keen project:describe --project XXXXXXXXXXXXXXX --master-key AAAAAAAAAAAAAA
+$ keen projects:describe --project XXXXXXXXXXXXXXX --master-key AAAAAAAAAAAAAA
 ```
 
 Shorter aliases exist as well: `-p` for project, `-k` for master key, `-r` for read key, and `-w` for write key.
 
 ``` shell
-$ keen project:describe -p XXXXXXXXXXXXXXX -k AAAAAAAAAAAAAA
+$ keen projects:describe -p XXXXXXXXXXXXXXX -k AAAAAAAAAAAAAA
 ```
 
 ### Usage
@@ -85,10 +86,10 @@ keen-cli has a variety of commands, and most are namespaced for clarity.
 
 ##### Projects
 
-* `project:open` - Open the Project Overview page in a browser
-* `project:workbench` - Open the Project Workbench page in a browser
-* `project:describe` - Get data about the project. Uses the [project row resource](https://keen.io/docs/api/reference/#project-row-resource).
-* `project:collections` - Get schema information about the project's collections. Uses the [event resource](https://keen.io/docs/api/reference/#event-resource).
+* `projects:open` - Open the Project Overview page in a browser
+* `projects:workbench` - Open the Project Workbench page in a browser
+* `projects:describe` - Get data about the project. Uses the [project row resource](https://keen.io/docs/api/reference/#project-row-resource).
+* `projects:collections` - Get schema information about the project's collections. Uses the [event resource](https://keen.io/docs/api/reference/#event-resource).
 
 ##### Events
 
@@ -97,51 +98,58 @@ keen-cli has a variety of commands, and most are namespaced for clarity.
 Parameters:
 
 + `--collection`, `-c`: The collection to add the event to. Alternately you can set `KEEN_COLLECTION_NAME` on the environment if you're working with the same collection frequently.
-+ `--data`, `-d`: The properties of the event. The value can be JSON or `key=value` pairs delimited by `&` (just like a query string). Data can also be piped in via STDIN.
-+ `--file`, `-f`: The name of a file that contains newline delimited JSON
-+ `--csv`: Specify that the file is in CSV format
++ `--batch-size`: Batch size of events posted to Keen, defaults to 1000.
+
+Input source parameters:
+
++ `--data`, `-d`: Pass an event body on the command line. Make sure to use quotes where necessary.
++ `--file`, `-f`: The name of a file containing events.
+
+You can also pass input via `STDIN`.
+
+If not otherwise specified, the format of data from any source is assumed to be newline-delimited JSON. CSV and query string-like input is also supported. The associated params:
+
++ `--csv`: Specify CSV format. The first line must contain column names. Column names containing `.`, such as `keen.timestamp`, will be converted to nested properties.
++ `--params`: Specify "params" format. Params format looks like `property1=value1&property2=value` etc.
 
 Various examples:
 
 ``` shell
-# create an empty event
+# add an empty event
 $ keen events:add --collection cli-tests
 
 # use the shorter form of collection
 $ keen events:add -c cli-tests
 
-# add a blank event to a collection specified in the .env file:
+# add a blank event to a collection specified in a .env file:
 # KEEN_COLLECTION_NAME=cli-tests
 $ keen events:add
 
-# create an event from JSON
-$ keen events:add -c cli-tests -d "{ \"username\" : \"dzello\", \"zsh\": 1 }"
+# add an event from JSON using the --data parameter
+$ keen events:add -c cli-tests --data "{ \"username\" : \"dzello\", \"zsh\": 1 }"
 
-# create an event from key value pairs
-$ keen events:add -c cli-tests -d "username=dzello&zsh=1"
+# add an event from key value pairs
+$ keen events:add -c cli-tests -data "username=dzello&zsh=1" --params
 
 # pipe in events as JSON
 $ echo "{ \"username\" : \"dzello\", \"zsh\": 1 }" | keen events:add -c cli-tests
 
-# pipe in events in querystring format
-$ echo "username=dzello&zsh=1" | keen events:add -c cli-test
-
-# specify a file that contains newline delimited json
+# add events from a file that contains newline delimited json
+# { "apple" : "sauce" }
+# { "banana" : "pudding" }
+# { "cherry" : "pie" }
 $ keen events:add --file events.json
 
-# specify a file in CSV format
-$ keen events:add --csv --file events.csv
-
-# pipe in events from a file of newline delimited json
-# { "username" : "dzello", "zsh" : 1 }
-# { "username" : "dkador", "zsh" : 1 }
-# { "username" : "gphat", "zsh" : 1 }
-$ cat events.json | keen events:add -c cli-test
+# add events from a file in CSV format; the first row should be columns
+# fruit,dish
+# apple,sauce
+# banana,pudding
+$ keen events:add --file events.csv --csv
 ```
 
 ##### Queries
 
-`queries:run` - Runs a query and prints the result in pretty JSON.
+`queries:run` - Runs a query and prints the result
 
 Parameters:
 
@@ -158,7 +166,11 @@ Parameters:
 + `--property-names`: A comma-separated list of property names. Extractions only.
 + `--latest`: Number of latest events to retrieve. Extractions only.
 + `--email`: Send extraction results via email, asynchronously. Extractions only.
-+ `--data`, `-d`: Specify query parameters as JSON instead of query params. Data can also be piped in via STDIN.
+
+Input source parameters:
++ `--data`, `-d`: Specify query parameters as JSON instead of query params.
+
+You can also pass input via `STDIN`.
 
 Some examples:
 
@@ -215,6 +227,16 @@ $ keen queries:run --collection minecraft-deaths --analysis-type extraction --pr
 $ echo "{ \"event_collection\" : \"minecraft-deaths\", \"target_property\": \"level\" }" | keen queries:run -a average
 ```
 
+**Query URL Generation**
+
+Run `keen` with no arguments to see the full list of aliases.
+
+`queries:url` - Generates the URL of a query, but does not run it.
+
+The same parameters apply as `queries:run`, in addition to one extra.
+
++ `--exclude-api-key`: Prevent the API key query param from being included in the output
+ 
 **Query Aliases**
 
 For each type of analysis (e.g. count, average, extraction, etc.) there is an alias that can be used
@@ -228,10 +250,16 @@ $ keen minimum -c cpu-checks -y iowait
 0.17
 ```
 
-Run `keen` with no arguments to see the full list of aliases.
+### Global parameters
+
+Parameters that apply to most commands include:
+
++ `--pretty`: Prettify API response JSON. Defaults to true, set `--pretty=false` to prevent
++ `--silent`: Silence any output. Defaults to false.
 
 ### Changelog
 
++ 0.1.6 - Big refactoring to make importing events much cleaner and batching happen automatically. Also adds `queries:url`.
 + 0.1.5 – Support adding events from files with `--file`. Optionally add from CSV with `--csv`.
 + 0.1.4 – Support absolute timeframes via `--start` and `--end` flags
 + 0.1.3 – Add querying via JSON. Add query aliases. Add support for extraction fields.

data/keen-cli.gemspec

@@ -12,7 +12,7 @@ Gem::Specification.new do |s|
   s.description = "Record events and run queries from the comfort of your command line"
   s.license     = "MIT"
 
-  s.add_dependency "keen", ">= 0.8.4"
+  s.add_dependency "keen", ">= 0.8.6"
   s.add_dependency "thor"
   s.add_dependency "dotenv"
 

data/lib/keen-cli/batch_processor.rb

@@ -0,0 +1,104 @@
+module KeenCli
+
+  class BatchProcessor
+
+    # defaults
+    DEFAULT_BATCH_SIZE = 1000
+    DEFAULT_CSV_OPTIONS = { :converters => :all }
+
+    # public options, set in constructor
+    attr_accessor :collection
+    attr_accessor :batch_size
+    attr_accessor :params
+    attr_accessor :csv
+    attr_accessor :csv_options
+    attr_accessor :pretty
+    attr_accessor :silent
+
+    # internal state tracking
+    attr_accessor :size
+    attr_accessor :events
+
+    def initialize(collection, options={})
+      self.collection = collection
+      self.batch_size = options[:batch_size] || DEFAULT_BATCH_SIZE
+      self.csv = options[:csv]
+      self.params = options[:params]
+      self.csv_options = DEFAULT_CSV_OPTIONS.merge(options[:csv_options] || {})
+      self.events = []
+      self.pretty = options[:pretty]
+      self.silent = options[:silent]
+      self.reset
+    end
+
+    def add(line)
+
+      # if we're in CSV mode and don't have headers let's try and set them
+      if self.csv && !self.csv_options.has_key?(:headers)
+
+        set_headers(line)
+        return
+
+      end
+
+      if self.csv
+
+        csv_row = CSV.parse_line(line, self.csv_options)
+        raise "Could not parse! #{line}" unless csv_row
+
+        hash = row_to_hash(csv_row)
+
+      elsif self.params
+
+        hash = Utils.parse_data_as_querystring(line)
+
+      else
+
+        hash = JSON.parse(line)
+
+      end
+
+      self.events.push(hash)
+      self.size += 1
+
+      self.flush if self.size >= self.batch_size
+
+    end
+
+    def flush
+      publish_batch(self.collection, self.events) if self.size > 0
+      reset
+    end
+
+    def reset
+      self.size = 0
+      self.events.clear
+    end
+
+    private
+
+    def set_headers(line)
+      csv_row = CSV.parse_line(line, self.csv_options)
+      self.csv_options[:headers] = csv_row.to_a
+    end
+
+    def publish_batch(collection, events)
+      batches = {}
+      batches[collection] = events
+      Keen.publish_batch(batches).tap do |result|
+        Utils.out_json(result, :pretty => self.pretty, :silent => self.silent)
+      end
+    end
+
+    def row_to_hash(csv_row)
+      naive_hash = csv_row.to_hash
+      naive_hash.map do |main_key, main_value|
+        main_key.to_s.split('.').reverse.inject(main_value) do |value, key|
+          {key => value}
+        end
+      end.inject(&:deep_merge)
+    end
+
+  end
+
+end

data/lib/keen-cli/cli.rb

@@ -5,250 +5,25 @@ require 'csv'
 
 require 'keen-cli/utils'
 
-module KeenCli
-
-  class CLI < Thor
-
-    def self.shared_options
-      option :project, :aliases => ['-p']
-      option :"master-key", :aliases => ['-k']
-      option :"read-key", :aliases => ['-r']
-      option :"write-key", :aliases => ['-w']
-    end
+require 'keen-cli/shared'
 
-    def self.data_options
-      option :data, :aliases => ['-d']
-    end
-
-    def self.file_options
-      option :file, :aliases => ['-f']
-      option :csv
-    end
+require 'keen-cli/projects'
+require 'keen-cli/events'
+require 'keen-cli/queries'
 
-    def self.collection_options
-      option :collection, :aliases => ['-c']
-    end
+module KeenCli
 
-    def self.query_options
-      self.collection_options
-      option :"analysis-type", :aliases => ['-a']
-      option :"group-by", :aliases => ['-g']
-      option :"target-property", :aliases => ['-y']
-      option :interval, :aliases => ['-i']
-      option :timeframe, :aliases => ['-t']
-      option :filters, :aliases => ['-f']
-      option :percentile
-      option :"property-names"
-      option :latest
-      option :email
-      option :start, :aliases => ['s']
-      option :end, :aliases => ['e']
-    end
+  class CLI < Thor
 
     desc 'version', 'Print the keen-cli version'
     map %w(-v --version) => :version
 
-    def version
-      "keen-cli version #{KeenCli::VERSION}".tap do |s|
-        puts s
-      end
-    end
-
-    desc 'project:describe', 'Print information about a project'
-    map 'project:describe' => :project_describe
-    shared_options
-
-    def project_describe
-      Utils.process_options!(options)
-      Keen.project_info.tap do |info|
-        puts JSON.pretty_generate(info)
-      end
-    end
-
-    desc 'project:collections', 'Print information about a project\'s collections'
-    map 'project:collections' => :project_collections
-    shared_options
-
-    def project_collections
-      Utils.process_options!(options)
-      Keen.event_collections.tap do |collections|
-        puts JSON.pretty_generate(collections)
-      end
-    end
-
-    desc 'project:open', 'Open a project\'s overview page in a browser'
-    map 'project:open' => :project_open
-    shared_options
-
-    def project_open
-      Utils.process_options!(options)
-      "https://keen.io/project/#{Keen.project_id}".tap do |project_url|
-        `open #{project_url}`
-      end
-    end
-
-    desc 'project:workbench', 'Open a project\'s workbench page in a browser'
-    map 'project:workbench' => :project_workbench
-    shared_options
-
-    def project_workbench
-      Utils.process_options!(options)
-      "https://keen.io/project/#{Keen.project_id}/workbench".tap do |project_url|
-        `open #{project_url}`
-      end
-    end
-
-    desc 'queries:run', 'Run a query and print the result'
-    map 'queries:run' => :queries_run
     shared_options
-    query_options
-    data_options
-    def queries_run(analysis_type=nil)
-
-      Utils.process_options!(options)
-
-      # work with a new set of options
-      q_options = {}
-
-      data = nil
-
-      if $stdin.tty?
-        data = options[:data]
-      else
-        ARGV.clear
-        ARGF.each_line do |line|
-          data = line
-        end
-      end
-
-      # if data is provided, parse it and merge it
-      unless data.nil?
-        data_options = JSON.parse(data)
-        q_options.merge!(data_options)
-      end
-
-      # convert dashes to underscores, and merge all into q_options
-      q_options.merge!(options.inject({}) do |memo, element|
-        if ['analysis-type', 'group-by', 'target-property', 'property-names'].include?(element.first)
-          memo[element.first.sub('-', '_')] = element.last
-        else
-          memo[element.first] = element.last
-        end
-        memo
-      end)
-
-      collection = Utils.get_collection_name(q_options)
-      analysis_type = analysis_type || q_options["analysis_type"]
-
-      # delete fields that shouldn't be passed to keen-gem as options
-      q_options.delete("collection")
-      q_options.delete("event_collection")
-      q_options.delete("data")
-      q_options.delete("analysis_type")
-
-      if property_names = q_options.delete("property_names")
-        q_options[:property_names] = property_names.split(",")
-      end
-
-      if start_time = q_options.delete("start")
-        q_options[:timeframe] = { :start => start_time }
-      end
-
-      if end_time = q_options.delete("end")
-        q_options[:timeframe] = q_options[:timeframe] || {}
-        q_options[:timeframe][:end] = end_time
-      end
-
-      raise "No analysis type given!" unless analysis_type
-      raise "No collection given!" unless collection
-
-      Keen.send(analysis_type, collection, q_options).tap do |result|
-        if result.is_a?(Hash) || result.is_a?(Array)
-          puts JSON.pretty_generate(result)
-        else
-          puts result
-        end
-      end
-    end
-
-    ANALYSIS_TYPES = %w(average count count-unique extraction median minimum maximum sum percentile select-unique)
-
-    ANALYSIS_TYPES.each do |analysis_type|
-      underscored_analysis_type = analysis_type.sub('-', '_')
-      desc analysis_type, "Alias for queries:run -c #{underscored_analysis_type}"
-      map analysis_type => method_name = "queries_run_#{underscored_analysis_type}"
-      shared_options
-      query_options
-      data_options
-      self.send(:define_method, method_name) { queries_run(underscored_analysis_type) }
-    end
-
-    desc 'events:add', 'Add one or more events and print the result'
-    map 'events:add' => :events_add
-
-    shared_options
-    data_options
-    file_options
-    collection_options
-
-    def events_add
-
-      events = []
-      collection = Utils.get_collection_name(options)
-
-      if options[:csv]
-
-        data = File.read(options[:file])
-        csv = CSV.new(data, :headers => true, :header_converters => :symbol, :converters => :all)
-        events = csv.to_a.map {|row| row.to_hash }
-
-      else
-
-        if $stdin.tty?
-          if data = options[:data]
-            events.push(data)
-          elsif file = options[:file]
-            File.readlines(file).each do |line|
-              events.push(line)
-            end
-          else
-            events.push({})
-          end
-        else
-          ARGV.clear
-          ARGF.each_line do |line|
-            events.push(line)
-          end
-        end
-
-        events = events.map do |event|
-          begin
-            JSON.parse(event)
-          rescue
-            begin
-              Utils.parse_data_as_querystring(event)
-            rescue
-              event
-            end
-          end
-        end
-
-      end
-
-      if events.length > 1
-
-        Keen.publish_batch(collection => events).tap do |result|
-          puts JSON.pretty_generate(result)
-        end
-
-      else
-
-        Keen.publish(collection, events.first).tap do |result|
-          puts JSON.pretty_generate(result)
-        end
 
+    def version
+      "keen-cli version #{KeenCli::VERSION}".tap do |s|
+        Utils.out(s, options)
       end
-
     end
 
   end

data/lib/keen-cli/events.rb

@@ -0,0 +1,70 @@
+require 'keen-cli/batch_processor'
+
+module KeenCli
+
+  class CLI < Thor
+
+    def self.events_options
+      option :csv
+      option :params
+      option :'batch-size'
+    end
+
+    desc 'events:add', 'Add one or more events and print the result'
+    map 'events:add' => :events_add
+
+    shared_options
+    data_options
+    file_options
+    collection_options
+    events_options
+
+    def events_add
+
+      Utils.process_options!(options)
+
+      collection = Utils.get_collection_name(options)
+
+      batch_processor = BatchProcessor.new(collection,
+        :csv => options[:csv],
+        :params => options[:params],
+        :pretty => options[:pretty],
+        :silent => options[:silent],
+        :batch_size => options[:'batch-size'])
+
+      total_events = 0
+
+      if data = options[:data]
+        batch_processor.add(data)
+        total_events += 1
+      end
+
+      if file = options[:file]
+        File.readlines(file).each do |line|
+          batch_processor.add(line)
+          total_events += 1
+        end
+      end
+
+      if !$stdin.tty?
+        ARGV.clear
+        ARGF.each_line do |line|
+          batch_processor.add(line)
+          total_events += 1
+        end
+      end
+
+      if $stdin.tty? && data.nil? && file.nil?
+        batch_processor.add("{}")
+        total_events += 1
+      end
+
+      batch_processor.flush
+
+      total_events
+
+    end
+
+  end
+
+end