keen-cli 0.1.5 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -18,24 +18,25 @@ Verify the `keen` command is in your path by running it:
18
18
 
19
19
  ``` shell
20
20
  Commands:
21
- keen average # Alias for queries:run -c average
22
- keen count # Alias for queries:run -c count
23
- keen count-unique # Alias for queries:run -c count_unique
24
- keen events:add # Add one or more events and print the result
25
- keen extraction # Alias for queries:run -c extraction
26
- keen help [COMMAND] # Describe available commands or one specific command
27
- keen maximum # Alias for queries:run -c maximum
28
- keen median # Alias for queries:run -c median
29
- keen minimum # Alias for queries:run -c minimum
30
- keen percentile # Alias for queries:run -c percentile
31
- keen project:collections # Print information about a project's collections
32
- keen project:describe # Print information about a project
33
- keen project:open # Open a project's overview page in a browser
34
- keen project:workbench # Open a project's workbench page in a browser
35
- keen queries:run # Run a query and print the result
36
- keen select-unique # Alias for queries:run -c select_unique
37
- keen sum # Alias for queries:run -c sum
38
- keen version # Print the keen-cli version
21
+ keen average # Alias for queries:run -a average
22
+ keen count # Alias for queries:run -a count
23
+ keen count-unique # Alias for queries:run -a count_unique
24
+ keen events:add # Add one or more events and print the result
25
+ keen extraction # Alias for queries:run -a extraction
26
+ keen help [COMMAND] # Describe available commands or one specific command
27
+ keen maximum # Alias for queries:run -a maximum
28
+ keen median # Alias for queries:run -a median
29
+ keen minimum # Alias for queries:run -a minimum
30
+ keen percentile # Alias for queries:run -a percentile
31
+ keen projects:collections # Print information about a project's collections
32
+ keen projects:describe # Print information about a project
33
+ keen projects:open # Open a project's overview page in a browser
34
+ keen projects:workbench # Open a project's workbench page in a browser
35
+ keen queries:run # Run a query and print the result
36
+ keen queries:url # Print the URL for a query
37
+ keen select-unique # Alias for queries:run -a select_unique
38
+ keen sum # Alias for queries:run -a sum
39
+ keen version # Print the keen-cli version
39
40
  ```
40
41
 
41
42
  You should see information about available commands.
@@ -60,7 +61,7 @@ If you run `keen` from a directory with this .env file, it will assume the proje
60
61
  To override the project context use the `--project` option:
61
62
 
62
63
  ``` shell
63
- $ keen project:describe --project XXXXXXXXXXXXXXX
64
+ $ keen projects:describe --project XXXXXXXXXXXXXXX
64
65
  ```
65
66
 
66
67
  Similar overrides are available for specifiying API keys: `--master-key`, `--read-key` and `--write-key`.
@@ -68,13 +69,13 @@ Similar overrides are available for specifiying API keys: `--master-key`, `--rea
68
69
  For example:
69
70
 
70
71
  ``` shell
71
- $ keen project:describe --project XXXXXXXXXXXXXXX --master-key AAAAAAAAAAAAAA
72
+ $ keen projects:describe --project XXXXXXXXXXXXXXX --master-key AAAAAAAAAAAAAA
72
73
  ```
73
74
 
74
75
  Shorter aliases exist as well: `-p` for project, `-k` for master key, `-r` for read key, and `-w` for write key.
75
76
 
76
77
  ``` shell
77
- $ keen project:describe -p XXXXXXXXXXXXXXX -k AAAAAAAAAAAAAA
78
+ $ keen projects:describe -p XXXXXXXXXXXXXXX -k AAAAAAAAAAAAAA
78
79
  ```
79
80
 
80
81
  ### Usage
@@ -85,10 +86,10 @@ keen-cli has a variety of commands, and most are namespaced for clarity.
85
86
 
86
87
  ##### Projects
87
88
 
88
- * `project:open` - Open the Project Overview page in a browser
89
- * `project:workbench` - Open the Project Workbench page in a browser
90
- * `project:describe` - Get data about the project. Uses the [project row resource](https://keen.io/docs/api/reference/#project-row-resource).
91
- * `project:collections` - Get schema information about the project's collections. Uses the [event resource](https://keen.io/docs/api/reference/#event-resource).
89
+ * `projects:open` - Open the Project Overview page in a browser
90
+ * `projects:workbench` - Open the Project Workbench page in a browser
91
+ * `projects:describe` - Get data about the project. Uses the [project row resource](https://keen.io/docs/api/reference/#project-row-resource).
92
+ * `projects:collections` - Get schema information about the project's collections. Uses the [event resource](https://keen.io/docs/api/reference/#event-resource).
92
93
 
93
94
  ##### Events
94
95
 
@@ -97,51 +98,58 @@ keen-cli has a variety of commands, and most are namespaced for clarity.
97
98
  Parameters:
98
99
 
99
100
  + `--collection`, `-c`: The collection to add the event to. Alternately you can set `KEEN_COLLECTION_NAME` on the environment if you're working with the same collection frequently.
100
- + `--data`, `-d`: The properties of the event. The value can be JSON or `key=value` pairs delimited by `&` (just like a query string). Data can also be piped in via STDIN.
101
- + `--file`, `-f`: The name of a file that contains newline delimited JSON
102
- + `--csv`: Specify that the file is in CSV format
101
+ + `--batch-size`: Batch size of events posted to Keen, defaults to 1000.
102
+
103
+ Input source parameters:
104
+
105
+ + `--data`, `-d`: Pass an event body on the command line. Make sure to use quotes where necessary.
106
+ + `--file`, `-f`: The name of a file containing events.
107
+
108
+ You can also pass input via `STDIN`.
109
+
110
+ If not otherwise specified, the format of data from any source is assumed to be newline-delimited JSON. CSV and query string-like input is also supported. The associated params:
111
+
112
+ + `--csv`: Specify CSV format. The first line must contain column names. Column names containing `.`, such as `keen.timestamp`, will be converted to nested properties.
113
+ + `--params`: Specify "params" format. Params format looks like `property1=value1&property2=value` etc.
103
114
 
104
115
  Various examples:
105
116
 
106
117
  ``` shell
107
- # create an empty event
118
+ # add an empty event
108
119
  $ keen events:add --collection cli-tests
109
120
 
110
121
  # use the shorter form of collection
111
122
  $ keen events:add -c cli-tests
112
123
 
113
- # add a blank event to a collection specified in the .env file:
124
+ # add a blank event to a collection specified in a .env file:
114
125
  # KEEN_COLLECTION_NAME=cli-tests
115
126
  $ keen events:add
116
127
 
117
- # create an event from JSON
118
- $ keen events:add -c cli-tests -d "{ \"username\" : \"dzello\", \"zsh\": 1 }"
128
+ # add an event from JSON using the --data parameter
129
+ $ keen events:add -c cli-tests --data "{ \"username\" : \"dzello\", \"zsh\": 1 }"
119
130
 
120
- # create an event from key value pairs
121
- $ keen events:add -c cli-tests -d "username=dzello&zsh=1"
131
+ # add an event from key value pairs
132
+ $ keen events:add -c cli-tests -data "username=dzello&zsh=1" --params
122
133
 
123
134
  # pipe in events as JSON
124
135
  $ echo "{ \"username\" : \"dzello\", \"zsh\": 1 }" | keen events:add -c cli-tests
125
136
 
126
- # pipe in events in querystring format
127
- $ echo "username=dzello&zsh=1" | keen events:add -c cli-test
128
-
129
- # specify a file that contains newline delimited json
137
+ # add events from a file that contains newline delimited json
138
+ # { "apple" : "sauce" }
139
+ # { "banana" : "pudding" }
140
+ # { "cherry" : "pie" }
130
141
  $ keen events:add --file events.json
131
142
 
132
- # specify a file in CSV format
133
- $ keen events:add --csv --file events.csv
134
-
135
- # pipe in events from a file of newline delimited json
136
- # { "username" : "dzello", "zsh" : 1 }
137
- # { "username" : "dkador", "zsh" : 1 }
138
- # { "username" : "gphat", "zsh" : 1 }
139
- $ cat events.json | keen events:add -c cli-test
143
+ # add events from a file in CSV format; the first row should be columns
144
+ # fruit,dish
145
+ # apple,sauce
146
+ # banana,pudding
147
+ $ keen events:add --file events.csv --csv
140
148
  ```
141
149
 
142
150
  ##### Queries
143
151
 
144
- `queries:run` - Runs a query and prints the result in pretty JSON.
152
+ `queries:run` - Runs a query and prints the result
145
153
 
146
154
  Parameters:
147
155
 
@@ -158,7 +166,11 @@ Parameters:
158
166
  + `--property-names`: A comma-separated list of property names. Extractions only.
159
167
  + `--latest`: Number of latest events to retrieve. Extractions only.
160
168
  + `--email`: Send extraction results via email, asynchronously. Extractions only.
161
- + `--data`, `-d`: Specify query parameters as JSON instead of query params. Data can also be piped in via STDIN.
169
+
170
+ Input source parameters:
171
+ + `--data`, `-d`: Specify query parameters as JSON instead of query params.
172
+
173
+ You can also pass input via `STDIN`.
162
174
 
163
175
  Some examples:
164
176
 
@@ -215,6 +227,16 @@ $ keen queries:run --collection minecraft-deaths --analysis-type extraction --pr
215
227
  $ echo "{ \"event_collection\" : \"minecraft-deaths\", \"target_property\": \"level\" }" | keen queries:run -a average
216
228
  ```
217
229
 
230
+ **Query URL Generation**
231
+
232
+ Run `keen` with no arguments to see the full list of aliases.
233
+
234
+ `queries:url` - Generates the URL of a query, but does not run it.
235
+
236
+ The same parameters apply as `queries:run`, in addition to one extra.
237
+
238
+ + `--exclude-api-key`: Prevent the API key query param from being included in the output
239
+
218
240
  **Query Aliases**
219
241
 
220
242
  For each type of analysis (e.g. count, average, extraction, etc.) there is an alias that can be used
@@ -228,10 +250,16 @@ $ keen minimum -c cpu-checks -y iowait
228
250
  0.17
229
251
  ```
230
252
 
231
- Run `keen` with no arguments to see the full list of aliases.
253
+ ### Global parameters
254
+
255
+ Parameters that apply to most commands include:
256
+
257
+ + `--pretty`: Prettify API response JSON. Defaults to true, set `--pretty=false` to prevent
258
+ + `--silent`: Silence any output. Defaults to false.
232
259
 
233
260
  ### Changelog
234
261
 
262
+ + 0.1.6 - Big refactoring to make importing events much cleaner and batching happen automatically. Also adds `queries:url`.
235
263
  + 0.1.5 – Support adding events from files with `--file`. Optionally add from CSV with `--csv`.
236
264
  + 0.1.4 – Support absolute timeframes via `--start` and `--end` flags
237
265
  + 0.1.3 – Add querying via JSON. Add query aliases. Add support for extraction fields.
data/keen-cli.gemspec CHANGED
@@ -12,7 +12,7 @@ Gem::Specification.new do |s|
12
12
  s.description = "Record events and run queries from the comfort of your command line"
13
13
  s.license = "MIT"
14
14
 
15
- s.add_dependency "keen", ">= 0.8.4"
15
+ s.add_dependency "keen", ">= 0.8.6"
16
16
  s.add_dependency "thor"
17
17
  s.add_dependency "dotenv"
18
18
 
@@ -0,0 +1,104 @@
1
+ module KeenCli
2
+
3
+ class BatchProcessor
4
+
5
+ # defaults
6
+ DEFAULT_BATCH_SIZE = 1000
7
+ DEFAULT_CSV_OPTIONS = { :converters => :all }
8
+
9
+ # public options, set in constructor
10
+ attr_accessor :collection
11
+ attr_accessor :batch_size
12
+ attr_accessor :params
13
+ attr_accessor :csv
14
+ attr_accessor :csv_options
15
+ attr_accessor :pretty
16
+ attr_accessor :silent
17
+
18
+ # internal state tracking
19
+ attr_accessor :size
20
+ attr_accessor :events
21
+
22
+ def initialize(collection, options={})
23
+ self.collection = collection
24
+ self.batch_size = options[:batch_size] || DEFAULT_BATCH_SIZE
25
+ self.csv = options[:csv]
26
+ self.params = options[:params]
27
+ self.csv_options = DEFAULT_CSV_OPTIONS.merge(options[:csv_options] || {})
28
+ self.events = []
29
+ self.pretty = options[:pretty]
30
+ self.silent = options[:silent]
31
+ self.reset
32
+ end
33
+
34
+ def add(line)
35
+
36
+ # if we're in CSV mode and don't have headers let's try and set them
37
+ if self.csv && !self.csv_options.has_key?(:headers)
38
+
39
+ set_headers(line)
40
+ return
41
+
42
+ end
43
+
44
+ if self.csv
45
+
46
+ csv_row = CSV.parse_line(line, self.csv_options)
47
+ raise "Could not parse! #{line}" unless csv_row
48
+
49
+ hash = row_to_hash(csv_row)
50
+
51
+ elsif self.params
52
+
53
+ hash = Utils.parse_data_as_querystring(line)
54
+
55
+ else
56
+
57
+ hash = JSON.parse(line)
58
+
59
+ end
60
+
61
+ self.events.push(hash)
62
+ self.size += 1
63
+
64
+ self.flush if self.size >= self.batch_size
65
+
66
+ end
67
+
68
+ def flush
69
+ publish_batch(self.collection, self.events) if self.size > 0
70
+ reset
71
+ end
72
+
73
+ def reset
74
+ self.size = 0
75
+ self.events.clear
76
+ end
77
+
78
+ private
79
+
80
+ def set_headers(line)
81
+ csv_row = CSV.parse_line(line, self.csv_options)
82
+ self.csv_options[:headers] = csv_row.to_a
83
+ end
84
+
85
+ def publish_batch(collection, events)
86
+ batches = {}
87
+ batches[collection] = events
88
+ Keen.publish_batch(batches).tap do |result|
89
+ Utils.out_json(result, :pretty => self.pretty, :silent => self.silent)
90
+ end
91
+ end
92
+
93
+ def row_to_hash(csv_row)
94
+ naive_hash = csv_row.to_hash
95
+ naive_hash.map do |main_key, main_value|
96
+ main_key.to_s.split('.').reverse.inject(main_value) do |value, key|
97
+ {key => value}
98
+ end
99
+ end.inject(&:deep_merge)
100
+ end
101
+
102
+ end
103
+
104
+ end
data/lib/keen-cli/cli.rb CHANGED
@@ -5,250 +5,25 @@ require 'csv'
5
5
 
6
6
  require 'keen-cli/utils'
7
7
 
8
- module KeenCli
9
-
10
- class CLI < Thor
11
-
12
- def self.shared_options
13
- option :project, :aliases => ['-p']
14
- option :"master-key", :aliases => ['-k']
15
- option :"read-key", :aliases => ['-r']
16
- option :"write-key", :aliases => ['-w']
17
- end
8
+ require 'keen-cli/shared'
18
9
 
19
- def self.data_options
20
- option :data, :aliases => ['-d']
21
- end
22
-
23
- def self.file_options
24
- option :file, :aliases => ['-f']
25
- option :csv
26
- end
10
+ require 'keen-cli/projects'
11
+ require 'keen-cli/events'
12
+ require 'keen-cli/queries'
27
13
 
28
- def self.collection_options
29
- option :collection, :aliases => ['-c']
30
- end
14
+ module KeenCli
31
15
 
32
- def self.query_options
33
- self.collection_options
34
- option :"analysis-type", :aliases => ['-a']
35
- option :"group-by", :aliases => ['-g']
36
- option :"target-property", :aliases => ['-y']
37
- option :interval, :aliases => ['-i']
38
- option :timeframe, :aliases => ['-t']
39
- option :filters, :aliases => ['-f']
40
- option :percentile
41
- option :"property-names"
42
- option :latest
43
- option :email
44
- option :start, :aliases => ['s']
45
- option :end, :aliases => ['e']
46
- end
16
+ class CLI < Thor
47
17
 
48
18
  desc 'version', 'Print the keen-cli version'
49
19
  map %w(-v --version) => :version
50
20
 
51
- def version
52
- "keen-cli version #{KeenCli::VERSION}".tap do |s|
53
- puts s
54
- end
55
- end
56
-
57
- desc 'project:describe', 'Print information about a project'
58
- map 'project:describe' => :project_describe
59
- shared_options
60
-
61
- def project_describe
62
- Utils.process_options!(options)
63
- Keen.project_info.tap do |info|
64
- puts JSON.pretty_generate(info)
65
- end
66
- end
67
-
68
- desc 'project:collections', 'Print information about a project\'s collections'
69
- map 'project:collections' => :project_collections
70
- shared_options
71
-
72
- def project_collections
73
- Utils.process_options!(options)
74
- Keen.event_collections.tap do |collections|
75
- puts JSON.pretty_generate(collections)
76
- end
77
- end
78
-
79
- desc 'project:open', 'Open a project\'s overview page in a browser'
80
- map 'project:open' => :project_open
81
- shared_options
82
-
83
- def project_open
84
- Utils.process_options!(options)
85
- "https://keen.io/project/#{Keen.project_id}".tap do |project_url|
86
- `open #{project_url}`
87
- end
88
- end
89
-
90
- desc 'project:workbench', 'Open a project\'s workbench page in a browser'
91
- map 'project:workbench' => :project_workbench
92
- shared_options
93
-
94
- def project_workbench
95
- Utils.process_options!(options)
96
- "https://keen.io/project/#{Keen.project_id}/workbench".tap do |project_url|
97
- `open #{project_url}`
98
- end
99
- end
100
-
101
- desc 'queries:run', 'Run a query and print the result'
102
- map 'queries:run' => :queries_run
103
21
  shared_options
104
- query_options
105
- data_options
106
- def queries_run(analysis_type=nil)
107
-
108
- Utils.process_options!(options)
109
-
110
- # work with a new set of options
111
- q_options = {}
112
-
113
- data = nil
114
-
115
- if $stdin.tty?
116
- data = options[:data]
117
- else
118
- ARGV.clear
119
- ARGF.each_line do |line|
120
- data = line
121
- end
122
- end
123
-
124
- # if data is provided, parse it and merge it
125
- unless data.nil?
126
- data_options = JSON.parse(data)
127
- q_options.merge!(data_options)
128
- end
129
-
130
- # convert dashes to underscores, and merge all into q_options
131
- q_options.merge!(options.inject({}) do |memo, element|
132
- if ['analysis-type', 'group-by', 'target-property', 'property-names'].include?(element.first)
133
- memo[element.first.sub('-', '_')] = element.last
134
- else
135
- memo[element.first] = element.last
136
- end
137
- memo
138
- end)
139
-
140
- collection = Utils.get_collection_name(q_options)
141
- analysis_type = analysis_type || q_options["analysis_type"]
142
-
143
- # delete fields that shouldn't be passed to keen-gem as options
144
- q_options.delete("collection")
145
- q_options.delete("event_collection")
146
- q_options.delete("data")
147
- q_options.delete("analysis_type")
148
-
149
- if property_names = q_options.delete("property_names")
150
- q_options[:property_names] = property_names.split(",")
151
- end
152
-
153
- if start_time = q_options.delete("start")
154
- q_options[:timeframe] = { :start => start_time }
155
- end
156
-
157
- if end_time = q_options.delete("end")
158
- q_options[:timeframe] = q_options[:timeframe] || {}
159
- q_options[:timeframe][:end] = end_time
160
- end
161
-
162
- raise "No analysis type given!" unless analysis_type
163
- raise "No collection given!" unless collection
164
-
165
- Keen.send(analysis_type, collection, q_options).tap do |result|
166
- if result.is_a?(Hash) || result.is_a?(Array)
167
- puts JSON.pretty_generate(result)
168
- else
169
- puts result
170
- end
171
- end
172
- end
173
-
174
- ANALYSIS_TYPES = %w(average count count-unique extraction median minimum maximum sum percentile select-unique)
175
-
176
- ANALYSIS_TYPES.each do |analysis_type|
177
- underscored_analysis_type = analysis_type.sub('-', '_')
178
- desc analysis_type, "Alias for queries:run -c #{underscored_analysis_type}"
179
- map analysis_type => method_name = "queries_run_#{underscored_analysis_type}"
180
- shared_options
181
- query_options
182
- data_options
183
- self.send(:define_method, method_name) { queries_run(underscored_analysis_type) }
184
- end
185
-
186
- desc 'events:add', 'Add one or more events and print the result'
187
- map 'events:add' => :events_add
188
-
189
- shared_options
190
- data_options
191
- file_options
192
- collection_options
193
-
194
- def events_add
195
-
196
- events = []
197
- collection = Utils.get_collection_name(options)
198
-
199
- if options[:csv]
200
-
201
- data = File.read(options[:file])
202
- csv = CSV.new(data, :headers => true, :header_converters => :symbol, :converters => :all)
203
- events = csv.to_a.map {|row| row.to_hash }
204
-
205
- else
206
-
207
- if $stdin.tty?
208
- if data = options[:data]
209
- events.push(data)
210
- elsif file = options[:file]
211
- File.readlines(file).each do |line|
212
- events.push(line)
213
- end
214
- else
215
- events.push({})
216
- end
217
- else
218
- ARGV.clear
219
- ARGF.each_line do |line|
220
- events.push(line)
221
- end
222
- end
223
-
224
- events = events.map do |event|
225
- begin
226
- JSON.parse(event)
227
- rescue
228
- begin
229
- Utils.parse_data_as_querystring(event)
230
- rescue
231
- event
232
- end
233
- end
234
- end
235
-
236
- end
237
-
238
- if events.length > 1
239
-
240
- Keen.publish_batch(collection => events).tap do |result|
241
- puts JSON.pretty_generate(result)
242
- end
243
-
244
- else
245
-
246
- Keen.publish(collection, events.first).tap do |result|
247
- puts JSON.pretty_generate(result)
248
- end
249
22
 
23
+ def version
24
+ "keen-cli version #{KeenCli::VERSION}".tap do |s|
25
+ Utils.out(s, options)
250
26
  end
251
-
252
27
  end
253
28
 
254
29
  end
@@ -0,0 +1,70 @@
1
+ require 'keen-cli/batch_processor'
2
+
3
+ module KeenCli
4
+
5
+ class CLI < Thor
6
+
7
+ def self.events_options
8
+ option :csv
9
+ option :params
10
+ option :'batch-size'
11
+ end
12
+
13
+ desc 'events:add', 'Add one or more events and print the result'
14
+ map 'events:add' => :events_add
15
+
16
+ shared_options
17
+ data_options
18
+ file_options
19
+ collection_options
20
+ events_options
21
+
22
+ def events_add
23
+
24
+ Utils.process_options!(options)
25
+
26
+ collection = Utils.get_collection_name(options)
27
+
28
+ batch_processor = BatchProcessor.new(collection,
29
+ :csv => options[:csv],
30
+ :params => options[:params],
31
+ :pretty => options[:pretty],
32
+ :silent => options[:silent],
33
+ :batch_size => options[:'batch-size'])
34
+
35
+ total_events = 0
36
+
37
+ if data = options[:data]
38
+ batch_processor.add(data)
39
+ total_events += 1
40
+ end
41
+
42
+ if file = options[:file]
43
+ File.readlines(file).each do |line|
44
+ batch_processor.add(line)
45
+ total_events += 1
46
+ end
47
+ end
48
+
49
+ if !$stdin.tty?
50
+ ARGV.clear
51
+ ARGF.each_line do |line|
52
+ batch_processor.add(line)
53
+ total_events += 1
54
+ end
55
+ end
56
+
57
+ if $stdin.tty? && data.nil? && file.nil?
58
+ batch_processor.add("{}")
59
+ total_events += 1
60
+ end
61
+
62
+ batch_processor.flush
63
+
64
+ total_events
65
+
66
+ end
67
+
68
+ end
69
+
70
+ end