td 0.11.1 → 0.11.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,15 +1,15 @@
1
1
  ---
2
2
  !binary "U0hBMQ==":
3
3
  metadata.gz: !binary |-
4
- MjhhYWI5NWE1ZmI3ZTkwZjY0OTNhMGE3N2VjNjUzNzk2ZWY2OGJkYQ==
4
+ M2EwMGZhM2U3ZmE4ZWE3YzE4ZGM4NTA3NDVmYzZlZTFmMmVkZDdiMg==
5
5
  data.tar.gz: !binary |-
6
- NDZmMDJlMDQyZmViMjY5NTNmMTE0ODg5ZWE2MGYxZjJhYTg4Y2VhZQ==
6
+ MzUwZGIzYTVjOGE2Yzc1NDcyYWYzZTE2YjNlZjNkMzY3YTljYTY4OQ==
7
7
  SHA512:
8
8
  metadata.gz: !binary |-
9
- NGZmMTk4ZTk5ZWFhYzc4YmEyZTE0ZmI0MjA2ODYzOWQxNzQ0ZTc5OGRjZDhh
10
- OTI0NjJiZjA3MDhiZmFjNjg3MThjNjI1ZjcyZjNiOTNiOWU5MjQ2M2FhOWRh
11
- ZTI4OGI2ZTk1YTEwYmVkZGJmNGU4Y2YzYTY1ZDc5MmNkYTI5ZTQ=
9
+ ZDE3YWM1OGQ5MWZmNTIxM2Q4NDNhMDdlMDUwYTcxNzYwZWUyNGU4YjRhNWFk
10
+ ZjZjOTFjNWRjZmQ0YWVhNmFlYTQyZDIyMTQ1NTBjNTA0NjI5YzlhMTMwMGE0
11
+ MDRlNDFiZDZiMTE1YjBlZmZkM2RkMzIxNDQ3NDc2OGEzNGMwY2M=
12
12
  data.tar.gz: !binary |-
13
- NjNiNTI4MGI4MDY0MGYzMjZiMTBmNzBhYzJmNjNmZjcxZDgxZWVjNTgzYzUx
14
- ZGZiMzViYmQ0ZjQwNWIwZmI5MWQ5ZmExYTA3MzZkZDlkY2RmNTEzNDMxYzNj
15
- ZGJhZTFlZDExYjc4YzdkYzUwZmJjNjc3NTlkZGQ5OTI2MDQ4NjY=
13
+ OTY4NWJkYmJiNmU5NTRjYTUyZTA0MDg5ODQ1ZmM0NmMwMjY2MmUzMjhiZTdk
14
+ ZGE5MWQ1OWI4MzQ0YTlkMzcyY2YyMWVhNTc1ZjhhMTVkZTAyOTExZWU1OWM1
15
+ M2QzNjM5MTQ1ZmY0NzA0NTNjNTI4ZjQwNWIxNmMzYzY3YWY4YWM=
data/ChangeLog CHANGED
@@ -1,3 +1,32 @@
1
+ == 2014-06-18 version 0.11.2
2
+
3
+ * Improve the update experience by showing a download progress indicator
4
+ * Fix issue on Windows OS where the JAR version file handle was not closed
5
+ and caused permissions access problems to the file in subsequent attempts
6
+ to open the file
7
+ * Add support for other/alternative backends using the -e / --endpoint
8
+ option. The backend can be also saved on the configuration file of use
9
+ via the 'td server:endpoint' or when running 'td -e <newendpoint> account'
10
+ command. The TD_API_SERVER environment variable still takes precendence
11
+ over all these settings
12
+ * Removed Treasure Data Result Output database and table name client side
13
+ validation since it's performed in the API backend
14
+ * Suppress stack traces for APIError exceptions from the client library
15
+ * Changed the default HTTP port from 80 to 443 when interacting with the API
16
+ server to support SSL
17
+ * Support working entirely without a configuration file (hence without
18
+ having to run 'td account' prior to using the CLI) by passing the -k option
19
+ on the command line
20
+ * don't wipe out the current JAR file if an attempt to retrieve the most updated
21
+ JAR file failed because of network connectivity problems
22
+ * Show the job result only for finished jobs: unfinished jobs don't have any
23
+ result to display anyway)
24
+ * Honor the HTTP_PROXY setting also when updating the JAR file or the Toolbelt
25
+ itself
26
+ * Fix parsing of the HTTP_PROXY variable for the Bulk Loader's usage
27
+ * Allow optionally disabling the JAR file auto-update feature by setting the
28
+ TD_TOOLBELT_JAR_UPDATE environment variable to 0
29
+
1
30
  == 2014-04-29 version 0.11.1
2
31
 
3
32
  * Fix Treasure Data query result output database and table validation
@@ -57,6 +57,70 @@ Then run following commands on MinGW Shell:
57
57
  $ bundle install # don't use "--path" option
58
58
  $ rake exe:build # don't use "bundle exec"
59
59
 
60
+ == Testing Hooks
61
+
62
+ The CLI implements several hooks to enable/disable/trigger special behaviors.
63
+ These hooks are expressed as environment variables and can therefore be provided in several ways:
64
+
65
+ === How to Use
66
+
67
+ * Unix / Linux / MacOSX
68
+ * environment variable export in the shell the command is executed. The setting remains active until the shell is closed. E.g.:
69
+
70
+ $ export TD_TOOLBELT_DEBUG=1
71
+
72
+ * in the shell configuration file, to be active in any new shell that is opened. E.g.: add
73
+
74
+ export TD_TOOLBELT_DEBUG=1
75
+
76
+ to <tt>~/.bashrc</tt> or equivalent shell configuration file.
77
+ To make the setting active in the current shell, source the configuration file, e.g.:
78
+
79
+ $ source ~/.bashrc
80
+
81
+ * on the command line at runtime (active only for the duration of the command). E.g.:
82
+
83
+ $ TD_TOOLBELT_DEBUG=1 td ....
84
+
85
+ * as alias on in the current shell. The setting remains active until the shell is closed. E.g.:
86
+
87
+ $ alias td='TD_TOOLBELT_DEBUG=1 td'
88
+
89
+ * as alias in configuration file, to be active in any new shell that is opened. E.g.:
90
+
91
+ alias td='TD_TOOLBELT_DEBUG=1 td'`
92
+
93
+ to <tt>~/.bashrc</tt> or equivalent shell configuration file.
94
+ To make the setting active in the current shell, source the configuration file, e.g.:
95
+
96
+ $ source ~/.bashrc
97
+
98
+ * Windows
99
+ * in the command prompt the command is executed. The setting remains active until the command prompt window is closed. E.g.:
100
+
101
+ cmd> set TD_TOOLBELT_DEBUG=1
102
+
103
+ * as a global environment variable in the system settings. It will be active for all new command prompt windows.
104
+
105
+ These are the available hooks:
106
+
107
+ * Enable debugging mode:
108
+
109
+ $ TD_TOOLBELT_DEBUG=1
110
+
111
+ * JAR auto update (enabled by default is not specified). This setting does not affect <tt>import:jar_update</tt>:
112
+ * Enable:
113
+
114
+ $ TD_TOOLBELT_JAR_UPDATE=1
115
+
116
+ * Disable:
117
+
118
+ $ TD_TOOLBELT_JAR_UPDATE=0
119
+
120
+ * Specify an alternative endpoint to use to update the toolbelt (default: http://toolbelt.treasuredata.com):
121
+
122
+ $ TD_TOOLBELT_UPDATE_ROOT="http://toolbelt.treasuredata.com"
123
+
60
124
 
61
125
  = Copyright
62
126
 
data/Rakefile CHANGED
@@ -42,6 +42,17 @@ def install_use_gems(target_dir)
42
42
  ENV['GEM_PATH'] = ''
43
43
  USE_GEMS.each {|gem|
44
44
  begin
45
+ # this is a hack to have the dependency handling for the 'td' gem
46
+ # pick up a local gem for 'td-client' so as to be able to build
47
+ # and test the 'toolbelt' package without publishing the 'td-client'
48
+ # gem on rubygems.com
49
+ unless ENV['TD_TOOLBELT_LOCAL_CLIENT_GEM'].nil?
50
+ unless File.exists? ENV['TD_TOOLBELT_LOCAL_CLIENT_GEM']
51
+ raise "Cannot find gem file with path #{ENV['TD_TOOLBELT_LOCAL_CLIENT_GEM']}"
52
+ end
53
+ puts "Copy local gem #{ENV['TD_TOOLBELT_LOCAL_CLIENT_GEM']} to #{Dir.pwd}"
54
+ FileUtils.cp File.expand_path(ENV['TD_TOOLBELT_LOCAL_CLIENT_GEM']), Dir.pwd
55
+ end
45
56
  Gem::GemRunner.new.run ["install", gem, "--no-rdoc", "--no-ri"]
46
57
  rescue Gem::SystemExitException => e
47
58
  unless e.exit_code.zero?
@@ -4,7 +4,7 @@ task 'exe:build' => :build do
4
4
  create_build_dir('exe') do |dir|
5
5
  # create ./installers/
6
6
  FileUtils.mkdir_p "installers"
7
- installer_path = download_resource('http://rubyforge.org/frs/download.php/76054/rubyinstaller-1.9.3-p194.exe')
7
+ installer_path = download_resource('http://dl.bintray.com/oneclick/rubyinstaller/rubyinstaller-1.9.3-p545.exe?direct')
8
8
  FileUtils.cp installer_path, "installers/rubyinstaller.exe"
9
9
 
10
10
  variables = {
@@ -18,10 +18,10 @@ task 'pkg:build' => :build do
18
18
 
19
19
  zip_files(project_root_path('pkg/td-update-pkg.zip'), 'td-client.build')
20
20
 
21
- # crete ./bundle/td-client.pkg/Bom
21
+ # create ./bundle/td-client.pkg/Bom
22
22
  sh "mkbom -s td-client.build bundle/td-client.pkg/Bom"
23
23
 
24
- # crete ./bundle/td-client.pkg/Scripts/
24
+ # create ./bundle/td-client.pkg/Scripts/
25
25
  install_resource 'pkg/postinstall', 'bundle/td-client.pkg/Scripts/postinstall', 0755
26
26
 
27
27
  variables = {
@@ -13,11 +13,22 @@ module Command
13
13
 
14
14
  user_name = op.cmd_parse
15
15
 
16
+ endpoint = nil
17
+ # user may be calling 'td account' with the -e / --endpoint
18
+ # option, which we want to preserve and save
19
+ begin
20
+ endpoint = Config.endpoint
21
+ rescue ConfigNotFoundError => e
22
+ # the endpoint is neither stored in the config file
23
+ # nor passed as option on the command line
24
+ end
25
+
16
26
  conf = nil
17
27
  begin
18
28
  conf = Config.read
19
29
  rescue ConfigError
20
30
  end
31
+
21
32
  if conf && conf['account.apikey']
22
33
  unless force
23
34
  if conf['account.user']
@@ -68,7 +79,10 @@ module Command
68
79
 
69
80
  begin
70
81
  # enalbe SSL for the authentication
71
- client = Client.authenticate(user_name, password, :ssl=>true)
82
+ opts = {}
83
+ opts[:ssl] = true
84
+ opts[:endpoint] = endpoint if endpoint
85
+ client = Client.authenticate(user_name, password, opts)
72
86
  rescue TreasureData::AuthError
73
87
  $stderr.puts "User name or password mismatched."
74
88
  end
@@ -77,28 +91,27 @@ module Command
77
91
  end
78
92
  return unless client
79
93
 
80
- $stderr.puts "Authenticated successfully."
94
+ puts "Authenticated successfully."
81
95
 
82
96
  conf ||= Config.new
83
97
  conf["account.user"] = user_name
84
98
  conf["account.apikey"] = client.apikey
99
+ conf['account.endpoint'] = endpoint if endpoint
85
100
  conf.save
86
101
 
87
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "db:create <db_name>' to create a database."
102
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "db:create <db_name>' to create a database."
88
103
  end
89
104
 
90
105
  def account_usage(op)
91
106
  op.cmd_parse
92
107
 
93
108
  client = get_client
94
-
95
109
  a = client.account
96
110
 
97
111
  $stderr.puts "Storage: #{a.storage_size_string}"
98
112
  end
99
113
 
100
114
  private
101
-
102
115
  if Helpers.on_windows?
103
116
  require 'Win32API'
104
117
 
@@ -18,7 +18,7 @@ module Command
18
18
 
19
19
  if rows.empty?
20
20
  $stderr.puts "There are no access controls."
21
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "acl:grant <subject> <action> <scope>' to grant permissions."
21
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "acl:grant <subject> <action> <scope>' to grant permissions."
22
22
  end
23
23
  end
24
24
 
@@ -55,9 +55,8 @@ module Command
55
55
  conf["account.apikey"] = apikey
56
56
  conf.save
57
57
 
58
- $stderr.puts "API key is set."
59
-
60
- $stderr.puts "Use '#{$prog} db:create <db_name>' to create a database."
58
+ puts "API key is set."
59
+ puts "Use '#{$prog} db:create <db_name>' to create a database."
61
60
  end
62
61
 
63
62
  end
@@ -20,7 +20,7 @@ module Command
20
20
 
21
21
  if rows.empty?
22
22
  $stderr.puts "There are no bulk import sessions."
23
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "bulk_import:create <name> <db> <table>' to create a session."
23
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "bulk_import:create <name> <db> <table>' to create a session."
24
24
  end
25
25
  end
26
26
 
@@ -61,7 +61,7 @@ module Command
61
61
  bi = client.bulk_import(name)
62
62
  unless bi
63
63
  $stderr.puts "Bulk import session '#{name}' does not exist."
64
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "bulk_import:create <name> <db> <table>' to create a session."
64
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "bulk_import:create <name> <db> <table>' to create a session."
65
65
  exit 1
66
66
  end
67
67
 
@@ -184,7 +184,7 @@ module Command
184
184
  job = client.perform_bulk_import(name)
185
185
 
186
186
  $stderr.puts "Job #{job.job_id} is queued."
187
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "job:show [-w] #{job.job_id}' to show the status."
187
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "job:show [-w] #{job.job_id}' to show the status."
188
188
  end
189
189
  end
190
190
 
@@ -246,7 +246,7 @@ module Command
246
246
  if bi.status == 'performing'
247
247
  $stderr.puts "Bulk import session '#{name}' is already performing."
248
248
  $stderr.puts "Add '-f' option to force start."
249
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "job:kill #{bi.job_id}' to cancel the last trial."
249
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "job:kill #{bi.job_id}' to cancel the last trial."
250
250
  exit 1
251
251
  elsif bi.status == 'ready'
252
252
  $stderr.puts "Bulk import session '#{name}' is already ready to commit."
@@ -259,7 +259,7 @@ module Command
259
259
  job = client.perform_bulk_import(name)
260
260
 
261
261
  $stderr.puts "Job #{job.job_id} is queued."
262
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "job:show [-w] #{job.job_id}' to show the status."
262
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "job:show [-w] #{job.job_id}' to show the status."
263
263
 
264
264
  if wait
265
265
  require 'td/command/job' # wait_job
@@ -330,13 +330,13 @@ module Command
330
330
  bi = bis.find {|bi| name == bi.name }
331
331
  unless bi
332
332
  $stderr.puts "Bulk import session '#{name}' does not exist."
333
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "bulk_import:create <name> <db> <table>' to create a session."
333
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "bulk_import:create <name> <db> <table>' to create a session."
334
334
  exit 1
335
335
  end
336
336
 
337
337
  if bi.status == "uploading" || bi.status == "performing"
338
338
  $stderr.puts "Bulk import session '#{name}' is not performed."
339
- $stderr.puts "Use '#{$prog} " + Config.cl_apikey_string + "bulk_import:perform <name>' to run."
339
+ $stderr.puts "Use '#{$prog} " + Config.cl_options_string + "bulk_import:perform <name>' to run."
340
340
  exit 1
341
341
  end
342
342
 
@@ -17,6 +17,9 @@ module Command
17
17
  class BulkImportExecutionError < ArgumentError
18
18
  end
19
19
 
20
+ class UpdateError < ArgumentError
21
+ end
22
+
20
23
  private
21
24
  def initialize
22
25
  @render_indent = ''
@@ -26,15 +29,30 @@ module Command
26
29
  unless opts.has_key?(:ssl)
27
30
  opts[:ssl] = Config.secure
28
31
  end
32
+
33
+ # apikey is mandatory
29
34
  apikey = Config.apikey
30
- unless apikey
31
- raise ConfigError, "Account is not configured."
35
+ raise ConfigError, "Account is not configured." unless apikey
36
+
37
+ # optional, if not provided a default is used from the ruby client library
38
+ begin
39
+ if Config.endpoint
40
+ opts[:endpoint] = Config.endpoint
41
+ end
42
+ rescue ConfigNotFoundError => e
43
+ # rescue the ConfigNotFoundError exception which originates when
44
+ # the config file is not found because the check on the apikey
45
+ # guarantees that the API key has been provided on the command
46
+ # line and that's good enough to continue since the default
47
+ # endpoint will be used in place of this definition.
32
48
  end
49
+
33
50
  opts[:user_agent] = "TD: #{TOOLBELT_VERSION}"
34
51
  if h = ENV['TD_API_HEADERS']
35
52
  pairs = h.split("\n")
36
53
  opts[:headers] = Hash[pairs.map {|pair| pair.split('=', 2) }]
37
54
  end
55
+
38
56
  Client.new(apikey, opts)
39
57
  end
40
58
 
@@ -111,7 +129,7 @@ EOS
111
129
  end
112
130
  end
113
131
 
114
- def humanize_time(time, is_ms = false)
132
+ def self.humanize_time(time, is_ms = false)
115
133
  if time.nil?
116
134
  return ''
117
135
  end
@@ -148,7 +166,7 @@ EOS
148
166
  end
149
167
 
150
168
  # assumed to
151
- def humanize_elapsed_time(start, finish)
169
+ def self.humanize_elapsed_time(start, finish)
152
170
  if start
153
171
  if !finish
154
172
  finish = Time.now.utc
@@ -211,5 +229,72 @@ EOS
211
229
  end
212
230
  end
213
231
 
214
- end
215
- end
232
+ def self.validate_api_endpoint(endpoint)
233
+ require 'uri'
234
+
235
+ uri = URI.parse(endpoint)
236
+ unless uri.kind_of?(URI::HTTP) || uri.kind_of?(URI::HTTPS)
237
+ raise ParameterConfigurationError,
238
+ "API server endpoint URL must use 'http' or 'https' protocol. Example format: 'https://api.treasuredata.com'"
239
+ end
240
+
241
+ if !(md = /(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})/.match(uri.host)).nil? # IP address
242
+ md[1..-1].each { |v|
243
+ if v.to_i < 0 || v.to_i > 255
244
+ raise ParameterConfigurationError,
245
+ "API server IP address must a 4 integers tuple, with every integer in the [0,255] range. Example format: 'https://1.2.3.4'"
246
+ end
247
+ }
248
+ else # host name validation
249
+ unless uri.host =~ /\.treasure\-?data\.com$/
250
+ raise ParameterConfigurationError,
251
+ "API server endpoint URL must end with '.treasuredata.com' or '.treasure-data.com'. Example format: 'https://api.treasuredata.com'"
252
+ end
253
+ unless uri.host =~ /[\d\w\.]+\.treasure\-?data\.com$/
254
+ raise ParameterConfigurationError,
255
+ "API server endpoint URL must have prefix before '.treasuredata.com' or '.treasure-data.com'. Example format: 'https://api.treasuredata.com'."
256
+ end
257
+ end
258
+ end
259
+
260
+ def self.get_http_class
261
+ # use Net::HTTP::Proxy in place of Net::HTTP if a proxy is provided
262
+ http_proxy = ENV['HTTP_PROXY']
263
+ if http_proxy
264
+ http_proxy = (http_proxy =~ /\Ahttp:\/\/(.*)\z/) ? $~[1] : http_proxy
265
+ host, port = http_proxy.split(':', 2)
266
+ port = (port ? port.to_i : 80)
267
+ return Net::HTTP::Proxy(host, port)
268
+ else
269
+ return Net::HTTP
270
+ end
271
+ end
272
+
273
+ class DownloadProgressIndicator
274
+ def initialize(msg, start_time, periodicity)
275
+ @base_msg = msg
276
+ @start_time = start_time
277
+ @last_time = start_time
278
+ @periodicity = periodicity
279
+
280
+ print @base_msg + " " * 10
281
+ end
282
+ end
283
+
284
+ class TimeBasedDownloadProgressIndicator < DownloadProgressIndicator
285
+ def update
286
+ # progress indicator
287
+ if (time = Time.now.to_i) - @last_time > @periodicity
288
+ msg = "\r#{@base_msg}: #{Command.humanize_elapsed_time(@start_time, time)} elapsed"
289
+ print msg + " " * 10
290
+ @last_time = time
291
+ end
292
+ end
293
+
294
+ def finish
295
+ puts "\r#{@base_msg}...done" + " " * 20
296
+ end
297
+ end
298
+
299
+ end # module Command
300
+ end # module TrasureData