elasticshelf 0.0.4 → 0.0.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 17341df9c00b09d1a9627fedc1b198469cdd7873
4
- data.tar.gz: 32c212f38fe54af22d16159dfadfe5948bafc226
3
+ metadata.gz: d7968094ac39d4574712c7eb852c759f1937a276
4
+ data.tar.gz: 3212b4fb19424ed90dcc19f4a4c07979629c8827
5
5
  SHA512:
6
- metadata.gz: 309f27d6d4b3bdc13222b2ab84f9d578ab1f0882abb2a8e86d7a657854e8084e8921ad1d46c700e86a039c96b033b9541c67d99c4958e381a80b136b38a6fdd1
7
- data.tar.gz: 646a30138b4cdea5c0d1f99dd9a2c8b2b255a0ae29d820eca3f6d632ccec048797664a9216f2f61756fe94386824281004951a32204fd45f737cddfb67939a2c
6
+ metadata.gz: bb262605adb4b05ddb5d2919d480f77b30c56a6d959f33fe0cc8e61e0866109ee3382c3826267df1b5df9d5881d5c555754e0ced1afdc4de0e025f65534ff928
7
+ data.tar.gz: 5cdae33b76fe925e5cdba056490c13741010f6b9c260e2c273a3264e814ed6f46d2f0d4e42e7397f4dbfc47656f7089b58ff03586d950a474ffc67a50a968553
data/README.md CHANGED
@@ -1,14 +1,15 @@
1
1
  # Elasticshelf
2
2
 
3
- Ruby integrations for Elasticsearch indices management using close, delete, snapshot and restore.
3
+ Manage Elasticsearch indices using wither, close, open, delete, snapshot, and restore.
4
4
 
5
- Inspired by Curator at https://github.com/elasticsearch/curator.git, but if
6
- you are an ophidiophobe, no worries, this is written in Ruby :-)
5
+ Most of the index actions can be performed by setting an expiry via cutoff days.
7
6
 
8
- Honestly, there is nothing special here that can not already be done with the elasticsearch-ruby gem,
9
- but it's just packaged up as a convenience gem.
7
+ The gem may be used within a Rails app or in a Ruby script for cron jobs.
10
8
 
11
- The goal is to perform the following on Elasticsearch indices:
9
+ > Note: this is inspired by Curator at https://github.com/elasticsearch/curator.git,
10
+ > but if you are an ophidiophobe, no worries, this is written in Ruby :-)
11
+
12
+ The following actions may be performed on Elasticsearch indices:
12
13
  * **wither** (i.e. disable bloom filter)
13
14
  * **close**
14
15
  * **open**
@@ -152,7 +153,7 @@ es.snapshot_expired_indices
152
153
  es.indices = "index1" ... single
153
154
  es.indices = "index1,index2" ... multiples
154
155
  es.repo = 'name_of_the_snapshot_repository'
155
- es.snapshot = 'name_of_this_snapshot'
156
+ es.snapshot = 'name_of_the_snapshot'
156
157
  es.snapshot_expired_indices
157
158
  ```
158
159
 
@@ -168,7 +169,7 @@ es.snapshot_get
168
169
  ### Snapshot delete
169
170
 
170
171
  ```
171
- es.snapshot = 'name_of_this_snapshot'
172
+ es.snapshot = 'name_of_the_snapshot'
172
173
  es.snapshot_delete
173
174
  ```
174
175
 
@@ -176,8 +177,12 @@ es.snapshot_delete
176
177
 
177
178
  ```
178
179
  es.repo = 'name_of_the_snapshot_repository'
179
- es.snapshot = "name_of_this_snapshot"
180
- es.indices = "index1"
180
+ es.snapshot = "name_of_the_snapshot"
181
+ - do one of these:
182
+ 1. es.indices = "" # restores all indices in the snapshot
183
+ 2. es.indices = "index1,index2" # restore multiple indices
184
+ 3. es.indices = "index1" # restore a single index
185
+ - note: es.snapshot_get can be used to list indices in a snapshot
181
186
  es.snapshot_restore
182
187
  puts "errors=#{es.errors.inspect}"
183
188
  puts "results=#{es.results.inspect}"
@@ -201,8 +206,37 @@ es.version ... of this gem
201
206
  * can not snapshot a closed index
202
207
  * can not restore an open index from a snapshot
203
208
  * deleting all (i.e. '*' or '_all') indices is not allowed
204
- * both es.errors and es.results should be checked after most usage
205
- * exceptions are not re-raised from the elasticsearch-ruby gem, but the class/message are echoed in es.errors
209
+ * both es.errors and es.results should be checked after any action
210
+ * exceptions are not re-raised from the elasticsearch-ruby gem, but the exception's class+message are copied into es.errors
211
+ * the following defaults are set, but may be overridden:
212
+ * es.wait_for_completion = true
213
+ * es.ignore_unavailable = true
214
+ * es.include_global_state = false
215
+ * es.date_separator = '.'
216
+ * es.indices_prefix = 'logstash-'
217
+ * es.repo_type = 'fs'
218
+ * es.repo_compressed = true
219
+
220
+ ### Actions
221
+ 1. **get_settings_index** name
222
+ 1. **find_expired_indices**
223
+ 1. **open_index** name
224
+ 1. **index_closed?** name
225
+ 1. **close_index** name
226
+ 1. **close_expired_indices**
227
+ 1. **delete_index** name
228
+ 1. **delete_expired_indices**
229
+ 1. **wither_index** name
230
+ 1. **wither_expired_indices**
231
+ 1. **snapshot_create_repository**
232
+ 1. **snapshot_get_repository**
233
+ 1. **snapshot_delete_repository**
234
+ 1. **snapshot_expired_indices**
235
+ 1. **snapshot_create**
236
+ 1. **snapshot_get**
237
+ 1. **snapshot_delete**
238
+ 1. **snapshot_restore**
239
+
206
240
 
207
241
  ## Contributing
208
242
 
data/lib/elasticshelf.rb CHANGED
@@ -9,30 +9,33 @@ module Elasticshelf
9
9
 
10
10
  attr_reader :get_info, :version
11
11
 
12
- attr_reader :expired_indices, :indices_affected, :indices_rejected, :indices_closed, :indices_deleted, :indices_withered
12
+ attr_reader :indices_expired, :indices_affected, :indices_rejected, :indices_closed, :indices_deleted, :indices_withered
13
13
 
14
14
  attr_accessor :indices_prefix, :date_separator, :cutoff_days
15
15
 
16
- attr_accessor :repo, :repo_type, :repo_location, :repo_compressed, :snapshot, :indices
16
+ attr_accessor :repo, :repo_type, :repo_location, :repo_compressed
17
+
18
+ attr_accessor :snapshot, :indices
17
19
 
18
20
  attr_accessor :wait_for_completion, :ignore_unavailable, :include_global_state
19
21
 
20
- attr_accessor :snapshot_expired_indices_name
22
+ attr_accessor :snapshot_indices_expired_name
21
23
 
22
24
  def new(arguments={})
23
25
  @client = Elasticsearch::Transport::Client.new(arguments)
24
26
  @wait_for_completion = true
25
27
  @ignore_unavailable = true
26
28
  @include_global_state = false
27
- @snapshot_expired_indices_name = nil
29
+ @snapshot_indices_expired_name = nil
28
30
  @date_separator = '.'
29
31
  @indices_prefix = 'logstash-'
30
32
  @cutoff_days = -1
31
33
  @repo_type = 'fs'
32
34
  @repo_compressed = true
35
+ @indices = []
33
36
  @indices_affected = {}
34
37
  @indices_rejected = {}
35
- @expired_indices = {}
38
+ @indices_expired = {}
36
39
  @indices_closed = {}
37
40
  @indices_deleted = {}
38
41
  @indices_withered = {}
@@ -52,7 +55,7 @@ module Elasticshelf
52
55
  return {} if blank? @date_separator
53
56
  return {} if @cutoff_days.nil?
54
57
  return {} if @cutoff_days.to_i < 0
55
- @expired_indices = {}
58
+ @indices_expired = {}
56
59
  # cutoff_date = cutoff_days_as_date
57
60
  # cutoff_date_beginning_of_day = set_time_to_beginning_of_day(cutoff_date.strftime("%Y.%m.%d"))
58
61
  cutoff_date_beginning_of_day = cutoff_days_as_date
@@ -68,7 +71,7 @@ module Elasticshelf
68
71
  index_time = set_time_to_beginning_of_day(unprefixed_index_name, separator=@date_separator)
69
72
  expiry = (cutoff_date_beginning_of_day - index_time).to_i
70
73
  if expiry > 0
71
- @expired_indices[index_name] = expiry
74
+ @indices_expired[index_name] = expiry
72
75
  end
73
76
  end
74
77
  end
@@ -95,8 +98,8 @@ module Elasticshelf
95
98
  end
96
99
 
97
100
  def close_expired_indices
98
- find_expired_indices if blank? @expired_indices
99
- @expired_indices.each do |k,v|
101
+ find_expired_indices if blank? @indices_expired
102
+ @indices_expired.each do |k,v|
100
103
  close_index(k)
101
104
  puts "closed index '#{k}' expired #{v} days ago"
102
105
  end
@@ -114,8 +117,8 @@ module Elasticshelf
114
117
 
115
118
  def delete_expired_indices
116
119
  reset_errors_results
117
- find_expired_indices if blank? @expired_indices
118
- @expired_indices.each do |k,v|
120
+ find_expired_indices if blank? @indices_expired
121
+ @indices_expired.each do |k,v|
119
122
  delete_index(k)
120
123
  puts "deleted '#{k}' expired #{v} days ago"
121
124
  end
@@ -142,8 +145,8 @@ module Elasticshelf
142
145
 
143
146
  def wither_expired_indices
144
147
  reset_errors_results
145
- find_expired_indices if blank? @expired_indices
146
- @expired_indices.each do |k,v|
148
+ find_expired_indices if blank? @indices_expired
149
+ @indices_expired.each do |k,v|
147
150
  wither_index(k)
148
151
  puts "withered '#{k}' expired #{v} days ago"
149
152
  end
@@ -204,12 +207,12 @@ module Elasticshelf
204
207
  def snapshot_expired_indices
205
208
  @results = @errors = {}
206
209
  find_expired_indices
207
- if blank? @expired_indices
210
+ if blank? @indices_expired
208
211
  @results = {"result" => "no expired indices found"}
209
212
  return
210
213
  end
211
214
  open_indices = []
212
- @expired_indices.keys.each do |index_name|
215
+ @indices_expired.keys.each do |index_name|
213
216
  # only open indices can be used in a snapshot:
214
217
  if index_closed?(index_name)
215
218
  @indices_rejected[index_name] = "closed index can't snapshot"
@@ -224,28 +227,14 @@ module Elasticshelf
224
227
  end
225
228
  @indices = open_indices.join(',')
226
229
  # allow user to override this name:
227
- if blank? @snapshot_expired_indices_name
230
+ if blank? @snapshot_indices_expired_name
228
231
  @snapshot = (Time.now.utc.to_s.gsub(' ', '_') +
229
232
  "_#{@indices_prefix.chomp('*')}" +
230
233
  "_cutoff_#{@cutoff_days}_days").downcase
231
234
  else
232
- @snapshot = @snapshot_expired_indices_name
235
+ @snapshot = @snapshot_indices_expired_name
233
236
  end
234
237
  @results = snapshot_create
235
- # {"snapshot"=>{
236
- # "snapshot"=>"2014-04-02_19:23:35_utc_logstash-_cutoff_22_days",
237
- # "indices"=>[],
238
- # "state"=>"SUCCESS",
239
- # "start_time"=>"2014-04-02T19:23:35.715Z",
240
- # "start_time_in_millis"=>1396466615715,
241
- # "end_time"=>"2014-04-02T19:23:35.721Z",
242
- # "end_time_in_millis"=>1396466615721,
243
- # "duration_in_millis"=>6,
244
- # "failures"=>[],
245
- # "shards"=>{"total"=>0, "failed"=>0, "successful"=>0}
246
- # }}
247
- # @results['snapshot']['state'].downcase == 'success'
248
- # @results['snapshot']['failures']
249
238
  end
250
239
 
251
240
  def snapshot_create
@@ -295,30 +284,31 @@ module Elasticshelf
295
284
  end
296
285
 
297
286
  def snapshot_restore
298
- if blank? @indices
299
- @results = {"error" => "indices must be specified to restore from a snapshot"}
300
- return
301
- end
302
287
  try do
303
- # @example Restore from the `snapshot-1` snapshot
304
- # client.snapshot.restore repository: 'my-backups', snapshot: 'snapshot-1'
305
- # @example Restore a specific index under a different name
306
- # client.snapshot.restore repository: 'my-backups',
307
- # snapshot: 'snapshot-5',
308
- # body: {
309
- # rename_pattern: "^(.*)$",
310
- # rename_replacement: "restored_$1"
311
- # }
312
- @results = @client.snapshot.restore(
313
- repository: @repo,
314
- snapshot: @snapshot,
315
- wait_for_completion: @wait_for_completion,
316
- body: {
317
- indices: @indices,
318
- ignore_unavailable: true,
319
- include_global_state: false
320
- }
321
- )
288
+ if blank? @indices
289
+ # the whole snapshot:
290
+ @results = @client.snapshot.restore(
291
+ repository: @repo,
292
+ snapshot: @snapshot,
293
+ wait_for_completion: @wait_for_completion,
294
+ body: {
295
+ ignore_unavailable: @ignore_unavailable,
296
+ include_global_state: @include_global_state
297
+ }
298
+ )
299
+ else
300
+ # note: @indices must be a string of comma separated index names
301
+ @results = @client.snapshot.restore(
302
+ repository: @repo,
303
+ snapshot: @snapshot,
304
+ wait_for_completion: @wait_for_completion,
305
+ body: {
306
+ indices: @indices,
307
+ ignore_unavailable: @ignore_unavailable,
308
+ include_global_state: @include_global_state
309
+ }
310
+ )
311
+ end
322
312
  end
323
313
  end
324
314
 
@@ -1,4 +1,4 @@
1
1
  module Elasticshelf
2
2
  # Elasticshelf::VERSION
3
- VERSION = "0.0.4"
3
+ VERSION = "0.0.5"
4
4
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: elasticshelf
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.4
4
+ version: 0.0.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - cLee Smith