consul-templaterb 1.25.2 → 1.26.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (15) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +11 -0
  3. data/INTERNALS.md +2 -0
  4. data/README.md +2 -0
  5. data/TemplateAPI.md +175 -164
  6. data/docs/article-06_Template-based_discovery_with_consul-templaterb.md +123 -0
  7. data/docs/images/consul-templaterb.png +0 -0
  8. data/docs/images/consul-templaterb.svg +3 -0
  9. data/lib/consul/async/consul_endpoint.rb +9 -4
  10. data/lib/consul/async/consul_template.rb +44 -34
  11. data/lib/consul/async/vault_endpoint.rb +9 -4
  12. data/lib/consul/async/version.rb +1 -1
  13. data/samples/all_services.txt.erb +1 -1
  14. data/samples/all_services_multi_agents.txt.erb +69 -0
  15. metadata +6 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 939468d29eba6433fe6f2c93c1ca6eb8a2522d6827a7c6fa99e49415a1b2b589
4
- data.tar.gz: 052ccf9fce6c083e6925439d0e2505eecbeb2b80371f418dc8b9c7ad6f42c65f
3
+ metadata.gz: 79f77f92ae3811c48a823724fb4fb242b78fb78713de126cc035de4089a5e607
4
+ data.tar.gz: 3aa6c6f9c1aca03006963dd03ab30c3ba9ceabc4ff88c3a0d7fa8470a024b2d2
5
5
  SHA512:
6
- metadata.gz: 0bd42ba387e49d7355153d557bbc2f4723f5eae92ee733ed5e21f550ed6eda7c5b914f1a38bbfa569d7df3061c94d8c4da62e814ceb87d328ac8cfd9cb59c490
7
- data.tar.gz: d4675c40d58c9bafddefef6ce6e71e2a0bf61bafe256db7e3512f889416f3ed1cd5388a651d7334dd7afc88da660677a7b684afcabfaed077ef4e5a49dab9221
6
+ metadata.gz: ed5763f1c109d678a867dddbe77a2b4e8d0b9adde1663f80e9bb6db2812c104f84edcd9f5c929d8e1f4763e9b4fa3af138da87c58af1285d6f1b0c6f5924760d
7
+ data.tar.gz: af7a53d991a486b4050632576b36c2cb1fe5e899bee75ec73a2e6bd76bb18e350913e5703b69ee8a05aa51b2d64a64a60b8c17dcc1e071b8cac3544331de0600
data/CHANGELOG.md CHANGED
@@ -2,6 +2,17 @@
2
2
 
3
3
  ## (UNRELEASED)
4
4
 
5
+ NEW FEATURES:
6
+
7
+ ## 1.26.0 (March 5, 2020)
8
+
9
+ * Using `agent: http://vault_or_consul_agent:port>` on most methods will now override the agent
10
+ used to perform Consul queries. It might be useful for very large clusters or large WAN federations,
11
+ because you can perform some requests on some agents, and some others on other agenrs. Might
12
+ also be useful to federate data from preprod/prod for instance.
13
+ See [samples/all_services_multi_agents.txt.erb](samples/all_services_multi_agents.txt.erb) for an
14
+ example.
15
+
5
16
  ## 1.25.2 (February 29, 2020)
6
17
 
7
18
  BUGFIX:
data/INTERNALS.md CHANGED
@@ -12,6 +12,8 @@ the list of services registered in the cluster).
12
12
 
13
13
  For each endpoint, `consul-templaterb` performs a watch by getting information about the last transaction on this endpoints, storing it locally and watching at Consul to get updates about the data within this endpoint.
14
14
 
15
+ ![Main Loop](docs/images/consul-templaterb.svg)
16
+
15
17
  ## How it works
16
18
 
17
19
  There is a loop in the engine, every second, all templates are rendered using the `ERB` template engine.
data/README.md CHANGED
@@ -33,6 +33,8 @@ with fully [working examples](samples/).
33
33
  It also allows displaying a very nice and hi-performance HTML5 UI for Consul,
34
34
  see [consul-ui](samples/consul-ui) for details.
35
35
 
36
+ There is an article [docs/article-06_Template-based_discovery_with_consul-templaterb.md](docs/article-06_Template-based_discovery_with_consul-templaterb.md) summarizing why we developped this tool (also available on [medium](https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457)).
37
+
36
38
  ## Video introduction to consul-templaterb
37
39
 
38
40
  [![Introduction to Consul-templaterb](docs/images/consul-ui_001.png)](https://youtu.be/zLzrLGLLl4Q)
data/TemplateAPI.md CHANGED
@@ -7,6 +7,10 @@ annoted with `[]`.
7
7
  Most of them support the optional dc attribute to use data from another datacenter. If the `dc`
8
8
  attribute is not specified, the function will output data from the current datacenter.
9
9
 
10
+ Starting with version 1.26.0, most methods also support the
11
+ `agent: "http://address_of_vault_or_consul_agent:port"` to use another agent than the one specified in command line. It opens the way to use different agents/DC or to create templates federating several different not-linked clusters (ex: prod/preprod).
12
+ Example on how using it: [samples/all_services_multi_agents.txt.erb](samples/all_services_multi_agents.txt.erb).
13
+
10
14
  To ease template development, `consul-templaterb` supports HOT reload of templates, thus it is possible to
11
15
  develop the templates interactively. While developing, it is possible to use the switch `--hot-reload=keep`,
12
16
  thus the application will display a warning if the template is invalid and won't stop
@@ -44,154 +48,162 @@ to avoid having to write .result in all templates, some shortcuts have been adde
44
48
 
45
49
  Also available for all results:
46
50
 
47
- ### Common methods available for all objects
48
-
49
- * .all?
50
- * .any?
51
- * .assoc
52
- * .chunk
53
- * .chunk_while
54
- * .class
55
- * .clear
56
- * .collect
57
- * .collect_concat
58
- * .compact
59
- * .count
60
- * .cycle
61
- * .detect
62
- * .dig
63
- * .display
64
- * .drop
65
- * .drop_while
66
- * .dup
67
- * .each
68
- * .each_cons
69
- * .each_entry
70
- * .each_slice
71
- * .each_with_index
72
- * .each_with_object
73
- * .empty?
74
- * .entries
75
- * .enum_for
76
- * .eql?
77
- * .equal?
78
- * .extend
79
- * .fetch
80
- * .find
81
- * .find_all
82
- * .find_index
83
- * .first
84
- * .flat_map
85
- * .flatten
86
- * .grep
87
- * .grep_v
88
- * .group_by
89
- * .hash
90
- * .include?
91
- * .index
92
- * .inject
93
- * .inspect
94
- * .is_a?
95
- * .itself
96
- * .keep_if
97
- * .kind_of?
98
- * .lazy
99
- * .length
100
- * .map
101
- * .max
102
- * .max_by
103
- * .member?
104
- * .min
105
- * .min_by
106
- * .minmax
107
- * .minmax_by
108
- * .nil?
109
- * .none?
110
- * .object_id
111
- * .one?
112
- * .partition
113
- * .pp
114
- * .rassoc
115
- * .reduce
116
- * .reject
117
- * .replace
118
- * .reverse_each
119
- * .select
120
- * .shift
121
- * .size
122
- * .slice
123
- * .slice_after
124
- * .slice_before
125
- * .slice_when
126
- * .sort
127
- * .sort_by
128
- * .sum
129
- * .take
130
- * .take_while
131
- * .tap
132
- * .to_a
133
- * .to_enum
134
- * .to_h
135
- * .to_s
136
- * .uniq
137
- * .values_at
138
- * .yield_self
139
- * .zip
140
-
141
- ### Methods available for Array objects
142
-
143
- * .append
144
- * .at
145
- * .bsearch
146
- * .bsearch_index
147
- * .combination
148
- * .concat
149
- * .each_index
150
- * .fill
151
- * .insert
152
- * .join
153
- * .last
154
- * .pack
155
- * .permutation
156
- * .pop
157
- * .prepend
158
- * .product
159
- * .push
160
- * .repeated_combination
161
- * .repeated_permutation
162
- * .reverse
163
- * .rindex
164
- * .rotate
165
- * .sample
166
- * .shuffle
167
- * .to_ary
168
- * .transpose
169
- * .unshift
170
-
171
- ### Methods available for hash objects
172
-
173
- * .compare_by_identity
174
- * .compare_by_identity?
175
- * .each_key
176
- * .each_pair
177
- * .each_value
178
- * .fetch_values
179
- * .has_key?
180
- * .has_value?
181
- * .invert
182
- * .key
183
- * .key?
184
- * .keys
185
- * .merge
186
- * .rehash
187
- * .store
188
- * .to_hash
189
- * .to_proc
190
- * .transform_keys
191
- * .transform_values
192
- * .update
193
- * .value?
194
- * .values
51
+ <details id="common-methods-available-for-all-objects">
52
+ <summary>Common methods available for all objects</summary>
53
+ <div class="samples">
54
+ <ul>
55
+ <li>.all?</li>
56
+ <li>.any?</li>
57
+ <li>.assoc</li>
58
+ <li>.chunk</li>
59
+ <li>.chunk_while</li>
60
+ <li>.class</li>
61
+ <li>.clear</li>
62
+ <li>.collect</li>
63
+ <li>.collect_concat</li>
64
+ <li>.compact</li>
65
+ <li>.count</li>
66
+ <li>.cycle</li>
67
+ <li>.detect</li>
68
+ <li>.dig</li>
69
+ <li>.display</li>
70
+ <li>.drop</li>
71
+ <li>.drop_while</li>
72
+ <li>.dup</li>
73
+ <li>.each</li>
74
+ <li>.each_cons</li>
75
+ <li>.each_entry</li>
76
+ <li>.each_slice</li>
77
+ <li>.each_with_index</li>
78
+ <li>.each_with_object</li>
79
+ <li>.empty?</li>
80
+ <li>.entries</li>
81
+ <li>.enum_for</li>
82
+ <li>.eql?</li>
83
+ <li>.equal?</li>
84
+ <li>.extend</li>
85
+ <li>.fetch</li>
86
+ <li>.find</li>
87
+ <li>.find_all</li>
88
+ <li>.find_index</li>
89
+ <li>.first</li>
90
+ <li>.flat_map</li>
91
+ <li>.flatten</li>
92
+ <li>.grep</li>
93
+ <li>.grep_v</li>
94
+ <li>.group_by</li>
95
+ <li>.hash</li>
96
+ <li>.include?</li>
97
+ <li>.index</li>
98
+ <li>.inject</li>
99
+ <li>.inspect</li>
100
+ <li>.is_a?</li>
101
+ <li>.itself</li>
102
+ <li>.keep_if</li>
103
+ <li>.kind_of?</li>
104
+ <li>.lazy</li>
105
+ <li>.length</li>
106
+ <li>.map</li>
107
+ <li>.max</li>
108
+ <li>.max_by</li>
109
+ <li>.member?</li>
110
+ <li>.min</li>
111
+ <li>.min_by</li>
112
+ <li>.minmax</li>
113
+ <li>.minmax_by</li>
114
+ <li>.nil?</li>
115
+ <li>.none?</li>
116
+ <li>.object_id</li>
117
+ <li>.one?</li>
118
+ <li>.partition</li>
119
+ <li>.pp</li>
120
+ <li>.rassoc</li>
121
+ <li>.reduce</li>
122
+ <li>.reject</li>
123
+ <li>.replace</li>
124
+ <li>.reverse_each</li>
125
+ <li>.select</li>
126
+ <li>.shift</li>
127
+ <li>.size</li>
128
+ <li>.slice</li>
129
+ <li>.slice_after</li>
130
+ <li>.slice_before</li>
131
+ <li>.slice_when</li>
132
+ <li>.sort</li>
133
+ <li>.sort_by</li>
134
+ <li>.sum</li>
135
+ <li>.take</li>
136
+ <li>.take_while</li>
137
+ <li>.tap</li>
138
+ <li>.to_a</li>
139
+ <li>.to_enum</li>
140
+ <li>.to_h</li>
141
+ <li>.to_s</li>
142
+ <li>.uniq</li>
143
+ <li>.values_at</li>
144
+ <li>.yield_self</li>
145
+ <li>.zip</li>
146
+ </ul></div></details>
147
+
148
+ <details>
149
+ <summary id="methods-available-for-array-objects">Methods available for Array objects</summary>
150
+ <div class="samples">
151
+ <ul>
152
+ <li>.append</li>
153
+ <li>.at</li>
154
+ <li>.bsearch</li>
155
+ <li>.bsearch_index</li>
156
+ <li>.combination</li>
157
+ <li>.concat</li>
158
+ <li>.each_index</li>
159
+ <li>.fill</li>
160
+ <li>.insert</li>
161
+ <li>.join</li>
162
+ <li>.last</li>
163
+ <li>.pack</li>
164
+ <li>.permutation</li>
165
+ <li>.pop</li>
166
+ <li>.prepend</li>
167
+ <li>.product</li>
168
+ <li>.push</li>
169
+ <li>.repeated_combination</li>
170
+ <li>.repeated_permutation</li>
171
+ <li>.reverse</li>
172
+ <li>.rindex</li>
173
+ <li>.rotate</li>
174
+ <li>.sample</li>
175
+ <li>.shuffle</li>
176
+ <li>.to_ary</li>
177
+ <li>.transpose</li>
178
+ <li>.unshift</li>
179
+ </ul></div></details>
180
+
181
+ <details id="Methods-available-for-hash-objects">
182
+ <summary>Methods available for hash objects</summary>
183
+ <ul>
184
+ <li>.compare_by_identity</li>
185
+ <li>.compare_by_identity?</li>
186
+ <li>.each_key</li>
187
+ <li>.each_pair</li>
188
+ <li>.each_value</li>
189
+ <li>.fetch_values</li>
190
+ <li>.has_key?</li>
191
+ <li>.has_value?</li>
192
+ <li>.invert</li>
193
+ <li>.key</li>
194
+ <li>.key?</li>
195
+ <li>.keys</li>
196
+ <li>.merge</li>
197
+ <li>.rehash</li>
198
+ <li>.store</li>
199
+ <li>.to_hash</li>
200
+ <li>.to_proc</li>
201
+ <li>.transform_keys</li>
202
+ <li>.transform_values</li>
203
+ <li>.update</li>
204
+ <li>.value?</li>
205
+ <li>.values</li>
206
+ </ul></div></details>
195
207
 
196
208
  ## coordinate
197
209
 
@@ -203,7 +215,7 @@ The coordinate object allow to interact with the coordinates of DCs and nodes as
203
215
  [List the Wan Coordinates](https://www.consul.io/api/coordinate.html#read-wan-coordinates) from local DC to
204
216
  other DCs. If dc is set, it will perform the same operation but from another DC.
205
217
 
206
- ### coordinate.nodes([dc: datacenter])
218
+ ### coordinate.nodes([dc: datacenter], [agent: consul_agent_address])
207
219
 
208
220
  [Read all LAN nodes coordinates](https://www.consul.io/api/coordinate.html#read-lan-coordinates-for-all-nodes).
209
221
  If If dc is set, it will perform the same operation but for another DC.
@@ -213,7 +225,7 @@ If If dc is set, it will perform the same operation but for another DC.
213
225
  Computes the rtt between 2 nodes returned by `coordinate.nodes` or `coordinate.datacenters`. A re-implementation of Golang sample code
214
226
  [Working with Coordinates](https://www.consul.io/docs/internals/coordinates.html#working-with-coordinates).
215
227
 
216
- ## datacenters()
228
+ ## datacenters([agent: consul_agent_address])
217
229
 
218
230
  [Get the list of datacenters as string array](https://www.consul.io/api/catalog.html#list-datacenters).
219
231
 
@@ -233,7 +245,7 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
233
245
  </div>
234
246
  </details>
235
247
 
236
- ## services([dc: datacenter], [tag: tagToFilterWith])
248
+ ## services([dc: datacenter], [tag: tagToFilterWith], [agent: consul_agent_address])
237
249
 
238
250
  [List the services matching the optional tag filter](https://www.consul.io/api/catalog.html#list-services),
239
251
  if tag is not specified, will match all the services. Note that this endpoint performs client side tag
@@ -270,7 +282,7 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
270
282
  </div>
271
283
  </details>
272
284
 
273
- ## service(serviceName, [dc: datacenter], [tag: tagToFilterWith], [passing: true])
285
+ ## service(serviceName, [dc: datacenter], [tag: tagToFilterWith], [passing: true], [agent: consul_agent_address])
274
286
 
275
287
  [List the instances](https://www.consul.io/api/health.html#list-nodes-for-service) of a service having the given
276
288
  optional tag. If no tag is specified, will return all instances of service. By default, it will return all the
@@ -342,7 +354,7 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
342
354
  </div>
343
355
  </details>
344
356
 
345
- ## nodes([dc: datacenter])
357
+ ## nodes([dc: datacenter], [agent: consul_agent_address])
346
358
 
347
359
  [List all the nodes of selected datacenter](https://www.consul.io/api/catalog.html#list-nodes). No filtering is
348
360
  applied.
@@ -363,20 +375,20 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
363
375
  </div>
364
376
  </details>
365
377
 
366
- ## node(nodeNameOrId, [dc: datacenter])
378
+ ## node(nodeNameOrId, [dc: datacenter], [agent: consul_agent_address])
367
379
 
368
380
  [List all the services of a given Node](https://www.consul.io/api/catalog.html#list-services-for-node) using its
369
381
  name or its ID. If DC is specified, will lookup for given node in another datacenter.
370
382
 
371
- ## checks_for_node(name, dc: nil, passing: false, tag: nil)
383
+ ## checks_for_node(name, dc: nil, passing: false, tag: nil, [agent: consul_agent_address])
372
384
 
373
385
  [Find all the checks](https://www.consul.io/api/health.html#list-checks-for-node) of a given node name.
374
386
 
375
- ## checks_for_service(name, dc: nil, passing: false, tag: nil)
387
+ ## checks_for_service(name, dc: nil, passing: false, tag: nil, [agent: consul_agent_address])
376
388
 
377
389
  [Find all the checks](https://www.consul.io/api/health.html#list-checks-for-service) of a given service.
378
390
 
379
- ## kv(name, [dc: nil], [keys: false], [recurse: false])
391
+ ## kv(name, [dc: nil], [keys: false], [recurse: false], [agent: consul_agent_address])
380
392
 
381
393
  [Read keys from KV Store](https://www.consul.io/api/kv.html#read-key). It can be used for both listing the keys and
382
394
  getting the values. See the file in samples [keys.html.erb](samples/keys.html.erb) for a working example.
@@ -451,7 +463,7 @@ value123 : <%= result.get_decoded('/my/multiple/values/value123') %>
451
463
  Since `kv('/my/multiple/values', recurse: true)` will retrieve all values at once, it might be more
452
464
  efficient in some cases than retrieving all values one by one.
453
465
 
454
- ## agent_members(wan: false)
466
+ ## agent_members(wan: false, [agent: consul_agent_address])
455
467
 
456
468
  [Get the Serf information](https://www.consul.io/api/agent.html#list-members) from Consul Agent point of view.
457
469
  This is a list of Serf information containing serf information. This information is not consistent and should be used with care, most of the time, you should prefer `nodes()`.
@@ -470,13 +482,13 @@ The returned value is an array containing the following objects containing the f
470
482
 
471
483
  See [samples/members.json.erb](samples/members.json.erb) for example of usage.
472
484
 
473
- ## agent_metrics()
485
+ ## agent_metrics([agent: consul_agent_address])
474
486
 
475
487
  [Get the metrics of Consul Agent](https://www.consul.io/api/agent.html#view-metrics). Since this endpoint does
476
488
  not support blocking queries, data will be refreshed every few seconds, but will not use blocking queries
477
489
  mechanism.
478
490
 
479
- ## agent_self()
491
+ ## agent_self([agent: consul_agent_address])
480
492
 
481
493
  [Get the configuration of Consul Agent](https://www.consul.io/api/agent.html#read-configuration).
482
494
  Since this endpoint does not support blocking queries, data will be refreshed every few seconds,
@@ -542,7 +554,7 @@ consul-templaterb --template "source.html.erb:dest.html:reload_command:params.ya
542
554
 
543
555
  See [samples/consul-ui/consul-services-ui.html.erb](samples/consul-ui/consul-services-ui.html.erb) for example of usage.
544
556
 
545
- ## secrets(prefix)
557
+ ## secrets(prefix, [agent: vault_agent_address])
546
558
 
547
559
  It requires that a Vault token is given either in parameter or in environment variable
548
560
  The [policies](https://www.vaultproject.io/docs/concepts/policies.html) should be properly set.
@@ -568,7 +580,7 @@ Full example: [samples/vault-ldap.txt.erb](samples/vault-ldap.txt.erb)
568
580
  </div>
569
581
  </details>
570
582
 
571
- ## secret(path, [data = nil])
583
+ ## secret(path, [data = nil], [agent: vault_agent_address])
572
584
 
573
585
  It requires that a Vault token is given either in parameter or in environment variable
574
586
  The [policies](https://www.vaultproject.io/docs/concepts/policies.html) should be properly set.
@@ -689,5 +701,4 @@ Here are templates rendered by consul-templaterb:
689
701
  ```
690
702
 
691
703
  </div>
692
- </details>
693
-
704
+ </details>
data/docs/article-06_Template-based_discovery_with_consul-templaterb.md ADDED
@@ -0,0 +1,123 @@
1
+ # Template-based discovery with consul-templaterb
2
+
3
+ Source of article https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
4
+
5
+ # How and why Criteo built a template-based system for Consul
6
+
7
+ [consul-templaterb](https://github.com/criteo/consul-templaterb/) is an OpenSource Software (OSS) written in Ruby that allows you to very quickly create UIs for Consul or configuration files for your beloved software to interact with Consul. This article describes why we wrote it and why you might be interested in using it.
8
+
9
+ # The context
10
+
11
+ At the end of 2017, Consul started becoming a very important piece in Criteo’s infrastructure as it was responsible for dealing with the discovery of all major business applications at Criteo. At that time, Criteo had more than 25,000 servers on production and some datacenters had more than 5,000 agents per datacenter. At the same time, we started building our Load-Balancer stack based on the Consul state. So it became more and more important to make it efficient, stable and expand it with new features.
12
+
13
+ Criteo wanted to move towards HAProxy for its load-balancing stack, thus we started generating some configuration for HAProxy with all HTTP services with the template-based configuration generator consul-template. However, things were a bit more complicated than expected because the program consul-template was using almost all bandwidth available just to discover all services. Moreover, the system was constantly updating HAProxy configuration.
14
+
15
+ As explained in our first article “[Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)”, Consul includes a notification system to ensure your systems are always up-to-date with the content of the discovery. When this system starts to push too many notifications, on large services, the amount of data sent to all agents can be quite impressive (several megabytes/s for a single service having hundreds of instances).
16
+
17
+ Thus, we started investigating the issue and figured out that it was linked to a race condition in large clusters that caused all listeners of Consul to be notified continuously. This issue was eventually solved by our [Pull-Request #3899](https://github.com/hashicorp/consul/pull/3899) (and quite a few others after that), but the investigation showed that it was quite hard to investigate the root cause of such issues.
18
+
19
+ Therefore, I started a simple tool to watch all the services matching some patterns and report the bandwidth usage, but I also wanted to prove it would be possible to generate all configurations for HAProxy with an optimized tool, thus having a templating mechanism. The [Pull Request #1066](https://github.com/hashicorp/consul-template/pull/1066) to fix the issue on the consul-template itself was not yet considered (and it was hard to have something better), so I decided to implement it in Ruby with asynchronous I/Os (Input/Outputs) because I was very frustrated by the Go templating language used in consul-template and was in love with ERB templates.
20
+
21
+ # The goals
22
+
23
+ From scratch, I wanted to take the various shortcoming of consul-template into account:
24
+
25
+ * The ability to generate high-level code to generate various formats (YAML, JSON, XML): it is dangerous, error-prone and painful to generate JSON or XML content using text templating. People fighting with YAML indentation will tell you that.
26
+ * The ability to use a Turing complete and effective language to perform simple transformation such as sorting (Consul by default has some rules to sort the output, but it might not be predictable, for instance, based on the round-trip between servers).
27
+ * Being able to push real-life optimizations (for instance, we don’t care about being notified immediately when a change occurs, we want to be notified once before dumping the configuration file, not 100 times without taking it into account), based on the type of query performed.
28
+ * Being able to scale nicely with Consul and avoid overloading the cluster for nothing in case of bug (for instance, we use a penalty for services changing too much, we rate-limit the notifications).
29
+ * Being very simple to use, hiding all the complexity to the template creator and using optimizations to reduce the number of calls to Consul, meaning being as purely functional as possible by hiding all I/O operations and taking decisions based on our experience with Consul internals.
30
+ * Be evolutive: When some functions or fields are missing in a consul-template, you have to do a pull request on consul-template to add your new fields/methods, then you can work. It would be nice if any field of endpoints would be supported natively (so new versions of Consul can be used with an old version of our tool).
31
+ * Support for hot-reload of templates, and nice error messages with the precise line when an error occurs, so writing templates should be very fast and efficient.
32
+ * Have most of the consul-template features including babysitting of processes, spawn commands when files do change…
33
+
34
+ # Internals: make your template engine in Ruby
35
+
36
+ ## At startup
37
+
38
+ The engine collects all template sources and computes the destinations. Each template and its destination also register some commands to run when the destination does change. Those objects save the last time the template has been read (to allow hot-reload) and the last binary content associated with them (to detect binary changes).
39
+
40
+ ## The main loop
41
+
42
+ First, the main loop is working on pending events. Those events are in fact the processing of the I/O operations querying the Consul agent. But every second by default, the main loop also evaluates templates which are Ruby code. When this is just regular code, this is evaluated normally, but when the code called is part of the [I/O API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md), the code does the following:
43
+
44
+ 1. Checks in a registry if this method has been called with the same parameters, if not, create it, mark it as dirty (so the registry knows we are waiting for data from Consul), launch the request asynchronously to Consul, store it in the registry. This will be translated into events that will be run as events in the main loop.
45
+ 2. Return the object stored within the registry. By default, this object returns the empty object or collection according to its type. When the request finally gets a response from Consul, the object is marked as “non-dirty” and the result is put within the registry, so the next evaluation returns the content of the requests.
46
+
47
+ Once all code has been evaluated and the result has been stored in memory, the main loop checks whether the template is using “dirty” objects in the registry. Having at least one dirty object means that some requests are still in flight and we don’t have the result yet, so basically means the template, while evaluated, has been evaluated with fake (empty) data and is not yet ready.
48
+
49
+ However, if the template is not using any dirty object it means we did receive all the data and we can render the template on disk. We first check if our last rendering was different. If the new rendering phase was not different, nothing did change, however, if the data rendered is different, the file is stored to disk and commands might be evaluated (to notify a program to reload for instance).
50
+
51
+ All the I/O are done asynchronously (using [eventmachine](https://github.com/eventmachine/eventmachine)) but performed within the main loop, the one also performing the rendering, so, there is no possibility of threading issues.
52
+
53
+ ![Main loop behavior, in purple, the async fetch, in yellow, the rendering](images/consul-templaterb.png)
54
+
55
+ ## Handling of I/O
56
+
57
+ The Input/Outputs are handled very carefully with regards to what we explained in our previous article “[Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)” and adds a few tricks:
58
+
59
+ * When a given endpoint is changing too fast with any real change (the data is the same as last call), a penalty is applied, so the endpoint won’t be queried before a few seconds (this allows to work well even with old Consul versions with patch [#3899](https://github.com/hashicorp/consul/pull/3899)).
60
+ * Some endpoints [are configured specifically](https://github.com/criteo/consul-templaterb/blob/master/bin/consul-templaterb#L61) to avoid too many calls (for instance, we don’t consider we add a new DC every 30 seconds).
61
+ * Some specific behavior exists for non-existing service (was causing lots of issues with Prometheus for instance, see [Prometheus PR #3814](https://github.com/prometheus/prometheus/pull/3814)), so we will work on the old Consul version with our [PR #4810](https://github.com/hashicorp/consul/pull/4810).
62
+ * By default, when some errors occur, try to limit the number of calls to avoid increasing the pressure on the cluster.
63
+ * Collect statistics that can be used in your templates. For instance, the [Prometheus exporter uses it to know whether a service is unstable or not](https://github.com/criteo/consul-templaterb/blob/master/samples/metrics.erb#L88).
64
+
65
+ ## The hard part about async I/Os with Ruby
66
+
67
+ This program was my first project dealing with complex code with important performance issues. I first discovered that Ruby async I/O code is really hard and most libraries are not dealing with it properly. Many libraries just lie (by creating threads and pretending to be async). The only library I found handling it correctly was [eventmachine](https://github.com/eventmachine/eventmachine/).
68
+
69
+ While the library is petty mature (the project is quite old, 9 years old), I had lots of issues very quickly. I spent quite some time making it work on my targeted operating systems (Linux, Mac OS, Windows).
70
+
71
+ * On Windows, installation is painful (see https://github.com/criteo/consul-templaterb/#quick-install-on-windows) and due to limitation into the Ruby runtime, it does not support more than 2048 file descriptors with native implementation (while this works with Linux Subsystem on Windows 10).
72
+ * On all Operating Systems, opening lots of HTTP connections concurrently led to crashes (see https://github.com/igrigorik/em-http-request/issues/315, [#604](https://github.com/eventmachine/eventmachine/issues/604), [#770](https://github.com/eventmachine/eventmachine/issues/770), and [#824](https://github.com/eventmachine/eventmachine/issues/824)) that took me lots of time to debug (including native C code debugging), quite frustrating for such mature library (the good news is that eventmachine is now fixed thanks to [consul-templaterb](https://github.com/criteo/consul-templaterb/)).
73
+
74
+ While those problems are now solved and you can use it properly, it took me weeks to solve.
75
+
76
+ Furthermore, the lack of HTTP/2 is a bit worrisome today as more and more systems allow it (it would avoid the lack of file descriptors on very large clusters).
77
+
78
+ ## Some fun features
79
+
80
+ * All templates in the repository are unit tested for each release of consul-templaterb, so all samples are working on your local installation.
81
+ * You can generate lots of formats: [Services in XML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.xml.erb), [JSON](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb), [YAML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb#L58) using Hash and just dumping in the right format!
82
+ * You can load templates from Consul Key/Value store, so you can change templates dynamically from Consul K/V Store (we use this feature in for our Prometheus configurations for instance as described in “[Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)”).
83
+ * You can also query Vault / some JSON APIs very easily.
84
+ * You can have a very high-performance UI (see [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)) easily for your own use-cases. This UI is generated in real-time as static files and can be served by your favorite webserver (we do use nginx on our side), so it scales indefinitely if you have lots of users.
85
+ * Huge Performance gap with consul-template: in version 1.0 of Consul, consul-template was using 800Mb/s to scrape all services in one of large DCs, consul-templaterb is using less than 100kb/s (800x reduction!) to generate the full UI of Consul and fetch all services in our largest DC now.
86
+ * Good abstraction with impressive performance compared to other projects such as [consult](https://github.com/veracross/consult) (no need to deal with kind of “magic TTL”) or consul-template itself: the APIs are simpler, you have to deal less with Consul specific optimizations and the performance is better.
87
+
88
+ # Usages at Criteo (non-exhaustive)
89
+
90
+ We use this executable for many things at Criteo (see “Inversion of Control with Consul”):
91
+
92
+ * Automatic Alerting / Prometheus configuration
93
+ * [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)s and its Consul Timeline (keep track of the history of all changes for all services)
94
+ * List of assets per DC/Racks
95
+ * Monitoring of services/racks
96
+ * Ownership enforcement
97
+ * Live generation of configuration for several programs/services
98
+
99
+ # Try it quickly
100
+
101
+ If you don’t want to bother configuring and tuning it, you can try it very quickly with our docker images: In 2 minutes (download included), you have a full scalable UI with Consul, serving static files with nginx and keeping a history of all changes on your services live.
102
+
103
+ ![Consul UI in action, scalable UI with excellent performance](images/consul-ui_001.png)
104
+
105
+ # More on the subject
106
+
107
+ * The [consul-templaterb API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md) to write templates, with links to real-world examples in the samples directory.
108
+ * [INTERNALS.md](../INTERNALS.md) on Github (describes the objects in the code)
109
+ * Slides at HashiConf ’19: “[Consul Templates on Steroids](https://fr.slideshare.net/PierreSouchay/2019-hashiconf-consultemplaterb)”
110
+ * A video introduction of consul-templaterb:
111
+
112
+ [![Quick presentation of consul-templaterb](images/consul-ui_001.png)](https://youtu.be/zLzrLGLLl4Q)
113
+
114
+ # Other articles from the Discovery team
115
+
116
+ * [Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)
117
+ * [Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)
118
+ * [Anatomy of a bug: When Consul has too much to deliver for the big day](https://medium.com/criteo-labs/anatomy-of-a-bug-when-consul-has-too-much-to-deliver-for-the-big-day-4904d19a46a4)
119
+ * [Inversion of Control for the Infrastructure with Consul](https://medium.com/criteo-labs/inversion-of-control-for-the-infrastructure-with-consul-b894877b33a4)
120
+ * [Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)
121
+
122
+ Pierre Souchay, 2020-03-03
123
+ Also published on https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
data/docs/images/consul-templaterb.svg ADDED
@@ -0,0 +1,3 @@
1
+ <?xml version="1.0" encoding="UTF-8"?>
2
+ <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
3
+ <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="896px" height="455px" viewBox="-0.5 -0.5 896 455" content="&lt;mxfile host=&quot;www.draw.io&quot; modified=&quot;2020-03-02T13:27:25.463Z&quot; agent=&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:73.0) Gecko/20100101 Firefox/73.0&quot; etag=&quot;Qxj7PV3jLHrlLdg27esA&quot; version=&quot;12.7.9&quot; type=&quot;device&quot;&gt;&lt;diagram name=&quot;Page-1&quot; id=&quot;58cdce13-f638-feb5-8d6f-7d28b1aa9fa0&quot;&gt;7VzZkps4FP0aP3YXIDY/9pqpTDI1k04myVNKBtlmgpEL5HY7Xz8SSCwSm21wO4n7IUELAt977rmLZE/A3erlTQzXy/fYR+HE0PyXCbifGIauG4D+x3p2WY9jT7OORRz4fFLR8RT8QLxT472bwEdJZSLBOCTButrp4ShCHqn0wTjG2+q0OQ6rT13DBVI6njwYqr2fA58sxefStGLgDxQslvzRrsUHZtD7vojxJuLPmxhgnv5lwyso1uLzkyX08bbUBR4m4C7GmGRXq5c7FDLZCrFl9z02jObvHaOI9LkBuMZsqs8sS9dnvjZ3rvgKzzDcIPER0hclOyEcugLVA23cLskqpH06vaQfZM3GEwJj8kQgYePzIAzvcIjj9EagpX9sMonxd1Qamc/5SCo65BeLZtJhQx5eBR6/DuEMhbe5rMVKEY7Sx+KIPMJVEDL0/YtiH0aQd3Oo6QZv170dDINFRPs8KkNEB2/9IKYgCzDrTPCGqfBWFbSQHIoJeil1ccG/QXiFSLyjU8Qo4CDYibaVtbcF5MSUZQlsog9ykC/ylQtF0wuu6556t2v0bock/fTP9HLBLj8gqpyYTvuIVuuQ6ZjPoY8rTau5cxvQT0T5ABLIwBR7iZg0i+Xb5MUk+FUhsl0GBD2tocdGt5SNqrCEscdVbpgqIJlheu2APBMIDgA3w5TgJjioBLe8r4w3ewi8eT6yfei7tgl1B8IrU6UVn1Iwb+KYLPECRzB8KHrLem2XPoXpDfMCbKU1irIeLne3Vd099Sb0E6I5wyh6CcgXhotri7e+cpSw6/uXcmPHG40apQwTe1wm2nL+58dvH9/Opt/evFt9fvjw9p/vVzr3pZRpF0hwQoM9M6G2giNG1I6D56rfq9N0eiuVKtyVJqxxEJGktPLfrKPA3NSoYs6RvJE0HWht0+lF9vyGZ+kSvnUJt5lg+V0tb2FKL21a0kKZ4JWFUhvIRXsYDU8VGi6oVoswSWMe2oEIZcALMw7DjI72msyoRGBaGwaCJDVa6O8u6h9I/aZjVdRvuK/pGK1Wx4jCGd7+HD4xopL4Ijwfa3wtPCRrFm4xbQm/KHypXvakuV89wpc2cW3Zjbb624F9qeJ38iRS+J2p1cuBHeB3DAhmnmFp0Jtrtu74Qha/Rjh2GBx0FQ7jR1V9aUJRmPASv4bG2sjiMG2eR/xrypGF1R4Ad8zviIBNp/7uEQJXFY/gODzmKZRK+wIbuoqNM0Rxi1sb0nkJ++/0XuZpnBewq9gDZr/saxDsmUrE/BceCI6VjF6vUFWBTpHtnxkcGyHXBtRecKyPk3o4UAHbseFoTSUqtKVYakwqbA/he2OvJgJucZMX7HVhb5wwft+QwJaR2RESdMyvhgQHwLdNnO27Ae+DJAmiBZ2HY1Gd6rkdMEfEW7IKlrZD/e+KUYYDKunHvpsFS7yabZLuesikvmhQg8aWOkJVU666naPX7efoQ2zo1KrRUNT4FSWKjOgnJlJxSK6yMLkEHgxv+MAq8P2UtqhKgh9wli7FLJijn65r3U6se7bWhuCEE8Uwcral+mzdvplZI2djLDGDPtbyyCA/ybfAYLKLvN7In8EktRYc0X9CmJD0lV7OYQ8N6b6FnDoXMrUdAO3J2KXChpLgkaZcLQmCaY0tj1USbAvjSxj7HAdpRRhvyHpDxlTxz1YNPkr18japccrNgFrVO2OkMnnwVZPKnGF96IhwUor3DosvzR7xpT1OJmNKuxNGv6pw90LT0apD9TGl28dNflr7+5wrybxpz8mtPndvExMnnmBmCLfeJn7O+ba45R3Ga975HyJkx1HOwiLJTgtr1J2qPV5rAHRgnSsPxQHVE4oZ1bPYnN7cwQHtlpUvOLxJuScwqaN4V8XrfRa75dnLJaray7W6r7jPWqvhI3e9fvea9SA0AFQaqM9lnVHcqymBEshZYl/3CqTsX1mowb3uWz+SC+0mjwca36t9/kj1I/Uc043PEthVXjMqijmXzGUgepU0fdJjTPU2qx5lymAQoW1RCqGBEPep5xWFKds7Q8df+qk4tu7EaL3CQCvCtGvNAKACsitBe0fysCOl3QC49GGHUbFr2dVAw6QvXv4DvZi5WFdMxPN5gkZJjgy14n6zXoepg6P5UBisglFLPL9DvGlar13EM4zjAs7zCh37E1Q374yzC6xLJY++hxKUhQy9I0ocKLhTXrjjwHzH/HGCO0Pd73hi3/oSXjyL7zIXf5Io79ckL/lAzatvQRhHfmfnki4PEMoZffPlhkhu4HTZBAdWo0/FqM6e6XLH/JEY9cgTQ2dmIwec4G/bijrRFk/vYGUcy9ItySDkw7t9LUteyOx59G1fy7KA9Bz3FJbiKLHHA8V2nJ4IkiOMvmd/9g8nBvDvhvRtD1BzXMitce/uaO7dHci9N+8zX9x7Jwk5PUnIPW/3DqRyjrLQQCRkSKcjgX6KBEitbtfsNrMj6L22msc9f9dgYMUPk7SwGf95Fb7+JD8KU/l9kWYuaeS+K+1a10XQcyhkxy/KiXOZIyY82rVz7t8i+GkocaxvtmgNifi5pjy6dHjxJCkMUAvYpz9r/CpcB7TJWXMdbRa/J5VNL360Czz8Dw==&lt;/diagram&gt;&lt;/mxfile&gt;"><defs/><g><ellipse cx="60" cy="89" rx="11" ry="11" fill="#000000" stroke="#ff0000" transform="rotate(90,60,89)" pointer-events="all"/><rect x="155" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 156px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Render Template</div><div>with data srcs<br /></div></div></div></div></foreignObject><text x="215" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Render Template...</text></switch></g><path d="M 835 59 L 835 19 Q 835 9 825 9 L 225 9 Q 215 9 215 19 L 215 56.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 210.5 48.88 L 215 57.88 L 219.5 48.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="615" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 616px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Template not completed</div></div></div></foreignObject><text x="675" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Template not complet...</text></switch></g><rect x="390" y="219" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 249px; margin-left: 391px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Template is ready</div></div></div></foreignObject><text x="450" y="253" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Template is ready</text></switch></g><path d="M 735 89 L 745 89 Q 755 89 763.88 89 L 772.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 764.88 93.5 L 773.88 89 L 764.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 75 89 L 152.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 144.88 93.5 L 153.88 89 L 144.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 275 89 L 305 89 Q 315 89 325 89 L 382.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 374.88 93.5 L 383.88 89 L 374.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 390 249 L 342.5 249 Q 332.5 249 332.5 239 L 332.5 199 Q 332.5 189 322.5 189 L 277.24 189" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 285.12 184.5 L 276.12 189 L 285.12 193.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 450 154 L 450 216.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 445.5 208.88 L 450 217.88 L 454.5 208.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe flex-start; width: 1px; height: 1px; padding-top: 187px; margin-left: 452px;"><div style="box-sizing: border-box; font-size: 0; text-align: left; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">No</div></div></div></foreignObject><text x="452" y="191" fill="#000000" font-family="Verdana" font-size="12px">No</text></switch></g><path d="M 515 89 L 595 89 Q 605 89 608.88 89 L 612.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 604.88 93.5 L 613.88 89 L 604.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 450 24 L 515 89 L 450 154 L 385 89 Z" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 128px; height: 1px; padding-top: 89px; margin-left: 386px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Missing or not</div><div>fetched yet</div><div>resources?</div></div></div></div></foreignObject><text x="450" y="93" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Missing or not...</text></switch></g><rect x="535" y="74" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 84px; margin-left: 555px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Yes</div></div></div></foreignObject><text x="555" y="88" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Yes</text></switch></g><rect x="390" y="334" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 364px; margin-left: 391px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Fetch data async</div><div>based on last idx<br /></div></div></div></div></foreignObject><text x="450" y="368" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Fetch data async...</text></switch></g><rect x="155" y="159" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 189px; margin-left: 156px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Write output</div></div></div></foreignObject><text x="215" y="193" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Write output</text></switch></g><path d="M 215 159 L 215 121.24" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 219.5 129.12 L 215 120.12 L 210.5 129.12" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 63.23 222.82 L 56.77 215.18 L 149.86 136.52 L 143.4 128.89 L 176 120.98 L 162.77 151.8 L 156.31 144.16 Z" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 169px; margin-left: 117px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; "><div>Update</div><div>data</div><div>async</div></div></div></div></foreignObject><text x="117" y="172" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">Update...</text></switch></g><rect x="0" y="219" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 249px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Data fetched</div></div></div></foreignObject><text x="60" y="253" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Data fetched</text></switch></g><path d="M 390 364 L 285 364 Q 275 364 265 364 L 247.24 364" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 255.12 359.5 L 246.12 364 L 255.12 368.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="775" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 776px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Add missing resources</div></div></div></foreignObject><text x="835" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Add missing resources</text></switch></g><path d="M 830 119 L 840 119 L 840 304 L 850 304 L 835 334 L 820 304 L 830 304 Z" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 229px; margin-left: 765px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">Add new data to fetch</div></div></div></foreignObject><text x="765" y="232" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">Add new data to fetch</text></switch></g><rect x="0" y="394" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 424px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Apply ratelimit</div></div></div></foreignObject><text x="60" y="428" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Apply ratelimit</text></switch></g><path d="M 60 279 L 60 299 Q 60 309 60 319 L 60 391.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 55.5 383.88 L 60 392.88 L 64.5 383.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="775" y="334" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 364px; margin-left: 776px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Start fetching new resources</div></div></div></foreignObject><text x="835" y="368" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Start fetching new r...</text></switch></g><path d="M 775 364 L 685 364 Q 675 364 665 364 L 512.24 364" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 520.12 359.5 L 511.12 364 L 520.12 368.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 120 424 L 440 424 Q 450 424 450 414 L 450 396.24" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 454.5 404.12 L 450 395.12 L 445.5 404.12" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 205 324 L 245 364 L 205 404 L 165 364 Z" fill="#e1d5e7" stroke="#9673a6" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 364px; margin-left: 166px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Error?</div></div></div></foreignObject><text x="205" y="368" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Error?</text></switch></g><path d="M 205 324 L 205 259 Q 205 249 195 249 L 122.24 249" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 130.12 244.5 L 121.12 249 L 130.12 253.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 254px; margin-left: 206px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; "><div>No</div></div></div></div></foreignObject><text x="206" y="257" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">No</text></switch></g><path d="M 165 364 L 100 364 Q 90 364 90 374 L 90 391.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 85.5 383.88 L 90 392.88 L 94.5 383.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 365px; margin-left: 119px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">Yes</div></div></div></foreignObject><text x="119" y="368" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Yes</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
data/lib/consul/async/consul_endpoint.rb CHANGED
@@ -50,10 +50,15 @@ module Consul
50
50
  end
51
51
  end
52
52
 
53
- def create(path)
53
+ def create(path, agent: nil)
54
54
  return self unless @paths[path.to_sym]
55
55
 
56
- ConsulConfiguration.new(base_url: ch(path, :base_url),
56
+ base_url = ch(path, :base_url)
57
+ if agent
58
+ agent = "http://#{agent}" unless agent.start_with? 'http', 'https'
59
+ base_url = agent
60
+ end
61
+ ConsulConfiguration.new(base_url: base_url,
57
62
  debug: ch(path, :debug),
58
63
  token: ch(path, :token),
59
64
  retry_duration: ch(path, :retry_duration),
@@ -128,8 +133,8 @@ module Consul
128
133
  # So, it basically performs all the optimizations to keep updated with Consul internal state.
129
134
  class ConsulEndpoint
130
135
  attr_reader :conf, :path, :x_consul_index, :queue, :stats, :last_result, :enforce_json_200, :start_time, :default_value, :query_params
131
- def initialize(conf, path, enforce_json_200 = true, query_params = {}, default_value = '[]')
132
- @conf = conf.create(path)
136
+ def initialize(conf, path, enforce_json_200 = true, query_params = {}, default_value = '[]', agent = nil)
137
+ @conf = conf.create(path, agent: agent)
133
138
  @default_value = default_value
134
139
  @path = path
135
140
  @queue = EM::Queue.new
data/lib/consul/async/consul_template.rb CHANGED
@@ -48,19 +48,23 @@ module Consul
48
48
  end
49
49
 
50
50
  # Return the coordinates of datacenters
51
- def datacenters(dc: nil)
51
+ def datacenters(dc: nil, agent: nil)
52
52
  path = '/v1/coordinate/datacenters'
53
53
  query_params = {}
54
54
  query_params[:dc] = dc if dc
55
- @endp_manager.create_if_missing(path, query_params) { ConsulTemplateNodes.new(ConsulEndpoint.new(@endp_manager.consul_conf, path, true, query_params, '[]')) }
55
+ @endp_manager.create_if_missing(path, query_params, agent: agent) do
56
+ ConsulTemplateNodes.new(ConsulEndpoint.new(@endp_manager.consul_conf, path, true, query_params, '[]', agent: agent))
57
+ end
56
58
  end
57
59
 
58
60
  # Returns the coordinates for all nodes of DC
59
- def nodes(dc: nil)
61
+ def nodes(dc: nil, agent: nil)
60
62
  path = '/v1/coordinate/nodes'
61
63
  query_params = {}
62
64
  query_params[:dc] = dc if dc
63
- @endp_manager.create_if_missing(path, query_params) { ConsulTemplateNodes.new(ConsulEndpoint.new(@endp_manager.consul_conf, path, true, query_params, '[]')) }
65
+ @endp_manager.create_if_missing(path, query_params, agent: agent) do
66
+ ConsulTemplateNodes.new(ConsulEndpoint.new(@endp_manager.consul_conf, path, true, query_params, '[]', agent: agent))
67
+ end
64
68
  end
65
69
 
66
70
  # Computes the RTT between 2 nodes
@@ -124,7 +128,7 @@ module Consul
124
128
  end
125
129
 
126
130
  # https://www.consul.io/api/health.html#list-nodes-for-service
127
- def service(name, dc: nil, passing: false, tag: nil)
131
+ def service(name, dc: nil, passing: false, tag: nil, agent: nil)
128
132
  raise 'You must specify a name for a service' if name.nil?
129
133
 
130
134
  path = "/v1/health/service/#{name}"
@@ -132,70 +136,70 @@ module Consul
132
136
  query_params[:dc] = dc if dc
133
137
  query_params[:passing] = passing if passing
134
138
  query_params[:tag] = tag if tag
135
- create_if_missing(path, query_params) { ConsulTemplateService.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
139
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateService.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
136
140
  end
137
141
 
138
142
  # https://www.consul.io/api/health.html#list-checks-for-service
139
- def checks_for_service(name, dc: nil, passing: false)
143
+ def checks_for_service(name, dc: nil, passing: false, agent: nil)
140
144
  raise 'You must specify a name for a service' if name.nil?
141
145
 
142
146
  path = "/v1/health/checks/#{name}"
143
147
  query_params = {}
144
148
  query_params[:dc] = dc if dc
145
149
  query_params[:passing] = passing if passing
146
- create_if_missing(path, query_params) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
150
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
147
151
  end
148
152
 
149
153
  # https://www.consul.io/api/health.html#list-checks-for-node
150
- def checks_for_node(name, dc: nil, passing: false)
154
+ def checks_for_node(name, dc: nil, passing: false, agent: nil)
151
155
  raise 'You must specify a name for a service' if name.nil?
152
156
 
153
157
  path = "/v1/health/node/#{name}"
154
158
  query_params = {}
155
159
  query_params[:dc] = dc if dc
156
160
  query_params[:passing] = passing if passing
157
- create_if_missing(path, query_params) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
161
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
158
162
  end
159
163
 
160
164
  # https://www.consul.io/api/catalog.html#list-nodes
161
- def nodes(dc: nil)
165
+ def nodes(dc: nil, agent: nil)
162
166
  path = '/v1/catalog/nodes'
163
167
  query_params = {}
164
168
  query_params[:dc] = dc if dc
165
- create_if_missing(path, query_params) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
169
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
166
170
  end
167
171
 
168
172
  # https://www.consul.io/api/catalog.html#list-services-for-node
169
- def node(name_or_id, dc: nil)
173
+ def node(name_or_id, dc: nil, agent: nil)
170
174
  path = "/v1/catalog/node/#{name_or_id}"
171
175
  query_params = {}
172
176
  query_params[:dc] = dc if dc
173
- create_if_missing(path, query_params) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
177
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
174
178
  end
175
179
 
176
180
  # https://www.consul.io/api/agent.html#read-configuration
177
- def agent_self
181
+ def agent_self(agent: nil)
178
182
  path = '/v1/agent/self'
179
183
  query_params = {}
180
184
  default_value = '{"Config":{}, "Coord":{}, "Member":{}, "Meta":{}, "Stats":{}}'
181
- create_if_missing(path, query_params) { ConsulAgentSelf.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
185
+ create_if_missing(path, query_params, agent: agent) { ConsulAgentSelf.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
182
186
  end
183
187
 
184
188
  # https://www.consul.io/api/agent.html#view-metrics
185
- def agent_metrics
189
+ def agent_metrics(agent: nil)
186
190
  path = '/v1/agent/metrics'
187
191
  query_params = {}
188
192
  default_value = '{"Gauges":[], "Points":[], "Member":{}, "Counters":[], "Samples":{}}'
189
- create_if_missing(path, query_params) { ConsulAgentMetrics.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
193
+ create_if_missing(path, query_params, agent: agent) { ConsulAgentMetrics.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
190
194
  end
191
195
 
192
196
  # https://www.consul.io/api/agent.html#list-members
193
- def agent_members(wan: false)
197
+ def agent_members(wan: false, agent: nil)
194
198
  path = '/v1/agent/members'
195
199
  query_params = {}
196
200
  query_params['wan'] = true if wan
197
201
  default_value = '[]'
198
- create_if_missing(path, query_params) { ConsulTemplateMembers.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
202
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateMembers.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
199
203
  end
200
204
 
201
205
  # Return a param of template
@@ -212,51 +216,57 @@ module Consul
212
216
  end
213
217
 
214
218
  # https://www.consul.io/api/catalog.html#list-services
215
- def services(dc: nil, tag: nil)
219
+ def services(dc: nil, tag: nil, agent: nil)
216
220
  path = '/v1/catalog/services'
217
221
  query_params = {}
218
222
  query_params[:dc] = dc if dc
219
223
  # Tag filtering is performed on client side
220
224
  query_params[:tag] = tag if tag
221
- create_if_missing(path, query_params) { ConsulTemplateServices.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
225
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateServices.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
222
226
  end
223
227
 
224
228
  # https://www.consul.io/api/catalog.html#list-datacenters
225
- def datacenters
229
+ def datacenters(agent: nil)
226
230
  path = '/v1/catalog/datacenters'
227
231
  query_params = {}
228
- create_if_missing(path, query_params) { ConsulTemplateDatacenters.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
232
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateDatacenters.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
229
233
  end
230
234
 
231
235
  # https://www.consul.io/api/kv.html#read-key
232
- def kv(name = nil, dc: nil, keys: nil, recurse: false)
236
+ def kv(name = nil, dc: nil, keys: nil, recurse: false, agent: nil)
233
237
  path = "/v1/kv/#{name}"
234
238
  query_params = {}
235
239
  query_params[:dc] = dc if dc
236
240
  query_params[:recurse] = recurse if recurse
237
241
  query_params[:keys] = keys if keys
238
242
  default_value = '[]'
239
- create_if_missing(path, query_params) { ConsulTemplateKV.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value), name) }
243
+ create_if_missing(path, query_params, agent: agent) { ConsulTemplateKV.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value), name) }
240
244
  end
241
245
 
242
- def secrets(path = '')
246
+ def secrets(path = '', agent: nil)
243
247
  raise "You need to provide a vault token to use 'secret' keyword" if vault_conf.token.nil?
244
248
 
245
249
  path = "/v1/#{path}".gsub(%r{/{2,}}, '/')
246
250
  query_params = { list: 'true' }
247
- create_if_missing(path, query_params, vault_conf.fail_fast_errors, vault_conf.max_consecutive_errors_on_endpoint) do
248
- ConsulTemplateVaultSecretList.new(VaultEndpoint.new(vault_conf, path, 'GET', true, query_params, JSON.generate(data: { keys: [] })))
251
+ create_if_missing(path, query_params,
252
+ fail_fast_errors: vault_conf.fail_fast_errors,
253
+ max_consecutive_errors_on_endpoint: vault_conf.max_consecutive_errors_on_endpoint,
254
+ agent: agent) do
255
+ ConsulTemplateVaultSecretList.new(VaultEndpoint.new(vault_conf, path, 'GET', true, query_params, JSON.generate(data: { keys: [] }), agent: agent))
249
256
  end
250
257
  end
251
258
 
252
- def secret(path = '', post_data = nil)
259
+ def secret(path = '', post_data = nil, agent: nil)
253
260
  raise "You need to provide a vault token to use 'secret' keyword" if vault_conf.token.nil?
254
261
 
255
262
  path = "/v1/#{path}".gsub(%r{/{2,}}, '/')
256
263
  query_params = {}
257
264
  method = post_data ? 'POST' : 'GET'
258
- create_if_missing(path, query_params, vault_conf.fail_fast_errors, vault_conf.max_consecutive_errors_on_endpoint) do
259
- ConsulTemplateVaultSecret.new(VaultEndpoint.new(vault_conf, path, method, true, query_params, JSON.generate(data: {})))
265
+ create_if_missing(path, query_params,
266
+ fail_fast_errors: vault_conf.fail_fast_errors,
267
+ max_consecutive_errors_on_endpoint: vault_conf.max_consecutive_errors_on_endpoint,
268
+ agent: agent) do
269
+ ConsulTemplateVaultSecret.new(VaultEndpoint.new(vault_conf, path, method, true, query_params, JSON.generate(data: {}), agent: agent))
260
270
  end
261
271
  end
262
272
 
@@ -372,10 +382,10 @@ module Consul
372
382
  VaultEndpoint.new(vault_conf, path, :POST, {}, {})
373
383
  end
374
384
 
375
- def create_if_missing(path, query_params, fail_fast_errors = @fail_fast_errors, max_consecutive_errors_on_endpoint = @max_consecutive_errors_on_endpoint)
385
+ def create_if_missing(path, query_params, fail_fast_errors: @fail_fast_errors, max_consecutive_errors_on_endpoint: @max_consecutive_errors_on_endpoint, agent: nil)
376
386
  fqdn = path.dup
377
387
  query_params.each_pair do |k, v|
378
- fqdn = "#{fqdn}&#{k}=#{v}"
388
+ fqdn = "#{agent}#{fqdn}&#{k}=#{v}"
379
389
  end
380
390
  tpl = @endpoints[fqdn]
381
391
  unless tpl
data/lib/consul/async/vault_endpoint.rb CHANGED
@@ -46,10 +46,15 @@ module Consul
46
46
  end
47
47
  end
48
48
 
49
- def create(path)
49
+ def create(path, agent: nil)
50
50
  return self unless @paths[path.to_sym]
51
51
 
52
- VaultConfiguration.new(base_url: ch(path, :base_url),
52
+ base_url = ch(path, :base_url)
53
+ if agent
54
+ agent = "http://#{agent}" unless agent.start_with? 'http', 'https'
55
+ base_url = agent
56
+ end
57
+ VaultConfiguration.new(base_url: base_url,
53
58
  debug: ch(path, :debug),
54
59
  token: ch(path, :token),
55
60
  retry_duration: ch(path, :retry_duration),
@@ -117,8 +122,8 @@ module Consul
117
122
  class VaultEndpoint
118
123
  attr_reader :conf, :path, :http_method, :queue, :stats, :last_result, :enforce_json_200, :start_time, :default_value, :query_params
119
124
 
120
- def initialize(conf, path, http_method = 'GET', enforce_json_200 = true, query_params = {}, default_value = '{}', post_data = {})
121
- @conf = conf.create(path)
125
+ def initialize(conf, path, http_method = 'GET', enforce_json_200 = true, query_params = {}, default_value = '{}', post_data = {}, agent: nil)
126
+ @conf = conf.create(path, agent: agent)
122
127
  @default_value = default_value
123
128
  @path = path
124
129
  @http_method = http_method
data/lib/consul/async/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  module Consul
2
2
  module Async
3
- VERSION = '1.25.2'.freeze
3
+ VERSION = '1.26.0'.freeze
4
4
  end
5
5
  end
data/samples/all_services.txt.erb CHANGED
@@ -9,7 +9,7 @@
9
9
  require 'set'
10
10
 
11
11
  # Services to hide
12
- services_blacklist_raw = (ENV['EXCLUDE_SERVICES'] || '.*netsvc-probe.*,.*consul-probed.*').split(',')
12
+ services_blacklist_raw = (ENV['EXCLUDE_SERVICES'] || '.*netsvc-probe.*,.*consul-probed.*,.lbl7.*').split(',')
13
13
  services_blacklist = services_blacklist_raw.map { |v| Regexp.new(v) }
14
14
 
15
15
  num_services={}
data/samples/all_services_multi_agents.txt.erb ADDED
@@ -0,0 +1,69 @@
1
+ __________________________________________________________________
2
+ | DC | Services |______________Instances________________| Nodes |
3
+ | | | Passing | Warning | Critic | Total | |
4
+ |-------+----------+---------+---------+---------+---------+-------|
5
+ <%
6
+ # This template list all instances on all DCs
7
+ # And aggregates a list of Services, Services Instances/status
8
+ # And nodes.
9
+ require 'set'
10
+
11
+
12
+ def find_best_agent_for_dc(dc)
13
+ agent = service('consul-relay', passing: true, dc:dc).first
14
+ agent = service('consul-agent-http', passing: true, dc:dc).first unless agent
15
+ service('consul', passing: true, dc:dc).first unless agent
16
+ return nil unless agent
17
+ port = agent['Service']['Port'] || 8500
18
+ "http://#{agent.service_address}:#{port}"
19
+ end
20
+
21
+ # Services to hide
22
+ services_blacklist_raw = (ENV['EXCLUDE_SERVICES'] || '.*netsvc-probe.*,.*consul-probed.*').split(',')
23
+ services_blacklist = services_blacklist_raw.map { |v| Regexp.new(v) }
24
+
25
+ num_services={}
26
+ num_instances={}
27
+ all_states = ['passing', 'warning', 'critical', 'total']
28
+ distinct_services=Set.new
29
+ datacenters().each do |dc|
30
+ next unless find_best_agent_for_dc(dc)
31
+ num_services[dc] = 0
32
+ num_instances[dc] = {
33
+ 'passing' => 0,
34
+ 'warning' => 0,
35
+ 'critical' => 0,
36
+ 'total' => 0
37
+ }
38
+ services(dc:dc, agent: find_best_agent_for_dc(dc)).each do |service_name, tags|
39
+ next if services_blacklist.any? {|r| r.match(service_name)}
40
+ distinct_services.add(service_name)
41
+ num_services[dc]+=1
42
+ service(service_name, dc:dc, agent: find_best_agent_for_dc(dc)).each do |snode|
43
+ num_instances[dc][snode.status]+=1
44
+ num_instances[dc]['total']+=1
45
+ end
46
+ end
47
+ end
48
+ num_instances_total={
49
+ 'passing' => 0,
50
+ 'warning' => 0,
51
+ 'critical' => 0,
52
+ 'total' => 0
53
+ }
54
+ num_nodes_total=0
55
+ num_services.each do |dc, num_services_for_dc|
56
+ all_states.each do |s|
57
+ num_instances_total[s] += num_instances[dc][s]
58
+ end
59
+ num_nodes=nodes(dc:dc, agent: find_best_agent_for_dc(dc)).count
60
+ num_nodes_total+=num_nodes
61
+ %>| <%= dc.rjust(5) %> | <%= num_services_for_dc.to_s.rjust(8) %> |<% all_states.each do |status|
62
+ %> <%= num_instances[dc][status].to_s.rjust(7) %> |<% end %> <%= num_nodes.to_s.rjust(5) %> |
63
+ <%
64
+ end
65
+ %>|-------+----------+---------+---------+---------+---------+-------|
66
+ | TOTAL | <%= distinct_services.count.to_s.rjust(8) %> |<% all_states.each do |status|
67
+ %> <%= num_instances_total[status].to_s.rjust(7) %> |<%
68
+ end %> <%= num_nodes_total.to_s.rjust(5) %> |
69
+ '_______|__________|_________|_________|_________|_________|_______'
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: consul-templaterb
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.25.2
4
+ version: 1.26.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - SRE Core Services
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-02-29 00:00:00.000000000 Z
11
+ date: 2020-03-05 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: em-http-request
@@ -180,6 +180,9 @@ files:
180
180
  - bin/consul-templaterb
181
181
  - consul-templaterb.gemspec
182
182
  - docker-nginx-conf/nginx.conf
183
+ - docs/article-06_Template-based_discovery_with_consul-templaterb.md
184
+ - docs/images/consul-templaterb.png
185
+ - docs/images/consul-templaterb.svg
183
186
  - docs/images/consul-ui_001.png
184
187
  - lib/consul/async/consul_endpoint.rb
185
188
  - lib/consul/async/consul_template.rb
@@ -193,6 +196,7 @@ files:
193
196
  - lib/consul/async/vault_endpoint.rb
194
197
  - lib/consul/async/version.rb
195
198
  - samples/all_services.txt.erb
199
+ - samples/all_services_multi_agents.txt.erb
196
200
  - samples/all_templates.erb
197
201
  - samples/consul-ui/README.md
198
202
  - samples/consul-ui/common/footer.html.erb