consul-templaterb 1.25.2 → 1.26.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/INTERNALS.md +2 -0
- data/README.md +2 -0
- data/TemplateAPI.md +175 -164
- data/docs/article-06_Template-based_discovery_with_consul-templaterb.md +123 -0
- data/docs/images/consul-templaterb.png +0 -0
- data/docs/images/consul-templaterb.svg +3 -0
- data/lib/consul/async/consul_endpoint.rb +9 -4
- data/lib/consul/async/consul_template.rb +44 -34
- data/lib/consul/async/vault_endpoint.rb +9 -4
- data/lib/consul/async/version.rb +1 -1
- data/samples/all_services.txt.erb +1 -1
- data/samples/all_services_multi_agents.txt.erb +69 -0
- metadata +6 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 79f77f92ae3811c48a823724fb4fb242b78fb78713de126cc035de4089a5e607
|
4
|
+
data.tar.gz: 3aa6c6f9c1aca03006963dd03ab30c3ba9ceabc4ff88c3a0d7fa8470a024b2d2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ed5763f1c109d678a867dddbe77a2b4e8d0b9adde1663f80e9bb6db2812c104f84edcd9f5c929d8e1f4763e9b4fa3af138da87c58af1285d6f1b0c6f5924760d
|
7
|
+
data.tar.gz: af7a53d991a486b4050632576b36c2cb1fe5e899bee75ec73a2e6bd76bb18e350913e5703b69ee8a05aa51b2d64a64a60b8c17dcc1e071b8cac3544331de0600
|
data/CHANGELOG.md
CHANGED
@@ -2,6 +2,17 @@
|
|
2
2
|
|
3
3
|
## (UNRELEASED)
|
4
4
|
|
5
|
+
NEW FEATURES:
|
6
|
+
|
7
|
+
## 1.26.0 (March 5, 2020)
|
8
|
+
|
9
|
+
* Using `agent: http://vault_or_consul_agent:port>` on most methods will now override the agent
|
10
|
+
used to perform Consul queries. It might be useful for very large clusters or large WAN federations,
|
11
|
+
because you can perform some requests on some agents, and some others on other agenrs. Might
|
12
|
+
also be useful to federate data from preprod/prod for instance.
|
13
|
+
See [samples/all_services_multi_agents.txt.erb](samples/all_services_multi_agents.txt.erb) for an
|
14
|
+
example.
|
15
|
+
|
5
16
|
## 1.25.2 (February 29, 2020)
|
6
17
|
|
7
18
|
BUGFIX:
|
data/INTERNALS.md
CHANGED
@@ -12,6 +12,8 @@ the list of services registered in the cluster).
|
|
12
12
|
|
13
13
|
For each endpoint, `consul-templaterb` performs a watch by getting information about the last transaction on this endpoints, storing it locally and watching at Consul to get updates about the data within this endpoint.
|
14
14
|
|
15
|
+
![Main Loop](docs/images/consul-templaterb.svg)
|
16
|
+
|
15
17
|
## How it works
|
16
18
|
|
17
19
|
There is a loop in the engine, every second, all templates are rendered using the `ERB` template engine.
|
data/README.md
CHANGED
@@ -33,6 +33,8 @@ with fully [working examples](samples/).
|
|
33
33
|
It also allows displaying a very nice and hi-performance HTML5 UI for Consul,
|
34
34
|
see [consul-ui](samples/consul-ui) for details.
|
35
35
|
|
36
|
+
There is an article [docs/article-06_Template-based_discovery_with_consul-templaterb.md](docs/article-06_Template-based_discovery_with_consul-templaterb.md) summarizing why we developped this tool (also available on [medium](https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457)).
|
37
|
+
|
36
38
|
## Video introduction to consul-templaterb
|
37
39
|
|
38
40
|
[![Introduction to Consul-templaterb](docs/images/consul-ui_001.png)](https://youtu.be/zLzrLGLLl4Q)
|
data/TemplateAPI.md
CHANGED
@@ -7,6 +7,10 @@ annoted with `[]`.
|
|
7
7
|
Most of them support the optional dc attribute to use data from another datacenter. If the `dc`
|
8
8
|
attribute is not specified, the function will output data from the current datacenter.
|
9
9
|
|
10
|
+
Starting with version 1.26.0, most methods also support the
|
11
|
+
`agent: "http://address_of_vault_or_consul_agent:port"` to use another agent than the one specified in command line. It opens the way to use different agents/DC or to create templates federating several different not-linked clusters (ex: prod/preprod).
|
12
|
+
Example on how using it: [samples/all_services_multi_agents.txt.erb](samples/all_services_multi_agents.txt.erb).
|
13
|
+
|
10
14
|
To ease template development, `consul-templaterb` supports HOT reload of templates, thus it is possible to
|
11
15
|
develop the templates interactively. While developing, it is possible to use the switch `--hot-reload=keep`,
|
12
16
|
thus the application will display a warning if the template is invalid and won't stop
|
@@ -44,154 +48,162 @@ to avoid having to write .result in all templates, some shortcuts have been adde
|
|
44
48
|
|
45
49
|
Also available for all results:
|
46
50
|
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
|
122
|
-
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
|
129
|
-
|
130
|
-
|
131
|
-
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
144
|
-
|
145
|
-
|
146
|
-
|
147
|
-
|
148
|
-
|
149
|
-
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
|
158
|
-
|
159
|
-
|
160
|
-
|
161
|
-
|
162
|
-
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
|
167
|
-
|
168
|
-
|
169
|
-
|
170
|
-
|
171
|
-
|
172
|
-
|
173
|
-
|
174
|
-
|
175
|
-
|
176
|
-
|
177
|
-
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
-
|
182
|
-
|
183
|
-
|
184
|
-
|
185
|
-
|
186
|
-
|
187
|
-
|
188
|
-
|
189
|
-
|
190
|
-
|
191
|
-
|
192
|
-
|
193
|
-
|
194
|
-
|
51
|
+
<details id="common-methods-available-for-all-objects">
|
52
|
+
<summary>Common methods available for all objects</summary>
|
53
|
+
<div class="samples">
|
54
|
+
<ul>
|
55
|
+
<li>.all?</li>
|
56
|
+
<li>.any?</li>
|
57
|
+
<li>.assoc</li>
|
58
|
+
<li>.chunk</li>
|
59
|
+
<li>.chunk_while</li>
|
60
|
+
<li>.class</li>
|
61
|
+
<li>.clear</li>
|
62
|
+
<li>.collect</li>
|
63
|
+
<li>.collect_concat</li>
|
64
|
+
<li>.compact</li>
|
65
|
+
<li>.count</li>
|
66
|
+
<li>.cycle</li>
|
67
|
+
<li>.detect</li>
|
68
|
+
<li>.dig</li>
|
69
|
+
<li>.display</li>
|
70
|
+
<li>.drop</li>
|
71
|
+
<li>.drop_while</li>
|
72
|
+
<li>.dup</li>
|
73
|
+
<li>.each</li>
|
74
|
+
<li>.each_cons</li>
|
75
|
+
<li>.each_entry</li>
|
76
|
+
<li>.each_slice</li>
|
77
|
+
<li>.each_with_index</li>
|
78
|
+
<li>.each_with_object</li>
|
79
|
+
<li>.empty?</li>
|
80
|
+
<li>.entries</li>
|
81
|
+
<li>.enum_for</li>
|
82
|
+
<li>.eql?</li>
|
83
|
+
<li>.equal?</li>
|
84
|
+
<li>.extend</li>
|
85
|
+
<li>.fetch</li>
|
86
|
+
<li>.find</li>
|
87
|
+
<li>.find_all</li>
|
88
|
+
<li>.find_index</li>
|
89
|
+
<li>.first</li>
|
90
|
+
<li>.flat_map</li>
|
91
|
+
<li>.flatten</li>
|
92
|
+
<li>.grep</li>
|
93
|
+
<li>.grep_v</li>
|
94
|
+
<li>.group_by</li>
|
95
|
+
<li>.hash</li>
|
96
|
+
<li>.include?</li>
|
97
|
+
<li>.index</li>
|
98
|
+
<li>.inject</li>
|
99
|
+
<li>.inspect</li>
|
100
|
+
<li>.is_a?</li>
|
101
|
+
<li>.itself</li>
|
102
|
+
<li>.keep_if</li>
|
103
|
+
<li>.kind_of?</li>
|
104
|
+
<li>.lazy</li>
|
105
|
+
<li>.length</li>
|
106
|
+
<li>.map</li>
|
107
|
+
<li>.max</li>
|
108
|
+
<li>.max_by</li>
|
109
|
+
<li>.member?</li>
|
110
|
+
<li>.min</li>
|
111
|
+
<li>.min_by</li>
|
112
|
+
<li>.minmax</li>
|
113
|
+
<li>.minmax_by</li>
|
114
|
+
<li>.nil?</li>
|
115
|
+
<li>.none?</li>
|
116
|
+
<li>.object_id</li>
|
117
|
+
<li>.one?</li>
|
118
|
+
<li>.partition</li>
|
119
|
+
<li>.pp</li>
|
120
|
+
<li>.rassoc</li>
|
121
|
+
<li>.reduce</li>
|
122
|
+
<li>.reject</li>
|
123
|
+
<li>.replace</li>
|
124
|
+
<li>.reverse_each</li>
|
125
|
+
<li>.select</li>
|
126
|
+
<li>.shift</li>
|
127
|
+
<li>.size</li>
|
128
|
+
<li>.slice</li>
|
129
|
+
<li>.slice_after</li>
|
130
|
+
<li>.slice_before</li>
|
131
|
+
<li>.slice_when</li>
|
132
|
+
<li>.sort</li>
|
133
|
+
<li>.sort_by</li>
|
134
|
+
<li>.sum</li>
|
135
|
+
<li>.take</li>
|
136
|
+
<li>.take_while</li>
|
137
|
+
<li>.tap</li>
|
138
|
+
<li>.to_a</li>
|
139
|
+
<li>.to_enum</li>
|
140
|
+
<li>.to_h</li>
|
141
|
+
<li>.to_s</li>
|
142
|
+
<li>.uniq</li>
|
143
|
+
<li>.values_at</li>
|
144
|
+
<li>.yield_self</li>
|
145
|
+
<li>.zip</li>
|
146
|
+
</ul></div></details>
|
147
|
+
|
148
|
+
<details>
|
149
|
+
<summary id="methods-available-for-array-objects">Methods available for Array objects</summary>
|
150
|
+
<div class="samples">
|
151
|
+
<ul>
|
152
|
+
<li>.append</li>
|
153
|
+
<li>.at</li>
|
154
|
+
<li>.bsearch</li>
|
155
|
+
<li>.bsearch_index</li>
|
156
|
+
<li>.combination</li>
|
157
|
+
<li>.concat</li>
|
158
|
+
<li>.each_index</li>
|
159
|
+
<li>.fill</li>
|
160
|
+
<li>.insert</li>
|
161
|
+
<li>.join</li>
|
162
|
+
<li>.last</li>
|
163
|
+
<li>.pack</li>
|
164
|
+
<li>.permutation</li>
|
165
|
+
<li>.pop</li>
|
166
|
+
<li>.prepend</li>
|
167
|
+
<li>.product</li>
|
168
|
+
<li>.push</li>
|
169
|
+
<li>.repeated_combination</li>
|
170
|
+
<li>.repeated_permutation</li>
|
171
|
+
<li>.reverse</li>
|
172
|
+
<li>.rindex</li>
|
173
|
+
<li>.rotate</li>
|
174
|
+
<li>.sample</li>
|
175
|
+
<li>.shuffle</li>
|
176
|
+
<li>.to_ary</li>
|
177
|
+
<li>.transpose</li>
|
178
|
+
<li>.unshift</li>
|
179
|
+
</ul></div></details>
|
180
|
+
|
181
|
+
<details id="Methods-available-for-hash-objects">
|
182
|
+
<summary>Methods available for hash objects</summary>
|
183
|
+
<ul>
|
184
|
+
<li>.compare_by_identity</li>
|
185
|
+
<li>.compare_by_identity?</li>
|
186
|
+
<li>.each_key</li>
|
187
|
+
<li>.each_pair</li>
|
188
|
+
<li>.each_value</li>
|
189
|
+
<li>.fetch_values</li>
|
190
|
+
<li>.has_key?</li>
|
191
|
+
<li>.has_value?</li>
|
192
|
+
<li>.invert</li>
|
193
|
+
<li>.key</li>
|
194
|
+
<li>.key?</li>
|
195
|
+
<li>.keys</li>
|
196
|
+
<li>.merge</li>
|
197
|
+
<li>.rehash</li>
|
198
|
+
<li>.store</li>
|
199
|
+
<li>.to_hash</li>
|
200
|
+
<li>.to_proc</li>
|
201
|
+
<li>.transform_keys</li>
|
202
|
+
<li>.transform_values</li>
|
203
|
+
<li>.update</li>
|
204
|
+
<li>.value?</li>
|
205
|
+
<li>.values</li>
|
206
|
+
</ul></div></details>
|
195
207
|
|
196
208
|
## coordinate
|
197
209
|
|
@@ -203,7 +215,7 @@ The coordinate object allow to interact with the coordinates of DCs and nodes as
|
|
203
215
|
[List the Wan Coordinates](https://www.consul.io/api/coordinate.html#read-wan-coordinates) from local DC to
|
204
216
|
other DCs. If dc is set, it will perform the same operation but from another DC.
|
205
217
|
|
206
|
-
### coordinate.nodes([dc: datacenter])
|
218
|
+
### coordinate.nodes([dc: datacenter], [agent: consul_agent_address])
|
207
219
|
|
208
220
|
[Read all LAN nodes coordinates](https://www.consul.io/api/coordinate.html#read-lan-coordinates-for-all-nodes).
|
209
221
|
If If dc is set, it will perform the same operation but for another DC.
|
@@ -213,7 +225,7 @@ If If dc is set, it will perform the same operation but for another DC.
|
|
213
225
|
Computes the rtt between 2 nodes returned by `coordinate.nodes` or `coordinate.datacenters`. A re-implementation of Golang sample code
|
214
226
|
[Working with Coordinates](https://www.consul.io/docs/internals/coordinates.html#working-with-coordinates).
|
215
227
|
|
216
|
-
## datacenters()
|
228
|
+
## datacenters([agent: consul_agent_address])
|
217
229
|
|
218
230
|
[Get the list of datacenters as string array](https://www.consul.io/api/catalog.html#list-datacenters).
|
219
231
|
|
@@ -233,7 +245,7 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
|
|
233
245
|
</div>
|
234
246
|
</details>
|
235
247
|
|
236
|
-
## services([dc: datacenter], [tag: tagToFilterWith])
|
248
|
+
## services([dc: datacenter], [tag: tagToFilterWith], [agent: consul_agent_address])
|
237
249
|
|
238
250
|
[List the services matching the optional tag filter](https://www.consul.io/api/catalog.html#list-services),
|
239
251
|
if tag is not specified, will match all the services. Note that this endpoint performs client side tag
|
@@ -270,7 +282,7 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
|
|
270
282
|
</div>
|
271
283
|
</details>
|
272
284
|
|
273
|
-
## service(serviceName, [dc: datacenter], [tag: tagToFilterWith], [passing: true])
|
285
|
+
## service(serviceName, [dc: datacenter], [tag: tagToFilterWith], [passing: true], [agent: consul_agent_address])
|
274
286
|
|
275
287
|
[List the instances](https://www.consul.io/api/health.html#list-nodes-for-service) of a service having the given
|
276
288
|
optional tag. If no tag is specified, will return all instances of service. By default, it will return all the
|
@@ -342,7 +354,7 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
|
|
342
354
|
</div>
|
343
355
|
</details>
|
344
356
|
|
345
|
-
## nodes([dc: datacenter])
|
357
|
+
## nodes([dc: datacenter], [agent: consul_agent_address])
|
346
358
|
|
347
359
|
[List all the nodes of selected datacenter](https://www.consul.io/api/catalog.html#list-nodes). No filtering is
|
348
360
|
applied.
|
@@ -363,20 +375,20 @@ Full example: [samples/consul_template.txt.erb](samples/consul_template.txt.erb)
|
|
363
375
|
</div>
|
364
376
|
</details>
|
365
377
|
|
366
|
-
## node(nodeNameOrId, [dc: datacenter])
|
378
|
+
## node(nodeNameOrId, [dc: datacenter], [agent: consul_agent_address])
|
367
379
|
|
368
380
|
[List all the services of a given Node](https://www.consul.io/api/catalog.html#list-services-for-node) using its
|
369
381
|
name or its ID. If DC is specified, will lookup for given node in another datacenter.
|
370
382
|
|
371
|
-
## checks_for_node(name, dc: nil, passing: false, tag: nil)
|
383
|
+
## checks_for_node(name, dc: nil, passing: false, tag: nil, [agent: consul_agent_address])
|
372
384
|
|
373
385
|
[Find all the checks](https://www.consul.io/api/health.html#list-checks-for-node) of a given node name.
|
374
386
|
|
375
|
-
## checks_for_service(name, dc: nil, passing: false, tag: nil)
|
387
|
+
## checks_for_service(name, dc: nil, passing: false, tag: nil, [agent: consul_agent_address])
|
376
388
|
|
377
389
|
[Find all the checks](https://www.consul.io/api/health.html#list-checks-for-service) of a given service.
|
378
390
|
|
379
|
-
## kv(name, [dc: nil], [keys: false], [recurse: false])
|
391
|
+
## kv(name, [dc: nil], [keys: false], [recurse: false], [agent: consul_agent_address])
|
380
392
|
|
381
393
|
[Read keys from KV Store](https://www.consul.io/api/kv.html#read-key). It can be used for both listing the keys and
|
382
394
|
getting the values. See the file in samples [keys.html.erb](samples/keys.html.erb) for a working example.
|
@@ -451,7 +463,7 @@ value123 : <%= result.get_decoded('/my/multiple/values/value123') %>
|
|
451
463
|
Since `kv('/my/multiple/values', recurse: true)` will retrieve all values at once, it might be more
|
452
464
|
efficient in some cases than retrieving all values one by one.
|
453
465
|
|
454
|
-
## agent_members(wan: false)
|
466
|
+
## agent_members(wan: false, [agent: consul_agent_address])
|
455
467
|
|
456
468
|
[Get the Serf information](https://www.consul.io/api/agent.html#list-members) from Consul Agent point of view.
|
457
469
|
This is a list of Serf information containing serf information. This information is not consistent and should be used with care, most of the time, you should prefer `nodes()`.
|
@@ -470,13 +482,13 @@ The returned value is an array containing the following objects containing the f
|
|
470
482
|
|
471
483
|
See [samples/members.json.erb](samples/members.json.erb) for example of usage.
|
472
484
|
|
473
|
-
## agent_metrics()
|
485
|
+
## agent_metrics([agent: consul_agent_address])
|
474
486
|
|
475
487
|
[Get the metrics of Consul Agent](https://www.consul.io/api/agent.html#view-metrics). Since this endpoint does
|
476
488
|
not support blocking queries, data will be refreshed every few seconds, but will not use blocking queries
|
477
489
|
mechanism.
|
478
490
|
|
479
|
-
## agent_self()
|
491
|
+
## agent_self([agent: consul_agent_address])
|
480
492
|
|
481
493
|
[Get the configuration of Consul Agent](https://www.consul.io/api/agent.html#read-configuration).
|
482
494
|
Since this endpoint does not support blocking queries, data will be refreshed every few seconds,
|
@@ -542,7 +554,7 @@ consul-templaterb --template "source.html.erb:dest.html:reload_command:params.ya
|
|
542
554
|
|
543
555
|
See [samples/consul-ui/consul-services-ui.html.erb](samples/consul-ui/consul-services-ui.html.erb) for example of usage.
|
544
556
|
|
545
|
-
## secrets(prefix)
|
557
|
+
## secrets(prefix, [agent: vault_agent_address])
|
546
558
|
|
547
559
|
It requires that a Vault token is given either in parameter or in environment variable
|
548
560
|
The [policies](https://www.vaultproject.io/docs/concepts/policies.html) should be properly set.
|
@@ -568,7 +580,7 @@ Full example: [samples/vault-ldap.txt.erb](samples/vault-ldap.txt.erb)
|
|
568
580
|
</div>
|
569
581
|
</details>
|
570
582
|
|
571
|
-
## secret(path, [data = nil])
|
583
|
+
## secret(path, [data = nil], [agent: vault_agent_address])
|
572
584
|
|
573
585
|
It requires that a Vault token is given either in parameter or in environment variable
|
574
586
|
The [policies](https://www.vaultproject.io/docs/concepts/policies.html) should be properly set.
|
@@ -689,5 +701,4 @@ Here are templates rendered by consul-templaterb:
|
|
689
701
|
```
|
690
702
|
|
691
703
|
</div>
|
692
|
-
</details>
|
693
|
-
|
704
|
+
</details>
|
@@ -0,0 +1,123 @@
|
|
1
|
+
# Template-based discovery with consul-templaterb
|
2
|
+
|
3
|
+
Source of article https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
|
4
|
+
|
5
|
+
# How and why Criteo built a template-based system for Consul
|
6
|
+
|
7
|
+
[consul-templaterb](https://github.com/criteo/consul-templaterb/) is an OpenSource Software (OSS) written in Ruby that allows you to very quickly create UIs for Consul or configuration files for your beloved software to interact with Consul. This article describes why we wrote it and why you might be interested in using it.
|
8
|
+
|
9
|
+
# The context
|
10
|
+
|
11
|
+
At the end of 2017, Consul started becoming a very important piece in Criteo’s infrastructure as it was responsible for dealing with the discovery of all major business applications at Criteo. At that time, Criteo had more than 25,000 servers on production and some datacenters had more than 5,000 agents per datacenter. At the same time, we started building our Load-Balancer stack based on the Consul state. So it became more and more important to make it efficient, stable and expand it with new features.
|
12
|
+
|
13
|
+
Criteo wanted to move towards HAProxy for its load-balancing stack, thus we started generating some configuration for HAProxy with all HTTP services with the template-based configuration generator consul-template. However, things were a bit more complicated than expected because the program consul-template was using almost all bandwidth available just to discover all services. Moreover, the system was constantly updating HAProxy configuration.
|
14
|
+
|
15
|
+
As explained in our first article “[Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)”, Consul includes a notification system to ensure your systems are always up-to-date with the content of the discovery. When this system starts to push too many notifications, on large services, the amount of data sent to all agents can be quite impressive (several megabytes/s for a single service having hundreds of instances).
|
16
|
+
|
17
|
+
Thus, we started investigating the issue and figured out that it was linked to a race condition in large clusters that caused all listeners of Consul to be notified continuously. This issue was eventually solved by our [Pull-Request #3899](https://github.com/hashicorp/consul/pull/3899) (and quite a few others after that), but the investigation showed that it was quite hard to investigate the root cause of such issues.
|
18
|
+
|
19
|
+
Therefore, I started a simple tool to watch all the services matching some patterns and report the bandwidth usage, but I also wanted to prove it would be possible to generate all configurations for HAProxy with an optimized tool, thus having a templating mechanism. The [Pull Request #1066](https://github.com/hashicorp/consul-template/pull/1066) to fix the issue on the consul-template itself was not yet considered (and it was hard to have something better), so I decided to implement it in Ruby with asynchronous I/Os (Input/Outputs) because I was very frustrated by the Go templating language used in consul-template and was in love with ERB templates.
|
20
|
+
|
21
|
+
# The goals
|
22
|
+
|
23
|
+
From scratch, I wanted to take the various shortcoming of consul-template into account:
|
24
|
+
|
25
|
+
* The ability to generate high-level code to generate various formats (YAML, JSON, XML): it is dangerous, error-prone and painful to generate JSON or XML content using text templating. People fighting with YAML indentation will tell you that.
|
26
|
+
* The ability to use a Turing complete and effective language to perform simple transformation such as sorting (Consul by default has some rules to sort the output, but it might not be predictable, for instance, based on the round-trip between servers).
|
27
|
+
* Being able to push real-life optimizations (for instance, we don’t care about being notified immediately when a change occurs, we want to be notified once before dumping the configuration file, not 100 times without taking it into account), based on the type of query performed.
|
28
|
+
* Being able to scale nicely with Consul and avoid overloading the cluster for nothing in case of bug (for instance, we use a penalty for services changing too much, we rate-limit the notifications).
|
29
|
+
* Being very simple to use, hiding all the complexity to the template creator and using optimizations to reduce the number of calls to Consul, meaning being as purely functional as possible by hiding all I/O operations and taking decisions based on our experience with Consul internals.
|
30
|
+
* Be evolutive: When some functions or fields are missing in a consul-template, you have to do a pull request on consul-template to add your new fields/methods, then you can work. It would be nice if any field of endpoints would be supported natively (so new versions of Consul can be used with an old version of our tool).
|
31
|
+
* Support for hot-reload of templates, and nice error messages with the precise line when an error occurs, so writing templates should be very fast and efficient.
|
32
|
+
* Have most of the consul-template features including babysitting of processes, spawn commands when files do change…
|
33
|
+
|
34
|
+
# Internals: make your template engine in Ruby
|
35
|
+
|
36
|
+
## At startup
|
37
|
+
|
38
|
+
The engine collects all template sources and computes the destinations. Each template and its destination also register some commands to run when the destination does change. Those objects save the last time the template has been read (to allow hot-reload) and the last binary content associated with them (to detect binary changes).
|
39
|
+
|
40
|
+
## The main loop
|
41
|
+
|
42
|
+
First, the main loop is working on pending events. Those events are in fact the processing of the I/O operations querying the Consul agent. But every second by default, the main loop also evaluates templates which are Ruby code. When this is just regular code, this is evaluated normally, but when the code called is part of the [I/O API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md), the code does the following:
|
43
|
+
|
44
|
+
1. Checks in a registry if this method has been called with the same parameters, if not, create it, mark it as dirty (so the registry knows we are waiting for data from Consul), launch the request asynchronously to Consul, store it in the registry. This will be translated into events that will be run as events in the main loop.
|
45
|
+
2. Return the object stored within the registry. By default, this object returns the empty object or collection according to its type. When the request finally gets a response from Consul, the object is marked as “non-dirty” and the result is put within the registry, so the next evaluation returns the content of the requests.
|
46
|
+
|
47
|
+
Once all code has been evaluated and the result has been stored in memory, the main loop checks whether the template is using “dirty” objects in the registry. Having at least one dirty object means that some requests are still in flight and we don’t have the result yet, so basically means the template, while evaluated, has been evaluated with fake (empty) data and is not yet ready.
|
48
|
+
|
49
|
+
However, if the template is not using any dirty object it means we did receive all the data and we can render the template on disk. We first check if our last rendering was different. If the new rendering phase was not different, nothing did change, however, if the data rendered is different, the file is stored to disk and commands might be evaluated (to notify a program to reload for instance).
|
50
|
+
|
51
|
+
All the I/O are done asynchronously (using [eventmachine](https://github.com/eventmachine/eventmachine)) but performed within the main loop, the one also performing the rendering, so, there is no possibility of threading issues.
|
52
|
+
|
53
|
+
![Main loop behavior, in purple, the async fetch, in yellow, the rendering](images/consul-templaterb.png)
|
54
|
+
|
55
|
+
## Handling of I/O
|
56
|
+
|
57
|
+
The Input/Outputs are handled very carefully with regards to what we explained in our previous article “[Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)” and adds a few tricks:
|
58
|
+
|
59
|
+
* When a given endpoint is changing too fast with any real change (the data is the same as last call), a penalty is applied, so the endpoint won’t be queried before a few seconds (this allows to work well even with old Consul versions with patch [#3899](https://github.com/hashicorp/consul/pull/3899)).
|
60
|
+
* Some endpoints [are configured specifically](https://github.com/criteo/consul-templaterb/blob/master/bin/consul-templaterb#L61) to avoid too many calls (for instance, we don’t consider we add a new DC every 30 seconds).
|
61
|
+
* Some specific behavior exists for non-existing service (was causing lots of issues with Prometheus for instance, see [Prometheus PR #3814](https://github.com/prometheus/prometheus/pull/3814)), so we will work on the old Consul version with our [PR #4810](https://github.com/hashicorp/consul/pull/4810).
|
62
|
+
* By default, when some errors occur, try to limit the number of calls to avoid increasing the pressure on the cluster.
|
63
|
+
* Collect statistics that can be used in your templates. For instance, the [Prometheus exporter uses it to know whether a service is unstable or not](https://github.com/criteo/consul-templaterb/blob/master/samples/metrics.erb#L88).
|
64
|
+
|
65
|
+
## The hard part about async I/Os with Ruby
|
66
|
+
|
67
|
+
This program was my first project dealing with complex code with important performance issues. I first discovered that Ruby async I/O code is really hard and most libraries are not dealing with it properly. Many libraries just lie (by creating threads and pretending to be async). The only library I found handling it correctly was [eventmachine](https://github.com/eventmachine/eventmachine/).
|
68
|
+
|
69
|
+
While the library is petty mature (the project is quite old, 9 years old), I had lots of issues very quickly. I spent quite some time making it work on my targeted operating systems (Linux, Mac OS, Windows).
|
70
|
+
|
71
|
+
* On Windows, installation is painful (see https://github.com/criteo/consul-templaterb/#quick-install-on-windows) and due to limitation into the Ruby runtime, it does not support more than 2048 file descriptors with native implementation (while this works with Linux Subsystem on Windows 10).
|
72
|
+
* On all Operating Systems, opening lots of HTTP connections concurrently led to crashes (see https://github.com/igrigorik/em-http-request/issues/315, [#604](https://github.com/eventmachine/eventmachine/issues/604), [#770](https://github.com/eventmachine/eventmachine/issues/770), and [#824](https://github.com/eventmachine/eventmachine/issues/824)) that took me lots of time to debug (including native C code debugging), quite frustrating for such mature library (the good news is that eventmachine is now fixed thanks to [consul-templaterb](https://github.com/criteo/consul-templaterb/)).
|
73
|
+
|
74
|
+
While those problems are now solved and you can use it properly, it took me weeks to solve.
|
75
|
+
|
76
|
+
Furthermore, the lack of HTTP/2 is a bit worrisome today as more and more systems allow it (it would avoid the lack of file descriptors on very large clusters).
|
77
|
+
|
78
|
+
## Some fun features
|
79
|
+
|
80
|
+
* All templates in the repository are unit tested for each release of consul-templaterb, so all samples are working on your local installation.
|
81
|
+
* You can generate lots of formats: [Services in XML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.xml.erb), [JSON](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb), [YAML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb#L58) using Hash and just dumping in the right format!
|
82
|
+
* You can load templates from Consul Key/Value store, so you can change templates dynamically from Consul K/V Store (we use this feature in for our Prometheus configurations for instance as described in “[Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)”).
|
83
|
+
* You can also query Vault / some JSON APIs very easily.
|
84
|
+
* You can have a very high-performance UI (see [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)) easily for your own use-cases. This UI is generated in real-time as static files and can be served by your favorite webserver (we do use nginx on our side), so it scales indefinitely if you have lots of users.
|
85
|
+
* Huge Performance gap with consul-template: in version 1.0 of Consul, consul-template was using 800Mb/s to scrape all services in one of large DCs, consul-templaterb is using less than 100kb/s (800x reduction!) to generate the full UI of Consul and fetch all services in our largest DC now.
|
86
|
+
* Good abstraction with impressive performance compared to other projects such as [consult](https://github.com/veracross/consult) (no need to deal with kind of “magic TTL”) or consul-template itself: the APIs are simpler, you have to deal less with Consul specific optimizations and the performance is better.
|
87
|
+
|
88
|
+
# Usages at Criteo (non-exhaustive)
|
89
|
+
|
90
|
+
We use this executable for many things at Criteo (see “Inversion of Control with Consul”):
|
91
|
+
|
92
|
+
* Automatic Alerting / Prometheus configuration
|
93
|
+
* [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)s and its Consul Timeline (keep track of the history of all changes for all services)
|
94
|
+
* List of assets per DC/Racks
|
95
|
+
* Monitoring of services/racks
|
96
|
+
* Ownership enforcement
|
97
|
+
* Live generation of configuration for several programs/services
|
98
|
+
|
99
|
+
# Try it quickly
|
100
|
+
|
101
|
+
If you don’t want to bother configuring and tuning it, you can try it very quickly with our docker images: In 2 minutes (download included), you have a full scalable UI with Consul, serving static files with nginx and keeping a history of all changes on your services live.
|
102
|
+
|
103
|
+
![Consul UI in action, scalable UI with excellent performance](images/consul-ui_001.png)
|
104
|
+
|
105
|
+
# More on the subject
|
106
|
+
|
107
|
+
* The [consul-templaterb API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md) to write templates, with links to real-world examples in the samples directory.
|
108
|
+
* [INTERNALS.md](../INTERNALS.md) on Github (describes the objects in the code)
|
109
|
+
* Slides at HashiConf ’19: “[Consul Templates on Steroids](https://fr.slideshare.net/PierreSouchay/2019-hashiconf-consultemplaterb)”
|
110
|
+
* A video introduction of consul-templaterb:
|
111
|
+
|
112
|
+
[![Quick presentation of consul-templaterb](images/consul-ui_001.png)](https://youtu.be/zLzrLGLLl4Q)
|
113
|
+
|
114
|
+
# Other articles from the Discovery team
|
115
|
+
|
116
|
+
* [Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)
|
117
|
+
* [Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)
|
118
|
+
* [Anatomy of a bug: When Consul has too much to deliver for the big day](https://medium.com/criteo-labs/anatomy-of-a-bug-when-consul-has-too-much-to-deliver-for-the-big-day-4904d19a46a4)
|
119
|
+
* [Inversion of Control for the Infrastructure with Consul](https://medium.com/criteo-labs/inversion-of-control-for-the-infrastructure-with-consul-b894877b33a4)
|
120
|
+
* [Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)
|
121
|
+
|
122
|
+
Pierre Souchay, 2020-03-03
|
123
|
+
Also published on https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
|
Binary file
|
@@ -0,0 +1,3 @@
|
|
1
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
2
|
+
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
3
|
+
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="896px" height="455px" viewBox="-0.5 -0.5 896 455" content="<mxfile host="www.draw.io" modified="2020-03-02T13:27:25.463Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:73.0) Gecko/20100101 Firefox/73.0" etag="Qxj7PV3jLHrlLdg27esA" version="12.7.9" type="device"><diagram name="Page-1" id="58cdce13-f638-feb5-8d6f-7d28b1aa9fa0">7VzZkps4FP0aP3YXIDY/9pqpTDI1k04myVNKBtlmgpEL5HY7Xz8SSCwSm21wO4n7IUELAt977rmLZE/A3erlTQzXy/fYR+HE0PyXCbifGIauG4D+x3p2WY9jT7OORRz4fFLR8RT8QLxT472bwEdJZSLBOCTButrp4ShCHqn0wTjG2+q0OQ6rT13DBVI6njwYqr2fA58sxefStGLgDxQslvzRrsUHZtD7vojxJuLPmxhgnv5lwyso1uLzkyX08bbUBR4m4C7GmGRXq5c7FDLZCrFl9z02jObvHaOI9LkBuMZsqs8sS9dnvjZ3rvgKzzDcIPER0hclOyEcugLVA23cLskqpH06vaQfZM3GEwJj8kQgYePzIAzvcIjj9EagpX9sMonxd1Qamc/5SCo65BeLZtJhQx5eBR6/DuEMhbe5rMVKEY7Sx+KIPMJVEDL0/YtiH0aQd3Oo6QZv170dDINFRPs8KkNEB2/9IKYgCzDrTPCGqfBWFbSQHIoJeil1ccG/QXiFSLyjU8Qo4CDYibaVtbcF5MSUZQlsog9ykC/ylQtF0wuu6556t2v0bock/fTP9HLBLj8gqpyYTvuIVuuQ6ZjPoY8rTau5cxvQT0T5ABLIwBR7iZg0i+Xb5MUk+FUhsl0GBD2tocdGt5SNqrCEscdVbpgqIJlheu2APBMIDgA3w5TgJjioBLe8r4w3ewi8eT6yfei7tgl1B8IrU6UVn1Iwb+KYLPECRzB8KHrLem2XPoXpDfMCbKU1irIeLne3Vd099Sb0E6I5wyh6CcgXhotri7e+cpSw6/uXcmPHG40apQwTe1wm2nL+58dvH9/Opt/evFt9fvjw9p/vVzr3pZRpF0hwQoM9M6G2giNG1I6D56rfq9N0eiuVKtyVJqxxEJGktPLfrKPA3NSoYs6RvJE0HWht0+lF9vyGZ+kSvnUJt5lg+V0tb2FKL21a0kKZ4JWFUhvIRXsYDU8VGi6oVoswSWMe2oEIZcALMw7DjI72msyoRGBaGwaCJDVa6O8u6h9I/aZjVdRvuK/pGK1Wx4jCGd7+HD4xopL4Ijwfa3wtPCRrFm4xbQm/KHypXvakuV89wpc2cW3Zjbb624F9qeJ38iRS+J2p1cuBHeB3DAhmnmFp0Jtrtu74Qha/Rjh2GBx0FQ7jR1V9aUJRmPASv4bG2sjiMG2eR/xrypGF1R4Ad8zviIBNp/7uEQJXFY/gODzmKZRK+wIbuoqNM0Rxi1sb0nkJ++/0XuZpnBewq9gDZr/saxDsmUrE/BceCI6VjF6vUFWBTpHtnxkcGyHXBtRecKyPk3o4UAHbseFoTSUqtKVYakwqbA/he2OvJgJucZMX7HVhb5wwft+QwJaR2RESdMyvhgQHwLdNnO27Ae+DJAmiBZ2HY1Gd6rkdMEfEW7IKlrZD/e+KUYYDKunHvpsFS7yabZLuesikvmhQg8aWOkJVU666naPX7efoQ2zo1KrRUNT4FSWKjOgnJlJxSK6yMLkEHgxv+MAq8P2UtqhKgh9wli7FLJijn65r3U6se7bWhuCEE8Uwcral+mzdvplZI2djLDGDPtbyyCA/ybfAYLKLvN7In8EktRYc0X9CmJD0lV7OYQ8N6b6FnDoXMrUdAO3J2KXChpLgkaZcLQmCaY0tj1USbAvjSxj7HAdpRRhvyHpDxlTxz1YNPkr18japccrNgFrVO2OkMnnwVZPKnGF96IhwUor3DosvzR7xpT1OJmNKuxNGv6pw90LT0apD9TGl28dNflr7+5wrybxpz8mtPndvExMnnmBmCLfeJn7O+ba45R3Ga975HyJkx1HOwiLJTgtr1J2qPV5rAHRgnSsPxQHVE4oZ1bPYnN7cwQHtlpUvOLxJuScwqaN4V8XrfRa75dnLJaray7W6r7jPWqvhI3e9fvea9SA0AFQaqM9lnVHcqymBEshZYl/3CqTsX1mowb3uWz+SC+0mjwca36t9/kj1I/Uc043PEthVXjMqijmXzGUgepU0fdJjTPU2qx5lymAQoW1RCqGBEPep5xWFKds7Q8df+qk4tu7EaL3CQCvCtGvNAKACsitBe0fysCOl3QC49GGHUbFr2dVAw6QvXv4DvZi5WFdMxPN5gkZJjgy14n6zXoepg6P5UBisglFLPL9DvGlar13EM4zjAs7zCh37E1Q374yzC6xLJY++hxKUhQy9I0ocKLhTXrjjwHzH/HGCO0Pd73hi3/oSXjyL7zIXf5Io79ckL/lAzatvQRhHfmfnki4PEMoZffPlhkhu4HTZBAdWo0/FqM6e6XLH/JEY9cgTQ2dmIwec4G/bijrRFk/vYGUcy9ItySDkw7t9LUteyOx59G1fy7KA9Bz3FJbiKLHHA8V2nJ4IkiOMvmd/9g8nBvDvhvRtD1BzXMitce/uaO7dHci9N+8zX9x7Jwk5PUnIPW/3DqRyjrLQQCRkSKcjgX6KBEitbtfsNrMj6L22msc9f9dgYMUPk7SwGf95Fb7+JD8KU/l9kWYuaeS+K+1a10XQcyhkxy/KiXOZIyY82rVz7t8i+GkocaxvtmgNifi5pjy6dHjxJCkMUAvYpz9r/CpcB7TJWXMdbRa/J5VNL360Czz8Dw==</diagram></mxfile>"><defs/><g><ellipse cx="60" cy="89" rx="11" ry="11" fill="#000000" stroke="#ff0000" transform="rotate(90,60,89)" pointer-events="all"/><rect x="155" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 156px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Render Template</div><div>with data srcs<br /></div></div></div></div></foreignObject><text x="215" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Render Template...</text></switch></g><path d="M 835 59 L 835 19 Q 835 9 825 9 L 225 9 Q 215 9 215 19 L 215 56.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 210.5 48.88 L 215 57.88 L 219.5 48.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="615" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 616px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Template not completed</div></div></div></foreignObject><text x="675" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Template not complet...</text></switch></g><rect x="390" y="219" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 249px; margin-left: 391px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Template is ready</div></div></div></foreignObject><text x="450" y="253" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Template is ready</text></switch></g><path d="M 735 89 L 745 89 Q 755 89 763.88 89 L 772.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 764.88 93.5 L 773.88 89 L 764.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 75 89 L 152.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 144.88 93.5 L 153.88 89 L 144.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 275 89 L 305 89 Q 315 89 325 89 L 382.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 374.88 93.5 L 383.88 89 L 374.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 390 249 L 342.5 249 Q 332.5 249 332.5 239 L 332.5 199 Q 332.5 189 322.5 189 L 277.24 189" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 285.12 184.5 L 276.12 189 L 285.12 193.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 450 154 L 450 216.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 445.5 208.88 L 450 217.88 L 454.5 208.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe flex-start; width: 1px; height: 1px; padding-top: 187px; margin-left: 452px;"><div style="box-sizing: border-box; font-size: 0; text-align: left; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">No</div></div></div></foreignObject><text x="452" y="191" fill="#000000" font-family="Verdana" font-size="12px">No</text></switch></g><path d="M 515 89 L 595 89 Q 605 89 608.88 89 L 612.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 604.88 93.5 L 613.88 89 L 604.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 450 24 L 515 89 L 450 154 L 385 89 Z" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 128px; height: 1px; padding-top: 89px; margin-left: 386px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Missing or not</div><div>fetched yet</div><div>resources?</div></div></div></div></foreignObject><text x="450" y="93" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Missing or not...</text></switch></g><rect x="535" y="74" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 84px; margin-left: 555px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Yes</div></div></div></foreignObject><text x="555" y="88" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Yes</text></switch></g><rect x="390" y="334" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 364px; margin-left: 391px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Fetch data async</div><div>based on last idx<br /></div></div></div></div></foreignObject><text x="450" y="368" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Fetch data async...</text></switch></g><rect x="155" y="159" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 189px; margin-left: 156px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Write output</div></div></div></foreignObject><text x="215" y="193" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Write output</text></switch></g><path d="M 215 159 L 215 121.24" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 219.5 129.12 L 215 120.12 L 210.5 129.12" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 63.23 222.82 L 56.77 215.18 L 149.86 136.52 L 143.4 128.89 L 176 120.98 L 162.77 151.8 L 156.31 144.16 Z" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 169px; margin-left: 117px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; "><div>Update</div><div>data</div><div>async</div></div></div></div></foreignObject><text x="117" y="172" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">Update...</text></switch></g><rect x="0" y="219" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 249px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Data fetched</div></div></div></foreignObject><text x="60" y="253" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Data fetched</text></switch></g><path d="M 390 364 L 285 364 Q 275 364 265 364 L 247.24 364" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 255.12 359.5 L 246.12 364 L 255.12 368.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="775" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 776px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Add missing resources</div></div></div></foreignObject><text x="835" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Add missing resources</text></switch></g><path d="M 830 119 L 840 119 L 840 304 L 850 304 L 835 334 L 820 304 L 830 304 Z" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 229px; margin-left: 765px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">Add new data to fetch</div></div></div></foreignObject><text x="765" y="232" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">Add new data to fetch</text></switch></g><rect x="0" y="394" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 424px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Apply ratelimit</div></div></div></foreignObject><text x="60" y="428" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Apply ratelimit</text></switch></g><path d="M 60 279 L 60 299 Q 60 309 60 319 L 60 391.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 55.5 383.88 L 60 392.88 L 64.5 383.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="775" y="334" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 364px; margin-left: 776px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Start fetching new resources</div></div></div></foreignObject><text x="835" y="368" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Start fetching new r...</text></switch></g><path d="M 775 364 L 685 364 Q 675 364 665 364 L 512.24 364" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 520.12 359.5 L 511.12 364 L 520.12 368.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 120 424 L 440 424 Q 450 424 450 414 L 450 396.24" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 454.5 404.12 L 450 395.12 L 445.5 404.12" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 205 324 L 245 364 L 205 404 L 165 364 Z" fill="#e1d5e7" stroke="#9673a6" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 364px; margin-left: 166px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Error?</div></div></div></foreignObject><text x="205" y="368" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Error?</text></switch></g><path d="M 205 324 L 205 259 Q 205 249 195 249 L 122.24 249" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 130.12 244.5 L 121.12 249 L 130.12 253.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 254px; margin-left: 206px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; "><div>No</div></div></div></div></foreignObject><text x="206" y="257" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">No</text></switch></g><path d="M 165 364 L 100 364 Q 90 364 90 374 L 90 391.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 85.5 383.88 L 90 392.88 L 94.5 383.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 365px; margin-left: 119px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">Yes</div></div></div></foreignObject><text x="119" y="368" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Yes</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
|
@@ -50,10 +50,15 @@ module Consul
|
|
50
50
|
end
|
51
51
|
end
|
52
52
|
|
53
|
-
def create(path)
|
53
|
+
def create(path, agent: nil)
|
54
54
|
return self unless @paths[path.to_sym]
|
55
55
|
|
56
|
-
|
56
|
+
base_url = ch(path, :base_url)
|
57
|
+
if agent
|
58
|
+
agent = "http://#{agent}" unless agent.start_with? 'http', 'https'
|
59
|
+
base_url = agent
|
60
|
+
end
|
61
|
+
ConsulConfiguration.new(base_url: base_url,
|
57
62
|
debug: ch(path, :debug),
|
58
63
|
token: ch(path, :token),
|
59
64
|
retry_duration: ch(path, :retry_duration),
|
@@ -128,8 +133,8 @@ module Consul
|
|
128
133
|
# So, it basically performs all the optimizations to keep updated with Consul internal state.
|
129
134
|
class ConsulEndpoint
|
130
135
|
attr_reader :conf, :path, :x_consul_index, :queue, :stats, :last_result, :enforce_json_200, :start_time, :default_value, :query_params
|
131
|
-
def initialize(conf, path, enforce_json_200 = true, query_params = {}, default_value = '[]')
|
132
|
-
@conf = conf.create(path)
|
136
|
+
def initialize(conf, path, enforce_json_200 = true, query_params = {}, default_value = '[]', agent = nil)
|
137
|
+
@conf = conf.create(path, agent: agent)
|
133
138
|
@default_value = default_value
|
134
139
|
@path = path
|
135
140
|
@queue = EM::Queue.new
|
@@ -48,19 +48,23 @@ module Consul
|
|
48
48
|
end
|
49
49
|
|
50
50
|
# Return the coordinates of datacenters
|
51
|
-
def datacenters(dc: nil)
|
51
|
+
def datacenters(dc: nil, agent: nil)
|
52
52
|
path = '/v1/coordinate/datacenters'
|
53
53
|
query_params = {}
|
54
54
|
query_params[:dc] = dc if dc
|
55
|
-
@endp_manager.create_if_missing(path, query_params
|
55
|
+
@endp_manager.create_if_missing(path, query_params, agent: agent) do
|
56
|
+
ConsulTemplateNodes.new(ConsulEndpoint.new(@endp_manager.consul_conf, path, true, query_params, '[]', agent: agent))
|
57
|
+
end
|
56
58
|
end
|
57
59
|
|
58
60
|
# Returns the coordinates for all nodes of DC
|
59
|
-
def nodes(dc: nil)
|
61
|
+
def nodes(dc: nil, agent: nil)
|
60
62
|
path = '/v1/coordinate/nodes'
|
61
63
|
query_params = {}
|
62
64
|
query_params[:dc] = dc if dc
|
63
|
-
@endp_manager.create_if_missing(path, query_params
|
65
|
+
@endp_manager.create_if_missing(path, query_params, agent: agent) do
|
66
|
+
ConsulTemplateNodes.new(ConsulEndpoint.new(@endp_manager.consul_conf, path, true, query_params, '[]', agent: agent))
|
67
|
+
end
|
64
68
|
end
|
65
69
|
|
66
70
|
# Computes the RTT between 2 nodes
|
@@ -124,7 +128,7 @@ module Consul
|
|
124
128
|
end
|
125
129
|
|
126
130
|
# https://www.consul.io/api/health.html#list-nodes-for-service
|
127
|
-
def service(name, dc: nil, passing: false, tag: nil)
|
131
|
+
def service(name, dc: nil, passing: false, tag: nil, agent: nil)
|
128
132
|
raise 'You must specify a name for a service' if name.nil?
|
129
133
|
|
130
134
|
path = "/v1/health/service/#{name}"
|
@@ -132,70 +136,70 @@ module Consul
|
|
132
136
|
query_params[:dc] = dc if dc
|
133
137
|
query_params[:passing] = passing if passing
|
134
138
|
query_params[:tag] = tag if tag
|
135
|
-
create_if_missing(path, query_params) { ConsulTemplateService.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
139
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateService.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
136
140
|
end
|
137
141
|
|
138
142
|
# https://www.consul.io/api/health.html#list-checks-for-service
|
139
|
-
def checks_for_service(name, dc: nil, passing: false)
|
143
|
+
def checks_for_service(name, dc: nil, passing: false, agent: nil)
|
140
144
|
raise 'You must specify a name for a service' if name.nil?
|
141
145
|
|
142
146
|
path = "/v1/health/checks/#{name}"
|
143
147
|
query_params = {}
|
144
148
|
query_params[:dc] = dc if dc
|
145
149
|
query_params[:passing] = passing if passing
|
146
|
-
create_if_missing(path, query_params) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
150
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
147
151
|
end
|
148
152
|
|
149
153
|
# https://www.consul.io/api/health.html#list-checks-for-node
|
150
|
-
def checks_for_node(name, dc: nil, passing: false)
|
154
|
+
def checks_for_node(name, dc: nil, passing: false, agent: nil)
|
151
155
|
raise 'You must specify a name for a service' if name.nil?
|
152
156
|
|
153
157
|
path = "/v1/health/node/#{name}"
|
154
158
|
query_params = {}
|
155
159
|
query_params[:dc] = dc if dc
|
156
160
|
query_params[:passing] = passing if passing
|
157
|
-
create_if_missing(path, query_params) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
161
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateChecks.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
158
162
|
end
|
159
163
|
|
160
164
|
# https://www.consul.io/api/catalog.html#list-nodes
|
161
|
-
def nodes(dc: nil)
|
165
|
+
def nodes(dc: nil, agent: nil)
|
162
166
|
path = '/v1/catalog/nodes'
|
163
167
|
query_params = {}
|
164
168
|
query_params[:dc] = dc if dc
|
165
|
-
create_if_missing(path, query_params) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
169
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
166
170
|
end
|
167
171
|
|
168
172
|
# https://www.consul.io/api/catalog.html#list-services-for-node
|
169
|
-
def node(name_or_id, dc: nil)
|
173
|
+
def node(name_or_id, dc: nil, agent: nil)
|
170
174
|
path = "/v1/catalog/node/#{name_or_id}"
|
171
175
|
query_params = {}
|
172
176
|
query_params[:dc] = dc if dc
|
173
|
-
create_if_missing(path, query_params) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
|
177
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateNodes.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
|
174
178
|
end
|
175
179
|
|
176
180
|
# https://www.consul.io/api/agent.html#read-configuration
|
177
|
-
def agent_self
|
181
|
+
def agent_self(agent: nil)
|
178
182
|
path = '/v1/agent/self'
|
179
183
|
query_params = {}
|
180
184
|
default_value = '{"Config":{}, "Coord":{}, "Member":{}, "Meta":{}, "Stats":{}}'
|
181
|
-
create_if_missing(path, query_params) { ConsulAgentSelf.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
|
185
|
+
create_if_missing(path, query_params, agent: agent) { ConsulAgentSelf.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
|
182
186
|
end
|
183
187
|
|
184
188
|
# https://www.consul.io/api/agent.html#view-metrics
|
185
|
-
def agent_metrics
|
189
|
+
def agent_metrics(agent: nil)
|
186
190
|
path = '/v1/agent/metrics'
|
187
191
|
query_params = {}
|
188
192
|
default_value = '{"Gauges":[], "Points":[], "Member":{}, "Counters":[], "Samples":{}}'
|
189
|
-
create_if_missing(path, query_params) { ConsulAgentMetrics.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
|
193
|
+
create_if_missing(path, query_params, agent: agent) { ConsulAgentMetrics.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
|
190
194
|
end
|
191
195
|
|
192
196
|
# https://www.consul.io/api/agent.html#list-members
|
193
|
-
def agent_members(wan: false)
|
197
|
+
def agent_members(wan: false, agent: nil)
|
194
198
|
path = '/v1/agent/members'
|
195
199
|
query_params = {}
|
196
200
|
query_params['wan'] = true if wan
|
197
201
|
default_value = '[]'
|
198
|
-
create_if_missing(path, query_params) { ConsulTemplateMembers.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
|
202
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateMembers.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value)) }
|
199
203
|
end
|
200
204
|
|
201
205
|
# Return a param of template
|
@@ -212,51 +216,57 @@ module Consul
|
|
212
216
|
end
|
213
217
|
|
214
218
|
# https://www.consul.io/api/catalog.html#list-services
|
215
|
-
def services(dc: nil, tag: nil)
|
219
|
+
def services(dc: nil, tag: nil, agent: nil)
|
216
220
|
path = '/v1/catalog/services'
|
217
221
|
query_params = {}
|
218
222
|
query_params[:dc] = dc if dc
|
219
223
|
# Tag filtering is performed on client side
|
220
224
|
query_params[:tag] = tag if tag
|
221
|
-
create_if_missing(path, query_params) { ConsulTemplateServices.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
|
225
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateServices.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '{}')) }
|
222
226
|
end
|
223
227
|
|
224
228
|
# https://www.consul.io/api/catalog.html#list-datacenters
|
225
|
-
def datacenters
|
229
|
+
def datacenters(agent: nil)
|
226
230
|
path = '/v1/catalog/datacenters'
|
227
231
|
query_params = {}
|
228
|
-
create_if_missing(path, query_params) { ConsulTemplateDatacenters.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
232
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateDatacenters.new(ConsulEndpoint.new(consul_conf, path, true, query_params, '[]')) }
|
229
233
|
end
|
230
234
|
|
231
235
|
# https://www.consul.io/api/kv.html#read-key
|
232
|
-
def kv(name = nil, dc: nil, keys: nil, recurse: false)
|
236
|
+
def kv(name = nil, dc: nil, keys: nil, recurse: false, agent: nil)
|
233
237
|
path = "/v1/kv/#{name}"
|
234
238
|
query_params = {}
|
235
239
|
query_params[:dc] = dc if dc
|
236
240
|
query_params[:recurse] = recurse if recurse
|
237
241
|
query_params[:keys] = keys if keys
|
238
242
|
default_value = '[]'
|
239
|
-
create_if_missing(path, query_params) { ConsulTemplateKV.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value), name) }
|
243
|
+
create_if_missing(path, query_params, agent: agent) { ConsulTemplateKV.new(ConsulEndpoint.new(consul_conf, path, true, query_params, default_value), name) }
|
240
244
|
end
|
241
245
|
|
242
|
-
def secrets(path = '')
|
246
|
+
def secrets(path = '', agent: nil)
|
243
247
|
raise "You need to provide a vault token to use 'secret' keyword" if vault_conf.token.nil?
|
244
248
|
|
245
249
|
path = "/v1/#{path}".gsub(%r{/{2,}}, '/')
|
246
250
|
query_params = { list: 'true' }
|
247
|
-
create_if_missing(path, query_params,
|
248
|
-
|
251
|
+
create_if_missing(path, query_params,
|
252
|
+
fail_fast_errors: vault_conf.fail_fast_errors,
|
253
|
+
max_consecutive_errors_on_endpoint: vault_conf.max_consecutive_errors_on_endpoint,
|
254
|
+
agent: agent) do
|
255
|
+
ConsulTemplateVaultSecretList.new(VaultEndpoint.new(vault_conf, path, 'GET', true, query_params, JSON.generate(data: { keys: [] }), agent: agent))
|
249
256
|
end
|
250
257
|
end
|
251
258
|
|
252
|
-
def secret(path = '', post_data = nil)
|
259
|
+
def secret(path = '', post_data = nil, agent: nil)
|
253
260
|
raise "You need to provide a vault token to use 'secret' keyword" if vault_conf.token.nil?
|
254
261
|
|
255
262
|
path = "/v1/#{path}".gsub(%r{/{2,}}, '/')
|
256
263
|
query_params = {}
|
257
264
|
method = post_data ? 'POST' : 'GET'
|
258
|
-
create_if_missing(path, query_params,
|
259
|
-
|
265
|
+
create_if_missing(path, query_params,
|
266
|
+
fail_fast_errors: vault_conf.fail_fast_errors,
|
267
|
+
max_consecutive_errors_on_endpoint: vault_conf.max_consecutive_errors_on_endpoint,
|
268
|
+
agent: agent) do
|
269
|
+
ConsulTemplateVaultSecret.new(VaultEndpoint.new(vault_conf, path, method, true, query_params, JSON.generate(data: {}), agent: agent))
|
260
270
|
end
|
261
271
|
end
|
262
272
|
|
@@ -372,10 +382,10 @@ module Consul
|
|
372
382
|
VaultEndpoint.new(vault_conf, path, :POST, {}, {})
|
373
383
|
end
|
374
384
|
|
375
|
-
def create_if_missing(path, query_params, fail_fast_errors
|
385
|
+
def create_if_missing(path, query_params, fail_fast_errors: @fail_fast_errors, max_consecutive_errors_on_endpoint: @max_consecutive_errors_on_endpoint, agent: nil)
|
376
386
|
fqdn = path.dup
|
377
387
|
query_params.each_pair do |k, v|
|
378
|
-
fqdn = "#{fqdn}&#{k}=#{v}"
|
388
|
+
fqdn = "#{agent}#{fqdn}&#{k}=#{v}"
|
379
389
|
end
|
380
390
|
tpl = @endpoints[fqdn]
|
381
391
|
unless tpl
|
@@ -46,10 +46,15 @@ module Consul
|
|
46
46
|
end
|
47
47
|
end
|
48
48
|
|
49
|
-
def create(path)
|
49
|
+
def create(path, agent: nil)
|
50
50
|
return self unless @paths[path.to_sym]
|
51
51
|
|
52
|
-
|
52
|
+
base_url = ch(path, :base_url)
|
53
|
+
if agent
|
54
|
+
agent = "http://#{agent}" unless agent.start_with? 'http', 'https'
|
55
|
+
base_url = agent
|
56
|
+
end
|
57
|
+
VaultConfiguration.new(base_url: base_url,
|
53
58
|
debug: ch(path, :debug),
|
54
59
|
token: ch(path, :token),
|
55
60
|
retry_duration: ch(path, :retry_duration),
|
@@ -117,8 +122,8 @@ module Consul
|
|
117
122
|
class VaultEndpoint
|
118
123
|
attr_reader :conf, :path, :http_method, :queue, :stats, :last_result, :enforce_json_200, :start_time, :default_value, :query_params
|
119
124
|
|
120
|
-
def initialize(conf, path, http_method = 'GET', enforce_json_200 = true, query_params = {}, default_value = '{}', post_data = {})
|
121
|
-
@conf = conf.create(path)
|
125
|
+
def initialize(conf, path, http_method = 'GET', enforce_json_200 = true, query_params = {}, default_value = '{}', post_data = {}, agent: nil)
|
126
|
+
@conf = conf.create(path, agent: agent)
|
122
127
|
@default_value = default_value
|
123
128
|
@path = path
|
124
129
|
@http_method = http_method
|
data/lib/consul/async/version.rb
CHANGED
@@ -9,7 +9,7 @@
|
|
9
9
|
require 'set'
|
10
10
|
|
11
11
|
# Services to hide
|
12
|
-
services_blacklist_raw = (ENV['EXCLUDE_SERVICES'] || '.*netsvc-probe.*,.*consul-probed.*').split(',')
|
12
|
+
services_blacklist_raw = (ENV['EXCLUDE_SERVICES'] || '.*netsvc-probe.*,.*consul-probed.*,.lbl7.*').split(',')
|
13
13
|
services_blacklist = services_blacklist_raw.map { |v| Regexp.new(v) }
|
14
14
|
|
15
15
|
num_services={}
|
@@ -0,0 +1,69 @@
|
|
1
|
+
__________________________________________________________________
|
2
|
+
| DC | Services |______________Instances________________| Nodes |
|
3
|
+
| | | Passing | Warning | Critic | Total | |
|
4
|
+
|-------+----------+---------+---------+---------+---------+-------|
|
5
|
+
<%
|
6
|
+
# This template list all instances on all DCs
|
7
|
+
# And aggregates a list of Services, Services Instances/status
|
8
|
+
# And nodes.
|
9
|
+
require 'set'
|
10
|
+
|
11
|
+
|
12
|
+
def find_best_agent_for_dc(dc)
|
13
|
+
agent = service('consul-relay', passing: true, dc:dc).first
|
14
|
+
agent = service('consul-agent-http', passing: true, dc:dc).first unless agent
|
15
|
+
service('consul', passing: true, dc:dc).first unless agent
|
16
|
+
return nil unless agent
|
17
|
+
port = agent['Service']['Port'] || 8500
|
18
|
+
"http://#{agent.service_address}:#{port}"
|
19
|
+
end
|
20
|
+
|
21
|
+
# Services to hide
|
22
|
+
services_blacklist_raw = (ENV['EXCLUDE_SERVICES'] || '.*netsvc-probe.*,.*consul-probed.*').split(',')
|
23
|
+
services_blacklist = services_blacklist_raw.map { |v| Regexp.new(v) }
|
24
|
+
|
25
|
+
num_services={}
|
26
|
+
num_instances={}
|
27
|
+
all_states = ['passing', 'warning', 'critical', 'total']
|
28
|
+
distinct_services=Set.new
|
29
|
+
datacenters().each do |dc|
|
30
|
+
next unless find_best_agent_for_dc(dc)
|
31
|
+
num_services[dc] = 0
|
32
|
+
num_instances[dc] = {
|
33
|
+
'passing' => 0,
|
34
|
+
'warning' => 0,
|
35
|
+
'critical' => 0,
|
36
|
+
'total' => 0
|
37
|
+
}
|
38
|
+
services(dc:dc, agent: find_best_agent_for_dc(dc)).each do |service_name, tags|
|
39
|
+
next if services_blacklist.any? {|r| r.match(service_name)}
|
40
|
+
distinct_services.add(service_name)
|
41
|
+
num_services[dc]+=1
|
42
|
+
service(service_name, dc:dc, agent: find_best_agent_for_dc(dc)).each do |snode|
|
43
|
+
num_instances[dc][snode.status]+=1
|
44
|
+
num_instances[dc]['total']+=1
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
48
|
+
num_instances_total={
|
49
|
+
'passing' => 0,
|
50
|
+
'warning' => 0,
|
51
|
+
'critical' => 0,
|
52
|
+
'total' => 0
|
53
|
+
}
|
54
|
+
num_nodes_total=0
|
55
|
+
num_services.each do |dc, num_services_for_dc|
|
56
|
+
all_states.each do |s|
|
57
|
+
num_instances_total[s] += num_instances[dc][s]
|
58
|
+
end
|
59
|
+
num_nodes=nodes(dc:dc, agent: find_best_agent_for_dc(dc)).count
|
60
|
+
num_nodes_total+=num_nodes
|
61
|
+
%>| <%= dc.rjust(5) %> | <%= num_services_for_dc.to_s.rjust(8) %> |<% all_states.each do |status|
|
62
|
+
%> <%= num_instances[dc][status].to_s.rjust(7) %> |<% end %> <%= num_nodes.to_s.rjust(5) %> |
|
63
|
+
<%
|
64
|
+
end
|
65
|
+
%>|-------+----------+---------+---------+---------+---------+-------|
|
66
|
+
| TOTAL | <%= distinct_services.count.to_s.rjust(8) %> |<% all_states.each do |status|
|
67
|
+
%> <%= num_instances_total[status].to_s.rjust(7) %> |<%
|
68
|
+
end %> <%= num_nodes_total.to_s.rjust(5) %> |
|
69
|
+
'_______|__________|_________|_________|_________|_________|_______'
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: consul-templaterb
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.26.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- SRE Core Services
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-03-05 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: em-http-request
|
@@ -180,6 +180,9 @@ files:
|
|
180
180
|
- bin/consul-templaterb
|
181
181
|
- consul-templaterb.gemspec
|
182
182
|
- docker-nginx-conf/nginx.conf
|
183
|
+
- docs/article-06_Template-based_discovery_with_consul-templaterb.md
|
184
|
+
- docs/images/consul-templaterb.png
|
185
|
+
- docs/images/consul-templaterb.svg
|
183
186
|
- docs/images/consul-ui_001.png
|
184
187
|
- lib/consul/async/consul_endpoint.rb
|
185
188
|
- lib/consul/async/consul_template.rb
|
@@ -193,6 +196,7 @@ files:
|
|
193
196
|
- lib/consul/async/vault_endpoint.rb
|
194
197
|
- lib/consul/async/version.rb
|
195
198
|
- samples/all_services.txt.erb
|
199
|
+
- samples/all_services_multi_agents.txt.erb
|
196
200
|
- samples/all_templates.erb
|
197
201
|
- samples/consul-ui/README.md
|
198
202
|
- samples/consul-ui/common/footer.html.erb
|