consul-templaterb 1.28.0 → 1.31.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +63 -30
- data/TemplateAPI.md +1 -1
- data/bin/consul-templaterb +5 -0
- data/lib/consul/async/consul_endpoint.rb +28 -2
- data/lib/consul/async/version.rb +1 -1
- data/samples/consul-ui/.preferences.rb.samples +1 -0
- data/samples/consul-ui/README.md +6 -0
- data/samples/consul-ui/consul_services.json.erb +14 -1
- data/samples/consul-ui/decorators.js.erb +7 -0
- data/samples/consul-ui/js/nodes.js +1 -1
- data/samples/consul-ui/js/service.js +1 -1
- data/samples/consul-ui/js/utils.js +8 -5
- metadata +32 -49
- data/.gitignore +0 -41
- data/.rspec +0 -2
- data/.rubocop.yml +0 -57
- data/.ruby_app +0 -0
- data/.travis.yml +0 -19
- data/CODE_OF_CONDUCT.md +0 -76
- data/CONTRIBUTING.md +0 -25
- data/Dockerfile +0 -15
- data/Gemfile +0 -5
- data/INTERNALS.md +0 -49
- data/LICENSE.txt +0 -201
- data/Rakefile +0 -8
- data/consul-templaterb.gemspec +0 -42
- data/docker-nginx-conf/nginx.conf +0 -27
- data/docs/article-06_Template-based_discovery_with_consul-templaterb.md +0 -124
- data/docs/images/consul-templaterb.png +0 -0
- data/docs/images/consul-templaterb.svg +0 -3
- data/docs/images/consul-ui_001.png +0 -0
data/Gemfile
DELETED
data/INTERNALS.md
DELETED
@@ -1,49 +0,0 @@
|
|
1
|
-
# consul-templaterb Internal Architecture Overview
|
2
|
-
|
3
|
-
## Consul basics
|
4
|
-
|
5
|
-
`consul-templaterb` is a ruby library (or `GEM`) with a single binary that does render `.erb` files based on Consul exposed endpoints.
|
6
|
-
|
7
|
-
Consul is a kind of distributed K/V tree store that exposes most of his APIs using HTTP endpoints with a way to watch for changes based on an Index. Each part of the Consul Tree has its own Index reflecting the last id of transaction for this
|
8
|
-
part of the tree. It means that Consul has a kind of transaction ID for each subtree of the data.
|
9
|
-
|
10
|
-
Consul also exposes some specific endpoints to get some discovery specific endpoints (eg: `/v1/catalog/services` to get
|
11
|
-
the list of services registered in the cluster).
|
12
|
-
|
13
|
-
For each endpoint, `consul-templaterb` performs a watch by getting information about the last transaction on this endpoints, storing it locally and watching at Consul to get updates about the data within this endpoint.
|
14
|
-
|
15
|
-
![Main Loop](docs/images/consul-templaterb.svg)
|
16
|
-
|
17
|
-
## How it works
|
18
|
-
|
19
|
-
There is a loop in the engine, every second, all templates are rendered using the `ERB` template engine.
|
20
|
-
|
21
|
-
Each template is assigned to a `ConsulTemplateRender` object that keeps tracks of changes on disk of template (in order
|
22
|
-
to be able to hot-reload templates) as well as monitoring if the template is completely rendered, aka all data from
|
23
|
-
Consul is properly retrieved and consistent.
|
24
|
-
|
25
|
-
At each rendering, each template using a Consul Endpoint (for instance, calling `datacenters()`) does register itself
|
26
|
-
in `EndPointsManager`. If the endpoint is already registered, fetch the result if available and return the value, if not,
|
27
|
-
the endpoint is tagged as dirty. When the rendering of a template is completed, a check is performed to see if some endpoints used by the template are still marked as dirty. Being dirty means that some data from Consul is missing, so
|
28
|
-
that the result is not the definitive one. If no endpoint is dirty, then the result of template can be rendered on disk
|
29
|
-
and template is complete.
|
30
|
-
|
31
|
-
At startup, in order to converge faster, templates are rendered more quickly that every 1s to speed up startup time for first convergence, meaning that the delay between each rendering of template might be very fast, especially at startup (so, DO NOT USE Ruby I/O in templates unless you really know what you are doing)
|
32
|
-
|
33
|
-
### EndPointsManager
|
34
|
-
|
35
|
-
`EndPointsManager` is the object responsible of keeping track the endpoints (basically the I/O) of the whole library.
|
36
|
-
When a new Consul endpoint is added by a template, it is registered in that object. Each time a template is rendered,
|
37
|
-
all the endpoints are marked as used. When some endpoints are not used for a while (a few rendering loops), the
|
38
|
-
`EndPointsManager` will garbage collect those endpoints.
|
39
|
-
|
40
|
-
### ConsulTemplateEngine
|
41
|
-
|
42
|
-
When `consul-templaterb` starts, it creates an [ConsulTemplateEngine](lib/consul/async/consul_template_engine.rb) that
|
43
|
-
aggregates all options, template files and parameters. The engine then starts a loop running every second (this can be changed) that:
|
44
|
-
* perform a rendering of each of the `.erb` template files using a `ConsulTemplateRender` object
|
45
|
-
* when all template are rendered the first time, will start some program(s) if specified (the `--exec` parameters)
|
46
|
-
|
47
|
-
### ConsulTemplateRender
|
48
|
-
|
49
|
-
This object keeps track of template, its last state (rendered or not), and whether the file template needs to be reloaded (aka if file has been modified on filesystem).
|
data/LICENSE.txt
DELETED
@@ -1,201 +0,0 @@
|
|
1
|
-
Apache License
|
2
|
-
Version 2.0, January 2004
|
3
|
-
http://www.apache.org/licenses/
|
4
|
-
|
5
|
-
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
6
|
-
|
7
|
-
1. Definitions.
|
8
|
-
|
9
|
-
"License" shall mean the terms and conditions for use, reproduction,
|
10
|
-
and distribution as defined by Sections 1 through 9 of this document.
|
11
|
-
|
12
|
-
"Licensor" shall mean the copyright owner or entity authorized by
|
13
|
-
the copyright owner that is granting the License.
|
14
|
-
|
15
|
-
"Legal Entity" shall mean the union of the acting entity and all
|
16
|
-
other entities that control, are controlled by, or are under common
|
17
|
-
control with that entity. For the purposes of this definition,
|
18
|
-
"control" means (i) the power, direct or indirect, to cause the
|
19
|
-
direction or management of such entity, whether by contract or
|
20
|
-
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
21
|
-
outstanding shares, or (iii) beneficial ownership of such entity.
|
22
|
-
|
23
|
-
"You" (or "Your") shall mean an individual or Legal Entity
|
24
|
-
exercising permissions granted by this License.
|
25
|
-
|
26
|
-
"Source" form shall mean the preferred form for making modifications,
|
27
|
-
including but not limited to software source code, documentation
|
28
|
-
source, and configuration files.
|
29
|
-
|
30
|
-
"Object" form shall mean any form resulting from mechanical
|
31
|
-
transformation or translation of a Source form, including but
|
32
|
-
not limited to compiled object code, generated documentation,
|
33
|
-
and conversions to other media types.
|
34
|
-
|
35
|
-
"Work" shall mean the work of authorship, whether in Source or
|
36
|
-
Object form, made available under the License, as indicated by a
|
37
|
-
copyright notice that is included in or attached to the work
|
38
|
-
(an example is provided in the Appendix below).
|
39
|
-
|
40
|
-
"Derivative Works" shall mean any work, whether in Source or Object
|
41
|
-
form, that is based on (or derived from) the Work and for which the
|
42
|
-
editorial revisions, annotations, elaborations, or other modifications
|
43
|
-
represent, as a whole, an original work of authorship. For the purposes
|
44
|
-
of this License, Derivative Works shall not include works that remain
|
45
|
-
separable from, or merely link (or bind by name) to the interfaces of,
|
46
|
-
the Work and Derivative Works thereof.
|
47
|
-
|
48
|
-
"Contribution" shall mean any work of authorship, including
|
49
|
-
the original version of the Work and any modifications or additions
|
50
|
-
to that Work or Derivative Works thereof, that is intentionally
|
51
|
-
submitted to Licensor for inclusion in the Work by the copyright owner
|
52
|
-
or by an individual or Legal Entity authorized to submit on behalf of
|
53
|
-
the copyright owner. For the purposes of this definition, "submitted"
|
54
|
-
means any form of electronic, verbal, or written communication sent
|
55
|
-
to the Licensor or its representatives, including but not limited to
|
56
|
-
communication on electronic mailing lists, source code control systems,
|
57
|
-
and issue tracking systems that are managed by, or on behalf of, the
|
58
|
-
Licensor for the purpose of discussing and improving the Work, but
|
59
|
-
excluding communication that is conspicuously marked or otherwise
|
60
|
-
designated in writing by the copyright owner as "Not a Contribution."
|
61
|
-
|
62
|
-
"Contributor" shall mean Licensor and any individual or Legal Entity
|
63
|
-
on behalf of whom a Contribution has been received by Licensor and
|
64
|
-
subsequently incorporated within the Work.
|
65
|
-
|
66
|
-
2. Grant of Copyright License. Subject to the terms and conditions of
|
67
|
-
this License, each Contributor hereby grants to You a perpetual,
|
68
|
-
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
69
|
-
copyright license to reproduce, prepare Derivative Works of,
|
70
|
-
publicly display, publicly perform, sublicense, and distribute the
|
71
|
-
Work and such Derivative Works in Source or Object form.
|
72
|
-
|
73
|
-
3. Grant of Patent License. Subject to the terms and conditions of
|
74
|
-
this License, each Contributor hereby grants to You a perpetual,
|
75
|
-
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
76
|
-
(except as stated in this section) patent license to make, have made,
|
77
|
-
use, offer to sell, sell, import, and otherwise transfer the Work,
|
78
|
-
where such license applies only to those patent claims licensable
|
79
|
-
by such Contributor that are necessarily infringed by their
|
80
|
-
Contribution(s) alone or by combination of their Contribution(s)
|
81
|
-
with the Work to which such Contribution(s) was submitted. If You
|
82
|
-
institute patent litigation against any entity (including a
|
83
|
-
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
84
|
-
or a Contribution incorporated within the Work constitutes direct
|
85
|
-
or contributory patent infringement, then any patent licenses
|
86
|
-
granted to You under this License for that Work shall terminate
|
87
|
-
as of the date such litigation is filed.
|
88
|
-
|
89
|
-
4. Redistribution. You may reproduce and distribute copies of the
|
90
|
-
Work or Derivative Works thereof in any medium, with or without
|
91
|
-
modifications, and in Source or Object form, provided that You
|
92
|
-
meet the following conditions:
|
93
|
-
|
94
|
-
(a) You must give any other recipients of the Work or
|
95
|
-
Derivative Works a copy of this License; and
|
96
|
-
|
97
|
-
(b) You must cause any modified files to carry prominent notices
|
98
|
-
stating that You changed the files; and
|
99
|
-
|
100
|
-
(c) You must retain, in the Source form of any Derivative Works
|
101
|
-
that You distribute, all copyright, patent, trademark, and
|
102
|
-
attribution notices from the Source form of the Work,
|
103
|
-
excluding those notices that do not pertain to any part of
|
104
|
-
the Derivative Works; and
|
105
|
-
|
106
|
-
(d) If the Work includes a "NOTICE" text file as part of its
|
107
|
-
distribution, then any Derivative Works that You distribute must
|
108
|
-
include a readable copy of the attribution notices contained
|
109
|
-
within such NOTICE file, excluding those notices that do not
|
110
|
-
pertain to any part of the Derivative Works, in at least one
|
111
|
-
of the following places: within a NOTICE text file distributed
|
112
|
-
as part of the Derivative Works; within the Source form or
|
113
|
-
documentation, if provided along with the Derivative Works; or,
|
114
|
-
within a display generated by the Derivative Works, if and
|
115
|
-
wherever such third-party notices normally appear. The contents
|
116
|
-
of the NOTICE file are for informational purposes only and
|
117
|
-
do not modify the License. You may add Your own attribution
|
118
|
-
notices within Derivative Works that You distribute, alongside
|
119
|
-
or as an addendum to the NOTICE text from the Work, provided
|
120
|
-
that such additional attribution notices cannot be construed
|
121
|
-
as modifying the License.
|
122
|
-
|
123
|
-
You may add Your own copyright statement to Your modifications and
|
124
|
-
may provide additional or different license terms and conditions
|
125
|
-
for use, reproduction, or distribution of Your modifications, or
|
126
|
-
for any such Derivative Works as a whole, provided Your use,
|
127
|
-
reproduction, and distribution of the Work otherwise complies with
|
128
|
-
the conditions stated in this License.
|
129
|
-
|
130
|
-
5. Submission of Contributions. Unless You explicitly state otherwise,
|
131
|
-
any Contribution intentionally submitted for inclusion in the Work
|
132
|
-
by You to the Licensor shall be under the terms and conditions of
|
133
|
-
this License, without any additional terms or conditions.
|
134
|
-
Notwithstanding the above, nothing herein shall supersede or modify
|
135
|
-
the terms of any separate license agreement you may have executed
|
136
|
-
with Licensor regarding such Contributions.
|
137
|
-
|
138
|
-
6. Trademarks. This License does not grant permission to use the trade
|
139
|
-
names, trademarks, service marks, or product names of the Licensor,
|
140
|
-
except as required for reasonable and customary use in describing the
|
141
|
-
origin of the Work and reproducing the content of the NOTICE file.
|
142
|
-
|
143
|
-
7. Disclaimer of Warranty. Unless required by applicable law or
|
144
|
-
agreed to in writing, Licensor provides the Work (and each
|
145
|
-
Contributor provides its Contributions) on an "AS IS" BASIS,
|
146
|
-
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
147
|
-
implied, including, without limitation, any warranties or conditions
|
148
|
-
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
149
|
-
PARTICULAR PURPOSE. You are solely responsible for determining the
|
150
|
-
appropriateness of using or redistributing the Work and assume any
|
151
|
-
risks associated with Your exercise of permissions under this License.
|
152
|
-
|
153
|
-
8. Limitation of Liability. In no event and under no legal theory,
|
154
|
-
whether in tort (including negligence), contract, or otherwise,
|
155
|
-
unless required by applicable law (such as deliberate and grossly
|
156
|
-
negligent acts) or agreed to in writing, shall any Contributor be
|
157
|
-
liable to You for damages, including any direct, indirect, special,
|
158
|
-
incidental, or consequential damages of any character arising as a
|
159
|
-
result of this License or out of the use or inability to use the
|
160
|
-
Work (including but not limited to damages for loss of goodwill,
|
161
|
-
work stoppage, computer failure or malfunction, or any and all
|
162
|
-
other commercial damages or losses), even if such Contributor
|
163
|
-
has been advised of the possibility of such damages.
|
164
|
-
|
165
|
-
9. Accepting Warranty or Additional Liability. While redistributing
|
166
|
-
the Work or Derivative Works thereof, You may choose to offer,
|
167
|
-
and charge a fee for, acceptance of support, warranty, indemnity,
|
168
|
-
or other liability obligations and/or rights consistent with this
|
169
|
-
License. However, in accepting such obligations, You may act only
|
170
|
-
on Your own behalf and on Your sole responsibility, not on behalf
|
171
|
-
of any other Contributor, and only if You agree to indemnify,
|
172
|
-
defend, and hold each Contributor harmless for any liability
|
173
|
-
incurred by, or claims asserted against, such Contributor by reason
|
174
|
-
of your accepting any such warranty or additional liability.
|
175
|
-
|
176
|
-
END OF TERMS AND CONDITIONS
|
177
|
-
|
178
|
-
APPENDIX: How to apply the Apache License to your work.
|
179
|
-
|
180
|
-
To apply the Apache License to your work, attach the following
|
181
|
-
boilerplate notice, with the fields enclosed by brackets "[]"
|
182
|
-
replaced with your own identifying information. (Don't include
|
183
|
-
the brackets!) The text should be enclosed in the appropriate
|
184
|
-
comment syntax for the file format. We also recommend that a
|
185
|
-
file or class name and description of purpose be included on the
|
186
|
-
same "printed page" as the copyright notice for easier
|
187
|
-
identification within third-party archives.
|
188
|
-
|
189
|
-
Copyright [yyyy] [name of copyright owner]
|
190
|
-
|
191
|
-
Licensed under the Apache License, Version 2.0 (the "License");
|
192
|
-
you may not use this file except in compliance with the License.
|
193
|
-
You may obtain a copy of the License at
|
194
|
-
|
195
|
-
http://www.apache.org/licenses/LICENSE-2.0
|
196
|
-
|
197
|
-
Unless required by applicable law or agreed to in writing, software
|
198
|
-
distributed under the License is distributed on an "AS IS" BASIS,
|
199
|
-
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
200
|
-
See the License for the specific language governing permissions and
|
201
|
-
limitations under the License.
|
data/Rakefile
DELETED
data/consul-templaterb.gemspec
DELETED
@@ -1,42 +0,0 @@
|
|
1
|
-
lib = File.expand_path('lib', __dir__)
|
2
|
-
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
3
|
-
require 'consul/async/version'
|
4
|
-
|
5
|
-
Gem::Specification.new do |spec|
|
6
|
-
spec.name = 'consul-templaterb'
|
7
|
-
|
8
|
-
spec.version = Consul::Async::VERSION
|
9
|
-
spec.authors = ['SRE Core Services']
|
10
|
-
spec.email = ['sre-core-services@criteo.com']
|
11
|
-
|
12
|
-
spec.summary = 'Implementation of Consul template using Ruby and .erb templating language'
|
13
|
-
spec.homepage = 'https://rubygems.org/gems/consul-templaterb'
|
14
|
-
spec.description = 'A ruby implementation of Consul Template with support of erb templating ' \
|
15
|
-
'with hi-performance on large clusters and advanced process management features.'
|
16
|
-
spec.metadata = { 'bug_tracker_uri' => 'https://github.com/criteo/consul-templaterb/issues',
|
17
|
-
'changelog_uri' => 'https://github.com/criteo/consul-templaterb/blob/master/CHANGELOG.md',
|
18
|
-
'homepage_uri' => 'https://github.com/criteo/consul-templaterb',
|
19
|
-
'source_code_uri' => 'https://github.com/criteo/consul-templaterb' }
|
20
|
-
spec.license = 'Apache v2'
|
21
|
-
|
22
|
-
spec.files = `git ls-files -z`.split("\x0").reject do |f|
|
23
|
-
f.match(%r{^(test|spec|features)/})
|
24
|
-
end
|
25
|
-
spec.bindir = 'bin'
|
26
|
-
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
27
|
-
spec.extra_rdoc_files = ['README.md', 'CHANGELOG.md', 'TemplateAPI.md']
|
28
|
-
spec.require_paths = ['lib']
|
29
|
-
spec.required_ruby_version = '>= 2.4.0'
|
30
|
-
|
31
|
-
spec.add_runtime_dependency 'em-http-request', '>= 1.1.5'
|
32
|
-
spec.add_runtime_dependency 'eventmachine', '>= 1.2.7'
|
33
|
-
spec.add_runtime_dependency 'parallel', '>= 1.2.7'
|
34
|
-
|
35
|
-
spec.add_development_dependency 'bundler', '>= 1.14'
|
36
|
-
spec.add_development_dependency 'rake', '~> 12.3.3'
|
37
|
-
spec.add_development_dependency 'rspec', '~> 3.0'
|
38
|
-
spec.add_development_dependency 'rspec_junit_formatter'
|
39
|
-
spec.add_development_dependency 'rubocop', '0.80.0'
|
40
|
-
spec.add_development_dependency 'rubocop-junit-formatter'
|
41
|
-
spec.add_development_dependency 'webmock'
|
42
|
-
end
|
@@ -1,27 +0,0 @@
|
|
1
|
-
daemon off;
|
2
|
-
worker_processes 2;
|
3
|
-
|
4
|
-
error_log /usr/src/app/error.log info;
|
5
|
-
|
6
|
-
events {
|
7
|
-
worker_connections 128;
|
8
|
-
}
|
9
|
-
|
10
|
-
http {
|
11
|
-
server_tokens off;
|
12
|
-
include /etc/nginx/mime.types;
|
13
|
-
charset utf-8;
|
14
|
-
|
15
|
-
access_log /usr/src/app/access.log combined;
|
16
|
-
|
17
|
-
server {
|
18
|
-
server_name localhost;
|
19
|
-
listen 0.0.0.0:80;
|
20
|
-
root /usr/src/app/samples/consul-ui;
|
21
|
-
sendfile on;
|
22
|
-
|
23
|
-
error_page 500 502 503 504 /50x.html;
|
24
|
-
|
25
|
-
}
|
26
|
-
|
27
|
-
}
|
@@ -1,124 +0,0 @@
|
|
1
|
-
# Template-based discovery with consul-templaterb
|
2
|
-
|
3
|
-
Source of article https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
|
4
|
-
|
5
|
-
# How and why Criteo built a template-based system for Consul
|
6
|
-
|
7
|
-
[consul-templaterb](https://github.com/criteo/consul-templaterb/) is an OpenSource Software (OSS) written in Ruby that allows you to very quickly create UIs for Consul or configuration files for your beloved software to interact with Consul. This article describes why we wrote it and why you might be interested in using it.
|
8
|
-
|
9
|
-
# The context
|
10
|
-
|
11
|
-
At the end of 2017, Consul started becoming a very important piece in Criteo’s infrastructure as it was responsible for dealing with the discovery of all major business applications at Criteo. At that time, Criteo had more than 25,000 servers on production and some datacenters had more than 5,000 agents per datacenter. At the same time, we started building our Load-Balancer stack based on the Consul state. So it became more and more important to make it efficient, stable and expand it with new features.
|
12
|
-
|
13
|
-
Criteo wanted to move towards HAProxy for its load-balancing stack, thus we started generating some configuration for HAProxy with all HTTP services with the template-based configuration generator consul-template. However, things were a bit more complicated than expected because the program consul-template was using almost all bandwidth available just to discover all services. Moreover, the system was constantly updating HAProxy configuration.
|
14
|
-
|
15
|
-
As explained in our first article “[Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)”, Consul includes a notification system to ensure your systems are always up-to-date with the content of the discovery. When this system starts to push too many notifications, on large services, the amount of data sent to all agents can be quite impressive (several megabytes/s for a single service having hundreds of instances).
|
16
|
-
|
17
|
-
Thus, we started investigating the issue and figured out that it was linked to a race condition in large clusters that caused all listeners of Consul to be notified continuously. This issue was eventually solved by our [Pull-Request #3899](https://github.com/hashicorp/consul/pull/3899) (and quite a few others after that), but the investigation showed that it was quite hard to investigate the root cause of such issues.
|
18
|
-
|
19
|
-
Therefore, I started a simple tool to watch all the services matching some patterns and report the bandwidth usage, but I also wanted to prove it would be possible to generate all configurations for HAProxy with an optimized tool, thus having a templating mechanism. The [Pull Request #1066](https://github.com/hashicorp/consul-template/pull/1066) to fix the issue on the consul-template itself was not yet considered (and it was hard to have something better), so I decided to implement it in Ruby with asynchronous I/Os (Input/Outputs) because I was very frustrated by the Go templating language used in consul-template and was in love with ERB templates.
|
20
|
-
|
21
|
-
# The goals
|
22
|
-
|
23
|
-
From scratch, I wanted to take the various shortcoming of consul-template into account:
|
24
|
-
|
25
|
-
* The ability to generate high-level code to generate various formats (YAML, JSON, XML): it is dangerous, error-prone and painful to generate JSON or XML content using text templating. People fighting with YAML indentation will tell you that.
|
26
|
-
* The ability to use a Turing complete and effective language to perform simple transformation such as sorting (Consul by default has some rules to sort the output, but it might not be predictable, for instance, based on the round-trip between servers).
|
27
|
-
* Being able to push real-life optimizations (for instance, we don’t care about being notified immediately when a change occurs, we want to be notified once before dumping the configuration file, not 100 times without taking it into account), based on the type of query performed.
|
28
|
-
* Being able to scale nicely with Consul and avoid overloading the cluster for nothing in case of bug (for instance, we use a penalty for services changing too much, we rate-limit the notifications).
|
29
|
-
* Being very simple to use, hiding all the complexity to the template creator and using optimizations to reduce the number of calls to Consul, meaning being as purely functional as possible by hiding all I/O operations and taking decisions based on our experience with Consul internals.
|
30
|
-
* Be evolutive: When some functions or fields are missing in a consul-template, you have to do a pull request on consul-template to add your new fields/methods, then you can work. It would be nice if any field of endpoints would be supported natively (so new versions of Consul can be used with an old version of our tool).
|
31
|
-
* Support for hot-reload of templates, and nice error messages with the precise line when an error occurs, so writing templates should be very fast and efficient.
|
32
|
-
* Have most of the consul-template features including babysitting of processes, spawn commands when files do change…
|
33
|
-
|
34
|
-
# Internals: make your template engine in Ruby
|
35
|
-
|
36
|
-
## At startup
|
37
|
-
|
38
|
-
The engine collects all template sources and computes the destinations. Each template and its destination also register some commands to run when the destination does change. Those objects save the last time the template has been read (to allow hot-reload) and the last binary content associated with them (to detect binary changes).
|
39
|
-
|
40
|
-
## The main loop
|
41
|
-
|
42
|
-
First, the main loop is working on pending events. Those events are in fact the processing of the I/O operations querying the Consul agent. But every second by default, the main loop also evaluates templates which are Ruby code. When this is just regular code, this is evaluated normally, but when the code called is part of the [I/O API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md), the code does the following:
|
43
|
-
|
44
|
-
1. Checks in a registry if this method has been called with the same parameters, if not, create it, mark it as dirty (so the registry knows we are waiting for data from Consul), launch the request asynchronously to Consul, store it in the registry. This will be translated into events that will be run as events in the main loop.
|
45
|
-
2. Return the object stored within the registry. By default, this object returns the empty object or collection according to its type. When the request finally gets a response from Consul, the object is marked as “non-dirty” and the result is put within the registry, so the next evaluation returns the content of the requests.
|
46
|
-
|
47
|
-
Once all code has been evaluated and the result has been stored in memory, the main loop checks whether the template is using “dirty” objects in the registry. Having at least one dirty object means that some requests are still in flight and we don’t have the result yet, so basically means the template, while evaluated, has been evaluated with fake (empty) data and is not yet ready.
|
48
|
-
|
49
|
-
However, if the template is not using any dirty object it means we did receive all the data and we can render the template on disk. We first check if our last rendering was different. If the new rendering phase was not different, nothing did change, however, if the data rendered is different, the file is stored to disk and commands might be evaluated (to notify a program to reload for instance).
|
50
|
-
|
51
|
-
All the I/O are done asynchronously (using [eventmachine](https://github.com/eventmachine/eventmachine)) but performed within the main loop, the one also performing the rendering, so, there is no possibility of threading issues.
|
52
|
-
|
53
|
-
![Main loop behavior, in purple, the async fetch, in yellow, the rendering](images/consul-templaterb.png)
|
54
|
-
|
55
|
-
## Handling of I/O
|
56
|
-
|
57
|
-
The Input/Outputs are handled very carefully with regards to what we explained in our previous article “[Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)” and adds a few tricks:
|
58
|
-
|
59
|
-
* When a given endpoint is changing too fast with any real change (the data is the same as last call), a penalty is applied, so the endpoint won’t be queried before a few seconds (this allows to work well even with old Consul versions with patch [#3899](https://github.com/hashicorp/consul/pull/3899)).
|
60
|
-
* Some endpoints [are configured specifically](https://github.com/criteo/consul-templaterb/blob/master/bin/consul-templaterb#L61) to avoid too many calls (for instance, we don’t consider we add a new DC every 30 seconds).
|
61
|
-
* Some specific behavior exists for non-existing service (was causing lots of issues with Prometheus for instance, see [Prometheus PR #3814](https://github.com/prometheus/prometheus/pull/3814)), so we will work on the old Consul version with our [PR #4810](https://github.com/hashicorp/consul/pull/4810).
|
62
|
-
* By default, when some errors occur, try to limit the number of calls to avoid increasing the pressure on the cluster.
|
63
|
-
* Collect statistics that can be used in your templates. For instance, the [Prometheus exporter uses it to know whether a service is unstable or not](https://github.com/criteo/consul-templaterb/blob/master/samples/metrics.erb#L88).
|
64
|
-
|
65
|
-
## The hard part about async I/Os with Ruby
|
66
|
-
|
67
|
-
This program was my first project dealing with complex code with important performance issues. I first discovered that Ruby async I/O code is really hard and most libraries are not dealing with it properly. Many libraries just lie (by creating threads and pretending to be async). The only library I found handling it correctly was [eventmachine](https://github.com/eventmachine/eventmachine/).
|
68
|
-
|
69
|
-
While the library is petty mature (the project is quite old, 9 years old), I had lots of issues very quickly. I spent quite some time making it work on my targeted operating systems (Linux, Mac OS, Windows).
|
70
|
-
|
71
|
-
* On Windows, installation is painful (see https://github.com/criteo/consul-templaterb/#quick-install-on-windows) and due to limitation into the Ruby runtime, it does not support more than 2048 file descriptors with native implementation (while this works with Linux Subsystem on Windows 10).
|
72
|
-
* On all Operating Systems, opening lots of HTTP connections concurrently led to crashes (see https://github.com/igrigorik/em-http-request/issues/315, [#604](https://github.com/eventmachine/eventmachine/issues/604), [#770](https://github.com/eventmachine/eventmachine/issues/770), and [#824](https://github.com/eventmachine/eventmachine/issues/824)) that took me lots of time to debug (including native C code debugging), quite frustrating for such mature library (the good news is that eventmachine is now fixed thanks to [consul-templaterb](https://github.com/criteo/consul-templaterb/)).
|
73
|
-
|
74
|
-
While those problems are now solved and you can use it properly, it took me weeks to solve.
|
75
|
-
|
76
|
-
Furthermore, the lack of HTTP/2 is a bit worrisome today as more and more systems allow it (it would avoid the lack of file descriptors on very large clusters).
|
77
|
-
|
78
|
-
## Some fun features
|
79
|
-
|
80
|
-
* All templates in the repository are unit tested for each release of consul-templaterb, so all samples are working on your local installation.
|
81
|
-
* You can generate lots of formats: [Services in XML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.xml.erb), [JSON](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb), [YAML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb#L58) using Hash and just dumping in the right format!
|
82
|
-
* You can load templates from Consul Key/Value store, so you can change templates dynamically from Consul K/V Store (we use this feature in for our Prometheus configurations for instance as described in “[Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)”).
|
83
|
-
* You can also query Vault / some JSON APIs very easily.
|
84
|
-
* You can have a very high-performance UI (see [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)) easily for your own use-cases. This UI is generated in real-time as static files and can be served by your favorite webserver (we do use nginx on our side), so it scales indefinitely if you have lots of users.
|
85
|
-
* Huge Performance gap with consul-template: in version 1.0 of Consul, consul-template was using 800Mb/s to scrape all services in one of large DCs, consul-templaterb is using less than 100kb/s (800x reduction!) to generate the full UI of Consul and fetch all services in our largest DC now.
|
86
|
-
* Good abstraction with impressive performance compared to other projects such as [consult](https://github.com/veracross/consult) (no need to deal with kind of “magic TTL”) or consul-template itself: the APIs are simpler, you have to deal less with Consul specific optimizations and the performance is better.
|
87
|
-
|
88
|
-
# Usages at Criteo (non-exhaustive)
|
89
|
-
|
90
|
-
We use this executable for many things at Criteo (see “Inversion of Control with Consul”):
|
91
|
-
|
92
|
-
* Automatic Alerting / Prometheus configuration
|
93
|
-
* [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)s and its Consul Timeline (keep track of the history of all changes for all services)
|
94
|
-
* List of assets per DC/Racks
|
95
|
-
* Monitoring of services/racks
|
96
|
-
* Ownership enforcement
|
97
|
-
* Live generation of configuration for several programs/services
|
98
|
-
|
99
|
-
# Try it quickly
|
100
|
-
|
101
|
-
If you don’t want to bother configuring and tuning it, you can try it very quickly with our docker images: In 2 minutes (download included), you have a full scalable UI with Consul, serving static files with nginx and keeping a history of all changes on your services live.
|
102
|
-
|
103
|
-
![Consul UI in action, scalable UI with excellent performance](images/consul-ui_001.png)
|
104
|
-
|
105
|
-
# More on the subject
|
106
|
-
|
107
|
-
* The [consul-templaterb API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md) to write templates, with links to real-world examples in the samples directory.
|
108
|
-
* [INTERNALS.md](../INTERNALS.md) on Github (describes the objects in the code)
|
109
|
-
* Slides at HashiConf ’19: “[Consul Templates on Steroids](https://fr.slideshare.net/PierreSouchay/2019-hashiconf-consultemplaterb)”
|
110
|
-
* A video introduction of consul-templaterb:
|
111
|
-
|
112
|
-
[![Quick presentation of consul-templaterb](images/consul-ui_001.png)](https://youtu.be/zLzrLGLLl4Q)
|
113
|
-
|
114
|
-
# Other articles from the Discovery team
|
115
|
-
|
116
|
-
* [Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)
|
117
|
-
* [Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)
|
118
|
-
* [Anatomy of a bug: When Consul has too much to deliver for the big day](https://medium.com/criteo-labs/anatomy-of-a-bug-when-consul-has-too-much-to-deliver-for-the-big-day-4904d19a46a4)
|
119
|
-
* [Inversion of Control for the Infrastructure with Consul](https://medium.com/criteo-labs/inversion-of-control-for-the-infrastructure-with-consul-b894877b33a4)
|
120
|
-
* [Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)
|
121
|
-
|
122
|
-
Pierre Souchay, 2020-03-03
|
123
|
-
|
124
|
-
Also published on https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
|