dropsonde 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 2972f1e6d2deff8b7826689d08c28f46b496f98af11ee98222f0cf9d67ea9d5e
4
+ data.tar.gz: ffc4b8c331ed8e18ee635f986877c88f590afb42d8bdbbb601d96ac9d37f6f65
5
+ SHA512:
6
+ metadata.gz: 0e8a91a089a3cee0b96896216c21daa94b9715b06ab539b32956016ed8d563ca698e39245e4188f5b0d52f14bb05e46fe4fbb2484ceff80a23fa3e3742cf09b2
7
+ data.tar.gz: 7ad9668cc0f244f241ed7335f4f52b3745f437e73e41b60c302e9daa17354d7f75da8b35c8610d264361c7f95753cc65b0a4dc087f6af1244dabfc40c8a75004
data/LICENSE ADDED
@@ -0,0 +1,202 @@
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
@@ -0,0 +1,152 @@
1
+ # Dropsonde
2
+
3
+ A simple telemetry probe for gathering non-identifiable information about Puppet
4
+ infrastructures.
5
+
6
+
7
+ ## Overview
8
+
9
+ We both know that you hate telemetry as much as I do. So what makes this different?
10
+ At its core, the purpose and design of this module is for your own benefit as much
11
+ as it is for ours. Think back to the time you last visited the Forge to find a
12
+ module. Chances are that you discovered many modules that claimed to solve your
13
+ problem and it was relatively difficult choosing between them. Surfacing usage
14
+ data in a way that lets you signal approval simply by using a module is the
15
+ primary goal of this project.
16
+
17
+ > **This means that the best way for you to help yourself find new modules is to
18
+ > install this telemetry tool and, along with your peers, share your module usage
19
+ > data.** ✅
20
+
21
+
22
+ ## Design
23
+
24
+ So what sort of information is gathered? Dropsonde identifies information like
25
+ which public modules and classes are used, and what platforms they're used on.
26
+ It identifies the component classes that your profile modules declares so we can
27
+ better support what you are doing with your profiles, *but without exposing your
28
+ profile names.* The key point here is that it looks for usage patterns for *public
29
+ modules only* and explicitly does its best to keep your internal code private to you.
30
+
31
+ You can see exactly what will be phoned home by running the command:
32
+
33
+ ```
34
+ $ dropsonde preview
35
+ ```
36
+
37
+ Dropsonde is a simple telemetry probe designed to run as a regular cron job.
38
+ Metrics are defined by plugins that gather data, but also export a schema that
39
+ constrains the data allowed to be reported on. Dropsonde will reject metrics
40
+ that don't meet these constraints. The backend database is also defined by this
41
+ schema so the system cannot gather any data that's not described in the schema.
42
+
43
+ See the full schema of all enabled plugins by running the command:
44
+
45
+ ```
46
+ $ dropsonde schema
47
+ ```
48
+
49
+ All information in the report is keyed off a non-reversible SHA512 hash site-id
50
+ to make it unidentifiable; this report cannot be linked back to you or to your
51
+ infrastructure. Now that said, we know that the more bits of data shared about a
52
+ specific site, the easier it is to fingerprint that site. See
53
+ [Panopticlick](https://panopticlick.eff.org) for a real-world example.
54
+
55
+ To mitigate this concern, we aggregate data in a two step process. The data
56
+ submitted by Dropsonde is stored in a private dataset. ACLs limit access to only
57
+ certain number of employees. Then each week, a job runs to generate a sanitized
58
+ and aggregated form of the data for the week which goes into the public dataset
59
+ that all our tooling actually depends on and uses. This dataset includes patterns
60
+ and aggregate data, but does not include any individual records whatsoever.
61
+
62
+ For example, this aggregated data might include records that show a count of how
63
+ many sites are using various combinations of modules together, but it will never
64
+ include a record showing the full list of modules that any single site is using.
65
+
66
+ With your own Google Cloud account, you can use that [dataset](https://console.cloud.google.com/bigquery?project=puppetlabs.com:api-project-53122606061)
67
+ in your own tooling and you can see/contribute to the aggregation queries in its
68
+ own [repository](https://github.com/puppetlabs/dropsonde-aggregation).
69
+
70
+
71
+ ## Privacy
72
+
73
+ Dropsonde will not collect the names or titles of internal modules, classes,
74
+ facts, types, etc. It will gather the names of the component classes that your
75
+ profiles use, but it will not gather the names of your profiles themselves. In
76
+ order to maintain data integrity, reports are keyed off a non-reversible site-id.
77
+ This means that there's no direct link from a record to you or your infrastructure,
78
+ but it does mean that the data could be used for fingerprinting. To mitigate that,
79
+ the data goes through an additional aggregation step that removes this ID and
80
+ presents only usage patterns publicly.
81
+
82
+ If you identify an unmitigated privacy concern then please inform us as soon as
83
+ possible: [privacy@puppet.com](mailto:privacy@puppet.com)
84
+
85
+
86
+ ## Installation
87
+
88
+ This is distributed as a Ruby gem. Simply `gem install dropsonde`
89
+
90
+
91
+ ## Configuration
92
+
93
+ Any command line arguments can also be specified in `/etc/puppetlabs/telemetry.yaml`.
94
+ For example the config file below will disable Forge module cache updating and
95
+ will not report the `:puppetfiles` metrics.
96
+
97
+
98
+ ``` yaml
99
+ ---
100
+ :update: false
101
+ :blacklist:
102
+ - puppetfiles
103
+ ```
104
+
105
+ The `puppetlabs-dropsonde` Puppet module manages this configuration for you.
106
+
107
+
108
+ ## Running
109
+
110
+ Run `dropsonde --help` to see usage information.
111
+
112
+ * `preview`
113
+ * Generate and print out an example telemetry report in human readable form
114
+ * Annotated with descriptions of each plugin and each metric gathered.
115
+ * `schema`
116
+ * Generate and print out the complete combined schema.
117
+ * `list`
118
+ * See a quick list of the available metrics and what they do.
119
+ * `submit`
120
+ * Generate and submit a telemetry report. This will be exactly the same as
121
+ the human readable form, just in JSON format.
122
+ * `update`
123
+ * Once a week, the list of public modules on the Forge will be updated. This
124
+ command will manually force that cache update to happen.
125
+
126
+
127
+ ## Architecture
128
+
129
+ Dropsonde is a simple telemetry probe designed to run as a regular cron job. It
130
+ will gather metrics defined by self-contained plugins that each defines its own
131
+ partial schema and then gathers the data to meet that schema.
132
+
133
+ Metrics plugins live in the `lib/dropsonde/metrics` directory. See the existing
134
+ examples for the API. All data returned must match the schema the plugin defines
135
+ and each column in the schema must be documented.
136
+
137
+ Dropsonde maintains a cache listing all the public modules existing on the Forge.
138
+ This is used for identifying public modules and ensuring that information about
139
+ your internal modules is not leaked outside your network. Once a week, this cache
140
+ of modules is updated.
141
+
142
+
143
+ ## Limitations
144
+
145
+ This is super early in development and has not yet been battle tested.
146
+
147
+
148
+ Contact
149
+ -------
150
+
151
+ community@puppet.com
152
+
@@ -0,0 +1,84 @@
1
+ #!/usr/bin/env ruby
2
+ require 'gli'
3
+ require 'dropsonde'
4
+ require 'puppet'
5
+
6
+ class Dropsonde
7
+ extend GLI::App
8
+
9
+ Puppet.initialize_settings
10
+
11
+ program_desc 'A simple telemetry tool for Puppet infrastructures'
12
+ config_file '/etc/puppetlabs/telemetry.yaml'
13
+
14
+ desc 'Verbose logging'
15
+ switch [:verbose, :v]
16
+
17
+ desc 'Auto update the Forge module name cache if expired'
18
+ switch [:update], :default_value => true
19
+
20
+ desc 'Path to cache directory'
21
+ flag [:cachepath], :default_value => "#{Puppet.settings[:vardir]}/dropsonde"
22
+
23
+ desc 'Forge module cache ttl in days'
24
+ flag [:ttl], :default_value => 7, :type => Integer
25
+
26
+ desc 'List of metrics to omit'
27
+ flag [:blacklist, :b], :type => Array
28
+
29
+ desc 'Any number or string used to generate the randomized site ID.'
30
+ flag [:seed]
31
+
32
+ pre do |global, command, options, args|
33
+ Dropsonde.settings = global
34
+ Dropsonde::Cache.init(global[:cachepath], global[:ttl], global[:update])
35
+ end
36
+
37
+ desc 'Manually update the Forge module name cache'
38
+ command :update do |c|
39
+ c.action do |global, options, args|
40
+ Dropsonde::Cache.update
41
+ end
42
+ end
43
+
44
+ desc 'Generate a complete schema set'
45
+ command :schema do |c|
46
+ c.action do |global, options, args|
47
+ Dropsonde.generate_schema
48
+ end
49
+ end
50
+
51
+ desc 'List all available metrics'
52
+ command :list do |c|
53
+ c.action do |global, options, args|
54
+ Dropsonde.list_metrics
55
+ end
56
+ end
57
+
58
+ desc 'Generate an example telemetry report'
59
+ command :preview do |c|
60
+ c.desc 'The output format to use'
61
+ c.flag [:format], :default_value => 'human'
62
+
63
+ c.action do |global, options, args|
64
+ Dropsonde::Cache.autoupdate
65
+ Dropsonde.generate_report(options[:format])
66
+ end
67
+ end
68
+
69
+ desc 'Submit a telemetry report'
70
+ command :submit do |c|
71
+ c.desc 'Telemetry endpoint'
72
+ c.flag [:endpoint], :default_value => 'https://dev.dujour.k8s.puppet.net'
73
+
74
+ c.desc 'Telemetry port'
75
+ c.flag [:port], :default_value => 443, :type => Integer
76
+
77
+ c.action do |global, options, args|
78
+ Dropsonde::Cache.autoupdate
79
+ Dropsonde.submit_report(options[:endpoint], options[:port])
80
+ end
81
+ end
82
+ end
83
+
84
+ exit Dropsonde.run(ARGV)
@@ -0,0 +1,84 @@
1
+ require 'json'
2
+ require 'httpclient'
3
+ require 'puppetdb'
4
+ require 'inifile'
5
+
6
+ class Dropsonde
7
+ require 'dropsonde/cache'
8
+ require 'dropsonde/metrics'
9
+ require 'dropsonde/monkeypatches'
10
+
11
+ @@pdbclient = nil
12
+ @@settings = {}
13
+ def self.settings=(arg)
14
+ raise "Requires a Hash to set all settings at once, not a #{arg.class}" unless arg.is_a? Hash
15
+ @@settings = arg
16
+ end
17
+
18
+ def self.settings
19
+ @@settings
20
+ end
21
+
22
+ def self.generate_schema
23
+ puts JSON.pretty_generate(Dropsonde::Metrics.new.schema)
24
+ end
25
+
26
+ def self.list_metrics
27
+ puts
28
+ puts Dropsonde::Metrics.new.list
29
+ end
30
+
31
+ def self.generate_report(format)
32
+ case format
33
+ when 'json'
34
+ puts JSON.pretty_generate(Dropsonde::Metrics.new.report)
35
+ when 'human'
36
+ puts
37
+ puts Dropsonde::Metrics.new.preview
38
+ else
39
+ raise "unknown format"
40
+ end
41
+ end
42
+
43
+ def self.submit_report(endpoint, port)
44
+ client = HTTPClient.new()
45
+ result = client.post("#{endpoint}:#{port}",
46
+ :header => {'Content-Type' => 'application/json'},
47
+ :body => Dropsonde::Metrics.new.report.to_json
48
+ )
49
+
50
+ if result.status == 200
51
+ data = JSON.parse(result.body)
52
+ if data['newer']
53
+ puts 'A newer version of the telemetry client is available:'
54
+ puts " -- #{data['link']}"
55
+ else
56
+ puts data['message']
57
+ end
58
+ else
59
+ puts 'Failed to submit report'
60
+ puts JSON.pretty_generate(result.body) if Dropsonde.settings[:verbose]
61
+ exit 1
62
+ end
63
+ end
64
+
65
+ def self.puppetDB
66
+ return @@pdbclient if @@pdbclient
67
+
68
+ config = File.join(Puppet.settings[:confdir], 'puppetdb.conf')
69
+
70
+ return unless File.file? config
71
+
72
+ server = IniFile.load(config)['main']['server_urls'].split(',').first
73
+
74
+ @@pdbclient = PuppetDB::Client.new({
75
+ :server => server,
76
+ :pem => {
77
+ 'key' => Puppet.settings[:hostprivkey],
78
+ 'cert' => Puppet.settings[:hostcert],
79
+ 'ca_file' => Puppet.settings[:localcacert],
80
+ }
81
+ })
82
+ end
83
+
84
+ end
@@ -0,0 +1,76 @@
1
+ require 'date'
2
+ require 'json'
3
+ require 'fileutils'
4
+ require 'puppet_forge'
5
+
6
+ class Dropsonde::Cache
7
+ @@autoupdate = false
8
+
9
+ def self.init(path, ttl, autoupdate)
10
+ FileUtils.mkdir_p(path)
11
+ @@path = "#{File.expand_path(path)}/forge.json"
12
+ @@ttl = ttl
13
+ @@autoupdate = autoupdate
14
+
15
+ if File.file? @@path
16
+ @@cache = JSON.parse(File.read(@@path))
17
+ else
18
+ @@cache = {
19
+ 'timestamp' => '2000-1-1', # long before any puppet modules were released!
20
+ 'modules' => [],
21
+ }
22
+ end
23
+
24
+ PuppetForge.user_agent = "Dropsonde Telemetry Client/0.0.1"
25
+ end
26
+
27
+ def self.modules
28
+ @@cache['modules']
29
+ end
30
+
31
+ def self.forgeModule?(mod)
32
+ case mod
33
+ when Puppet::Module
34
+ modname = mod.forge_slug
35
+ when Hash
36
+ modname = mod[:name] || mod['name']
37
+ when String
38
+ modname = mod
39
+ end
40
+ return unless modname
41
+
42
+ modules.include? modname.tr('/','-')
43
+ end
44
+
45
+ def self.update
46
+ iter = PuppetForge::Module.all(:sort_by => 'latest_release')
47
+ newest = DateTime.parse(@@cache['timestamp'])
48
+
49
+ @@cache['timestamp'] = iter.first.created_at
50
+
51
+ until iter.next.nil?
52
+ # stop once we reach modules we've already cached
53
+ break if DateTime.parse(iter.first.created_at) <= newest
54
+
55
+ @@cache['modules'].concat iter.map {|mod| mod.slug }
56
+
57
+ iter = iter.next
58
+ print '.'
59
+ end
60
+ @@cache['modules'].sort!
61
+ @@cache['modules'].uniq!
62
+
63
+ File.write(@@path, JSON.pretty_generate(@@cache))
64
+ end
65
+
66
+ def self.autoupdate
67
+ return unless @@autoupdate
68
+
69
+ update unless File.file? @@path
70
+
71
+ if (Date.today - File.mtime(@@path).to_date).to_i > @@ttl
72
+ update
73
+ end
74
+ end
75
+
76
+ end
@@ -0,0 +1,184 @@
1
+ require 'little-plugger'
2
+
3
+ class Dropsonde::Metrics
4
+ extend LittlePlugger( :path => 'dropsonde/metrics', :module => Dropsonde::Metrics)
5
+
6
+ def initialize
7
+ Dropsonde::Metrics.disregard_plugins(*Dropsonde.settings[:blacklist])
8
+ Dropsonde::Metrics.initialize_plugins
9
+ end
10
+
11
+ def siteid
12
+ return @siteid if @siteid
13
+
14
+ sha2 = Digest::SHA512.new
15
+ sha2.update Puppet.settings[:certname]
16
+ sha2.update Puppet.settings[:cacert]
17
+ sha2.update Dropsonde.settings[:seed] if Dropsonde.settings[:seed]
18
+ @siteid = sha2.hexdigest
19
+ @siteid
20
+ end
21
+
22
+ def list
23
+ str = " Loaded telemetry plugins\n"
24
+ str << " ===============================\n\n"
25
+ Dropsonde::Metrics.plugins.each do |name, plugin|
26
+ str << name.to_s
27
+ str << "\n--------\n"
28
+ str << plugin.description.strip
29
+ str << "\n\n"
30
+ end
31
+ if Dropsonde.settings[:blacklist]
32
+ str << "Disabled plugins:\n"
33
+ str << " #{Dropsonde.settings[:blacklist].join(', ')}"
34
+ end
35
+ str
36
+ end
37
+
38
+ def schema
39
+ schema = skeleton_schema
40
+ Dropsonde::Metrics.plugins.each do |name, plugin|
41
+ schema.concat(sanity_check_schema(plugin))
42
+ end
43
+ check_for_duplicates(schema)
44
+ schema
45
+ end
46
+
47
+ def preview
48
+ str = " Puppet Telemetry Report Preview\n"
49
+ str << " ===============================\n\n"
50
+ Dropsonde::Metrics.plugins.each do |name, plugin|
51
+ schema = plugin.schema
52
+
53
+ plugin.setup
54
+ data = sanity_check_data(plugin)
55
+ plugin.cleanup
56
+
57
+ str << plugin.name+"\n"
58
+ str << "-------------------------------\n"
59
+ str << plugin.description
60
+ data.each do |row|
61
+ key = row.keys.first
62
+ values = row.values.first
63
+
64
+ desc = schema.find {|item| item[:name].to_sym == key.to_sym}[:description]
65
+ str << "- #{key}: #{desc}\n"
66
+ values.each do |item|
67
+ str << " #{item}\n"
68
+ end
69
+ end
70
+ str << "\n\n"
71
+ end
72
+ str << "Site ID:\n"
73
+ str << siteid
74
+ str
75
+ end
76
+
77
+ def report
78
+ snapshots = {}
79
+ Dropsonde::Metrics.plugins.each do |name, plugin|
80
+ plugin.setup
81
+ sanity_check_data(plugin).each do |row|
82
+ snapshots[row.keys.first] = {
83
+ 'value' => row.values.first,
84
+ 'timestamp' => Time.now.iso8601,
85
+ }
86
+ end
87
+ plugin.cleanup
88
+ end
89
+
90
+ results = skeleton_report
91
+ results[:'self-service-analytics'][:snapshots] = snapshots
92
+ results
93
+ end
94
+
95
+ def sanity_check_data(plugin)
96
+ data = plugin.run
97
+ keys_data = data.map {|item| item.keys }.flatten.map(&:to_s)
98
+ keys_schema = plugin.schema.map {|item| item[:name] }
99
+
100
+ disallowed = (keys_data - keys_schema)
101
+
102
+ raise "ERROR: The #{plugin.name} plugin exported the following keys not documented in the schema: #{disallowed}" unless disallowed.empty?
103
+
104
+ data
105
+ end
106
+
107
+ def sanity_check_schema(plugin)
108
+ schema = plugin.schema
109
+
110
+ if schema.class != Array or schema.find {|item| item.class != Hash}
111
+ raise "The #{plugin.name} plugin schema is not an array of hashes"
112
+ end
113
+
114
+ error = ''
115
+ [:name, :type, :description].each do |field|
116
+ count = schema.reject {|item| item[field] }.count
117
+ next if count == 0
118
+
119
+ error << "The #{plugin.name} plugin schema has #{count} missing #{field}s\n"
120
+ end
121
+ raise error unless error.empty?
122
+
123
+ schema
124
+ end
125
+
126
+ def check_for_duplicates(schema)
127
+ keys = schema.map {|col| col[:name] }
128
+ dupes = keys.select{ |e| keys.count(e) > 1 }.uniq
129
+
130
+ raise "The schema defines duplicate keys: #{dupes}" unless dupes.empty?
131
+ end
132
+
133
+ def skeleton_schema
134
+ [
135
+ {
136
+ "description": "An ID that's unique for each checkin to Dujour.",
137
+ "mode": "NULLABLE",
138
+ "name": "message_id",
139
+ "type": "STRING"
140
+ },
141
+ {
142
+ "description": "A unique identifier for a site, derived as a hash of the CA certificate and optional seed.",
143
+ "mode": "NULLABLE",
144
+ "name": "site_id",
145
+ "type": "BYTES"
146
+ },
147
+ {
148
+ "description": "The name of the product.",
149
+ "mode": "NULLABLE",
150
+ "name": "product",
151
+ "type": "STRING"
152
+ },
153
+ {
154
+ "description": "Version of the project.",
155
+ "mode": "NULLABLE",
156
+ "name": "version",
157
+ "type": "STRING"
158
+ },
159
+ {
160
+ "description": "Time the checkin to Dujour occurred.",
161
+ "mode": "NULLABLE",
162
+ "name": "timestamp",
163
+ "type": "TIMESTAMP"
164
+ },
165
+ {
166
+ "description": "IP Address of node checking in to Dujour.",
167
+ "mode": "NULLABLE",
168
+ "name": "ip",
169
+ "type": "STRING"
170
+ }
171
+ ]
172
+ end
173
+
174
+ def skeleton_report
175
+ {
176
+ "product": "popularity-module",
177
+ "version": "1.0.0",
178
+ "site_id": siteid,
179
+ "self-service-analytics": {
180
+ "snapshots": { }
181
+ }
182
+ }
183
+ end
184
+ end
@@ -0,0 +1,70 @@
1
+ class Dropsonde::Metrics::Dependencies
2
+ def self.initialize_dependencies
3
+ # require any libraries needed here -- no need to load puppet; it's already initialized
4
+ # All plugins are initialized before any metrics are generated.
5
+ end
6
+
7
+ def self.description
8
+ <<~EOF
9
+ This group of metrics discovers dependencies between modules in all
10
+ environments. It will omit dependencies on private modules.
11
+ EOF
12
+ end
13
+
14
+ def self.schema
15
+ # return an array of hashes of a partial schema to be merged into the complete schema
16
+ # See https://cloud.google.com/bigquery/docs/schemas#specifying_a_json_schema_file
17
+ [
18
+ {
19
+ "fields": [
20
+ {
21
+ "description": "The depended on module name",
22
+ "mode": "NULLABLE",
23
+ "name": "name",
24
+ "type": "STRING"
25
+ },
26
+ {
27
+ "description": "The depended on module version requirement",
28
+ "mode": "NULLABLE",
29
+ "name": "version_requirement",
30
+ "type": "STRING"
31
+ }
32
+ ],
33
+ "description": "List of modules that private modules in all environments depend on.",
34
+ "mode": "REPEATED",
35
+ "name": "dependencies",
36
+ "type": "RECORD"
37
+ }
38
+ ]
39
+ end
40
+
41
+ def self.setup
42
+ # run just before generating this metric
43
+ end
44
+
45
+ def self.run
46
+ # return an array of hashes representing the data to be merged into the combined checkin
47
+ environments = Puppet.lookup(:environments).list.map{|e|e.name}
48
+ modules = environments.map do |env|
49
+ Puppet.lookup(:environments).get(env).modules
50
+ end.flatten
51
+
52
+ # we want only PUBLIC modules that PRIVATE modules depend on
53
+ dependencies = modules.map do|mod|
54
+ next unless mod.dependencies
55
+ next if Dropsonde::Cache.forgeModule? mod # skip unless this is a private module
56
+
57
+ # and return a list of all public modules it depends on
58
+ mod.dependencies.select {|mod| Dropsonde::Cache.forgeModule? mod }
59
+ end.flatten.compact
60
+
61
+ [
62
+ { :dependencies => dependencies },
63
+ ]
64
+
65
+ end
66
+
67
+ def self.cleanup
68
+ # run just after generating this metric
69
+ end
70
+ end
@@ -0,0 +1,114 @@
1
+ class Dropsonde::Metrics::Modules
2
+ def self.initialize_modules
3
+ # require any libraries needed here -- no need to load puppet; it's already initialized
4
+ # All plugins are initialized before any metrics are generated.
5
+ end
6
+
7
+ def self.description
8
+ <<~EOF
9
+ This group of metrics exports name & version information about the public
10
+ modules installed in all environments, ignoring private modules.
11
+ EOF
12
+ end
13
+
14
+ def self.schema
15
+ # return an array of hashes of a partial schema to be merged into the complete schema
16
+ # See https://cloud.google.com/bigquery/docs/schemas#specifying_a_json_schema_file
17
+ [
18
+ {
19
+ "fields": [
20
+ {
21
+ "description": "The module name",
22
+ "mode": "NULLABLE",
23
+ "name": "name",
24
+ "type": "STRING"
25
+ },
26
+ {
27
+ "description": "The module slug (author-name)",
28
+ "mode": "NULLABLE",
29
+ "name": "slug",
30
+ "type": "STRING"
31
+ },
32
+ {
33
+ "description": "The module version",
34
+ "mode": "NULLABLE",
35
+ "name": "version",
36
+ "type": "STRING"
37
+ }
38
+ ],
39
+ "description": "List of modules in all environments.",
40
+ "mode": "REPEATED",
41
+ "name": "modules",
42
+ "type": "RECORD"
43
+ },
44
+ {
45
+ "fields": [
46
+ {
47
+ "description": "The class name",
48
+ "mode": "NULLABLE",
49
+ "name": "name",
50
+ "type": "STRING"
51
+ },
52
+ {
53
+ "description": "How many nodes it is declared on",
54
+ "mode": "NULLABLE",
55
+ "name": "count",
56
+ "type": "INTEGER"
57
+ }
58
+ ],
59
+ "description": "List of classes and counts in all environments.",
60
+ "mode": "REPEATED",
61
+ "name": "classes",
62
+ "type": "RECORD"
63
+ }
64
+ ]
65
+ end
66
+
67
+ def self.setup
68
+ # run just before generating this metric
69
+ end
70
+
71
+ def self.run
72
+ # return an array of hashes representing the data to be merged into the combined checkin
73
+ environments = Puppet.lookup(:environments).list.map{|e|e.name}
74
+ modules = environments.map do |env|
75
+ Puppet.lookup(:environments).get(env).modules.map do|mod|
76
+ next unless mod.forge_module?
77
+
78
+ {
79
+ :name => mod.name,
80
+ :slug => mod.forge_slug,
81
+ :version => mod.version,
82
+ }
83
+ end
84
+ end.flatten.compact.uniq
85
+
86
+ if Dropsonde.puppetDB
87
+ # classes and how many nodes they're enforced on
88
+ results = Dropsonde.puppetDB.request( '',
89
+ 'resources[certname, type, title] { type = "Class" }'
90
+ ).data
91
+
92
+ # select only classes from public modules
93
+ classes = results.map do |klass|
94
+ next unless modules.find {|mod| mod[:name] == klass['title'].split('::').first.downcase }
95
+
96
+ {
97
+ :name => klass['title'],
98
+ :count => results.count {|row| row['title'] == klass['title']},
99
+ }
100
+ end.compact.uniq
101
+ else
102
+ classes = []
103
+ end
104
+
105
+ [
106
+ { :modules => modules },
107
+ { :classes => classes },
108
+ ]
109
+ end
110
+
111
+ def self.cleanup
112
+ # run just after generating this metric
113
+ end
114
+ end
@@ -0,0 +1,76 @@
1
+ class Dropsonde::Metrics::Puppetfiles
2
+ def self.initialize_puppetfiles
3
+ # require any libraries needed here -- no need to load puppet; it's already initialized
4
+ # All plugins are initialized before any metrics are generated.
5
+ require 'ripper'
6
+ end
7
+
8
+ def self.description
9
+ <<~EOF
10
+ This generates interesting stats about Puppetfiles used in your environments,
11
+ including whether your Puppetfiles have Ruby code in them.
12
+ EOF
13
+ end
14
+
15
+ def self.schema
16
+ # return an array of hashes of a partial schema to be merged into the complete schema
17
+ # See https://cloud.google.com/bigquery/docs/schemas#specifying_a_json_schema_file
18
+ [
19
+ {
20
+ "fields": [
21
+ {
22
+ "description": "The method name",
23
+ "mode": "NULLABLE",
24
+ "name": "name",
25
+ "type": "STRING"
26
+ },
27
+ {
28
+ "description": "How many times is it used",
29
+ "mode": "NULLABLE",
30
+ "name": "count",
31
+ "type": "INTEGER"
32
+ }
33
+ ],
34
+ "description": "Ruby methods used in Puppetfiles.",
35
+ "mode": "REPEATED",
36
+ "name": "puppetfile_ruby_methods",
37
+ "type": "RECORD"
38
+ }
39
+ ]
40
+ end
41
+
42
+ def self.setup
43
+ # run just before generating this metric
44
+ end
45
+
46
+ def self.run
47
+ methods = Dir.entries(Puppet.settings[:environmentpath]).map do |entry|
48
+ puppetfile = File.join(Puppet.settings[:environmentpath], entry, 'Puppetfile')
49
+
50
+ next if entry.start_with? '.'
51
+ next unless File.file? puppetfile
52
+
53
+ tokens = Ripper.sexp(File.read(puppetfile)).flatten
54
+ indices = tokens.map.with_index {|a, i| a == :command ? i : nil}.compact
55
+
56
+ indices.map {|i| tokens[i+2] }
57
+ end.flatten.compact
58
+
59
+ methods.reject! {|name| ['mod', 'forge', 'moduledir'].include? name }
60
+
61
+ methods = methods.uniq.map do |name|
62
+ {
63
+ :name => name,
64
+ :count => methods.count(name),
65
+ }
66
+ end
67
+
68
+ [
69
+ { :puppetfile_ruby_methods => methods },
70
+ ]
71
+ end
72
+
73
+ def self.cleanup
74
+ # run just after generating this metric
75
+ end
76
+ end
@@ -0,0 +1,17 @@
1
+ module Puppet
2
+ class Module
3
+
4
+ unless Module.method_defined? :"forge_module?"
5
+ def forge_module?
6
+ Dropsonde::Cache.forgeModule? self
7
+ end
8
+ end
9
+
10
+ unless Module.method_defined? :forge_slug
11
+ def forge_slug
12
+ self.forge_name.tr('/','-') rescue nil
13
+ end
14
+ end
15
+
16
+ end
17
+ end
metadata ADDED
@@ -0,0 +1,182 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: dropsonde
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1
5
+ platform: ruby
6
+ authors:
7
+ - Ben Ford
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2020-03-20 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: json
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: gli
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: httpclient
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: little-plugger
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ - !ruby/object:Gem::Dependency
70
+ name: puppet
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - ">="
74
+ - !ruby/object:Gem::Version
75
+ version: '0'
76
+ type: :runtime
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - ">="
81
+ - !ruby/object:Gem::Version
82
+ version: '0'
83
+ - !ruby/object:Gem::Dependency
84
+ name: puppet_forge
85
+ requirement: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - ">="
88
+ - !ruby/object:Gem::Version
89
+ version: '0'
90
+ type: :runtime
91
+ prerelease: false
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - ">="
95
+ - !ruby/object:Gem::Version
96
+ version: '0'
97
+ - !ruby/object:Gem::Dependency
98
+ name: semantic_puppet
99
+ requirement: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - ">="
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ type: :runtime
105
+ prerelease: false
106
+ version_requirements: !ruby/object:Gem::Requirement
107
+ requirements:
108
+ - - ">="
109
+ - !ruby/object:Gem::Version
110
+ version: '0'
111
+ - !ruby/object:Gem::Dependency
112
+ name: inifile
113
+ requirement: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - ">="
116
+ - !ruby/object:Gem::Version
117
+ version: '0'
118
+ type: :runtime
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - ">="
123
+ - !ruby/object:Gem::Version
124
+ version: '0'
125
+ - !ruby/object:Gem::Dependency
126
+ name: puppetdb-ruby
127
+ requirement: !ruby/object:Gem::Requirement
128
+ requirements:
129
+ - - ">="
130
+ - !ruby/object:Gem::Version
131
+ version: '0'
132
+ type: :runtime
133
+ prerelease: false
134
+ version_requirements: !ruby/object:Gem::Requirement
135
+ requirements:
136
+ - - ">="
137
+ - !ruby/object:Gem::Version
138
+ version: '0'
139
+ description: |
140
+ Dropsonde is a simple telemetry probe designed to run as a regular cron job. It
141
+ will gather metrics defined by self-contained plugins that each defines its own
142
+ partial schema and then gathers the data to meet that schema.
143
+ email: ben.ford@puppet.com
144
+ executables:
145
+ - dropsonde
146
+ extensions: []
147
+ extra_rdoc_files: []
148
+ files:
149
+ - LICENSE
150
+ - README.md
151
+ - bin/dropsonde
152
+ - lib/dropsonde.rb
153
+ - lib/dropsonde/cache.rb
154
+ - lib/dropsonde/metrics.rb
155
+ - lib/dropsonde/metrics/dependencies.rb
156
+ - lib/dropsonde/metrics/modules.rb
157
+ - lib/dropsonde/metrics/puppetfiles.rb
158
+ - lib/dropsonde/monkeypatches.rb
159
+ homepage:
160
+ licenses:
161
+ - Apache 2
162
+ metadata: {}
163
+ post_install_message:
164
+ rdoc_options: []
165
+ require_paths:
166
+ - lib
167
+ required_ruby_version: !ruby/object:Gem::Requirement
168
+ requirements:
169
+ - - ">="
170
+ - !ruby/object:Gem::Version
171
+ version: '0'
172
+ required_rubygems_version: !ruby/object:Gem::Requirement
173
+ requirements:
174
+ - - ">="
175
+ - !ruby/object:Gem::Version
176
+ version: '0'
177
+ requirements: []
178
+ rubygems_version: 3.0.3
179
+ signing_key:
180
+ specification_version: 4
181
+ summary: A simple telemetry probe for gathering usage information about Puppet infrastructures.
182
+ test_files: []