dropsonde 0.0.3 → 0.0.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0ac6f28ed9d64982c400f6a7460b57f44396020e6f17f04f587cc6e5007ff115
-  data.tar.gz: ef5a2ac1957b4b45c756b0e4bbc5ae66291706afabd3a7b80e899702f8aa6757
+  metadata.gz: b2b344789cd257a8cae81d53150c1ab50fe0cf20716c4fce306446a2c207c7e2
+  data.tar.gz: 563788ccfcf13c75feb8529e4870ba65a0679a1d461eda2413cd401c0c52decd
 SHA512:
-  metadata.gz: d9bda687091ec5a1d6bd8ca7e5f1414e2c489e3a1c66a970b879d7f6a2d6ab82e11f079eebfea13ed7a710ebe20b4c0aefd0a08f1af17547e3a5cd120c587847
-  data.tar.gz: e96d699c38c0d158bce376f13d77d9bafa3ef5d4020e104d32694df04fcc68c91bb458d2bb858f9d120dc5b9146ffd3bfde450535cfe0b971b331b349e7dc78b
+  metadata.gz: 5006cf40dd17e018b6af805ae6192d5cc8b1842b981a4eba6895966a7ace7ee13365b9bae976169adaad1399ee9ca767d10b56cef56542ba5420cd04cecafdb6
+  data.tar.gz: ebda439d8baeec73d04887fb03b745acb2a11b9af8621f8992ad585b9ea9380f4c5732f88a5c17661f8123cba8125dc781d2929302a0954511bd6fae221c78f4

data/CHANGELOG.md

@@ -0,0 +1,58 @@
+# Changelog
+
+## [0.0.7](https://github.com/puppetlabs/dropsonde/tree/0.0.7) (2022-02-10)
+
+[Full Changelog](https://github.com/puppetlabs/dropsonde/compare/v0.0.6...0.0.7)
+
+**Merged pull requests:**
+
+- update server language [\#20](https://github.com/puppetlabs/dropsonde/pull/20) ([binford2k](https://github.com/binford2k))
+- Updates the telemetry endpoint to a stable URL [\#19](https://github.com/puppetlabs/dropsonde/pull/19) ([MikaelSmith](https://github.com/MikaelSmith))
+- docs updates now that it's shipped [\#18](https://github.com/puppetlabs/dropsonde/pull/18) ([binford2k](https://github.com/binford2k))
+
+## [v0.0.6](https://github.com/puppetlabs/dropsonde/tree/v0.0.6) (2021-11-17)
+
+[Full Changelog](https://github.com/puppetlabs/dropsonde/compare/v0.0.5...v0.0.6)
+
+**Merged pull requests:**
+
+- \(MAINT\) Update gemspec license param to 'Apache-2.0' [\#17](https://github.com/puppetlabs/dropsonde/pull/17) ([sanfrancrisko](https://github.com/sanfrancrisko))
+- \(MAINT\) Release prep for 0.0.6 [\#16](https://github.com/puppetlabs/dropsonde/pull/16) ([sanfrancrisko](https://github.com/sanfrancrisko))
+- \(MAINT\) Fix Rubocop violations [\#15](https://github.com/puppetlabs/dropsonde/pull/15) ([sanfrancrisko](https://github.com/sanfrancrisko))
+- \(SERVER-3079\) Allow overriding Puppet's base settings [\#14](https://github.com/puppetlabs/dropsonde/pull/14) ([Magisus](https://github.com/Magisus))
+- Load Ruby's CA certificates instead of using httpclient defaults [\#13](https://github.com/puppetlabs/dropsonde/pull/13) ([Magisus](https://github.com/Magisus))
+- \(IAC-1676\) - Add nightly workflow [\#12](https://github.com/puppetlabs/dropsonde/pull/12) ([adrianiurca](https://github.com/adrianiurca))
+- \(IAC-1659\) - Add Acceptance Tests [\#11](https://github.com/puppetlabs/dropsonde/pull/11) ([adrianiurca](https://github.com/adrianiurca))
+- \(IAC-1624\) - Expand unit test case suit [\#9](https://github.com/puppetlabs/dropsonde/pull/9) ([adrianiurca](https://github.com/adrianiurca))
+- \(IAC-1620\) - replace travis workflow with github actions [\#8](https://github.com/puppetlabs/dropsonde/pull/8) ([adrianiurca](https://github.com/adrianiurca))
+- \(IAC-1605\) - add static checks - rubocop [\#5](https://github.com/puppetlabs/dropsonde/pull/5) ([adrianiurca](https://github.com/adrianiurca))
+- Allow to specify siteid with a flag [\#4](https://github.com/puppetlabs/dropsonde/pull/4) ([raphink](https://github.com/raphink))
+
+## [v0.0.5](https://github.com/puppetlabs/dropsonde/tree/v0.0.5) (2020-06-11)
+
+[Full Changelog](https://github.com/puppetlabs/dropsonde/compare/v0.0.4...v0.0.5)
+
+## [v0.0.4](https://github.com/puppetlabs/dropsonde/tree/v0.0.4) (2020-06-05)
+
+[Full Changelog](https://github.com/puppetlabs/dropsonde/compare/v0.0.3...v0.0.4)
+
+## [v0.0.3](https://github.com/puppetlabs/dropsonde/tree/v0.0.3) (2020-05-24)
+
+[Full Changelog](https://github.com/puppetlabs/dropsonde/compare/v0.0.2...v0.0.3)
+
+## [v0.0.2](https://github.com/puppetlabs/dropsonde/tree/v0.0.2) (2020-05-08)
+
+[Full Changelog](https://github.com/puppetlabs/dropsonde/compare/v0.0.1...v0.0.2)
+
+## [v0.0.1](https://github.com/puppetlabs/dropsonde/tree/v0.0.1) (2020-02-27)
+
+[Full Changelog](https://github.com/puppetlabs/dropsonde/compare/502bce39de366c8d6816676e0cf7ca6284b90734...v0.0.1)
+
+**Merged pull requests:**
+
+- Add simple puppetfile metric [\#2](https://github.com/puppetlabs/dropsonde/pull/2) ([binford2k](https://github.com/binford2k))
+- MVP workable tool [\#1](https://github.com/puppetlabs/dropsonde/pull/1) ([binford2k](https://github.com/binford2k))
+
+
+
+\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*

data/README.md

@@ -15,8 +15,7 @@ data in a way that lets you signal approval simply by using a module is the
 primary goal of this project.
 
 > **This means that the best way for you to help yourself find new modules is to
-> install this telemetry tool and, along with your peers, share your module usage
-> data.** ✅
+> opt in and use this telemetry tool to share your module usage data.** ✅
 
 
 ## Design
@@ -34,19 +33,24 @@ You can see exactly what will be phoned home by running the command:
 $ dropsonde preview
 ```
 
-Dropsonde is a simple telemetry probe designed to run as a regular cron job.
-Metrics are defined by plugins that gather data, but also export a schema that
-constrains the data allowed to be reported on. Dropsonde will reject metrics
-that don't meet these constraints. The backend database is also defined by this
-schema so the system cannot gather any data that's not described in the schema.
+> *Note:* as of Puppetserver 7.5.0, Dropsonde is bundled in as a standard `puppetserver`
+> command. You should run it as `puppetserver dropsonde` (or `/opt/puppetlabs/bin/puppetserver
+> dropsonde` if your `$PATH` isn't configured.)
+
+Dropsonde is a simple telemetry probe designed to run both as a command-line
+tool and optionally as a regularly scheduled task. Metrics are defined by
+plugins that gather data, but also export a schema that constrains the data
+allowed to be reported on. Dropsonde will reject metrics that don't meet these
+constraints. The backend database is also defined by this schema so the system
+cannot gather any data that's not described in the schema.
 
 See the full schema of all enabled plugins by running the command:
 
 ```
-$ dropsonde schema
+$ dropsonde dev schema
 ```
 
-All information in the report is keyed off a non-reversible SHA512 hash site-id
+All information in the report is keyed off a non-reversible SHA512 hashed site-id
 to make it unidentifiable; this report cannot be linked back to you or to your
 infrastructure. Now that said, we know that the more bits of data shared about a
 specific site, the easier it is to fingerprint that site. See
@@ -63,6 +67,8 @@ For example, this aggregated data might include records that show a count of how
 many sites are using various combinations of modules together, but it will never
 include a record showing the full list of modules that any single site is using.
 
+![Data aggregation workflow](./aggregation.png)
+
 With your own Google Cloud account, you can use that [dataset](https://console.cloud.google.com/bigquery?p=dataops-puppet-public-data&d=community&t=forge_modules&page=table)
 in your own tooling and you can see/contribute to the aggregation queries in its
 own [repository](https://github.com/puppetlabs/dropsonde-aggregation).
@@ -85,9 +91,11 @@ possible: [privacy@puppet.com](mailto:privacy@puppet.com)
 
 ## Installation
 
-This is distributed as a Ruby gem. Simply `gem install dropsonde`. There's a
-[Puppet module](https://github.com/puppetlabs/puppetlabs-dropsonde) to manage it
-if that's more your thing.
+As of Puppetserver 7.5.0, Dropsonde is bundled in as a standard `puppetserver`
+command. If you're running an older version, you can install it yourself with
+`gem install dropsonde` or let the
+[Puppet module](https://github.com/puppetlabs/puppetlabs-dropsonde) manage it
+for you.
 
 
 ## Configuration
@@ -100,11 +108,33 @@ will not report the `:puppetfiles` metrics.
 ``` yaml
 ---
 :update: false
-:blacklist:
+:disable:
   - puppetfiles
 ```
 
-The `puppetlabs-dropsonde` Puppet module manages this configuration for you.
+The [`puppetlabs-dropsonde`](https://github.com/puppetlabs/puppetlabs-dropsonde)
+Puppet module manages this configuration for you.
+
+
+## Scheduled Reporting
+
+Dropsonde is intended to run on a regular schedule to submit usage reports back
+to Puppet. The [`puppetlabs-dropsonde`](https://github.com/puppetlabs/puppetlabs-dropsonde)
+Puppet module will manage this for you, or you can follow the instructions below
+to configure a schedule manually.
+
+If you're running Puppetserver 7.5.0 or above, it's already set up so you can
+simply opt in. Add or update the following stanza of
+`/etc/puppetlabs/puppetserver/conf.d/puppetserver.conf`:
+
+``` hocon
+dropsonde: {
+    enabled: true
+}
+```
+
+If you're running an older version of Puppetserver, then create a cron job that
+runs `dropsonde submit` once a week.
 
 
 ## Running
@@ -137,7 +167,7 @@ Developer comands
 
 ## Architecture
 
-Dropsonde is a simple telemetry probe designed to run as a regular cron job. It
+Dropsonde is a simple telemetry probe designed to run as a scheduled task. It
 will gather metrics defined by self-contained plugins that each defines its own
 partial schema and then gathers the data to meet that schema.
 
@@ -153,7 +183,8 @@ of modules is updated.
 
 ## Limitations
 
-This is super early in development and has not yet been battle tested.
+The output format of individual metrics is not yet formally defined. If you write
+tooling to use it, then make your tooling resilient to changes.
 
 
 Contact

data/bin/dropsonde

@@ -5,6 +5,9 @@ require 'dropsonde'
 class Dropsonde
   extend GLI::App
 
+  @cache = nil
+  @puppetdb_session = Dropsonde.new
+
   program_desc 'A simple telemetry tool for Puppet infrastructures'
   config_file  "#{File.dirname(Puppet.settings[:confdir])}/telemetry.yaml"
   version      Dropsonde::VERSION
@@ -22,20 +25,26 @@ class Dropsonde
   flag [:ttl], :default_value => 7, :type => Integer
 
   desc 'List of metrics to omit'
-  flag [:blacklist, :b], :type => Array
+  flag [:disable, :d], :type => Array
+
+  desc 'Only load these metrics'
+  flag [:enable, :e], :type => Array
 
   desc 'Any number or string used to generate the randomized site ID.'
   flag [:seed]
 
+  desc 'Static site ID'
+  flag [:siteid]
+
   pre do |global, command, options, args|
     Dropsonde.settings = global
-    Dropsonde::Cache.init(global[:cachepath], global[:ttl], global[:update])
+    @cache = Dropsonde::Cache.new(global[:cachepath], global[:ttl], global[:update])
   end
 
   desc 'Manually update the Forge module name cache'
   command :update do |c|
     c.action do |global, options, args|
-      Dropsonde::Cache.update
+      @cache.update
     end
   end
 
@@ -52,21 +61,21 @@ class Dropsonde
     c.flag [:format], :default_value => 'human'
 
     c.action do |global, options, args|
-      Dropsonde::Cache.autoupdate
-      Dropsonde.generate_report(options[:format])
+      @cache.autoupdate
+      Dropsonde.generate_report(options[:format], @puppetdb_session)
     end
   end
 
   desc 'Submit a telemetry report'
   command :submit do |c|
     c.desc 'Telemetry endpoint'
-    c.flag [:endpoint], :default_value => 'https://prod.dujour.k8s.puppet.net'
+    c.flag [:endpoint], :default_value => 'https://updates.puppet.com'
 
     c.desc 'Telemetry port'
     c.flag [:port], :default_value => 443, :type => Integer
 
     c.action do |global, options, args|
-      Dropsonde::Cache.autoupdate
+      @cache.autoupdate
       Dropsonde.submit_report(options[:endpoint], options[:port])
     end
   end
@@ -102,13 +111,13 @@ class Dropsonde
                 This is in jsonl format for direct upload to BigQuery."
     t.command :example do |c|
       c.desc 'How many rows to generate'
-      c.flag [:size], :default_value => 100
+      c.flag [:size], :default_value => 100, :type => Integer
 
       c.desc 'Filename for the output (in jsonl format).'
       c.flag [:filename], :default_value => 'example.jsonl'
 
       c.action do |global, options, args|
-        Dropsonde::Cache.autoupdate
+        @cache.autoupdate
         Dropsonde.generate_example(options[:size], options[:filename])
       end
     end

data/lib/dropsonde/cache.rb

@@ -1,34 +1,41 @@
+# frozen_string_literal: true
+
 require 'date'
 require 'json'
 require 'fileutils'
 require 'puppet_forge'
 
+# cache class
 class Dropsonde::Cache
-  @@autoupdate = false
+  @autoupdate = false
 
-  def self.init(path, ttl, autoupdate)
+  def initialize(path, ttl, autoupdate)
     FileUtils.mkdir_p(path)
-    @@path = "#{File.expand_path(path)}/forge.json"
-    @@ttl  = ttl
-    @@autoupdate = autoupdate
-
-    if File.file? @@path
-      @@cache = JSON.parse(File.read(@@path))
-    else
-      @@cache = {
-                'timestamp' => '2000-1-1',  # long before any puppet modules were released!
-                'modules'   => [],
-              }
-    end
-
-    PuppetForge.user_agent = "Dropsonde Telemetry Client/0.0.1"
+    @path = "#{File.expand_path(path)}/forge.json"
+    @ttl  = ttl
+    @autoupdate = autoupdate
+
+    @@cache = if File.file? @path # rubocop:disable Style/ClassVars
+                JSON.parse(File.read(@path))
+              else
+                {
+                  'timestamp' => '2000-1-1', # long before any puppet modules were released!
+                  'modules' => [],
+                }
+              end
+
+    PuppetForge.user_agent = 'Dropsonde Telemetry Client/0.0.1'
   end
 
-  def self.modules
+  def modules
     @@cache['modules']
   end
 
-  def self.forgeModule?(mod)
+  def cache
+    @@cache
+  end
+
+  def self.forge_module?(mod)
     case mod
     when Puppet::Module
       modname = mod.forge_slug
@@ -39,12 +46,12 @@ class Dropsonde::Cache
     end
     return unless modname
 
-    modules.include? modname.tr('/','-')
+    @@cache['modules'].include? modname.tr('/', '-')
   end
 
-  def self.update
-    puts "Updating module cache..."
-    iter   = PuppetForge::Module.all(:sort_by => 'latest_release')
+  def update
+    puts 'Updating module cache...'
+    iter   = PuppetForge::Module.all(sort_by: 'latest_release')
     newest = DateTime.parse(@@cache['timestamp'])
 
     @@cache['timestamp'] = iter.first.updated_at
@@ -53,7 +60,7 @@ class Dropsonde::Cache
       # stop once we reach modules we've already cached
       break if DateTime.parse(iter.first.updated_at) <= newest
 
-      @@cache['modules'].concat iter.map {|mod| mod.slug }
+      @@cache['modules'].concat(iter.map { |mod| mod.slug })
 
       iter = iter.next
       print '.'
@@ -62,22 +69,19 @@ class Dropsonde::Cache
     @@cache['modules'].sort!
     @@cache['modules'].uniq!
 
-    File.write(@@path, JSON.pretty_generate(@@cache))
+    File.write(@path, JSON.pretty_generate(@@cache))
   end
 
-  def self.autoupdate
-    return unless @@autoupdate
+  def autoupdate
+    return unless @autoupdate
 
-    unless File.file? @@path
-      puts "Dropsonde caches a list of all Forge modules to ensure that it only reports"
-      puts "usage data on public modules. Generating this cache may take some time on"
+    unless File.file? @path
+      puts 'Dropsonde caches a list of all Forge modules to ensure that it only reports'
+      puts 'usage data on public modules. Generating this cache may take some time on'
       puts "the first run and you'll see your screen fill up with dots."
       update
     end
 
-    if (Date.today - File.mtime(@@path).to_date).to_i > @@ttl
-      update
-    end
+    return update if (Date.today - File.mtime(@path).to_date).to_i > @ttl
   end
-
 end

data/lib/dropsonde/metrics/dependencies.rb

@@ -1,3 +1,6 @@
+# frozen_string_literal: true
+
+# dependencies plugin
 class Dropsonde::Metrics::Dependencies
   def self.initialize_dependencies
     # require any libraries needed here -- no need to load puppet; it's already initialized
@@ -5,10 +8,10 @@ class Dropsonde::Metrics::Dependencies
   end
 
   def self.description
-    <<~EOF
+    <<~DESCRIPTION
       This group of metrics discovers dependencies between modules in all
       environments. It will omit dependencies on private modules.
-    EOF
+    DESCRIPTION
   end
 
   def self.schema
@@ -18,23 +21,23 @@ class Dropsonde::Metrics::Dependencies
       {
         "fields": [
           {
-            "description": "The depended on module name",
-            "mode": "NULLABLE",
-            "name": "name",
-            "type": "STRING"
+            "description": 'The depended on module name',
+            "mode": 'NULLABLE',
+            "name": 'name',
+            "type": 'STRING',
           },
           {
-            "description": "The depended on module version requirement",
-            "mode": "NULLABLE",
-            "name": "version_requirement",
-            "type": "STRING"
-          }
+            "description": 'The depended on module version requirement',
+            "mode": 'NULLABLE',
+            "name": 'version_requirement',
+            "type": 'STRING',
+          },
         ],
-        "description": "List of modules that private modules in all environments depend on.",
-        "mode": "REPEATED",
-        "name": "dependencies",
-        "type": "RECORD"
-      }
+        "description": 'List of modules that private modules in all environments depend on.',
+        "mode": 'REPEATED',
+        "name": 'dependencies',
+        "type": 'RECORD',
+      },
     ]
   end
 
@@ -42,26 +45,25 @@ class Dropsonde::Metrics::Dependencies
     # run just before generating this metric
   end
 
-  def self.run
+  def self.run(_puppetdb_session = nil)
     # return an array of hashes representing the data to be merged into the combined checkin
-    environments = Puppet.lookup(:environments).list.map{|e|e.name}
-    modules = environments.map do |env|
+    environments = Puppet.lookup(:environments).list.map { |e| e.name }
+    modules = environments.map { |env|
       Puppet.lookup(:environments).get(env).modules
-    end.flatten
+    }.flatten
 
     # we want only PUBLIC modules that PRIVATE modules depend on
-    dependencies = modules.map do|mod|
+    dependencies = modules.map { |mod|
       next unless mod.dependencies
-      next if Dropsonde::Cache.forgeModule? mod  # skip unless this is a private module
+      next if Dropsonde::Cache.forge_module? mod # skip unless this is a private module
 
       # and return a list of all public modules it depends on
-      mod.dependencies.select {|mod| Dropsonde::Cache.forgeModule? mod }
-    end.flatten.compact
+      mod.dependencies.select { |dep| Dropsonde::Cache.forge_module? dep }
+    }.flatten.compact
 
     [
-      { :dependencies => dependencies },
+      { dependencies: dependencies },
     ]
-
   end
 
   def self.example
@@ -69,14 +71,17 @@ class Dropsonde::Metrics::Dependencies
     # make it easier to write data aggregation queries without access to the
     # actual private data that users have submitted.
 
+    dropsonde_cache = Dropsonde::Cache.new('foo', 7, true)
     versions = ['>= 1.5.2', '>= 4.3.2', '>= 3.0.0 < 4.0.0', '>= 2.2.1 < 5.0.0', '>= 5.0.0 < 7.0.0', '>= 4.11.0']
     [
-      :dependencies => Dropsonde::Cache.modules
-                                  .sample(rand(250))
-                                  .map {|item| {
-                                    :name                => item,
-                                    :version_requirement => versions.sample,
-                                  }},
+      dependencies: dropsonde_cache.modules
+                                   .sample(rand(250))
+                                   .map do |item|
+                      {
+                        name: item,
+                        version_requirement: versions.sample,
+                      }
+                    end,
     ]
   end
 

data/lib/dropsonde/metrics/environments.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+# environments plugin
+class Dropsonde::Metrics::Environments
+  def self.initialize_environments
+    # Require any libraries needed here -- no need to load puppet or puppetdb;
+    # they're already loaded. This hook is named after the class name.
+    # All plugins are initialized at startup before any metrics are generated.
+  end
+
+  def self.description
+    # This is a Ruby squiggle heredoc; just a multi-line string with indentation removed
+    <<~DESCRIPTION
+      This group of metrics gathers information about environments.
+    DESCRIPTION
+  end
+
+  def self.schema
+    # return an array of hashes of a partial schema to be merged into the complete schema
+    [
+      {
+        "description": 'The number of environments',
+        "mode": 'NULLABLE',
+        "name": 'environment_count',
+        "type": 'INTEGER',
+      },
+    ]
+  end
+
+  def self.setup
+    # run just before generating this metric
+  end
+
+  def self.run(_puppetdb_session = nil)
+    # return an array of hashes representing the data to be merged into the combined checkin
+    [
+      environment_count: Puppet.lookup(:environments).list.count,
+    ]
+  end
+
+  def self.example
+    # this method is used to generate a table filled with randomized data to
+    # make it easier to write data aggregation queries without access to the
+    # actual private data that users have submitted.
+    [
+      environment_count: rand(1..100),
+    ]
+  end
+
+  def self.cleanup
+    # run just after generating this metric
+  end
+end