vm-client 1.0.0

data/lib/prometheus/client/data_stores/README.md

@@ -0,0 +1,306 @@
+# Custom Data Stores
+
+Stores are basically an abstraction over a Hash, whose keys are in turn a Hash of labels
+plus a metric name. The intention behind having different data stores is solving 
+different requirements for different production scenarios, or performance trade-offs.
+
+The most common of these scenarios are pre-fork servers like Unicorn, which have multiple
+separate processes gathering metrics. If each of these had their own store, the metrics
+reported on each Prometheus scrape would be different, depending on which process handles
+the request. Solving this requires some sort of shared storage between these processes, 
+and there are many ways to solve this problem, each with their own trade-offs.
+
+This abstraction allows us to easily plug in the most adequate store for each scenario.
+
+## Interface
+   
+`Store` exposes a `for_metric` method, which returns a store-specific and metric-specific 
+`MetricStore` object, which represents a "view" onto the actual internal storage for one
+particular metric. Each metric / collector object will have a references to this 
+`MetricStore` and interact with it directly. 
+
+The `MetricStore` class must expose `synchronize`, `set`, `increment`, `get` and `all_values`
+methods,  which are explained in the code sample below. Its initializer should be called 
+only by `Store#for_metric`, not directly.
+
+All values stored are `Float`s.
+
+Internally, a `Store` can store the data however it needs to, based on its requirements. 
+For example, a store that needs to work in a multi-process environment needs to have a 
+shared section of memory, via either Files, an MMap, an external database, or whatever the 
+implementor chooses for their particular use case.
+
+Each `Store` / `MetricStore` will also choose how to divide responsibilities over the 
+storage of values. For some use cases, each `MetricStore` may have their own individual 
+storage, whereas for others, the `Store` may own a central storage, and `MetricStore` 
+objects will access it through the `Store`. This depends on the design choices of each `Store`. 
+
+`Store` and `MetricStore` MUST be thread safe. This applies not only to operations on 
+stored values (`set`, `increment`), but `MetricStore` must also expose a `synchronize`
+method that would allow a Metric to increment multiple values atomically (Histograms need
+this, for example).
+
+Ideally, multiple keys should be modifiable simultaneously, but this is not a
+hard requirement.
+
+This is what the interface looks like, in practice:
+
+```ruby
+module Prometheus
+  module Client
+    module DataStores
+      class CustomStore
+      
+        # Return a MetricStore, which provides a view of the internal data store, 
+        # catering specifically to that metric.
+        #
+        # - `metric_settings` specifies configuration parameters for this metric 
+        #   specifically. These may or may not be necessary, depending on each specific
+        #   store and metric. The most obvious example of this is for gauges in 
+        #   multi-process environments, where the developer needs to choose how those 
+        #   gauges will get aggregated between all the per-process values.
+        # 
+        #   The settings that the store will accept, and what it will do with them, are
+        #   100% Store-specific. Each store should document what settings it will accept
+        #   and how to use them, so the developer using that store can pass the appropriate 
+        #   instantiating the Store itself, and the Metrics they declare.
+        #
+        # - `metric_type` is specified in case a store wants to validate that the settings
+        #   are valid for the metric being set up. It may go unused by most Stores
+        #
+        # Even if your store doesn't need these two parameters, the Store must expose them
+        # to make them swappable.   
+        def for_metric(metric_name, metric_type:, metric_settings: {})
+          # Generally, here a Store would validate that the settings passed in are valid,
+          # and raise if they aren't.
+          validate_metric_settings(metric_type: metric_type, 
+                                   metric_settings: metric_settings)
+          MetricStore.new(store: self, 
+                          metric_name: metric_name, 
+                          metric_type: metric_type, 
+                          metric_settings: metric_settings)
+        end
+
+        
+        # MetricStore manages the data for one specific metric. It's generally a view onto
+        # the central store shared by all metrics, but it could also hold the data itself
+        # if that's better for the specific scenario 
+        class MetricStore
+          # This constructor is internal to this store, so the signature doesn't need to
+          # be this. No one other than the Store should be creating MetricStores 
+          def initialize(store:, metric_name:, metric_type:, metric_settings:)
+          end
+
+          # Metrics may need to modify multiple values at once (Histograms do this, for 
+          # example). MetricStore needs to provide a way to synchronize those, in addition
+          # to all of the value modifications being thread-safe without a need for simple 
+          # Metrics to call `synchronize`
+          def synchronize
+            raise NotImplementedError
+          end
+
+          # Store a value for this metric and a set of labels
+          # Internally, may add extra "labels" to disambiguate values between,
+          # for example, different processes
+          def set(labels:, val:)
+            raise NotImplementedError
+          end
+
+          def increment(labels:, by: 1)
+            raise NotImplementedError
+          end
+  
+          # Return a value for a set of labels
+          # Will return the same value stored by `set`, as opposed to `all_values`, which 
+          # may aggregate multiple values.
+          #
+          # For example, in a multi-process scenario, `set` may add an extra internal
+          # label tagging the value with the process id. `get` will return the value for
+          # "this" process ID. `all_values` will return an aggregated value for all 
+          # process IDs.
+          def get(labels:)
+            raise NotImplementedError
+          end
+  
+          # Returns all the sets of labels seen by the Store, and the aggregated value for 
+          # each.
+          # 
+          # In some cases, this is just a matter of returning the stored value.
+          # 
+          # In other cases, the store may need to aggregate multiple values for the same
+          # set of labels. For example, in a multiple process it may need to `sum` the
+          # values of counters from each process. Or for `gauges`, it may need to take the
+          # `max`. This is generally specified in `metric_settings` when calling 
+          # `Store#for_metric`.
+          def all_values
+            raise NotImplementedError
+          end
+        end
+      end
+    end
+  end
+end
+```
+
+## Conventions
+
+- Your store MAY require or accept extra settings for each metric on the call to `for_metric`.
+- You SHOULD validate these parameters to make sure they are correct, and raise if they aren't. 
+- If your store needs to aggregate multiple values for the same metric (for example, in
+  a multi-process scenario), you MUST accept a setting to define how values are aggregated.
+  - This setting MUST be called `:aggregation`
+  - It MUST support, at least, `:sum`, `:max` and `:min`. 
+  - It MAY support other aggregation modes that may apply to your requirements.
+  - It MUST default to `:sum`
+
+## Testing your Store
+
+In order to make it easier to test your store, the basic functionality is tested using
+`shared_examples`:
+
+`it_behaves_like Prometheus::Client::DataStores`
+
+Follow the simple structure in `synchronized_spec.rb` for a starting point.
+
+Note that if your store stores data somewhere other than in-memory (in files, Redis, 
+databases, etc), you will need to do cleanup between tests in a `before` block.
+
+The tests for `DirectFileStore` have a good example at the top of the file. This file also
+has some examples on testing multi-process stores, checking that aggregation between 
+processes works correctly.
+
+## Benchmarking your custom data store
+
+If you are developing your own data store, you probably want to benchmark it to see how
+it compares to the built-in ones, and to make sure it achieves the performance you want.
+
+The Prometheus Ruby Client includes some benchmarks (in the `spec/benchmarks` directory)
+to help you with this, and also with validating that your store works correctly.
+
+The `README` in that directory contains more information what these benchmarks are for,
+and how to use them.
+
+## Extra Stores and Research
+
+In the process of abstracting stores away, and creating the built-in ones, GoCardless
+has created a good amount of research, benchmarks, and experimental stores, which 
+weren't useful to include in this repo, but may be a useful resource or starting point 
+if you are building your own store.
+
+Check out the [GoCardless Data Stores Experiments](https://github.com/gocardless/prometheus-client-ruby-data-stores-experiments) 
+repository for these.
+
+## Sample, imaginary multi-process Data Store
+
+This is just an example of how one could implement a data store, and a clarification on
+the "aggregation" point 
+
+Important: This is a **toy example**, intended simply to show how this could work / how to
+implement these interfaces.
+
+There are some key pieces of code missing, which are fairly uninteresting, this only shows
+the parts that illustrate the idea of storing multiple different values, and aggregating
+them
+
+```ruby
+module Prometheus
+  module Client
+    module DataStores
+      # Stores all the data in a magic data structure that keeps cross-process data, in a
+      # way that all processes can read it, but each can write only to their own set of
+      # keys.
+      # It doesn't care how that works, this is not an actual solution to anything,
+      # just an example of how the interface would work with something like that.
+      #
+      # Metric Settings have one possible key, `aggregation`, which must be one of
+      # `AGGREGATION_MODES`
+      class SampleMagicMultiprocessStore
+        AGGREGATION_MODES = [MAX = :max, MIN = :min, SUM = :sum]
+        DEFAULT_METRIC_SETTINGS = { aggregation: SUM }
+
+        def initialize
+          @internal_store = MagicHashSharedBetweenProcesses.new # PStore, for example
+        end
+
+        def for_metric(metric_name, metric_type:, metric_settings: {})
+          settings = DEFAULT_METRIC_SETTINGS.merge(metric_settings)
+          validate_metric_settings(metric_settings: settings)
+          MetricStore.new(store: self,
+                          metric_name: metric_name,
+                          metric_type: metric_type,
+                          metric_settings: settings)
+        end
+
+        private
+
+        def validate_metric_settings(metric_settings:)
+          raise unless metric_settings.has_key?(:aggregation)
+          raise unless metric_settings[:aggregation].in?(AGGREGATION_MODES)
+        end
+
+        class MetricStore
+          def initialize(store:, metric_name:, metric_type:, metric_settings:)
+            @store = store
+            @internal_store = store.internal_store
+            @metric_name = metric_name
+            @aggregation_mode = metric_settings[:aggregation]
+          end
+
+          def set(labels:, val:)
+            @internal_store[store_key(labels)] = val.to_f
+          end
+
+          def get(labels:)
+            @internal_store[store_key(labels)]
+          end
+
+          def all_values
+            non_aggregated_values = all_store_values.each_with_object({}) do |(labels, v), acc|
+              if labels["__metric_name"] == @metric_name
+                label_set = labels.reject { |k,_| k.in?("__metric_name", "__pid") }
+                acc[label_set] ||= []
+                acc[label_set] << v
+              end
+            end
+
+            # Aggregate all the different values for each label_set
+            non_aggregated_values.each_with_object({}) do |(label_set, values), acc|
+              acc[label_set] = aggregate(values)
+            end
+          end
+
+          private
+
+          def all_store_values
+            # This assumes there's a something common that all processes can write to, and
+            # it's magically synchronized (which is not true of a PStore, for example, but
+            # would of some sort of external data store like Redis, Memcached, SQLite)
+
+            # This could also have some sort of:
+            #    file_list = Dir.glob(File.join(path, '*.db')).sort
+            # which reads all the PStore files / MMapped files, etc, and returns a hash
+            # with all of them together, which then `values` and `label_sets` can use
+          end
+
+          # This method holds most of the key to how this Store works. Adding `_pid` as
+          # one of the labels, we hold each process's value separately, which we can 
+          # aggregate later 
+          def store_key(labels)
+            labels.merge(
+              {
+                "__metric_name" => @metric_name,
+                "__pid" => Process.pid
+              }
+            )
+          end
+
+          def aggregate(values)
+            # This is a horrible way to do this, just illustrating the point
+            values.send(@aggregation_mode)
+          end
+        end
+      end
+    end
+  end
+end
+```

data/lib/prometheus/client/data_stores/direct_file_store.rb

@@ -0,0 +1,368 @@
+require 'fileutils'
+require "cgi"
+
+module Prometheus
+  module Client
+    module DataStores
+      # Stores data in binary files, one file per process and per metric.
+      # This is generally the recommended store to use to deal with pre-fork servers and
+      # other "multi-process" scenarios.
+      #
+      # Each process will get a file for a metric, and it will manage its contents by
+      # storing keys next to binary-encoded Floats, and keeping track of the offsets of
+      # those Floats, to be able to update them directly as they increase.
+      #
+      # When exporting metrics, the process that gets scraped by Prometheus  will find
+      # all the files that apply to a metric, read their contents, and aggregate them
+      # (generally that means SUMming the values for each labelset).
+      #
+      # In order to do this, each Metric needs an `:aggregation` setting, specifying how
+      # to aggregate the multiple possible values we can get for each labelset. By default,
+      # Counters, Histograms and Summaries get `SUM`med, and Gauges will report `ALL`
+      # values, tagging each one with a `pid` label.
+      # For Gauges, it's also possible to set `SUM`, MAX` or `MIN` as aggregation, to get
+      # the highest / lowest value / or the sum of all the processes / threads.
+      #
+      # Before using this Store, please read the "`DirectFileStore` caveats and things to
+      # keep in mind" section of the main README in this repository. It includes a number
+      # of important things to keep in mind.
+
+      class DirectFileStore
+        class InvalidStoreSettingsError < StandardError; end
+        AGGREGATION_MODES = [MAX = :max, MIN = :min, SUM = :sum, ALL = :all, MOST_RECENT = :most_recent]
+        DEFAULT_METRIC_SETTINGS = { aggregation: SUM }
+        DEFAULT_GAUGE_SETTINGS = { aggregation: ALL }
+
+        def initialize(dir:)
+          @store_settings = { dir: dir }
+          FileUtils.mkdir_p(dir)
+        end
+
+        def for_metric(metric_name, metric_type:, metric_settings: {})
+          default_settings = DEFAULT_METRIC_SETTINGS
+          if metric_type == :gauge
+            default_settings = DEFAULT_GAUGE_SETTINGS
+          end
+
+          settings = default_settings.merge(metric_settings)
+          validate_metric_settings(metric_type, settings)
+
+          MetricStore.new(metric_name: metric_name,
+                          store_settings: @store_settings,
+                          metric_settings: settings)
+        end
+
+        private
+
+        def validate_metric_settings(metric_type, metric_settings)
+          unless metric_settings.has_key?(:aggregation) &&
+            AGGREGATION_MODES.include?(metric_settings[:aggregation])
+            raise InvalidStoreSettingsError,
+                  "Metrics need a valid :aggregation key"
+          end
+
+          unless (metric_settings.keys - [:aggregation]).empty?
+            raise InvalidStoreSettingsError,
+                  "Only :aggregation setting can be specified"
+          end
+
+          if metric_settings[:aggregation] == MOST_RECENT && metric_type != :gauge
+            raise InvalidStoreSettingsError,
+                  "Only :gauge metrics support :most_recent aggregation"
+          end
+        end
+
+        class MetricStore
+          attr_reader :metric_name, :store_settings
+
+          def initialize(metric_name:, store_settings:, metric_settings:)
+            @metric_name = metric_name
+            @store_settings = store_settings
+            @values_aggregation_mode = metric_settings[:aggregation]
+            @store_opened_by_pid = nil
+
+            @lock = Monitor.new
+          end
+
+          # Synchronize is used to do a multi-process Mutex, when incrementing multiple
+          # values at once, so that the other process, reading the file for export, doesn't
+          # get incomplete increments.
+          #
+          # `in_process_sync`, instead, is just used so that two threads don't increment
+          # the same value and get a context switch between read and write leading to an
+          # inconsistency
+          def synchronize
+            in_process_sync do
+              internal_store.with_file_lock do
+                yield
+              end
+            end
+          end
+
+          def set(labels:, val:)
+            in_process_sync do
+              internal_store.write_value(store_key(labels), val.to_f)
+            end
+          end
+
+          def increment(labels:, by: 1)
+            if @values_aggregation_mode == DirectFileStore::MOST_RECENT
+              raise InvalidStoreSettingsError,
+                    "The :most_recent aggregation does not support the use of increment"\
+                      "/decrement"
+            end
+
+            key = store_key(labels)
+            in_process_sync do
+              internal_store.increment_value(key, by.to_f)
+            end
+          end
+
+          def get(labels:)
+            in_process_sync do
+              internal_store.read_value(store_key(labels))
+            end
+          end
+
+          def all_values
+            stores_data = Hash.new{ |hash, key| hash[key] = [] }
+
+            # There's no need to call `synchronize` here. We're opening a second handle to
+            # the file, and `flock`ing it, which prevents inconsistent reads
+            stores_for_metric.each do |file_path|
+              begin
+                store = FileMappedDict.new(file_path, true)
+                store.all_values.each do |(labelset_qs, v, ts)|
+                  # Labels come as a query string, and CGI::parse returns arrays for each key
+                  # "foo=bar&x=y" => { "foo" => ["bar"], "x" => ["y"] }
+                  # Turn the keys back into symbols, and remove the arrays
+                  label_set = CGI::parse(labelset_qs).map do |k, vs|
+                    [k.to_sym, vs.first]
+                  end.to_h
+
+                  stores_data[label_set] << [v, ts]
+                end
+              ensure
+                store.close if store
+              end
+            end
+
+            # Aggregate all the different values for each label_set
+            aggregate_hash = Hash.new { |hash, key| hash[key] = 0.0 }
+            stores_data.each_with_object(aggregate_hash) do |(label_set, values), acc|
+              acc[label_set] = aggregate_values(values)
+            end
+          end
+
+          private
+
+          def in_process_sync
+            @lock.synchronize { yield }
+          end
+
+          def store_key(labels)
+            if @values_aggregation_mode == ALL
+              labels[:pid] = process_id
+            end
+
+            labels.to_a.sort.map{|k,v| "#{CGI::escape(k.to_s)}=#{CGI::escape(v.to_s)}"}.join('&')
+          end
+
+          def internal_store
+            if @store_opened_by_pid != process_id
+              @store_opened_by_pid = process_id
+              @internal_store = FileMappedDict.new(filemap_filename)
+            else
+              @internal_store
+            end
+          end
+
+          # Filename for this metric's PStore (one per process)
+          def filemap_filename
+            filename = "metric_#{ metric_name }___#{ process_id }.bin"
+            File.join(@store_settings[:dir], filename)
+          end
+
+          def stores_for_metric
+            Dir.glob(File.join(@store_settings[:dir], "metric_#{ metric_name }___*"))
+          end
+
+          def process_id
+            Process.pid
+          end
+
+          def aggregate_values(values)
+            # Each entry in the `values` array is a tuple of `value` and `timestamp`,
+            # so for all aggregations except `MOST_RECENT`, we need to only take the
+            # first value in each entry and ignore the second.
+            if @values_aggregation_mode == MOST_RECENT
+              latest_tuple = values.max { |a,b| a[1] <=> b[1] }
+              latest_tuple.first # return the value without the timestamp
+            else
+              values = values.map(&:first) # Discard timestamps
+
+              if @values_aggregation_mode == SUM
+                values.inject { |sum, element| sum + element }
+              elsif @values_aggregation_mode == MAX
+                values.max
+              elsif @values_aggregation_mode == MIN
+                values.min
+              elsif @values_aggregation_mode == ALL
+                values.first
+              else
+                raise InvalidStoreSettingsError,
+                      "Invalid Aggregation Mode: #{ @values_aggregation_mode }"
+              end
+            end
+          end
+        end
+
+        private_constant :MetricStore
+
+        # A dict of doubles, backed by an file we access directly as a byte array.
+        #
+        # The file starts with a 4 byte int, indicating how much of it is used.
+        # Then 4 bytes of padding.
+        # There's then a number of entries, consisting of a 4 byte int which is the
+        # size of the next field, a utf-8 encoded string key, padding to an 8 byte
+        # alignment, and then a 8 byte float which is the value, and then a 8 byte
+        # float which is the unix timestamp when the value was set.
+        class FileMappedDict
+          INITIAL_FILE_SIZE = 1024*1024
+
+          attr_reader :capacity, :used, :positions
+
+          def initialize(filename, readonly = false)
+            @positions = {}
+            @used = 0
+
+            open_file(filename, readonly)
+            @used = @f.read(4).unpack('l')[0] if @capacity > 0
+
+            if @used > 0
+              # File already has data. Read the existing values
+              with_file_lock { populate_positions }
+            else
+              # File is empty. Init the `used` counter, if we're in write mode
+              if !readonly
+                @used = 8
+                @f.seek(0)
+                @f.write([@used].pack('l'))
+              end
+            end
+          end
+
+          # Return a list of key-value pairs
+          def all_values
+            with_file_lock do
+              @positions.map do |key, pos|
+                @f.seek(pos)
+                value, timestamp = @f.read(16).unpack('dd')
+                [key, value, timestamp]
+              end
+            end
+          end
+
+          def read_value(key)
+            if !@positions.has_key?(key)
+              init_value(key)
+            end
+
+            pos = @positions[key]
+            @f.seek(pos)
+            @f.read(8).unpack('d')[0]
+          end
+
+          def write_value(key, value)
+            if !@positions.has_key?(key)
+              init_value(key)
+            end
+
+            now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            pos = @positions[key]
+            @f.seek(pos)
+            @f.write([value, now].pack('dd'))
+            @f.flush
+          end
+
+          def increment_value(key, by)
+            if !@positions.has_key?(key)
+              init_value(key)
+            end
+
+            pos = @positions[key]
+            @f.seek(pos)
+            value = @f.read(8).unpack('d')[0]
+
+            now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            @f.seek(-8, :CUR)
+            @f.write([value + by, now].pack('dd'))
+            @f.flush
+          end
+
+          def close
+            @f.close
+          end
+
+          def with_file_lock
+            @f.flock(File::LOCK_EX)
+            yield
+          ensure
+            @f.flock(File::LOCK_UN)
+          end
+
+          private
+
+          def open_file(filename, readonly)
+            mode = if readonly
+                     "r"
+                   elsif File.exist?(filename)
+                     "r+b"
+                   else
+                     "w+b"
+                   end
+
+            @f = File.open(filename, mode)
+            if @f.size == 0 && !readonly
+              resize_file(INITIAL_FILE_SIZE)
+            end
+            @capacity = @f.size
+          end
+
+          def resize_file(new_capacity)
+            @f.truncate(new_capacity)
+          end
+
+          # Initialize a value. Lock must be held by caller.
+          def init_value(key)
+            # Pad to be 8-byte aligned.
+            padded = key + (' ' * (8 - (key.length + 4) % 8))
+            value = [padded.length, padded, 0.0, 0.0].pack("lA#{padded.length}dd")
+            while @used + value.length > @capacity
+              @capacity *= 2
+              resize_file(@capacity)
+            end
+            @f.seek(@used)
+            @f.write(value)
+            @used += value.length
+            @f.seek(0)
+            @f.write([@used].pack('l'))
+            @f.flush
+            @positions[key] = @used - 16
+          end
+
+          # Read position of all keys. No locking is performed.
+          def populate_positions
+            @f.seek(8)
+            while @f.pos < @used
+              padded_len = @f.read(4).unpack('l')[0]
+              key = @f.read(padded_len).unpack("A#{padded_len}")[0].strip
+              @positions[key] = @f.pos
+              @f.seek(16, :CUR)
+            end
+          end
+        end
+      end
+    end
+  end
+end