infrawrench-bulk_instances 1.0.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bccbd61e8ed4801a2ca745978807a8ec752b6b04cc8b5042cf92b01a7406ffeb
-  data.tar.gz: 529c9965e71e9a194204cfdbba3b78be46ba6779a225d4e580ab83621df2f3db
+  metadata.gz: 35025be8d677ed34cdfea296faca613c02d6ffe5c88d35266ad33fc36947d704
+  data.tar.gz: 3035dee6a0a3f8f86923ab7fb1b1c16c02fbc0e2aff9798a18d4640b4518138c
 SHA512:
-  metadata.gz: a26d838f6e2da662c1dd77bea3f021e509a679fb45b14dd86496d238f2dc71417f85a8c96651d90d2402e0be6244b8919b71e9da1171af308f883ea126f6c896
-  data.tar.gz: 4045a6ef4cb8451a56de51a4b6215eb0ed2c7f9418b9cfcc7f665fa1e4c221dc1a91b2b80020e79392166e3a8b75ddfc6a86f2723919bbb29e27485e7c271a8a
+  metadata.gz: fd9380c0871c15a34ebe92328f868d6347f6a3b6ce3722a137318d44c48b70cc22573fc38d696a9f66584ea604e97feacfeeecd3772bf87d6285ba93167d11fe
+  data.tar.gz: 33c0e9e0a4e9855840c34f2a93d4d4903a8f813d495daadf782e0057ba3b9df45f34c2f17699f155a7dd509ad62629cf603f046c07f9f9dea9c1510a684da852

data/README.md

@@ -1,7 +1,7 @@
 # infrawrench-bulk_instances
 
-AWS EC2 instance attributes and prices, with their full history, from the bundle
-Infrawrench publishes at `instances-bundles.infrawrench.com`.
+AWS instance attributes and prices (EC2, RDS, ElastiCache and more), with their full
+history, from the bundles Infrawrench publishes at `instances-bundles.infrawrench.com`.
 
 ```sh
 gem install infrawrench-bulk_instances
@@ -29,16 +29,69 @@ m5.pricing_history.reserved # every reserved price change, oldest first
 Any Infrawrench bearer credential works as `token:` (an `iwk_` API key, an agent
 credential or a WorkOS access token). It defaults to `INFRAWRENCH_API_KEY`.
 
+## Other AWS platforms
+
+`download_aws_pricing_data(platform, **options)` loads any platform's bundle;
+`download_aws_ec2_pricing_data(**options)` is the same as `download_aws_pricing_data("ec2", **options)`.
+`Infrawrench::BulkInstances::AWS_PLATFORMS` lists the platforms:
+
+| Platform | AWS service |
+| --- | --- |
+| `ec2` | EC2 |
+| `rds` | RDS |
+| `elasticache` | ElastiCache |
+| `opensearch` | OpenSearch Service |
+| `redshift` | Redshift |
+| `docdb` | DocumentDB |
+| `neptune` | Neptune |
+| `memorydb` | MemoryDB |
+| `dax` | DynamoDB Accelerator |
+| `sagemaker` | SageMaker |
+| `emr` | EMR |
+| `gamelift` | GameLift |
+
+The fields that pick a price differ by platform. `data.variant_keys` names them, and every
+price has a `variant` hash keyed by them:
+
+```ruby
+rds = Infrawrench::BulkInstances.download_aws_pricing_data("rds")
+rds.variant_keys # ["databaseEngine", "databaseEdition", "licenseModel", "deploymentOption", ...]
+rds["db.m5.large"].pricing.on_demand.find do |p|
+  p.region_code == "us-east-1" && p.variant["databaseEngine"] == "MySQL" &&
+    p.variant["deploymentOption"] == "Multi-AZ"
+end&.hourly
+```
+
+`operating_system`, `tenancy`, `license_model`, `pre_installed_sw` and `capacitystatus` are EC2's
+variant fields. On other platforms they are `""`, unless the platform has a field with that name
+(RDS has `licenseModel`).
+
 Requires Ruby 3.2 or later, and liblzma (part of macOS and of practically every
 Linux distribution; `xz-utils`/`xz` otherwise) to decompress downloads.
 
+## One instance
+
+`Infrawrench::BulkInstances.fetch_aws_instance(platform, instance_type, ...)` fetches a single
+instance from the Infrawrench API instead of downloading the platform's whole bundle. It returns
+the same `AWSInstance` that `get` does, or `nil` if the platform has no instance with that name:
+
+```ruby
+m5 = Infrawrench::BulkInstances.fetch_aws_instance("ec2", "m5.large")
+m5&.pricing&.on_demand&.length
+```
+
+It takes `token:` like the bundle download, and `base_url:` for where the API is served from
+(`https://app.infrawrench.com` by default). Nothing is cached. The API syncs each new bundle within
+about half an hour of it being published, so it can briefly trail `download_aws_pricing_data`.
+
 ## Caching
 
 The bundle is downloaded xz-compressed (about 20 MB) and kept decompressed (about
 210 MB) in `~/.infrawrench/aws_pricing_data` (`cache_path:`), so a cached load skips
 decompression. Each load asks the server which bundle is latest and downloads it
 only if it is not cached, deleting the older one. If the server cannot be
-reached, the newest cached bundle is used instead. Pass `cache_pricing_data: false`
+reached, the newest cached bundle is used instead. Each platform's bundle is cached
+under its own name (`rds-bundle-…json`), so loading one never evicts another. Pass `cache_pricing_data: false`
 to always download and never touch the disk. The cache is shared with the other
 Infrawrench SDKs.
 

data/lib/infrawrench/bulk_instances/aws_instance.rb

@@ -138,7 +138,7 @@ module Infrawrench
       private
 
       def od_key(r)
-        [r.region, r.operating_system, r.tenancy, r.license_model, r.pre_installed_sw, r.capacitystatus]
+        [r.region, *r.variant.values]
       end
     end
   end

data/lib/infrawrench/bulk_instances/aws_pricing_data.rb

@@ -2,7 +2,7 @@
 
 module Infrawrench
   module BulkInstances
-    # The AWS EC2 pricing data, keyed by instance type.
+    # One AWS platform's pricing data, keyed by instance type.
     #
     # Nothing is decoded until it is asked for: #get wraps the raw rows, and each
     # reader on the returned AWSInstance decodes its part on first access and keeps
@@ -15,10 +15,12 @@ module Infrawrench
       # index into +compression_table+:
       #
       #   {
+      #     "offer" => "AmazonRDS", # EC2 bundles leave it out
       #     "compression_table" => [String, ...],
       #     "snapshots" => [Integer, ...],
       #     "regions" => [[location, [[key, value], ...]], ...],
-      #     "variants" => [[os, tenancy, license, sw, capacity], ...],
+      #     "variant_keys" => [String, ...], # EC2 bundles leave it out (see EC2_VARIANT_KEYS)
+      #     "variants" => [[value, ...], ...], # one value per variant key
       #     "reserved_terms" => [[lease, offering, purchase], ...],
       #     "instances" => { "m5.large" => [attrs, attr_changes, base_price, on_demand, reserved, created_at, deleted_at] },
       #   }
@@ -50,6 +52,18 @@ module Infrawrench
       end
       alias [] get
 
+      # The AWS offer code, e.g. +AmazonEC2+ or +AmazonRDS+.
+      # @return [String]
+      def offer
+        @raw["offer"] || "AmazonEC2"
+      end
+
+      # The names of the fields that pick a price, which every price's +variant+ is keyed by.
+      # @return [Array<String>]
+      def variant_keys
+        @raw["variant_keys"] || EC2_VARIANT_KEYS
+      end
+
       # The publication date of every AWS price list in the bundle, oldest first.
       # @return [Array<Time>]
       def snapshots
@@ -82,13 +96,18 @@ module Infrawrench
       # @api private
       # @return [Variant]
       def variant(i)
-        @variants ||= @raw["variants"].map do |os, tenancy, license, sw, capacity|
+        @variants ||= @raw["variants"].map do |fields|
+          keys = variant_keys
+          raise "variant has #{fields.length} fields, expected #{keys.length}" if fields.length != keys.length
+
+          variant = keys.zip(fields).to_h { |k, v| [k, str(v)] }.freeze
           Variant.new(
-            operating_system: str(os),
-            tenancy: str(tenancy),
-            license_model: str(license),
-            pre_installed_sw: str(sw),
-            capacitystatus: str(capacity),
+            variant:,
+            operating_system: variant.fetch("operatingSystem", ""),
+            tenancy: variant.fetch("tenancy", ""),
+            license_model: variant.fetch("licenseModel", ""),
+            pre_installed_sw: variant.fetch("preInstalledSw", ""),
+            capacitystatus: variant.fetch("capacitystatus", ""),
           )
         end.freeze
         @variants[i]

data/lib/infrawrench/bulk_instances/types.rb

@@ -2,13 +2,17 @@
 
 module Infrawrench
   module BulkInstances
-    # Every price and price change starts with its location:
+    # Every price and price change starts with its location and variant:
     #
     # - +region+: the AWS location name, e.g. <tt>US East (N. Virginia)</tt>.
     # - +region_code+: e.g. +us-east-1+. +nil+ for locations the price list never gave a code.
+    # - +variant+: every field that picks the price, as a frozen <tt>Hash{String => String}</tt> keyed
+    #   by name in AWSPricingData#variant_keys order: for RDS, +databaseEngine+, +deploymentOption+ and so on.
+    # - +operating_system+, +tenancy+, +license_model+, +pre_installed_sw+, +capacitystatus+: EC2's
+    #   variant fields. For other platforms these are <tt>""</tt> unless the platform shares the field.
 
     # The product fields that pick a price.
-    Variant = Data.define(:operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus)
+    Variant = Data.define(:variant, :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus)
 
     ReservedTerm = Data.define(:lease_contract_length, :offering_class, :purchase_option)
 
@@ -18,7 +22,7 @@ module Infrawrench
     #   @return [Time] When this price took effect.
     OnDemandPrice = Data.define(
       :region, :region_code,
-      :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
+      :variant, :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
       :hourly, :since,
     )
 
@@ -30,7 +34,7 @@ module Infrawrench
     #   @return [Time]
     ReservedPrice = Data.define(
       :region, :region_code,
-      :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
+      :variant, :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
       :lease_contract_length, :offering_class, :purchase_option,
       :hourly, :upfront, :since,
     )
@@ -47,7 +51,7 @@ module Infrawrench
     #   @return [Float, nil] +nil+ means no longer offered from +at+ on.
     OnDemandPriceChange = Data.define(
       :at, :region, :region_code,
-      :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
+      :variant, :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
       :hourly,
     )
 
@@ -59,7 +63,7 @@ module Infrawrench
     #   @return [Float, nil]
     ReservedPriceChange = Data.define(
       :at, :region, :region_code,
-      :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
+      :variant, :operating_system, :tenancy, :license_model, :pre_installed_sw, :capacitystatus,
       :lease_contract_length, :offering_class, :purchase_option,
       :hourly, :upfront,
     )

data/lib/infrawrench/bulk_instances/version.rb

@@ -2,6 +2,6 @@
 
 module Infrawrench
   module BulkInstances
-    VERSION = "1.0.0"
+    VERSION = "1.2.0"
   end
 end

data/lib/infrawrench/bulk_instances.rb

@@ -14,9 +14,10 @@ require_relative "bulk_instances/aws_pricing_data"
 require_relative "bulk_instances/aws_instance"
 
 module Infrawrench
-  # AWS EC2 instance attributes and pricing, with full history, from Infrawrench.
+  # AWS instance attributes and pricing (EC2, RDS, ElastiCache and more), with full history, from Infrawrench.
   module BulkInstances
     DEFAULT_BASE_URL = "https://instances-bundles.infrawrench.com"
+    DEFAULT_API_URL = "https://app.infrawrench.com"
     USER_AGENT = "infrawrench/bulk-instances, ruby, 1.0"
 
     # What Net::HTTP raises when the server cannot be reached or the connection breaks.
@@ -42,12 +43,15 @@ module Infrawrench
     end
     private_constant :Util
 
+    # The AWS platforms Infrawrench publishes a bundle for. Any other lowercase name is
+    # requested as is, so a platform added later works without a new SDK.
+    AWS_PLATFORMS = %w[ec2 rds elasticache opensearch redshift docdb neptune memorydb dax sagemaker emr gamelift].freeze
+
+    # The variant keys of EC2 bundles, which don't list their own.
+    EC2_VARIANT_KEYS = %w[operatingSystem tenancy licenseModel preInstalledSw capacitystatus].freeze
+
     class << self
-      # Downloads the AWS EC2 pricing data compressed blob and decompresses it.
-      #
-      # The bucket's +latest+ object names the newest bundle. With caching on, a bundle
-      # is only downloaded when +latest+ names one that is not in the cache yet, and if
-      # +latest+ cannot be fetched at all (offline, outage) the newest cached bundle is used.
+      # Downloads the AWS EC2 pricing data. The same as <tt>download_aws_pricing_data("ec2", ...)</tt>.
       #
       # @param token [String, nil] Your Infrawrench API token. Defaults to the +INFRAWRENCH_API_KEY+ environment variable.
       # @param cache_pricing_data [Boolean] Whether to cache the pricing data.
@@ -56,6 +60,29 @@ module Infrawrench
       # @return [AWSPricingData]
       # @raise [AWSPricingDataError]
       def download_aws_ec2_pricing_data(token: nil, cache_pricing_data: true, cache_path: nil, base_url: nil)
+        download_aws_pricing_data("ec2", token:, cache_pricing_data:, cache_path:, base_url:)
+      end
+
+      # Downloads one AWS platform's pricing data compressed blob and decompresses it.
+      #
+      # The bucket's <tt><platform>-latest</tt> object (+latest+ for EC2) names the newest bundle. With
+      # caching on, a bundle is only downloaded when that names one that is not in the cache yet,
+      # and if it cannot be fetched at all (offline, outage) the newest cached bundle is used.
+      #
+      # @param platform [String, Symbol] One of AWS_PLATFORMS, e.g. +"rds"+, or any other lowercase platform name.
+      # @param token [String, nil] Your Infrawrench API token. Defaults to the +INFRAWRENCH_API_KEY+ environment variable.
+      # @param cache_pricing_data [Boolean] Whether to cache the pricing data.
+      # @param cache_path [String, nil] The cache path. Defaults to <tt>~/.infrawrench/aws_pricing_data</tt>.
+      # @param base_url [String, nil] Where the bundles are served from. Defaults to https://instances-bundles.infrawrench.com
+      # @return [AWSPricingData]
+      # @raise [AWSPricingDataError]
+      def download_aws_pricing_data(platform, token: nil, cache_pricing_data: true, cache_path: nil, base_url: nil)
+        unless (platform.is_a?(String) || platform.is_a?(Symbol)) && platform.match?(/\A[a-z0-9]+\z/)
+          raise AWSPricingDataError, %("#{platform}" is not a platform name.)
+        end
+
+        # EC2's keys have no prefix because they were published before the other platforms.
+        prefix = platform.to_s == "ec2" ? "" : "#{platform}-"
         token = ENV.fetch("INFRAWRENCH_API_KEY", nil) if token.nil?
         if token.nil? || token.empty?
           raise AWSPricingDataError, "No Infrawrench API token: pass `token:` or set INFRAWRENCH_API_KEY."
@@ -65,15 +92,15 @@ module Infrawrench
         cache_path = (cache_path || File.join(Dir.home, ".infrawrench", "aws_pricing_data")).to_s
 
         begin
-          latest = download("#{base_url}/latest", token, &:read_body).to_s.dup.force_encoding(Encoding::UTF_8).scrub.strip
+          latest = download("#{base_url}/#{prefix}latest", token, &:read_body).to_s.dup.force_encoding(Encoding::UTF_8).scrub.strip
           unless latest.match?(/\A[A-Za-z0-9._-]+\z/)
-            raise AWSPricingDataError, %(latest holds "#{latest}", which is not a timestamp.)
+            raise AWSPricingDataError, %(#{prefix}latest holds "#{latest}", which is not a timestamp.)
           end
 
-          name = "bundle-#{latest}.json"
+          name = "#{prefix}bundle-#{latest}.json"
         rescue StandardError => e
           # A rejected token is a real answer; anything else falls back to what we already have.
-          cached = newest_cached(cache_path) if cache_pricing_data && !(e.is_a?(AWSPricingDataError) && e.status == 401)
+          cached = newest_cached(cache_path, prefix) if cache_pricing_data && !(e.is_a?(AWSPricingDataError) && e.status == 401)
           raise if cached.nil?
 
           name = cached
@@ -92,9 +119,9 @@ module Infrawrench
             tmp = File.join(cache_path, "#{name}.#{Process.pid}.tmp")
             File.binwrite(tmp, json)
             File.rename(tmp, File.join(cache_path, name))
-            # Older bundles, including compressed ones cached by earlier versions of this SDK.
+            # This platform's older bundles, including compressed ones cached by earlier versions of this SDK.
             Dir.each_child(cache_path) do |f|
-              FileUtils.rm_f(File.join(cache_path, f)) if f != name && f.match?(/\Abundle-.+\.json(\.xz)?\z/)
+              FileUtils.rm_f(File.join(cache_path, f)) if f != name && f.match?(/\A#{prefix}bundle-.+\.json(\.xz)?\z/)
             end
           end
         end
@@ -102,10 +129,53 @@ module Infrawrench
         AWSPricingData.new(JSON.parse(json, freeze: true))
       end
 
+      # Fetches one instance from the Infrawrench API, without downloading the platform's bundle.
+      # Nothing is cached.
+      #
+      # The result is the same as <tt>download_aws_pricing_data(platform).get(instance_type)</tt>, from
+      # whichever bundle the API last synced, which can trail the newest bundle by up to half an hour.
+      #
+      # @param platform [String, Symbol] One of AWS_PLATFORMS, e.g. +"rds"+, or any other lowercase platform name.
+      # @param instance_type [String] The instance type, e.g. +"m5.large"+.
+      # @param token [String, nil] Your Infrawrench API token. Defaults to the +INFRAWRENCH_API_KEY+ environment variable.
+      # @param base_url [String, nil] Where the Infrawrench API is served from. Defaults to https://app.infrawrench.com
+      # @return [AWSInstance, nil] +nil+ if the platform has no instance with that name.
+      # @raise [AWSPricingDataError]
+      def fetch_aws_instance(platform, instance_type, token: nil, base_url: nil)
+        unless (platform.is_a?(String) || platform.is_a?(Symbol)) && platform.match?(/\A[a-z0-9]+\z/)
+          raise AWSPricingDataError, %("#{platform}" is not a platform name.)
+        end
+        raise AWSPricingDataError, "The instance type is empty." if instance_type.to_s.empty?
+
+        token = ENV.fetch("INFRAWRENCH_API_KEY", nil) if token.nil?
+        if token.nil? || token.empty?
+          raise AWSPricingDataError, "No Infrawrench API token: pass `token:` or set INFRAWRENCH_API_KEY."
+        end
+
+        base_url = (base_url || DEFAULT_API_URL).sub(%r{/+\z}, "")
+        # Percent-encoded, with spaces as %20: a + in a path is a plus.
+        name = URI.encode_www_form_component(instance_type.to_s).gsub("+", "%20")
+        begin
+          json = download("#{base_url}/api/v1/aws-bulk-pricing/#{platform}/#{name}", token, &:read_body)
+        rescue AWSPricingDataError => e
+          return nil if e.status == 404
+
+          raise
+        end
+
+        # The bundle's tables cut down to what this instance uses, and its row renumbered to match.
+        value = JSON.parse(json.to_s.dup.force_encoding(Encoding::UTF_8), freeze: true)
+        bundle = value.slice("offer", "compression_table", "regions", "variant_keys", "variants", "reserved_terms")
+        bundle["snapshots"] = []
+        bundle["instances"] = { value["name"] => value["instance"] }
+        AWSPricingData.new(bundle).get(value["name"])
+      end
+
       private
 
       # GETs +url+ and yields the successful response, retrying the 503 the bundles Worker
-      # answers when it cannot reach the Infrawrench API.
+      # answers when it cannot reach the Infrawrench API (and the API answers when it cannot
+      # reach its store).
       def download(url, token)
         uri = URI(url)
         (1..).each do |attempt|
@@ -135,9 +205,9 @@ module Infrawrench
         raise AWSPricingDataError, "GET #{url} failed: #{e.message}"
       end
 
-      # The newest bundle already in the cache directory, if any.
-      def newest_cached(cache_path)
-        Dir.children(cache_path).grep(/\Abundle-.+\.json\z/).max
+      # The newest bundle for a platform already in the cache directory, if any.
+      def newest_cached(cache_path, prefix)
+        Dir.children(cache_path).grep(/\A#{prefix}bundle-.+\.json\z/).max
       rescue SystemCallError
         nil
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: infrawrench-bulk_instances
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Infrawrench LLC
@@ -61,5 +61,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 3.6.9
 specification_version: 4
-summary: AWS EC2 instance attributes and pricing, with full history, from Infrawrench.
+summary: AWS instance attributes and pricing (EC2, RDS, ElastiCache and more), with
+  full history, from Infrawrench.
 test_files: []