newrelic-telemetry_sdk 0.1.0

data/README.md

@@ -0,0 +1,95 @@
+[![Community Project header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Community_Project.png)](https://opensource.newrelic.com/oss-category/#community-project)
+
+# New Relic Ruby Telemetry SDK
+The New Relic Ruby Telemetry SDK is an easy way to send data to New Relic. The SDK currently supports sending span/trace data via New Relic's [Trace API](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/trace-api/introduction-trace-api).
+
+Why is this cool?
+
+Send data to New Relic! No agent required.
+
+Our [Telemetry SDK](https://docs.newrelic.com/docs/data-ingest-apis/get-data-new-relic/new-relic-sdks/telemetry-sdks-send-custom-telemetry-data-new-relic) makes it easier for you to send your telemetry data to New Relic. We've covered all of the basics for you so you can focus on writing feature code directly related to your business need or interest.
+
+## Installation
+
+### With Bundler
+
+For using with Bundler, add the New Relic Telemetry SDK gem to your project's Gemfile.
+
+```ruby
+gem 'newrelic-telemetry_sdk'
+```
+
+and run `bundle install` to activate the new gem.
+
+### Without Bundler
+
+If you are not using Bundler, install the gem with:
+
+```bash
+gem install newrelic-telemetry_sdk
+```
+
+and then require the New Relic Ruby Telemetry SDK in your Ruby start-up sequence:
+
+```ruby
+require 'newrelic-telemetry_sdk'
+```
+
+## Getting Started
+
+### Sending your first span
+
+The example code assumes you've set the following environment variables:
+
+* NEW_RELIC_INSERT_KEY
+
+```
+NewRelic::TelemetrySdk.configure do |config|
+   config.api_insert_key = ENV["NEW_RELIC_INSERT_KEY"]
+end
+
+span = NewRelic::TelemetrySdk::Span.new
+sleep 1
+span.finish
+client = NewRelic::TelemetrySdk::TraceClient.new
+client.report span
+```
+
+For more detailed examples please see [our examples directory](./examples).
+
+## Testing
+
+Running the test suite is simple.  Just invoke:
+
+    bundle
+    bundle exec rake
+
+## Support
+
+New Relic hosts and moderates an online forum where customers can interact with New Relic employees as well as other customers to get help and share best practices. Like all official New Relic open source projects, there's a related Community topic in the New Relic Explorers Hub. You can find this project's topic/threads here:
+
+https://discuss.newrelic.com/t/new-relic-telemetry-sdk-for-ruby/114266
+
+## Contributing
+We encourage your contributions to improve newrelic-telemetry-sdk-ruby! Keep in mind when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.
+If you have any questions, or to execute our corporate CLA, required if your contribution is on behalf of a company,  please drop us an email at opensource@newrelic.com.
+
+## License
+newrelic-telemetry-sdk-ruby is licensed under the [Apache 2.0](http://apache.org/licenses/LICENSE-2.0.txt) License.
+
+## Find and use data
+
+Tips on how to find and query your data in New Relic:
+
+* [Find metric data](https://docs.newrelic.com/docs/data-ingest-apis/get-data-new-relic/metric-api/introduction-metric-api#find-data)
+* [Find event data](https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/introduction-event-api#find-data)
+* [Find trace/span data](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/trace-api/introduction-trace-api#view-data>)
+
+For general querying information, see:
+
+* [Query New Relic data](https://docs.newrelic.com/docs/using-new-relic/data/understand-data/query-new-relic-data)
+* [Intro to NRQL](https://docs.newrelic.com/docs/query-data/nrql-new-relic-query-language/getting-started/introduction-nrql)
+
+## Limitations
+
+The New Relic Telemetry APIs are rate limited. Please reference the documentation for [New Relic Metrics API](https://docs.newrelic.com/docs/introduction-new-relic-metric-api) and [New Relic Trace API requirements and limits](https://docs.newrelic.com/docs/apm/distributed-tracing/trace-api/trace-api-general-requirements-limits) on the specifics of the rate limits.

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task :default => :test

data/cla.md

@@ -0,0 +1,16 @@
+# NEW RELIC, INC.
+## INDIVIDUAL CONTRIBUTOR LICENSE AGREEMENT
+Thank you for your interest in contributing to the open source projects of New Relic, Inc. (“New Relic”). In order to clarify the intellectual property license granted with Contributions from any person or entity, New Relic must have a Contributor License Agreement ("Agreement") on file that has been signed by each Contributor, indicating agreement to the license terms below. This Agreement is for your protection as a Contributor as well as the protection of New Relic; it does not change your rights to use your own Contributions for any other purpose.
+
+You accept and agree to the following terms and conditions for Your present and future Contributions submitted to New Relic. Except for the licenses granted herein to New Relic and recipients of software distributed by New Relic, You reserve all right, title, and interest in and to Your Contributions.
+
+## Definitions.
+1. "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is entering into this Agreement with New Relic. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+2. "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to New Relic for inclusion in, or documentation of, any of the products managed or maintained by New Relic (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to New Relic or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, New Relic for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
+3. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to New Relic and to recipients of software distributed by New Relic a perpetual, worldwide, non-exclusive, no-charge, royalty-free, transferable, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works.
+4. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to New Relic and to recipients of software distributed by New Relic a perpetual, worldwide, non-exclusive, no-charge, royalty-free, transferable, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contributions alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that Your Contribution, or the Work to which You have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
+5. You represent that You are legally entitled to grant the above licenses. If Your employer(s) has rights to intellectual property that You create that includes Your Contributions, You represent that You have received permission to make Contributions on behalf of that employer, that Your employer has waived such rights for Your Contributions to New Relic, or that Your employer has executed a separate Agreement with New Relic.
+6. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which You are personally aware and which are associated with any part of Your Contributions.
+7. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
+8. Should You wish to submit work that is not Your original creation, You may submit it to New Relic separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which You are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
+9. You agree to notify New Relic of any facts or circumstances of which You become aware that would make these representations inaccurate in any respect.

data/examples/external_request/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gem 'newrelic-telemetry_sdk', path: "../../"

data/examples/external_request/simple.rb

@@ -0,0 +1,80 @@
+# This example is one of the simplest examples demonstrating sending one span 
+# at at time to the Trace API endpoint.  It will time how long it takes to retrive
+# an external URL's content and construct a span and send it to New Relic's server.
+# 
+# Usage: `API_KEY=<YOUR_API_KEY> bundle exec ruby simple.rb`
+require 'net/http'
+require 'bundler'
+Bundler.require
+
+# Fail fast if an API_KEY wasn't exported to the environment.
+unless ENV["API_KEY"]
+  raise "No API Key supplied.  Export API_KEY environment variable!" 
+end
+
+# Just a random ID generator -- A string of 16 (by default) hexadecimal digits.
+def random_id length=16
+  length.times.map{rand(16).to_s(16)}.join
+end
+
+def configure_sdk
+  NewRelic::TelemetrySdk.configure do |config|
+    config.api_insert_key = ENV["API_KEY"]
+  end
+end
+
+# Instantiates and memoizes the SDK client for sending Spans
+def trace_client
+  return @trace_client if @trace_client
+  configure_sdk
+  @trace_client = NewRelic::TelemetrySdk::TraceClient.new
+end
+
+# Wraps the given block by recording time to retrieve URL's resource, then
+# constructs the span object and sends it to the server.
+# The response from the given block is returned as the method's result
+def record_external_request
+  # Capture time elapsed to make the external request (given block)
+  start_time = Time.now
+  response = yield
+  finish_time = Time.now
+  duration = finish_time - start_time
+
+  # Set some custom attributes for the Span (some static, some dynamic)
+  custom_attributes = {
+    "name": "Net::HTTP#get",
+    "http.method": "GET",
+    "http.status": response.code,
+    "size": response.body.bytesize,
+  }
+
+  # Construct the span and send it.
+  span = NewRelic::TelemetrySdk::Span.new(
+    id: random_id(16),
+    trace_id: random_id(32),
+    start_time: start_time,
+    duration_ms: (duration * 1000).to_i,
+    name: "Net::HTTP#get",
+    custom_attributes: custom_attributes
+  )
+  trace_client.report span
+
+  # return the results from the given block/proc
+  response
+end
+
+# Retrieve the resource from given URL, recording a span
+# for the operation.
+def get_page url
+  uri = URI::parse(url)
+  response = record_external_request do
+    conn = Net::HTTP.new uri.host, uri.port
+    conn.use_ssl = true
+    conn.request_get("/")
+  end
+  puts "Fetched: #{uri}"
+  puts "Status: #{response.code}"
+  puts "Size: #{response.body.bytesize}"
+end
+
+get_page "https://google.com"

data/examples/external_request/with_harvester.rb

@@ -0,0 +1,89 @@
+require 'net/http'
+require 'bundler'
+Bundler.require
+
+unless ENV["API_KEY"]
+  raise "No API Key supplied.  Export API_KEY environment variable!" 
+end
+
+def configure_sdk
+  NewRelic::TelemetrySdk.configure do |config|
+    config.api_insert_key = ENV["API_KEY"]
+  end
+end
+
+def setup_buffer_harvesting common_attributes = {host: 'fake_host'}
+  configure_sdk
+  @harvester = NewRelic::TelemetrySdk::Harvester.new 
+  @trace_client = NewRelic::TelemetrySdk::TraceClient.new
+  # Creates a buffer with common attributes that will be added to all spans in the buffer
+  @buffer = NewRelic::TelemetrySdk::Buffer.new common_attributes
+  # Register the buffer with a name and the associated client
+  @harvester.register 'external_spans', @buffer, @trace_client
+  # Begins the harvester running in the background
+  @harvester.start
+end
+
+def stop_harvester
+  @harvester.stop
+end
+
+def random_id length=16
+  length.times.map{rand(16).to_s(16)}.join
+end
+
+def record_external_request
+  start_time = Time.now
+  response = yield
+  finish_time = Time.now
+  duration = finish_time - start_time
+
+  custom_attributes = {
+    "http.status": response.code,
+    "size": response.body.bytesize,
+  }
+
+  span = NewRelic::TelemetrySdk::Span.new(
+    id: random_id(16),
+    trace_id: random_id(32),
+    start_time: start_time,
+    duration_ms: (duration * 1000).to_i,
+    name: "Net::HTTP#get",
+    custom_attributes: custom_attributes
+  )
+  @buffer.record span
+
+  response
+end
+
+def get_page url
+  uri = URI::parse(url)
+  response = record_external_request do
+    conn = Net::HTTP.new uri.host, uri.port
+    conn.use_ssl = true
+    conn.request_get("/")
+  end
+  puts "Fetched: #{uri}"
+  puts "Status: #{response.code}"
+  puts "Size: #{response.body.bytesize}"
+end
+
+# Creates the buffer, client, and harvester
+common_attributes = {"name": "Net::HTTP#get", "http.method": "GET"}
+setup_buffer_harvesting common_attributes
+
+# Buffer holds multiple spans, spans are added to the buffer in the record_external_request_method
+get_page "https://google.com"
+get_page "https://google.com"
+get_page "https://google.com"
+
+# allow harvester time to send data, default interval is 5 seconds
+sleep 6 
+# buffer is now empty
+
+get_page "https://google.com"
+get_page "https://google.com"
+get_page "https://google.com"
+
+# harvester will flush remaining data in buffer when stopped
+stop_harvester

data/lib/newrelic/telemetry_sdk.rb

@@ -0,0 +1,38 @@
+# encoding: utf-8
+# frozen_string_literal: true
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-telemetry-sdk-ruby/blob/main/LICENSE for complete details.
+
+require 'net/http'
+require 'logger'
+require 'json'
+require 'zlib'
+require 'securerandom'
+require 'logger'
+
+module NewRelic
+  module TelemetrySdk
+    private
+
+    USER_AGENT_NAME = "NewRelic-Ruby-TelemetrySDK"
+    RFC7230_TOKEN = /^[!#$%&'*+\-.^_`|~0-9A-Za-z]+$/
+
+    NEW_RELIC_PREFIX = "NEW_RELIC"
+    API_INSERT_KEY = "#{NEW_RELIC_PREFIX}_API_INSERT_KEY"
+  end
+end
+
+require 'newrelic/telemetry_sdk/version'
+require 'newrelic/telemetry_sdk/util'
+require 'newrelic/telemetry_sdk/exception'
+
+require 'newrelic/telemetry_sdk/config'
+require 'newrelic/telemetry_sdk/configurator'
+require 'newrelic/telemetry_sdk/logger'
+
+require 'newrelic/telemetry_sdk/span'
+require 'newrelic/telemetry_sdk/harvester'
+require 'newrelic/telemetry_sdk/buffer'
+
+require 'newrelic/telemetry_sdk/clients/client'
+require 'newrelic/telemetry_sdk/clients/trace_client'

data/lib/newrelic/telemetry_sdk/buffer.rb

@@ -0,0 +1,65 @@
+# encoding: utf-8
+# frozen_string_literal: true
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-telemetry-sdk-ruby/blob/main/LICENSE for complete details.
+
+module NewRelic
+  module TelemetrySdk
+    # Buffers store discrete pieces of data (e.g. {Span Spans}) until they are
+    # sent via a timed {Harvester}. Batches of data may also be flushed
+    # from a buffer and sent directly through the client.
+    #
+    # @api public
+    class Buffer
+
+      include NewRelic::TelemetrySdk::Logger
+
+      # @!attribute common_attributes [optional, Hash]
+      # Attributes that should be added to every item in the batch
+      # e.g. +{host: 'my_host'}+
+      attr_accessor :common_attributes
+
+      # Record a discrete piece of data (e.g. a Span) into the buffer
+      # for batching purposes.
+      # @param common_attributes [optional, Hash]
+      #     Attributes that should be added to every item in the batch
+      #     e.g. +{host: 'my_host'}+
+      #
+      # @api public
+      def initialize common_attributes=nil
+        @items = []
+        @common_attributes = common_attributes
+        @lock = Mutex.new
+      end
+
+      # Record a discrete piece of data (e.g. a {Span}) into the {Buffer buffer}.
+      # @param item [Span, etc.]
+      #     A piece of data to record into the buffer. Must have a to_h method
+      #     for transformation.
+      #
+      # @api public
+      def record item
+        @lock.synchronize { @items << item }
+      rescue => e
+        log_error "Encountered error while recording in buffer", e
+      end
+
+      # Return a batch of data that has been collected in this buffer
+      # as an Array of Hashes. Also returns a Hash of any common attributes
+      # that have been set on the buffer to be attached to each individual data item.
+      #
+      # @api public
+      def flush
+        data = nil
+        @lock.synchronize do
+          data = @items
+          @items = []
+        end
+        data.map!(&:to_h)
+        return data, @common_attributes
+      rescue => e
+        log_error "Encountered error while flushing buffer", e
+      end
+    end
+  end
+end

data/lib/newrelic/telemetry_sdk/clients/client.rb

@@ -0,0 +1,262 @@
+# encoding: utf-8
+# frozen_string_literal: true
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-telemetry-sdk-ruby/blob/main/LICENSE for complete details.
+
+module NewRelic
+  module TelemetrySdk
+    # This class is a parent class for clients used to send data to the New Relic data
+    # ingest endpoints over HTTP (e.g. TraceClient for span data). Clients will automatically
+    # resend data if a recoverable error occurs. They will also automatically handle
+    # connection issues and New Relic errors.
+    #
+    # @api public
+    class Client
+      include NewRelic::TelemetrySdk::Logger
+
+      def initialize host:,
+                     path:,
+                     headers: {},
+                     use_gzip: true,
+                     payload_type:
+        @connection = set_up_connection host
+        @path = path
+        @headers = headers
+        @gzip_request = use_gzip
+        @payload_type = payload_type
+        @user_agent_products = nil
+        add_user_agent_header @headers
+        add_content_encoding_header @headers if @gzip_request
+        @connection_attempts = 0
+      end
+
+      # Reports a single item to a New Relic data ingest endpoint.
+      #
+      # @param item
+      #     a single point of data to send to New Relic (e.g. a Span). The item
+      #     should respond to the +#to_h+ method to return a Hash which is then serialized
+      #     and sent to the data ingest endpoint.
+      #
+      # @api public
+      def report item
+        # Report a batch of one pre-transformed item with no common attributes
+        report_batch [[item.to_h], nil]
+      rescue => e
+        log_error "Encountered error reporting item in client. Dropping data: 1 point of data", e
+      end
+
+      # Reports a batch of one or more items to a New Relic data ingest endpoint.
+      #
+      # @param batch_data [Array]
+      #     a two-part array contianing a Array of Hashes paired with a Hash of
+      #     common attributes.
+      #
+      # @api public
+      def report_batch batch_data
+        # We need to generate a version 4 uuid that will
+        # be used for each unique batch, including on retries.
+        # If a batch is split due to a 413 response,
+        # each smaller batch should have its own.
+
+        data, common_attributes = batch_data
+
+        @headers[:'x-request-id'] = SecureRandom.uuid
+
+        post_body = format_payload data, common_attributes
+        send_with_response_handling post_body, data, common_attributes
+      rescue => e
+        log_error "Encountered error reporting batch in client. Dropping data: #{data.size} points of data", e
+      end
+
+      # Allows creators of exporters and other product built on this SDK to provide information about
+      # their product for analytic purposes.  It may be called multiple times and is idempotent.
+      #
+      # @param product [String]
+      #     The name of the exporter or other product, e.g. NewRelic-Ruby-OpenTelemetry.
+      # @param version [optional, String]
+      #     The version number of the exporter or other product.
+      #
+      # Both product and version must conform to RFC 7230.
+      # @see https://github.com/newrelic/newrelic-telemetry-sdk-specs/blob/master/communication.md#extending-user-agent-with-exporter-product Communication with New Relic
+      #
+      # @api public
+      def add_user_agent_product product, version=nil
+        # The product token must be valid to add to the headers
+        if product !~ RFC7230_TOKEN
+          log_once :warn, "Product is not a valid RFC 7230 token"
+          return
+        end
+
+        # The version is ignored if invalid
+        if version && version !~ RFC7230_TOKEN
+          log_once :warn, "Product version is not a valid RFC 7230 token"
+          version = nil
+        end
+
+        entry = [product, version].compact.join("/")
+
+        # adds the product entry and updates the combined user agent
+        # header, ignoring duplicate product entries.
+        @user_agent_products ||= []
+        unless @user_agent_products.include? entry
+          @user_agent_products << entry
+          add_user_agent_header @headers
+        end
+      rescue => e
+        log_error "Encountered error adding user agent product", e
+      end
+
+    private
+
+      def api_insert_key
+        TelemetrySdk.config.api_insert_key
+      end
+
+      def max_retries
+        TelemetrySdk.config.max_retries
+      end
+
+      def backoff_factor
+        TelemetrySdk.config.backoff_factor
+      end
+
+      def backoff_max
+        TelemetrySdk.config.backoff_max
+      end
+
+      def send_with_response_handling post_body, data, common_attributes
+        response = send_request post_body
+
+        case response
+        when Net::HTTPSuccess # 200 - 299
+          @connection_attempts = 0 # reset count after sucessful connection
+          logger.debug "Successfully sent data to New Relic with response: #{response.code}"
+
+        when Net::HTTPBadRequest, # 400
+            Net::HTTPUnauthorized, # 401
+            Net::HTTPForbidden, # 403
+            Net::HTTPNotFound, # 404
+            Net::HTTPMethodNotAllowed, # 405
+            Net::HTTPConflict, # 409
+            Net::HTTPGone, # 410
+            Net::HTTPLengthRequired  # 411
+          log_once_and_drop_data response, data
+
+        when Net::HTTPRequestTimeOut # 408
+          log_and_retry response
+
+        when Net::HTTPRequestEntityTooLarge # 413
+          log_and_split_payload response, data, common_attributes
+
+        when Net::HTTPTooManyRequests # 429
+          log_and_retry_later response
+
+        else
+          log_and_retry_with_backoff response, data
+        end
+
+      rescue NewRelic::TelemetrySdk::RetriableServerResponseException
+        retry
+      end
+
+      def audit_logging_enabled?
+        TelemetrySdk.config.audit_logging_enabled
+      end
+
+      def send_request body
+        body = serialize body
+        log_json_payload body if audit_logging_enabled?
+        body = gzip_data body if @gzip_request
+        @connection.post @path, body, @headers
+      end
+
+      def log_json_payload payload
+        logger.debug "Sent payload: #{payload}"
+      end
+
+      def log_and_retry response
+        log_error response.message
+        raise NewRelic::TelemetrySdk::RetriableServerResponseException
+      end
+
+      def log_and_retry_later response
+        wait_time = response['Retry-After'].to_i
+        log_error "Connection error. Retrying in #{wait_time} seconds", response.message
+        sleep wait_time
+        raise NewRelic::TelemetrySdk::RetriableServerResponseException
+      end
+
+      def log_once_and_drop_data response, data
+        log_error_once response.class, response.message
+        log_error "Connection error. Dropping data: #{data.size} points of data"
+      end
+
+      def log_and_split_payload response, data, common_attributes
+        log_error "Payload too large. Splitting payload in half and attempting to resend.", response.message
+        if data.size > 1
+          # splits the data in half and calls report_batch for each half
+          midpoint = data.size/2.0
+          report_batch [data.first(midpoint.ceil), common_attributes]
+          report_batch [data.last(midpoint.floor), common_attributes]
+        else
+          # payload cannot be split, drop data
+          log_error "Unable to split payload. Dropping data: #{data.size} points of data"
+        end
+      end
+
+      def log_and_retry_with_backoff response, data
+        if @connection_attempts < max_retries
+          wait = backoff_strategy
+          log_error "Connection error. Retrying in #{wait} seconds.", response.message
+          sleep wait
+          raise NewRelic::TelemetrySdk::RetriableServerResponseException
+        else
+          log_error "Maximum retries reached. Dropping data: #{data.size} points of data"
+        end
+      end
+
+      def calculate_backoff_strategy
+        [backoff_max, (backoff_factor * (2**(@connection_attempts-1)).to_i)].min
+      end
+
+      def backoff_strategy
+        calculate_backoff_strategy.tap { @connection_attempts += 1 }
+      end
+
+      def format_payload data, common_attributes
+        post_body = { @payload_type => data }
+
+        if common_attributes
+          post_body[:common] = {}
+          post_body[:common][:attributes] = common_attributes
+        end
+
+        [post_body]
+      end
+
+      def add_user_agent_header headers
+        sdk_id = "#{USER_AGENT_NAME}/#{VERSION}"
+        headers[:'User-Agent'] = ([sdk_id] + Array(@user_agent_products)).join(" ")
+      end
+
+      def add_content_encoding_header headers
+        headers.merge!(:'content-encoding' => 'gzip')
+      end
+
+      def set_up_connection host
+        uri = URI(host)
+        conn = Net::HTTP.new uri.host, uri.port
+        conn.use_ssl = true
+        conn
+      end
+
+      def serialize data
+        JSON.generate data
+      end
+
+      def gzip_data data
+        Zlib.gzip data
+      end
+    end
+  end
+end