contentful_redis 0.0.1

data/README.md

@@ -0,0 +1,291 @@
+# ContentfulRedis
+A lightweight read-only contentful API wrapper which caches your responses in Redis.
+
+# Features
+- Lightweight easy to configure ruby contentful integration.
+- Faster load times due to having a Redis cache.
+- All content models responses are cached.
+- Webhooks update
+- Multiple space support
+- Preview and production API support on a single environment
+
+## WIP
+- Migrate tests
+- logger
+- Experiment Redis size optimization
+- auto clean up of dead Redis keys
+- code clean up
+
+ContentfulRedis also supports multiple API endpoints(preview and published) within a single application.
+
+## Installation
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'redis-store'
+gem 'contentful_redis'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install contentful_redis
+
+## Configuration
+
+Heres a default example, however, I will go over all of the individual configurations options below
+
+```ruby
+# config/initializers/contentful_redis.rb
+ContentfulRedis.configure do |config|
+  config.default_env = :preview # unless production env
+  config.model_scope = 'Contentful' # models live in a Contentful module
+  config.logging = true
+
+  config.spaces = { 
+    test_space: {
+      id: 'xxxx',
+      access_token: 'xxxx',
+      preview_access_token: 'xxxx'
+    }
+  }
+
+  config.redis = Redis::Store.new(
+    host: (ENV['REDIS_HOST']) || 'localhost',
+    port: 6379,
+    db: 1,
+    namespace: 'contentful'
+  )
+
+```
+
+### Spaces (required)
+Contentful Redis supports multiple space configurations with your first space being the default
+
+```ruby
+# config/initializers/contentful_redis.rb
+ContentfulRedis.configure do |config|
+  config.spaces = { 
+    test_space: {
+      id: 'xxxx',
+      access_token: 'xxxx',
+      preview_access_token: 'xxxx'
+    },
+
+    test_space_2: {
+      id: 'xxxy',
+      access_token: 'xxxx',
+      preview_access_token: 'xxxx'
+    }
+  }
+```
+
+To use a different space for a model override the classes `#space` method
+
+```ruby
+# app/models/my_model.rb
+class MyModel < ContentfulRedis::ModelBase
+  
+  # override default space
+  def self.space
+    ContentfulRedis.configuration.spaces[:test_space_2]
+  end
+end
+```
+
+### Redis (required)
+There are various ways you can integrate with Redis.
+I suggest using [redis-store](https://github.com/redis-store/redis-store) unless your application already has Redis adapter installed.
+I recommend having a separate Redis database for all of your contentful data so that you can isolate your application Redis from your content.
+
+```ruby
+# config/initializers/contentful_redis.rb
+ContentfulRedis.configure do |config|
+  config.redis = Redis::Store.new(
+    host: (ENV['REDIS_HOST']) || 'localhost',
+    port: 6379,
+    db: 1,
+    namespace: 'contentful'
+  )
+end
+```
+
+### Default env
+
+If unset the default call is to the `:published` data. however, setting default_env to `:preview` will request to the preview API.
+The Find methods can have an additional argument to force the non-default endpoint.
+
+```ruby
+# config/initializers/contentful_redis.rb
+ContentfulRedis.configure do |config|
+  # if unset defaults to :published
+  config.default_env = :preview
+end
+```
+
+### Model scope
+Set the scope for where your models live.
+
+```ruby
+# config/initializers/contentful_redis.rb
+ContentfulRedis.configure do |config|
+  config.model_scope = 'Contentful'
+end
+
+# app/models/contentful/page.rb
+module Contentful
+  class Page < ContentfulRedis::ModelBase
+
+  end
+end
+```
+
+## Models
+All content models will need to be defined, prior to integration especially when using references.
+The example model we are going to define has a slug(input field) and a body(references other content models)
+
+```ruby
+# app/models/page.rb
+class Page < ContentfulRedis::ModelBase
+  # allows the field to be queried from
+  define_searchable_fields :slug
+
+  # Set default readers which can return nil
+  attr_reader: :slug
+  
+  # define your desired return types manually
+  def body
+    @body || []
+  end
+end
+```
+
+### Querying
+
+All content models are found by their contentful ID. Contentful Redis only stores only one cache of the content model
+This Redis key is generated and is unique to a content model, space and endpoint.
+
+```ruby
+  Contentful::Page.find('<contentful_uid>')
+```
+
+Contentful Redis does not store a duplicate object from searchable attributes,
+Instead, it builds a glossary of searchable attributes mapping to their content models ids.
+These attributes are defined in the class declaration as `define_searchable_fields :slug`
+
+```ruby
+  Contentful::Page.find_by(slug: 'about-us') 
+```
+
+### Content model overriding
+
+Classes should match their content model name, however, if they don't you can override the classes `#name` method.
+
+```ruby
+# app/models/page.rb
+class Page < ContentfulRedis::ModelBase
+
+  # Overwrite to match contentful model using ruby class syntax
+  def self.name
+    'NameThatMatchesContentfulModel'
+  end
+end
+```
+
+## Webhooks
+
+Instead of creating rails specific implementation it is up to the developers to create your controllers and manage your webhook into your applications.
+See the [Contentful webhooks docs](https://www.contentful.com/developers/docs/concepts/webhooks/) creating your own
+
+Examples below will get you started!
+
+Required Contentful webhooks to update the Redis cache are:
+```json
+{
+  "id": "{ /payload/sys/id }",
+  "environment": "{ /payload/sys/environment/sys/id }",
+  "model": "{ /payload/sys/contentType/sys/id }"
+}
+```
+
+When pushing text attributes make sure you are using the correct language endpoint.
+```json
+{
+  "title": "{ /payload/fields/title/en-US }",
+  "slug": "{ /payload/fields/slug/en-US }",
+}
+```
+
+### Webhook Controllers
+
+#### Rails
+```ruby
+# app/controllers/contentful/webhook_controller.rb
+module Contentful
+  class WebhookController < ApplicationController
+    # before_action :some_auth_layer
+
+    def update
+      payload = JSON.parse request.raw_post
+      
+      contentful_model = ContentfulRedis::ClassFinder.search(payload['model'])
+      contentful_model.update(payload['id'])
+
+      render json: { status: :ok }
+    end
+
+    def delete
+      payload = JSON.parse request.raw_post
+      
+      contentful_model = ContentfulRedis::ClassFinder.search(payload['model'])
+      contentful_model.destroy(payload['id'])
+
+      render json: { status: :ok }
+    end
+  end
+end
+
+# config/routes
+#...
+namespace :contentful do
+  resource 'webhooks', only: :update, :delete
+end
+# ...
+```
+
+#### Other
+Feel free to create a PR for other ruby frameworks :)
+
+## Content Seeding
+Seeding the data is a great way to get started in building your content models, there is a couple of ways this can be done.
+
+Create a service object inside your application and get it to fetch the root pages of your content tree by their ID.
+The find method will build your Redis cache as well as link your content models with their searchable fields.
+
+```ruby
+# app/services/seed_content.rb
+class SeedContent
+  # trigger a cascading content model seeding process
+  def call
+    ['xxContentfulModelIdxx'].each do |page|
+      Contentful::Page.find(page)
+    end
+  end
+end
+```
+
+## Development
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a Git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/contentful-redis.
+
+## License
+
+The gem is available as open source under the terms of the [GNU General Public License](https://www.gnu.org/licenses/#GPL)

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "contentful_redis/rb"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/contentful-redis.gemspec

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+# rubocop:disable Metrics/BlockLength
+lib = File.expand_path('lib', __dir__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'contentful_redis'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'contentful_redis'
+  spec.version       = ContentfulRedis::VERSION
+  spec.authors       = ['DanHenton']
+  spec.email         = ['Dan.Henton@gmail.com']
+
+  spec.summary       = 'Contentful api wrapper which caches responses from contentful'
+  spec.homepage      = 'https://github.com/DigitalNZ/contentful-redis'
+  spec.license       = 'GNU'
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(File.expand_path(__dir__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  end
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  spec.add_dependency 'faraday'
+  spec.add_dependency 'redis-store'
+
+  spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'factory_bot'
+  spec.add_development_dependency 'pry'
+  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'rspec'
+  spec.add_development_dependency 'rubocop'
+  spec.add_development_dependency 'webmock'
+end
+# rubocop:enable Metrics/BlockLength

data/lib/contentful_redis/asset.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module ContentfulRedis
+  class Asset
+    attr_reader :id, :title, :description, :url, :details, :file_name, :content_type
+
+    def initialize(model)
+      @id           = model.dig('sys',    'id')
+      @title        = model.dig('fields', 'title')
+      @description  = model.dig('fields', 'description')
+      @url          = model.dig('fields', 'file', 'url')
+      @details      = model.dig('fields', 'file', 'details')
+      @file_name    = model.dig('fields', 'file', 'fileName')
+      @content_type = model.dig('fields', 'file', 'contentType')
+    end
+  end
+end

data/lib/contentful_redis/class_finder.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module ContentfulRedis
+  module ClassFinder
+    def self.search(type)
+      begin
+        "#{ContentfulRedis.configuration.model_scope}#{type.classify}".constantize
+      rescue NameError => _e
+        raise ContentfulRedis::Error::ClassNotFound, "Content type: #{type} is undefined"
+      end
+    end
+  end
+end

data/lib/contentful_redis/configuration.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module ContentfulRedis
+  class Configuration
+    attr_writer :model_scope
+    attr_accessor :spaces, :redis, :default_env, :logging
+
+    def model_scope
+      "#{@model_scope}::" unless @model_scope.nil?
+    end
+  end
+end

data/lib/contentful_redis/error.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module ContentfulRedis
+  module Error
+    class ArgumentError < StandardError; end
+    class RecordNotFound < StandardError; end
+    class ClassNotFound < StandardError; end
+    class InternalServerError < StandardError; end
+  end
+end

data/lib/contentful_redis/key_manager.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module ContentfulRedis
+  module KeyManager
+    class << self
+      # Links a contentful models attribute to its contentful_id
+      def attribute_index(klass, attribute)
+        "#{klass.space.fetch(:space_id)}/#{klass.content_model}/#{attribute}"
+      end
+
+      # Links content model request to its contentful json response
+      def content_model_key(space, endpoint, parameters)
+        "#{space.fetch(:space_id)}/#{endpoint}/#{parameters.map { |k, v| "#{k}-#{v}" }.join('/')}"
+      end
+    end
+  end
+end

data/lib/contentful_redis/model_base.rb

@@ -0,0 +1,150 @@
+# frozen_string_literal: true
+
+require_relative 'asset'
+require_relative 'request'
+require_relative 'key_manager'
+require_relative 'error'
+require_relative 'class_finder'
+
+# Base class for contentful redis intergation.
+module ContentfulRedis
+  class ModelBase
+    class << self
+      def find(id, env = nil)
+        parameters = { 'sys.id': id, content_type: content_model }
+
+        new(ContentfulRedis::Request.new(space, parameters, :get, request_env(env)).call)
+      end
+
+      def find_by(args, env = ContentfulRedis.configuration.default_env || :published)
+        raise ContentfulRedis::Error::ArgumentError, "#{args} contain fields which are not a declared as a searchable field" unless (args.keys - searchable_fields).empty?
+
+        id = args.values.map do |value|
+          key = ContentfulRedis::KeyManager.attribute_index(self, value)
+          key.nil? || key.empty? ? nil : ContentfulRedis.redis.get(key)
+        end.compact.first
+
+        raise ContentfulRedis::Error::RecordNotFound, 'Missing attribute in glossary' if id.nil?
+
+        find(id, env)
+      end
+
+      def update(id, env = nil)
+        parameters = { 'sys.id': id, content_type: content_model }
+
+        new(ContentfulRedis::Request.new(space, parameters, :update, request_env(env)).call)
+      end
+
+      def destroy(id, env = nil)
+        keys = []
+        keys << ContentfulRedis::KeyManager.content_model_key(space, request_env(env), 'sys.id': id, content_type: content_model)
+
+        searchable_fields.each do |field|
+          keys << ContentfulRedis::KeyManager.attribute_index(self, field)
+        end
+
+        ContentfulRedis.redis.del(*keys)
+      end
+
+      def space
+        ContentfulRedis.configuration.spaces.first[1]
+      end
+
+      def content_model
+        model_name = name.demodulize
+
+        "#{model_name[0].downcase}#{model_name[1..-1]}"
+      end
+
+      def searchable_fields
+        []
+      end
+
+      def define_searchable_fields(*fields)
+        instance_eval("def searchable_fields; #{fields}; end")
+      end
+
+      private
+
+      def request_env(env)
+        env || ContentfulRedis.configuration.default_env || :published
+      end
+    end
+
+    def initialize(model)
+      instance_variable_set(:@id, model['items'].first.dig('sys', 'id'))
+      self.class.send(:attr_reader, :id)
+
+      entries = entries_as_objects(model)
+
+      model['items'].first['fields'].each do |key, value|
+        value = case value
+                when Array
+                  value.map { |val| entries[val.dig('sys', 'id')] || val }
+                when Hash
+                  extract_object_from_hash(model, value, entries)
+                else
+                  value
+                end
+
+        instance_variable_set("@#{key.underscore}", value)
+      end
+
+      create_searchable_attribute_links if self.class.searchable_fields.any?
+    end
+
+    def content_type
+      self.class.name.demodulize.underscore
+    end
+
+    private
+
+    def entries_as_objects(model)
+      entries = model.dig('includes', 'Entry')
+
+      return {} if entries.nil? || entries.empty?
+
+      entries.each_with_object({}) do |entry, hash|
+        type = entry.dig('sys', 'contentType', 'sys', 'id')
+        id = entry.dig('sys', 'id')
+
+        hash[id] = ContentfulRedis::ClassFinder.search(type).find(id)
+      end
+    end
+
+    def extract_object_from_hash(model, value, entries)
+      entry_id = value.dig('sys', 'id')
+
+      assets = model.dig('includes', 'Asset')
+      asset = if !assets.nil? && assets.is_a?(Array)
+                model.dig('includes', 'Asset').first
+              end
+
+      if entries.key?(entry_id)
+        entries[entry_id]
+      elsif !asset.nil?
+        ContentfulRedis::Asset.new(asset)
+      else
+        value
+      end
+    end
+
+    def create_searchable_attribute_links
+      self.class.searchable_fields.each do |field|
+        begin
+          instance_attribute = send(field)
+        rescue NoMethodError => _e
+          raise ContentfulRedis::Error::ArgumentError, "Undefined attribute: #{field} when creating attribute glossary"
+        end
+
+        raise ContentfulRedis::Error::ArgumentError, 'Searchable fields cannot be blank and must be required' if instance_attribute.nil?
+        raise ContentfulRedis::Error::ArgumentError, 'Searchable fields must be singular and cannot be references' if instance_attribute.is_a?(Array)
+
+        key = ContentfulRedis::KeyManager.attribute_index(self.class, send(field))
+        next if ContentfulRedis.redis.exists(key)
+
+        ContentfulRedis.redis.set(key, id)
+      end
+    end
+  end
+end

data/lib/contentful_redis/request.rb

@@ -0,0 +1,98 @@
+# frozen_string_literal: true
+
+require_relative 'key_manager'
+require_relative 'error'
+
+# Request from contentful api and store in redis.
+# Atempt to fetch response from redis before requesting to the contentful api
+module ContentfulRedis
+  class Request
+    def initialize(space, params, action = :get, env = ContentfulRedis.configuration.default_env || :published)
+      @space = space
+      @action = action
+
+      if env.to_s.downcase == 'published'
+        @endpoint = 'cdn'
+        @access_token = @space[:access_token]
+      else
+        @endpoint = 'preview'
+        @access_token = @space[:preview_access_token]
+      end
+
+      params[:include] = 1
+
+      @parameters = params
+    end
+
+    def call
+      generated_key = ContentfulRedis::KeyManager.content_model_key(@space, @endpoint, @parameters)
+
+      return fetch_from_origin(generated_key) if @action == :update || !ContentfulRedis.redis.exists(generated_key)
+
+      JSON.parse(ContentfulRedis.redis.get(generated_key))
+    end
+
+    private
+
+    def fetch_from_origin(generated_key)
+      response = perform_request
+
+      raise ContentfulRedis::Error::RecordNotFound, 'Contentful entry was not found' if response.match?(/"total":0/)
+
+      ContentfulRedis.redis.set(generated_key, response)
+
+      JSON.parse(response)
+    end
+
+    def perform_request
+      res = faraday_connection.get do |req|
+        req.url "https://#{@endpoint}.contentful.com/spaces/#{@space[:space_id]}/environments/master/entries"
+
+        req.params = @parameters
+      end
+
+      catch_errors(res)
+
+      # decompress then use JSON.parse to remove any blank characters to reduce bytesize
+      # Even when we ask for gzip encoding if content model is small contentfull wont gzib the response body
+      #
+      # Futher storage optimizations can be made to reduce the total redis size.
+      begin
+        JSON.parse(Zlib::GzipReader.new(StringIO.new(res.body)).read).to_json
+      rescue Zlib::GzipFile::Error
+        JSON.parse(res.body).to_json
+      end
+    end
+
+    def faraday_connection
+      ::Faraday.new do |faraday|
+        faraday.request :url_encoded
+
+        if ContentfulRedis.configuration.logging
+          faraday.response :logger do |logger|
+            logger.filter(/(Authorization:)(.*)/, '\1[REMOVED]')
+          end
+        end
+
+        faraday.adapter Faraday.default_adapter
+        faraday.headers = {
+          'Authorization': "Bearer #{@access_token}",
+          'Content-Type': 'application/vnd.contentful.delivery.v1+json',
+          'Accept-Encoding': 'gzip'
+        }
+      end
+    end
+
+    def catch_errors(res)
+      send("__#{res.status.to_s[0]}00_error__") unless res.status == 200
+    end
+
+    def __400_error__
+      raise ContentfulRedis::Error::RecordNotFound, 'Contentful could not find the content entry'
+    end
+
+    def __500_error__
+      raise ContentfulRedis::Error::InternalServerError, 'An external Contentful error has occured'
+    end
+  end
+end