mongoid-cached-json 1.5.3 → 1.7.0

data/CONTRIBUTING.md

@@ -1,16 +1,16 @@
 Contributing to Mongoid::CachedJson
 ===================================
 
-Mongoid::CachedJson is work of [many of contributors](https://github.com/dblock/mongoid-cached-json/graphs/contributors). You're encouraged to submit [pull requests](https://github.com/dblock/mongoid-cached-json/pulls), [propose features and discuss issues](https://github.com/dblock/mongoid-cached-json/issues). When in doubt, ask a question in the [Google Group](http://groups.google.com/group/mongoid-cached-json).
+Mongoid::CachedJson is work of [many of contributors](https://github.com/mongoid/mongoid-cached-json/graphs/contributors). You're encouraged to submit [pull requests](https://github.com/mongoid/mongoid-cached-json/pulls), [propose features and discuss issues](https://github.com/mongoid/mongoid-cached-json/issues). When in doubt, ask a question in the [Google Group](http://groups.google.com/group/mongoid-cached-json).
 
 #### Fork the Project
 
-Fork the [project on Github](https://github.com/dblock/mongoid-cached-json) and check out your copy.
+Fork the [project on Github](https://github.com/mongoid/mongoid-cached-json) and check out your copy.
 
 ```
 git clone https://github.com/contributor/mongoid-cached-json.git
 cd mongoid-cached-json
-git remote add upstream https://github.com/dblock/mongoid-cached-json.git
+git remote add upstream https://github.com/mongoid/mongoid-cached-json.git
 ```
 
 #### Create a Topic Branch
@@ -63,6 +63,8 @@ Document any external behavior in the [README](README.md).
 
 Add a line to [CHANGELOG](CHANGELOG.md) under *Next Release*. Make it look like every other line, including your name and link to your Github account.
 
+Danger checks pull requests with `danger-changelog` and reports missing changelog entries in a PR comment. Include the pull request number in your entry once it is available.
+
 #### Commit Changes
 
 Make sure git knows your name and email address:
@@ -104,7 +106,7 @@ git push origin my-feature-branch -f
 Update the [CHANGELOG](CHANGELOG.md) with the pull request number. A typical entry looks as follows.
 
 ```
-* [#123](https://github.com/dblock/mongoid-cached-json/pull/123): Reticulated splines - [@contributor](https://github.com/contributor).
+* [#123](https://github.com/mongoid/mongoid-cached-json/pull/123): Reticulated splines - [@contributor](https://github.com/contributor).
 ```
 
 Amend your previous commit and force push the changes.

data/Dangerfile

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+danger.import_dangerfile(gem: 'danger-pr-comment')
+
+changelog.check!

data/Gemfile

@@ -1,27 +1,27 @@
-source 'http://rubygems.org'
+source 'https://rubygems.org'
 
 gemspec
 
-case version = ENV['MONGOID_VERSION'] || '~> 5.0'
-when /5/
-  gem 'mongoid', '~> 5.0'
-when /4/
-  gem 'mongoid', '~> 4.0'
-when /3/
-  gem 'mongoid', '~> 3.1'
-else
-  gem 'mongoid', version
-end
+version = ENV['MONGOID_VERSION'] || '9'
+gem 'bigdecimal' if RUBY_VERSION >= '3.4'
+gem 'logger' if RUBY_VERSION >= '3.4'
+gem 'mongoid', version =~ /\A\d+\z/ ? "~> #{version}.0" : version
 
 group :development do
+  if RUBY_VERSION >= '2.7'
+    gem 'danger'
+    gem 'danger-changelog', require: false
+    gem 'danger-pr-comment', require: false
+  end
+  gem 'dalli', RUBY_VERSION >= '3.0' ? '~> 3.0' : '~> 2.6'
   gem 'rake'
-  gem 'bundler', '~> 1.0'
+  gem 'rdoc', RUBY_VERSION >= '3.0' ? '>= 6' : '~> 4.2'
+  gem 'rubocop', ENV.fetch('RUBOCOP_VERSION', '>= 0')
   gem 'yard', '~> 0.6'
-  gem 'dalli', '~> 2.6'
-  gem 'rubocop', '0.33.0'
 end
 
 group :test do
   gem 'mongoid-compatibility'
   gem 'rspec', '~> 3.1'
+  gem 'simplecov', '~> 0.22' if ENV['COVERAGE'] == 'true'
 end

data/README.md

@@ -2,9 +2,10 @@ Mongoid::CachedJson
 ===================
 
 [![Gem Version](https://badge.fury.io/rb/mongoid-cached-json.svg)](http://badge.fury.io/rb/mongoid-cached-json)
-[![Build Status](https://travis-ci.org/dblock/mongoid-cached-json.svg?branch=master)](https://travis-ci.org/dblock/mongoid-cached-json)
-[![Dependency Status](https://gemnasium.com/dblock/mongoid-cached-json.svg)](https://gemnasium.com/dblock/mongoid-cached-json)
-[![Code Climate](https://codeclimate.com/github/dblock/mongoid-cached-json.svg)](https://codeclimate.com/github/dblock/mongoid-cached-json)
+[![Test](https://github.com/mongoid/mongoid-cached-json/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/mongoid/mongoid-cached-json/actions/workflows/test.yml?query=branch%3Amaster)
+[![Coverage Status](https://coveralls.io/repos/github/mongoid/mongoid-cached-json/badge.svg?branch=master)](https://coveralls.io/github/mongoid/mongoid-cached-json?branch=master)
+
+The Ruby 4.0 CI job measures library line coverage with SimpleCov and submits `coverage/.resultset.json` to Coveralls using GitHub's automatic token. Other matrix jobs do not submit duplicate reports. Run `COVERAGE=true bundle install` followed by `COVERAGE=true bundle exec rake spec` for a local report. The default-branch badge requires Coveralls to accept an upstream `master` build; configuring this workflow does not activate the external service.
 
 Typical `as_json` definitions may involve lots of database point queries and method calls. When returning collections of objects, a single call may yield hundreds of database queries that can take seconds. This library mitigates the problem by implementing a module called `CachedJson`.
 
@@ -12,16 +13,12 @@ Typical `as_json` definitions may involve lots of database point queries and met
 
 Using `Mongoid::CachedJson` we were able to cut our JSON API average response time by about a factor of 10.
 
-Compatibility
--------------
-
-This gem is compatible with Mongoid 3, 4 and 5.
-
 Resources
 ---------
 
 * [Need Help? Google Group](http://groups.google.com/group/mongoid-cached-json)
-* [Source Code](http://github.com/dblock/mongoid-cached-json)
+* [Source Code](http://github.com/mongoid/mongoid-cached-json)
+* [Upgrading](UPGRADING.md)
 
 Quickstart
 ----------
@@ -102,7 +99,7 @@ Defining Fields
 
 `Mongoid::CachedJson` supports the following options:
 
-* `:hide_as_child_json_when` is an optional function that hides the child JSON from `as_json` parent objects, eg. `cached_json hide_as_child_json_when: lambda { |instance| ! instance.secret? }`
+* `:hide_as_child_json_when` is an optional function that omits the child key from `as_json` parent objects when it returns true, eg. `cached_json hide_as_child_json_when: lambda { |instance| ! instance.secret? }`. A missing relationship remains a `nil` value.
 
 `Mongoid::CachedJson` field definitions support the following options:
 
@@ -111,8 +108,66 @@ Defining Fields
 * `:properties` can be one of `:short`, `:public`, `:all`, in this order
 * `:version` can be a single version for this field to appear in
 * `:versions` can be an array of versions for this field to appear in
+* `:cache` defaults to `true`; set it to `false` to recompute a field on every serialization (see below)
 * `:reference_properties` can be one of `:short`, `:public`, `:all`, default will select the reference properties format dynamically (see below)
 
+Per-Field Caching
+-----------------
+
+Use `cache: false` for a field whose value should be evaluated on every `as_json` call, even when the rest of the object's JSON comes from the cache. The field is excluded from the cached fragment and merged into the response after that fragment is fetched. Its `definition`, property/version filters, and configured transformations still apply.
+
+``` ruby
+class Product
+  include Mongoid::Document
+  include Mongoid::CachedJson
+
+  field :name
+  field :stock, type: Integer
+
+  json_fields \
+    name: {},
+    stock: { cache: false }
+end
+
+product = Product.create!(name: 'Widget', stock: 10)
+product.as_json # => { name: 'Widget', stock: 10 }
+
+# Unsaved changes illustrate which fields are recomputed on a cache hit.
+product.name = 'Renamed widget'
+product.stock = 9
+product.as_json # => { name: 'Widget', stock: 9 }
+```
+
+The cached `name` stays unchanged until its cache entry expires or is invalidated (for example, by saving the product); `stock` is read from the current model instance each time. This option does not reload the model from MongoDB: reload a stale instance yourself if another process has changed its stored values.
+
+For a `type: :reference` field, `cache: false` bypasses reads and writes of the referenced object's own JSON fragment through that relationship. The parent's other fields remain cached:
+
+``` ruby
+class Order
+  include Mongoid::Document
+  include Mongoid::CachedJson
+
+  field :number
+  belongs_to :product
+
+  json_fields \
+    number: {},
+    product: { type: :reference, cache: false }
+end
+
+order = Order.create!(number: '123', product: product)
+order.as_json # => { number: '123', product: { name: 'Renamed widget', stock: 9 } }
+```
+
+The `product` reference is recomputed even if the product has a cached JSON fragment from another serialization. This does not disable caching on the `Product` model itself: `product.as_json` and other relationships to that product still use their normal cache policies. References nested inside the product also retain their own cache policies; the bypass is not recursive.
+
+All Stored Fields
+-----------------
+
+Use `json_fields :all, display_name: { definition: :display_name }` to expand every Mongoid field declared so far (including `_id`) into an ordinary JSON field, then merge explicit definitions. An explicit definition overrides the generated definition for the same symbol key; for example, `json_fields :all, secret: { properties: :all }` restricts that field to the `:all` representation.
+
+Declare Mongoid fields before `json_fields :all`. Later declarations, dynamic attributes, and associations are not added automatically; declare references explicitly with `type: :reference`. This is opt-in and may expose sensitive stored data or foreign keys, so prefer an explicit allowlist for public APIs. Runtime `as_json(methods: ...)` selection remains unsupported: declare computed methods in `json_fields`.
+
 Reference Properties
 --------------------
 
@@ -120,7 +175,7 @@ When calling `as_json` on a model that contains references to other models the v
 
 * Use the value of the `:reference_properties` option, if specified.
 * For `:short` JSON, use `:short`.
-* For `:public` JSON, use `:short`.
+* For `:public` JSON, use `:public`.
 * For `:all` JSON, use `:all`.
 
 The dynamic selection where `:public` generates `:short` references allows to return smaller embedded collections, while `:all` allows to fetch deep data. Another way of looking at this is to say that a field in a `:short` JSON appears in collections, a field declared in the `:public` JSON appears for all users and the field declared in the `:all` JSON appears for object owners only.
@@ -178,6 +233,13 @@ person.as_json(version: :v2) # { first: 'John', middle: 'F.', last: 'Kennedy', n
 person.as_json(version: :v3) # { first: 'John', middle: 'F.', last: 'Kennedy', name: 'John F. Kennedy', born: 'May 29, 1917' }
 ```
 
+Schema Cache Versions
+---------------------
+
+Declare `json_fields cache_version: 2, name: {}` to add a model-specific schema namespace to its cache keys. Increment it when changing fields, computed values, or transformations in a deployment. This is independent of the API `version:` passed to `as_json`; callers need not change. Omitting `cache_version` preserves existing keys.
+
+New code stops reading old fragments, including referenced fragments of that model. Old entries are not deleted and remain until eviction or a cache-store TTL. The gem does not set a default TTL; configure `expires_in` on your cache store if needed. Updates expire only the current namespace. For rolling deployments or rollbacks, do not reuse a namespace containing stale values: coordinate invalidation or choose another version.
+
 Transformations
 ---------------
 
@@ -214,6 +276,28 @@ Turning Caching Off
 
 You can set `Mongoid::CachedJson.config.disable_caching = true`. It may be a good idea to set it to `ENV['DISABLE_JSON_CACHING']`, in case this turns out not to be The Solution To All Of Your Performance Problems (TM).
 
+To bypass shared-cache reads and writes for one model while retaining its JSON fields, versions, and property hierarchy, declare `json_fields disable_caching: true, name: {}`. This applies when the model is serialized directly, in collections, or as a reference. Other models, including its parent and referenced children, keep their own caching policy. Changes to an uncached child are visible even when its parent's scalar fields are cached.
+
+Cache Invalidation Boundaries
+-----------------------------
+
+Normal model updates and destroys expire that document's JSON fragments through Mongoid callbacks. Bulk or raw collection updates that bypass model callbacks cannot invalidate the cache automatically. Explicitly expire each affected document after such a write:
+
+``` ruby
+User.where(id: user_id).update_all(status: 'active')
+User.find(user_id).expire_cached_json
+```
+
+Concurrent writes have a separate boundary: an older in-memory instance can repopulate a fragment after another instance has saved newer values and expired the cache. Reload stale instances before serializing them, and expire their fragments if a stale writer may already have repopulated the cache:
+
+``` ruby
+user.reload
+user.expire_cached_json
+user.as_json
+```
+
+Reloading and expiring are not an atomic cross-process operation. This gem does not provide a distributed revision or compare-and-set protocol for cache stores. Applications requiring strict consistency across concurrent writers should coordinate persistence and cache invalidation themselves or disable caching for the affected data.
+
 Testing JSON
 ------------
 
@@ -293,6 +377,6 @@ See [CONTRIBUTING](CONTRIBUTING.md).
 Copyright and License
 ---------------------
 
-MIT License, see [LICENSE](https://github.com/dblock/mongoid-cached-json/blob/master/LICENSE.md) for details.
+MIT License, see [LICENSE](https://github.com/mongoid/mongoid-cached-json/blob/master/LICENSE.md) for details.
 
 (c) 2012-2014 [Artsy](https://artsy.net) and [Contributors](CHANGELOG.md)

data/Rakefile

@@ -1,7 +1,7 @@
 require 'rubygems'
 require 'bundler'
 
-require File.expand_path('../lib/mongoid-cached-json/version', __FILE__)
+require File.expand_path('lib/mongoid-cached-json/version', __dir__)
 
 Bundler.setup(:default, :development)
 Bundler::GemHelper.install_tasks

data/UPGRADING.md

@@ -0,0 +1,59 @@
+# Upgrading
+
+## Upgrading to 1.7.0
+
+### Hidden child references
+
+[#35](https://github.com/mongoid/mongoid-cached-json/pull/35) changes how a single child reference is serialized when the child's `hide_as_child_json_when` predicate returns `true`. Previously, an existing but hidden child and a missing child both appeared as `child: nil` (`"child": null` in JSON), so clients could not distinguish them. Now, an existing but hidden child's key is omitted; a missing child still appears as `child: nil`.
+
+Here, `parent` is a `SecretParent` document with a `has_one` relationship to a `SometimesSecret` document. The parent exposes that relationship as the JSON field `child`; the hiding predicate is defined on the child model:
+
+```ruby
+class SecretParent
+  include Mongoid::Document
+  include Mongoid::CachedJson
+
+  field :name
+  has_one :sometimes_secret
+
+  json_fields \
+    name: {},
+    child: { definition: :sometimes_secret, type: :reference }
+end
+
+class SometimesSecret
+  include Mongoid::Document
+  include Mongoid::CachedJson
+
+  field :secret
+  field :should_tell_secret, type: Boolean
+  belongs_to :secret_parent
+
+  json_fields \
+    hide_as_child_json_when: lambda { |child| !child.should_tell_secret },
+    secret: {}
+end
+
+parent = SecretParent.create!(name: 'Parent')
+parent.create_sometimes_secret!(secret: 'Afraid of the dark', should_tell_secret: false)
+
+# The child exists, but its hiding predicate returns true.
+# Before 1.7.0:
+parent.as_json(properties: :short)
+# => { name: 'Parent', child: nil }
+
+# From 1.7.0:
+parent.as_json(properties: :short)
+# => { name: 'Parent' }
+
+# By contrast, a parent with no child still includes an explicit nil.
+parent_without_child = SecretParent.create!(name: 'No child')
+parent_without_child.as_json(properties: :short)
+# => { name: 'No child', child: nil }
+```
+
+A genuinely missing relationship still includes `child: nil`. Requests using `properties: :all` and serialization of the child itself are unchanged.
+
+Update clients and response schemas that require the child key to allow its absence. In Ruby, `json[:child]` still returns `nil` for either case; use `json.key?(:child)` to distinguish an omitted key from an explicit `nil`.
+
+Invalidate existing cached JSON fragments when deploying this change so cached results use the new representation. If the cache is shared with other applications, invalidate only this application's cached JSON entries rather than clearing the entire store.

data/lib/mongoid-cached-json/cached_json.rb

@@ -1,14 +1,19 @@
 # encoding: utf-8
 module Mongoid
   module CachedJson
+    HIDDEN_JSON = :__mongoid_cached_json_hidden__
+
     extend ActiveSupport::Concern
 
     included do
       class_attribute :all_json_properties
       class_attribute :all_json_versions
       class_attribute :cached_json_field_defs
+      class_attribute :cached_json_uncached_field_defs
       class_attribute :cached_json_reference_defs
       class_attribute :hide_as_child_json_when
+      class_attribute :cached_json_cache_version
+      class_attribute :cached_json_disable_caching
     end
 
     module ClassMethods
@@ -17,11 +22,17 @@ module Mongoid
       # @param [ hash ] defs JSON field definition.
       #
       # @since 1.0
-      def json_fields(defs)
+      def json_fields(defs, additions = {})
+        if defs == :all
+          defs = Hash[fields.keys.map { |field| [field.to_sym, {}] }].merge(additions)
+        end
+        self.cached_json_cache_version = defs.delete(:cache_version)
+        self.cached_json_disable_caching = defs.delete(:disable_caching)
         self.hide_as_child_json_when = defs.delete(:hide_as_child_json_when) || lambda { |_a| false }
         self.all_json_properties = [:short, :public, :all]
         cached_json_defs = Hash[defs.map { |k, v| [k, { type: :callable, properties: :short, definition: k }.merge(v)] }]
         self.cached_json_field_defs = {}
+        self.cached_json_uncached_field_defs = {}
         self.cached_json_reference_defs = {}
         # Collect all versions for clearing cache
         self.all_json_versions = cached_json_defs.map do |_field, definition|
@@ -29,7 +40,10 @@ module Mongoid
         end.flatten.compact.uniq
         all_json_properties.each_with_index do |property, i|
           cached_json_field_defs[property] = Hash[cached_json_defs.find_all do |_field, definition|
-            all_json_properties.find_index(definition[:properties]) <= i && definition[:type] == :callable
+            all_json_properties.find_index(definition[:properties]) <= i && definition[:type] == :callable && definition[:cache] != false
+          end]
+          cached_json_uncached_field_defs[property] = Hash[cached_json_defs.find_all do |_field, definition|
+            all_json_properties.find_index(definition[:properties]) <= i && definition[:type] == :callable && definition[:cache] == false
           end]
           cached_json_reference_defs[property] = Hash[cached_json_defs.find_all do |_field, definition|
             all_json_properties.find_index(definition[:properties]) <= i && definition[:type] == :reference
@@ -49,8 +63,10 @@ module Mongoid
       def materialize_cached_json(clazz, id, object_reference, options)
         is_top_level_json = options[:is_top_level_json] || false
         object_reference = clazz.where(_id: id).first unless object_reference
-        if !object_reference || (!is_top_level_json && options[:properties] != :all && clazz.hide_as_child_json_when.call(object_reference))
+        if !object_reference
           nil
+        elsif !is_top_level_json && options[:properties] != :all && clazz.hide_as_child_json_when.call(object_reference)
+          HIDDEN_JSON
         else
           Hash[clazz.cached_json_field_defs[options[:properties]].map do |field, definition|
             # version match
@@ -65,6 +81,22 @@ module Mongoid
         end
       end
 
+      # Materialize JSON fields that opt out of shared caching.
+      def materialize_uncached_json(clazz, id, object_reference, options)
+        object_reference = clazz.where(_id: id).first unless object_reference
+        return {} unless object_reference
+
+        Hash[clazz.cached_json_uncached_field_defs[options[:properties]].map do |field, definition|
+          versions = ([definition[:version]] | Array(definition[:versions])).compact
+          next unless versions.empty? || versions.include?(options[:version])
+          json_value = (definition[:definition].is_a?(Symbol) ? object_reference.send(definition[:definition]) : definition[:definition].call(object_reference))
+          Mongoid::CachedJson.config.transform.each do |t|
+            json_value = t.call(field, definition, json_value)
+          end
+          [field, json_value]
+        end.compact]
+      end
+
       # Given an object definition in the form of either an object or a class, id pair,
       # grab the as_json representation from the cache if possible, otherwise create
       # the as_json representation by loading the object from the database. For any
@@ -84,7 +116,8 @@ module Mongoid
           id = object_def[:id]
         end
         key = cached_json_key(options, clazz, id)
-        json = { _ref: { _clazz: self, _key: key, _materialize_cached_json: [clazz, id, object_reference, options] } }
+        json = { _ref: { _clazz: self, _key: key, _disable_caching: clazz.cached_json_disable_caching,
+                        _materialize_cached_json: [clazz, id, object_reference, options] } }
         keys = KeyReferences.new
         keys.set_and_add(key, json)
         reference_defs = clazz.cached_json_reference_defs[options[:properties]]
@@ -98,6 +131,12 @@ module Mongoid
                 ref[:_parent] = json
                 ref[:_field] = field
               end
+              if definition[:cache] == false
+                references = reference.is_a?(Array) ? reference : [reference]
+                references.each do |item|
+                  item[:_ref][:_disable_caching] = true if item.is_a?(Hash) && item[:_ref]
+                end
+              end
               keys.merge_set(reference_keys)
               [field, reference]
             end])
@@ -109,7 +148,8 @@ module Mongoid
       # Cache key.
       def cached_json_key(options, cached_class, cached_id)
         base_class_name = cached_class.collection_name.to_s.singularize.camelize
-        "as_json/#{options[:version]}/#{base_class_name}/#{cached_id}/#{options[:properties]}/#{!!options[:is_top_level_json]}"
+        key = "as_json/#{options[:version]}/#{base_class_name}/#{cached_id}/#{options[:properties]}/#{!!options[:is_top_level_json]}"
+        cached_class.cached_json_cache_version.nil? ? key : "#{key}/schema/#{cached_class.cached_json_cache_version}"
       end
 
       # If the reference is a symbol, we may be lucky and be able to figure out the as_json
@@ -126,7 +166,13 @@ module Mongoid
           else
             clazz = reference_def[:metadata].class_name.constantize
           end
-          if reference_def[:metadata].relation == Mongoid::Relations::Referenced::ManyToMany
+          relation_class = if Mongoid::Compatibility::Version.mongoid7_or_newer?
+                             Mongoid::Association::Referenced::HasAndBelongsToMany::Proxy
+                           else
+                             Mongoid::Relations::Referenced::ManyToMany
+                           end
+
+          if reference_def[:metadata].relation == relation_class
             object_ids = object.send(key)
             if object_ids
               reference_json = object_ids.map do |id|
@@ -145,13 +191,11 @@ module Mongoid
           reference_def_definition = reference_def[:definition]
           reference = reference_def_definition.is_a?(Symbol) ? object.send(reference_def_definition) : reference_def_definition.call(object)
           reference_json = nil
-          if reference
-            if reference.respond_to?(:as_json_partial)
-              reference_keys, reference_json = reference.as_json_partial(options)
-              keys = keys ? keys.merge_set(reference_keys) : reference_keys
-            else
-              reference_json = reference.as_json(options)
-            end
+          if reference.respond_to?(:as_json_partial)
+            reference_keys, reference_json = reference.as_json_partial(options)
+            keys = keys ? keys.merge_set(reference_keys) : reference_keys
+          else
+            reference_json = reference.as_json(options)
           end
         end
         [keys, reference_json]
@@ -160,8 +204,10 @@ module Mongoid
 
     # Check whether the cache supports :read_multi and prefetch the data if it does.
     def self.materialize_json_references_with_read_multi(key_refs, partial_json)
-      unfrozen_keys = key_refs.keys.to_a.map(&:dup) if key_refs # see https://github.com/mperham/dalli/pull/320
-      read_multi = unfrozen_keys && Mongoid::CachedJson.config.cache.respond_to?(:read_multi)
+      if key_refs
+        unfrozen_keys = key_refs.select { |_key, refs| refs.any? { |ref| !ref[:_ref][:_disable_caching] } }.keys.map(&:dup)
+      end
+      read_multi = unfrozen_keys && !unfrozen_keys.empty? && Mongoid::CachedJson.config.cache.respond_to?(:read_multi)
       local_cache = read_multi ? Mongoid::CachedJson.config.cache.read_multi(*unfrozen_keys) : {}
       Mongoid::CachedJson.materialize_json_references(key_refs, local_cache, read_multi) if key_refs
       partial_json
@@ -174,7 +220,9 @@ module Mongoid
           _ref = ref.delete(:_ref)
           key = _ref[:_key]
           fetched_json = local_cache[key] if local_cache.key?(key)
-          unless fetched_json
+          if _ref[:_disable_caching]
+            fetched_json = _ref[:_clazz].materialize_cached_json(* _ref[:_materialize_cached_json])
+          elsif !fetched_json
             if read_multi
               # no value in cache, materialize and write
               fetched_json = (local_cache[key] = _ref[:_clazz].materialize_cached_json(* _ref[:_materialize_cached_json]))
@@ -186,8 +234,11 @@ module Mongoid
               end)
             end
           end
-          if fetched_json
+          if fetched_json == HIDDEN_JSON
+            _ref[:_parent].delete(_ref[:_field]) if _ref[:_parent]
+          elsif fetched_json
             ref.merge! fetched_json
+            ref.merge! _ref[:_clazz].materialize_uncached_json(* _ref[:_materialize_cached_json])
           elsif _ref[:_parent]
             # a single _ref that resolved to a nil
             _ref[:_parent][_ref[:_field]] = nil

data/lib/mongoid-cached-json/config.rb

@@ -3,7 +3,6 @@ module Mongoid
   module CachedJson
     module Config
       extend self
-      include ActiveSupport::Callbacks
 
       # Current configuration settings.
       attr_accessor :settings

data/lib/mongoid-cached-json/version.rb

@@ -1,5 +1,5 @@
 module Mongoid
   module CachedJson
-    VERSION = '1.5.3'
+    VERSION = '1.7.0'
   end
 end

data/mongoid-cached-json.gemspec

@@ -1,4 +1,4 @@
-$LOAD_PATH.push File.expand_path('../lib', __FILE__)
+$LOAD_PATH.push File.expand_path('lib', __dir__)
 require 'mongoid-cached-json/version'
 
 Gem::Specification.new do |s|
@@ -10,7 +10,7 @@ Gem::Specification.new do |s|
   s.required_rubygems_version = '>= 1.3.6'
   s.files = `git ls-files`.split("\n")
   s.require_paths = ['lib']
-  s.homepage = 'http://github.com/dblock/mongoid-cached-json'
+  s.homepage = 'http://github.com/mongoid/mongoid-cached-json'
   s.licenses = ['MIT']
   s.summary = 'Cached-json is a DSL for describing JSON representations of Mongoid models.'
   s.add_dependency 'mongoid', '>= 3.0'

data/spec/all_fields_spec.rb

@@ -0,0 +1,47 @@
+require 'spec_helper'
+
+describe 'Declaring all stored fields' do
+  before do
+    class AllFieldsDocument
+      include Mongoid::Document
+      include Mongoid::CachedJson
+      field :name
+      field :secret
+
+      def display_name
+        name.upcase
+      end
+
+      json_fields :all, secret: { properties: :all }, display_name: {}
+      field :added_later
+    end
+  end
+
+  after do
+    AllFieldsDocument.delete_all
+    Object.send(:remove_const, :AllFieldsDocument)
+  end
+
+  it 'expands fields at declaration time and adds computed fields' do
+    document = AllFieldsDocument.create!(name: 'example', secret: 'hidden', added_later: 'not declared')
+    expected = { _id: document.id, name: 'example', display_name: 'EXAMPLE' }
+    expect(document.as_json).to eq(expected)
+    expect([document].as_json).to eq([expected])
+    expect(document.as_json(properties: :all)).to eq(expected.merge(secret: 'hidden'))
+  end
+
+  it 'retains the normal field cache and invalidation behavior' do
+    document = AllFieldsDocument.create!(name: 'example')
+    document.as_json
+    document.name = 'unsaved'
+    expect(document.as_json[:name]).to eq('example')
+    document.save!
+    expect(document.as_json[:name]).to eq('unsaved')
+  end
+
+  it 'does not expose extra fields for ordinary explicit declarations' do
+    document = JsonFoobar.create!(foo: 'visible', bar: 'private')
+    expect(document.as_json).not_to have_key(:bar)
+    expect(document.as_json).not_to have_key(:_id)
+  end
+end

data/spec/cache_version_spec.rb

@@ -0,0 +1,58 @@
+require 'spec_helper'
+
+describe 'Schema cache versions' do
+  before do
+    class VersionedDocument
+      include Mongoid::Document
+      include Mongoid::CachedJson
+      field :value
+      json_fields cache_version: 1, value: {}
+    end
+  end
+
+  after do
+    VersionedDocument.delete_all
+    Object.send(:remove_const, :VersionedDocument)
+  end
+
+  [true, false].each do |bulk|
+    context "with bulk reads #{bulk}" do
+      before do
+        @store = ActiveSupport::Cache::MemoryStore.new
+        @store.singleton_class.send(:undef_method, :read_multi) unless bulk
+        Mongoid::CachedJson.config.cache = @store
+      end
+
+      it 'separates new schema fragments without changing the API version' do
+        document = VersionedDocument.create!(value: 'old')
+        expect(document.as_json).to eq(value: 'old')
+        document.value = 'new'
+        expect(document.as_json).to eq(value: 'old')
+        VersionedDocument.cached_json_cache_version = 2
+        expect(document.as_json).to eq(value: 'new')
+        VersionedDocument.cached_json_cache_version = 1
+        expect(document.as_json).to eq(value: 'old')
+      end
+
+      it 'invalidates the current namespace on a model update' do
+        document = VersionedDocument.create!(value: 'old')
+        [1, 2].each do |version|
+          VersionedDocument.cached_json_cache_version = version
+          document.as_json
+        end
+        document.update_attributes!(value: 'new')
+        expect(document.as_json).to eq(value: 'new')
+        VersionedDocument.cached_json_cache_version = 1
+        expect(document.as_json).to eq(value: 'old')
+      end
+
+      it 'uses the schema version for child fragments too' do
+        document = VersionedDocument.create!(value: 'old')
+        options = { properties: :short, version: :unspecified, is_top_level_json: false }
+        key = VersionedDocument.cached_json_key(options, VersionedDocument, document.id)
+        expect(key).to end_with('/false/schema/1')
+        expect([document].as_json).to eq([{ value: 'old' }])
+      end
+    end
+  end
+end