dolly 3.1.5 → 3.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c05943aabfd347226c64d622f7e2f872ad8f1f78e52e1229697a3a60364a5e53
-  data.tar.gz: 7a64b93ef1232cec80ca842f9d4e4ebb5dd1e260b967549d8b6ec0d12b0e929d
+  metadata.gz: 7a6b7aaee7e192101dddab520fbdc363a6e7379d1a52f08c36b3c21fe1371f21
+  data.tar.gz: 34e0686cb8726fc60d69f0c84457664c07e86fbbfec579aa0a56ad84c131ce1c
 SHA512:
-  metadata.gz: a50e0365e3dabf0620e6f0c2d13d57f029709616d02733758f2bb1317ef0d4906c4ec999dfef81f5dc95f94d94997dba36004ed1f3a7bf4f0be59f51895fe578
-  data.tar.gz: ff4faa8af117ef82db9c428387a0d31d24dd1d59f3e831d41fe4399ef03faf5ea710ac732bc39108a710191645fefdd31a5284a7448803de5dd1786893b704c0
+  metadata.gz: 6b43b48ccd6bfd844c68712dbd008c4e75cdff27d43c6c0fa885174e35cd170135b9975a352042a1382b29efa587e120648095a6971d089528d06d1227b4eec2
+  data.tar.gz: 374986fde329b1519c228a11fa844f140d8a7cf718b54d04b10f39c6071944bfe5852e2699bffdc34768a43720b4b46c52ad11d48dc2b6f481c8756d0767228a

data/README.md

@@ -1,6 +1,6 @@
-# Dolly3
+# Dolly
 
-The thing you move your couch with...
+CouchDB adapter for Ruby. The thing you move your couch with...
 
 ![example workflow](https://github.com/amco/dolly/actions/workflows/ruby.yml/badge.svg)
 
@@ -9,7 +9,7 @@ The thing you move your couch with...
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'dolly3'
+gem 'dolly'
 ```
 
 And then execute:
@@ -18,61 +18,112 @@ And then execute:
 
 Or install it yourself as:
 
-    $ gem install dolly3
+    $ gem install dolly
+
+Configure CouchDB in `config/couchdb.yml` (see the [sample config](config/couchdb.yml) in this repo).
 
 ## Usage
 
-TODO: Write usage instructions here
+Define documents as usual:
 
-## Development
+```ruby
+class FooBar < Dolly::Document
+  property :foo, :bar
+  timestamps!
+end
 
-After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+doc = FooBar.create(foo: 'a', bar: 'b')
+FooBar.find(doc.id)
+```
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+The public `Dolly::Connection` API (`get`, `post`, `put`, `delete`, `request`, `view`, `attach`, `uuids`, etc.) is unchanged for application code.
 
-## Contributing
+### Connection reuse (Curb)
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/dolly3.
+Dolly uses [Curb](https://github.com/taf2/curb) under the hood. Newer Curb releases keep HTTP connections alive, which previously leaked one TCP socket per Dolly request and could exhaust CouchDB's accept backlog during bursts (Sidekiq jobs, seeds, Passenger workers).
 
+Dolly now reuses **one `Curl::Easy` handle per thread** via internal `Dolly::Curl`
+collaborators (`Connection` for transport/retries, `WriteReconciler` for ambiguous writes):
 
-## Migrating from couch 1.x to 2.x
+- Persistent keep-alive to CouchDB without opening a new socket on every call
+- Safe for typical **Passenger** (one request per thread/process) and **Sidekiq** (one job per thread) deployments
+- Not safe under fiber-based servers that interleave multiple Dolly requests on a single thread
 
-[Official docs](https://docs.couchdb.org/en/2.3.1/install/index.html)
+### Stale keep-alive handling
+
+If CouchDB closes an idle keep-alive socket, Curb may raise a transport error (`GotNothingError`, `RecvError`, `SendError`, `PartialFileError`). Dolly always discards the dead handle, then:
+
+| Request | Behavior |
+|---------|----------|
+| `GET`, `HEAD` | Retry once on a fresh connection |
+| Read-only `POST` (`/_find`, `/_all_docs`) | Retry once on a fresh connection |
+| Document `PUT` | Reconcile by reading the document; succeed if content matches, retry once if the write clearly did not land, otherwise raise |
+| Document `DELETE` (with `rev`) | Reconcile by reading the document; succeed if gone/deleted, retry once if the same rev is still present, otherwise raise |
+| `POST /_bulk_docs` | Reconcile via `/_all_docs`; retry only documents that clearly did not apply |
+| Attachments and other writes | Do **not** blind-retry; raise |
+
+### `Dolly::AmbiguousWriteError`
+
+Raised when a write may or may not have been applied (transport failed after the request was possibly processed, and reconciliation cannot prove the outcome).
 
+```ruby
+begin
+  doc.save!
+rescue Dolly::AmbiguousWriteError => e
+  e.method # => :put
+  e.uri    # => #<URI::HTTP ...>
+  e.cause  # => original Curl::Err::*
+end
+```
 
-You will need to uninstall the couchdb service with brew
+Handle this in jobs/seeds by reloading or re-checking the document before retrying the business operation.
 
-`brew services stop couchdb`
-`brew services uninstall couchdb`
+## Development
 
-Download the application from the following source
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-http://couchdb.apache.org/#download
+```bash
+bundle exec rake test
+```
 
-launch fauxton and check your installation
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-Copy [this file](https://github.com/apache/couchdb/blob/master/rel/overlay/bin/couchup) into your filesystem
+## Contributing
 
+Bug reports and pull requests are welcome on GitHub at https://github.com/amco/dolly.
 
-make it executable
+## Migrating from CouchDB 1.x to 2.x
 
-`chmod +x couchup.py`
+[Official docs](https://docs.couchdb.org/en/2.3.1/install/index.html)
 
-and run it
+You will need to uninstall the couchdb service with brew:
 
-`./couchup.py -h`
+```bash
+brew services stop couchdb
+brew services uninstall couchdb
+```
 
-You might need to install python 3 and pip3 and the following libs
+Download the application from http://couchdb.apache.org/#download, launch Fauxton, and check your installation.
 
-`pip3 install requests progressbar2`
+Copy [this file](https://github.com/apache/couchdb/blob/master/rel/overlay/bin/couchup) into your filesystem, make it executable, and run it:
 
-move your .couch files into the specified `database_dir` in your [fauxton config](http://127.0.0.1:5984/_utils/#_config/couchdb@localhost)
+```bash
+chmod +x couchup.py
+./couchup.py -h
+```
 
+You might need Python 3, pip3, and:
 
+```bash
+pip3 install requests progressbar2
 ```
-$ ./couchup list           # Shows your unmigrated 1.x databases
-$ ./couchup replicate -a   # Replicates your 1.x DBs to 2.x
-$ ./couchup rebuild -a     # Optional; starts rebuilding your views
-$ ./couchup delete -a      # Deletes your 1.x DBs (careful!)
-$ ./couchup list           # Should show no remaining databases!
+
+Move your `.couch` files into the specified `database_dir` in your [Fauxton config](http://127.0.0.1:5984/_utils/#_config/couchdb@localhost), then:
+
+```bash
+./couchup list           # Shows your unmigrated 1.x databases
+./couchup replicate -a   # Replicates your 1.x DBs to 2.x
+./couchup rebuild -a     # Optional; starts rebuilding your views
+./couchup delete -a      # Deletes your 1.x DBs (careful!)
+./couchup list           # Should show no remaining databases!
 ```

data/lib/dolly/connection.rb

@@ -1,19 +1,17 @@
 # frozen_string_literal: true
 
-require 'curb'
-require 'oj'
 require 'cgi'
 require 'net/http'
 require 'dolly/request_header'
 require 'dolly/exceptions'
 require 'dolly/configuration'
+require 'dolly/curl'
 require 'refinements/string_refinements'
-require 'dolly/framework_helper'
 
 module Dolly
   class Connection
     include Dolly::Configuration
-    include Dolly::FrameworkHelper
+    include Dolly::Curl::ResponseFormatter
     attr_reader :db, :app_env
 
     DEFAULT_HEADER = { 'Content-Type' => 'application/json', 'Accept' => 'application/json' }
@@ -78,7 +76,7 @@ module Dolly
       body        = fetch_body(data)
       uri         = URI("#{base_uri}#{db_resource}")
 
-      conn = curl_method_call(method, uri, body) do |curl|
+      conn = curl_connection.request(method, uri, body) do |curl|
         if env['username'] && !env['username'].empty?
           curl.http_auth_types = :basic
           curl.username = env['username']
@@ -88,11 +86,20 @@ module Dolly
         headers.each { |k, v| curl.headers[k] = v } unless !headers || headers.empty?
       end
 
+      return conn unless conn.is_a?(::Curl::Easy)
+
       response_format(conn, method)
     end
 
     private
 
+    def curl_connection
+      @curl_connection ||= Dolly::Curl::Connection.new(
+        db_name: db_name,
+        reader: Dolly::Curl::Reader.new(self)
+      )
+    end
+
     def fetch_headers(data)
       return unless data.is_a?(Hash)
       Dolly::HeaderRequest.new(data&.delete(:headers))
@@ -105,28 +112,6 @@ module Dolly
       data&.merge!(data&.delete(:query) || {})
     end
 
-    def curl_method_call(method, uri, data, &block)
-      return Curl::Easy.http_head(uri.to_s, &block) if method.to_sym == :head
-      return Curl.delete(uri.to_s, &block) if method.to_sym == :delete
-      return Curl.send(method, uri, data, &block) if method.to_sym == :get
-      Curl.send(method, uri.to_s, data.to_json, &block)
-    end
-
-    def response_format(res, method)
-      raise Dolly::ResourceNotFound if res.status.to_i == 404
-      raise Dolly::ServerError.new(res.status.to_i) if (400..600).include? res.status.to_i
-      return res.header_str if method == :head
-
-      data = Oj.load(res.body_str, symbol_keys: true)
-      return data unless rails?
-      return data.with_indifferent_access if data.is_a?(Hash)
-      data
-    rescue Oj::ParseError
-      res.body_str
-    ensure
-      GC.start if res&.body_str&.length&.to_i > 250000
-    end
-
     def values_to_json hash
       hash.each_with_object({}) { |(k,v), h| h[k] = v.is_a?(Numeric) ? v : v.to_json }
     end

data/lib/dolly/curl/connection.rb

@@ -0,0 +1,150 @@
+# frozen_string_literal: true
+
+require 'curb'
+require 'cgi'
+require 'dolly/curl/stale_connection'
+require 'dolly/curl/write_reconciler'
+
+module Dolly
+  module Curl
+    # Thread-local Curb transport with stale-connection handling.
+    #
+    # Reuses one ::Curl::Easy handle per thread, retries safe reads once, and
+    # delegates ambiguous writes to WriteReconciler.
+    #
+    # Assumes one in-flight request per thread (not safe under fiber-based servers).
+    #
+    # +reader+ must respond to:
+    # - fetch_document(doc_id) -> Hash or nil
+    # - fetch_documents_by_ids(ids) -> Hash keyed by id
+    class Connection
+      MAX_STALE_RETRIES = 1
+      THREAD_STORE_KEY = :dolly_curl_easy
+      READ_ONLY_POST_SUFFIXES = %w[/_find /_all_docs].freeze
+
+      attr_reader :db_name, :reader
+
+      def initialize(db_name:, reader:, reconciler: nil)
+        @db_name = db_name
+        @reader = reader
+        @reconciler = reconciler
+      end
+
+      def request(method, uri, data = nil, retries = 0, &block)
+        perform(method, uri, data, &block)
+      rescue *StaleConnection::ERRORS => error
+        discard_handle
+
+        if safe_to_retry?(method, uri)
+          raise error if retries >= MAX_STALE_RETRIES
+
+          return request(method, uri, data, retries + 1, &block)
+        end
+
+        reconciler.reconcile(method, uri, data, error, &block)
+      end
+
+      def perform(method, uri, data = nil, &block)
+        handle = prepare_handle(uri, &block)
+        dispatch_http(handle, method, data)
+        handle
+      end
+
+      def discard_handle
+        store = Thread.current[THREAD_STORE_KEY]
+        return unless store
+
+        handle = store.delete(self.class)
+        handle&.close
+      rescue StandardError
+        nil
+      end
+
+      def thread_local_handle
+        store = Thread.current[THREAD_STORE_KEY] ||= {}
+        handle = store[self.class]
+        return handle unless handle.nil? || mocked_curl_class?(handle)
+
+        begin
+          handle&.close
+        rescue StandardError
+          nil
+        end
+
+        store[self.class] = ::Curl::Easy.new
+      end
+
+      private
+
+      def mocked_curl_class?(handle)
+        !handle.instance_of?(::Curl::Easy)
+      end
+
+      def reconciler
+        @reconciler ||= WriteReconciler.new(
+          transport: self,
+          reader: reader,
+          db_name: db_name
+        )
+      end
+
+      def prepare_handle(uri)
+        handle = thread_local_handle
+        handle.reset
+        handle.url = uri.to_s
+        handle.headers['Content-Type'] = 'application/json'
+        handle.headers['Accept'] = 'application/json'
+        yield handle if block_given?
+        handle
+      end
+
+      def dispatch_http(handle, method, data)
+        case method.to_sym
+        when :head then handle.http_head
+        when :delete then handle.http_delete
+        when :get then http_get(handle, data)
+        when :post then handle.http_post(payload_json(data))
+        when :put then handle.http_put(payload_json(data))
+        else
+          raise ArgumentError, "unsupported HTTP method: #{method}"
+        end
+      end
+
+      def http_get(handle, data)
+        apply_get_query!(handle, data)
+        handle.http_get
+      end
+
+      def payload_json(data)
+        return data if data.is_a?(String)
+
+        data.to_json
+      end
+
+      def apply_get_query!(handle, data)
+        return if data.nil? || data.empty?
+
+        query = data.map do |key, value|
+          "#{CGI.escape(key.to_s)}=#{CGI.escape(value.to_s)}"
+        end.join('&')
+        separator = handle.url.include?('?') ? '&' : '?'
+        handle.url = "#{handle.url}#{separator}#{query}"
+      end
+
+      def safe_to_retry?(method, uri)
+        case method.to_sym
+        when :get, :head
+          true
+        when :post
+          read_only_post?(uri)
+        else
+          false
+        end
+      end
+
+      def read_only_post?(uri)
+        READ_ONLY_POST_SUFFIXES.any? { |suffix| uri.path.end_with?(suffix) }
+      end
+    end
+  end
+end

data/lib/dolly/curl/document_helper.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require 'cgi'
+require 'oj'
+
+module Dolly
+  module Curl
+    # Document identity, URI parsing, and equality helpers for write reconciliation.
+    class DocumentHelper
+      IGNORED_COMPARE_KEYS = %i[_rev].freeze
+
+      def initialize(db_name:)
+        @db_name = db_name
+      end
+
+      def id_from_uri(uri)
+        prefix = "/#{@db_name}/"
+        path = uri.path
+        return nil unless path.start_with?(prefix)
+
+        rest = path[prefix.length..-1]
+        return nil if rest.nil? || rest.empty?
+        return nil if rest.start_with?('_')
+        return nil if rest.include?('/')
+
+        CGI.unescape(rest)
+      end
+
+      def rev_from_uri(uri)
+        return nil unless uri.query
+
+        CGI.parse(uri.query)['rev']&.first
+      end
+
+      def normalize(data)
+        return data.transform_keys(&:to_sym) if data.is_a?(Hash)
+        return nil if data.nil?
+
+        parsed = data.is_a?(String) ? Oj.load(data, symbol_keys: true) : nil
+        parsed.is_a?(Hash) ? parsed : nil
+      rescue Oj::ParseError
+        nil
+      end
+
+      def matches?(stored, intended)
+        intended.each do |key, expected|
+          key = key.to_sym
+          next if IGNORED_COMPARE_KEYS.include?(key)
+          return false unless value(stored, key) == expected
+        end
+        true
+      end
+
+      def value(doc, key)
+        return doc[key] if doc.key?(key)
+        return doc[key.to_s] if doc.key?(key.to_s)
+
+        nil
+      end
+
+      def id(doc)
+        value(doc, :_id)
+      end
+
+      def revision(doc)
+        value(doc, :_rev)
+      end
+
+      def deleted?(doc)
+        value(doc, :_deleted)
+      end
+
+      def docs_from_bulk_payload(data)
+        payload = normalize(data)
+        return [] unless payload
+
+        Array(payload[:docs]).map { |doc| normalize(doc) }.compact
+      end
+    end
+  end
+end

data/lib/dolly/curl/reader.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'cgi'
+require 'dolly/exceptions'
+
+module Dolly
+  module Curl
+    # Reads CouchDB state for write reconciliation via Connection's public API.
+    class Reader
+      def initialize(connection)
+        @connection = connection
+      end
+
+      def fetch_document(doc_id)
+        @connection.request(:get, CGI.escape(doc_id.to_s))
+      rescue Dolly::ResourceNotFound
+        nil
+      end
+
+      def fetch_documents_by_ids(ids)
+        response = @connection.post('_all_docs', keys: ids, query: { include_docs: true })
+        rows = response[:rows] || []
+        rows.each_with_object({}) do |row, memo|
+          memo[row[:id] || row[:key]] = row
+        end
+      end
+    end
+  end
+end

data/lib/dolly/curl/response_formatter.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require 'oj'
+require 'dolly/framework_helper'
+require 'dolly/exceptions'
+
+module Dolly
+  module Curl
+    module ResponseFormatter
+      include Dolly::FrameworkHelper
+
+      private
+
+      def response_format(res, method)
+        status = res.status.to_i
+        raise Dolly::ResourceNotFound if status == 404
+        raise Dolly::ServerError.new(status) if (400..600).include?(status)
+        return res.header_str if method == :head
+
+        data = Oj.load(res.body_str, symbol_keys: true)
+        return data unless rails?
+        return data.with_indifferent_access if data.is_a?(Hash)
+
+        data
+      rescue Oj::ParseError
+        res.body_str
+      end
+    end
+  end
+end

data/lib/dolly/curl/stale_connection.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+require 'curb'
+
+module Dolly
+  module Curl
+    module StaleConnection
+      ERRORS = [
+        ::Curl::Err::GotNothingError,
+        ::Curl::Err::RecvError,
+        ::Curl::Err::SendError,
+        ::Curl::Err::PartialFileError
+      ].freeze
+    end
+  end
+end