tobox 0.1.5 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c424bc9f0cdf31194c16e9295a37eed99bc143867a08622f909b2605298f15c6
-  data.tar.gz: 7076e13a73ec2b480f66c4be77354c6802fa52112590882ccb95bc284c1fd305
+  metadata.gz: ecf0e9576a8a59834eb6b84eed5c648a2fda3e50b4be49fba068f2bcfc785978
+  data.tar.gz: fb8dc2819591dfe63a177dc1f27b9a947712db00bf451a8f3633d305ee12172b
 SHA512:
-  metadata.gz: f806224e9cccddb59b1417e93343261aec881eff281fb112bc4a39ef7348215194d82c58596b88f89b052fe64bb492a65f2c560ee5ef218073d39d3f84bf4cc1
-  data.tar.gz: 43a66e0d419eb193d1d0dff35c4e86c84da180bd24e7504e0ded248e283ba737492bf05101d8e13b2e5f232dab649865ee943f3263e1d4d7a4cd9d27813594b7
+  metadata.gz: 4b0d7eae10175ce51332e32ad89613196f7b56d01f54670bd78a36af0d411081fda20af93f7c2eee0fd032da84f7c098ad662b6ce00e01b8ab603b6f1f06cc4f
+  data.tar.gz: 6db739a849af63cee915970ea1abc7df4a9d4aff91edb4c76cc4c78273e7231750771638ff0dc477bbb586fc24f0dfab1de793d875cd22bb395c5ff1d3772641

data/CHANGELOG.md

@@ -1,30 +1,72 @@
 ## [Unreleased]
 
-## [0.1.5] - 2002-10-06
+## [0.2.0] - 2022-12-05
+
+### Features
+
+#### Ordered event processing
+
+When the outbox table contains a `:group_id` table (and the producer fills up events with it), then a group of events with the same `:group_id` will be processed one by one, by order of insertion.
+
+```ruby
+# migration
+create_table(:outbox) do
+  column :message_group_id, :integer
+
+# tobox.rb
+message_group_column :group_id
+
+# event production
+DB[:outbox].insert(event_type: "order_created", message_group_id: order.id, ....
+DB[:outbox].insert(event_type: "billing_event_started", message_group_id: order.id, ....
+
+# order_created handled first, billing_event_started only after
+```
+
+#### on_error_worker callback
+
+The config option `on_error_worker { |error| }` gets called when an error happens in a worker **before** events are processed (p.ex. when the database connection becomes unhealthy). You can use it to report such errors to an error reporting system (the `sentry` plugin relies on it).
+
+```ruby
+# tobox.rb
+on_error_worker { |error| Sentry.capture_exception(error, hint: { background: false }) }
+```
+
+### Bugfixes
+
+Thread workers: when errors happen which bring down the workers (such as database becoming unresponsive), workers will be restarted.
+
+## [0.1.6] - 2022-10-06
+
+### Bugfixes
+
+Allow passing datadog options, initialize tracing from plugin.
+
+## [0.1.5] - 2022-10-06
 
 ### Bugfixes
 
 Fixing datadog plugin name.
 
-## [0.1.4] - 2002-10-06
+## [0.1.4] - 2022-10-06
 
 ### Bugfixes
 
 Actual fix for missing datadog constants.
 
-## [0.1.3] - 2002-10-06
+## [0.1.3] - 2022-10-06
 
 ### Bugfixes
 
 Datadog constants unproperly namespaced.
 
-## [0.1.2] - 2002-09-14
+## [0.1.2] - 2022-09-14
 
 ### Bugfixes
 
 Actual fix for foregoing json parsing.
 
-## [0.1.1] - 2002-09-14
+## [0.1.1] - 2022-09-14
 
 ### Chore
 

data/README.md

@@ -1,11 +1,33 @@
 # Tobox: Transactional outbox pattern implementation in ruby
 
 [![Gem Version](https://badge.fury.io/rb/tobox.svg)](http://rubygems.org/gems/tobox)
-[![pipeline status](https://gitlab.com/honeyryderchuck/tobox/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/tobox/pipelines?page=1&scope=all&ref=master)
-[![coverage report](https://gitlab.com/honeyryderchuck/tobox/badges/master/coverage.svg?job=coverage)](https://honeyryderchuck.gitlab.io/tobox/#_AllFiles)
+[![pipeline status](https://gitlab.com/os85/tobox/badges/master/pipeline.svg)](https://gitlab.com/os85/tobox/pipelines?page=1&scope=all&ref=master)
+[![coverage report](https://gitlab.com/os85/tobox/badges/master/coverage.svg?job=coverage)](https://os85.gitlab.io/tobox/#_AllFiles)
 
 Simple, data-first events processing framework based on the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html).
 
+<!-- TOC -->
+
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Configuration](#configuration)
+- [Event](#event)
+- [Features](#features)
+  - [Ordered event processing](#ordered-event-processing)
+- [Plugins](#plugins)
+  - [Zeitwerk](#zeitwerk)
+  - [Sentry](#sentry)
+  - [Datadog](#datadog)
+- [Supported Rubies](#supported-rubies)
+- [Rails support](#rails-support)
+- [Why?](#why)
+- [Development](#development)
+- [Contributing](#contributing)
+
+<!-- /TOC -->
+
+<a id="markdown-requirements" name="requirements"></a>
 ## Requirements
 
 `tobox` requires integration with RDBMS which supports `SKIP LOCKED` functionality. As of today, that's:
@@ -15,6 +37,7 @@ Simple, data-first events processing framework based on the [transactional outbo
 * Oracle
 * Microsoft SQL Server
 
+<a id="markdown-installation" name="installation"></a>
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -22,7 +45,7 @@ Add this line to your application's Gemfile:
 ```ruby
 gem "tobox"
 
-# You'll also need to aadd the right database client gem for the target RDBMS
+# You'll also need to add the right database client gem for the target RDBMS
 # ex, for postgresql:
 #
 # gem "pg"
@@ -37,6 +60,8 @@ Or install it yourself as:
 
     $ gem install tobox
 
+
+<a id="markdown-usage" name="usage"></a>
 ## Usage
 
 1. create the `outbox` table in your application's database:
@@ -80,11 +105,14 @@ end
 on("user_updated") do |event|
   # ...
 end
+on("user_created", "user_updated") do |event|
+  # ...
+end
 ```
 
 3. Start the `tobox` process
 
-```
+```bash
 > bundle exec tobox -C path/to/tobox.rb -r path/to/file_requiring_application_code.rb
 ```
 
@@ -135,11 +163,12 @@ CREATE TRIGGER order_created_outbox_event
   EXECUTE PROCEDURE order_created_outbox_event();
 ```
 
+<a id="markdown-configuration" name="configuration"></a>
 ## Configuration
 
 As mentioned above, configuration can be set in a particular file. The following options are configurable:
 
-### `environment``
+### `environment`
 
 Sets the application environment (either "development" or "production"). Can be set directly, or via `APP_ENV` environment variable (defaults to "development").
 
@@ -228,6 +257,15 @@ callback executed when an exception was raised while processing an event.
 on_error_event { |event, exception| Sentry.capture_exception(exception) }
 ```
 
+### `on_error_worker { |error| }`
+
+callback executed when an exception was raised in the worker, before processing events.
+
+
+```ruby
+on_error_worker { |exception| Sentry.capture_exception(exception) }
+```
+
 ### `message_to_arguments { |event| }`
 
 if exposing raw data to the `on` handlers is not what you'd want, you can always override the behaviour by providing an alternative "before/after fetcher" implementation.
@@ -257,6 +295,11 @@ Overrides the internal logger (an instance of `Logger`).
 
 Overrides the default log level ("info" when in "production" environment, "debug" otherwise).
 
+### group_column
+
+Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
+
+<a id="markdown-event" name="event"></a>
 ## Event
 
 The event is composed of the following properties:
@@ -269,20 +312,57 @@ The event is composed of the following properties:
 
 (*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.)
 
-## Rails support
+<a id="markdown-features" name="features"></a>
+## Features
 
-Rails is supported out of the box by adding the [sequel-activerecord_connection](https://github.com/janko/sequel-activerecord_connection) gem into your Gemfile, and requiring the rails application in the `tobox` cli call:
+There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`.
 
-```bash
-> bundle exec tobox -C path/to/tobox.rb -r path/to/rails_app/config/environment.rb
+<a id="markdown-ordered-event-processing" name="ordered-event-processing"></a>
+### Ordered event processing
+
+By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order.
+
+One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration.
+
+What you have to do is:
+
+1. add a "group id" column to the "outbox" table
+
+```ruby
+create_table(:outbox) do
+  primary_key :id
+  column :group_id, :integer
+  # The type is irrelevant, could also be :string, :uuid...
+  # ..
 ```
 
-In the `tobox` config, you can set the environment:
+2. set the "group_column" configuration
 
 ```ruby
-environment Rails.env
+# in your tobox.rb
+group_column :group_id
+index :group_id
 ```
 
+3. insert related outbox events with the same group id
+
+```ruby
+order = Order.new(
+  item_id: item.id,
+  price: 20_20,
+  currency: "EUR"
+)
+DB.transaction do
+  order.save
+  DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash)
+  DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash)
+end
+
+# "order_created" will be processed first
+# "billing_event_created" will only start processing once "order_created" finishes
+```
+
+<a id="markdown-plugins" name="plugins"></a>
 ## Plugins
 
 `tobox` ships with a very simple plugin system. (TODO: add docs).
@@ -296,6 +376,7 @@ plugin(:plugin_name)
 
 It ships with the following integrations.
 
+<a id="markdown-zeitwerk" name="zeitwerk"></a>
 ### Zeitwerk
 
 (requires the `zeitwerk` gem.)
@@ -310,6 +391,7 @@ zeitwerk_loader do |loader|
 end
 ```
 
+<a id="markdown-sentry" name="sentry"></a>
 ### Sentry
 
 (requires the `sentry-ruby` gem.)
@@ -321,6 +403,7 @@ Plugin for the [sentry](https://github.com/getsentry/sentry-ruby) ruby SDK for e
 plugin(:sentry)
 ```
 
+<a id="markdown-datadog" name="datadog"></a>
 ### Datadog
 
 (requires the `ddtrace` gem.)
@@ -337,10 +420,28 @@ end
 plugin(:datadog)
 ```
 
+<a id="markdown-supported-rubies" name="supported-rubies"></a>
 ## Supported Rubies
 
 All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby.
 
+
+<a id="markdown-rails-support" name="rails-support"></a>
+## Rails support
+
+Rails is supported out of the box by adding the [sequel-activerecord_connection](https://github.com/janko/sequel-activerecord_connection) gem into your Gemfile, and requiring the rails application in the `tobox` cli call:
+
+```bash
+> bundle exec tobox -C path/to/tobox.rb -r path/to/rails_app/config/environment.rb
+```
+
+In the `tobox` config, you can set the environment:
+
+```ruby
+environment Rails.env
+```
+
+<a id="markdown-why" name="why"></a>
 ## Why?
 
 ### Simple and lightweight, framework (and programming language) agnostic
@@ -375,10 +476,12 @@ By using the database as the message broker, `tobox` can rely on good old transa
 
 (The actual processing may change this to "at least once", as issues may happen before the event is successfully deleted from the outbox. Still, "at least once" is acceptable and solvable using idempotency mechanisms).
 
+<a id="markdown-development" name="development"></a>
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
+<a id="markdown-contributing" name="contributing"></a>
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://gitlab.com/honeyryderchuck/tobox.
+Bug reports and pull requests are welcome on GitHub at https://gitlab.com/os85/tobox.

data/lib/tobox/application.rb

@@ -5,10 +5,6 @@ module Tobox
     def initialize(configuration)
       @configuration = configuration
       @running = false
-    end
-
-    def start
-      return if @running
 
       worker = @configuration[:worker]
 
@@ -17,7 +13,12 @@ module Tobox
               when :fiber then FiberPool
               else worker
               end.new(@configuration)
+    end
+
+    def start
+      return if @running
 
+      @pool.start
       @running = true
     end
 

data/lib/tobox/configuration.rb

@@ -17,6 +17,7 @@ module Tobox
       log_level: nil,
       database_uri: nil,
       table: :outbox,
+      group_column: nil,
       max_attempts: 10,
       exponential_retry_factor: 4,
       wait_for_events_delay: 5,
@@ -60,8 +61,10 @@ module Tobox
       freeze
     end
 
-    def on(event, &callback)
-      (@handlers[event.to_sym] ||= []) << callback
+    def on(*events, &callback)
+      events.each do |event|
+        (@handlers[event.to_sym] ||= []) << callback
+      end
       self
     end
 
@@ -80,12 +83,17 @@ module Tobox
       self
     end
 
+    def on_error_worker(&callback)
+      (@lifecycle_events[:error_worker] ||= []) << callback
+      self
+    end
+
     def message_to_arguments(&callback)
       @arguments_handler = callback
       self
     end
 
-    def plugin(plugin, _options = nil, &block)
+    def plugin(plugin, **options, &block)
       raise Error, "Cannot add a plugin to a frozen config" if frozen?
 
       plugin = Plugins.load_plugin(plugin) if plugin.is_a?(Symbol)
@@ -93,11 +101,11 @@ module Tobox
       return if @plugins.include?(plugin)
 
       @plugins << plugin
-      plugin.load_dependencies(self, &block) if plugin.respond_to?(:load_dependencies)
+      plugin.load_dependencies(self, **options, &block) if plugin.respond_to?(:load_dependencies)
 
       extend(plugin::ConfigurationMethods) if defined?(plugin::ConfigurationMethods)
 
-      plugin.configure(self, &block) if plugin.respond_to?(:configure)
+      plugin.configure(self, **options, &block) if plugin.respond_to?(:configure)
     end
 
     def freeze

data/lib/tobox/fetcher.rb

@@ -19,6 +19,7 @@ module Tobox
       @db.loggers << @logger unless @configuration[:environment] == "production"
 
       @table = configuration[:table]
+      @group_column = configuration[:group_column]
       @exponential_retry_factor = configuration[:exponential_retry_factor]
 
       max_attempts = configuration[:max_attempts]
@@ -33,9 +34,6 @@ module Tobox
       @pick_next_sql = @ds.where(Sequel[@table][:attempts] < max_attempts) # filter out exhausted attempts
                           .where(run_at_conds)
                           .order(Sequel.desc(:run_at, nulls: :first), :id)
-                          .for_update
-                          .skip_locked
-                          .limit(1)
 
       @before_event_handlers = Array(@configuration.lifecycle_events[:before_event])
       @after_event_handlers = Array(@configuration.lifecycle_events[:after_event])
@@ -44,8 +42,34 @@ module Tobox
 
     def fetch_events(&blk)
       num_events = 0
-      @db.transaction do
-        event_ids = @pick_next_sql.select_map(:id) # lock starts here
+      @db.transaction(savepoint: false) do
+        if @group_column
+          group = @pick_next_sql.for_update
+                                .skip_locked
+                                .limit(1)
+                                .select(@group_column)
+
+          # get total from a group, to compare to the number of future locked rows.
+          total_from_group = @ds.where(@group_column => group).count
+
+          event_ids = @ds.where(@group_column => group)
+                         .order(Sequel.desc(:run_at, nulls: :first), :id)
+                         .for_update.skip_locked.select_map(:id)
+
+          if event_ids.size != total_from_group
+            # this happens if concurrent workers locked different rows from the same group,
+            # or when new rows from a given group have been inserted after the lock has been
+            # acquired
+            event_ids = []
+          end
+
+          # lock all, process 1
+          event_ids = event_ids[0, 1]
+        else
+          event_ids = @pick_next_sql.for_update
+                                    .skip_locked
+                                    .limit(1).select_map(:id) # lock starts here
+        end
 
         events = nil
         error = nil

data/lib/tobox/plugins/datadog/configuration.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "datadog/tracing/contrib"
 require "datadog/tracing/contrib/configuration/settings"
 require "datadog/tracing/span_operation"
 

data/lib/tobox/plugins/datadog.rb

@@ -88,15 +88,19 @@ module Tobox
           require "uri"
         end
 
-        def configure(config)
+        def configure(config, **datadog_options, &blk)
           event_handler = EventHandler.new(config)
           config.on_before_event(&event_handler.method(:on_start))
           config.on_after_event(&event_handler.method(:on_finish))
           config.on_error_event(&event_handler.method(:on_error))
+          ::Datadog.configure do |c|
+            c.tracing.instrument :tobox, datadog_options
+            yield(c) if blk
+          end
         end
       end
     end
 
-    register_plugin :tobox, Datadog
+    register_plugin :datadog, Datadog
   end
 end

data/lib/tobox/plugins/sentry.rb

@@ -130,6 +130,10 @@ module Tobox
           config.on_after_event(&event_handler.method(:on_finish))
           config.on_error_event(&event_handler.method(:on_error))
 
+          config.on_error_worker do |error|
+            ::Sentry.capture_exception(error, hint: { background: false })
+          end
+
           ::Sentry::Configuration.attr_reader(:tobox)
           ::Sentry::Configuration.add_post_initialization_callback do
             @tobox = Plugins::Sentry::Configuration.new

data/lib/tobox/pool/fiber_pool.rb

@@ -5,12 +5,9 @@ require "fiber_scheduler"
 
 module Tobox
   class FiberPool < Pool
-    class KillError < Interrupt; end
-
     def initialize(_configuration)
       Sequel.extension(:fiber_concurrency)
       super
-      @error_handlers = Array(@configuration.lifecycle_events[:error])
     end
 
     def start
@@ -19,14 +16,7 @@ module Tobox
 
         FiberScheduler do
           @workers.each_with_index do |wk, _idx|
-            Fiber.schedule do
-              wk.work
-            rescue KillError
-              # noop
-            rescue Exception => e # rubocop:disable Lint/RescueException
-              @error_handlers.each { |hd| hd.call(:tobox_error, e) }
-              raise e
-            end
+            Fiber.schedule { do_work(wk) }
           end
         end
       end

data/lib/tobox/pool/threaded_pool.rb

@@ -1,32 +1,22 @@
 # frozen_string_literal: true
 
+require "monitor"
+
 module Tobox
   class ThreadedPool < Pool
-    class KillError < Interrupt; end
-
     def initialize(_configuration)
+      @parent_thread = Thread.main
       @threads = []
+      @threads.extend(MonitorMixin)
       super
-      @error_handlers = Array(@configuration.lifecycle_events[:error])
     end
 
     def start
-      @workers.each_with_index do |wk, idx|
-        th = Thread.start do
-          Thread.current.name = "tobox-worker-#{idx}"
-
-          begin
-            wk.work
-          rescue KillError
-            # noop
-          rescue Exception => e # rubocop:disable Lint/RescueException
-            @error_handlers.each { |hd| hd.call(:tobox_error, e) }
-            raise e
-          end
-
-          @threads.delete(Thread.current)
+      @workers.each do |wk|
+        th = start_thread_worker(wk)
+        @threads.synchronize do
+          @threads << th
         end
-        @threads << th
       end
     end
 
@@ -51,5 +41,30 @@ module Tobox
         th.value # waits
       end
     end
+
+    private
+
+    def start_thread_worker(wrk)
+      Thread.start(wrk) do |worker|
+        Thread.current.name = worker.label
+
+        do_work(worker)
+
+        @threads.synchronize do
+          @threads.delete(Thread.current)
+
+          if worker.finished? && @running
+            idx = @workers.index(worker)
+
+            subst_worker = Worker.new(worker.label, @configuration)
+            @workers[idx] = subst_worker
+            subst_thread = start_thread_worker(subst_worker)
+            @threads << subst_thread
+          end
+          # all workers went down abruply, we need to kill the process.
+          # @parent_thread.raise(Interrupt) if wk.finished? && @threads.empty? && @running
+        end
+      end
+    end
   end
 end

data/lib/tobox/pool.rb

@@ -2,17 +2,39 @@
 
 module Tobox
   class Pool
+    class KillError < Interrupt; end
+
     def initialize(configuration)
       @configuration = configuration
+      @logger = @configuration.default_logger
       @num_workers = configuration[:concurrency]
       @workers = Array.new(@num_workers) do |idx|
         Worker.new("tobox-worker-#{idx}", configuration)
       end
-      start
+      @worker_error_handlers = Array(@configuration.lifecycle_events[:error_worker])
+      @running = true
     end
 
     def stop
+      return unless @running
+
       @workers.each(&:finish!)
+      @running = false
+    end
+
+    def do_work(wrk)
+      wrk.work
+    rescue KillError
+    # noop
+    rescue Exception => e # rubocop:disable Lint/RescueException
+      wrk.finish!
+      @logger.error do
+        "(worker: #{wrk.label}) -> " \
+          "crashed with error\n" \
+          "#{e.class}: #{e.message}\n" \
+          "#{e.backtrace.join("\n")}"
+      end
+      @worker_error_handlers.each { |hd| hd.call(e) }
     end
   end
 

data/lib/tobox/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Tobox
-  VERSION = "0.1.5"
+  VERSION = "0.2.0"
 end

data/lib/tobox/worker.rb

@@ -2,7 +2,10 @@
 
 module Tobox
   class Worker
+    attr_reader :label
+
     def initialize(label, configuration)
+      @label = label
       @wait_for_events_delay = configuration[:wait_for_events_delay]
       @handlers = configuration.handlers || {}
       @fetcher = Fetcher.new(label, configuration)
@@ -13,6 +16,10 @@ module Tobox
       define_singleton_method(:message_to_arguments, &message_to_arguments)
     end
 
+    def finished?
+      @finished
+    end
+
     def finish!
       @finished = true
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tobox
 version: !ruby/object:Gem::Version
-  version: 0.1.5
+  version: 0.2.0
 platform: ruby
 authors:
 - HoneyryderChuck
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-10-06 00:00:00.000000000 Z
+date: 2022-12-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sequel
@@ -52,15 +52,15 @@ files:
 - lib/tobox/pool/threaded_pool.rb
 - lib/tobox/version.rb
 - lib/tobox/worker.rb
-homepage: https://gitlab.com/honeyryderchuck/tobox
+homepage: https://gitlab.com/os85/tobox
 licenses: []
 metadata:
-  homepage_uri: https://gitlab.com/honeyryderchuck/tobox
+  homepage_uri: https://gitlab.com/os85/tobox
   allowed_push_host: https://rubygems.org
-  source_code_uri: https://gitlab.com/honeyryderchuck/tobox
-  bug_tracker_uri: https://gitlab.com/honeyryderchuck/tobox/issues
-  documentation_uri: https://gitlab.com/honeyryderchuck/tobox
-  changelog_uri: https://gitlab.com/honeyryderchuck/tobox/-/blob/master/CHANGELOG.md
+  source_code_uri: https://gitlab.com/os85/tobox
+  bug_tracker_uri: https://gitlab.com/os85/tobox/issues
+  documentation_uri: https://gitlab.com/os85/tobox
+  changelog_uri: https://gitlab.com/os85/tobox/-/blob/master/CHANGELOG.md
   rubygems_mfa_required: 'true'
 post_install_message:
 rdoc_options: []