cache_sweeper 0.1.1 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +21 -0
- data/README.md +125 -498
- data/lib/cache_sweeper/base.rb +11 -24
- data/lib/cache_sweeper/batch.rb +68 -0
- data/lib/cache_sweeper/callbacks.rb +13 -0
- data/lib/cache_sweeper/configuration.rb +53 -0
- data/lib/cache_sweeper/delete_job.rb +9 -0
- data/lib/cache_sweeper/delivery.rb +36 -0
- data/lib/cache_sweeper/dispatcher.rb +27 -0
- data/lib/cache_sweeper/executor_hook.rb +15 -0
- data/lib/cache_sweeper/key.rb +22 -0
- data/lib/cache_sweeper/log_subscriber.rb +25 -0
- data/lib/cache_sweeper/railtie.rb +21 -6
- data/lib/cache_sweeper/registry.rb +28 -0
- data/lib/cache_sweeper/rule.rb +77 -0
- data/lib/cache_sweeper/version.rb +3 -1
- data/lib/cache_sweeper.rb +27 -85
- metadata +66 -33
- data/CODE_OF_CONDUCT.md +0 -74
- data/lib/cache_sweeper/async_worker.rb +0 -114
- data/lib/cache_sweeper/dsl.rb +0 -32
- data/lib/cache_sweeper/flush_middleware.rb +0 -147
- data/lib/cache_sweeper/loader.rb +0 -360
- data/lib/cache_sweeper/logger.rb +0 -104
- data/lib/tasks/cache_sweeper.rake +0 -4
data/README.md
CHANGED
|
@@ -1,580 +1,207 @@
|
|
|
1
1
|
# CacheSweeper
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
- [Features](#features)
|
|
8
|
-
- [Installation](#installation)
|
|
9
|
-
- [Quick Start](#quick-start)
|
|
10
|
-
- [Configuration](#configuration)
|
|
11
|
-
- [Usage Examples](#usage-examples)
|
|
12
|
-
- [API Reference](#api-reference)
|
|
13
|
-
- [Logging](#logging)
|
|
14
|
-
- [Middleware](#middleware)
|
|
15
|
-
- [Testing](#testing)
|
|
16
|
-
- [Troubleshooting](#troubleshooting)
|
|
17
|
-
- [Contributing](#contributing)
|
|
18
|
-
- [License](#license)
|
|
19
|
-
|
|
20
|
-
## Features
|
|
21
|
-
|
|
22
|
-
- **Rule-based cache invalidation**: Define what changes should trigger cache clearing using a simple DSL
|
|
23
|
-
- **Flexible triggering**: Choose between instant cache deletion or request-level batching
|
|
24
|
-
- **Async processing**: Offload cache deletion to Sidekiq for scalability
|
|
25
|
-
- **Association support**: Invalidate cache for associated models when their attributes change
|
|
26
|
-
- **Multi-level configuration**: Control behavior globally, per-sweeper, or per-rule
|
|
27
|
-
- **Comprehensive logging**: Detailed logging for debugging with configurable levels
|
|
28
|
-
- **Performance monitoring**: Built-in performance logging for cache operations and timing
|
|
29
|
-
- **Error tracking**: Comprehensive error logging with context and stack traces
|
|
30
|
-
- **Clean model code**: Keep cache logic out of your models
|
|
31
|
-
- **Thread-safe**: Uses RequestStore for reliable multi-threaded operation
|
|
32
|
-
- **Efficient batch deletion**: Uses `Rails.cache.delete_multi` for optimal performance
|
|
33
|
-
- **Configurable batch sizes**: Control how many keys are deleted in each batch
|
|
34
|
-
|
|
35
|
-
## Installation
|
|
36
|
-
|
|
37
|
-
Add this line to your application's Gemfile:
|
|
38
|
-
|
|
39
|
-
```ruby
|
|
40
|
-
gem 'cache_sweeper'
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
Then run:
|
|
44
|
-
|
|
45
|
-
```sh
|
|
46
|
-
bundle install
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
### Dependencies
|
|
50
|
-
|
|
51
|
-
- **Rails**: 5.0+ (tested with Rails 6.x and 7.x)
|
|
52
|
-
- **Sidekiq**: Required for async cache deletion
|
|
53
|
-
- **RequestStore**: For thread-safe request-level storage
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
## Quick Start
|
|
57
|
-
|
|
58
|
-
### 1. Configure the Gem
|
|
59
|
-
|
|
60
|
-
Create `config/initializers/cache_sweeper.rb`:
|
|
61
|
-
|
|
62
|
-
```ruby
|
|
63
|
-
# Logging configuration
|
|
64
|
-
CacheSweeper.logger = Rails.logger
|
|
65
|
-
CacheSweeper.log_level = :info # :debug, :info, :warn, :error
|
|
66
|
-
|
|
67
|
-
# Global cache invalidation configuration (optional)
|
|
68
|
-
CacheSweeper.trigger = :request # :instant or :request
|
|
69
|
-
CacheSweeper.mode = :async # :async or :inline
|
|
70
|
-
CacheSweeper.queue = :low # Sidekiq queue name
|
|
71
|
-
CacheSweeper.sidekiq_options = { retry: false }
|
|
72
|
-
CacheSweeper.delete_multi_batch_size = 100 # Batch size for efficient cache deletion
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
### 2. Add Middleware (if using request-level batching)
|
|
76
|
-
|
|
77
|
-
Add to `config/application.rb`:
|
|
3
|
+
Declarative cache invalidation for ActiveRecord. Describe which model changes invalidate which cache keys in
|
|
4
|
+
dedicated sweeper classes, and CacheSweeper deletes them after the transaction commits: inline, batched per
|
|
5
|
+
request or job, or in the background through ActiveJob.
|
|
78
6
|
|
|
79
7
|
```ruby
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
8
|
+
class CatalogSweeper < CacheSweeper::Base
|
|
9
|
+
watch Product, only: %i[name price slug] do |product|
|
|
10
|
+
["products/#{product.slug}", "products/#{product.slug_before_last_save}", "products/index"]
|
|
11
|
+
end
|
|
84
12
|
|
|
85
|
-
|
|
13
|
+
watch Variant, only: :price, on: :update do |variant|
|
|
14
|
+
product_keys(variant.product)
|
|
15
|
+
end
|
|
86
16
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
sweeper_options trigger: :request, mode: :async, queue: :products
|
|
17
|
+
watch Review, on: %i[create destroy], if: :published? do |review|
|
|
18
|
+
product_keys(review.product)
|
|
19
|
+
end
|
|
91
20
|
|
|
92
|
-
|
|
93
|
-
watch attributes: [:name, :price], keys: ->(product) { ["product:#{product.id}"] }
|
|
21
|
+
private
|
|
94
22
|
|
|
95
|
-
|
|
96
|
-
|
|
23
|
+
def product_keys(product)
|
|
24
|
+
["products/#{product.slug}", [product, :card]]
|
|
25
|
+
end
|
|
97
26
|
end
|
|
98
27
|
```
|
|
99
28
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
```sh
|
|
103
|
-
bundle exec sidekiq
|
|
104
|
-
```
|
|
105
|
-
|
|
106
|
-
That's it! Your cache will now be automatically invalidated when products change.
|
|
107
|
-
|
|
108
|
-
## Configuration
|
|
109
|
-
|
|
110
|
-
### Global Configuration
|
|
29
|
+
## Requirements
|
|
111
30
|
|
|
112
|
-
|
|
31
|
+
- Ruby 3.3+
|
|
32
|
+
- Rails 7.1+ (ActiveRecord, ActiveJob, ActiveSupport)
|
|
113
33
|
|
|
114
|
-
|
|
115
|
-
# Logging configuration
|
|
116
|
-
CacheSweeper.logger = Rails.logger
|
|
117
|
-
CacheSweeper.log_level = :info # :debug, :info, :warn, :error
|
|
118
|
-
|
|
119
|
-
# Cache invalidation configuration (optional - has sensible defaults)
|
|
120
|
-
CacheSweeper.trigger = :request # :instant or :request
|
|
121
|
-
CacheSweeper.mode = :async # :async or :inline
|
|
122
|
-
CacheSweeper.queue = :low # Sidekiq queue name
|
|
123
|
-
CacheSweeper.sidekiq_options = { retry: false }
|
|
124
|
-
CacheSweeper.delete_multi_batch_size = 100 # Batch size for efficient cache deletion
|
|
125
|
-
```
|
|
126
|
-
|
|
127
|
-
### Sweeper-Level Configuration
|
|
128
|
-
|
|
129
|
-
Use the `sweeper_options` DSL for clean sweeper configuration:
|
|
34
|
+
## Installation
|
|
130
35
|
|
|
131
36
|
```ruby
|
|
132
|
-
|
|
133
|
-
sweeper_options trigger: :request, mode: :async, queue: :orders
|
|
134
|
-
# ... watch rules
|
|
135
|
-
end
|
|
37
|
+
gem "cache_sweeper"
|
|
136
38
|
```
|
|
137
39
|
|
|
138
|
-
|
|
40
|
+
Put sweepers in `app/sweepers`. They are loaded at boot and reloaded with the rest of your code in development.
|
|
139
41
|
|
|
140
|
-
|
|
42
|
+
## Watching models
|
|
141
43
|
|
|
142
44
|
```ruby
|
|
143
|
-
|
|
144
|
-
#
|
|
145
|
-
watch attributes: [:name], keys: ->(obj) { ["instant:#{obj.id}"] },
|
|
146
|
-
trigger: :instant, mode: :inline
|
|
147
|
-
|
|
148
|
-
# Async processing for less critical data
|
|
149
|
-
watch attributes: [:description], keys: ->(obj) { ["async:#{obj.id}"] },
|
|
150
|
-
trigger: :request, mode: :async, queue: :background
|
|
45
|
+
watch Model, on: %i[create update destroy], only: nil, if: nil, unless: nil, delivery: nil, queue: nil do |record|
|
|
46
|
+
# return one cache key or an array of keys
|
|
151
47
|
end
|
|
152
48
|
```
|
|
153
49
|
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
### Configuration Options
|
|
50
|
+
| Option | Meaning |
|
|
51
|
+
|------------|-----------------------------------------------------------------------------------------------|
|
|
52
|
+
| `on` | Events that trigger the sweep. Defaults to all three. |
|
|
53
|
+
| `only` | Attributes whose change triggers an `update` sweep. Creates and destroys always sweep. |
|
|
54
|
+
| `if` | Symbol (method on the record) or Proc (receives the record, runs on the sweeper) to require. |
|
|
55
|
+
| `unless` | Same as `if`, inverted. |
|
|
56
|
+
| `delivery` | Overrides the sweeper's delivery for this rule. |
|
|
57
|
+
| `queue` | ActiveJob queue for `:async` delivery. |
|
|
163
58
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
- **`sidekiq_options`**: Hash of Sidekiq options (e.g., `{ retry: false, backtrace: true }`)
|
|
168
|
-
- **`delete_multi_batch_size`**: Number of keys to delete in each batch (default: 100)
|
|
59
|
+
A sweeper can watch any number of models, and any number of sweepers can watch the same model, so group them by
|
|
60
|
+
what they cache rather than by model. The block and Proc conditions run on a sweeper instance, so private helper
|
|
61
|
+
methods are available to them. Sweeps run in `after_commit`, so rolled-back changes never invalidate anything.
|
|
169
62
|
|
|
170
|
-
|
|
63
|
+
### Keys
|
|
171
64
|
|
|
172
|
-
|
|
65
|
+
The block returns a single key or an array of keys. Keys are expanded exactly like `Rails.cache` expands them,
|
|
66
|
+
so anything you can pass to `Rails.cache.fetch` works, including records and composite keys. Nest composite keys
|
|
67
|
+
inside the returned array:
|
|
173
68
|
|
|
174
69
|
```ruby
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
watch attributes: [:name, :price], keys: ->(product) { ["product:#{product.id}"] }
|
|
70
|
+
watch Product do |product|
|
|
71
|
+
["products/index", [product, :card], { locale: I18n.locale, page: :home }]
|
|
178
72
|
end
|
|
179
73
|
```
|
|
180
74
|
|
|
181
|
-
|
|
75
|
+
`nil` and blank keys are dropped and duplicates removed. Use Rails' `*_before_last_save` readers to sweep keys
|
|
76
|
+
built from a previous value, such as an old slug.
|
|
182
77
|
|
|
183
|
-
|
|
184
|
-
# app/cache_sweepers/package_sweeper.rb
|
|
185
|
-
class PackageSweeper < CacheSweeper::Base
|
|
186
|
-
# Clear cache when package name changes
|
|
187
|
-
watch attributes: [:name], keys: ->(package) { ["package:#{package.id}"] }
|
|
188
|
-
|
|
189
|
-
# Clear cache when associated products change
|
|
190
|
-
watch :products, attributes: [:name], keys: ->(product) {
|
|
191
|
-
product.packages.map { |pkg| "package:#{pkg.id}" }
|
|
192
|
-
}
|
|
193
|
-
end
|
|
194
|
-
```
|
|
78
|
+
## Delivery
|
|
195
79
|
|
|
196
|
-
|
|
80
|
+
| Mode | When keys are deleted |
|
|
81
|
+
|------------|----------------------------------------------------------------------------------------------------|
|
|
82
|
+
| `:inline` | Immediately after commit. Default. |
|
|
83
|
+
| `:batched` | At the end of the current request or job, deduplicated into one `delete_multi`. |
|
|
84
|
+
| `:async` | At the end of the current request or job, enqueued as `CacheSweeper::DeleteJob`s. |
|
|
197
85
|
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
# Clear cache with custom condition
|
|
205
|
-
watch attributes: [:last_login_at], keys: ->(user) { ["users:active"] },
|
|
206
|
-
if: ->(user) { user.last_login_at_changed? && user.last_login_at > 1.day.ago }
|
|
207
|
-
end
|
|
208
|
-
```
|
|
209
|
-
|
|
210
|
-
### Mixed Configuration Sweeper
|
|
86
|
+
`:inline` is the default because it is the only mode where code running after the commit, including the rest of
|
|
87
|
+
the same request, never reads a stale entry. `:batched` and `:async` trade that for fewer cache round trips: for
|
|
88
|
+
a request, the keys are deleted after the response has been sent, so a write followed by a cached read in the
|
|
89
|
+
same request, or a request arriving in that gap, sees the old value. Opt into them per sweeper or per rule where
|
|
90
|
+
that is acceptable.
|
|
211
91
|
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
sweeper_options trigger: :request, mode: :async, queue: :orders
|
|
217
|
-
|
|
218
|
-
# Critical data - instant deletion
|
|
219
|
-
watch attributes: [:status], keys: ->(order) { ["order:#{order.id}"] },
|
|
220
|
-
trigger: :instant, mode: :inline
|
|
221
|
-
|
|
222
|
-
# Less critical data - async processing
|
|
223
|
-
watch attributes: [:notes], keys: ->(order) { ["order:#{order.id}:notes"] },
|
|
224
|
-
trigger: :request, mode: :async, queue: :background
|
|
225
|
-
|
|
226
|
-
# Association changes
|
|
227
|
-
watch :order_items, attributes: [:quantity, :price], keys: ->(order_item) {
|
|
228
|
-
["order:#{order_item.order_id}", "order:#{order_item.order_id}:total"]
|
|
229
|
-
}
|
|
230
|
-
end
|
|
231
|
-
```
|
|
92
|
+
Batching hooks into the Rails executor, which wraps web requests, ActiveJob and Sidekiq jobs, and ActionCable.
|
|
93
|
+
Pending keys are flushed even when the request raises, and early once `max_pending_keys` are waiting, so a
|
|
94
|
+
long job updating many records deletes as it goes instead of holding every key until it ends. Async keys are
|
|
95
|
+
split into jobs of `batch_size` keys and enqueued in bulk.
|
|
232
96
|
|
|
233
|
-
|
|
97
|
+
`rails runner` wraps the whole script in the executor, so a runner script is one unit of work: its keys are
|
|
98
|
+
deleted when it finishes, or every `max_pending_keys` keys. Outside an executor (console, rake tasks) `:batched`
|
|
99
|
+
deletes and `:async` enqueues straight away. To batch a console session or rake task, wrap it:
|
|
234
100
|
|
|
235
101
|
```ruby
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
# Only clear cache on create and destroy, not update
|
|
239
|
-
watch attributes: [:message], keys: ->(notification) { ["notifications:count"] },
|
|
240
|
-
on: [:create, :destroy]
|
|
241
|
-
|
|
242
|
-
# Use before_commit instead of after_commit
|
|
243
|
-
watch attributes: [:read_at], keys: ->(notification) { ["user:#{notification.user_id}:unread_count"] },
|
|
244
|
-
callback: :before_commit
|
|
102
|
+
Rails.application.executor.wrap do
|
|
103
|
+
Product.find_each { |product| product.update!(price: product.price * 1.1) }
|
|
245
104
|
end
|
|
246
105
|
```
|
|
247
106
|
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
### Sweeper DSL
|
|
251
|
-
|
|
252
|
-
#### `watch(association = nil, **options)`
|
|
253
|
-
|
|
254
|
-
Define cache invalidation rules.
|
|
255
|
-
|
|
256
|
-
**Parameters:**
|
|
257
|
-
|
|
258
|
-
- **`association`** (optional): Association name to watch (e.g., `:products`, `:order_items`)
|
|
259
|
-
- **`attributes`**: Array of attributes to watch for changes (e.g., `[:name, :price]`)
|
|
260
|
-
- **`keys`**: Proc or array of cache keys to invalidate
|
|
261
|
-
- **`if`**: Proc or method name for conditional invalidation
|
|
262
|
-
- **`trigger`**: `:instant` or `:request` (per rule)
|
|
263
|
-
- **`mode`**: `:async` or `:inline` (per rule)
|
|
264
|
-
- **`queue`**: Sidekiq queue name (per rule)
|
|
265
|
-
- **`sidekiq_options`**: Hash of Sidekiq options (per rule)
|
|
266
|
-
- **`callback`**: Callback type (`:after_commit`, `:before_commit`, etc.)
|
|
267
|
-
- **`on`**: Events to watch (`[:create, :update, :destroy]`)
|
|
268
|
-
|
|
269
|
-
**Examples:**
|
|
107
|
+
Set the delivery per sweeper or per rule:
|
|
270
108
|
|
|
271
109
|
```ruby
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
# Association watching
|
|
276
|
-
watch :products, attributes: [:name], keys: ->(product) { ["product:#{product.id}"] }
|
|
277
|
-
|
|
278
|
-
# Conditional invalidation
|
|
279
|
-
watch attributes: [:status], keys: ->(obj) { ["key"] },
|
|
280
|
-
if: ->(obj) { obj.status == 'published' }
|
|
110
|
+
class SearchSweeper < CacheSweeper::Base
|
|
111
|
+
delivery :async, queue: :low
|
|
281
112
|
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
```
|
|
286
|
-
|
|
287
|
-
#### `sweeper_options(**options)`
|
|
288
|
-
|
|
289
|
-
Configure sweeper-level behavior.
|
|
290
|
-
|
|
291
|
-
**Parameters:**
|
|
292
|
-
|
|
293
|
-
- **`trigger`**: `:instant` or `:request`
|
|
294
|
-
- **`mode`**: `:async` or `:inline`
|
|
295
|
-
- **`queue`**: Sidekiq queue name
|
|
296
|
-
- **`sidekiq_options`**: Hash of Sidekiq options
|
|
297
|
-
|
|
298
|
-
**Example:**
|
|
113
|
+
watch Product, only: :name do |product|
|
|
114
|
+
"search/#{product.category_id}"
|
|
115
|
+
end
|
|
299
116
|
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
117
|
+
watch Product, only: :price, delivery: :inline do |product|
|
|
118
|
+
"prices/#{product.id}"
|
|
119
|
+
end
|
|
303
120
|
end
|
|
304
121
|
```
|
|
305
122
|
|
|
306
|
-
|
|
123
|
+
Rule options win over sweeper options, which win over the global configuration.
|
|
307
124
|
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
Set the logger for cache actions.
|
|
311
|
-
|
|
312
|
-
#### `CacheSweeper.log_level = level`
|
|
313
|
-
|
|
314
|
-
Set minimum log level (`:debug`, `:info`, `:warn`, `:error`).
|
|
315
|
-
|
|
316
|
-
#### Global Configuration Attributes
|
|
317
|
-
|
|
318
|
-
Configure global cache invalidation behavior using direct attribute assignment:
|
|
319
|
-
|
|
320
|
-
**Example:**
|
|
321
|
-
|
|
322
|
-
```ruby
|
|
323
|
-
CacheSweeper.trigger = :request
|
|
324
|
-
CacheSweeper.mode = :async
|
|
325
|
-
CacheSweeper.queue = :low
|
|
326
|
-
CacheSweeper.sidekiq_options = { retry: false }
|
|
327
|
-
```
|
|
328
|
-
|
|
329
|
-
## Logging
|
|
330
|
-
|
|
331
|
-
The gem provides comprehensive logging to help debug cache invalidation issues.
|
|
332
|
-
|
|
333
|
-
### Log Levels
|
|
334
|
-
|
|
335
|
-
- **`:debug`** - All logging enabled (initialization, rule execution, performance, cache operations, async jobs, middleware)
|
|
336
|
-
- **`:info`** - Important events (initialization, cache operations, async jobs, middleware)
|
|
337
|
-
- **`:warn`** - Warnings and errors only
|
|
338
|
-
- **`:error`** - Errors only
|
|
339
|
-
|
|
340
|
-
### Default Log Levels
|
|
341
|
-
|
|
342
|
-
- **Development**: `:debug` (all logging enabled)
|
|
343
|
-
- **Production**: `:warn` (warnings and errors only)
|
|
344
|
-
- **Other environments**: `:info`
|
|
345
|
-
|
|
346
|
-
### Log Output
|
|
347
|
-
|
|
348
|
-
Log output includes:
|
|
349
|
-
|
|
350
|
-
- **Initialization**: Sweeper loading and model attachment
|
|
351
|
-
- **Rule execution**: Which rules are triggered, condition evaluation, cache key generation
|
|
352
|
-
- **Performance**: Timing for cache operations and rule execution
|
|
353
|
-
- **Cache operations**: Cache key invalidation details
|
|
354
|
-
- **Async jobs**: Job scheduling and execution status
|
|
355
|
-
- **Middleware**: Request-level batching and flushing
|
|
356
|
-
- **Errors**: Detailed error information with context and stack traces
|
|
357
|
-
|
|
358
|
-
### Example Log Output
|
|
359
|
-
|
|
360
|
-
```
|
|
361
|
-
[CacheSweeper] [2024-01-15 10:30:45.123] [INFO] Initialization: Processing sweeper: ProductSweeper
|
|
362
|
-
[CacheSweeper] [2024-01-15 10:30:45.124] [DEBUG] Rule execution: ProductSweeper -> Product#123
|
|
363
|
-
[CacheSweeper] [2024-01-15 10:30:45.125] [INFO] Cache operations: Deleted instantly: product:123
|
|
364
|
-
[CacheSweeper] [2024-01-15 10:30:45.126] [DEBUG] Performance: cache_invalidation took 2.456ms
|
|
365
|
-
```
|
|
366
|
-
|
|
367
|
-
### Debugging Configuration
|
|
125
|
+
## Configuration
|
|
368
126
|
|
|
369
127
|
```ruby
|
|
370
|
-
#
|
|
371
|
-
CacheSweeper.
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
#
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
128
|
+
# config/initializers/cache_sweeper.rb
|
|
129
|
+
CacheSweeper.configure do |config|
|
|
130
|
+
config.cache_store = Rails.cache # any ActiveSupport::Cache::Store, defaults to Rails.cache
|
|
131
|
+
config.delivery = :inline # :inline, :batched or :async
|
|
132
|
+
config.queue = :default # ActiveJob queue for :async
|
|
133
|
+
config.batch_size = 100 # keys per delete_multi call and per DeleteJob
|
|
134
|
+
config.max_pending_keys = 1_000 # flush a request's or job's batch early at this size
|
|
135
|
+
config.raise_errors = Rails.env.local? # raise sweep errors instead of reporting them
|
|
136
|
+
config.force_inline = false # delete inline regardless of declared delivery
|
|
137
|
+
config.sweeper_paths = ["app/sweepers"]
|
|
138
|
+
end
|
|
378
139
|
```
|
|
379
140
|
|
|
380
|
-
##
|
|
381
|
-
|
|
382
|
-
The `CacheSweeperFlushMiddleware` handles request-level batching. It automatically flushes all pending cache keys at the end of each request.
|
|
383
|
-
|
|
384
|
-
### Setup
|
|
385
|
-
|
|
386
|
-
Add to `config/application.rb`:
|
|
141
|
+
## Deleting manually
|
|
387
142
|
|
|
388
143
|
```ruby
|
|
389
|
-
|
|
144
|
+
CacheSweeper.delete("products/index", [product, :card])
|
|
390
145
|
```
|
|
391
146
|
|
|
392
|
-
|
|
147
|
+
## Errors
|
|
393
148
|
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
149
|
+
A failing key block or cache delete never breaks the request that committed the change. Errors go to
|
|
150
|
+
`Rails.error` (`ActiveSupport.error_reporter`) with `source: "cache_sweeper"` and are logged. With `raise_errors`
|
|
151
|
+
enabled, the default in development and test, they are raised instead. `DeleteJob` always raises so your queue
|
|
152
|
+
adapter can retry it.
|
|
397
153
|
|
|
398
|
-
|
|
154
|
+
## Instrumentation
|
|
399
155
|
|
|
400
|
-
|
|
156
|
+
| Event | Payload |
|
|
157
|
+
|---------------------------|--------------------------------------------------|
|
|
158
|
+
| `sweep.cache_sweeper` | `sweeper`, `record`, `event`, `keys`, `delivery` |
|
|
159
|
+
| `delete.cache_sweeper` | `keys` |
|
|
160
|
+
| `failure.cache_sweeper` | `error` and context |
|
|
401
161
|
|
|
402
|
-
|
|
403
|
-
# config/application.rb
|
|
404
|
-
config.middleware.use SomeOtherMiddleware
|
|
405
|
-
config.middleware.use CacheSweeperFlushMiddleware
|
|
406
|
-
```
|
|
162
|
+
Sweeps and deletes are logged at debug level and failures at error level.
|
|
407
163
|
|
|
408
164
|
## Testing
|
|
409
165
|
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
You can test sweepers using standard Rails/ActiveRecord test frameworks:
|
|
166
|
+
Requests and jobs in your test suite run inside the executor, so sweepers using `:batched` or `:async` only
|
|
167
|
+
sweep when the test's unit of work finishes. Force inline deletion in tests that assert on the cache:
|
|
413
168
|
|
|
414
169
|
```ruby
|
|
415
|
-
#
|
|
416
|
-
|
|
417
|
-
test "clears cache when product name changes" do
|
|
418
|
-
product = Product.create!(name: "Original Name")
|
|
419
|
-
|
|
420
|
-
# Mock cache
|
|
421
|
-
Rails.cache.expects(:delete).with("product:#{product.id}")
|
|
422
|
-
|
|
423
|
-
product.update!(name: "New Name")
|
|
424
|
-
end
|
|
425
|
-
end
|
|
170
|
+
# config/environments/test.rb
|
|
171
|
+
config.after_initialize { CacheSweeper.config.force_inline = true }
|
|
426
172
|
```
|
|
427
173
|
|
|
428
|
-
|
|
174
|
+
## Limitations
|
|
429
175
|
|
|
430
|
-
|
|
176
|
+
Sweeps rely on `after_commit`, so bulk operations that skip callbacks (`update_all`, `delete_all`, `insert_all`,
|
|
177
|
+
`upsert_all`) don't sweep. `only:` reads `saved_changes`, which reflects the record's last save within the
|
|
178
|
+
transaction.
|
|
431
179
|
|
|
432
|
-
|
|
433
|
-
# test/sweepers/async_sweeper_test.rb
|
|
434
|
-
class AsyncSweeperTest < ActiveSupport::TestCase
|
|
435
|
-
test "schedules async job for cache deletion" do
|
|
436
|
-
# Stub Sidekiq worker
|
|
437
|
-
CacheSweeper::AsyncWorker.expects(:perform_async).with(["key1", "key2"])
|
|
438
|
-
|
|
439
|
-
# Trigger the change
|
|
440
|
-
product = Product.create!(name: "Test Product")
|
|
441
|
-
end
|
|
442
|
-
end
|
|
443
|
-
```
|
|
444
|
-
|
|
445
|
-
### Testing with Sidekiq
|
|
446
|
-
|
|
447
|
-
For integration tests with Sidekiq:
|
|
180
|
+
## Development
|
|
448
181
|
|
|
449
|
-
```
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
product = Product.create!(name: "Test Product")
|
|
456
|
-
# Cache should be cleared synchronously
|
|
457
|
-
end
|
|
458
|
-
end
|
|
459
|
-
end
|
|
182
|
+
```sh
|
|
183
|
+
bundle install
|
|
184
|
+
bundle exec rake # rubocop, both suites, 100% line and branch coverage
|
|
185
|
+
bundle exec rspec spec/unit # ActiveRecord without Rails
|
|
186
|
+
bundle exec rspec spec/rails # Railtie, executor and reloading against a dummy app
|
|
187
|
+
RAILS_VERSION=7.2 bundle update && bundle exec rake spec
|
|
460
188
|
```
|
|
461
189
|
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
190
|
+
The two suites run in separate processes so the unit suite proves the gem works without Rails loaded. Their
|
|
191
|
+
coverage is merged before the threshold is enforced. The store specs run against Redis and Memcached when they
|
|
192
|
+
are reachable on their default ports, and CI always runs them.
|
|
465
193
|
|
|
466
|
-
|
|
194
|
+
CI also runs RuboCop, `bundle-audit` and a strict `gem build`; CodeQL scans the code weekly and on every PR.
|
|
467
195
|
|
|
468
|
-
|
|
196
|
+
### Releasing
|
|
469
197
|
|
|
470
|
-
|
|
198
|
+
Bump `CacheSweeper::VERSION`, update the CHANGELOG, then tag and push. The Release workflow runs the suite and
|
|
199
|
+
publishes to RubyGems through trusted publishing:
|
|
471
200
|
|
|
472
201
|
```sh
|
|
473
|
-
|
|
202
|
+
git tag v1.0.0 && git push origin v1.0.0
|
|
474
203
|
```
|
|
475
204
|
|
|
476
|
-
#### Sweepers Not Loading
|
|
477
|
-
|
|
478
|
-
**Problem**: Sweepers aren't being loaded or attached to models.
|
|
479
|
-
|
|
480
|
-
**Solution**:
|
|
481
|
-
1. Ensure sweepers are in `app/cache_sweepers/` directory
|
|
482
|
-
2. Ensure sweepers inherit from `CacheSweeper::Base`
|
|
483
|
-
3. Check that the sweeper files are named `*_sweeper.rb`
|
|
484
|
-
|
|
485
|
-
#### Cache Keys Not Being Cleared
|
|
486
|
-
|
|
487
|
-
**Problem**: Cache keys aren't being invalidated when models change.
|
|
488
|
-
|
|
489
|
-
**Solution**:
|
|
490
|
-
1. Enable debug logging: `CacheSweeper.log_level = :debug`
|
|
491
|
-
2. Check that the correct attributes are being watched
|
|
492
|
-
3. Verify cache key generation logic
|
|
493
|
-
4. Ensure the model callbacks are being triggered
|
|
494
|
-
|
|
495
|
-
#### Middleware Not Flushing
|
|
496
|
-
|
|
497
|
-
**Problem**: Request-level batching isn't flushing at the end of requests.
|
|
498
|
-
|
|
499
|
-
**Solution**:
|
|
500
|
-
1. Ensure `CacheSweeperFlushMiddleware` is added to the middleware stack
|
|
501
|
-
2. Check middleware order in `config/application.rb`
|
|
502
|
-
3. Verify that `trigger: :request` is being used
|
|
503
|
-
|
|
504
|
-
### Debugging Steps
|
|
505
|
-
|
|
506
|
-
1. **Enable comprehensive logging**:
|
|
507
|
-
```ruby
|
|
508
|
-
CacheSweeper.logger = Rails.logger
|
|
509
|
-
CacheSweeper.log_level = :debug
|
|
510
|
-
```
|
|
511
|
-
|
|
512
|
-
2. **Check sweeper loading**:
|
|
513
|
-
```ruby
|
|
514
|
-
# In Rails console
|
|
515
|
-
CacheSweeper::Base.descendants
|
|
516
|
-
```
|
|
517
|
-
|
|
518
|
-
3. **Verify model callbacks**:
|
|
519
|
-
```ruby
|
|
520
|
-
# In Rails console
|
|
521
|
-
Product._commit_callbacks.map(&:filter)
|
|
522
|
-
```
|
|
523
|
-
|
|
524
|
-
4. **Test cache key generation**:
|
|
525
|
-
```ruby
|
|
526
|
-
# In Rails console
|
|
527
|
-
product = Product.first
|
|
528
|
-
sweeper = ProductSweeper.new
|
|
529
|
-
# Test your key generation logic
|
|
530
|
-
```
|
|
531
|
-
|
|
532
|
-
### Performance Considerations
|
|
533
|
-
|
|
534
|
-
- **Use `:instant` trigger** for critical cache that must be cleared immediately
|
|
535
|
-
- **Use `:request` trigger** for less critical cache to reduce database load
|
|
536
|
-
- **Use `:async` mode** for high-volume applications to avoid blocking requests
|
|
537
|
-
- **Use `:inline` mode** for low-volume applications or when immediate consistency is required
|
|
538
|
-
- **Monitor Sidekiq queue sizes** to ensure async jobs are being processed
|
|
539
|
-
- **Optimize batch sizes**: Adjust `delete_multi_batch_size` based on your cache store's performance
|
|
540
|
-
- **Redis**: 100-500 keys per batch works well
|
|
541
|
-
- **Memcached**: 50-200 keys per batch is optimal
|
|
542
|
-
- **File store**: 10-50 keys per batch to avoid I/O bottlenecks
|
|
543
|
-
|
|
544
|
-
### Memory Usage
|
|
545
|
-
|
|
546
|
-
- Request-level batching stores cache keys in memory during the request
|
|
547
|
-
- For high-volume applications, consider using `:instant` trigger to avoid memory buildup
|
|
548
|
-
- Monitor RequestStore memory usage in production
|
|
549
|
-
|
|
550
|
-
## Contributing
|
|
551
|
-
|
|
552
|
-
1. Fork the repository
|
|
553
|
-
2. Create your feature branch (`git checkout -b my-new-feature`)
|
|
554
|
-
3. Commit your changes (`git commit -am 'Add some feature'`)
|
|
555
|
-
4. Push to the branch (`git push origin my-new-feature`)
|
|
556
|
-
5. Create a new Pull Request
|
|
557
|
-
|
|
558
|
-
### Development Setup
|
|
559
|
-
|
|
560
|
-
1. Clone the repository
|
|
561
|
-
2. Run `bundle install`
|
|
562
|
-
3. Run tests with `bundle exec rspec`
|
|
563
|
-
4. Run linting with `bundle exec rubocop`
|
|
564
|
-
|
|
565
|
-
### Code Style
|
|
566
|
-
|
|
567
|
-
- Follow Ruby style guidelines
|
|
568
|
-
- Write tests for new features
|
|
569
|
-
- Update documentation for API changes
|
|
570
|
-
- Use meaningful commit messages
|
|
571
|
-
|
|
572
205
|
## License
|
|
573
206
|
|
|
574
|
-
MIT
|
|
575
|
-
|
|
576
|
-
## Links
|
|
577
|
-
|
|
578
|
-
- [GitHub Repository](https://github.com/rafayqayyum/cache_sweeper)
|
|
579
|
-
- [MIT License](https://opensource.org/licenses/MIT)
|
|
580
|
-
- [RubyGems](https://rubygems.org/gems/cache_sweeper)
|
|
207
|
+
MIT. See [LICENSE.txt](LICENSE.txt).
|