cache_sweeper 0.1.0 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +21 -0
- data/README.md +125 -504
- data/lib/cache_sweeper/base.rb +11 -24
- data/lib/cache_sweeper/batch.rb +68 -0
- data/lib/cache_sweeper/callbacks.rb +13 -0
- data/lib/cache_sweeper/configuration.rb +53 -0
- data/lib/cache_sweeper/delete_job.rb +9 -0
- data/lib/cache_sweeper/delivery.rb +36 -0
- data/lib/cache_sweeper/dispatcher.rb +27 -0
- data/lib/cache_sweeper/executor_hook.rb +15 -0
- data/lib/cache_sweeper/key.rb +22 -0
- data/lib/cache_sweeper/log_subscriber.rb +25 -0
- data/lib/cache_sweeper/railtie.rb +21 -6
- data/lib/cache_sweeper/registry.rb +28 -0
- data/lib/cache_sweeper/rule.rb +77 -0
- data/lib/cache_sweeper/version.rb +3 -1
- data/lib/cache_sweeper.rb +27 -88
- metadata +66 -33
- data/CODE_OF_CONDUCT.md +0 -74
- data/lib/cache_sweeper/async_worker.rb +0 -121
- data/lib/cache_sweeper/dsl.rb +0 -32
- data/lib/cache_sweeper/flush_middleware.rb +0 -152
- data/lib/cache_sweeper/loader.rb +0 -368
- data/lib/cache_sweeper/logger.rb +0 -104
- data/lib/tasks/cache_sweeper.rake +0 -4
data/README.md
CHANGED
|
@@ -1,586 +1,207 @@
|
|
|
1
1
|
# CacheSweeper
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
- [Features](#features)
|
|
8
|
-
- [Installation](#installation)
|
|
9
|
-
- [Quick Start](#quick-start)
|
|
10
|
-
- [Configuration](#configuration)
|
|
11
|
-
- [Usage Examples](#usage-examples)
|
|
12
|
-
- [API Reference](#api-reference)
|
|
13
|
-
- [Logging](#logging)
|
|
14
|
-
- [Middleware](#middleware)
|
|
15
|
-
- [Testing](#testing)
|
|
16
|
-
- [Troubleshooting](#troubleshooting)
|
|
17
|
-
- [Contributing](#contributing)
|
|
18
|
-
- [License](#license)
|
|
19
|
-
|
|
20
|
-
## Features
|
|
21
|
-
|
|
22
|
-
- **Rule-based cache invalidation**: Define what changes should trigger cache clearing using a simple DSL
|
|
23
|
-
- **Flexible triggering**: Choose between instant cache deletion or request-level batching
|
|
24
|
-
- **Async processing**: Offload cache deletion to Sidekiq for scalability
|
|
25
|
-
- **Association support**: Invalidate cache for associated models when their attributes change
|
|
26
|
-
- **Multi-level configuration**: Control behavior globally, per-sweeper, or per-rule
|
|
27
|
-
- **Comprehensive logging**: Detailed logging for debugging with configurable levels
|
|
28
|
-
- **Performance monitoring**: Built-in performance logging for cache operations and timing
|
|
29
|
-
- **Error tracking**: Comprehensive error logging with context and stack traces
|
|
30
|
-
- **Clean model code**: Keep cache logic out of your models
|
|
31
|
-
- **Thread-safe**: Uses RequestStore for reliable multi-threaded operation
|
|
32
|
-
- **Efficient batch deletion**: Uses `Rails.cache.delete_multi` for optimal performance
|
|
33
|
-
- **Configurable batch sizes**: Control how many keys are deleted in each batch
|
|
34
|
-
|
|
35
|
-
## Installation
|
|
36
|
-
|
|
37
|
-
Add this line to your application's Gemfile:
|
|
3
|
+
Declarative cache invalidation for ActiveRecord. Describe which model changes invalidate which cache keys in
|
|
4
|
+
dedicated sweeper classes, and CacheSweeper deletes them after the transaction commits: inline, batched per
|
|
5
|
+
request or job, or in the background through ActiveJob.
|
|
38
6
|
|
|
39
7
|
```ruby
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
```sh
|
|
46
|
-
bundle install
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
### Dependencies
|
|
50
|
-
|
|
51
|
-
- **Rails**: 5.0+ (tested with Rails 6.x and 7.x)
|
|
52
|
-
- **Sidekiq**: Required for async cache deletion
|
|
53
|
-
- **RequestStore**: For thread-safe request-level storage
|
|
54
|
-
|
|
55
|
-
Add Sidekiq to your Gemfile:
|
|
56
|
-
|
|
57
|
-
```ruby
|
|
58
|
-
gem 'sidekiq'
|
|
59
|
-
gem 'request_store'
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
## Quick Start
|
|
63
|
-
|
|
64
|
-
### 1. Configure the Gem
|
|
65
|
-
|
|
66
|
-
Create `config/initializers/cache_sweeper.rb`:
|
|
67
|
-
|
|
68
|
-
```ruby
|
|
69
|
-
# Logging configuration
|
|
70
|
-
CacheSweeper.logger = Rails.logger
|
|
71
|
-
CacheSweeper.log_level = :info # :debug, :info, :warn, :error
|
|
72
|
-
|
|
73
|
-
# Global cache invalidation configuration (optional)
|
|
74
|
-
CacheSweeper.trigger = :request # :instant or :request
|
|
75
|
-
CacheSweeper.mode = :async # :async or :inline
|
|
76
|
-
CacheSweeper.queue = :low # Sidekiq queue name
|
|
77
|
-
CacheSweeper.sidekiq_options = { retry: false }
|
|
78
|
-
CacheSweeper.delete_multi_batch_size = 100 # Batch size for efficient cache deletion
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
### 2. Add Middleware (if using request-level batching)
|
|
82
|
-
|
|
83
|
-
Add to `config/application.rb`:
|
|
84
|
-
|
|
85
|
-
```ruby
|
|
86
|
-
config.middleware.use CacheSweeperFlushMiddleware
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
### 3. Create Your First Sweeper
|
|
8
|
+
class CatalogSweeper < CacheSweeper::Base
|
|
9
|
+
watch Product, only: %i[name price slug] do |product|
|
|
10
|
+
["products/#{product.slug}", "products/#{product.slug_before_last_save}", "products/index"]
|
|
11
|
+
end
|
|
90
12
|
|
|
91
|
-
|
|
13
|
+
watch Variant, only: :price, on: :update do |variant|
|
|
14
|
+
product_keys(variant.product)
|
|
15
|
+
end
|
|
92
16
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
sweeper_options trigger: :request, mode: :async, queue: :products
|
|
17
|
+
watch Review, on: %i[create destroy], if: :published? do |review|
|
|
18
|
+
product_keys(review.product)
|
|
19
|
+
end
|
|
97
20
|
|
|
98
|
-
|
|
99
|
-
watch attributes: [:name, :price], keys: ->(product) { ["product:#{product.id}"] }
|
|
21
|
+
private
|
|
100
22
|
|
|
101
|
-
|
|
102
|
-
|
|
23
|
+
def product_keys(product)
|
|
24
|
+
["products/#{product.slug}", [product, :card]]
|
|
25
|
+
end
|
|
103
26
|
end
|
|
104
27
|
```
|
|
105
28
|
|
|
106
|
-
|
|
29
|
+
## Requirements
|
|
107
30
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
That's it! Your cache will now be automatically invalidated when products change.
|
|
113
|
-
|
|
114
|
-
## Configuration
|
|
31
|
+
- Ruby 3.3+
|
|
32
|
+
- Rails 7.1+ (ActiveRecord, ActiveJob, ActiveSupport)
|
|
115
33
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
Configure in `config/initializers/cache_sweeper.rb`:
|
|
119
|
-
|
|
120
|
-
```ruby
|
|
121
|
-
# Logging configuration
|
|
122
|
-
CacheSweeper.logger = Rails.logger
|
|
123
|
-
CacheSweeper.log_level = :info # :debug, :info, :warn, :error
|
|
124
|
-
|
|
125
|
-
# Cache invalidation configuration (optional - has sensible defaults)
|
|
126
|
-
CacheSweeper.trigger = :request # :instant or :request
|
|
127
|
-
CacheSweeper.mode = :async # :async or :inline
|
|
128
|
-
CacheSweeper.queue = :low # Sidekiq queue name
|
|
129
|
-
CacheSweeper.sidekiq_options = { retry: false }
|
|
130
|
-
CacheSweeper.delete_multi_batch_size = 100 # Batch size for efficient cache deletion
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
### Sweeper-Level Configuration
|
|
134
|
-
|
|
135
|
-
Use the `sweeper_options` DSL for clean sweeper configuration:
|
|
34
|
+
## Installation
|
|
136
35
|
|
|
137
36
|
```ruby
|
|
138
|
-
|
|
139
|
-
sweeper_options trigger: :request, mode: :async, queue: :orders
|
|
140
|
-
# ... watch rules
|
|
141
|
-
end
|
|
37
|
+
gem "cache_sweeper"
|
|
142
38
|
```
|
|
143
39
|
|
|
144
|
-
|
|
40
|
+
Put sweepers in `app/sweepers`. They are loaded at boot and reloaded with the rest of your code in development.
|
|
145
41
|
|
|
146
|
-
|
|
42
|
+
## Watching models
|
|
147
43
|
|
|
148
44
|
```ruby
|
|
149
|
-
|
|
150
|
-
#
|
|
151
|
-
watch attributes: [:name], keys: ->(obj) { ["instant:#{obj.id}"] },
|
|
152
|
-
trigger: :instant, mode: :inline
|
|
153
|
-
|
|
154
|
-
# Async processing for less critical data
|
|
155
|
-
watch attributes: [:description], keys: ->(obj) { ["async:#{obj.id}"] },
|
|
156
|
-
trigger: :request, mode: :async, queue: :background
|
|
45
|
+
watch Model, on: %i[create update destroy], only: nil, if: nil, unless: nil, delivery: nil, queue: nil do |record|
|
|
46
|
+
# return one cache key or an array of keys
|
|
157
47
|
end
|
|
158
48
|
```
|
|
159
49
|
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
50
|
+
| Option | Meaning |
|
|
51
|
+
|------------|-----------------------------------------------------------------------------------------------|
|
|
52
|
+
| `on` | Events that trigger the sweep. Defaults to all three. |
|
|
53
|
+
| `only` | Attributes whose change triggers an `update` sweep. Creates and destroys always sweep. |
|
|
54
|
+
| `if` | Symbol (method on the record) or Proc (receives the record, runs on the sweeper) to require. |
|
|
55
|
+
| `unless` | Same as `if`, inverted. |
|
|
56
|
+
| `delivery` | Overrides the sweeper's delivery for this rule. |
|
|
57
|
+
| `queue` | ActiveJob queue for `:async` delivery. |
|
|
163
58
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
59
|
+
A sweeper can watch any number of models, and any number of sweepers can watch the same model, so group them by
|
|
60
|
+
what they cache rather than by model. The block and Proc conditions run on a sweeper instance, so private helper
|
|
61
|
+
methods are available to them. Sweeps run in `after_commit`, so rolled-back changes never invalidate anything.
|
|
167
62
|
|
|
168
|
-
###
|
|
63
|
+
### Keys
|
|
169
64
|
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
- **`sidekiq_options`**: Hash of Sidekiq options (e.g., `{ retry: false, backtrace: true }`)
|
|
174
|
-
- **`delete_multi_batch_size`**: Number of keys to delete in each batch (default: 100)
|
|
175
|
-
|
|
176
|
-
## Usage Examples
|
|
177
|
-
|
|
178
|
-
### Basic Sweeper
|
|
65
|
+
The block returns a single key or an array of keys. Keys are expanded exactly like `Rails.cache` expands them,
|
|
66
|
+
so anything you can pass to `Rails.cache.fetch` works, including records and composite keys. Nest composite keys
|
|
67
|
+
inside the returned array:
|
|
179
68
|
|
|
180
69
|
```ruby
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
watch attributes: [:name, :price], keys: ->(product) { ["product:#{product.id}"] }
|
|
70
|
+
watch Product do |product|
|
|
71
|
+
["products/index", [product, :card], { locale: I18n.locale, page: :home }]
|
|
184
72
|
end
|
|
185
73
|
```
|
|
186
74
|
|
|
187
|
-
|
|
75
|
+
`nil` and blank keys are dropped and duplicates removed. Use Rails' `*_before_last_save` readers to sweep keys
|
|
76
|
+
built from a previous value, such as an old slug.
|
|
188
77
|
|
|
189
|
-
|
|
190
|
-
# app/cache_sweepers/package_sweeper.rb
|
|
191
|
-
class PackageSweeper < CacheSweeper::Base
|
|
192
|
-
# Clear cache when package name changes
|
|
193
|
-
watch attributes: [:name], keys: ->(package) { ["package:#{package.id}"] }
|
|
194
|
-
|
|
195
|
-
# Clear cache when associated products change
|
|
196
|
-
watch :products, attributes: [:name], keys: ->(product) {
|
|
197
|
-
product.packages.map { |pkg| "package:#{pkg.id}" }
|
|
198
|
-
}
|
|
199
|
-
end
|
|
200
|
-
```
|
|
78
|
+
## Delivery
|
|
201
79
|
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
# Clear cache when user profile changes
|
|
208
|
-
watch attributes: [:name, :email], keys: ->(user) { ["user:#{user.id}", "user:#{user.id}:profile"] }
|
|
209
|
-
|
|
210
|
-
# Clear cache with custom condition
|
|
211
|
-
watch attributes: [:last_login_at], keys: ->(user) { ["users:active"] },
|
|
212
|
-
if: ->(user) { user.last_login_at_changed? && user.last_login_at > 1.day.ago }
|
|
213
|
-
end
|
|
214
|
-
```
|
|
80
|
+
| Mode | When keys are deleted |
|
|
81
|
+
|------------|----------------------------------------------------------------------------------------------------|
|
|
82
|
+
| `:inline` | Immediately after commit. Default. |
|
|
83
|
+
| `:batched` | At the end of the current request or job, deduplicated into one `delete_multi`. |
|
|
84
|
+
| `:async` | At the end of the current request or job, enqueued as `CacheSweeper::DeleteJob`s. |
|
|
215
85
|
|
|
216
|
-
|
|
86
|
+
`:inline` is the default because it is the only mode where code running after the commit, including the rest of
|
|
87
|
+
the same request, never reads a stale entry. `:batched` and `:async` trade that for fewer cache round trips: for
|
|
88
|
+
a request, the keys are deleted after the response has been sent, so a write followed by a cached read in the
|
|
89
|
+
same request, or a request arriving in that gap, sees the old value. Opt into them per sweeper or per rule where
|
|
90
|
+
that is acceptable.
|
|
217
91
|
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
sweeper_options trigger: :request, mode: :async, queue: :orders
|
|
223
|
-
|
|
224
|
-
# Critical data - instant deletion
|
|
225
|
-
watch attributes: [:status], keys: ->(order) { ["order:#{order.id}"] },
|
|
226
|
-
trigger: :instant, mode: :inline
|
|
227
|
-
|
|
228
|
-
# Less critical data - async processing
|
|
229
|
-
watch attributes: [:notes], keys: ->(order) { ["order:#{order.id}:notes"] },
|
|
230
|
-
trigger: :request, mode: :async, queue: :background
|
|
231
|
-
|
|
232
|
-
# Association changes
|
|
233
|
-
watch :order_items, attributes: [:quantity, :price], keys: ->(order_item) {
|
|
234
|
-
["order:#{order_item.order_id}", "order:#{order_item.order_id}:total"]
|
|
235
|
-
}
|
|
236
|
-
end
|
|
237
|
-
```
|
|
92
|
+
Batching hooks into the Rails executor, which wraps web requests, ActiveJob and Sidekiq jobs, and ActionCable.
|
|
93
|
+
Pending keys are flushed even when the request raises, and early once `max_pending_keys` are waiting, so a
|
|
94
|
+
long job updating many records deletes as it goes instead of holding every key until it ends. Async keys are
|
|
95
|
+
split into jobs of `batch_size` keys and enqueued in bulk.
|
|
238
96
|
|
|
239
|
-
|
|
97
|
+
`rails runner` wraps the whole script in the executor, so a runner script is one unit of work: its keys are
|
|
98
|
+
deleted when it finishes, or every `max_pending_keys` keys. Outside an executor (console, rake tasks) `:batched`
|
|
99
|
+
deletes and `:async` enqueues straight away. To batch a console session or rake task, wrap it:
|
|
240
100
|
|
|
241
101
|
```ruby
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
# Only clear cache on create and destroy, not update
|
|
245
|
-
watch attributes: [:message], keys: ->(notification) { ["notifications:count"] },
|
|
246
|
-
on: [:create, :destroy]
|
|
247
|
-
|
|
248
|
-
# Use before_commit instead of after_commit
|
|
249
|
-
watch attributes: [:read_at], keys: ->(notification) { ["user:#{notification.user_id}:unread_count"] },
|
|
250
|
-
callback: :before_commit
|
|
102
|
+
Rails.application.executor.wrap do
|
|
103
|
+
Product.find_each { |product| product.update!(price: product.price * 1.1) }
|
|
251
104
|
end
|
|
252
105
|
```
|
|
253
106
|
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
### Sweeper DSL
|
|
257
|
-
|
|
258
|
-
#### `watch(association = nil, **options)`
|
|
259
|
-
|
|
260
|
-
Define cache invalidation rules.
|
|
261
|
-
|
|
262
|
-
**Parameters:**
|
|
263
|
-
|
|
264
|
-
- **`association`** (optional): Association name to watch (e.g., `:products`, `:order_items`)
|
|
265
|
-
- **`attributes`**: Array of attributes to watch for changes (e.g., `[:name, :price]`)
|
|
266
|
-
- **`keys`**: Proc or array of cache keys to invalidate
|
|
267
|
-
- **`if`**: Proc or method name for conditional invalidation
|
|
268
|
-
- **`trigger`**: `:instant` or `:request` (per rule)
|
|
269
|
-
- **`mode`**: `:async` or `:inline` (per rule)
|
|
270
|
-
- **`queue`**: Sidekiq queue name (per rule)
|
|
271
|
-
- **`sidekiq_options`**: Hash of Sidekiq options (per rule)
|
|
272
|
-
- **`callback`**: Callback type (`:after_commit`, `:before_commit`, etc.)
|
|
273
|
-
- **`on`**: Events to watch (`[:create, :update, :destroy]`)
|
|
274
|
-
|
|
275
|
-
**Examples:**
|
|
107
|
+
Set the delivery per sweeper or per rule:
|
|
276
108
|
|
|
277
109
|
```ruby
|
|
278
|
-
|
|
279
|
-
|
|
110
|
+
class SearchSweeper < CacheSweeper::Base
|
|
111
|
+
delivery :async, queue: :low
|
|
280
112
|
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
# Conditional invalidation
|
|
285
|
-
watch attributes: [:status], keys: ->(obj) { ["key"] },
|
|
286
|
-
if: ->(obj) { obj.status == 'published' }
|
|
287
|
-
|
|
288
|
-
# Custom events
|
|
289
|
-
watch attributes: [:name], keys: ->(obj) { ["key"] },
|
|
290
|
-
on: [:create, :destroy]
|
|
291
|
-
```
|
|
292
|
-
|
|
293
|
-
#### `sweeper_options(**options)`
|
|
294
|
-
|
|
295
|
-
Configure sweeper-level behavior.
|
|
296
|
-
|
|
297
|
-
**Parameters:**
|
|
298
|
-
|
|
299
|
-
- **`trigger`**: `:instant` or `:request`
|
|
300
|
-
- **`mode`**: `:async` or `:inline`
|
|
301
|
-
- **`queue`**: Sidekiq queue name
|
|
302
|
-
- **`sidekiq_options`**: Hash of Sidekiq options
|
|
303
|
-
|
|
304
|
-
**Example:**
|
|
113
|
+
watch Product, only: :name do |product|
|
|
114
|
+
"search/#{product.category_id}"
|
|
115
|
+
end
|
|
305
116
|
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
117
|
+
watch Product, only: :price, delivery: :inline do |product|
|
|
118
|
+
"prices/#{product.id}"
|
|
119
|
+
end
|
|
309
120
|
end
|
|
310
121
|
```
|
|
311
122
|
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
#### `CacheSweeper.logger = logger`
|
|
315
|
-
|
|
316
|
-
Set the logger for cache actions.
|
|
317
|
-
|
|
318
|
-
#### `CacheSweeper.log_level = level`
|
|
319
|
-
|
|
320
|
-
Set minimum log level (`:debug`, `:info`, `:warn`, `:error`).
|
|
321
|
-
|
|
322
|
-
#### Global Configuration Attributes
|
|
323
|
-
|
|
324
|
-
Configure global cache invalidation behavior using direct attribute assignment:
|
|
325
|
-
|
|
326
|
-
**Example:**
|
|
327
|
-
|
|
328
|
-
```ruby
|
|
329
|
-
CacheSweeper.trigger = :request
|
|
330
|
-
CacheSweeper.mode = :async
|
|
331
|
-
CacheSweeper.queue = :low
|
|
332
|
-
CacheSweeper.sidekiq_options = { retry: false }
|
|
333
|
-
```
|
|
334
|
-
|
|
335
|
-
## Logging
|
|
336
|
-
|
|
337
|
-
The gem provides comprehensive logging to help debug cache invalidation issues.
|
|
338
|
-
|
|
339
|
-
### Log Levels
|
|
340
|
-
|
|
341
|
-
- **`:debug`** - All logging enabled (initialization, rule execution, performance, cache operations, async jobs, middleware)
|
|
342
|
-
- **`:info`** - Important events (initialization, cache operations, async jobs, middleware)
|
|
343
|
-
- **`:warn`** - Warnings and errors only
|
|
344
|
-
- **`:error`** - Errors only
|
|
345
|
-
|
|
346
|
-
### Default Log Levels
|
|
347
|
-
|
|
348
|
-
- **Development**: `:debug` (all logging enabled)
|
|
349
|
-
- **Production**: `:warn` (warnings and errors only)
|
|
350
|
-
- **Other environments**: `:info`
|
|
351
|
-
|
|
352
|
-
### Log Output
|
|
123
|
+
Rule options win over sweeper options, which win over the global configuration.
|
|
353
124
|
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
- **Initialization**: Sweeper loading and model attachment
|
|
357
|
-
- **Rule execution**: Which rules are triggered, condition evaluation, cache key generation
|
|
358
|
-
- **Performance**: Timing for cache operations and rule execution
|
|
359
|
-
- **Cache operations**: Cache key invalidation details
|
|
360
|
-
- **Async jobs**: Job scheduling and execution status
|
|
361
|
-
- **Middleware**: Request-level batching and flushing
|
|
362
|
-
- **Errors**: Detailed error information with context and stack traces
|
|
363
|
-
|
|
364
|
-
### Example Log Output
|
|
365
|
-
|
|
366
|
-
```
|
|
367
|
-
[CacheSweeper] [2024-01-15 10:30:45.123] [INFO] Initialization: Processing sweeper: ProductSweeper
|
|
368
|
-
[CacheSweeper] [2024-01-15 10:30:45.124] [DEBUG] Rule execution: ProductSweeper -> Product#123
|
|
369
|
-
[CacheSweeper] [2024-01-15 10:30:45.125] [INFO] Cache operations: Deleted instantly: product:123
|
|
370
|
-
[CacheSweeper] [2024-01-15 10:30:45.126] [DEBUG] Performance: cache_invalidation took 2.456ms
|
|
371
|
-
```
|
|
372
|
-
|
|
373
|
-
### Debugging Configuration
|
|
125
|
+
## Configuration
|
|
374
126
|
|
|
375
127
|
```ruby
|
|
376
|
-
#
|
|
377
|
-
CacheSweeper.
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
#
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
128
|
+
# config/initializers/cache_sweeper.rb
|
|
129
|
+
CacheSweeper.configure do |config|
|
|
130
|
+
config.cache_store = Rails.cache # any ActiveSupport::Cache::Store, defaults to Rails.cache
|
|
131
|
+
config.delivery = :inline # :inline, :batched or :async
|
|
132
|
+
config.queue = :default # ActiveJob queue for :async
|
|
133
|
+
config.batch_size = 100 # keys per delete_multi call and per DeleteJob
|
|
134
|
+
config.max_pending_keys = 1_000 # flush a request's or job's batch early at this size
|
|
135
|
+
config.raise_errors = Rails.env.local? # raise sweep errors instead of reporting them
|
|
136
|
+
config.force_inline = false # delete inline regardless of declared delivery
|
|
137
|
+
config.sweeper_paths = ["app/sweepers"]
|
|
138
|
+
end
|
|
384
139
|
```
|
|
385
140
|
|
|
386
|
-
##
|
|
387
|
-
|
|
388
|
-
The `CacheSweeperFlushMiddleware` handles request-level batching. It automatically flushes all pending cache keys at the end of each request.
|
|
389
|
-
|
|
390
|
-
### Setup
|
|
391
|
-
|
|
392
|
-
Add to `config/application.rb`:
|
|
141
|
+
## Deleting manually
|
|
393
142
|
|
|
394
143
|
```ruby
|
|
395
|
-
|
|
144
|
+
CacheSweeper.delete("products/index", [product, :card])
|
|
396
145
|
```
|
|
397
146
|
|
|
398
|
-
|
|
147
|
+
## Errors
|
|
399
148
|
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
149
|
+
A failing key block or cache delete never breaks the request that committed the change. Errors go to
|
|
150
|
+
`Rails.error` (`ActiveSupport.error_reporter`) with `source: "cache_sweeper"` and are logged. With `raise_errors`
|
|
151
|
+
enabled, the default in development and test, they are raised instead. `DeleteJob` always raises so your queue
|
|
152
|
+
adapter can retry it.
|
|
403
153
|
|
|
404
|
-
|
|
154
|
+
## Instrumentation
|
|
405
155
|
|
|
406
|
-
|
|
156
|
+
| Event | Payload |
|
|
157
|
+
|---------------------------|--------------------------------------------------|
|
|
158
|
+
| `sweep.cache_sweeper` | `sweeper`, `record`, `event`, `keys`, `delivery` |
|
|
159
|
+
| `delete.cache_sweeper` | `keys` |
|
|
160
|
+
| `failure.cache_sweeper` | `error` and context |
|
|
407
161
|
|
|
408
|
-
|
|
409
|
-
# config/application.rb
|
|
410
|
-
config.middleware.use SomeOtherMiddleware
|
|
411
|
-
config.middleware.use CacheSweeperFlushMiddleware
|
|
412
|
-
```
|
|
162
|
+
Sweeps and deletes are logged at debug level and failures at error level.
|
|
413
163
|
|
|
414
164
|
## Testing
|
|
415
165
|
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
You can test sweepers using standard Rails/ActiveRecord test frameworks:
|
|
166
|
+
Requests and jobs in your test suite run inside the executor, so sweepers using `:batched` or `:async` only
|
|
167
|
+
sweep when the test's unit of work finishes. Force inline deletion in tests that assert on the cache:
|
|
419
168
|
|
|
420
169
|
```ruby
|
|
421
|
-
#
|
|
422
|
-
|
|
423
|
-
test "clears cache when product name changes" do
|
|
424
|
-
product = Product.create!(name: "Original Name")
|
|
425
|
-
|
|
426
|
-
# Mock cache
|
|
427
|
-
Rails.cache.expects(:delete).with("product:#{product.id}")
|
|
428
|
-
|
|
429
|
-
product.update!(name: "New Name")
|
|
430
|
-
end
|
|
431
|
-
end
|
|
170
|
+
# config/environments/test.rb
|
|
171
|
+
config.after_initialize { CacheSweeper.config.force_inline = true }
|
|
432
172
|
```
|
|
433
173
|
|
|
434
|
-
|
|
174
|
+
## Limitations
|
|
435
175
|
|
|
436
|
-
|
|
176
|
+
Sweeps rely on `after_commit`, so bulk operations that skip callbacks (`update_all`, `delete_all`, `insert_all`,
|
|
177
|
+
`upsert_all`) don't sweep. `only:` reads `saved_changes`, which reflects the record's last save within the
|
|
178
|
+
transaction.
|
|
437
179
|
|
|
438
|
-
|
|
439
|
-
# test/sweepers/async_sweeper_test.rb
|
|
440
|
-
class AsyncSweeperTest < ActiveSupport::TestCase
|
|
441
|
-
test "schedules async job for cache deletion" do
|
|
442
|
-
# Stub Sidekiq worker
|
|
443
|
-
CacheSweeper::AsyncWorker.expects(:perform_async).with(["key1", "key2"])
|
|
444
|
-
|
|
445
|
-
# Trigger the change
|
|
446
|
-
product = Product.create!(name: "Test Product")
|
|
447
|
-
end
|
|
448
|
-
end
|
|
449
|
-
```
|
|
450
|
-
|
|
451
|
-
### Testing with Sidekiq
|
|
180
|
+
## Development
|
|
452
181
|
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
#
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
# Ensure Sidekiq is running
|
|
460
|
-
Sidekiq::Testing.inline! do
|
|
461
|
-
product = Product.create!(name: "Test Product")
|
|
462
|
-
# Cache should be cleared synchronously
|
|
463
|
-
end
|
|
464
|
-
end
|
|
465
|
-
end
|
|
182
|
+
```sh
|
|
183
|
+
bundle install
|
|
184
|
+
bundle exec rake # rubocop, both suites, 100% line and branch coverage
|
|
185
|
+
bundle exec rspec spec/unit # ActiveRecord without Rails
|
|
186
|
+
bundle exec rspec spec/rails # Railtie, executor and reloading against a dummy app
|
|
187
|
+
RAILS_VERSION=7.2 bundle update && bundle exec rake spec
|
|
466
188
|
```
|
|
467
189
|
|
|
468
|
-
|
|
190
|
+
The two suites run in separate processes so the unit suite proves the gem works without Rails loaded. Their
|
|
191
|
+
coverage is merged before the threshold is enforced. The store specs run against Redis and Memcached when they
|
|
192
|
+
are reachable on their default ports, and CI always runs them.
|
|
469
193
|
|
|
470
|
-
|
|
194
|
+
CI also runs RuboCop, `bundle-audit` and a strict `gem build`; CodeQL scans the code weekly and on every PR.
|
|
471
195
|
|
|
472
|
-
|
|
196
|
+
### Releasing
|
|
473
197
|
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
**Solution**: Ensure Sidekiq is running:
|
|
198
|
+
Bump `CacheSweeper::VERSION`, update the CHANGELOG, then tag and push. The Release workflow runs the suite and
|
|
199
|
+
publishes to RubyGems through trusted publishing:
|
|
477
200
|
|
|
478
201
|
```sh
|
|
479
|
-
|
|
202
|
+
git tag v1.0.0 && git push origin v1.0.0
|
|
480
203
|
```
|
|
481
204
|
|
|
482
|
-
#### Sweepers Not Loading
|
|
483
|
-
|
|
484
|
-
**Problem**: Sweepers aren't being loaded or attached to models.
|
|
485
|
-
|
|
486
|
-
**Solution**:
|
|
487
|
-
1. Ensure sweepers are in `app/cache_sweepers/` directory
|
|
488
|
-
2. Ensure sweepers inherit from `CacheSweeper::Base`
|
|
489
|
-
3. Check that the sweeper files are named `*_sweeper.rb`
|
|
490
|
-
|
|
491
|
-
#### Cache Keys Not Being Cleared
|
|
492
|
-
|
|
493
|
-
**Problem**: Cache keys aren't being invalidated when models change.
|
|
494
|
-
|
|
495
|
-
**Solution**:
|
|
496
|
-
1. Enable debug logging: `CacheSweeper.log_level = :debug`
|
|
497
|
-
2. Check that the correct attributes are being watched
|
|
498
|
-
3. Verify cache key generation logic
|
|
499
|
-
4. Ensure the model callbacks are being triggered
|
|
500
|
-
|
|
501
|
-
#### Middleware Not Flushing
|
|
502
|
-
|
|
503
|
-
**Problem**: Request-level batching isn't flushing at the end of requests.
|
|
504
|
-
|
|
505
|
-
**Solution**:
|
|
506
|
-
1. Ensure `CacheSweeperFlushMiddleware` is added to the middleware stack
|
|
507
|
-
2. Check middleware order in `config/application.rb`
|
|
508
|
-
3. Verify that `trigger: :request` is being used
|
|
509
|
-
|
|
510
|
-
### Debugging Steps
|
|
511
|
-
|
|
512
|
-
1. **Enable comprehensive logging**:
|
|
513
|
-
```ruby
|
|
514
|
-
CacheSweeper.logger = Rails.logger
|
|
515
|
-
CacheSweeper.log_level = :debug
|
|
516
|
-
```
|
|
517
|
-
|
|
518
|
-
2. **Check sweeper loading**:
|
|
519
|
-
```ruby
|
|
520
|
-
# In Rails console
|
|
521
|
-
CacheSweeper::Base.descendants
|
|
522
|
-
```
|
|
523
|
-
|
|
524
|
-
3. **Verify model callbacks**:
|
|
525
|
-
```ruby
|
|
526
|
-
# In Rails console
|
|
527
|
-
Product._commit_callbacks.map(&:filter)
|
|
528
|
-
```
|
|
529
|
-
|
|
530
|
-
4. **Test cache key generation**:
|
|
531
|
-
```ruby
|
|
532
|
-
# In Rails console
|
|
533
|
-
product = Product.first
|
|
534
|
-
sweeper = ProductSweeper.new
|
|
535
|
-
# Test your key generation logic
|
|
536
|
-
```
|
|
537
|
-
|
|
538
|
-
### Performance Considerations
|
|
539
|
-
|
|
540
|
-
- **Use `:instant` trigger** for critical cache that must be cleared immediately
|
|
541
|
-
- **Use `:request` trigger** for less critical cache to reduce database load
|
|
542
|
-
- **Use `:async` mode** for high-volume applications to avoid blocking requests
|
|
543
|
-
- **Use `:inline` mode** for low-volume applications or when immediate consistency is required
|
|
544
|
-
- **Monitor Sidekiq queue sizes** to ensure async jobs are being processed
|
|
545
|
-
- **Optimize batch sizes**: Adjust `delete_multi_batch_size` based on your cache store's performance
|
|
546
|
-
- **Redis**: 100-500 keys per batch works well
|
|
547
|
-
- **Memcached**: 50-200 keys per batch is optimal
|
|
548
|
-
- **File store**: 10-50 keys per batch to avoid I/O bottlenecks
|
|
549
|
-
|
|
550
|
-
### Memory Usage
|
|
551
|
-
|
|
552
|
-
- Request-level batching stores cache keys in memory during the request
|
|
553
|
-
- For high-volume applications, consider using `:instant` trigger to avoid memory buildup
|
|
554
|
-
- Monitor RequestStore memory usage in production
|
|
555
|
-
|
|
556
|
-
## Contributing
|
|
557
|
-
|
|
558
|
-
1. Fork the repository
|
|
559
|
-
2. Create your feature branch (`git checkout -b my-new-feature`)
|
|
560
|
-
3. Commit your changes (`git commit -am 'Add some feature'`)
|
|
561
|
-
4. Push to the branch (`git push origin my-new-feature`)
|
|
562
|
-
5. Create a new Pull Request
|
|
563
|
-
|
|
564
|
-
### Development Setup
|
|
565
|
-
|
|
566
|
-
1. Clone the repository
|
|
567
|
-
2. Run `bundle install`
|
|
568
|
-
3. Run tests with `bundle exec rspec`
|
|
569
|
-
4. Run linting with `bundle exec rubocop`
|
|
570
|
-
|
|
571
|
-
### Code Style
|
|
572
|
-
|
|
573
|
-
- Follow Ruby style guidelines
|
|
574
|
-
- Write tests for new features
|
|
575
|
-
- Update documentation for API changes
|
|
576
|
-
- Use meaningful commit messages
|
|
577
|
-
|
|
578
205
|
## License
|
|
579
206
|
|
|
580
|
-
MIT
|
|
581
|
-
|
|
582
|
-
## Links
|
|
583
|
-
|
|
584
|
-
- [GitHub Repository](https://github.com/rafayqayyum/cache_sweeper)
|
|
585
|
-
- [MIT License](https://opensource.org/licenses/MIT)
|
|
586
|
-
- [RubyGems](https://rubygems.org/gems/cache_sweeper) (when published)
|
|
207
|
+
MIT. See [LICENSE.txt](LICENSE.txt).
|