synchronisable 1.1.5 → 1.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +173 -23
- data/lib/synchronisable/configuration.rb +21 -0
- data/lib/synchronisable/gateway.rb +1 -1
- data/lib/synchronisable/helper/logging.rb +1 -1
- data/lib/synchronisable/model.rb +6 -0
- data/lib/synchronisable/source.rb +0 -2
- data/lib/synchronisable/synchronizer.rb +2 -2
- data/lib/synchronisable/version.rb +1 -1
- data/lib/synchronisable.rb +18 -23
- data/spec/synchronisable/synchronisable_spec.rb +3 -3
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 70388e8c1eef4b5cb5e12223a4201d02073248ad
|
4
|
+
data.tar.gz: 829ab2f2ebf18374eaaab7e34b97ca084c494224
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3c4b925a4fb1a39e8c7d440fea7803c2303f89b7fad7b951564aa800b18b21d48934691f686e9e112102e005df959792f67d08ba4529687b7e4a84977905e74e
|
7
|
+
data.tar.gz: 7f69d985173f10d17fb3e84474c7b1dc4c0d0d8e8f85eadf9d7cf42a6364aa3072606a0a275a09060a8ec28268998640664019ba2de87f9bae199a239cc58e14
|
data/README.md
CHANGED
@@ -7,11 +7,21 @@
|
|
7
7
|
|
8
8
|
# Synchronisable
|
9
9
|
|
10
|
-
|
11
|
-
with external resources (apis, services etc).
|
10
|
+
### :construction: this README & docs are work in progress :construction:
|
12
11
|
|
13
12
|
## Overview
|
14
13
|
|
14
|
+
Provides base fuctionality for active record models synchronization
|
15
|
+
with external resources. The remote source could be anything you like:
|
16
|
+
apis, services, site that you gonna parse and steal some data from it.
|
17
|
+
|
18
|
+
## Resources
|
19
|
+
|
20
|
+
* [Rubygems](https://rubygems.org/gems/synchronisable)
|
21
|
+
* [API](http://rdoc.info/github/vyorkin/synchronisable/master/frames)
|
22
|
+
* [Bugs](https://github.com/vyorkin/synchronisable/issues)
|
23
|
+
* [Development](https://travis-ci.org/vyorkin/synchronisable)
|
24
|
+
|
15
25
|
## Installation
|
16
26
|
|
17
27
|
Add this line to your application's Gemfile:
|
@@ -26,58 +36,195 @@ Or install it yourself as:
|
|
26
36
|
|
27
37
|
$ gem install synchronisable
|
28
38
|
|
39
|
+
Optionally, if you are using rails to run an initializer generator:
|
40
|
+
|
41
|
+
$ rails g synchronisable:install
|
42
|
+
|
43
|
+
## Rationale
|
44
|
+
|
45
|
+
Sometimes we need to sync our domain models (or some part of them)
|
46
|
+
with some kind of remote source. Its great if you can consume a well-done RESTful api
|
47
|
+
that is pretty close to you local domain models.
|
48
|
+
But unfortunately the remote data source could be just anything.
|
49
|
+
|
50
|
+
Actually this gem was made to consume data coming from a site parser :crying_cat_face:
|
51
|
+
|
52
|
+
Examples of the usage patterns are shown below.
|
53
|
+
You can find more by looking at the [dummy app](https://github.com/vyorkin/synchronisable/tree/master/spec/dummy/app)
|
54
|
+
[models](https://github.com/vyorkin/synchronisable/tree/master/spec/dummy/app/models) and
|
55
|
+
[synchronizers](https://github.com/vyorkin/synchronisable/tree/master/spec/dummy/app/synchronizers).
|
56
|
+
|
57
|
+
## Features
|
58
|
+
|
59
|
+
* Attribute mapping, `unique_id`
|
60
|
+
* Associations sync + `:includes` option to specify (restrict)
|
61
|
+
an association tree to be synchronized
|
62
|
+
* `before` and `after` callbacks to hook into sync process
|
63
|
+
* ???
|
64
|
+
|
65
|
+
## Configuration
|
66
|
+
|
67
|
+
For rails users there is a well-documented initializer.
|
68
|
+
Just run `rails g synchronisable:install` and you'll be fine.
|
69
|
+
|
70
|
+
Non-rails users can do so by using provided
|
71
|
+
`ActiveSupport::Configurable` interface. So here is the default settings:
|
72
|
+
|
73
|
+
```ruby
|
74
|
+
Synchronisable.configure do |config|
|
75
|
+
# Logging configuration
|
76
|
+
#
|
77
|
+
# Default logger fallbacks to `Rails.logger` if available, otherwise
|
78
|
+
# `STDOUT` will be used for output.
|
79
|
+
#
|
80
|
+
config.logging = {
|
81
|
+
:logger => defined?(Rails) ? Rails.logger : Logger.new(STDOUT)
|
82
|
+
:verbose => true,
|
83
|
+
:colorize => true
|
84
|
+
}
|
85
|
+
|
86
|
+
# If you want to restrict synchronized models.
|
87
|
+
# By default it will try to sync all models that have
|
88
|
+
# a `synchronisable` dsl instruction.
|
89
|
+
#
|
90
|
+
config.models = %w(Foo Bar)
|
91
|
+
end
|
92
|
+
```
|
93
|
+
|
29
94
|
## Usage
|
30
95
|
|
31
|
-
|
96
|
+
Imagine a situation when you have to periodically get data from
|
97
|
+
some remote source and store it locally.
|
98
|
+
Basically the task is to create local records if they don't exist
|
99
|
+
and update their attributes otherwise.
|
100
|
+
|
101
|
+
### Gateways
|
102
|
+
|
103
|
+
Thing that provides an access to an external system or resource
|
104
|
+
is called [gateway](http://martinfowler.com/eaaCatalog/gateway.html).
|
105
|
+
You can take a look at the base [gateway](https://github.com/vyorkin/synchronisable/blob/master/lib/synchronisable/gateway.rb)
|
106
|
+
class to get a clue what does it mean in terms of this gem
|
107
|
+
(btw fetching data from a remote source is not a purpose of this gem).
|
108
|
+
|
109
|
+
The main idea is that gateway implementation class has only 2 methods:
|
110
|
+
|
111
|
+
* `fetch(params = {})` – returns an array of hashes, each hash contains
|
112
|
+
an attributes that should be (somehow) mapped over your target model.
|
113
|
+
* `find(params)` – returns a single hash with remote attributes.
|
114
|
+
`params` here is only to have a choice between representing a single
|
115
|
+
or a composite identity.
|
116
|
+
|
117
|
+
### Models and synchronizers
|
118
|
+
|
119
|
+
The first step is to declare that your active record model is synchronizable.
|
120
|
+
You can do so by using corresponding `synchronisable` dsl instruction,
|
121
|
+
that optionally takes a synchonizer class to be used.
|
122
|
+
You should only specify it when the name can't be figured out
|
123
|
+
by the following convention: `ModelSynchronizer`.
|
124
|
+
So for example here we have a Tournament that has many Stages:
|
32
125
|
|
33
126
|
```ruby
|
34
|
-
class
|
35
|
-
has_many :
|
127
|
+
class Tournament < ActiveRecord::Base
|
128
|
+
has_many :stages
|
36
129
|
|
37
130
|
synchronisable
|
38
131
|
end
|
39
132
|
|
40
|
-
class
|
41
|
-
belongs_to :
|
133
|
+
class Stage < ActiveRecord::Base
|
134
|
+
belongs_to :tournament
|
42
135
|
|
43
|
-
synchronisable
|
136
|
+
synchronisable
|
44
137
|
end
|
45
138
|
```
|
46
139
|
|
47
|
-
|
48
|
-
synchronisable. You can do so by using corresponding dsl instruction,
|
49
|
-
that optionally takes a synchonizer class to be used. Actually,
|
50
|
-
the only reason to specify it its when it has a name, that can't be figured out
|
51
|
-
by the following convention: `ModelSynchronizer`.
|
52
|
-
|
53
|
-
After that you should define your model synchronizers
|
140
|
+
Lets define synchronizers:
|
54
141
|
|
55
142
|
```ruby
|
56
|
-
class
|
57
|
-
|
143
|
+
class TournamentSynchronizer < Synchronisable::Synchronizer
|
144
|
+
has_many :stages
|
58
145
|
|
146
|
+
remote_id :tour_id
|
147
|
+
unique_id { |attrs| attrs[:name] }
|
148
|
+
|
149
|
+
mappings(
|
150
|
+
:eman => :name,
|
151
|
+
:eman_trohs => :short_name,
|
152
|
+
:gninnigeb => :beginning,
|
153
|
+
:gnidge => :ending,
|
154
|
+
:tnerruc_si => :is_current
|
155
|
+
)
|
156
|
+
|
157
|
+
only :name, :beginning, :ending
|
158
|
+
|
159
|
+
gateway TournamentGateway
|
160
|
+
end
|
161
|
+
|
162
|
+
class StageSynchronizer < Synchronisable::Synchronizer
|
163
|
+
has_many :matches
|
164
|
+
|
165
|
+
remote_id :stage_id
|
166
|
+
|
167
|
+
mappings(
|
168
|
+
:tour_id => :tournament_id,
|
169
|
+
:gninnigeb => :beginning,
|
170
|
+
:gnidge => :ending,
|
171
|
+
:eman => :name,
|
172
|
+
:rebmun => :number
|
173
|
+
)
|
174
|
+
|
175
|
+
except :ignored_1, :ignored_2
|
176
|
+
|
177
|
+
gateway StageGateway
|
178
|
+
|
179
|
+
before_sync do |source|
|
180
|
+
source.local_attrs[:name] != 'ignored'
|
181
|
+
end
|
182
|
+
end
|
183
|
+
```
|
184
|
+
|
185
|
+
### Gateways vs `fetch` & `find` in synchronizers
|
186
|
+
|
187
|
+
TDOO: Blah blah blah... Need to describe the difference & use cases.
|
188
|
+
|
189
|
+
### Blah blah
|
190
|
+
|
191
|
+
### Blah blah
|
192
|
+
|
193
|
+
```ruby
|
194
|
+
class TournamentSynchronizer < Synchronisable::Synchronizer
|
59
195
|
mappings(
|
60
196
|
:t => :title,
|
61
197
|
:c => :content
|
62
198
|
)
|
63
199
|
|
200
|
+
remote_id :p_id
|
201
|
+
|
202
|
+
# Local attributes to ignore.
|
203
|
+
# These will not be set on your local record.
|
64
204
|
except :ignored_attr1, :ignored_attr42
|
65
205
|
|
206
|
+
# Declares that we want to sync comments after syncing this model.
|
207
|
+
# The resulting hash with remote attributes should contain `comment_ids`
|
66
208
|
has_many :comments
|
67
209
|
|
210
|
+
# Method that will be used to fetch all of the remote entities
|
68
211
|
fetch do
|
69
|
-
# return array of hashes with
|
70
|
-
|
212
|
+
# Somehow get and return an array of hashes with remote entity attibutes
|
213
|
+
[
|
214
|
+
{ t: 'first', c: 'i am the first post' },
|
215
|
+
{ t: 'second', c: 'content of the second post' }
|
216
|
+
]
|
71
217
|
end
|
72
218
|
|
219
|
+
# This method should return only one hash for the given id
|
73
220
|
find do |id|
|
74
|
-
# return a hash with
|
75
|
-
#
|
221
|
+
# return a hash with with remote entity attributes
|
222
|
+
# ...
|
76
223
|
end
|
77
224
|
|
78
|
-
#
|
79
|
-
|
225
|
+
#
|
80
226
|
before_record_sync do |source|
|
227
|
+
# return false if you want to skip syncing of this particular record
|
81
228
|
# ...
|
82
229
|
end
|
83
230
|
|
@@ -95,6 +242,7 @@ class PostSynchronizer < Synchronisable::Synchronizer
|
|
95
242
|
|
96
243
|
before_sync do |source|
|
97
244
|
# ...
|
245
|
+
# return false if you want to skip syncing of this particular record
|
98
246
|
end
|
99
247
|
|
100
248
|
after_sync do |source|
|
@@ -102,6 +250,8 @@ class PostSynchronizer < Synchronisable::Synchronizer
|
|
102
250
|
end
|
103
251
|
end
|
104
252
|
|
253
|
+
|
254
|
+
|
105
255
|
class MyCommentSynchronizer < Synchronisable::Synchronizer
|
106
256
|
remote_id :c_id
|
107
257
|
|
@@ -0,0 +1,21 @@
|
|
1
|
+
module Synchronisable
|
2
|
+
class Configuration
|
3
|
+
include ActiveSupport::Configurable
|
4
|
+
|
5
|
+
config_accessor :models do
|
6
|
+
{}
|
7
|
+
end
|
8
|
+
config_accessor :logging do
|
9
|
+
default_logger = -> { Logger.new(STDOUT) }
|
10
|
+
rails_logger = -> { Rails.logger || default_logger.() }
|
11
|
+
|
12
|
+
logger = defined?(Rails) ? rails_logger.() : default_logger.()
|
13
|
+
|
14
|
+
{
|
15
|
+
:logger => logger,
|
16
|
+
:verbose => true,
|
17
|
+
:colorize => true
|
18
|
+
}
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
data/lib/synchronisable/model.rb
CHANGED
@@ -33,6 +33,12 @@ module Synchronisable
|
|
33
33
|
class_attribute :synchronizer
|
34
34
|
has_one :import, as: :synchronisable, class_name: 'Synchronisable::Import'
|
35
35
|
|
36
|
+
scope :without_import, -> {
|
37
|
+
includes(:import)
|
38
|
+
.where(imports: { synchronisable_id: nil })
|
39
|
+
.references(:imports)
|
40
|
+
}
|
41
|
+
|
36
42
|
set_defaults(args)
|
37
43
|
end
|
38
44
|
|
@@ -27,7 +27,7 @@ module Synchronisable
|
|
27
27
|
source.try(:with_indifferent_access)
|
28
28
|
}
|
29
29
|
|
30
|
-
# Attributes
|
30
|
+
# Attributes to ignored.
|
31
31
|
attribute :except, converter: SYMBOL_ARRAY_CONVERTER
|
32
32
|
|
33
33
|
# The only attributes that will be used.
|
@@ -39,7 +39,7 @@ module Synchronisable
|
|
39
39
|
|
40
40
|
# Logger that will be used during synchronization
|
41
41
|
# of this particular model.
|
42
|
-
attribute :logger, default: -> { Synchronisable.logging[:logger] }
|
42
|
+
attribute :logger, default: -> { Synchronisable.config.logging[:logger] }
|
43
43
|
|
44
44
|
# Gateway to be used to get the remote data
|
45
45
|
#
|
data/lib/synchronisable.rb
CHANGED
@@ -11,44 +11,39 @@ require 'active_support/concern'
|
|
11
11
|
require 'synchronisable/bootstrap/i18n'
|
12
12
|
|
13
13
|
require 'synchronisable/version'
|
14
|
+
require 'synchronisable/configuration'
|
14
15
|
require 'synchronisable/models/import'
|
15
16
|
require 'synchronisable/synchronizer'
|
16
17
|
require 'synchronisable/model'
|
17
18
|
require 'synchronisable/gateway'
|
18
19
|
|
19
|
-
|
20
20
|
module Synchronisable
|
21
|
-
|
22
|
-
|
23
|
-
config_accessor :models do
|
24
|
-
{}
|
21
|
+
def self.config
|
22
|
+
@configuration ||= Configuration.new
|
25
23
|
end
|
26
|
-
config_accessor :logging do
|
27
|
-
default_logger = -> { Logger.new(STDOUT) }
|
28
|
-
rails_logger = -> { Rails.logger || default_logger.() }
|
29
|
-
|
30
|
-
logger = defined?(Rails) ? rails_logger.() : default_logger.()
|
31
24
|
|
32
|
-
|
33
|
-
|
34
|
-
:verbose => true,
|
35
|
-
:colorize => true
|
36
|
-
}
|
25
|
+
def self.configure
|
26
|
+
yield config
|
37
27
|
end
|
38
28
|
|
39
29
|
# Syncs models that are defined in {Synchronisable#models}
|
40
30
|
#
|
41
|
-
# @
|
42
|
-
#
|
43
|
-
#
|
44
|
-
#
|
31
|
+
# @overload sync(models, options)
|
32
|
+
# @param models [Array] array of models that should be synchronized.
|
33
|
+
# This take a precedence over models defined in {Synchronisable#models}.
|
34
|
+
# If this parameter is not specified and {Synchronisable#models} is empty,
|
35
|
+
# than it will try to sync only those models which have a corresponding synchronizers
|
36
|
+
# @param options [Hash] options that will be passed to controller
|
37
|
+
# @overload sync(models)
|
38
|
+
# @overlaod sync(options)
|
45
39
|
#
|
46
40
|
# @return [Array<[Synchronisable::Context]>] array of synchronization contexts
|
47
41
|
#
|
48
42
|
# @see Synchronisable::Context
|
49
|
-
def self.sync(*
|
50
|
-
|
51
|
-
source
|
43
|
+
def self.sync(*args)
|
44
|
+
options = args.extract_options!
|
45
|
+
source = source_models(args)
|
46
|
+
source.map { |model| model.sync(options) }
|
52
47
|
end
|
53
48
|
|
54
49
|
private
|
@@ -59,7 +54,7 @@ module Synchronisable
|
|
59
54
|
end
|
60
55
|
|
61
56
|
def self.default_models
|
62
|
-
models.map(&:safe_constantize).compact
|
57
|
+
config.models.map(&:safe_constantize).compact
|
63
58
|
end
|
64
59
|
|
65
60
|
def self.find_models
|
@@ -9,7 +9,7 @@ describe Synchronisable do
|
|
9
9
|
describe 'models specified in configuration' do
|
10
10
|
context 'only Team and Match' do
|
11
11
|
before :all do
|
12
|
-
Synchronisable.models = %w(Match Team)
|
12
|
+
Synchronisable.config.models = %w(Match Team)
|
13
13
|
end
|
14
14
|
|
15
15
|
it { is_expected.to change { Match.count }.by(1) }
|
@@ -22,7 +22,7 @@ describe Synchronisable do
|
|
22
22
|
|
23
23
|
context 'all' do
|
24
24
|
before :all do
|
25
|
-
Synchronisable.models = %w(
|
25
|
+
Synchronisable.config.models = %w(
|
26
26
|
Tournament Team
|
27
27
|
Match MatchPlayer Player
|
28
28
|
)
|
@@ -40,7 +40,7 @@ describe Synchronisable do
|
|
40
40
|
|
41
41
|
context 'when models setting is overriden in method call' do
|
42
42
|
before :all do
|
43
|
-
Synchronisable.models = %w(Team Match)
|
43
|
+
Synchronisable.config.models = %w(Team Match)
|
44
44
|
end
|
45
45
|
|
46
46
|
subject do
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: synchronisable
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.1.
|
4
|
+
version: 1.1.6
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Vasiliy Yorkin
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-
|
11
|
+
date: 2014-09-01 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|