multiwoven-integrations 0.1.18 → 0.1.20
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +19 -0
- data/README.md +84 -22
- data/lib/multiwoven/integrations/destination/facebook_custom_audience/client.rb +25 -28
- data/lib/multiwoven/integrations/rollout.rb +1 -1
- metadata +6 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 97435b81707f9ccad9b5240681e74028d231fd235fbd9ec6c4b6a24f52b1361e
|
4
|
+
data.tar.gz: 686d26cc447f1876009c520262cdd3846cfc0410a4ce1cdbbc3e3ec446ddc966
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: acaeea9547c243fd5d9901d1247d046807a600f69cc461df0462b718fb759e10d767faa8b2e9dd44f7d64740137a8a9148084e3e66101f6383458a54bf99576d
|
7
|
+
data.tar.gz: 650b6a32b16d3b8dd50bb88a4df44aebf49071539b6dfa57f82cb38ad1539d4ea154ace486afbf9e863b37301cd836c3503cd897f06e3a2b31a515ba2e1f8c09
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,24 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
+
## [0.3.0](https://www.github.com/Multiwoven/multiwoven-integrations/compare/v0.2.0...v0.3.0) (2024-02-14)
|
4
|
+
|
5
|
+
|
6
|
+
### Features
|
7
|
+
|
8
|
+
* Publish gem to rubygems ([#60](https://www.github.com/Multiwoven/multiwoven-integrations/issues/60)) ([04cfbb6](https://www.github.com/Multiwoven/multiwoven-integrations/commit/04cfbb60c78bc90d886ce691d5094fed9c6f078f))
|
9
|
+
|
10
|
+
## [0.2.0](https://www.github.com/Multiwoven/multiwoven-integrations/compare/v0.1.0...v0.2.0) (2024-02-14)
|
11
|
+
|
12
|
+
|
13
|
+
### Features
|
14
|
+
|
15
|
+
* Automate gem release ([#55](https://www.github.com/Multiwoven/multiwoven-integrations/issues/55)) ([1c3373b](https://www.github.com/Multiwoven/multiwoven-integrations/commit/1c3373bcedae60c6fb8171c54ce40de94f1537ce))
|
16
|
+
|
17
|
+
|
18
|
+
### Bug Fixes
|
19
|
+
|
20
|
+
* Klaviyo discover ([#23](https://www.github.com/Multiwoven/multiwoven-integrations/issues/23)) ([7c43db6](https://www.github.com/Multiwoven/multiwoven-integrations/commit/7c43db6a369403748ed29085bb5254405ad1c95e))
|
21
|
+
|
3
22
|
## [0.1.0] - 2023-12-01
|
4
23
|
|
5
24
|
- Initial release
|
data/README.md
CHANGED
@@ -1,43 +1,105 @@
|
|
1
|
-
|
1
|
+
<div align="center">
|
2
|
+
<a href="https://multiwoven.com?utm_source=github" target="_blank">
|
3
|
+
<img src="https://res.cloudinary.com/dspflukeu/image/upload/v1706696350/Multiwoven/Logo_hrkaxj.png" alt="Multiwoven" width="240" />
|
4
|
+
</a>
|
5
|
+
</div>
|
6
|
+
<br />
|
7
|
+
<p align="center">
|
8
|
+
<a href="http://badge.fury.io/rb/multiwoven-integrations">
|
9
|
+
<img src="https://badge.fury.io/rb/multiwoven-integrations.svg" alt="Gem Version">
|
10
|
+
</a>
|
11
|
+
<a href="https://github.com/Multiwoven/multiwoven-integrations/actions/workflows/ci.yml">
|
12
|
+
<img src="https://github.com/Multiwoven/multiwoven-integrations/actions/workflows/ci.yml/badge.svg" alt="CI">
|
13
|
+
</a>
|
14
|
+
<a href="https://codeclimate.com/repos/657d0a2a60265a2f2155ffca/maintainability">
|
15
|
+
<img src="https://api.codeclimate.com/v1/badges/d841270f1f7a966043c1/maintainability" alt="Maintainability">
|
16
|
+
</a>
|
17
|
+
<a href="https://codeclimate.com/repos/657d0a2a60265a2f2155ffca/test_coverage">
|
18
|
+
<img src="https://api.codeclimate.com/v1/badges/d841270f1f7a966043c1/test_coverage" alt="Test Coverage">
|
19
|
+
</a>
|
20
|
+
</p>
|
21
|
+
<h2 align="center">The open-source Reverse ETL platform for data teams</h2>
|
22
|
+
|
23
|
+
<p align="center">
|
24
|
+
<br />
|
25
|
+
<a href="https://docs.multiwoven.com" rel="">
|
26
|
+
<strong>Explore the docs »</strong>
|
27
|
+
</a>
|
28
|
+
<br />
|
29
|
+
<br />
|
30
|
+
<a href="https://join.slack.com/t/multiwoven/shared_invite/zt-2bnjye26u-~lu_FFOMLpChOYxvovep7g">Slack</a>
|
31
|
+
·
|
32
|
+
<a href="https://multiwoven.com">Website</a>
|
33
|
+
·
|
34
|
+
<a href="https://github.com/Multiwoven/multiwoven-integrations/issues">Report Issues</a>
|
35
|
+
·
|
36
|
+
<a href="https://github.com/orgs/Multiwoven/projects/4">Roadmap</a>
|
37
|
+
</p>
|
38
|
+
|
39
|
+
## Multiwoven Integrations
|
40
|
+
|
41
|
+
Multiwoven integrations is the collection of connectors built on top of [Multiwoven protocol](https://docs.multiwoven.com/guides/architecture/multiwoven-protocol).
|
42
|
+
|
43
|
+
Multiwoven protocol is an open source standard for moving data between data sources to any third-part destinations.
|
44
|
+
Anyone can build a connetor with basic ruby knowledge using the protocol.
|
45
|
+
|
46
|
+
### Installation
|
2
47
|
|
3
|
-
|
4
|
-
[![Maintainability](https://api.codeclimate.com/v1/badges/d841270f1f7a966043c1/maintainability)](https://codeclimate.com/repos/657d0a2a60265a2f2155ffca/maintainability)
|
5
|
-
[![Test Coverage](https://api.codeclimate.com/v1/badges/d841270f1f7a966043c1/test_coverage)](https://codeclimate.com/repos/657d0a2a60265a2f2155ffca/test_coverage)
|
48
|
+
Install the gem and add to the application's Gemfile by executing:
|
6
49
|
|
7
|
-
|
50
|
+
$ bundle add multiwoven-integrations
|
8
51
|
|
9
|
-
Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/multiwoven/integrations`. To experiment with that code, run `bin/console` for an interactive prompt.
|
10
52
|
|
11
|
-
|
53
|
+
### Usage
|
12
54
|
|
13
|
-
|
55
|
+
#### Source
|
56
|
+
```
|
57
|
+
source = Multiwoven::Integrations::Source::[CONNECTOR_NAME]::Client.new
|
58
|
+
source.read(sync_config)
|
59
|
+
```
|
60
|
+
#### Destination
|
14
61
|
|
15
|
-
|
62
|
+
```
|
63
|
+
destination = Multiwoven::Integrations::Destination::[CONNECTOR_NAME]::Client.new
|
64
|
+
destination.write(sync_config, records)
|
65
|
+
```
|
16
66
|
|
17
|
-
|
67
|
+
#### Supported methods
|
68
|
+
Please refer [Multiwoven Protocol](https://docs.multiwoven.com/guides/architecture/multiwoven-protocol) to understand more about the supported methods on source and destination
|
18
69
|
|
19
|
-
|
70
|
+
## Development
|
20
71
|
|
21
|
-
|
72
|
+
- **Install Dependencies**
|
73
|
+
- Command: `bin/setup`
|
74
|
+
- Description: After checking out the repo, run this command to install dependencies.
|
22
75
|
|
23
|
-
|
76
|
+
- **Run Tests**
|
77
|
+
- Command: `rake spec`
|
78
|
+
- Description: Run this command to execute the tests.
|
24
79
|
|
25
|
-
|
80
|
+
- **Interactive Prompt**
|
81
|
+
- Command: `bin/console`
|
82
|
+
- Description: For an interactive prompt that allows you to experiment, run this command.
|
26
83
|
|
27
|
-
|
84
|
+
- **Install Gem Locally**
|
85
|
+
- Command: `bundle exec rake install`
|
86
|
+
- Description: To install this gem onto your local machine, run this command.
|
28
87
|
|
29
|
-
|
88
|
+
- **Release New Version**
|
89
|
+
- Steps:
|
90
|
+
1. Update the version number in `rollout.rb`.
|
91
|
+
2. Command: `bundle exec rake release`
|
92
|
+
3. Description: This command will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
30
93
|
|
31
|
-
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
32
94
|
|
33
95
|
## Contributing
|
34
96
|
|
35
|
-
Bug reports and pull requests are welcome on GitHub
|
97
|
+
Bug reports and pull requests are welcome on [GitHub](https://github.com/Multiwoven/multiwoven-integrations). This project aims to be a safe, welcoming space for collaboration.
|
36
98
|
|
37
|
-
##
|
99
|
+
## Code of Conduct
|
38
100
|
|
39
|
-
|
101
|
+
Contributors are expected to adhere to the project's [code of conduct](https://github.com/Multiwoven/multiwoven-integrations/blob/main/CODE_OF_CONDUCT.md)
|
40
102
|
|
41
|
-
##
|
103
|
+
## License
|
42
104
|
|
43
|
-
|
105
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
@@ -3,7 +3,8 @@
|
|
3
3
|
module Multiwoven::Integrations::Destination
|
4
4
|
module FacebookCustomAudience
|
5
5
|
include Multiwoven::Integrations::Core
|
6
|
-
class Client < DestinationConnector
|
6
|
+
class Client < DestinationConnector # rubocop:disable Metrics/ClassLength
|
7
|
+
MAX_CHUNK_SIZE = 10_000
|
7
8
|
def check_connection(connection_config)
|
8
9
|
connection_config = connection_config.with_indifferent_access
|
9
10
|
access_token = connection_config[:access_token]
|
@@ -54,8 +55,8 @@ module Multiwoven::Integrations::Destination
|
|
54
55
|
url = generate_url(sync_config, connection_config)
|
55
56
|
write_success = 0
|
56
57
|
write_failure = 0
|
57
|
-
records.
|
58
|
-
payload = create_payload(
|
58
|
+
records.each_slice(MAX_CHUNK_SIZE) do |chunk|
|
59
|
+
payload = create_payload(chunk, sync_config.stream.json_schema.with_indifferent_access)
|
59
60
|
response = Multiwoven::Integrations::Core::HttpClient.request(
|
60
61
|
url,
|
61
62
|
sync_config.stream.request_method,
|
@@ -63,28 +64,22 @@ module Multiwoven::Integrations::Destination
|
|
63
64
|
headers: auth_headers(access_token)
|
64
65
|
)
|
65
66
|
if success?(response)
|
66
|
-
write_success +=
|
67
|
+
write_success += chunk.size
|
67
68
|
else
|
68
|
-
write_failure +=
|
69
|
+
write_failure += chunk.size
|
69
70
|
end
|
70
71
|
rescue StandardError => e
|
71
|
-
|
72
|
-
|
73
|
-
)
|
74
|
-
write_failure += 1
|
72
|
+
handle_exception("FACEBOOK:RECORD:WRITE:EXCEPTION", "error", e)
|
73
|
+
write_failure += chunk.size
|
75
74
|
end
|
75
|
+
|
76
76
|
tracker = Multiwoven::Integrations::Protocol::TrackingMessage.new(
|
77
77
|
success: write_success,
|
78
78
|
failed: write_failure
|
79
79
|
)
|
80
80
|
tracker.to_multiwoven_message
|
81
81
|
rescue StandardError => e
|
82
|
-
|
83
|
-
handle_exception(
|
84
|
-
"FACEBOOK:WRITE:EXCEPTION",
|
85
|
-
"error",
|
86
|
-
e
|
87
|
-
)
|
82
|
+
handle_exception("FACEBOOK:WRITE:EXCEPTION", "error", e)
|
88
83
|
end
|
89
84
|
|
90
85
|
private
|
@@ -93,28 +88,30 @@ module Multiwoven::Integrations::Destination
|
|
93
88
|
sync_config.stream.url.gsub("{audience_id}", connection_config[:audience_id])
|
94
89
|
end
|
95
90
|
|
96
|
-
def create_payload(
|
97
|
-
schema, data = extract_schema_and_data(
|
91
|
+
def create_payload(records, json_schema)
|
92
|
+
schema, data = extract_schema_and_data(records, json_schema)
|
98
93
|
{
|
99
94
|
"payload" => {
|
100
95
|
"schema" => schema,
|
101
|
-
"data" =>
|
96
|
+
"data" => data
|
102
97
|
}
|
103
98
|
}
|
104
99
|
end
|
105
100
|
|
106
|
-
def extract_schema_and_data(
|
101
|
+
def extract_schema_and_data(records, json_schema)
|
107
102
|
schema_properties = json_schema[:properties]
|
108
|
-
schema =
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
103
|
+
schema = records.first.keys.map(&:to_s).map(&:upcase)
|
104
|
+
data = []
|
105
|
+
records.each do |record|
|
106
|
+
encrypted_data_array = []
|
107
|
+
record.with_indifferent_access.each do |key, value|
|
108
|
+
schema_key = key.upcase
|
109
|
+
encrypted_value = schema_properties[schema_key] && schema_properties[schema_key]["x-hashRequired"] ? Digest::SHA256.hexdigest(value.to_s) : value
|
110
|
+
encrypted_data_array << encrypted_value
|
111
|
+
end
|
112
|
+
data << encrypted_data_array
|
115
113
|
end
|
116
|
-
|
117
|
-
[schema, encrypted_data_array]
|
114
|
+
[schema, data]
|
118
115
|
end
|
119
116
|
|
120
117
|
def auth_headers(access_token)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: multiwoven-integrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.20
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Subin T P
|
8
|
-
autorequire:
|
8
|
+
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-02-
|
11
|
+
date: 2024-02-14 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -347,7 +347,7 @@ metadata:
|
|
347
347
|
homepage_uri: https://www.multiwoven.com/
|
348
348
|
source_code_uri: https://github.com/Multiwoven/multiwoven-integrations
|
349
349
|
changelog_uri: https://github.com/Multiwoven/multiwoven-integrations/blob/master/CHANGELOG.md
|
350
|
-
post_install_message:
|
350
|
+
post_install_message:
|
351
351
|
rdoc_options: []
|
352
352
|
require_paths:
|
353
353
|
- lib
|
@@ -362,8 +362,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
362
362
|
- !ruby/object:Gem::Version
|
363
363
|
version: '0'
|
364
364
|
requirements: []
|
365
|
-
rubygems_version: 3.
|
366
|
-
signing_key:
|
365
|
+
rubygems_version: 3.4.1
|
366
|
+
signing_key:
|
367
367
|
specification_version: 4
|
368
368
|
summary: Integration suite for open source reverse ETL platform
|
369
369
|
test_files: []
|