decidim-cdtb 0.1.2 → 0.1.4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.rubocop.yml +1 -1
- data/CHANGELOG.md +9 -1
- data/Gemfile +1 -0
- data/Gemfile.lock +4 -2
- data/README.md +59 -23
- data/lib/decidim/cdtb/engine.rb +5 -1
- data/lib/decidim/cdtb/multitenants/org_by_host_like.rb +4 -4
- data/lib/decidim/cdtb/spam/user_spam_detector.rb +86 -0
- data/lib/decidim/cdtb/task.rb +3 -3
- data/lib/decidim/cdtb/tasks.rb +2 -0
- data/lib/decidim/cdtb/tasks_utils.rb +20 -6
- data/lib/decidim/cdtb/upgrades/install_gem_migrations_step.rb +18 -0
- data/lib/decidim/cdtb/upgrades/upgrade_modules_task.rb +42 -0
- data/lib/decidim/cdtb/upgrades/validate_migrations_task.rb +60 -0
- data/lib/decidim/cdtb/version.rb +1 -1
- data/lib/decidim/cdtb.rb +8 -0
- data/lib/generators/cdtb/USAGE +8 -0
- data/lib/generators/cdtb/templates/validate_migrations.yml +52 -0
- data/lib/generators/cdtb/validate_migrations_ci_generator.rb +12 -0
- data/lib/tasks/spam.rake +13 -0
- data/lib/tasks/upgrade.rake +20 -0
- metadata +11 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 515d740942f3089a3cb906fab890191dc4b19f97c98ccff17fac5aac2aa83d9e
|
4
|
+
data.tar.gz: 557fa8a269b25d0ac8b15c522d16072600c0b5a0d39fee2250cf9c80299486ab
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 519c7234a2cff3adba2732e07e4fb9c2aa111fb0803ac8cac8180a9683c501676bb2f0aa96059f1a5cedd063db98471f70e1321d33be0e8db4a9947341df3b6a
|
7
|
+
data.tar.gz: be0c056d4b91a543a528628b4f820fd06a7913899c12f3fa4e5db646ddbd1599970fcba41d67d583472b53abe2eef447e8b0d4aede8d83222617e3f3d80f9df8
|
data/.rubocop.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,6 +1,14 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
-
## [0.1.
|
3
|
+
## [0.1.4] - 2024-01-30 (Peus grans com gegants)
|
4
|
+
|
5
|
+
- Add users spam detector task
|
6
|
+
|
7
|
+
## [0.1.3] - 2023-06-23 (Tan iguals com especials)
|
8
|
+
|
9
|
+
- Validate migrations task
|
10
|
+
|
11
|
+
## [0.1.2] - 2023-06-02 (Empastifada amb melmelada)
|
4
12
|
|
5
13
|
- Add anonymize rake task
|
6
14
|
|
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
decidim-cdtb (0.1.
|
4
|
+
decidim-cdtb (0.1.4)
|
5
5
|
decidim (>= 0.26.2)
|
6
6
|
rails (>= 6)
|
7
7
|
ruby-progressbar
|
@@ -700,6 +700,7 @@ GEM
|
|
700
700
|
actionpack (>= 5.2)
|
701
701
|
activesupport (>= 5.2)
|
702
702
|
sprockets (>= 3.0.0)
|
703
|
+
sqlite3 (1.6.3-x86_64-linux)
|
703
704
|
ssrf_filter (1.1.1)
|
704
705
|
system_test_html_screenshots (0.2.0)
|
705
706
|
actionpack (>= 5.2, < 6.1.a)
|
@@ -771,6 +772,7 @@ DEPENDENCIES
|
|
771
772
|
rake (~> 13.0)
|
772
773
|
rspec (~> 3.0)
|
773
774
|
rubocop
|
775
|
+
sqlite3
|
774
776
|
|
775
777
|
BUNDLED WITH
|
776
|
-
2.
|
778
|
+
2.4.22
|
data/README.md
CHANGED
@@ -8,16 +8,16 @@ This is CodiTramuntana's Decidim Toolbelt (cdtb), a gem to help managing Decidim
|
|
8
8
|
Install the gem and add to the application's Gemfile by executing:
|
9
9
|
|
10
10
|
$ bundle add decidim-cdtb
|
11
|
+
$ bundle install
|
11
12
|
|
12
|
-
|
13
|
+
Install CI tests on your app with:
|
13
14
|
|
14
|
-
$
|
15
|
+
$ bin/rails generate cdtb:validate_migrations_ci
|
15
16
|
|
16
|
-
## Usage
|
17
17
|
|
18
|
-
|
18
|
+
## Usage
|
19
19
|
|
20
|
-
|
20
|
+
### Organizations information
|
21
21
|
|
22
22
|
Returns information regarding the organizations in a multitenant installation that match a search term ignorecase.
|
23
23
|
|
@@ -36,7 +36,7 @@ bin/rake cdtb:org_by_host_like[vila]
|
|
36
36
|
host: localhost, time_zone: Madrid, locales: ca + [ca, es, oc], available authorizations: [postal_letter, members_picker_authorization_handler]
|
37
37
|
```
|
38
38
|
|
39
|
-
|
39
|
+
### Fix nicknames
|
40
40
|
|
41
41
|
In a previous version than Decidim v0.25 a validation to the `Decidim::User.nickname` was added with a migration to fix existing nicknames. But the migration was only taking into acocunt managed (impersonated) users.
|
42
42
|
|
@@ -48,27 +48,21 @@ To execute the task run:
|
|
48
48
|
bin/rake cdtb:fix_nicknames
|
49
49
|
```
|
50
50
|
|
51
|
-
|
51
|
+
### Anonymize production dump
|
52
52
|
|
53
53
|
Anonymize rake task was taken from https://github.com/AjuntamentdeBarcelona/decidim-barcelona
|
54
54
|
|
55
55
|
Available rake tasks:
|
56
56
|
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
```bin/rake cdtb:anonymize:user_groups``` anonymizes user groups
|
66
|
-
|
67
|
-
```bin/rake cdtb:anonymize:system_admins``` anonymizes system admins
|
57
|
+
- `bin/rake cdtb:anonymize:check` allows you to check if you can anonymize production dump
|
58
|
+
- `bin/rake cdtb:anonymize:all` anonymizes whole production dump (without proposals)
|
59
|
+
- `bin/rake cdtb:anonymize:users` anonymizes users
|
60
|
+
- `bin/rake cdtb:anonymize:proposals` anonymizes proposals
|
61
|
+
- `bin/rake cdtb:anonymize:user_groups` anonymizes user groups
|
62
|
+
- `bin/rake cdtb:anonymize:system_admins` anonymizes system admins
|
63
|
+
- `bin/rake cdtb:anonymize:paper_trail` anonymizes paper trails
|
68
64
|
|
69
|
-
|
70
|
-
|
71
|
-
#### Migrate ActiveStorage service from S3 to local
|
65
|
+
### Migrate ActiveStorage service from S3 to local
|
72
66
|
|
73
67
|
To migrate from S3 to local storage, the identified steps will be:
|
74
68
|
|
@@ -82,6 +76,48 @@ To migrate from S3 to local storage, the identified steps will be:
|
|
82
76
|
`bin/rake cache:clear`
|
83
77
|
5. Restart the Rails server
|
84
78
|
|
79
|
+
### Detect spam
|
80
|
+
|
81
|
+
To detect spam in Decidim.
|
82
|
+
|
83
|
+
#### Detect spam users
|
84
|
+
Detects users susceptible of being spammers. It can run on all organizations or be scoped to a single organization by passing the organization ID as the rake task parameter.
|
85
|
+
|
86
|
+
This rake task export a .csv with a list of all the searched users. A column indicates if each user is suspicious of being a spammer or not.
|
87
|
+
The columns in the CSV are: "ID, "Is suspicious?", "Name", "Email", "Nickname", "Personal URL", "About"
|
88
|
+
|
89
|
+
Examples:
|
90
|
+
`bin/rake cdtb:spam:users[org_id]` --> find users in organization with an id.
|
91
|
+
`bin/rake cdtb:spam:users` --> find all users in all organizations.
|
92
|
+
|
93
|
+
To set custom words in the rake, you can override it with an initalizer:
|
94
|
+
|
95
|
+
```
|
96
|
+
Decidim::Cdtb.configure do |config|
|
97
|
+
config.spam_words = ENV["CDTB_SPAM_WORDS"]&.split(",")
|
98
|
+
end
|
99
|
+
```
|
100
|
+
|
101
|
+
### Upgrades:
|
102
|
+
|
103
|
+
#### Upgrade modules
|
104
|
+
|
105
|
+
Upgrades the gems with engines in them. All, Decidim modules and standard Rails engines.
|
106
|
+
|
107
|
+
TO-DO To be finished
|
108
|
+
|
109
|
+
#### Validate migrations
|
110
|
+
|
111
|
+
Validates that migrations from all gems in the Gemfile have already been installed.
|
112
|
+
|
113
|
+
```
|
114
|
+
bin/rake cdtb:upgrades:validate_migrations
|
115
|
+
```
|
116
|
+
|
117
|
+
See the [Installation](#installation) chapter to install a GitHub Action on your app that will run this validation on your CI.
|
118
|
+
|
119
|
+
TO-DO also check that all migrations have been executed and the schema.rb does not change
|
120
|
+
|
85
121
|
## Development
|
86
122
|
|
87
123
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
@@ -107,7 +143,7 @@ bundle exec rspec spec
|
|
107
143
|
|
108
144
|
## Contributing
|
109
145
|
|
110
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
146
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/CodiTramuntana/decidim-cdtb. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/CodiTramuntana/decidim-cdtb/blob/master/CODE_OF_CONDUCT.md).
|
111
147
|
|
112
148
|
## License
|
113
149
|
|
@@ -115,4 +151,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
|
|
115
151
|
|
116
152
|
## Code of Conduct
|
117
153
|
|
118
|
-
Everyone interacting in the Decidim::Cdtb project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/
|
154
|
+
Everyone interacting in the Decidim::Cdtb project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/CodiTramuntana/decidim-cdtb/blob/master/CODE_OF_CONDUCT.md).
|
data/lib/decidim/cdtb/engine.rb
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
module Decidim
|
4
4
|
module Cdtb
|
5
|
-
#
|
5
|
+
# This module's engine
|
6
6
|
class Engine < ::Rails::Engine
|
7
7
|
isolate_namespace Decidim::Cdtb
|
8
8
|
|
@@ -10,6 +10,10 @@ module Decidim
|
|
10
10
|
# Workaround for https://stackoverflow.com/questions/72970170/upgrading-to-rails-6-1-6-1-causes-psychdisallowedclass-tried-to-load-unspecif
|
11
11
|
Rails.application.config.active_record.use_yaml_unsafe_load = true
|
12
12
|
end
|
13
|
+
|
14
|
+
config.after_initialize do
|
15
|
+
Decidim::Cdtb.config.spam_regexp = Regexp.union(Decidim::Cdtb.config.spam_words)
|
16
|
+
end
|
13
17
|
end
|
14
18
|
end
|
15
19
|
end
|
@@ -30,9 +30,9 @@ module Decidim
|
|
30
30
|
locales: "#{org.default_locale} + [#{org.available_locales&.join(", ")}]",
|
31
31
|
available_authorizations: org.available_authorizations&.join(", ")
|
32
32
|
}
|
33
|
-
|
33
|
+
do_log_info(h.to_yaml)
|
34
34
|
end
|
35
|
-
|
35
|
+
do_log_info("---------------------------------------------------------")
|
36
36
|
end
|
37
37
|
end
|
38
38
|
|
@@ -47,7 +47,7 @@ module Decidim
|
|
47
47
|
end
|
48
48
|
|
49
49
|
def show_full_info(org)
|
50
|
-
|
50
|
+
do_log_info(org.attributes.to_yaml)
|
51
51
|
end
|
52
52
|
|
53
53
|
def show_summary_info(org)
|
@@ -57,7 +57,7 @@ module Decidim
|
|
57
57
|
locales: "#{org.default_locale} + [#{org.available_locales&.join(", ")}]",
|
58
58
|
available_authorizations: org.available_authorizations&.join(", ")
|
59
59
|
}
|
60
|
-
|
60
|
+
do_log_info(h.to_yaml)
|
61
61
|
end
|
62
62
|
end
|
63
63
|
end
|
@@ -0,0 +1,86 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "csv"
|
4
|
+
|
5
|
+
module Decidim
|
6
|
+
module Cdtb
|
7
|
+
module Spam
|
8
|
+
# Detect spam behavior in users
|
9
|
+
#
|
10
|
+
class UserSpamDetector < ::Decidim::Cdtb::Task
|
11
|
+
# rubocop:disable Style/RedundantRegexpEscape
|
12
|
+
URL_REGEX = %r{(https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|
|
13
|
+
www\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|https?:\/\/(?:www\.|
|
14
|
+
(?!www))[a-zA-Z0-9]+\.[^\s]{2,}|www\.[a-zA-Z0-9]+\.[^\s]{2,})}.freeze
|
15
|
+
# rubocop:enable Style/RedundantRegexpEscape
|
16
|
+
|
17
|
+
def initialize(organization = nil)
|
18
|
+
@organization = organization
|
19
|
+
progress_bar = { title: "Decidim::User" }
|
20
|
+
super("SPAM DETECTOR", progress_bar: progress_bar)
|
21
|
+
end
|
22
|
+
|
23
|
+
def prepare_execution(_ctx)
|
24
|
+
@users = if @organization.present?
|
25
|
+
Decidim::User.where(organization: @organization)
|
26
|
+
else
|
27
|
+
Decidim::User.all
|
28
|
+
end
|
29
|
+
|
30
|
+
@num_users = @users.count
|
31
|
+
log_task_info("Checking #{@num_users} users...")
|
32
|
+
end
|
33
|
+
|
34
|
+
def total_items
|
35
|
+
@num_users
|
36
|
+
end
|
37
|
+
|
38
|
+
def do_execution(context)
|
39
|
+
progress_bar = context[:progress_bar]
|
40
|
+
|
41
|
+
CSV.open("spam_users.csv", "w") do |csv|
|
42
|
+
csv_headers = ["ID", "Is suspicious?", "Name", "Email", "Nickname", "Personal URL", "About"]
|
43
|
+
csv << csv_headers
|
44
|
+
|
45
|
+
@users.find_each do |user|
|
46
|
+
suspicious = "NO"
|
47
|
+
|
48
|
+
if spam_user?(user)
|
49
|
+
suspicious = "YES"
|
50
|
+
@num_applied+= 1
|
51
|
+
end
|
52
|
+
|
53
|
+
csv << [user.id, suspicious, user.name, user.email, user.nickname, user.personal_url, user.about]
|
54
|
+
|
55
|
+
progress_bar.increment
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
|
60
|
+
def end_execution(_ctx)
|
61
|
+
if @num_applied.positive?
|
62
|
+
log_task_step("#{@num_applied} suspicious users")
|
63
|
+
log_task_step("Suspicious users list exported to spam_users.csv")
|
64
|
+
else
|
65
|
+
log_task_step("There are not suspicious users!!")
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
def spam_user?(user)
|
70
|
+
has_spam_word?(user) || has_spam_url?(user)
|
71
|
+
end
|
72
|
+
|
73
|
+
private
|
74
|
+
|
75
|
+
def has_spam_word?(user)
|
76
|
+
[user.name, user.about, user.nickname,
|
77
|
+
user.personal_url, user.about].compact.join("||").match?(Decidim::Cdtb.config.spam_regexp)
|
78
|
+
end
|
79
|
+
|
80
|
+
def has_spam_url?(user)
|
81
|
+
!!(user&.about =~ URL_REGEX || user.name =~ URL_REGEX)
|
82
|
+
end
|
83
|
+
end
|
84
|
+
end
|
85
|
+
end
|
86
|
+
end
|
data/lib/decidim/cdtb/task.rb
CHANGED
@@ -22,21 +22,21 @@ module Decidim
|
|
22
22
|
def init
|
23
23
|
log_task_title(@title)
|
24
24
|
@start_time= Time.zone.now
|
25
|
-
|
25
|
+
do_log_info("▶️ Starting at #{@start_time}")
|
26
26
|
end
|
27
27
|
|
28
28
|
def execute!
|
29
29
|
init
|
30
30
|
ctx= {}
|
31
|
-
ctx[:progress_bar]= ProgressBar.create(total: total_items, title: title) if has_progress?
|
32
31
|
prepare_execution(ctx)
|
32
|
+
ctx[:progress_bar]= ProgressBar.create(total: total_items, title: title) if has_progress?
|
33
33
|
do_execution(ctx)
|
34
34
|
end_execution(ctx)
|
35
35
|
finish
|
36
36
|
end
|
37
37
|
|
38
38
|
def finish
|
39
|
-
|
39
|
+
do_log_info("⏱️ Took #{Time.zone.now - @start_time} seconds")
|
40
40
|
log_task_end
|
41
41
|
end
|
42
42
|
|
data/lib/decidim/cdtb/tasks.rb
CHANGED
@@ -4,5 +4,7 @@ require "decidim/cdtb/tasks_utils"
|
|
4
4
|
require "decidim/cdtb/task"
|
5
5
|
require "decidim/cdtb/fixes/nickname_fixer"
|
6
6
|
require "decidim/cdtb/multitenants/org_by_host_like"
|
7
|
+
require "decidim/cdtb/spam/user_spam_detector"
|
7
8
|
require "decidim/cdtb/storage/local_sharding"
|
8
9
|
require "decidim/cdtb/storage/set_local_on_blobs"
|
10
|
+
require "decidim/cdtb/upgrades/validate_migrations_task"
|
@@ -10,26 +10,40 @@ module Decidim
|
|
10
10
|
Rails.logger
|
11
11
|
end
|
12
12
|
|
13
|
-
def
|
13
|
+
def do_log_info(msg)
|
14
14
|
puts msg
|
15
15
|
logger.info(msg)
|
16
16
|
end
|
17
17
|
|
18
|
+
def do_log_error(msg)
|
19
|
+
puts msg
|
20
|
+
logger.error(msg)
|
21
|
+
end
|
22
|
+
|
18
23
|
def log_task_title(title)
|
19
|
-
|
24
|
+
do_log_info("⚙️ #{title}")
|
20
25
|
end
|
21
26
|
|
22
27
|
def log_task_step(description)
|
23
|
-
|
28
|
+
do_log_info("➡️ #{description}")
|
24
29
|
end
|
25
30
|
|
26
31
|
def log_task_info(info)
|
27
|
-
|
32
|
+
do_log_info("ℹ️ #{info}")
|
33
|
+
end
|
34
|
+
|
35
|
+
def log_task_failure(msg)
|
36
|
+
@failed= true
|
37
|
+
do_log_error("⚠️ #{msg}")
|
28
38
|
end
|
29
39
|
|
30
40
|
def log_task_end
|
31
|
-
end_comment=
|
32
|
-
|
41
|
+
end_comment= if defined?(@failed) && @failed
|
42
|
+
"❌ Ended with errors!"
|
43
|
+
else
|
44
|
+
"✅ Done."
|
45
|
+
end
|
46
|
+
do_log_info(end_comment)
|
33
47
|
end
|
34
48
|
end
|
35
49
|
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Decidim
|
4
|
+
module Cdtb
|
5
|
+
module Upgrades
|
6
|
+
# Invokes rails to install gem migrations
|
7
|
+
#
|
8
|
+
class InstallGemMigrationsStep
|
9
|
+
# Pass +gem_names+ to define from which gems to install migrations.
|
10
|
+
def install!(gem_names)
|
11
|
+
cmd= "#{Rails.root.join("bin/rails")} railties:install:migrations"
|
12
|
+
env_vars= "FROM=#{gem_names.join(",")}"
|
13
|
+
`#{cmd} #{env_vars}`
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
@@ -0,0 +1,42 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Decidim
|
4
|
+
module Cdtb
|
5
|
+
module Upgrades
|
6
|
+
# Upgrades the gems with engines in them. All, Decidim modules and standard Rails engines.
|
7
|
+
#
|
8
|
+
class UpgradeModulesTask < ::Decidim::Cdtb::Task
|
9
|
+
def initialize
|
10
|
+
progress_bar= { title: "Modules" }
|
11
|
+
super("UPGRADE MODULES", progress_bar: progress_bar)
|
12
|
+
end
|
13
|
+
|
14
|
+
def prepare_execution(_ctx)
|
15
|
+
log_task_info("Have you updated the versions of your modules in the Gemfile (y/n)?")
|
16
|
+
response= $stdin.gets
|
17
|
+
if response&.downcase == "y"
|
18
|
+
get the number of modules
|
19
|
+
else
|
20
|
+
@exit= true
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
def total_items
|
25
|
+
@num_users
|
26
|
+
end
|
27
|
+
|
28
|
+
def do_execution(context)
|
29
|
+
# progress_bar= context[:progress_bar]
|
30
|
+
|
31
|
+
# system("bundle update decidim")
|
32
|
+
# system("bin/rails decidim:upgrade")
|
33
|
+
# system("bin/rails db:migrate")
|
34
|
+
end
|
35
|
+
|
36
|
+
def end_execution(_ctx)
|
37
|
+
log_task_step("#{@num_applied} users nicknamized")
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
@@ -0,0 +1,60 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative "install_gem_migrations_step"
|
4
|
+
|
5
|
+
module Decidim
|
6
|
+
module Cdtb
|
7
|
+
module Upgrades
|
8
|
+
# Validates that all Decidim modules have the migrations installed.
|
9
|
+
#
|
10
|
+
class ValidateMigrationsTask < ::Decidim::Cdtb::Task
|
11
|
+
STEPS_IN_DO_EXECUTION= 2
|
12
|
+
|
13
|
+
def initialize
|
14
|
+
progress_bar= { title: "Modules" }
|
15
|
+
super("VALIDATE MODULES MIGRATIONS", progress_bar: progress_bar)
|
16
|
+
end
|
17
|
+
|
18
|
+
def prepare_execution(_ctx)
|
19
|
+
all_railties= Rails.application.migration_railties
|
20
|
+
railties_w_migrations= all_railties.select do |railtie|
|
21
|
+
railtie.respond_to?(:paths) && railtie.paths["db/migrate"].first.present?
|
22
|
+
end
|
23
|
+
@gem_names= railties_w_migrations.map(&:railtie_name)
|
24
|
+
|
25
|
+
log_task_info("Found #{@gem_names.size} gems with migrations. Validating.....")
|
26
|
+
end
|
27
|
+
|
28
|
+
def total_items
|
29
|
+
STEPS_IN_DO_EXECUTION
|
30
|
+
end
|
31
|
+
|
32
|
+
def pending_migrations?
|
33
|
+
@pending_migrations.present?
|
34
|
+
end
|
35
|
+
|
36
|
+
def do_execution(context)
|
37
|
+
progress_bar= context[:progress_bar]
|
38
|
+
|
39
|
+
output= install_gem_migrations
|
40
|
+
|
41
|
+
progress_bar.increment
|
42
|
+
|
43
|
+
@pending_migrations= output.lines.select { |l| l.include?("Copied migration") }
|
44
|
+
|
45
|
+
progress_bar.increment
|
46
|
+
end
|
47
|
+
|
48
|
+
def end_execution(_ctx)
|
49
|
+
log_task_step("#{@gem_names.size} gems validated")
|
50
|
+
log_task_failure(@pending_migrations.join("\n")) if pending_migrations?
|
51
|
+
end
|
52
|
+
|
53
|
+
def install_gem_migrations
|
54
|
+
install_step= InstallGemMigrationsStep.new
|
55
|
+
install_step.install!(@gem_names)
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
60
|
+
end
|
data/lib/decidim/cdtb/version.rb
CHANGED
data/lib/decidim/cdtb.rb
CHANGED
@@ -2,9 +2,17 @@
|
|
2
2
|
|
3
3
|
require_relative "cdtb/version"
|
4
4
|
require_relative "cdtb/engine"
|
5
|
+
require_relative "cdtb/tasks"
|
5
6
|
|
6
7
|
module Decidim
|
8
|
+
# Cdtb configuration
|
7
9
|
module Cdtb
|
10
|
+
include ActiveSupport::Configurable
|
11
|
+
|
8
12
|
class Error < StandardError; end
|
13
|
+
|
14
|
+
config_accessor :spam_words do
|
15
|
+
%w[viagra sex game free crypto crack xxx luck girls vip download]
|
16
|
+
end
|
9
17
|
end
|
10
18
|
end
|
@@ -0,0 +1,52 @@
|
|
1
|
+
name: "[CI] Validate migrations"
|
2
|
+
|
3
|
+
on:
|
4
|
+
push:
|
5
|
+
branches:
|
6
|
+
- master
|
7
|
+
- release/*
|
8
|
+
- "*-stable"
|
9
|
+
pull_request:
|
10
|
+
|
11
|
+
env:
|
12
|
+
DB_DATABASE: app
|
13
|
+
DB_USERNAME: postgres
|
14
|
+
DB_PASSWORD: postgres
|
15
|
+
RUBY_VERSION: 3.1.3
|
16
|
+
|
17
|
+
jobs:
|
18
|
+
test:
|
19
|
+
runs-on: ubuntu-latest
|
20
|
+
env:
|
21
|
+
SECRET_KEY_BASE: "secret_key_base"
|
22
|
+
|
23
|
+
steps:
|
24
|
+
- uses: actions/checkout@v2.0.0
|
25
|
+
with:
|
26
|
+
fetch-depth: 1
|
27
|
+
- uses: ruby/setup-ruby@master
|
28
|
+
with:
|
29
|
+
ruby-version: ${{ env.RUBY_VERSION }}
|
30
|
+
bundler-cache: true
|
31
|
+
- name: Recover Ruby dependency cache
|
32
|
+
uses: actions/cache@v1
|
33
|
+
with:
|
34
|
+
path: ./vendor/bundle
|
35
|
+
key: ${{ runner.OS }}-app-rubydeps-${{ hashFiles('Gemfile.lock') }}
|
36
|
+
restore-keys: |
|
37
|
+
${{ runner.OS }}-app-rubydeps-${{ env.cache-name }}-
|
38
|
+
${{ runner.OS }}-app-rubydeps-
|
39
|
+
|
40
|
+
- name: Set bundle local config vendor/bundle path
|
41
|
+
run: bundle config set --local path 'vendor/bundle'
|
42
|
+
|
43
|
+
- name: Install Ruby deps
|
44
|
+
uses: nick-invision/retry@v2
|
45
|
+
with:
|
46
|
+
timeout_minutes: 10
|
47
|
+
max_attempts: 3
|
48
|
+
retry_on: error
|
49
|
+
command: bundle install --jobs 4 --retry 3
|
50
|
+
|
51
|
+
- name: Run specs
|
52
|
+
run: bin/rails cdtb:upgrades:validate_migrations
|
@@ -0,0 +1,12 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Cdtb
|
4
|
+
# Generates the GitHub workflow that validates that the app has the migrations from all gems already installed
|
5
|
+
class ValidateMigrationsCiGenerator < Rails::Generators::Base
|
6
|
+
source_root File.expand_path("templates", __dir__)
|
7
|
+
|
8
|
+
def copy_github_workflow_file
|
9
|
+
copy_file "validate_migrations.yml", ".github/workflows/validate_migrations.yml"
|
10
|
+
end
|
11
|
+
end
|
12
|
+
end
|
data/lib/tasks/spam.rake
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
namespace :cdtb do
|
4
|
+
namespace :spam do
|
5
|
+
desc "Show a list with users suspected of spam"
|
6
|
+
task :users, %i[org_id] => :environment do |_task, args|
|
7
|
+
organization = args.org_id.present? ? Decidim::Organization.find(args.org_id) : nil
|
8
|
+
|
9
|
+
detector = ::Decidim::Cdtb::Spam::UserSpamDetector.new(organization)
|
10
|
+
detector.execute!
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# A set of utils to manage Decidim Organizations.
|
5
|
+
#
|
6
|
+
namespace :cdtb do
|
7
|
+
namespace :upgrades do
|
8
|
+
require "decidim/cdtb/tasks_utils"
|
9
|
+
include ::Decidim::Cdtb::TasksUtils
|
10
|
+
|
11
|
+
desc <<~EODESC
|
12
|
+
Validates that migrations from all gems in the Gemfile have been installed.
|
13
|
+
EODESC
|
14
|
+
task validate_migrations: [:environment] do
|
15
|
+
task= ::Decidim::Cdtb::Upgrades::ValidateMigrationsTask.new
|
16
|
+
task.execute!
|
17
|
+
raise("There are pending migrations") if task.pending_migrations?
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: decidim-cdtb
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.4
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Oliver Valls
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2024-02-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: decidim
|
@@ -105,16 +105,25 @@ files:
|
|
105
105
|
- lib/decidim/cdtb/engine.rb
|
106
106
|
- lib/decidim/cdtb/fixes/nickname_fixer.rb
|
107
107
|
- lib/decidim/cdtb/multitenants/org_by_host_like.rb
|
108
|
+
- lib/decidim/cdtb/spam/user_spam_detector.rb
|
108
109
|
- lib/decidim/cdtb/storage/local_sharding.rb
|
109
110
|
- lib/decidim/cdtb/storage/set_local_on_blobs.rb
|
110
111
|
- lib/decidim/cdtb/task.rb
|
111
112
|
- lib/decidim/cdtb/tasks.rb
|
112
113
|
- lib/decidim/cdtb/tasks_utils.rb
|
114
|
+
- lib/decidim/cdtb/upgrades/install_gem_migrations_step.rb
|
115
|
+
- lib/decidim/cdtb/upgrades/upgrade_modules_task.rb
|
116
|
+
- lib/decidim/cdtb/upgrades/validate_migrations_task.rb
|
113
117
|
- lib/decidim/cdtb/version.rb
|
118
|
+
- lib/generators/cdtb/USAGE
|
119
|
+
- lib/generators/cdtb/templates/validate_migrations.yml
|
120
|
+
- lib/generators/cdtb/validate_migrations_ci_generator.rb
|
114
121
|
- lib/tasks/anonymize.rake
|
115
122
|
- lib/tasks/cdtb.rake
|
116
123
|
- lib/tasks/multitenants.rake
|
124
|
+
- lib/tasks/spam.rake
|
117
125
|
- lib/tasks/storage.rake
|
126
|
+
- lib/tasks/upgrade.rake
|
118
127
|
- sig/decidim/cdtb.rbs
|
119
128
|
homepage: http://github.com/CodiTramunana/cdtb
|
120
129
|
licenses:
|