decidim-cdtb 0.1.2

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023 Oliver Valls
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,118 @@
+# Decidim::Cdtb
+
+This is CodiTramuntana's Decidim Toolbelt (cdtb), a gem to help managing Decidim applications.
+
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+    $ bundle add decidim-cdtb
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+    $ gem install decidim-cdtb
+
+## Usage
+
+### Rake tasks
+
+#### Organizations information
+
+Returns information regarding the organizations in a multitenant installation that match a search term ignorecase.
+
+
+The following will return all the attributes for all organizations that contain the "vila" term in its host name:
+
+```
+bin/rake cdtb:org_by_host_like[vila,true]
+```
+
+With the `full` argument set to `true` will return the most relevant attributes:
+
+```
+bin/rake cdtb:org_by_host_like[vila]
+>>> Organization [1] Sant Boi de Llobregat:
+host: localhost, time_zone: Madrid, locales: ca + [ca, es, oc], available authorizations: [postal_letter, members_picker_authorization_handler]
+```
+
+#### Fix nicknames
+
+In a previous version than Decidim v0.25 a validation to the `Decidim::User.nickname` was added with a migration to fix existing nicknames. But the migration was only taking into acocunt managed (impersonated) users.
+
+This task iterates (with `find_each`) over all non managed users and nicknamizes the nickname.
+
+To execute the task run:
+
+```
+bin/rake cdtb:fix_nicknames
+```
+
+#### Anonymize production dump
+
+Anonymize rake task was taken from https://github.com/AjuntamentdeBarcelona/decidim-barcelona
+
+Available rake tasks:
+
+```bin/rake cdtb:anonymize:check``` allows you to check if you can anonymize production dump
+
+```bin/rake cdtb:anonymize:all``` anonymizes whole production dump (without proposals)
+
+```bin/rake cdtb:anonymize:users``` anonymizes users
+
+```bin/rake cdtb:anonymize:proposals``` anonymizes proposals
+
+```bin/rake cdtb:anonymize:user_groups``` anonymizes user groups
+
+```bin/rake cdtb:anonymize:system_admins``` anonymizes system admins
+
+```bin/rake cdtb:anonymize:paper_trail``` anonymizes paper trails
+
+#### Migrate ActiveStorage service from S3 to local
+
+To migrate from S3 to local storage, the identified steps will be:
+
+1. Download the assets to a temporary directory:
+    `aws s3 sync s3://bucket-name tmp/storage/`
+2. Move the downloaded assets into the local storage directory doing the sharding:
+    `bin/rake cdtb:s3_to_local:do_sharding`
+3. Update all blobs to use the local service
+    `bin/rake cdtb:s3_to_local:set_local_service_on_blobs`
+4. Clean the cache:
+    `bin/rake cache:clear`
+5. Restart the Rails server
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Run tests
+
+Node 16.9.1 is required!
+
+Create a dummy app:
+
+```bash
+bin/rails decidim:generate_external_test_app
+```
+
+And run tests:
+
+```bash
+bundle exec rspec spec
+```
+
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/decidim-cdtb. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/decidim-cdtb/blob/master/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the Decidim::Cdtb project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/decidim-cdtb/blob/master/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "decidim/dev/common_rake"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/app/jobs/application_job.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module Decidim
+  # Base Job for all Cdtb jobs
+  class ApplicationJob < ActiveJob::Base
+    # Automatically retry jobs that encountered a deadlock
+    retry_on ActiveRecord::Deadlocked
+
+    # Most jobs are safe to ignore if the underlying records are no longer available
+    discard_on ActiveJob::DeserializationError
+  end
+end

data/app/jobs/cdtb/fix_nickname_job.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Cdtb
+  # Fixes the nickname of the Decidim::User with the given `user_id`.
+  class FixNicknameJob < ApplicationJob
+    queue_as :default
+
+    def perform(user_id)
+      user= Decidim::User.find(user_id)
+      previous= user.nickname
+
+      nickname = Decidim::User.nicknamize(previous, organization: user.organization)
+      user.update_attribute(:nickname, nickname)
+
+      Rails.logger.info "#{user.id}-#{user.email}: #{previous} => #{user.nickname}"
+    end
+  end
+end

data/decidim-cdtb.gemspec

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+require_relative "lib/decidim/cdtb/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "decidim-cdtb"
+  spec.version = Decidim::Cdtb::VERSION
+  spec.authors = ["Oliver Valls"]
+  spec.email = ["199462+tramuntanal@users.noreply.github.com"]
+
+  spec.summary = "CodiTramuntana's Decidim Toolbelt (cdtb)."
+  spec.description = "A gem to help managing Decidim applications."
+  spec.homepage = "http://github.com/CodiTramunana/cdtb"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 2.7.5"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+  spec.metadata["changelog_uri"] = "#{spec.homepage}/CHANGELOG.md"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (f == __FILE__) || f.match(%r{\A(?:(?:bin|test|spec|features)/|\.(?:git|travis|circleci)|appveyor)})
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "decidim", Decidim::Cdtb::DECIDIM_MIN_VERSION
+  spec.add_dependency "rails", ">= 6"
+  spec.add_dependency "ruby-progressbar"
+
+  spec.add_development_dependency "decidim-dev", Decidim::Cdtb::DECIDIM_MIN_VERSION
+  spec.add_development_dependency "faker"
+end

data/lib/decidim/cdtb/engine.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Cdtb
+    # Census have no public app (see AdminEngine)
+    class Engine < ::Rails::Engine
+      isolate_namespace Decidim::Cdtb
+
+      initializer "psych.tmp.fix" do |_app|
+        # Workaround for https://stackoverflow.com/questions/72970170/upgrading-to-rails-6-1-6-1-causes-psychdisallowedclass-tried-to-load-unspecif
+        Rails.application.config.active_record.use_yaml_unsafe_load = true
+      end
+    end
+  end
+end

data/lib/decidim/cdtb/fixes/nickname_fixer.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Cdtb
+    module Fixes
+      # Fixes invalid Decidim::User#nickname
+      #
+      class NicknameFixer < ::Decidim::Cdtb::Task
+        def initialize
+          progress_bar= { title: "Decidim::User" }
+          super("FIX NICKNAMES", progress_bar: progress_bar)
+        end
+
+        def prepare_execution(_ctx)
+          @num_users= Decidim::User.count
+          log_task_info("Checking #{@num_users} users...")
+        end
+
+        def total_items
+          @num_users
+        end
+
+        def do_execution(context)
+          progress_bar= context[:progress_bar]
+
+          Decidim::User.find_each do |user|
+            Decidim::User.validators_on(:nickname).each do |validator|
+              validator.validate_each(user, :nickname, user.nickname)
+            end
+
+            if user.errors[:nickname].any?
+              ::Cdtb::FixNicknameJob.perform_later(user.id)
+              @num_applied+= 1
+            end
+            progress_bar.increment
+          end
+        end
+
+        def end_execution(_ctx)
+          log_task_step("#{@num_applied} users nicknamized")
+        end
+      end
+    end
+  end
+end

data/lib/decidim/cdtb/multitenants/org_by_host_like.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Cdtb
+    module Multitenants
+      # Finds information about the Organization, or Organizations, searching by the :host_term argument ignorecase.
+      # Set :full (second param) to `true` for full information
+      #
+      class OrgByHostLike < ::Decidim::Cdtb::Task
+        def initialize(host_term, full_info)
+          @host_term= host_term
+          @show_full_info= full_info == "true"
+          super("ORG BY HOST LIKE")
+        end
+
+        def prepare_execution(_ctx)
+          @query = Decidim::Organization.where("host ilike ?", "%#{@host_term}%")
+          log_task_info("Found #{@query.count} organizations")
+        end
+
+        def do_execution(_ctx)
+          @query.find_each do |org|
+            log_task_step("Organization [#{org.id}] #{org.name}:")
+            if show_full_info?
+              show_full_info(org)
+            else
+              h= {
+                host: org.host,
+                time_zone: org.time_zone,
+                locales: "#{org.default_locale} + [#{org.available_locales&.join(", ")}]",
+                available_authorizations: org.available_authorizations&.join(", ")
+              }
+              do_log(h.to_yaml)
+            end
+            do_log("---------------------------------------------------------")
+          end
+        end
+
+        #----------------------------------------------------------------
+
+        private
+
+        #----------------------------------------------------------------
+
+        def show_full_info?
+          @show_full_info
+        end
+
+        def show_full_info(org)
+          do_log(org.attributes.to_yaml)
+        end
+
+        def show_summary_info(org)
+          h= {
+            host: org.host,
+            time_zone: org.time_zone,
+            locales: "#{org.default_locale} + [#{org.available_locales&.join(", ")}]",
+            available_authorizations: org.available_authorizations&.join(", ")
+          }
+          do_log(h.to_yaml)
+        end
+      end
+    end
+  end
+end

data/lib/decidim/cdtb/storage/local_sharding.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Cdtb
+    module Storage
+      # Given that all assets has already been copied from S3 to storage/,
+      # this task performs the sharding of the downloaded files.
+      #
+      # This step is required because in S3 all assets are stored flat at the same level (directory),
+      # but local service stores the files with sharding.
+      class LocalSharding < ::Decidim::Cdtb::Task
+        def initialize
+          progress_bar= { title: "ActiveStorage::Blob" }
+          super("S3 to local: DO SHARDING", progress_bar: progress_bar)
+        end
+
+        def prepare_execution(_ctx)
+          @num_blobs= ActiveStorage::Blob.count
+          log_task_info("Checking #{@num_blobs} blobs...")
+        end
+
+        def total_items
+          @num_blobs
+        end
+
+        def do_execution(context)
+          progress_bar= context[:progress_bar]
+
+          ActiveStorage::Blob.find_each do |blob|
+            path= ActiveStorage::Blob.service.path_for(blob.key)
+            src_file= Rails.root.join("tmp/storage", blob.key)
+            if File.exist?(src_file)
+              shard_asset(blob, path)
+              @num_applied+= 1
+            else
+              logger.warn "File Not Found or directory: #{path}"
+            end
+            progress_bar.increment
+          end
+        end
+
+        def end_execution(_ctx)
+          log_task_info("#{@num_applied} blobs sharded")
+        end
+
+        #----------------------------------------------------------------
+
+        private
+
+        #----------------------------------------------------------------
+
+        def shard_asset(blob, path)
+          blob_dir= File.dirname path
+          logger.info "Creating dir: #{blob_dir}"
+          FileUtils.mkdir_p(blob_dir)
+          FileUtils.mv Rails.root.join("tmp/storage", blob.key), path, force: true
+          logger.info "Sharding for file: #{path}"
+        end
+      end
+    end
+  end
+end

data/lib/decidim/cdtb/storage/set_local_on_blobs.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Cdtb
+    module Storage
+      # Updates all ActiveStorage::Blob rows in the DB to use the :local service.
+      class SetLocalOnBlobs < ::Decidim::Cdtb::Task
+        def initialize
+          super("S3 to local: FORCE LOCAL SERVICE")
+        end
+
+        def prepare_execution(_ctx)
+          @num_blobs= ActiveStorage::Blob.count
+          log_task_info("Updating #{@num_blobs} blobs...")
+        end
+
+        def total_items
+          @num_blobs
+        end
+
+        def do_execution(_context)
+          ActiveStorage::Blob.update(service_name: "local")
+        end
+
+        def end_execution(_ctx)
+          log_task_info("Blobs updated")
+        end
+
+        #----------------------------------------------------------------
+
+        #----------------------------------------------------------------
+      end
+    end
+  end
+end

data/lib/decidim/cdtb/task.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+require "decidim/cdtb/tasks_utils"
+
+module Decidim
+  module Cdtb
+    # Parent class with common behaviour for all tasks.
+    #
+    class Task
+      include Decidim::Cdtb::TasksUtils
+
+      # title: The title shown at the begining of the Task
+      # progress_bar: A hash with one key: :title for the title of the ProgressBar.
+      def initialize(title, progress_bar: nil)
+        @title= title
+        @progress_bar= progress_bar
+        @num_applied = 0
+      end
+
+      attr_reader :num_applied, :title
+
+      def init
+        log_task_title(@title)
+        @start_time= Time.zone.now
+        do_log("▶️  Starting at #{@start_time}")
+      end
+
+      def execute!
+        init
+        ctx= {}
+        ctx[:progress_bar]= ProgressBar.create(total: total_items, title: title) if has_progress?
+        prepare_execution(ctx)
+        do_execution(ctx)
+        end_execution(ctx)
+        finish
+      end
+
+      def finish
+        do_log("⏱️  Took #{Time.zone.now - @start_time} seconds")
+        log_task_end
+      end
+
+      #################################
+
+      protected
+
+      #################################
+
+      # May be used by subclasses for preparing before executing the task
+      def prepare_execution(context); end
+
+      # Subclasses must implement the steps of the task overriding this method.
+      def do_execution(context); end
+
+      # May be used by subclasses for doing whatever after executing the task
+      def end_execution(context); end
+
+      def has_progress?
+        @progress_bar.present?
+      end
+
+      # The number of items to be processed.
+      # Required by the progress bar.
+      def total_items; end
+    end
+  end
+end

data/lib/decidim/cdtb/tasks.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "decidim/cdtb/tasks_utils"
+require "decidim/cdtb/task"
+require "decidim/cdtb/fixes/nickname_fixer"
+require "decidim/cdtb/multitenants/org_by_host_like"
+require "decidim/cdtb/storage/local_sharding"
+require "decidim/cdtb/storage/set_local_on_blobs"

data/lib/decidim/cdtb/tasks_utils.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require "ruby-progressbar"
+
+module Decidim
+  module Cdtb
+    # Reusable utils for Cdtb Rake tasks.
+    module TasksUtils
+      def logger
+        Rails.logger
+      end
+
+      def do_log(msg)
+        puts msg
+        logger.info(msg)
+      end
+
+      def log_task_title(title)
+        do_log("⚙️  #{title}")
+      end
+
+      def log_task_step(description)
+        do_log("➡️  #{description}")
+      end
+
+      def log_task_info(info)
+        do_log("ℹ️  #{info}")
+      end
+
+      def log_task_end
+        end_comment= "✅ Done."
+        do_log(end_comment)
+      end
+    end
+  end
+end

data/lib/decidim/cdtb/version.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Cdtb
+    VERSION = "0.1.2"
+    DECIDIM_MIN_VERSION = ">= 0.26.2"
+  end
+end

data/lib/decidim/cdtb.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require_relative "cdtb/version"
+require_relative "cdtb/engine"
+
+module Decidim
+  module Cdtb
+    class Error < StandardError; end
+  end
+end