decidim 0.20.0 → 0.23.1.rc1

data/docs/advanced/how_to_fix_metrics.md

@@ -0,0 +1,238 @@
+# How to fix metrics
+
+At the request of some instances, we have analyzed the issues related to metrics and looked for possible solutions.
+
+## Problems
+
+We have identified two main problems:
+
+- Metrics generation crashing, which cause `MetricJob`s to run again and again.
+- Peaks in generated metrics, sudden changes from day to day when displaying metrics.
+
+### Metrics generation crashing
+
+We have identified only one culprit here: "orphans" records, meaning records whose related component or participatory space cannot be found in the database. This is because in a previous decidim release `PartipatorySpaces` could be deleted but they were not deleted properly. So any application that has deleted a participatory space in the past, will probably have unrelated records that will make some metrics calculation crash.
+
+### Peaks in generated metrics
+
+If somehow the metrics jobs fail to execute for a period of time, big differences can appear in metrics. So first make sure that you have metrics for every day, if not [generate them](https://github.com/decidim/decidim/blob/master/docs/advanced/metrics.md).
+
+If you have metrics generated for almost everyday and still see drastic changes from day to day, take into account that changing the visibility of a component or participatory space (making them private or unpublishing them) will naturally cause big differences in generated metrics.
+
+Finally, if you see that the differences in some days are multiples of a previous generated metric, meaning suddenly you have exactly the double or the triple of a calculated metric, it's very likely that you have duplicate generated metrics. We have only seen this problem with instances using Sidekiq, not Delayed Job. We do not know the cause of this, but it seems to be a known issue [Avoiding duplicate jobs in Sidekiq](https://blog.francium.tech/avoiding-duplicate-jobs-in-sidekiq-dcbb1aca1e20).
+
+## Solutions
+
+We cannot offer a definitive solution for duplicate metrics, other than to delete old duplicate metrics and generate them again. If this problem persists, however, consider using Delayed Job.
+For a given metric type (`rake decidim:metrics:list`) that has duplicates:
+
+- Option 1: Remove individually each metric record per day.
+- Option 2: Delete all metric records and recalculate them. [CHANGELOG](https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics) of decidim version 0.18 has an example for "participants".
+
+For orphan records, you can do the following:
+
+- Back up the database.
+- Delete orphan records fromt the console (code is below).
+- Delete "comments" metrics and recalculate them following the [aforementioned example](https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics).
+
+### Some queries that may help
+
+```ruby
+    GROUP_BY_FIELDS= %w(
+      day
+      metric_type
+      decidim_organization_id
+      participatory_space_type
+      participatory_space_id
+      related_object_type
+      related_object_id
+      decidim_category_id).join(', ')
+
+    def remove_duplicates
+      sql= <<~EOSQL.strip
+      DELETE FROM decidim_metrics WHERE decidim_metrics.id NOT IN
+        (SELECT id FROM (
+          SELECT DISTINCT ON (#{GROUP_BY_FIELDS}) * FROM decidim_metrics));
+      EOSQL
+    end
+
+    # DELETE FROM decidim_metrics WHERE decidim_metrics.id NOT IN \n  (SELECT id FROM (\n    SELECT DISTINCT ON (day, metric_type, decidim_organization_id, participatory_space_type, participatory_space_id, related_object_type, related_object_id, decidim_category_id) * FROM decidim_metrics));
+    def count_duplicates
+      sql= <<~EOSQL.strip
+        SELECT count(1), #{GROUP_BY_FIELDS} FROM decidim_metrics GROUP BY #{GROUP_BY_FIELDS} HAVING COUNT(1) > 1;
+      EOSQL
+    end
+```
+
+### Delete orphan records
+
+"proposals", "meetings", "accountability", "debates", "pages", "budgets", "surveys"
+
+#### Proposals
+
+Delete proposals whose component does not have a participatory space and delete components of a proposal type that do not have a participatory space
+
+```ruby
+Decidim::Component.where(manifest_name: "proposals").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    Decidim::Proposals::Proposal.where(component: c).destroy_all
+    c.destroy
+  end
+}
+```
+
+Delete proposals that do not have a component
+
+```ruby
+Decidim::Proposals::Proposal.find_each(batch_size: 100) { |proposal|
+  proposal.delete if proposal.component.blank?
+}
+````
+
+#### Meetings
+
+Delete meetings whose component has no participatory space and delete components of meeting type that do not have a participatory space
+
+```ruby
+Decidim::Component.where(manifest_name: "meetings").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    Decidim::Meetings::Meeting.where(component: c).destroy_all
+    c.destroy
+  end
+}
+```
+
+Delete meetings that do not have a component
+
+```ruby
+Decidim::Meetings::Meeting.find_each(batch_size: 100) { |meeting|
+  meeting.delete if meeting.component.blank?
+}
+````
+
+#### Debates
+
+Delete debates that its component has no participatory space and the debate components that do not have a participatory space
+
+```ruby
+Decidim::Component.where(manifest_name: "debates").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    Decidim::Debates::Debate.where(component: c).destroy_all
+    c.destroy
+  end
+}
+```
+
+Destroy debates that do not have a component
+
+```ruby
+Decidim::Debates::Debate.find_each(batch_size: 100) { |debate|
+  debate.delete if debate.component.blank?
+}
+```
+
+#### Posts
+
+Destroy posts whose component has no participatory space and blog components that do not have a participatory space
+
+```ruby
+Decidim::Component.where(manifest_name: "blogs").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    Decidim::Blogs::Post.where(component: c).destroy_all
+    c.destroy
+  end
+}
+```
+
+Destroy posts that do not have a component
+
+```ruby
+Decidim::Blogs::Post.find_each(batch_size: 100) { |post|
+  post.delete if post.component.blank?
+}
+```
+
+#### Accountability
+
+Destroy results whose component has no participatory space and components of accountability type that do not have a participatory space
+
+```ruby
+Decidim::Component.where(manifest_name: "accountability").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    Decidim::Accountability::Result.where(component: c).destroy_all
+    c.destroy
+  end
+}
+```
+
+Destroy results that do not have a component
+
+```ruby
+Decidim::Accountability::Result.find_each(batch_size: 100) { |result|
+  result.delete if result.component.blank?
+}
+```
+
+#### Pages
+
+Destroy page components that do not have a participatory space
+
+```ruby
+Decidim::Component.where(manifest_name: "pages").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    c.destroy
+  end
+}
+```
+
+#### Budgets
+
+Destroy projects whose component has no participatory space and budget components that do not have a participatory space
+
+```ruby
+Decidim::Component.where(manifest_name: "budgets").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    Decidim::Budgets::Project.where(component: c).destroy_all
+    c.destroy
+  end
+}
+```
+
+Destroy results that do not have a component
+
+```ruby
+Decidim::Budgets::Project.find_each(batch_size: 100) { |project|
+  project.delete if project.component.blank?
+}
+```
+
+#### Surveys
+
+```ruby
+Decidim::Component.where(manifest_name: "surveys").find_each(batch_size: 100) { |c|
+  if c.participatory_space.blank?
+    Decidim::Surveys::Survey.where(component: c).destroy_all
+    c.destroy
+  end
+}
+```
+
+Destroy surveys that do not have a component
+
+```ruby
+Decidim::Surveys::Survey.find_each(batch_size: 100) { |survey|
+  survey.delete if survey.component.blank?
+}
+```
+
+#### Comments
+
+Destroy comments whose commentable root is a proposal that does not have a participatory space.
+
+```ruby
+proposal_ids = Decidim::Comments::Comment.where(decidim_root_commentable_type: "Decidim::Proposals::Proposal").pluck(:decidim_root_commentable_id)
+
+proposal_ids_without_space = Decidim::Proposals::Proposal.where(id: proposal_ids).find_all{|p| p.participatory_space.blank? }.pluck(:id)
+
+Decidim::Comments::Comment.where(decidim_root_commentable_type: "Decidim::Proposals::Proposal", decidim_root_commentable_id: proposal_ids_without_space).destroy_all
+```

data/docs/advanced/machine_translation_service.md

@@ -0,0 +1,12 @@
+# Create your own machine translation service
+
+You can use the `Decidim::Dev::DummyTranslator` service as a base. Any new translator service will need to implement the same API as this class.
+
+## Integrating with async services
+
+Some translation services are async, which means that some extra work is needed. This is the main overview:
+
+- The Translation service will only send the translation request. It should have a way to send what resource, field and target locale are related to that translation.
+- You'll need to create a custom controller in your application to receive the callback from the translation service when the translation is finished
+- From that new endpoint, find a way to find the related resource, field and target locale. Then start a `Decidim::MachineTranslationSaveJob` with that data. This job will handle how to save the data in the DB.
+

data/docs/advanced/metrics.md

@@ -37,6 +37,7 @@ Metrics calculations must be executed everyday. Some `rake task` have been added
 ```ruby
 bundle exec rake decidim:metrics:list
 ```
+
 Currently, available metrics are:
 
 - **users**, created `Users`
@@ -66,7 +67,7 @@ Only available for `ParticipatorySpaces` (restricted to `ParticipatoryProcesses`
 
 ## Configuration
 
-- A **crontab** line must be added to your server to maintain them updated daily. You could use [Whenever](https://github.com/javan/whenever) to manage it directly from the APP
+- A **crontab** line must be added to your server to maintain them updated daily. You could use [Whenever](https://github.com/javan/whenever) to manage it directly from the APP. You probably want to schedule a `bundle exec rake decidim:metrics:all` every night.
 - An **ActiveJob** queue, like [Sidekiq](https://github.com/mperham/sidekiq) or [DelayedJob](https://github.com/collectiveidea/delayed_job/)
 
 ## Persistence
@@ -78,20 +79,17 @@ persist metrics from all times and types.
 The `decidim_metrics` table has the following fields:
 
 - `day`: the day for which the metric has been computed.
-- `metric_type`: the type of the metric. One of: users, proposals,
-accepted_proposals, supports, assemblies.
+- `metric_type`: the type of the metric. One of: users, proposals, accepted_proposals, supports, assemblies.
 - `cumulative`: quantity accumulated to day ”day”.
 - `quantity`:  quantity for the current day, ”day”.
-- `decidim_organization_id`: the FK to the organization to which this Metric
-belongs to.
-- `participatory_space_type` + `participatory_space_id`: the FK to the
-participatory space to which this Metric belongs to, if any.
-- `related_object_type` + `related_object_id`: the FK to the object to which
-this Metric belongs to, if any.
+- `decidim_organization_id`: the FK to the organization to which this Metric belongs to.
+- `participatory_space_type` + `participatory_space_id`: the FK to the participatory space to which this Metric belongs to, if any.
+- `related_object_type` + `related_object_id`: the FK to the object to which this Metric belongs to, if any.
 - `decidim_category_id`: the FK to the category for this Metric, if any.
 
 Relations around `decidim_metrics` table:
-```
+
+```ascii
                                                     +------------------------+
 +--------------+                                    | ParticipatoryProcesses |
 | Organization |                               +----+------------------------+

data/docs/advanced/newsletter_templates.md

@@ -0,0 +1,64 @@
+# Newsletter templates
+
+The newsletter templates allow the user to select a template amongst a set of of them, and use it as base to send newsletters. This allow for more customization on newsletter, tematic newsletters, etc.
+
+Code-wise, they use the content blocks system internally, so [check the docs](https://github.com/decidim/decidim/blob/master/docs/advanced/content_blocks.md) for that section first.
+
+## Adding a new template
+
+You'll first need to register the template as a content block, but specifying `:newsletter_template` as its scope:
+
+```ruby
+Decidim.content_blocks.register(:newsletter_template, :my_template) do |content_block|
+  content_block.cell "decidim/newsletter_templates/my_template"
+  content_block.settings_form_cell = "decidim/newsletter_templates/my_template_settings_form"
+  content_block.public_name_key "decidim.newsletter_templates.my_template.name"
+
+  content_block.images = [
+    {
+      name: :main_image,
+      uploader: "Decidim::NewsletterTemplateImageUploader",
+      preview: -> { ActionController::Base.helpers.asset_path("decidim/placeholder.jpg") }
+    }
+  ]
+
+  content_block.settings do |settings|
+    settings.attribute(
+      :body,
+      type: :text,
+      translated: true,
+      preview: -> { ([I18n.t("decidim.newsletter_templates.my_template.body_preview")] * 100).join(" ") }
+    )
+  end
+end
+```
+
+You'll need to add this into an initializer. Note that if you're adding this from a module, then you need to add it from the `engine.rb` file (check the docs for content blocks for more info).
+
+This is the simplest template. It has a single attribute, in this case a translatable chunk of text. Let's go line by line.
+
+Inside the block, first we define the path of the cell we'll use to render the email publicly. This cell will receive the `Decidim::ContentBlock` object, which will contain the attributes and any image we have (one in this example). In order to render cells, please note that emails have a very picky HTML syntax, so we suggest using some specialized tools to design the template, export the layout to HTML and render that through the cell. We suggest you make this cell inherit from `Decidim::NewsletterTemplates::BaseCell` for convenience.
+
+Then we define the cell that will be used to render the form in the admin section. This form needs to show inputs for the attributes defined when registering the template. It will receive the `form` object to render the input fields. We suggest this cell to inherit from `Decidim::NewsletterTemplates::BaseSettingsFormCell`.
+
+In the third line inside the block we define the I18n path to the public name of the template. This name will serve as identifier for the users who write the newsletters, so be sure to make it descriptive.
+
+After that we define the images this newsletter supports. We give it a unique name, the class name of the uploader we'll use (the example one is the default one, but you might want to customize this value) and a way to preview this image. This preview image will only be used in the "Preview" page of the template, it's not a fallback. If you want a fallback, please implement it through a custom uploader (see `carrierwave`'s docs for that). There's no limit of the amount of images you add.
+
+Finally, we define the attributes for the newsletter. In this case we define a body attribute, which is a translatable text. Whether this text require an editor or not will be defined by the settings cell. We also have a way to preview that attribute. There's no limit on the number of attributes you define.
+
+## Interpolating the recipient name
+
+Decidim accepts `%{name}` as a placeholder for the recipient name. If you want your template to use it, you'll need to call `parse_interpolations` in your public cell:
+
+```ruby
+class Decidim::NewsletterTemplates::MyTemplate < Decidim::ViewModel
+  include Decidim::NewslettersHelper
+
+  def body
+    parse_interpolations(uninterpolated_body, recipient_user, newsletter.id)
+  end
+end
+```
+
+The newsletter subject is automatically interpolated.

data/docs/advanced/notifications.md

@@ -0,0 +1,114 @@
+# Notifications
+
+In Decidim, notifications may mean two things:
+
+- he concept of notifying an event to a user. This is the wider use of "notification".
+- the notification's participant space, which lists the `Decidim::Notification`s she has received.
+
+So, in the wider sense, notifications are messages that are sent to the users, admins or participants, when something interesting occurs in the platform.
+
+Each notification is sent via two communication channels: email and internal notifications.
+
+## A Decidim Event
+
+Technically, a Decidim event is nothing but an `ActiveSupport::Notification` with a payload of the form
+
+```ruby
+ActiveSupport::Notifications.publish(
+  event,
+  event_class: event_class.name,
+  resource: resource,
+  affected_users: affected_users.uniq.compact,
+  followers: followers.uniq.compact,
+  extra: extra
+)
+```
+
+To publish an event to send a notification, Decidim's `EventManager` should be used:
+
+```ruby
+# Note the convention between the `event` key, and the `event_class` that will be used later to wrap the payload and be used as the email or notification model.
+data = {
+  event: "decidim.events.comments.comment_created",
+  event_class: Decidim::Comments::CommentCreatedEvent,
+  resource: comment.root_commentable,
+  extra: {
+    comment_id: comment.id
+  },
+  affected_users: [user1, user2],
+  followers: [user3, user4]
+}
+
+Decidim::EventsManager.publish(data)
+```
+
+Both, `EmailNotificationGenerator` and `NotificationGenerator` are use the same arguments:
+
+- **event**: A String with the name of the event.
+- **event_class**: A class that wraps the event.
+- **resource**: an instance of a class implementing the `Decidim::Resource` concern.
+- **followers**: a collection of Users that receive the notification because they're following it.
+- **affected_users**: a collection of Users that receive the notification because they're affected by it.
+- **force_send**: boolean indicating if EventPublisherJob should skip the `notifiable?` check it performs before notifying.
+- **extra**: a Hash with extra information to be included in the notification.
+
+Again, both generators will check for each user
+
+- in the *followers* array, if she has the `notification_types` set to "all" or "followed-only".
+- in the *affected_users* array, if she has the `notification_types` set to "all" or "own-only".
+
+Event names must start with "decidim.events." (the `event` data key). This way `Decidim::EventPublisherJob` will automatically process them. Otherwise no artifact in Decidim will process them, and will be the developer's responsibility to subscribe to them and process.
+
+Sometimes, when something that must be notified to users happen, a service is defined to manage the logic involved to decide which events should be published. See for example `Decidim::Comments::NewCommentNotificationCreator`.
+
+Please refer to [Ruby on Rails Notifications documentation](https://api.rubyonrails.org/classes/ActiveSupport/Notifications.html) if you need to hack the Decidim's events system.
+
+## How Decidim's `EventPublisherJob` processes the events?
+
+The `EventPublisherJob` in Decidim's core engine subscribes to all notifications matching the regular expression `/^decidim\.events\./`. This is, starting with "decidim.events.". It will then be invoked when an imaginary event named "decidim.events.harmonica_blues" is published.
+
+When invoked it simply performs some validations and enqueue an `EmailNotificationGeneratorJob` and a `NotificationGeneratorJob`.
+
+The validations it performs check if the resource, the component, or the participatory space are published (when the concept applies to the artifact).
+
+## The \*Event class
+
+Generates the email and notification messages from the information related with the notification.
+
+Event classes are subclasses of `Decidim::Events::SimpleEvent`.
+A subset of the payload of the notification is passed to the event class's constructor:
+
+- The `resource`
+- The `event` name
+- The notified user, either from the `followers` or from the `affected_users` arrays
+- The `extra` hash, with content specific for the given SimpleEvent subclass
+- The user_role, either :follower or :affected_user
+
+With the previous information the event class is able to generate the following contents.
+
+Developers will be able to customize those messages by adding translations to the `config/locales/en.yml` file of the corresponding module.
+The keys to be used will have the translation scope corresponding to the event name ("decidim.events.comments.comment_by_followed_user" for example) and the key will be the content to override (email_subject, email_intro, etc.)
+
+### Email contents
+
+The following are the parts of the notification email:
+
+- *email_subject*, to be customized
+- email_greeting, with a good default, usually there's no need to cusomize it
+- *email_intro*, to be customized
+- *resource_text* (optional), rendered `html_safe` if present
+- *resource_url*, a link to the involved resource if resource_url and resource_title are present
+- *email_outro*
+
+All contents except the `email_greeting` use to require customization on each notification.
+
+### Notification contents
+
+Only the `notification_title` is generated in the event class. The rest of the contents are produced by the templates from the `resource` and the `notification` objects.
+
+## Testing notifications
+
+- Test that the event has been published (usually a command test)
+- Test the event returns the expected contents for the email and the notification.
+
+Developers should we aware when adding URLs in the email's content, be sure to use absolute URLs and not relative paths.