decidim 0.23.6 → 0.24.0.rc1

data/docs/modules/develop/pages/guide_development_app.adoc

@@ -0,0 +1,44 @@
+= Development App
+
+In order to start developing you will need what is called a `development_app`. This is nearly the same as a new Decidim app (that you can create with `decidim app_name`) but with a Gemfile pre-configured for local development and some other small config modifications.
+You need it in order to have a Rails application configured to lookup Decidim modules from your filesystem. This way changes in your modules will be directly observed by this `development_app`.
+
+You can create a `development_app` from inside the project's root folder with the command:
+
+[source,console]
+----
+git clone https://github.com/decidim/decidim.git
+cd decidim
+bundle install
+bundle exec rake development_app
+cd development_app
+----
+
+A development_app/ entry appears in the .gitignore file, so you don't have to worry about committing the development app by mistake.
+
+On creation, this steps are automatically invoked by the generator:
+
+* create a `config/database.yml`
+* `bundle install`
+* `bin/rails decidim:upgrade`
+* `bin/rails db:migrate db:seed`
+
+If the default database.yml does not suit your needs you can always configure it at your will and run this steps manually.
+
+The last command will set your database and add some example data (called "seed data") so that you can start trying Decidim. We don't recommend using seed data for production environments, but it's useful for local development and staging environments.
+
+Once the app is created you are ready to start the server:
+
+* `bin/rails s`
+
+== Migrations
+
+When creating new migrations in Decidim's modules, you will need to "apply" this migrations to your development_app. The way to do this is by copying the migration from your module into the db/migrate dir of your development_app. Luckily we already have a script that automates this: it copies all missing migrations in development_app/db/migrate. The command is:
+
+[source,console]
+----
+bin/rails decidim:upgrade
+----
+
+Anyway we recommend re-creating your development_app every once in a while.
+

data/docs/modules/develop/pages/guide_development_with_custom_seed_data.adoc

@@ -0,0 +1,31 @@
+= Development with custom seed data
+
+The seed data is not only useful for local development, but also for staging environments to showcase your work to your clients. In this case, you might need to customize some of that seed data to make sure it fits your needs.
+
+== Customizing seed data via the `db/seeds.rb` file
+
+One way to customize the seed data is to use the `db/seeds.rb` file in your application. You'll see a call to generate the Decidim seeds:
+
+[source,ruby]
+----
+Decidim.seed!
+----
+
+This will generate all the needed seed data. But after that you might want to, for example, create some specific scopes, or change the default user password. You can do anything from that file.
+
+== Customizing the seed data via environment variables
+
+Some specific pieces of the seed data can be customized via environment variables. This is specially useful to set the SMTP credentials or the System Admin password.
+
+If you set the variables _before_ creating the seeds data, the values in those variables will be used.
+
+* `SMTP_FROM_LABEL`: The name used to send emails.
+* `SMTP_FROM_EMAIL`: The email where emails will be sent from.
+* `SMTP_USERNAME`: The username required by your email sending system.
+* `SMTP_PASSWORD`: The password required by your email sending system.
+* `SMTP_ADDRESS`: The address where your email sending system lives.
+* `SMTP_PORT`: The port used to connect to your email sending system.
+* `DECIDIM_SYSTEM_USER_PASSWORD`: The password for the System user. Useful to hide access to the System panel, where the SMTP config can be retrieved in plain text.
+
+This is useful, for example, to set up a staging application open to anyone, and use Sendgrid as a mailing system. By changing the default password only you will be able to access the System panel, so nobody will be able to see the credentials used to connect to Sendgrid.
+

data/docs/modules/develop/pages/guide_development_with_localhost_ssl.adoc

@@ -0,0 +1,63 @@
+= Testing SSL in localhost
+
+In some cases is useful to access the `localhost` environment under SSL (ie: `https://localhost:3000/`).
+
+A typical use case is when you want to test login integration with 3rd party systems using OAuth (which in the latest versions only the HTTPS protocol is allowed).
+
+Using Puma, the default server for Rails, you can easily do the trick by creating a self-signed certificate and using it when starting your development environment.
+
+== Create a self-signed certificate for localhost
+
+First you need a SSL certificate for the `localhost` domain (you only need to execute this command once):
+
+[source,bash]
+----
+openssl req -x509 -out localhost.crt -keyout localhost.key \
+  -newkey rsa:2048 -nodes -sha256 \
+  -subj '/CN=localhost' -extensions EXT -config <( \
+   printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
+----
+
+This will generate two files: `localhost.crt` and `localhost.key`, you can store them anywhere you want to use them later. In this example we'll suppose we have them in the folder `certs` in our root user home:
+
+[source,bash]
+----
+mkdir ~/certs/
+mv localhost.crt ~/certs/
+mv localhost.key ~/certs/
+----
+
+== Starting development in SSL mode
+
+Once you have your certificate is generated, just use the next command in order to start puma instead of the typical `bin/rails s`:
+
+[source,bash]
+----
+bin/rails s -b "ssl://127.0.0.1:3000?key=$HOME/certs/localhost.key&cert=$HOME/certs/localhost.crt"
+----
+
+Now you are ready to visit your favorite browser at the address `https://localhost:3000/` (note "https").
+
+Note: Your browser is going to complain as this is a self-signed certificate, that's ok for development, just add an exception and accept the certificate.
+
+Also take into account that starting Puma in SSL mode will disable accessing it in non-ssl mode (normal `http` protocol).
+
+== Testing tenants using lvh.me
+
+You can also test the multi-tenant capabilities of Decidim by using alternative domains or subdomains that points to your local machine. `lvh.me` is a service that does just that without configuring anything in your machine (an alternative is to add entries in your `/etc/hosts` with a testing domain of your choice). Just point your browser to any subdomain of `lvh.me` and you'll be in your own machine.
+
+Just access your `/system` admin and create new organization with some subdomain of `lvh.me`: `organization.lvh.me`, `tenant2.lvh.me`, etc.
+
+You can combine this with the previously generated certificate (your browser is going to complaint but just tell it to proceed vising the site).
+
+Finally remember to add the port as `lvh.me` do not forwards anything, for instance (use `http` or `https` depending on how you've started Rails):
+
+`http://someorg.lvh.me:3000/`
+
+`https://someorg.lvh.me:3000/`
+
+`http://anotherorg.lvh.me:3000/`
+
+`https://anotherorg.lvh.me:3000/`
+
+etc.

data/docs/modules/develop/pages/guide_example_apps.adoc

@@ -0,0 +1,59 @@
+= Example Apps
+
+If you have worked in the past with Ruby on Rails, you'll notice quickly that Decidim is *not* your typical Ruby on Rails Vanilla App.
+
+We've tried using http://decide.es[Consul] but we found some problems on reutilization, adaptation, modularization and configuration. You can read more about that on https://alabs.gitbooks.io/propuesta-de-cambios-en-la-arquitectura-de-consul/content/[Propuesta de Cambios de Arquitectura de Consul (in spanish only)]. As a summary, we decided to try with a model based in https://guides.rubyonrails.org/engines.html[RoR Engines]. A good resource regarding this approach is https://cbra.info/[Component Based Rails Applications].
+
+== What this means
+
+This means that:
+
+. Decidim is a https://rubygems.org/gems/decidim[gem], available in rubygems.org. You can install it with the command `gem install decidim`
+. Decidim is a command line generator of Ruby on Rails applications. After you've install it you can generate a new RoR app with `decidim HelloWorld`
+. This app has a dependency the gem itself. You can see this in the Gemfile of the generated app.
+. This app can be extended by making changes in the app (see xref:customize:index.adoc[Customising]) and by working with https://decidim.org/modules[Modules].
+
+== Examples
+
+Here are some good examples of what can be done with Decidim:
+
+=== Helsinki OmaStadi
+
+See the website at https://omastadi.hel.fi/
+
+See the source code at https://github.com/City-of-Helsinki/decidim-helsinki/
+
+image::helsinki.png[Helsinki OmaStadi]
+
+Some highlights:
+
+* They have their https://github.com/City-of-Helsinki/decidim-helsinki/blob/b9a09e570eb6090dee93f2ee73b5951882c74755/Gemfile[Gemfile] with lots of modules with their own code. Some of them are also in ruybgems (like `decidim-access_requests`, `decidim-antivirus`, `decidim-mpassid`, `decidim-process_groups_content_block`, `decidim-suomifi`, `decidim-term_customizer`), others are only in Github (`decidim-accountability_simple`, `decidim-apiauth`, `decidim-combined_budgeting`, `decidim-plans`, `decidim-redirects`). They even use a module made by other organization in the community (`decidim-budgets_enhanced`)
+* They have their UI customized to their needs. See the https://github.com/City-of-Helsinki/decidim-helsinki/tree/a7396a312cc04198654a86d66c2a7de556c212af/app/assets/stylesheets[CSS] and https://github.com/City-of-Helsinki/decidim-helsinki/tree/a7396a312cc04198654a86d66c2a7de556c212af/app/views[HTML (views)].
+* They have an awesome READE explaining how it works and what changes have made.
+
+=== Decidim Barcelona
+
+See the website at https://decidim.barcelona
+
+See the source code at https://github.com/AjuntamentdeBarcelona/decidim-barcelona
+
+image::barcelona.png[Decidim Barcelona]
+
+Some highlights:
+
+* In the https://github.com/AjuntamentdeBarcelona/decidim-barcelona/blob/4d88ec6106c5f29354a1fd069dd50d7d123e492a/Gemfile#L7[Gemfile], they use some modules that are local only, as folders in the repository using the "path" Bundler directive: `decidim-dataviz`, `decidim-stats`, `decidim-valid_auth`
+* For the footer (FEDER logo notice) they use the `deface` gem instead of overriding. See https://github.com/AjuntamentdeBarcelona/decidim-barcelona/pull/300[Pull Request].
+
+=== inDICEs
+
+See the website at https://participate.indices-culture.eu/
+
+See the source code at https://github.com/Platoniq/decidim-indices
+
+image::indices.png[inDICEs]
+
+Some highlights:
+
+* In the https://github.com/Platoniq/decidim-indices/blob/ef6d862900ed440aa5ee94c9618648650f3342c6/Gemfile#L13[Gemfile], they use modules on their git repositories: `decidim-decidim_awesome`, `decidim-direct_verifications`, `decidim-notify` and `decidim-term_customizer`.
+* They have their UI customized to their needs.
+

data/docs/modules/develop/pages/guide_git_conventions.adoc

@@ -0,0 +1,75 @@
+= Git Conventions
+
+== GitFlow Branching model
+
+The Decidim respository follows the GitFlow branching model. There are good documentations on it at:
+
+* https://nvie.com/posts/a-successful-git-branching-model/[the original post]
+* https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow[Atlassian tutorial]
+
+This model introduces the `develop` branch as a kind of queue for new features to enter into the next release.
+
+In summary, Decidim developers that work on `+feature/...+` or `+fix/...+` branches will branch off from `develop` and must be merged back into `develop`.
+
+Then, to start a new feature branch off from `develop` in the following way:
+
+[source,bash]
+----
+git checkout develop
+git checkout -b feature/xxx
+----
+
+Implement the feature, and open a Pull Request as normal, but against `develop` branch. As this is the most common operation, `develop` is the default branch instead of `master`.
+
+Please note that Decidim does not use the `master` branch.
+
+== Naming branches
+
+We would like to have all branches following this namings:
+
+|===
+| Branch prefix | Comment
+
+| chore/
+| Internal work. For instance, automatisms, etc. No production code change.
+
+| ci/
+| For continous integration related tasks. No production code change.
+
+| deps/
+| For dependency management tasks.
+
+| doc/
+| For changes to the documentation.
+
+| feature/
+| For new features for the users or for the Decidim command.
+
+| fix/
+| For feature bugfixing.
+
+| release/
+| With MAYOR.MINOR-stable. For instance, release/0.22-stable
+
+| refactor/
+| For refactorings related with production code.
+
+| test/
+| When adding missing tests, refactoring tests, improving coverage, etc.
+
+| backport/
+| We only offer support for the last mayor version.
+|===
+
+== Git commit messages and Pull Request titles
+
+We recommend following https://chris.beams.io/posts/git-commit/[this guide] for making good git commit messages. It also applies to Pull Request titles. The summary is:
+
+. Separate subject from body with a blank line
+. Limit the subject line to 50 characters
+. Capitalize the subject line
+. Do not end the subject line with a period
+. Use the imperative mood in the subject line
+. Wrap the body at 72 characters
+. Use the body to explain what and why vs. how
+

data/docs/modules/develop/pages/guide_github_projects.adoc

@@ -0,0 +1,42 @@
+= GitHub Projects Workflow
+
+This is an internal document on how we're organizing the development of Decidim with external contractors through Github projects.
+
+There are three teams:
+
+1. Product team: formed by representatives of the Barcelona City Council, Localret and Decidim Association who make functional decisions. They have the product vision and ensure that all the developments that are merged are coherent and consistent. You can summon them by mentioning in GitHub at @decidim/product
+2. Contractors: depends on the project. Most of the current developments are funded by the city of Barcelona, and the teams depend on the contract that has won the public procurement process. If there's any published contract we link it in the description of the project.
+3. Maintainers: developers that make the releases, review the Pull Requests, update dependencies and generally take care of all the Decidim source code. You can summon them by mentioning in GitHub at @decidim/core
+
+We follow some rules of Agile Development (mostly Scrum and KanBan).
+
+== Explanation of the columns
+
+=== Product Backlog
+
+The Product team prepares all the issues following the same template and order them by priority.
+
+=== Sprint Backlog
+
+Issues fully refined than will be tackled on the Sprint. Each Sprint has a duration of 3 weeks.
+The issues in this column will be agreed upon at the beginning of the Sprint between the Product Team and Contractors.
+
+=== Doing
+
+Issues on active development by Contractors.
+
+=== Ready
+
+Issues completed by Contractors. Ready to be tested and reviewed by Product Team.
+
+=== QA Testing
+
+Issues on functional testing and review by Product Team.
+
+=== Technical Review
+
+Quality and technical peer review by Maintainers.
+
+=== Done
+
+Merged by Maintainers. The issue can be closed.

data/docs/modules/develop/pages/guide_semver.adoc

@@ -0,0 +1,7 @@
+= Semantic Versioning
+
+For releases we follow https://semver.org/[Semantic Versioning recommendations]. Some details in our case:
+
+* Until v1 there could be changes in the API. We'll let you know on the Release Notes for a given version
+* Upgrading on patch versions should be safe (for instance from 0.20.0 to 0.20.1). The only things that we add are bug fixes and new languages.
+

data/docs/{advanced/how_to_fix_metrics.md → modules/develop/pages/how_to_fix_metrics.adoc}

@@ -1,43 +1,44 @@
-# How to fix metrics
+= How to fix metrics
 
 At the request of some instances, we have analyzed the issues related to metrics and looked for possible solutions.
 
-## Problems
+== Problems
 
 We have identified two main problems:
 
-- Metrics generation crashing, which cause `MetricJob`s to run again and again.
-- Peaks in generated metrics, sudden changes from day to day when displaying metrics.
+* Metrics generation crashing, which cause ``MetricJob``s to run again and again.
+* Peaks in generated metrics, sudden changes from day to day when displaying metrics.
 
-### Metrics generation crashing
+=== Metrics generation crashing
 
 We have identified only one culprit here: "orphans" records, meaning records whose related component or participatory space cannot be found in the database. This is because in a previous decidim release `PartipatorySpaces` could be deleted but they were not deleted properly. So any application that has deleted a participatory space in the past, will probably have unrelated records that will make some metrics calculation crash.
 
-### Peaks in generated metrics
+=== Peaks in generated metrics
 
-If somehow the metrics jobs fail to execute for a period of time, big differences can appear in metrics. So first make sure that you have metrics for every day, if not [generate them](https://github.com/decidim/decidim/blob/master/docs/advanced/metrics.md).
+If somehow the metrics jobs fail to execute for a period of time, big differences can appear in metrics. So first make sure that you have metrics for every day, if not https://github.com/decidim/decidim/blob/develop/docs/advanced/metrics.md[generate them].
 
 If you have metrics generated for almost everyday and still see drastic changes from day to day, take into account that changing the visibility of a component or participatory space (making them private or unpublishing them) will naturally cause big differences in generated metrics.
 
-Finally, if you see that the differences in some days are multiples of a previous generated metric, meaning suddenly you have exactly the double or the triple of a calculated metric, it's very likely that you have duplicate generated metrics. We have only seen this problem with instances using Sidekiq, not Delayed Job. We do not know the cause of this, but it seems to be a known issue [Avoiding duplicate jobs in Sidekiq](https://blog.francium.tech/avoiding-duplicate-jobs-in-sidekiq-dcbb1aca1e20).
+Finally, if you see that the differences in some days are multiples of a previous generated metric, meaning suddenly you have exactly the double or the triple of a calculated metric, it's very likely that you have duplicate generated metrics. We have only seen this problem with instances using Sidekiq, not Delayed Job. We do not know the cause of this, but it seems to be a known issue https://blog.francium.tech/avoiding-duplicate-jobs-in-sidekiq-dcbb1aca1e20[Avoiding duplicate jobs in Sidekiq].
 
-## Solutions
+== Solutions
 
 We cannot offer a definitive solution for duplicate metrics, other than to delete old duplicate metrics and generate them again. If this problem persists, however, consider using Delayed Job.
 For a given metric type (`rake decidim:metrics:list`) that has duplicates:
 
-- Option 1: Remove individually each metric record per day.
-- Option 2: Delete all metric records and recalculate them. [CHANGELOG](https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics) of decidim version 0.18 has an example for "participants".
+* Option 1: Remove individually each metric record per day.
+* Option 2: Delete all metric records and recalculate them. https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics[CHANGELOG] of decidim version 0.18 has an example for "participants".
 
 For orphan records, you can do the following:
 
-- Back up the database.
-- Delete orphan records fromt the console (code is below).
-- Delete "comments" metrics and recalculate them following the [aforementioned example](https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics).
+* Back up the database.
+* Delete orphan records fromt the console (code is below).
+* Delete "comments" metrics and recalculate them following the https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics[aforementioned example].
 
-### Some queries that may help
+=== Some queries that may help
 
-```ruby
+[source,ruby]
+----
     GROUP_BY_FIELDS= %w(
       day
       metric_type
@@ -62,177 +63,193 @@ For orphan records, you can do the following:
         SELECT count(1), #{GROUP_BY_FIELDS} FROM decidim_metrics GROUP BY #{GROUP_BY_FIELDS} HAVING COUNT(1) > 1;
       EOSQL
     end
-```
+----
 
-### Delete orphan records
+=== Delete orphan records
 
 "proposals", "meetings", "accountability", "debates", "pages", "budgets", "surveys"
 
-#### Proposals
+==== Proposals
 
 Delete proposals whose component does not have a participatory space and delete components of a proposal type that do not have a participatory space
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "proposals").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     Decidim::Proposals::Proposal.where(component: c).destroy_all
     c.destroy
   end
 }
-```
+----
 
 Delete proposals that do not have a component
 
-```ruby
+[source,ruby]
+----
 Decidim::Proposals::Proposal.find_each(batch_size: 100) { |proposal|
   proposal.delete if proposal.component.blank?
 }
-````
+----
 
-#### Meetings
+==== Meetings
 
 Delete meetings whose component has no participatory space and delete components of meeting type that do not have a participatory space
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "meetings").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     Decidim::Meetings::Meeting.where(component: c).destroy_all
     c.destroy
   end
 }
-```
+----
 
 Delete meetings that do not have a component
 
-```ruby
+[source,ruby]
+----
 Decidim::Meetings::Meeting.find_each(batch_size: 100) { |meeting|
   meeting.delete if meeting.component.blank?
 }
-````
+----
 
-#### Debates
+==== Debates
 
 Delete debates that its component has no participatory space and the debate components that do not have a participatory space
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "debates").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     Decidim::Debates::Debate.where(component: c).destroy_all
     c.destroy
   end
 }
-```
+----
 
 Destroy debates that do not have a component
 
-```ruby
+[source,ruby]
+----
 Decidim::Debates::Debate.find_each(batch_size: 100) { |debate|
   debate.delete if debate.component.blank?
 }
-```
+----
 
-#### Posts
+==== Posts
 
 Destroy posts whose component has no participatory space and blog components that do not have a participatory space
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "blogs").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     Decidim::Blogs::Post.where(component: c).destroy_all
     c.destroy
   end
 }
-```
+----
 
 Destroy posts that do not have a component
 
-```ruby
+[source,ruby]
+----
 Decidim::Blogs::Post.find_each(batch_size: 100) { |post|
   post.delete if post.component.blank?
 }
-```
+----
 
-#### Accountability
+==== Accountability
 
 Destroy results whose component has no participatory space and components of accountability type that do not have a participatory space
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "accountability").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     Decidim::Accountability::Result.where(component: c).destroy_all
     c.destroy
   end
 }
-```
+----
 
 Destroy results that do not have a component
 
-```ruby
+[source,ruby]
+----
 Decidim::Accountability::Result.find_each(batch_size: 100) { |result|
   result.delete if result.component.blank?
 }
-```
+----
 
-#### Pages
+==== Pages
 
 Destroy page components that do not have a participatory space
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "pages").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     c.destroy
   end
 }
-```
+----
 
-#### Budgets
+==== Budgets
 
 Destroy projects whose component has no participatory space and budget components that do not have a participatory space
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "budgets").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     Decidim::Budgets::Project.where(component: c).destroy_all
     c.destroy
   end
 }
-```
+----
 
 Destroy results that do not have a component
 
-```ruby
+[source,ruby]
+----
 Decidim::Budgets::Project.find_each(batch_size: 100) { |project|
   project.delete if project.component.blank?
 }
-```
+----
 
-#### Surveys
+==== Surveys
 
-```ruby
+[source,ruby]
+----
 Decidim::Component.where(manifest_name: "surveys").find_each(batch_size: 100) { |c|
   if c.participatory_space.blank?
     Decidim::Surveys::Survey.where(component: c).destroy_all
     c.destroy
   end
 }
-```
+----
 
 Destroy surveys that do not have a component
 
-```ruby
+[source,ruby]
+----
 Decidim::Surveys::Survey.find_each(batch_size: 100) { |survey|
   survey.delete if survey.component.blank?
 }
-```
+----
 
-#### Comments
+==== Comments
 
 Destroy comments whose commentable root is a proposal that does not have a participatory space.
 
-```ruby
+[source,ruby]
+----
 proposal_ids = Decidim::Comments::Comment.where(decidim_root_commentable_type: "Decidim::Proposals::Proposal").pluck(:decidim_root_commentable_id)
 
 proposal_ids_without_space = Decidim::Proposals::Proposal.where(id: proposal_ids).find_all{|p| p.participatory_space.blank? }.pluck(:id)
 
 Decidim::Comments::Comment.where(decidim_root_commentable_type: "Decidim::Proposals::Proposal", decidim_root_commentable_id: proposal_ids_without_space).destroy_all
-```
+----