decidim 0.23.6 → 0.24.0.rc1

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of decidim might be problematic. Click here for more details.

Files changed (114) hide show
  1. checksums.yaml +4 -4
  2. data/Rakefile +1 -0
  3. data/docs/README.adoc +74 -0
  4. data/docs/antora.yml +7 -0
  5. data/docs/modules/configure/pages/environment_variables.adoc +69 -0
  6. data/docs/modules/configure/pages/index.adoc +16 -0
  7. data/docs/modules/configure/pages/initializer.adoc +376 -0
  8. data/docs/modules/customize/assets/images/header-snippet.png +0 -0
  9. data/docs/modules/customize/assets/images/menu.png +0 -0
  10. data/docs/modules/customize/assets/images/organization-colors.png +0 -0
  11. data/docs/modules/customize/pages/authorizations.adoc +22 -0
  12. data/docs/{customization/code.md → modules/customize/pages/code.adoc} +12 -9
  13. data/docs/{customization/gemfile.md → modules/customize/pages/gemfile.adoc} +5 -4
  14. data/docs/modules/customize/pages/images.adoc +7 -0
  15. data/docs/modules/customize/pages/javascript.adoc +59 -0
  16. data/docs/modules/customize/pages/menu.adoc +25 -0
  17. data/docs/modules/customize/pages/oauth.adoc +33 -0
  18. data/docs/modules/customize/pages/styles.adoc +64 -0
  19. data/docs/modules/customize/pages/texts.adoc +30 -0
  20. data/docs/modules/customize/pages/users_registration_mode.adoc +17 -0
  21. data/docs/{customization/views.md → modules/customize/pages/views.adoc} +13 -13
  22. data/docs/modules/develop/assets/images/barcelona.png +0 -0
  23. data/docs/modules/develop/assets/images/helsinki.png +0 -0
  24. data/docs/modules/develop/assets/images/indices.png +0 -0
  25. data/docs/{advanced/api.md → modules/develop/pages/api.adoc} +2 -2
  26. data/docs/{advanced/authorship.md → modules/develop/pages/authorable.adoc} +5 -5
  27. data/docs/modules/develop/pages/c4_component.adoc +91 -0
  28. data/docs/modules/develop/pages/c4_container.adoc +42 -0
  29. data/docs/modules/develop/pages/c4_context.adoc +35 -0
  30. data/docs/{advanced/components.md → modules/develop/pages/components.adoc} +47 -10
  31. data/docs/{advanced/content_blocks.md → modules/develop/pages/content_blocks.adoc} +16 -13
  32. data/docs/{advanced/content_processors.md → modules/develop/pages/content_processors.adoc} +25 -19
  33. data/docs/modules/develop/pages/data-picker.adoc +85 -0
  34. data/docs/modules/develop/pages/deploy.adoc +15 -0
  35. data/docs/modules/develop/pages/docker.adoc +12 -0
  36. data/docs/{advanced/embeddable.md → modules/develop/pages/embeddable.adoc} +6 -6
  37. data/docs/{advanced/endorsable.md → modules/develop/pages/endorsable.adoc} +31 -25
  38. data/docs/{advanced/fixing_locales.md → modules/develop/pages/fixing_locales.adoc} +36 -23
  39. data/docs/{advanced/followers.md → modules/develop/pages/followable.adoc} +9 -8
  40. data/docs/modules/develop/pages/guide.adoc +16 -0
  41. data/docs/modules/develop/pages/guide_architecture.adoc +17 -0
  42. data/docs/modules/develop/pages/guide_changelog.adoc +8 -0
  43. data/docs/modules/develop/pages/guide_commands.adoc +86 -0
  44. data/docs/modules/develop/pages/guide_development_app.adoc +44 -0
  45. data/docs/modules/develop/pages/guide_development_with_custom_seed_data.adoc +31 -0
  46. data/docs/modules/develop/pages/guide_development_with_localhost_ssl.adoc +63 -0
  47. data/docs/modules/develop/pages/guide_example_apps.adoc +59 -0
  48. data/docs/modules/develop/pages/guide_git_conventions.adoc +75 -0
  49. data/docs/modules/develop/pages/guide_github_projects.adoc +42 -0
  50. data/docs/modules/develop/pages/guide_semver.adoc +7 -0
  51. data/docs/{advanced/how_to_fix_metrics.md → modules/develop/pages/how_to_fix_metrics.adoc} +76 -59
  52. data/docs/modules/develop/pages/machine_translations.adoc +42 -0
  53. data/docs/modules/develop/pages/managing_translations_i18n.adoc +24 -0
  54. data/docs/modules/develop/pages/maps.adoc +499 -0
  55. data/docs/modules/develop/pages/metrics.adoc +119 -0
  56. data/docs/{advanced/modules.md → modules/develop/pages/modules.adoc} +16 -6
  57. data/docs/{advanced/newsletter_templates.md → modules/develop/pages/newsletter_templates.adoc} +12 -10
  58. data/docs/{advanced/notifications.md → modules/develop/pages/notifications.adoc} +40 -38
  59. data/docs/{advanced/open-data.md → modules/develop/pages/open-data.adoc} +4 -3
  60. data/docs/modules/develop/pages/permissions.adoc +92 -0
  61. data/docs/{advanced/profiling.md → modules/develop/pages/profiling.adoc} +15 -12
  62. data/docs/modules/develop/pages/releases.adoc +148 -0
  63. data/docs/modules/develop/pages/reportable.adoc +31 -0
  64. data/docs/modules/develop/pages/security.adoc +33 -0
  65. data/docs/{advanced/share_tokens.md → modules/develop/pages/share_tokens.adoc} +18 -14
  66. data/docs/{advanced/templates.md → modules/develop/pages/templates.adoc} +14 -12
  67. data/docs/{advanced/testing.md → modules/develop/pages/testing.adoc} +21 -20
  68. data/docs/{advanced/activity_log.md → modules/develop/pages/traceable.adoc} +31 -26
  69. data/docs/modules/develop/pages/turbolinks.adoc +7 -0
  70. data/docs/{advanced/view_hooks.md → modules/develop/pages/view_hooks.adoc} +29 -23
  71. data/docs/modules/develop/pages/view_models_aka_cells.adoc +105 -0
  72. data/docs/modules/install/pages/checklist.adoc +39 -0
  73. data/docs/modules/install/pages/index.adoc +148 -0
  74. data/docs/{manual-installation.md → modules/install/pages/manual.adoc} +54 -42
  75. data/docs/modules/install/pages/update.adoc +95 -0
  76. data/docs/{services/activejob.md → modules/services/pages/activejob.adoc} +3 -3
  77. data/docs/modules/services/pages/elections_bulletin_board.adoc +52 -0
  78. data/docs/{services/etherpad.md → modules/services/pages/etherpad.adoc} +15 -12
  79. data/docs/modules/services/pages/maps.adoc +311 -0
  80. data/docs/modules/services/pages/smtp.adoc +10 -0
  81. data/docs/modules/services/pages/social_providers.adoc +122 -0
  82. data/lib/decidim/gem_manager.rb +5 -5
  83. data/lib/decidim/version.rb +1 -1
  84. metadata +137 -100
  85. data/README.md +0 -157
  86. data/docs/advanced/add_authorizable_action.md +0 -63
  87. data/docs/advanced/adding_fixtures_aka_dummy_content.md +0 -9
  88. data/docs/advanced/data-picker.md +0 -83
  89. data/docs/advanced/deploy.md +0 -9
  90. data/docs/advanced/how_to_create_a_module.md +0 -9
  91. data/docs/advanced/machine_translation_service.md +0 -12
  92. data/docs/advanced/managing_translations_i18n.md +0 -24
  93. data/docs/advanced/metrics.md +0 -114
  94. data/docs/advanced/permissions.md +0 -23
  95. data/docs/advanced/releases.md +0 -114
  96. data/docs/advanced/tradeoffs.md +0 -14
  97. data/docs/advanced/view_models_aka_cells.md +0 -99
  98. data/docs/checklist.md +0 -55
  99. data/docs/customization/authorizations.md +0 -5
  100. data/docs/customization/images.md +0 -7
  101. data/docs/customization/javascript.md +0 -9
  102. data/docs/customization/machine_translations.md +0 -30
  103. data/docs/customization/maps.md +0 -610
  104. data/docs/customization/oauth.md +0 -50
  105. data/docs/customization/styles.md +0 -11
  106. data/docs/customization/texts.md +0 -27
  107. data/docs/customization/users_registration_mode.md +0 -17
  108. data/docs/development_guide.md +0 -166
  109. data/docs/getting_started.md +0 -191
  110. data/docs/possible_flows_for_proposal.png +0 -0
  111. data/docs/services/analytics.md +0 -23
  112. data/docs/services/elections_bulletin_board.md +0 -38
  113. data/docs/services/maps.md +0 -362
  114. data/docs/services/social_providers.md +0 -98
@@ -0,0 +1,44 @@
1
+ = Development App
2
+
3
+ In order to start developing you will need what is called a `development_app`. This is nearly the same as a new Decidim app (that you can create with `decidim app_name`) but with a Gemfile pre-configured for local development and some other small config modifications.
4
+ You need it in order to have a Rails application configured to lookup Decidim modules from your filesystem. This way changes in your modules will be directly observed by this `development_app`.
5
+
6
+ You can create a `development_app` from inside the project's root folder with the command:
7
+
8
+ [source,console]
9
+ ----
10
+ git clone https://github.com/decidim/decidim.git
11
+ cd decidim
12
+ bundle install
13
+ bundle exec rake development_app
14
+ cd development_app
15
+ ----
16
+
17
+ A development_app/ entry appears in the .gitignore file, so you don't have to worry about committing the development app by mistake.
18
+
19
+ On creation, this steps are automatically invoked by the generator:
20
+
21
+ * create a `config/database.yml`
22
+ * `bundle install`
23
+ * `bin/rails decidim:upgrade`
24
+ * `bin/rails db:migrate db:seed`
25
+
26
+ If the default database.yml does not suit your needs you can always configure it at your will and run this steps manually.
27
+
28
+ The last command will set your database and add some example data (called "seed data") so that you can start trying Decidim. We don't recommend using seed data for production environments, but it's useful for local development and staging environments.
29
+
30
+ Once the app is created you are ready to start the server:
31
+
32
+ * `bin/rails s`
33
+
34
+ == Migrations
35
+
36
+ When creating new migrations in Decidim's modules, you will need to "apply" this migrations to your development_app. The way to do this is by copying the migration from your module into the db/migrate dir of your development_app. Luckily we already have a script that automates this: it copies all missing migrations in development_app/db/migrate. The command is:
37
+
38
+ [source,console]
39
+ ----
40
+ bin/rails decidim:upgrade
41
+ ----
42
+
43
+ Anyway we recommend re-creating your development_app every once in a while.
44
+
@@ -0,0 +1,31 @@
1
+ = Development with custom seed data
2
+
3
+ The seed data is not only useful for local development, but also for staging environments to showcase your work to your clients. In this case, you might need to customize some of that seed data to make sure it fits your needs.
4
+
5
+ == Customizing seed data via the `db/seeds.rb` file
6
+
7
+ One way to customize the seed data is to use the `db/seeds.rb` file in your application. You'll see a call to generate the Decidim seeds:
8
+
9
+ [source,ruby]
10
+ ----
11
+ Decidim.seed!
12
+ ----
13
+
14
+ This will generate all the needed seed data. But after that you might want to, for example, create some specific scopes, or change the default user password. You can do anything from that file.
15
+
16
+ == Customizing the seed data via environment variables
17
+
18
+ Some specific pieces of the seed data can be customized via environment variables. This is specially useful to set the SMTP credentials or the System Admin password.
19
+
20
+ If you set the variables _before_ creating the seeds data, the values in those variables will be used.
21
+
22
+ * `SMTP_FROM_LABEL`: The name used to send emails.
23
+ * `SMTP_FROM_EMAIL`: The email where emails will be sent from.
24
+ * `SMTP_USERNAME`: The username required by your email sending system.
25
+ * `SMTP_PASSWORD`: The password required by your email sending system.
26
+ * `SMTP_ADDRESS`: The address where your email sending system lives.
27
+ * `SMTP_PORT`: The port used to connect to your email sending system.
28
+ * `DECIDIM_SYSTEM_USER_PASSWORD`: The password for the System user. Useful to hide access to the System panel, where the SMTP config can be retrieved in plain text.
29
+
30
+ This is useful, for example, to set up a staging application open to anyone, and use Sendgrid as a mailing system. By changing the default password only you will be able to access the System panel, so nobody will be able to see the credentials used to connect to Sendgrid.
31
+
@@ -0,0 +1,63 @@
1
+ = Testing SSL in localhost
2
+
3
+ In some cases is useful to access the `localhost` environment under SSL (ie: `https://localhost:3000/`).
4
+
5
+ A typical use case is when you want to test login integration with 3rd party systems using OAuth (which in the latest versions only the HTTPS protocol is allowed).
6
+
7
+ Using Puma, the default server for Rails, you can easily do the trick by creating a self-signed certificate and using it when starting your development environment.
8
+
9
+ == Create a self-signed certificate for localhost
10
+
11
+ First you need a SSL certificate for the `localhost` domain (you only need to execute this command once):
12
+
13
+ [source,bash]
14
+ ----
15
+ openssl req -x509 -out localhost.crt -keyout localhost.key \
16
+ -newkey rsa:2048 -nodes -sha256 \
17
+ -subj '/CN=localhost' -extensions EXT -config <( \
18
+ printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
19
+ ----
20
+
21
+ This will generate two files: `localhost.crt` and `localhost.key`, you can store them anywhere you want to use them later. In this example we'll suppose we have them in the folder `certs` in our root user home:
22
+
23
+ [source,bash]
24
+ ----
25
+ mkdir ~/certs/
26
+ mv localhost.crt ~/certs/
27
+ mv localhost.key ~/certs/
28
+ ----
29
+
30
+ == Starting development in SSL mode
31
+
32
+ Once you have your certificate is generated, just use the next command in order to start puma instead of the typical `bin/rails s`:
33
+
34
+ [source,bash]
35
+ ----
36
+ bin/rails s -b "ssl://127.0.0.1:3000?key=$HOME/certs/localhost.key&cert=$HOME/certs/localhost.crt"
37
+ ----
38
+
39
+ Now you are ready to visit your favorite browser at the address `https://localhost:3000/` (note "https").
40
+
41
+ Note: Your browser is going to complain as this is a self-signed certificate, that's ok for development, just add an exception and accept the certificate.
42
+
43
+ Also take into account that starting Puma in SSL mode will disable accessing it in non-ssl mode (normal `http` protocol).
44
+
45
+ == Testing tenants using lvh.me
46
+
47
+ You can also test the multi-tenant capabilities of Decidim by using alternative domains or subdomains that points to your local machine. `lvh.me` is a service that does just that without configuring anything in your machine (an alternative is to add entries in your `/etc/hosts` with a testing domain of your choice). Just point your browser to any subdomain of `lvh.me` and you'll be in your own machine.
48
+
49
+ Just access your `/system` admin and create new organization with some subdomain of `lvh.me`: `organization.lvh.me`, `tenant2.lvh.me`, etc.
50
+
51
+ You can combine this with the previously generated certificate (your browser is going to complaint but just tell it to proceed vising the site).
52
+
53
+ Finally remember to add the port as `lvh.me` do not forwards anything, for instance (use `http` or `https` depending on how you've started Rails):
54
+
55
+ `http://someorg.lvh.me:3000/`
56
+
57
+ `https://someorg.lvh.me:3000/`
58
+
59
+ `http://anotherorg.lvh.me:3000/`
60
+
61
+ `https://anotherorg.lvh.me:3000/`
62
+
63
+ etc.
@@ -0,0 +1,59 @@
1
+ = Example Apps
2
+
3
+ If you have worked in the past with Ruby on Rails, you'll notice quickly that Decidim is *not* your typical Ruby on Rails Vanilla App.
4
+
5
+ We've tried using http://decide.es[Consul] but we found some problems on reutilization, adaptation, modularization and configuration. You can read more about that on https://alabs.gitbooks.io/propuesta-de-cambios-en-la-arquitectura-de-consul/content/[Propuesta de Cambios de Arquitectura de Consul (in spanish only)]. As a summary, we decided to try with a model based in https://guides.rubyonrails.org/engines.html[RoR Engines]. A good resource regarding this approach is https://cbra.info/[Component Based Rails Applications].
6
+
7
+ == What this means
8
+
9
+ This means that:
10
+
11
+ . Decidim is a https://rubygems.org/gems/decidim[gem], available in rubygems.org. You can install it with the command `gem install decidim`
12
+ . Decidim is a command line generator of Ruby on Rails applications. After you've install it you can generate a new RoR app with `decidim HelloWorld`
13
+ . This app has a dependency the gem itself. You can see this in the Gemfile of the generated app.
14
+ . This app can be extended by making changes in the app (see xref:customize:index.adoc[Customising]) and by working with https://decidim.org/modules[Modules].
15
+
16
+ == Examples
17
+
18
+ Here are some good examples of what can be done with Decidim:
19
+
20
+ === Helsinki OmaStadi
21
+
22
+ See the website at https://omastadi.hel.fi/
23
+
24
+ See the source code at https://github.com/City-of-Helsinki/decidim-helsinki/
25
+
26
+ image::helsinki.png[Helsinki OmaStadi]
27
+
28
+ Some highlights:
29
+
30
+ * They have their https://github.com/City-of-Helsinki/decidim-helsinki/blob/b9a09e570eb6090dee93f2ee73b5951882c74755/Gemfile[Gemfile] with lots of modules with their own code. Some of them are also in ruybgems (like `decidim-access_requests`, `decidim-antivirus`, `decidim-mpassid`, `decidim-process_groups_content_block`, `decidim-suomifi`, `decidim-term_customizer`), others are only in Github (`decidim-accountability_simple`, `decidim-apiauth`, `decidim-combined_budgeting`, `decidim-plans`, `decidim-redirects`). They even use a module made by other organization in the community (`decidim-budgets_enhanced`)
31
+ * They have their UI customized to their needs. See the https://github.com/City-of-Helsinki/decidim-helsinki/tree/a7396a312cc04198654a86d66c2a7de556c212af/app/assets/stylesheets[CSS] and https://github.com/City-of-Helsinki/decidim-helsinki/tree/a7396a312cc04198654a86d66c2a7de556c212af/app/views[HTML (views)].
32
+ * They have an awesome READE explaining how it works and what changes have made.
33
+
34
+ === Decidim Barcelona
35
+
36
+ See the website at https://decidim.barcelona
37
+
38
+ See the source code at https://github.com/AjuntamentdeBarcelona/decidim-barcelona
39
+
40
+ image::barcelona.png[Decidim Barcelona]
41
+
42
+ Some highlights:
43
+
44
+ * In the https://github.com/AjuntamentdeBarcelona/decidim-barcelona/blob/4d88ec6106c5f29354a1fd069dd50d7d123e492a/Gemfile#L7[Gemfile], they use some modules that are local only, as folders in the repository using the "path" Bundler directive: `decidim-dataviz`, `decidim-stats`, `decidim-valid_auth`
45
+ * For the footer (FEDER logo notice) they use the `deface` gem instead of overriding. See https://github.com/AjuntamentdeBarcelona/decidim-barcelona/pull/300[Pull Request].
46
+
47
+ === inDICEs
48
+
49
+ See the website at https://participate.indices-culture.eu/
50
+
51
+ See the source code at https://github.com/Platoniq/decidim-indices
52
+
53
+ image::indices.png[inDICEs]
54
+
55
+ Some highlights:
56
+
57
+ * In the https://github.com/Platoniq/decidim-indices/blob/ef6d862900ed440aa5ee94c9618648650f3342c6/Gemfile#L13[Gemfile], they use modules on their git repositories: `decidim-decidim_awesome`, `decidim-direct_verifications`, `decidim-notify` and `decidim-term_customizer`.
58
+ * They have their UI customized to their needs.
59
+
@@ -0,0 +1,75 @@
1
+ = Git Conventions
2
+
3
+ == GitFlow Branching model
4
+
5
+ The Decidim respository follows the GitFlow branching model. There are good documentations on it at:
6
+
7
+ * https://nvie.com/posts/a-successful-git-branching-model/[the original post]
8
+ * https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow[Atlassian tutorial]
9
+
10
+ This model introduces the `develop` branch as a kind of queue for new features to enter into the next release.
11
+
12
+ In summary, Decidim developers that work on `+feature/...+` or `+fix/...+` branches will branch off from `develop` and must be merged back into `develop`.
13
+
14
+ Then, to start a new feature branch off from `develop` in the following way:
15
+
16
+ [source,bash]
17
+ ----
18
+ git checkout develop
19
+ git checkout -b feature/xxx
20
+ ----
21
+
22
+ Implement the feature, and open a Pull Request as normal, but against `develop` branch. As this is the most common operation, `develop` is the default branch instead of `master`.
23
+
24
+ Please note that Decidim does not use the `master` branch.
25
+
26
+ == Naming branches
27
+
28
+ We would like to have all branches following this namings:
29
+
30
+ |===
31
+ | Branch prefix | Comment
32
+
33
+ | chore/
34
+ | Internal work. For instance, automatisms, etc. No production code change.
35
+
36
+ | ci/
37
+ | For continous integration related tasks. No production code change.
38
+
39
+ | deps/
40
+ | For dependency management tasks.
41
+
42
+ | doc/
43
+ | For changes to the documentation.
44
+
45
+ | feature/
46
+ | For new features for the users or for the Decidim command.
47
+
48
+ | fix/
49
+ | For feature bugfixing.
50
+
51
+ | release/
52
+ | With MAYOR.MINOR-stable. For instance, release/0.22-stable
53
+
54
+ | refactor/
55
+ | For refactorings related with production code.
56
+
57
+ | test/
58
+ | When adding missing tests, refactoring tests, improving coverage, etc.
59
+
60
+ | backport/
61
+ | We only offer support for the last mayor version.
62
+ |===
63
+
64
+ == Git commit messages and Pull Request titles
65
+
66
+ We recommend following https://chris.beams.io/posts/git-commit/[this guide] for making good git commit messages. It also applies to Pull Request titles. The summary is:
67
+
68
+ . Separate subject from body with a blank line
69
+ . Limit the subject line to 50 characters
70
+ . Capitalize the subject line
71
+ . Do not end the subject line with a period
72
+ . Use the imperative mood in the subject line
73
+ . Wrap the body at 72 characters
74
+ . Use the body to explain what and why vs. how
75
+
@@ -0,0 +1,42 @@
1
+ = GitHub Projects Workflow
2
+
3
+ This is an internal document on how we're organizing the development of Decidim with external contractors through Github projects.
4
+
5
+ There are three teams:
6
+
7
+ 1. Product team: formed by representatives of the Barcelona City Council, Localret and Decidim Association who make functional decisions. They have the product vision and ensure that all the developments that are merged are coherent and consistent. You can summon them by mentioning in GitHub at @decidim/product
8
+ 2. Contractors: depends on the project. Most of the current developments are funded by the city of Barcelona, and the teams depend on the contract that has won the public procurement process. If there's any published contract we link it in the description of the project.
9
+ 3. Maintainers: developers that make the releases, review the Pull Requests, update dependencies and generally take care of all the Decidim source code. You can summon them by mentioning in GitHub at @decidim/core
10
+
11
+ We follow some rules of Agile Development (mostly Scrum and KanBan).
12
+
13
+ == Explanation of the columns
14
+
15
+ === Product Backlog
16
+
17
+ The Product team prepares all the issues following the same template and order them by priority.
18
+
19
+ === Sprint Backlog
20
+
21
+ Issues fully refined than will be tackled on the Sprint. Each Sprint has a duration of 3 weeks.
22
+ The issues in this column will be agreed upon at the beginning of the Sprint between the Product Team and Contractors.
23
+
24
+ === Doing
25
+
26
+ Issues on active development by Contractors.
27
+
28
+ === Ready
29
+
30
+ Issues completed by Contractors. Ready to be tested and reviewed by Product Team.
31
+
32
+ === QA Testing
33
+
34
+ Issues on functional testing and review by Product Team.
35
+
36
+ === Technical Review
37
+
38
+ Quality and technical peer review by Maintainers.
39
+
40
+ === Done
41
+
42
+ Merged by Maintainers. The issue can be closed.
@@ -0,0 +1,7 @@
1
+ = Semantic Versioning
2
+
3
+ For releases we follow https://semver.org/[Semantic Versioning recommendations]. Some details in our case:
4
+
5
+ * Until v1 there could be changes in the API. We'll let you know on the Release Notes for a given version
6
+ * Upgrading on patch versions should be safe (for instance from 0.20.0 to 0.20.1). The only things that we add are bug fixes and new languages.
7
+
@@ -1,43 +1,44 @@
1
- # How to fix metrics
1
+ = How to fix metrics
2
2
 
3
3
  At the request of some instances, we have analyzed the issues related to metrics and looked for possible solutions.
4
4
 
5
- ## Problems
5
+ == Problems
6
6
 
7
7
  We have identified two main problems:
8
8
 
9
- - Metrics generation crashing, which cause `MetricJob`s to run again and again.
10
- - Peaks in generated metrics, sudden changes from day to day when displaying metrics.
9
+ * Metrics generation crashing, which cause ``MetricJob``s to run again and again.
10
+ * Peaks in generated metrics, sudden changes from day to day when displaying metrics.
11
11
 
12
- ### Metrics generation crashing
12
+ === Metrics generation crashing
13
13
 
14
14
  We have identified only one culprit here: "orphans" records, meaning records whose related component or participatory space cannot be found in the database. This is because in a previous decidim release `PartipatorySpaces` could be deleted but they were not deleted properly. So any application that has deleted a participatory space in the past, will probably have unrelated records that will make some metrics calculation crash.
15
15
 
16
- ### Peaks in generated metrics
16
+ === Peaks in generated metrics
17
17
 
18
- If somehow the metrics jobs fail to execute for a period of time, big differences can appear in metrics. So first make sure that you have metrics for every day, if not [generate them](https://github.com/decidim/decidim/blob/master/docs/advanced/metrics.md).
18
+ If somehow the metrics jobs fail to execute for a period of time, big differences can appear in metrics. So first make sure that you have metrics for every day, if not https://github.com/decidim/decidim/blob/develop/docs/advanced/metrics.md[generate them].
19
19
 
20
20
  If you have metrics generated for almost everyday and still see drastic changes from day to day, take into account that changing the visibility of a component or participatory space (making them private or unpublishing them) will naturally cause big differences in generated metrics.
21
21
 
22
- Finally, if you see that the differences in some days are multiples of a previous generated metric, meaning suddenly you have exactly the double or the triple of a calculated metric, it's very likely that you have duplicate generated metrics. We have only seen this problem with instances using Sidekiq, not Delayed Job. We do not know the cause of this, but it seems to be a known issue [Avoiding duplicate jobs in Sidekiq](https://blog.francium.tech/avoiding-duplicate-jobs-in-sidekiq-dcbb1aca1e20).
22
+ Finally, if you see that the differences in some days are multiples of a previous generated metric, meaning suddenly you have exactly the double or the triple of a calculated metric, it's very likely that you have duplicate generated metrics. We have only seen this problem with instances using Sidekiq, not Delayed Job. We do not know the cause of this, but it seems to be a known issue https://blog.francium.tech/avoiding-duplicate-jobs-in-sidekiq-dcbb1aca1e20[Avoiding duplicate jobs in Sidekiq].
23
23
 
24
- ## Solutions
24
+ == Solutions
25
25
 
26
26
  We cannot offer a definitive solution for duplicate metrics, other than to delete old duplicate metrics and generate them again. If this problem persists, however, consider using Delayed Job.
27
27
  For a given metric type (`rake decidim:metrics:list`) that has duplicates:
28
28
 
29
- - Option 1: Remove individually each metric record per day.
30
- - Option 2: Delete all metric records and recalculate them. [CHANGELOG](https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics) of decidim version 0.18 has an example for "participants".
29
+ * Option 1: Remove individually each metric record per day.
30
+ * Option 2: Delete all metric records and recalculate them. https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics[CHANGELOG] of decidim version 0.18 has an example for "participants".
31
31
 
32
32
  For orphan records, you can do the following:
33
33
 
34
- - Back up the database.
35
- - Delete orphan records fromt the console (code is below).
36
- - Delete "comments" metrics and recalculate them following the [aforementioned example](https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics).
34
+ * Back up the database.
35
+ * Delete orphan records fromt the console (code is below).
36
+ * Delete "comments" metrics and recalculate them following the https://github.com/decidim/decidim/blob/release/0.18-stable/CHANGELOG.md#participants-metrics[aforementioned example].
37
37
 
38
- ### Some queries that may help
38
+ === Some queries that may help
39
39
 
40
- ```ruby
40
+ [source,ruby]
41
+ ----
41
42
  GROUP_BY_FIELDS= %w(
42
43
  day
43
44
  metric_type
@@ -62,177 +63,193 @@ For orphan records, you can do the following:
62
63
  SELECT count(1), #{GROUP_BY_FIELDS} FROM decidim_metrics GROUP BY #{GROUP_BY_FIELDS} HAVING COUNT(1) > 1;
63
64
  EOSQL
64
65
  end
65
- ```
66
+ ----
66
67
 
67
- ### Delete orphan records
68
+ === Delete orphan records
68
69
 
69
70
  "proposals", "meetings", "accountability", "debates", "pages", "budgets", "surveys"
70
71
 
71
- #### Proposals
72
+ ==== Proposals
72
73
 
73
74
  Delete proposals whose component does not have a participatory space and delete components of a proposal type that do not have a participatory space
74
75
 
75
- ```ruby
76
+ [source,ruby]
77
+ ----
76
78
  Decidim::Component.where(manifest_name: "proposals").find_each(batch_size: 100) { |c|
77
79
  if c.participatory_space.blank?
78
80
  Decidim::Proposals::Proposal.where(component: c).destroy_all
79
81
  c.destroy
80
82
  end
81
83
  }
82
- ```
84
+ ----
83
85
 
84
86
  Delete proposals that do not have a component
85
87
 
86
- ```ruby
88
+ [source,ruby]
89
+ ----
87
90
  Decidim::Proposals::Proposal.find_each(batch_size: 100) { |proposal|
88
91
  proposal.delete if proposal.component.blank?
89
92
  }
90
- ````
93
+ ----
91
94
 
92
- #### Meetings
95
+ ==== Meetings
93
96
 
94
97
  Delete meetings whose component has no participatory space and delete components of meeting type that do not have a participatory space
95
98
 
96
- ```ruby
99
+ [source,ruby]
100
+ ----
97
101
  Decidim::Component.where(manifest_name: "meetings").find_each(batch_size: 100) { |c|
98
102
  if c.participatory_space.blank?
99
103
  Decidim::Meetings::Meeting.where(component: c).destroy_all
100
104
  c.destroy
101
105
  end
102
106
  }
103
- ```
107
+ ----
104
108
 
105
109
  Delete meetings that do not have a component
106
110
 
107
- ```ruby
111
+ [source,ruby]
112
+ ----
108
113
  Decidim::Meetings::Meeting.find_each(batch_size: 100) { |meeting|
109
114
  meeting.delete if meeting.component.blank?
110
115
  }
111
- ````
116
+ ----
112
117
 
113
- #### Debates
118
+ ==== Debates
114
119
 
115
120
  Delete debates that its component has no participatory space and the debate components that do not have a participatory space
116
121
 
117
- ```ruby
122
+ [source,ruby]
123
+ ----
118
124
  Decidim::Component.where(manifest_name: "debates").find_each(batch_size: 100) { |c|
119
125
  if c.participatory_space.blank?
120
126
  Decidim::Debates::Debate.where(component: c).destroy_all
121
127
  c.destroy
122
128
  end
123
129
  }
124
- ```
130
+ ----
125
131
 
126
132
  Destroy debates that do not have a component
127
133
 
128
- ```ruby
134
+ [source,ruby]
135
+ ----
129
136
  Decidim::Debates::Debate.find_each(batch_size: 100) { |debate|
130
137
  debate.delete if debate.component.blank?
131
138
  }
132
- ```
139
+ ----
133
140
 
134
- #### Posts
141
+ ==== Posts
135
142
 
136
143
  Destroy posts whose component has no participatory space and blog components that do not have a participatory space
137
144
 
138
- ```ruby
145
+ [source,ruby]
146
+ ----
139
147
  Decidim::Component.where(manifest_name: "blogs").find_each(batch_size: 100) { |c|
140
148
  if c.participatory_space.blank?
141
149
  Decidim::Blogs::Post.where(component: c).destroy_all
142
150
  c.destroy
143
151
  end
144
152
  }
145
- ```
153
+ ----
146
154
 
147
155
  Destroy posts that do not have a component
148
156
 
149
- ```ruby
157
+ [source,ruby]
158
+ ----
150
159
  Decidim::Blogs::Post.find_each(batch_size: 100) { |post|
151
160
  post.delete if post.component.blank?
152
161
  }
153
- ```
162
+ ----
154
163
 
155
- #### Accountability
164
+ ==== Accountability
156
165
 
157
166
  Destroy results whose component has no participatory space and components of accountability type that do not have a participatory space
158
167
 
159
- ```ruby
168
+ [source,ruby]
169
+ ----
160
170
  Decidim::Component.where(manifest_name: "accountability").find_each(batch_size: 100) { |c|
161
171
  if c.participatory_space.blank?
162
172
  Decidim::Accountability::Result.where(component: c).destroy_all
163
173
  c.destroy
164
174
  end
165
175
  }
166
- ```
176
+ ----
167
177
 
168
178
  Destroy results that do not have a component
169
179
 
170
- ```ruby
180
+ [source,ruby]
181
+ ----
171
182
  Decidim::Accountability::Result.find_each(batch_size: 100) { |result|
172
183
  result.delete if result.component.blank?
173
184
  }
174
- ```
185
+ ----
175
186
 
176
- #### Pages
187
+ ==== Pages
177
188
 
178
189
  Destroy page components that do not have a participatory space
179
190
 
180
- ```ruby
191
+ [source,ruby]
192
+ ----
181
193
  Decidim::Component.where(manifest_name: "pages").find_each(batch_size: 100) { |c|
182
194
  if c.participatory_space.blank?
183
195
  c.destroy
184
196
  end
185
197
  }
186
- ```
198
+ ----
187
199
 
188
- #### Budgets
200
+ ==== Budgets
189
201
 
190
202
  Destroy projects whose component has no participatory space and budget components that do not have a participatory space
191
203
 
192
- ```ruby
204
+ [source,ruby]
205
+ ----
193
206
  Decidim::Component.where(manifest_name: "budgets").find_each(batch_size: 100) { |c|
194
207
  if c.participatory_space.blank?
195
208
  Decidim::Budgets::Project.where(component: c).destroy_all
196
209
  c.destroy
197
210
  end
198
211
  }
199
- ```
212
+ ----
200
213
 
201
214
  Destroy results that do not have a component
202
215
 
203
- ```ruby
216
+ [source,ruby]
217
+ ----
204
218
  Decidim::Budgets::Project.find_each(batch_size: 100) { |project|
205
219
  project.delete if project.component.blank?
206
220
  }
207
- ```
221
+ ----
208
222
 
209
- #### Surveys
223
+ ==== Surveys
210
224
 
211
- ```ruby
225
+ [source,ruby]
226
+ ----
212
227
  Decidim::Component.where(manifest_name: "surveys").find_each(batch_size: 100) { |c|
213
228
  if c.participatory_space.blank?
214
229
  Decidim::Surveys::Survey.where(component: c).destroy_all
215
230
  c.destroy
216
231
  end
217
232
  }
218
- ```
233
+ ----
219
234
 
220
235
  Destroy surveys that do not have a component
221
236
 
222
- ```ruby
237
+ [source,ruby]
238
+ ----
223
239
  Decidim::Surveys::Survey.find_each(batch_size: 100) { |survey|
224
240
  survey.delete if survey.component.blank?
225
241
  }
226
- ```
242
+ ----
227
243
 
228
- #### Comments
244
+ ==== Comments
229
245
 
230
246
  Destroy comments whose commentable root is a proposal that does not have a participatory space.
231
247
 
232
- ```ruby
248
+ [source,ruby]
249
+ ----
233
250
  proposal_ids = Decidim::Comments::Comment.where(decidim_root_commentable_type: "Decidim::Proposals::Proposal").pluck(:decidim_root_commentable_id)
234
251
 
235
252
  proposal_ids_without_space = Decidim::Proposals::Proposal.where(id: proposal_ids).find_all{|p| p.participatory_space.blank? }.pluck(:id)
236
253
 
237
254
  Decidim::Comments::Comment.where(decidim_root_commentable_type: "Decidim::Proposals::Proposal", decidim_root_commentable_id: proposal_ids_without_space).destroy_all
238
- ```
255
+ ----