lcms-engine 0.1.2 → 0.3.1

data/docs/google-cloud-platform-setup.md

@@ -0,0 +1,106 @@
+An application calls the Google Application Script in order to post-process generated lesson materials. Instructions below contains all the necessary steps need to set up the integration.
+
+All environment variables and user names provided here have been set up to automate deployment on [Cloud66](https://www.cloud66.com) service stacks. If you plan to use another service provider please change all the variables accordingly.
+
+1. [Set up](#set-up)
+2. [How to update Google Application Script](#how-to-update-google-application-script)
+
+## Set up
+
+1. Login to google (all google docs will be saved under this account)
+
+2. Create project (i.e. LCMS-dev) at [google cloud](https://console.cloud.google.com)
+
+3. Enable Google Drive API, Google Apps Script Execution API for that [project](https://console.cloud.google.com/apis/library)
+
+4. Set up google auth
+
+#### Credentials
+
+Create credentials:
+  - type oAuth Client ID
+  - Application type: Other,
+  - Aplication Name: up to you, i.e. lcms-cli-dev
+
+Download client secret JSON file.
+
+###### Local development
+
+Copy downloaded JSON file to `config/google/client_secret.json`
+
+###### Server
+
+Make sure you're executing commands under _cloud66-user_
+```bash
+$ sudo -i -u cloud66-user
+```
+
+Make sure that the following directory exists on a server and is symlinked correctly
+```bash
+$ mkdir -p "$STACK_BASE/shared/google"
+$ chown cloud66-user:cloud66-user "$STACK_BASE/shared/google"
+$ ln -nsf "$STACK_BASE/shared/google" "$STACK_PATH/config/google"
+```
+
+Copy downloaded JSON file and set appropriate ownership and access rights for it (where `cloud66-user` is a system username under which web application is running)
+
+```bash
+$ cp client_secret.json $STACK_BASE/shared/google/
+$ cd $STACK_BASE/shared/google
+$ sudo chown cloud66-user:cloud66-user client_secret.json
+$ sudo chmod 755 client_secret.json
+```
+
+#### Application token
+
+If you generate token on a remote server, then make sure you're executing commands under _cloud66-user_
+
+```bash
+$ sudo -i -u cloud66-user
+```
+
+Run rake task. You will be asked to go by link, give app permissions and paste code into the terminal
+
+```bash
+$ bundle exec rake google:setup_auth
+```
+
+This will create `config/google/app_token.yaml`. Change the ownership of the `app_token.yaml` otherwise it will not be accessible for the application:
+
+```bash
+$ cd $STACK_BASE/shared/google
+$ chown cloud66-user:app_writers app_token.yaml
+$ sudo chmod g+w app_token.yaml
+```
+
+5. Add script
+- go to https://www.google.com/script/start/
+- copy-paste content of `config/scripts/Code.gs` there and save
+- at top menu Resources->Cloud Platform Project set project from step 2 (you need to paste *project number*)
+- at top menu Publish->Deploy As API Executable set version v1, access Only myself
+- save Current API ID somewhere, click Close on that annoying window (update will not close it)
+- choose any function and run it, it'll request permissions - grant them (there will be security warnings, just ignore them)
+
+6. Create folder (all materials will be saved there) on Google Drive, give view only to all by link, keep folder id (last part of url), copy to the root [LANDSCAPE](https://docs.google.com/document/d/1pXQDNKYOJYT6OTPnp8gsTWAydg5B9GTRibaWspmX4oE), [PORTRAIT](https://docs.google.com/document/d/1ijuZhGQXkPBxcZT4DRyNVY-qmI0xyVvSzVFqckOpsCc) templates and keep their IDs.
+
+7. Update corresponding env-file in the project
+
+```
+GOOGLE_APPLICATION_FOLDER_ID=saved from step 6
+GOOGLE_APPLICATION_SCRIPT_ID=saved id from step 5
+GOOGLE_APPLICATION_TEMPLATE_PORTRAIT=saved from step 6
+GOOGLE_APPLICATION_TEMPLATE_LANSCAPE=saved from step 6
+```
+
+## How to update Google Application Script
+
+1. Proceed to Google Drive for an account which has been used to create the Google Application Script (see _Add script_ section in [Set up](#set-up))
+
+2. Update script content
+
+3. Re-publish the script:
+
+- `Publish->Deploy As API Executable`
+- Set new version
+- `Update` & `Close`
+- Choose any function and run it, it'll request permissions - grant them (there will be security warnings, just ignore them)

data/lcms-engine.gemspec

@@ -19,112 +19,92 @@ Gem::Specification.new do |s|
   s.files = `git ls-files`.split("\n")
   s.test_files = `git ls-files -- spec/*`.split("\n")
 
-  s.required_ruby_version = '>= 2.5'
+  s.required_ruby_version = '>= 2.6'
 
-  s.add_dependency 'active_model_serializers', '~> 0.9.3'
   s.add_dependency 'activejob-retry', '~> 0.6.3'
-  s.add_dependency 'acts-as-taggable-on', '~> 5.0'
-  s.add_dependency 'acts_as_list', '~> 0.9.10'
-  s.add_dependency 'addressable', '~> 2.5.1'
-  s.add_dependency 'airbrake', ' ~> 10'
-  s.add_dependency 'autoprefixer-rails', '~> 6.4.0'
-  s.add_dependency 'aws-sdk-rails', '~> 1.0'
-  s.add_dependency 'backport_new_renderer', '~> 1.0.0'
-  s.add_dependency 'bitly', '~> 1.0'
-  s.add_dependency 'bootstrap-sass', '~> 3.4.1'
-  s.add_dependency 'bullet', '~> 5.7.3'
-  s.add_dependency 'carrierwave', '~> 1.0'
-  s.add_dependency 'ckeditor', '~> 4'
-  s.add_dependency 'closure_tree', '~> 7'
-  s.add_dependency 'coffee-rails', '~> 4.1.1'
-  s.add_dependency 'combine_pdf', '~> 1.0.15'
-  s.add_dependency 'daemons', '~> 1.2', '>= 1.2.3'
-  s.add_dependency 'devise', '~> 4'
-  s.add_dependency 'elasticsearch-dsl', '~> 0.1.6'
-  s.add_dependency 'elasticsearch-model', '~> 6.0'
-  s.add_dependency 'elasticsearch-persistence', '~> 6.0'
-  s.add_dependency 'elasticsearch-rails', '~> 6.0'
-  s.add_dependency 'fog-aws', '~> 2.0'
-  s.add_dependency 'font-awesome-sass', '~> 4.7.0'
-  s.add_dependency 'foreman', '~> 0.78.0'
-  s.add_dependency 'foundation-rails', '~> 6.2.1'
-  s.add_dependency 'google-api-client', '~> 0.30'
-  s.add_dependency 'heap', '~> 1.0'
-  s.add_dependency 'hiredis', '~> 0.6'
-  s.add_dependency 'httparty', '~> 0.16'
-  s.add_dependency 'jbuilder', '~> 2.0'
-  s.add_dependency 'jquery-rails', '~> 4.3'
-  s.add_dependency 'js-routes', '~> 1.1.2'
-  s.add_dependency 'lt-google-api', '~> 0.1'
-  s.add_dependency 'lt-lcms', '~> 0.2'
-  s.add_dependency 'migration_data', '~> 0.2.1'
-  s.add_dependency 'mini_magick', '~> 4.9', '>= 4.9.4'
-  s.add_dependency 'nested_form', '~> 0.3'
-  s.add_dependency 'newrelic_rpm', '~> 6.9'
+  s.add_dependency 'active_model_serializers', '~> 0.10.10'
+  s.add_dependency 'acts_as_list', '~> 1.0'
+  s.add_dependency 'acts-as-taggable-on', '~> 6.5'
+  s.add_dependency 'addressable', '~> 2.7'
+  s.add_dependency 'airbrake', '~> 10.0'
+  s.add_dependency 'autoprefixer-rails', '~> 9.7'
+  s.add_dependency 'aws-sdk-rails', '~> 3.1'
+  s.add_dependency 'aws-sdk-s3', '~> 1'
+  s.add_dependency 'bitly', '~> 1.1', '>= 1.1.2'
+  s.add_dependency 'bootstrap-sass', '~> 3.4'
+  s.add_dependency 'bullet', '~> 6.1', '>= 6.1.0'
+  s.add_dependency 'carrierwave', '~> 2.1'
+  s.add_dependency 'ckeditor', '~> 4.3'
+  s.add_dependency 'closure_tree', '~> 7.1'
+  s.add_dependency 'combine_pdf', '~> 1.0'
+  s.add_dependency 'daemons', '~> 1.3', '>= 1.3.1'
+  s.add_dependency 'devise', '~> 4.7', '>= 4.7.1'
+  s.add_dependency 'elasticsearch-dsl', '~> 0.1.9'
+  s.add_dependency 'elasticsearch-model', '~> 7.0'
+  s.add_dependency 'elasticsearch-persistence', '~> 7.0'
+  s.add_dependency 'elasticsearch-rails', '~> 7.0'
+  s.add_dependency 'fog-aws', '~> 3.5', '>= 3.5.2'
+  s.add_dependency 'font-awesome-sass', '~> 5.12'
+  s.add_dependency 'foundation-rails', '~> 6.6', '>= 6.6.1'
+  s.add_dependency 'google-api-client', '< 1'
+  s.add_dependency 'hiredis', '~> 0.6.3'
+  s.add_dependency 'httparty', '~> 0.18'
+  s.add_dependency 'jbuilder', '~> 2.10'
+  s.add_dependency 'jquery-rails', '~> 4.3', '>= 4.3.5'
+  s.add_dependency 'js-routes', '~> 1.4', '>= 1.4.9'
+  s.add_dependency 'lt-google-api', '~> 0.2', '>= 0.2.2'
+  s.add_dependency 'lt-lcms', '~> 0.4', '>= 0.4.1'
+  s.add_dependency 'migration_data', '~> 0.6'
+  s.add_dependency 'mini_magick', '~> 4.10', '>= 4.10.1'
+  s.add_dependency 'nested_form', '~> 0.3.2'
   s.add_dependency 'nikkou', '~> 0.0.5'
-  s.add_dependency 'nokogiri', '~> 1.10', '>= 1.10.4'
-  s.add_dependency 'oj', '~> 3'
-  s.add_dependency 'oj_mimic_json', '~> 1.0.1'
-  s.add_dependency 'pandoc-ruby', '~> 2.0.1'
-  s.add_dependency 'pdfjs_viewer-rails', '~> 0.0.9'
-  s.add_dependency 'pg', '0.20'
-  s.add_dependency 'pg_search', '~> 2.1'
-  s.add_dependency 'postgres_ext', '~> 3.0'
-  s.add_dependency 'rack-mini-profiler', '~> 2'
-  s.add_dependency 'rails', '~> 4.2.11'
-  s.add_dependency 'rails-assets-classnames', '~> 2.2.3'
-  s.add_dependency 'rails-assets-es6-promise', '~> 3.1.2'
-  s.add_dependency 'rails-assets-eventemitter3', '~> 1.2.0'
-  s.add_dependency 'rails-assets-fetch', '~> 0.11.0'
-  s.add_dependency 'rails-assets-jstree', '~> 3.3.4'
-  s.add_dependency 'rails-assets-knockout', '~> 3.3.0'
-  s.add_dependency 'rails-assets-lodash', '~> 4.17.11'
-  s.add_dependency 'rails-assets-selectize', '~> 0.12.1'
-  s.add_dependency 'ransack', '~> 1.7.0'
-  s.add_dependency 'react-rails', '~> 2.6.0'
-  s.add_dependency 'readthis', '~> 2'
-  s.add_dependency 'redis', '< 4'
-  s.add_dependency 'resque', '~> 2'
-  s.add_dependency 'resque-scheduler', '~> 4.4'
-  s.add_dependency 'rest-client', '~> 1.8'
-  s.add_dependency 'ruby-progressbar', '~> 1.7', '>= 1.7.5'
+  s.add_dependency 'nokogiri', '~> 1.10', '>= 1.10.8'
+  s.add_dependency 'oj', '~> 3.10', '>= 3.10.2'
+  s.add_dependency 'oj_mimic_json', '~> 1.0', '>= 1.0.1'
+  s.add_dependency 'pandoc-ruby', '~> 2.0', '>= 2.0.2'
+  s.add_dependency 'pdfjs_viewer-rails', '~> 0.3.1'
+  s.add_dependency 'pg', '~> 1.2', '>= 1.2.2'
+  s.add_dependency 'pg_search', '~> 2.3', '>= 2.3.2'
+  s.add_dependency 'rack-mini-profiler', '~> 1.1', '>= 1.1.6'
+  s.add_dependency 'rails', '~> 5.2', '>= 5.2.4'
+  s.add_dependency 'ransack', '~> 2.3', '>= 2.3.2'
+  s.add_dependency 'react-rails', '~> 2.6', '>= 2.6.1'
+  s.add_dependency 'redis', '~> 4.1', '>= 4.1.3'
+  s.add_dependency 'resque', '~> 2.0', '>= 2.0.0'
+  s.add_dependency 'resque-scheduler', '~> 4.4', '>= 4.4.0'
+  s.add_dependency 'rest-client', '~> 2.1', '>= 2.1.0'
+  s.add_dependency 'ruby-progressbar', '~> 1.10', '>= 1.10.1'
   s.add_dependency 'rubyzip', '~> 2'
-  s.add_dependency 'sanitize', '~> 5.0'
-  s.add_dependency 'sass-rails', '~> 5.0'
-  s.add_dependency 'simple_form', '~> 3.5'
-  s.add_dependency 'staccato', '~> 0.4.7'
-  s.add_dependency 'thor', '0.19.1'
+  s.add_dependency 'sanitize', '>= 5.1', '< 5.3'
+  s.add_dependency 'sass-rails', '~> 5.1', '>= 5.1.0'
+  s.add_dependency 'simple_form', '~> 5.0', '>= 5.0.2'
+  s.add_dependency 'staccato', '~> 0.5.3'
   s.add_dependency 'truncate_html', '~> 0.9.3'
-  s.add_dependency 'turbolinks', '~> 5.1.0'
-  s.add_dependency 'uglifier', '~> 3.0', '>= 3.0.4'
-  s.add_dependency 'validate_url', '~> 1.0'
-  s.add_dependency 'virtus', '~> 1.0.5'
-  s.add_dependency 'will_paginate', '~> 3.0.7'
-  s.add_dependency 'will_paginate-bootstrap', '~> 1'
+  s.add_dependency 'turbolinks', '~> 5.2', '>= 5.2.1'
+  s.add_dependency 'validate_url', '~> 1.0', '>= 1.0.8'
+  s.add_dependency 'virtus', '~> 1.0', '>= 1.0.5'
+  s.add_dependency 'will_paginate', '~> 3.2', '>= 3.2.1'
+  s.add_dependency 'will_paginate-bootstrap', '~> 1.0', '>= 1.0.2'
 
-  s.add_development_dependency 'capybara-screenshot', '~> 1'
-  s.add_development_dependency 'database_cleaner', '~> 1'
+  s.add_development_dependency 'capybara', '~> 3.31'
+  s.add_development_dependency 'database_cleaner-active_record', '~> 1.8'
   s.add_development_dependency 'dotenv-rails', '~> 2.2'
   s.add_development_dependency 'email_spec', '~> 2.1'
   s.add_development_dependency 'factory_bot', '~> 5'
   s.add_development_dependency 'faker', '~> 2.1'
-  s.add_development_dependency 'guard-livereload', '~> 2.5', '>= 2.5.2'
-  s.add_development_dependency 'mailcatcher', '~> 0.7'
-  s.add_development_dependency 'overcommit', '~> 0.49.1'
+  s.add_development_dependency 'overcommit', '~> 0.57'
   s.add_development_dependency 'pry-byebug', '~> 3.7'
   s.add_development_dependency 'pry-rails', '~> 0.3.5'
-  s.add_development_dependency 'puma', '~> 3'
-  s.add_development_dependency 'quiet_assets', '~> 1.1'
-  s.add_development_dependency 'rack-livereload', '~> 0.3.16'
   s.add_development_dependency 'rspec-rails', '~> 3.8'
-  s.add_development_dependency 'rubocop', '~> 0.59.2'
+  s.add_development_dependency 'rubocop', '~> 0.93.1'
   s.add_development_dependency 'safe_attributes', '~> 1.0.10'
-  s.add_development_dependency 'sdoc', '~> 0.4.0'
+  s.add_development_dependency 'sdoc', '~> 2'
   s.add_development_dependency 'seedbank', '~> 0.3'
   s.add_development_dependency 'selenium-webdriver', '~> 3.142', '>= 3.142.7'
   s.add_development_dependency 'shoulda-matchers', '~> 4.1'
   s.add_development_dependency 'spring', '~> 2.1'
-  s.add_development_dependency 'spring-commands-rspec', '~> 1'
+  s.add_development_dependency 'spring-commands-rspec', '~> 1.0'
   s.add_development_dependency 'traceroute', '~> 0.8'
-  s.add_development_dependency 'webpacker', '~> 4.0.6'
+  s.add_development_dependency 'webdrivers', '~> 4.0'
+  s.add_development_dependency 'webpacker', '~> 4.0'
 end

data/lib/doc_template.rb

@@ -19,7 +19,7 @@ module DocTemplate
     sanitizer: 'Lcms::Engine::HtmlSanitizer'
   }.freeze
 
-  FULL_TAG ||= /\[([^\]:\s]*)?\s*:?\s*([^\]]*?)?\]/mo
+  FULL_TAG ||= /\[([^\]:\s]*)?\s*:?\s*([^\]]*?)?\]/mo.freeze
   START_TAG ||= '\[[^\]]*'
 
   STARTTAG_XPATH ||= 'span[contains(., "[")]'

data/lib/doc_template/document.rb

@@ -2,7 +2,7 @@
 
 module DocTemplate
   class Document
-    MAX_PARSE_ITERATIONS = 100
+    MAX_PARSE_ITERATIONS = 300
 
     # Contains the list of tags for which no parts should be created
     TAGS_WITHOUT_PARTS = [
@@ -178,7 +178,7 @@ module DocTemplate
 
       parts << {
         anchor: parsed_tag.anchor.to_s,
-        content: sanitized_content.squish,
+        content: parsed_tag.try(:without_squish?) ? sanitized_content : sanitized_content.squish,
         context_type: @opts[:context_type],
         data: parsed_tag.tag_data,
         materials: parsed_tag.materials,

data/lib/doc_template/objects/agenda_metadata.rb

@@ -62,7 +62,7 @@ module DocTemplate
 
       attribute :children, Array[Group]
 
-      def self.build_from(data)
+      def self.build_from(data) # rubocop:disable Metrics/AbcSize
         copy = Marshal.load Marshal.dump(data)
         agenda_data =
           copy.map do |d|

data/lib/doc_template/objects/metadata_helpers.rb

@@ -3,7 +3,7 @@
 module DocTemplate
   module Objects
     module MetadataHelpers
-      SEPARATOR = /\s*[,;]\s*/
+      SEPARATOR = /\s*[,;]\s*/.freeze
 
       def self.build_anchor_from(item)
         [

data/lib/doc_template/objects/toc_metadata.rb

@@ -18,7 +18,7 @@ module DocTemplate
         attribute :time, Integer, default: 0
         attribute :title, String
 
-        def excluded?(excludes, ela = false)
+        def excluded?(excludes, ela: false)
           # Do not exclude parent if all children are optional and deselected
           return false if ela && children.all?(&:optional)
           return excludes.exclude?(anchor) if optional
@@ -27,7 +27,7 @@ module DocTemplate
           children.any? && children.all? { |c| c.excluded?(excludes) }
         end
 
-        def time_with(excludes)
+        def time_with(excludes) # rubocop:disable Metrics/PerceivedComplexity
           # Optional and nothing to exclude explicitly
           return excludes.include?(anchor) ? time : 0 if optional
           # General and excluded explicitly

data/lib/doc_template/tables/activity.rb

@@ -10,10 +10,9 @@ module DocTemplate
 
       def parse(fragment, *args)
         template_type = args.extract_options![:template_type].presence || 'core'
-        path = ".//table/*/tr[1]/td//*[case_insensitive_equals(text(),'#{HEADER_LABEL}')]"
         idx = 0
         [].tap do |result|
-          fragment.xpath(path, XpathFunctions.new).each do |el|
+          fragment.xpath(xpath_meta_headers, XpathFunctions.new).each do |el|
             table = el.ancestors('table').first
             data = fetch table
 

data/lib/doc_template/tables/base.rb

@@ -3,9 +3,9 @@
 module DocTemplate
   module Tables
     class Base
-      SPLIT_REGEX = /[,;\r\n]/
+      SPLIT_REGEX = /[,;\r\n]/.freeze
 
-      attr_reader :data
+      attr_reader :errors, :data
 
       def self.parse(fragment, *args)
         new.parse(fragment, *args)
@@ -13,6 +13,7 @@ module DocTemplate
 
       def initialize
         @data = {}
+        @errors = []
         @table_exists = false
       end
 
@@ -132,6 +133,10 @@ module DocTemplate
           end
         end
       end
+
+      def xpath_meta_headers
+        ".//table/*/tr[1]/td[1][case_insensitive_equals(normalize-space(),'#{self.class::HEADER_LABEL}')]"
+      end
     end
   end
 end

data/lib/doc_template/tables/section.rb

@@ -9,8 +9,7 @@ module DocTemplate
       MATERIALS_KEY = 'section-materials'
 
       def parse(fragment, *args)
-        path = ".//table/*/tr[1]/td//*[case_insensitive_equals(text(),'#{HEADER_LABEL}')]"
-        section_tables = fragment.xpath(path, XpathFunctions.new)
+        section_tables = fragment.xpath(xpath_meta_headers, XpathFunctions.new)
 
         # # Allows to handle ELA as Math:: inject fake section
         return fake_section(fragment) if section_tables.empty? && args.extract_options![:force_inject]

data/lib/doc_template/tags/activity_metadata_type_tag.rb

@@ -7,7 +7,7 @@ module DocTemplate
       include ERB::Util
 
       TAG_NAME = 'activity-metadata-type'
-      TASK_RE = /(\[task:\s(#)\])/i
+      TASK_RE = /(\[task:\s(#)\])/i.freeze
       TEMPLATES = {
         default: 'activity.html.erb',
         gdoc: 'gdoc/activity.html.erb'

data/lib/doc_template/tags/answer_space_tag.rb

@@ -6,7 +6,7 @@ module DocTemplate
       SPACE_SIZE = {
         s: 5, # small = 5 lines
         m: 10, # medium = 10 lines
-        l: 20, # large = 20 lines
+        l: 20 # large = 20 lines
       }.freeze
       TAG_NAME = 'answer-space'
 

data/lib/doc_template/tags/base_tag.rb

@@ -3,8 +3,8 @@
 module DocTemplate
   module Tags
     class BaseTag
-      SOFT_RETURN_RE = /([[:graph:]]+\[|\][[:graph:]]+)/
-      UNICODE_SPACES_RE = /(\u0020|\u00A0|\u1680|\u180E|[\u2000-\u200B]|\u202F|\u205F|\u3000|\uFEFF)/
+      SOFT_RETURN_RE = /([[:graph:]]+\[|\][[:graph:]]+)/.freeze
+      UNICODE_SPACES_RE = /(\u0020|\u00A0|\u1680|\u180E|[\u2000-\u200B]|\u202F|\u205F|\u3000|\uFEFF)/.freeze
 
       attr_reader :anchor, :content
 
@@ -16,7 +16,7 @@ module DocTemplate
         def tag_with_html_regexp
           raise NotImplementedError unless const_defined?(:TAG_NAME)
 
-          @tag_with_html_regexp ||= /\[[^\]]*#{self::TAG_NAME}[[^\:,;.]]*:?\s?[^\]]*\]/i
+          @tag_with_html_regexp ||= /\[[^\]]*#{self::TAG_NAME}[[^:,;.]]*:?\s?[^\]]*\]/i
         end
 
         def template_path_for(name)