jekyll-algolia 1.0.1 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2de0f9bc4ac08d49c4f78093c469d778b1d69001
-  data.tar.gz: dcce0aafc47b6661063e645f03bed64b3cc849d9
+  metadata.gz: 18ee9d2d725bb531703ddc05d9f31250959d81ff
+  data.tar.gz: 93115f75f4d41d301898b77809732453d5c195f2
 SHA512:
-  metadata.gz: 8c898016647a648a10d3aeaa270aeca5257399a47a124e55902581ae935cedab196295204f6e4177d58b4a56b544fd54f9825b8dc7198995fa2dfa367de1c875
-  data.tar.gz: 486d9709cd1de0ed485d921e2c22640776c36fc7d45edabceff1f45e9f50f65879b7e0971daa9d038ffbe5d8e5143bb256cd00429203f0590c17a243eba25bca
+  metadata.gz: fed961094843fdf503620f86fcb83313c96b68061ce0d170b1a66f018d60cf1edf0a68a9dd5df7f17dee5c2a6e6e4fa05e677ae5a743383c859c42f3443e6bba
+  data.tar.gz: d66ea9ca93b8cafc3749711f17d27f03753ddfc6f4de70d2cc94296e5209b9247f2e3bbbefbf924414e5fe3c380f13c13fac87f80b8eed9322ccb0338d2dad2b

data/CONTRIBUTING.md

@@ -1,10 +1,9 @@
 Hi collaborator!
 
-If you have a fix or a new feature, please start by checking in the
-[issues](https://github.com/algolia/jekyll-algolia/issues) if it is
-already referenced. If not, feel free to open one.
+If you have a fix or a new feature, please start by checking in the [issues][1]
+if it is already referenced. If not, feel free to open one.
 
-We use [pull requests](https://github.com/algolia/jekyll-algolia/pulls)
+We use [pull requests][2]
 for collaboration. The workflow is as follow:
 
 - Create a local branch, starting from `develop`
@@ -20,23 +19,18 @@ Start by running `bundle install` to get all the dependencies up to date.
 
 ## Testing
 
-Run `rake test` to launch all tests. You can run `rake test_details` to get an
-output with more details about the tests.
+Run `rake test` to launch the test suite. Run `./scripts/test_all_ruby_versions`
+to run the test on all the supported ruby versions (requires `rvm`).
 
 ## TDD
 
-// TODO
-
-## Testing different ruby versions
-
-You can test the gem across all the supported Ruby versions by running
-`./scripts/test_all_ruby_versions`. Note that you will need to have RVM
-installed for this to work.
+Run `rake watch` to start a watcher on the code and test files. Whenever you
+update the code, the relevant tests will be run. Incredibly useful for TDD.
 
 ## Testing local changes on an existing Jekyll website
 
 If you want to test the plugin on an existing Jekyll website while developping,
-I suggest updating the website `Gemfile` to point to the correct local directory
+we suggest updating the website `Gemfile` to point to the correct local directory
 
 ```ruby
 group :jekyll_plugins do
@@ -44,10 +38,19 @@ group :jekyll_plugins do
 end
 ```
 
+## Linting
+
+Run `rake lint` to check the style of all ruby files. Run `rake
+lint:auto_correct` to try to automatically correct the potential violations.
+It's always a good practice to double check the modification after an
+auto-correct.
+
 # Git Hooks
 
-If you plan on submitting a PR, I suggest you install the git hooks located in
-`./scripts/git_hook`.
+If you plan on submitting a PR, we suggest you install the git hooks located in
+`./scripts/git_hook`. Those hooks will run the linter on each commit, and the
+tests before each push. This greatly help reduce the chances of breaking the
+build on Travis.
 
 The easiest way is to create a symlink from your `.git/hooks` folder:
 
@@ -67,28 +70,36 @@ the git tags, create the gem and push it to Rubygems.
 
 ## Requirements
 
-To run this project, you will need:
+The documentation website uses Metalsmith (and not Jekyll), so you'll need:
 
-- Node.js >= v9.2.0, use nvm - [install instructions](https://github.com/creationix/nvm#install-script)
-- Yarn >= v1.3.2 - [install instructions](https://yarnpkg.com/en/docs/install#alternatives-tab)
+- Node.js >= v9.2.0, use nvm - [install instructions][3]
+- Yarn >= v1.3.2 - [install instructions][4]
 
 ## Development
 
-```sh
-yarn
-yarn start
-```
+All the documentation source files live in the `./docs-src` folder.
 
-Go to <http://localhost:3000>.
+To serve a local version of the documentation (including livereload), run `rake
+docs:serve`. The documentation will be available on
+[localhost:3000](http://localhost:3000/).
 
-## Update docs
+This will create a `./docs-dev` folder and serve files from there. This folder
+is ignored by git.
 
-```sh
-yarn docs:update
-git push
-```
+## Deploying docs
 
+To update the documentation website, you should run `rake docs:deploy` from the
+`develop` branch. This will merge `develop` into master, build the documentation
+into `docs` and push it. The content of the `./docs` folder will then be server
+by GitHub pages.
 
 # Project owner
 
-[@pixelastic](https://github.com/pixelastic)
+[@pixelastic][5]
+
+
+[1]: https://github.com/algolia/jekyll-algolia/issues
+[2]: https://github.com/algolia/jekyll-algolia/pulls
+[3]: https://github.com/creationix/nvm#install-script
+[4]: https://yarnpkg.com/en/docs/install#alternatives-tab
+[5]: https://github.com/pixelastic

data/README.md

@@ -6,7 +6,7 @@ Status][3]](https://coveralls.io/github/algolia/jekyll-algolia?branch=master)
 [![Code Climate][4]](https://codeclimate.com/github/algolia/jekyll-algolia)
 ![Jekyll >= 3.6.0][5] ![Ruby >= 2.3.0][6]
 
-Jekyll plugin to automatically index your content on Algolia.
+Add fast and relevant search to your Jekyll site.
 
 ## Usage
 

data/lib/jekyll-algolia.rb

@@ -33,6 +33,8 @@ module Jekyll
 
       exit 1 unless Configurator.assert_valid_credentials
 
+      Configurator.warn_of_deprecated_options
+
       if Configurator.dry_run?
         Logger.log('W:==== THIS IS A DRY RUN ====')
         Logger.log('W:  - No records will be pushed to your index')
@@ -72,9 +74,10 @@ module Jekyll
 
     # Public: Get access to the time at which the command was run
     #
-    # Jekyll will override some date with the current time, and we'll need to
-    # keep them as nil, so we have to compare to this date to assume it has been
-    # overwritten
+    # Jekyll will always set the updated time of pages to the time of the build
+    # run. The plugin needs those values to stay at nil if they did not change,
+    # so we'll keep track of the time at build time and revert any page build at
+    # that time to nil.
     def self.start_time
       @start_time
     end

data/lib/jekyll/algolia/configurator.rb

@@ -12,7 +12,6 @@ module Jekyll
         'files_to_exclude' => nil,
         'nodes_to_index' => 'p',
         'indexing_batch_size' => 1000,
-        'indexing_mode' => 'diff',
         'settings' => {
           'distinct' => true,
           'attributeForDistinct' => 'url',
@@ -129,18 +128,6 @@ module Jekyll
         ALGOLIA_DEFAULTS['settings'].merge(user_settings)
       end
 
-      # Public: Return the current indexing mode
-      #
-      # Default mode is `diff`, but users can configure their own by updating
-      # the `indexing_mode` config in _config.yml. The only other authorized
-      # value is `atomic`. If an unrecognized mode is defined, it defaults to
-      # `diff`.
-      def self.indexing_mode
-        mode = algolia('indexing_mode') || ALGOLIA_DEFAULTS['indexing_mode']
-        return 'diff' unless %w[diff atomic].include?(mode)
-        mode
-      end
-
       # Public: Check that all credentials are set
       #
       # Returns true if everything is ok, false otherwise. Will display helpful
@@ -197,6 +184,20 @@ module Jekyll
         return true if value == true
         false
       end
+
+      # Public: Check for any deprecated config option and warn the user
+      def self.warn_of_deprecated_options
+        # indexing_mode is no longer used
+        return if algolia('indexing_mode').nil?
+
+        # rubocop:disable Metrics/LineLength
+        Logger.log('I:')
+        Logger.log('W:[jekyll-algolia] You are using the algolia.indexing_mode option which has been deprecated in v1.1')
+        Logger.log('I:    Indexing is now always using an atomic diff algorithm.')
+        Logger.log('I:    This option is no longer necessary, you can remove it from your _config.yml')
+        Logger.log('I:')
+        # rubocop:enable Metrics/LineLength
+      end
     end
   end
 end

data/lib/jekyll/algolia/error_handler.rb

@@ -47,7 +47,6 @@ module Jekyll
       def self.identify(error, context = {})
         known_errors = %w[
           unknown_application_id
-          invalid_credentials_for_tmp_index
           invalid_credentials
           record_too_big
           unknown_settings
@@ -156,29 +155,6 @@ module Jekyll
         { 'application_id' => app_id }
       end
 
-      # Public: Check if credentials specifically can't access the _tmp index
-      #
-      # _context - Not used
-      #
-      # If the error happens on a _tmp folder, it might mean that the key does
-      # not have access to the _tmp indices and the error message will reflect
-      # that.
-      def self.invalid_credentials_for_tmp_index?(error, _context = {})
-        details = error_hash(error.message)
-
-        index_name_tmp = details['index_name']
-        if details['message'] != 'Index not allowed with this API key' ||
-           index_name_tmp !~ /_tmp$/
-          return false
-        end
-
-        {
-          'application_id' => Configurator.application_id,
-          'index_name' => Configurator.index_name,
-          'index_name_tmp' => index_name_tmp
-        }
-      end
-
       # Public: Check if the credentials are working
       #
       # _context - Not used

data/lib/jekyll/algolia/indexer.rb

@@ -45,45 +45,17 @@ module Jekyll
         ::Algolia::Index.new(index_name)
       end
 
-      # Public: Update records of the specified index
+      # Public: Check if an index exists
       #
-      # index - Algolia Index to update
-      # records - Array of records to update
+      # index_name - Name of the index
       #
-      # New records will be automatically added. Technically existing records
-      # should be updated but this case should never happen as changing a record
-      # content will change its objectID as well.
-      #
-      # Does nothing in dry run mode
-      def self.update_records(index, records)
-        batch_size = Configurator.algolia('indexing_batch_size')
-        records.each_slice(batch_size) do |batch|
-          Logger.log("I:Pushing #{batch.size} records")
-          next if Configurator.dry_run?
-          begin
-            index.add_objects!(batch)
-          rescue StandardError => error
-            ErrorHandler.stop(error, records: records)
-          end
-        end
-      end
-
-      # Public: Delete records whose objectIDs are passed
-      #
-      # index - Algolia Index to target
-      # ids - Array of objectIDs to delete
-      #
-      # Does nothing in dry run mode
-      def self.delete_records_by_id(index, ids)
-        return if ids.empty?
-        Logger.log("I:Deleting #{ids.length} records")
-        return if Configurator.dry_run?
-
-        begin
-          index.delete_objects!(ids)
-        rescue StandardError => error
-          ErrorHandler.stop(error)
-        end
+      # Note: there is no API endpoint to do that, so we try to get the settings
+      # instead, which will fail if the index does not exist
+      def self.index?(index_name)
+        index(index_name).get_settings
+        return true
+      rescue StandardError
+        return false
       end
 
       # Public: Returns an array of all the objectIDs in the index
@@ -119,113 +91,69 @@ module Jekyll
       # Public: Update settings of the index
       #
       # index - The Algolia Index
-      # settings - The hash of settings to pass to the index
       #
       # Does nothing in dry run mode
-      def self.update_settings(index, settings)
+      # Settings will only be updated in the first push, and if custom settings
+      # are defined in _config.yml. Otherwise, they are left untouched, allowing
+      # users to configure them through their dashboard.
+      def self.update_settings(index)
+        has_custom_settings = !Configurator.algolia('settings').nil?
+        index_exists = index?(index.name)
+
+        # No need to update the settings if the index is already configured and
+        # the user did not specify custom settings
+        return if index_exists && !has_custom_settings
+
         Logger.verbose('I:Updating settings')
         return if Configurator.dry_run?
+        settings = Configurator.settings
         begin
-          index.set_settings(settings)
+          index.set_settings!(settings)
         rescue StandardError => error
           ErrorHandler.stop(error, settings: settings)
         end
       end
 
-      # Public: Index content following the `diff` indexing mode
+      # Public: Update records of the index
       #
-      # records - Array of local records
+      # index_name - The Algolia index
+      # old_records_ids - Ids of records to delete from the index
+      # new_records - Records to add to the index
       #
-      # The `diff` indexing mode will only push new content to the index and
-      # remove old content from it. It won't touch records that haven't been
-      # updated. It will be a bit slower as it will first need to get the list
-      # of all records in the index, but it will consume less operations.
-      def self.run_diff_mode(records)
-        index = index(Configurator.index_name)
-
-        # Update settings
-        update_settings(index, Configurator.settings)
-
-        # Getting list of objectID in remote and locally
-        remote_ids = remote_object_ids(index)
-        local_ids = local_object_ids(records)
-
-        old_records_ids = remote_ids - local_ids
-        new_records_ids = local_ids - remote_ids
-        if old_records_ids.empty? && new_records_ids.empty?
+      # Note: All operations will be done in one batch, assuring an atomic
+      # update
+      # Does nothing in dry run mode
+      def self.update_records(index_name, old_records_ids, new_records)
+        # Stop if nothing to change
+        if old_records_ids.empty? && new_records.empty?
           Logger.log('I:Nothing to index. Your content is already up to date.')
           return
         end
 
-        Logger.log("I:Updating records in index #{index.name}...")
-
-        # Delete remote records that are no longer available locally
-        delete_records_by_id(index, old_records_ids)
+        Logger.log("I:Updating records in index #{index_name}...")
+        Logger.log("I:Records to delete: #{old_records_ids.length}")
+        Logger.log("I:Records to add:    #{new_records.length}")
+        return if Configurator.dry_run?
 
-        # Add only records that are not yet already in the remote
-        new_records = records.select do |record|
-          new_records_ids.include?(record[:objectID])
+        operations = new_records.map do |new_record|
+          { action: 'addObject', indexName: index_name, body: new_record }
         end
-        update_records(index, new_records)
-
-        Logger.log('I:✔ Indexing complete')
-      end
-
-      # Public: Get the settings of the remote index
-      #
-      # index - The Algolia Index
-      def self.remote_settings(index)
-        index.get_settings
-      rescue StandardError => error
-        ErrorHandler.stop(error)
-      end
-
-      # Public: Rename an index
-      #
-      # old_name - Current name of the index
-      # new_name - New name of the index
-      #
-      # Does nothing in dry run mode
-      def self.rename_index(old_name, new_name)
-        Logger.verbose("I:Renaming `#{old_name}` to `#{new_name}`")
-        return if Configurator.dry_run?
-        begin
-          ::Algolia.move_index(old_name, new_name)
-        rescue StandardError => error
-          ErrorHandler.stop(error, new_name: new_name)
+        old_records_ids.each do |object_id|
+          operations << {
+            action: 'deleteObject', indexName: index_name,
+            body: { objectID: object_id }
+          }
         end
-      end
-
-      # Public: Index content following the `atomic` indexing mode
-      #
-      # records - Array of records to push
-      #
-      # The `atomic` indexing mode will push all records to a brand new index,
-      # configure it, and then overwrite the previous index with this new one.
-      # For the end-user, it will make all the changes in one go, making sure
-      # people are always searching into a fully configured index. It will
-      # consume more operations, but will never leave the index in a transient
-      # state.
-      def self.run_atomic_mode(records)
-        index_name = Configurator.index_name
-        index = index(index_name)
-        index_tmp_name = "#{Configurator.index_name}_tmp"
-        index_tmp = index(index_tmp_name)
-
-        Logger.verbose("I:Using `#{index_tmp_name}` as temporary index")
 
-        # Copying original settings to the new index
-        remote_settings = remote_settings(index)
-        new_settings = remote_settings.merge(Configurator.settings)
-        update_settings(index_tmp, new_settings)
-
-        # Pushing everthing to a brand new index
-        update_records(index_tmp, records)
-
-        # Renaming the new index in place of the old
-        rename_index(index_tmp_name, index_name)
-
-        Logger.log('I:✔ Indexing complete')
+        # Run the batches in slices if they are too large
+        batch_size = Configurator.algolia('indexing_batch_size')
+        operations.each_slice(batch_size) do |slice|
+          begin
+            ::Algolia.batch!(slice)
+          rescue StandardError => error
+            ErrorHandler.stop(error)
+          end
+        end
       end
 
       # Public: Push all records to Algolia and configure the index
@@ -234,10 +162,8 @@ module Jekyll
       def self.run(records)
         init
 
-        record_count = records.length
-
         # Indexing zero record is surely a misconfiguration
-        if record_count.zero?
+        if records.length.zero?
           files_to_exclude = Configurator.algolia('files_to_exclude').join(', ')
           Logger.known_message(
             'no_records_found',
@@ -247,14 +173,25 @@ module Jekyll
           exit 1
         end
 
-        indexing_mode = Configurator.indexing_mode
-        Logger.verbose("I:Indexing mode: #{indexing_mode}")
-        case indexing_mode
-        when 'diff'
-          run_diff_mode(records)
-        when 'atomic'
-          run_atomic_mode(records)
+        index_name = Configurator.index_name
+        index = index(index_name)
+
+        # Update settings
+        update_settings(index)
+
+        # Getting list of objectID in remote and locally
+        remote_ids = remote_object_ids(index)
+        local_ids = local_object_ids(records)
+
+        # Getting list of what to add and what to delete
+        old_records_ids = remote_ids - local_ids
+        new_records_ids = local_ids - remote_ids
+        new_records = records.select do |record|
+          new_records_ids.include?(record[:objectID])
         end
+        update_records(index_name, old_records_ids, new_records)
+
+        Logger.log('I:✔ Indexing complete')
       end
     end
   end

data/lib/jekyll/algolia/version.rb

@@ -2,6 +2,6 @@
 
 module Jekyll
   module Algolia
-    VERSION = '1.0.1'
+    VERSION = '1.1.0'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jekyll-algolia
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.1.0
 platform: ruby
 authors:
 - Tim Carry
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-12-22 00:00:00.000000000 Z
+date: 2018-01-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: algolia_html_extractor
@@ -257,7 +257,6 @@ files:
 - CONTRIBUTING.md
 - README.md
 - lib/errors/invalid_credentials.txt
-- lib/errors/invalid_credentials_for_tmp_index.txt
 - lib/errors/invalid_index_name.txt
 - lib/errors/missing_api_key.txt
 - lib/errors/missing_application_id.txt

data/lib/errors/invalid_credentials_for_tmp_index.txt

@@ -1,17 +0,0 @@
-E: [✗ Error] Invalid credentials for temporary index
-E:
-E: The jekyll-algolia plugin could not access your index with the API key you
-E: provided.
-W:
-W: When using the `atomic` indexing mode, we will push all your content to a 
-W: temporary index, then overwrite the actual index with it. The API key you
-W: defined should have access rights on both.
-I: 
-I: Make sure your API key has access:
-I: - {index_name}
-I: - {index_name_tmp}
-I:
-I: You can configure API keys from your dashboard:
-I:    https://www.algolia.com/apps/{application_id}/api-keys/restricted
-I:
-I: