ultrasphinx 1.9 → 1.11

data/CHANGELOG

@@ -1,4 +1,10 @@
 
+v1.11. Update to Sphinx 0.9.8-rc2.
+
+v1.10. Add geodistance support (Jeremy Seitz, Mark Lane). Move Postgres stored procedures into a migration generator (Lang Riley). Duplicate text :sortable columns so that they are also searchable.
+
+v1.9.1. Add ultrasphinx:index:merge task for index merging, and a note in DEPLOYMENT_NOTES about how to use it.
+
 v1.9. Delta indexing. ERb now supported in .base files. Allow setting the searched indexes at runtime.
 
 v1.8.1. Use multifind/multiget for record loading; avoid using HashWithIndifferentAccess internally for speed; other minor performance improvements.

data/DEPLOYMENT_NOTES

@@ -17,13 +17,15 @@ It's easy to keep the search daemon and the indexer running in a production envi
 
   */6 * * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \
     rake ultrasphinx:index:delta >> /log/ultrasphinx-index.log 2>&1'
-  1 4 * * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \
+  1 4 * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \
     rake ultrasphinx:index:main >> /log/ultrasphinx-index.log 2>&1'
   */3 * * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \
     rake ultrasphinx:daemon:start >> /log/ultrasphinx-daemon.log 2>&1'
 
 The first line reindexes the delta index every 10 minutes. The second line reindexes the main index once a day at 4am. The third line will try to restart the search daemon every three minutes. If it's already running, nothing happens. 
 
+Note that you can use <tt>ultrasphinx:index:merge</tt> in place of <tt>ultrasphinx:index:main</tt>. It will be faster, but will not remove deleted records from the index.
+
 Of course if you don't have any models with deltas, don't include the <tt>ultrasphinx:index:delta</tt> task.
 
 If you are under severe memory limitations you might want to manage the daemon with Monit instead, so you can keep a closer eye on it. The search daemon is extremely reliable, so don't bother with fancy monitoring infrastructure unless you're sure you need it.

data/Manifest

@@ -37,6 +37,7 @@ test/integration/app/app/helpers/application_helper.rb
 test/integration/app/app/helpers/sellers_helper.rb
 test/integration/app/app/helpers/states_helper.rb
 test/integration/app/app/helpers/users_helper.rb
+test/integration/app/app/models/category.rb
 test/integration/app/app/models/geo/address.rb
 test/integration/app/app/models/geo/country.rb
 test/integration/app/app/models/geo/state.rb
@@ -71,7 +72,6 @@ test/integration/app/config/environments/test.rb
 test/integration/app/config/locomotive.yml
 test/integration/app/config/routes.rb
 test/integration/app/config/ultrasphinx/default.base
-test/integration/app/config/ultrasphinx/development.conf
 test/integration/app/config/ultrasphinx/development.conf.canonical
 test/integration/app/db/migrate/001_create_users.rb
 test/integration/app/db/migrate/002_create_sellers.rb
@@ -82,7 +82,8 @@ test/integration/app/db/migrate/006_add_deleted_to_user.rb
 test/integration/app/db/migrate/007_add_lat_and_long_to_address.rb
 test/integration/app/db/migrate/008_add_mission_statement_to_seller.rb
 test/integration/app/db/migrate/009_create_countries.rb
-test/integration/app/db/schema.rb
+test/integration/app/db/migrate/010_create_categories.rb
+test/integration/app/db/migrate/011_categories_sellers.rb
 test/integration/app/doc/README_FOR_APP
 test/integration/app/public/404.html
 test/integration/app/public/500.html
@@ -115,6 +116,8 @@ test/integration/app/script/process/spawner
 test/integration/app/script/runner
 test/integration/app/script/server
 test/integration/app/test/fixtures/addresses.yml
+test/integration/app/test/fixtures/categories.yml
+test/integration/app/test/fixtures/categories_sellers.yml
 test/integration/app/test/fixtures/countries.yml
 test/integration/app/test/fixtures/sellers.yml
 test/integration/app/test/fixtures/states.yml
@@ -125,6 +128,7 @@ test/integration/app/test/functional/states_controller_test.rb
 test/integration/app/test/functional/users_controller_test.rb
 test/integration/app/test/test_helper.rb
 test/integration/app/test/unit/address_test.rb
+test/integration/app/test/unit/category_test.rb
 test/integration/app/test/unit/country_test.rb
 test/integration/app/test/unit/seller_test.rb
 test/integration/app/test/unit/state_test.rb
@@ -153,6 +157,7 @@ vendor/riddle/README
 vendor/riddle/spec/fixtures/data/anchor.bin
 vendor/riddle/spec/fixtures/data/any.bin
 vendor/riddle/spec/fixtures/data/boolean.bin
+vendor/riddle/spec/fixtures/data/comment.bin
 vendor/riddle/spec/fixtures/data/distinct.bin
 vendor/riddle/spec/fixtures/data/field_weights.bin
 vendor/riddle/spec/fixtures/data/filter.bin
@@ -166,6 +171,8 @@ vendor/riddle/spec/fixtures/data/filter_range_exclude.bin
 vendor/riddle/spec/fixtures/data/group.bin
 vendor/riddle/spec/fixtures/data/index.bin
 vendor/riddle/spec/fixtures/data/index_weights.bin
+vendor/riddle/spec/fixtures/data/keywords_with_hits.bin
+vendor/riddle/spec/fixtures/data/keywords_without_hits.bin
 vendor/riddle/spec/fixtures/data/phrase.bin
 vendor/riddle/spec/fixtures/data/rank_mode.bin
 vendor/riddle/spec/fixtures/data/simple.bin
@@ -179,6 +186,7 @@ vendor/riddle/spec/fixtures/sql/conf.example.yml
 vendor/riddle/spec/fixtures/sql/data.sql
 vendor/riddle/spec/fixtures/sql/structure.sql
 vendor/riddle/spec/functional/excerpt_spec.rb
+vendor/riddle/spec/functional/keywords_spec.rb
 vendor/riddle/spec/functional/search_spec.rb
 vendor/riddle/spec/functional/update_spec.rb
 vendor/riddle/spec/spec_helper.rb

data/RAKE_TASKS

@@ -7,6 +7,7 @@ These Rake tasks are made available to your Rails app:
 <tt>ultrasphinx:index</tt>:: Reindex and rotate all indexes.
 <tt>ultrasphinx:index:delta</tt>:: Reindex and rotate the delta index.
 <tt>ultrasphinx:index:main</tt>:: Reindex and rotate the main index.
+<tt>ultrasphinx:index:merge</tt>:: Merge the delta index into the main index.
 <tt>ultrasphinx:daemon:restart</tt>:: Restart the search daemon.
 <tt>ultrasphinx:daemon:start</tt>:: Start the search daemon.
 <tt>ultrasphinx:daemon:stop</tt>:: Stop the search daemon.
@@ -14,4 +15,4 @@ These Rake tasks are made available to your Rails app:
 <tt>ultrasphinx:spelling:build</tt>:: Rebuild the custom spelling dictionary. You may need to use <tt>sudo</tt> if your Aspell folder is not writable by the app user.
 <tt>ultrasphinx:bootstrap</tt>:: Bootstrap a full Sphinx environment by running configure, index, then daemon:start.
 
-All tasks have shortcuts. Use <tt>us:conf</tt>, <tt>us:index</tt>, <tt>us:main</tt>, <tt>us:delta</tt>, <tt>us:restart</tt>, <tt>us:start</tt>, <tt>us:stop</tt>, <tt>us:stat</tt>, <tt>us:spell</tt>, and <tt>us:boot</tt>.
+All tasks have shortcuts. Use <tt>us:conf</tt>, <tt>us:index</tt>, <tt>us:main</tt>, <tt>us:delta</tt>, <tt>us:merge</tt>, <tt>us:restart</tt>, <tt>us:start</tt>, <tt>us:stop</tt>, <tt>us:stat</tt>, <tt>us:spell</tt>, and <tt>us:boot</tt>.

data/README

@@ -1,4 +1,3 @@
-
 Ultrasphinx 
 
 Ruby on Rails configurator and client to the Sphinx full text search engine.
@@ -14,7 +13,7 @@ If you use this software, please {make a donation}[http://blog.evanweaver.com/do
 == Requirements
 
 * MySQL 5.0, or PostgreSQL 8.2
-* Sphinx 0.9.8-rc1
+* Sphinx 0.9.8-rc2
 * Rails 2.0.2
 
 More recent versions than listed are usually ok.
@@ -25,13 +24,14 @@ Sphinx/Ultrasphinx is the fastest and most stable Rails fulltext search solution
 
 Features include:
 
-* searching and ranking across orthogonal models
+* searching and ranking across multiple models
 * delta index support
 * excerpt highlighting
 * Google-style query parser
 * spellcheck
 * faceting on text, date, and numeric fields
-* field weighting, merging, and aliases
+* field weighting, merging, and aliasing
+* geodistance
 * <tt>belongs_to</tt> and <tt>has_many</tt> includes
 * drop-in compatibility with will_paginate[http://err.lighthouseapp.com/projects/466/home]
 * drop-in compatibility with Interlock[http://blog.evanweaver.com/files/doc/fauna/interlock/]
@@ -50,7 +50,7 @@ You also need the <tt>chronic</tt> gem:
   sudo gem install chronic
 
 Then, install the plugin:
-  script/plugin install -x svn://rubyforge.org/var/svn/fauna/ultrasphinx/trunk
+  script/plugin install git://github.com/fauna/ultrasphinx.git
  
 Next, copy the <tt>examples/default.base</tt> file to <tt>RAILS_ROOT/config/ultrasphinx/default.base</tt>. This file sets up the  Sphinx daemon options such as port, host, and index location.
   
@@ -124,7 +124,7 @@ See DEPLOYMENT_NOTES[link:files/DEPLOYMENT_NOTES.html].
 
 Note that since Ultrasphinx preloads indexed models, you need to make sure those models have their own dependencies in place early in the boot process. This may require adjusting the general plugin load order or moving monkey-patches from <tt>lib/</tt> to <tt>vendor/plugins/</tt>.
 
-PostgreSQL 8.2 and higher are well supported. However, make sure you have executed <tt>CREATE LANGUAGE plpgsql;</tt> at least once. This step does not need to be repeated, so depending on your DB permissions, you might be able to put it in a migration.
+PostgreSQL 8.2 and higher are well supported. However, make sure the stored procedure migration gets generated correctly. If you did not install the plugin via <tt>script/install</tt>, run <tt>script/generate ultrasphinx_migration</tt> by hand.
 
 == Reporting problems
 

data/Rakefile

@@ -0,0 +1,27 @@
+
+require 'echoe'
+
+Echoe.new("ultrasphinx") do |p|
+  p.project = "fauna"
+  p.summary = "Ruby on Rails configurator and client to the Sphinx fulltext search engine."
+  p.url = "http://blog.evanweaver.com/files/doc/fauna/ultrasphinx/"  
+  p.docs_host = "blog.evanweaver.com:~/www/bax/public/files/doc/"  
+  p.rdoc_pattern = /is_indexed.rb|search.rb|spell.rb|ultrasphinx.rb|^README|TODO|DEPLOY|RAKE_TASKS|CHANGELOG|^LICENSE/
+  p.dependencies = "chronic"
+  p.test_pattern = ["test/integration/*.rb", "test/unit/*.rb"]
+  p.rcov_options << '--include-file test\/integration\/app\/vendor\/plugins\/ultrasphinx\/lib\/.*\.rb --include-file '
+end
+
+desc "Run all the tests for every database adapter" 
+task "test_all" do
+  ['mysql', 'postgresql'].each do |adapter|
+    ENV['DB'] = adapter
+    ENV['PRODUCTION'] = nil
+    STDERR.puts "#{'='*80}\nDevelopment mode for #{adapter}\n#{'='*80}"
+    system("rake test:multi_rails:all")
+  
+    ENV['PRODUCTION'] = '1'
+    STDERR.puts "#{'='*80}\nProduction mode for #{adapter}\n#{'='*80}"
+    system("rake test:multi_rails:all")    
+  end
+end

data/examples/default.base

@@ -34,7 +34,7 @@ searchd
 {  
   # What interface the search daemon should listen on and where to store its logs
   address = 0.0.0.0
-  port = 3312
+  port = 3313
   seamless_rotate = 1
   log = <%= path %>log/searchd.log
   query_log = <%= path %>log/query.log
@@ -52,7 +52,7 @@ client
   
   # How your application connects to the search daemon (not necessarily the same as above)
   server_host = localhost
-  server_port = 3312
+  server_port = 3313
 }
 
 # Individual SQL source options
@@ -79,6 +79,11 @@ index
   # Enable these if you need wildcard searching. They will slow down indexing significantly.
   #   min_infix_len = 1    
   #   enable_star = 1 
+
+  # # URL search options
+  # # add " @, /, :," before " a-z," in the charset_table and uncomment prefix_fields
+  # to seach URL and email addresses
+  # prefix_fields = url, domain
     
   charset_type = utf-8 # or sbcs (Single Byte Character Set)
   charset_table = 0..9, A..Z->a..z, -, _, ., &, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F,U+C5->U+E5, U+E5, U+C4->U+E4, U+E4, U+D6->U+F6, U+F6, U+16B, U+0c1->a, U+0c4->a, U+0c9->e, U+0cd->i, U+0d3->o, U+0d4->o, U+0da->u, U+0dd->y, U+0e1->a, U+0e4->a, U+0e9->e, U+0ed->i, U+0f3->o, U+0f4->o, U+0fa->u, U+0fd->y, U+104->U+105, U+105, U+106->U+107, U+10c->c, U+10d->c, U+10e->d, U+10f->d, U+116->U+117, U+117, U+118->U+119, U+11a->e, U+11b->e, U+12E->U+12F, U+12F, U+139->l, U+13a->l, U+13d->l, U+13e->l, U+141->U+142, U+142, U+143->U+144, U+144,U+147->n, U+148->n, U+154->r, U+155->r, U+158->r, U+159->r, U+15A->U+15B, U+15B, U+160->s, U+160->U+161, U+161->s, U+164->t, U+165->t, U+16A->U+16B, U+16B, U+16e->u, U+16f->u, U+172->U+173, U+173, U+179->U+17A, U+17A, U+17B->U+17C, U+17C, U+17d->z, U+17e->z,

data/init.rb

@@ -1,2 +1,3 @@
 
 require 'ultrasphinx'
+require 'ultrasphinx/fields'

data/lib/ultrasphinx/autoload.rb

@@ -6,7 +6,6 @@ class Rails::Initializer
   def after_initialize_with_ultrasphinx_configuration
     after_initialize_without_ultrasphinx_configuration
     Ultrasphinx::Configure.load_constants
-    Ultrasphinx.verify_database_name
   end     
   
   alias_method_chain :after_initialize, :ultrasphinx_configuration

data/lib/ultrasphinx/configure.rb

@@ -16,21 +16,20 @@ module Ultrasphinx
                   filename = filename[0..-4]
                   begin                
                     File.basename(filename).camelize.constantize
-                  rescue NameError => e
+                  rescue LoadError, NameError => e
                     filename.camelize.constantize
                   end
                 end
               rescue Object => e
-                say "warning: possibly critical autoload error on #{filename}"
-                say e.inspect
-                #say e.backtrace.join("\n") if RAILS_ENV == "development"
+                say "warning: critical autoload error on #{filename}; try referencing \"#{filename.camelize}\" directly in the console"
+                say e.backtrace.join("\n") if RAILS_ENV == "development"
               end
             end 
           end
         end
   
         # Build the field-to-type mappings.
-        Fields.instance.configure(MODEL_CONFIGURATION)
+        Ultrasphinx::Fields.instance.configure(MODEL_CONFIGURATION)
       end
       
                     
@@ -40,7 +39,9 @@ module Ultrasphinx
         load_constants
               
         say "rebuilding configurations for #{RAILS_ENV} environment" 
-        say "available models are #{MODEL_CONFIGURATION.keys.to_sentence}"
+        # stable sort classes by name rather than rely on hash order
+        model_list = MODEL_CONFIGURATION.keys.sort
+        say "available models are #{model_list.to_sentence}"
         File.open(CONF_PATH, "w") do |conf|              
           conf.puts global_header            
           say "generating SQL"    
@@ -49,9 +50,8 @@ module Ultrasphinx
             sources = []
             cached_groups = Fields.instance.groups.join("\n")
 
-            MODEL_CONFIGURATION.each_with_index do |model_and_options, class_id|              
-              # This relies on hash sort order being deterministic per-machine
-              model, options = model_and_options
+            model_list.each_with_index do |model, class_id|
+              options = MODEL_CONFIGURATION[model]
               klass = model.constantize
               source = "#{model.tableize.gsub('/', '__')}_#{index}"
  
@@ -89,13 +89,16 @@ module Ultrasphinx
       def setup_source_database(klass)
         # Supporting Postgres now
         connection_settings = klass.connection.instance_variable_get("@config")
+        raise ConfigurationError, "Unsupported database adapter" unless connection_settings || defined?(JRUBY_VERSION)
 
         adapter_defaults = DEFAULTS[ADAPTER]
         raise ConfigurationError, "Unsupported database adapter" unless adapter_defaults
 
         conf = [adapter_defaults]                  
         connection_settings.reverse_merge(CONNECTION_DEFAULTS).each do |key, value|
-          conf << "#{CONFIG_MAP[key]} = #{value}" if CONFIG_MAP[key]          
+          if CONFIG_MAP[key] && !SOURCE_SETTINGS[CONFIG_MAP[key]]
+            conf << "#{CONFIG_MAP[key]} = #{value}"
+          end
         end                 
         conf.sort.join("\n")
       end
@@ -114,8 +117,14 @@ module Ultrasphinx
               # Warning about the sortable problem
               # XXX Kind of in an odd place, but I want to happen at index time
               Ultrasphinx.say "warning; text sortable columns on #{klass.name} will return wrong results with partial delta indexing"
-            end            
-            string = "#{source_string} > #{SQL_FUNCTIONS[ADAPTER]['delta']._interpolate(INDEXER_SETTINGS['delta'])}";
+            end
+
+            delta = INDEXER_SETTINGS['delta']
+            if delta 
+              string = "#{source_string} > #{SQL_FUNCTIONS[ADAPTER]['delta']._interpolate(delta)}";
+            else
+              raise ConfigurationError, "No 'indexer { delta }' setting specified in '#{BASE_PATH}'"
+            end
           else
             Ultrasphinx.say "warning; #{klass.name} will reindex the entire table during delta indexing"
           end
@@ -123,8 +132,18 @@ module Ultrasphinx
       end
       
       
-      def setup_source_arrays(index, klass, fields, class_id, conditions)        
-        condition_strings = Array(conditions).map do |condition| 
+      def setup_source_arrays(index, klass, fields, class_id, conditions, order)
+        if((klass.column_names.include? klass.inheritance_column) and (klass.superclass != ActiveRecord::Base))
+          if conditions.nil?
+            conditions = ""
+          else
+            conditions += " AND  "
+          end
+          conditions +=  "#{klass.table_name}.#{klass.inheritance_column} = '#{klass.name.to_s}'  "
+        end
+     	
+
+	condition_strings = Array(conditions).map do |condition| 
           "(#{condition})"
         end
         
@@ -132,7 +151,7 @@ module Ultrasphinx
           "(#{klass.table_name}.#{klass.primary_key} * #{MODEL_CONFIGURATION.size} + #{class_id}) AS id", 
           "#{class_id} AS class_id", "'#{klass.name}' AS class"]
         remaining_columns = fields.types.keys - ["class", "class_id"]        
-        [column_strings, [], condition_strings, [], false, remaining_columns]
+        [column_strings, [], condition_strings, [], false, remaining_columns, order]
       end
       
       
@@ -154,9 +173,9 @@ module Ultrasphinx
             
       def build_source(index, fields, model, options, class_id, klass, source, groups)
                 
-        column_strings, join_strings, condition_strings, group_bys, use_distinct, remaining_columns = 
+        column_strings, join_strings, condition_strings, group_bys, use_distinct, remaining_columns, order = 
           setup_source_arrays(
-            index, klass, fields, class_id, options['conditions'])
+            index, klass, fields, class_id, options['conditions'], options['order'])
             
         delta_condition = 
           build_delta_condition(
@@ -182,18 +201,18 @@ module Ultrasphinx
           SOURCE_SETTINGS._to_conf_string,
           setup_source_database(klass),
           range_select_string(klass, delta_condition),
-          build_query(klass, column_strings, join_strings, condition_strings, use_distinct, group_bys),
+          build_query(klass, column_strings, join_strings, condition_strings, use_distinct, group_bys, order),
           "\n" + groups,
           query_info_string(klass, class_id),
           "}\n\n"]
       end
       
       
-      def build_query(klass, column_strings, join_strings, condition_strings, use_distinct, group_bys)
+      def build_query(klass, column_strings, join_strings, condition_strings, use_distinct, group_bys, order)
         
         primary_key = "#{klass.table_name}.#{klass.primary_key}"
         group_bys = case ADAPTER
-          when 'mysql'
+          when 'mysql', 'jdbcmysql'
             primary_key
           when 'postgresql'
             # Postgres is very fussy about GROUP_BY 
@@ -211,7 +230,8 @@ module Ultrasphinx
           join_strings.uniq,
           "WHERE #{primary_key} >= $start AND #{primary_key} <= $end",
           condition_strings.uniq.map {|condition| "AND #{condition}" },
-          "GROUP BY #{group_bys}"
+          "GROUP BY #{group_bys}",
+          ("ORDER BY #{order}" if order)
         ].flatten.compact.join(" ")
       end
       
@@ -225,10 +245,10 @@ module Ultrasphinx
       
 
       def build_regular_fields(klass, fields, entries, column_strings, join_strings, group_bys, remaining_columns)          
-        entries.to_a.each do |entry|
-          source_string = "#{entry['table_alias']}.#{entry['field']}"
+        entries.to_a.each do |entry|          
+          source_string = "#{entry['table_alias']}.#{entry['field']}"          
           group_bys << source_string
-          column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], column_strings, remaining_columns)
+          column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], entry['sortable'], column_strings, remaining_columns)
         end
         
         [column_strings, join_strings, group_bys, remaining_columns]
@@ -242,7 +262,7 @@ module Ultrasphinx
           association = get_association(klass, entry)
 
           # You can use 'class_name' and 'association_sql' to associate to a model that doesn't actually 
-          # have an association
+          # have an association.
           join_klass = association ? association.class_name.constantize : entry['class_name'].constantize
                         
           raise ConfigurationError, "Unknown association from #{klass} to #{entry['class_name'] || entry['association_name']}" if not association and not entry['association_sql']
@@ -259,8 +279,8 @@ module Ultrasphinx
           end
           
           source_string = "#{entry['table_alias']}.#{entry['field']}"
-          group_bys << source_string
-          column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], column_strings, remaining_columns)                         
+          group_bys << source_string unless entry['dont_group_by']
+          column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], entry['sortable'], column_strings, remaining_columns)                         
         end
         
         [column_strings, join_strings, group_bys, remaining_columns]
@@ -272,26 +292,28 @@ module Ultrasphinx
           if entry['field']
             # Group concats
   
-            # Only has_many's or explicit sql right now
+            # Only has_many's or explicit sql right now.
             association = get_association(klass, entry)
             
             # You can use 'class_name' and 'association_sql' to associate to a model that doesn't actually 
-            # have an association
+            # have an association. The automatic choice of a table alias chosen might be kind of strange.
             join_klass = association ? association.class_name.constantize : entry['class_name'].constantize
         
             join_strings = install_join_unless_association_sql(entry['association_sql'], nil, join_strings) do 
-              # XXX make sure foreign key is right for polymorphic relationships
+              # XXX The foreign key is not verified for polymorphic relationships.
               association = get_association(klass, entry)
               "LEFT OUTER JOIN #{join_klass.table_name} AS #{entry['table_alias']} ON #{klass.table_name}.#{klass.primary_key} = #{entry['table_alias']}.#{association.primary_key_name}" + 
-                (entry['conditions'] ? " AND (#{entry['conditions']})" : "")
+                # XXX Is this valid?
+                (entry['conditions'] ? " AND (#{entry['conditions']})" : "") 
             end
             
             source_string = "#{entry['table_alias']}.#{entry['field']}"
+            order_string = ("ORDER BY #{entry['order']}" if entry['order'])
             # We are using the field in an aggregate, so we don't want to add it to group_bys
-            source_string = SQL_FUNCTIONS[ADAPTER]['group_concat']._interpolate(source_string)
+            source_string = SQL_FUNCTIONS[ADAPTER]['group_concat']._interpolate(source_string, order_string)
             use_distinct = true
             
-            column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], column_strings, remaining_columns)
+            column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], entry['sortable'], column_strings, remaining_columns)
             
           elsif entry['fields']
             # Regular concats
@@ -301,7 +323,7 @@ module Ultrasphinx
               group_bys << subsource_string
             end.join(', ') + ")"
             
-            column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], column_strings, remaining_columns)              
+            column_strings, remaining_columns = install_field(fields, source_string, entry['as'], entry['function_sql'], entry['facet'], entry['sortable'], column_strings, remaining_columns)              
 
           else
             raise ConfigurationError, "Invalid concatenate parameters for #{model}: #{entry.inspect}."
@@ -311,7 +333,7 @@ module Ultrasphinx
         [column_strings, join_strings, group_bys, use_distinct, remaining_columns]
       end
       
-    
+          
       def build_index(index, sources)
         ["\n# Index configuration\n\n",
           "index #{index}\n{",
@@ -323,12 +345,18 @@ module Ultrasphinx
       end
       
     
-      def install_field(fields, source_string, as, function_sql, with_facet, column_strings, remaining_columns)
+      def install_field(fields, source_string, as, function_sql, with_facet, with_sortable, column_strings, remaining_columns)
         source_string = function_sql._interpolate(source_string) if function_sql
 
         column_strings << fields.cast(source_string, as)
         remaining_columns.delete(as)
         
+        # Generate duplicate text fields for sorting
+        if with_sortable
+          column_strings << fields.cast(source_string, "#{as}_sortable")
+          remaining_columns.delete("#{as}_sortable")
+        end
+        
         # Generate hashed integer fields for text grouping
         if with_facet
           column_strings << "#{SQL_FUNCTIONS[ADAPTER]['hash']._interpolate(source_string)} AS #{as}_facet"