oedipus 0.0.1.pre1 → 0.0.1.pre2

data/.gitignore

@@ -6,3 +6,5 @@ spec/data/index/*
 spec/data/binlog/*
 spec/data/searchd.*
 spec/data/sphinx.*
+lib/oedipus/oedipus.so
+tmp/

data/README.md

@@ -5,53 +5,108 @@ real-time indexes and multi and/or faceted searches.
 
 It is not a clone of the PHP API, rather it is written from the ground up,
 wrapping the SphinxQL API offered by searchd.  Nor is it a plugin for
-ActiveRecord or DataMapper... though this is planned in separate gems.
+ActiveRecord or DataMapper... though this will follow in separate gems.
 
-It will provide some higher level of abstraction in terms of the ease with
-which faceted search may be implemented, though it will remain light and simple.
+Oedipus provides a level of abstraction in terms of the ease with which faceted
+search may be implemented, while remaining light and simple.
+
+Data structures are managed using core ruby data type (Array and Hash), ensuring
+simplicity and flexibilty.
 
 ## Current Status
 
 This gem is in development.  It is not ready for production use.  I work for
 a company called Flippa.com, which currently implements faceted search in a PHP
 part of the website, using a slightly older version of Sphinx with lesser
-support for SphinxQL.  Once a month the developers at Flippa are given three days
-to work on a project of their own choice.  This is my 'Triple Time' project.
+support for SphinxQL.  We want to move this search across to the ruby codebase
+of the website, but are held back by ruby's lack of support for Sphinx 2.
+
+Once a month the developers at Flippa are given three days to work on a project of
+their own choice.  This is my 'Triple Time' project.
 
-I anticipate another week or two of development before I can consider this project
+I anticipate another week or so of development before I can consider this project
 production-ready.
 
+## Dependencies
+
+  * ruby (>= 1.9)
+  * sphinx (>= 2.0.2)
+  * mysql.h / client development libraries (>= 4.1)
+
+The gem builds a small (tiny) native extension for interfacing with mysql, as
+existing gems either did not support multi-queries, or were too flaky
+(i.e. ruby-mysql). I was also concerned about potential conflicts with any
+specific ORMs users may be using.  I will add a pure-ruby option in due course
+(it requires implementing a relatively small subset of the mysql 4.1/5.0 protocol).
+
 ## Usage
 
-Not all of the following features are currently implemented, but the interface
-style is as follows.
+The following features are all currently implemented, but more are coming.
+
+### Connecting to Sphinx
 
 ``` ruby
 require "oedipus"
 
 sphinx = Oedipus.connect('localhost:9306') # sphinxql host
+```
+
+### Inserting
+
+``` ruby
+sphinx[:articles].insert(
+  7,
+  title:     "Badgers in the wild",
+  body:      "A big long wodge of text",
+  author_id: 4,
+  views:     102
+)
+```
+
+### Replacing
 
-# insert a record into the 'articles' real-time index
-record = sphinx[:articles].insert(
+``` ruby
+sphinx[:articles].replace(
   7,
   title:     "Badgers in the wild",
   body:      "A big long wodge of text",
   author_id: 4,
   views:     102
 )
-# The attributes (but not the indexed fields) are returned
-# => { id: 7,  author_id: 4, views: 102 }
+```
+
+### Updating
 
-# updating a record
-record = sphinx[:articles].update(7, views: 103)
-# The new attributes (but not the indexed fields) are returned
-# => { id: 7,  author_id: 4, views: 103 }
+``` ruby
+sphinx[:articles].update(7, views: 103)
+```
 
-# deleting a record
+### Deleting
+
+``` ruby
 sphinx[:articles].delete(7)
 # => true
+```
+
+### Fetching a known document (by ID)
 
-# searching the index
+``` ruby
+record = sphinx[:articles].fetch(7)
+# => { id: 7, views: 984, author_id: 3 }
+```
+
+### Fulltext searching
+
+You perform queries by invoking `#search` on the index.
+
+Oedipus makes no attempt to provide an abstraction layer for the fulltext
+query itself.  I believe this would not be flexible enough.  Sphinx fulltext
+queries are extremely featureful, very dense and concise; a ruby solution
+would only be lengthier and harder to understand, IMHO.  Perhaps such an
+abstraction could be provided by a separate gem.
+
+
+``` ruby
 results = sphinx[:articles].search("badgers", limit: 2)
 
 # Meta deta indicates the overall number of matched records, while the ':records'
@@ -67,25 +122,108 @@ results = sphinx[:articles].search("badgers", limit: 2)
 #     { id: 11, author_id: 6, views: 23 }
 #   ]
 # }
+```
+
+### Attribute filters
 
-# using attribute filters
-results = sphinx[:articles].search(
+Result formatting is the same as for a fulltext search.  You can add as many
+filters as you like.
+
+``` ruby
+# equality
+sphinx[:articles].search(
   "example",
   author_id: 7
 )
-# => (the same results, filtered by author)
 
-# performing a faceted search
+# less than or equal
+sphinx[:articles].search(
+  "example",
+  views: -Float::INFINITY..100
+)
+
+sphinx[:articles].search(
+  "example",
+  views: Oedipus.lte(100)
+)
+
+# greater than
+sphinx[:articles].search(
+  "example",
+  views: 100...Float::INFINITY
+)
+
+sphinx[:articles].search(
+  "example",
+  views: Oedipus.gt(100)
+)
+
+# not equal
+sphinx[:articles].search(
+  "example",
+  author_id: Oedipus.not(7)
+)
+
+# between
+sphinx[:articles].search(
+  "example",
+  views: 50..100
+)
+
+sphinx[:articles].search(
+  "example",
+  views: 50...100
+)
+
+# not between
+sphinx[:articles].search(
+  "example",
+  views: Oedipus.not(50..100)
+)
+
+sphinx[:articles].search(
+  "example",
+  views: Oedipus.not(50...100)
+)
+
+# IN( ... )
+sphinx[:articles].search(
+  "example",
+  author_id: [7, 22]
+)
+
+# NOT IN( ... )
+sphinx[:articles].search(
+  "example",
+  author_id: Oedipus.not([7, 22])
+)
+```
+
+### Faceted searching
+
+A faceted search takes a base query and a set of additional queries that are
+variations on it.  Oedipus makes this simple by allowing your facets to inherit
+from the base query.
+
+Oedipus allows you to replace '%{query}' in your facets with whatever was in the
+original query.  This can be useful if you want to provide facets that only
+perform the search in the title of the document (`"@title (%{query})"`) for example.
+
+Each facet is given a name, which is used to reference them in the results.
+
+Sphinx optimizes the queries by figuring out what the common parts are.  Currently
+it does two optimizations, though in future this will likely improve further, so
+using this technique to do your faceted searches is a good idea.
+
+``` ruby
 results = sphinx[:articles].facted_search(
   "badgers",
   facets: {
     popular:         { views: 100..10000 },
-    farming:         "farming",
-    popular_farming: ["farming", { views: 100..10000 } ]
+    also_farming:    "%{query} & farming",
+    popular_farming: ["%{query} & farming", views: 100..10000 ]
   }
 )
-# The main results are returned in the ':records' array, and all the facets in
-# the ':facets' Hash.
 # => {
 #   total_found: 987,
 #   time: 0.000,
@@ -96,7 +234,7 @@ results = sphinx[:articles].facted_search(
 #       time: 0.000,
 #       records: [ ... ]
 #     },
-#     farming: {
+#     also_farming: {
 #       total_found: 123,
 #       time: 0.000,
 #       records: [ ... ]
@@ -108,17 +246,21 @@ results = sphinx[:articles].facted_search(
 #     }
 #   }
 # }
-# 
-# When performing a faceted search, the primary search is used as the basis for
-# each facet, so they can be considered refinements.
+```
+
+### General purpose multi-search
+
+If you want to execute multiple queries in a batch that are not related to each
+other (which would be a faceted search), then you can use `#multi_search`.
 
-# performing a mutli-search
+You pass a Hash of named queries and get a Hash of named resultsets.
+
+``` ruby
 results = sphinx[:articles].multi_search(
-  badgers: ["badgers", { limit: 30 }],
-  frogs:   "frogs AND wetlands",
-  rabbits: ["rabbits OR burrows", { view_count: 20..100 }]
+  badgers: ["badgers", limit: 30],
+  frogs:   "frogs & wetlands",
+  rabbits: ["rabbits | burrows", view_count: 20..100]
 )
-# The results are returned in a 2-dimensional Hash, keyed as sent in the query
 # => {
 #   badgers: {
 #     ...
@@ -130,19 +272,68 @@ results = sphinx[:articles].multi_search(
 #     ...
 #   }
 # }
-# 
-# Unlike with a faceted search, the queries in a multi-search do not have to be
-# related to one another.
 ```
 
-## Future Plans
+### Limits and offsets
+
+Note that Sphinx applies a limit of 20 by default, so you probably want to specify
+a limit yourself.  You are bound by your `max_matches` setting in sphinx.conf.
+
+Note that the meta data will still indicate the actual number of results that matched;
+you simply get a smaller collection of materialized records.
+
+``` ruby
+sphinx[:articles].search("bobcats", limit: 50)
+sphinx[:articles].search("bobcats", limit: 50, offset: 150)
+```
+
+### Ordering
+
+``` ruby
+sphinx[:articles].search("badgers", order: { views: :asc })
+```
+
+## Running the specs
 
-I plan to release gems for integration with DataMapper and ActiveRecord.  DataMapper
-first, since that's what we use.
+There are both unit tests and integration tests in the specs/ directory.  By default they
+will both run, but in order for the integration specs to work, you need a locally
+installed copy of Sphinx [1].  You then execute the specs as follows:
+
+    SEARCHD=/path/to/bin/searchd bundle exec rake spec
+
+If you don't have Sphinx installed locally, you cannot run the integration specs (they need
+to write config files and start and stop sphinx internally).
+
+To run the unit tests alone, without the need for Sphinx:
+
+    bundle exec rake spec:unit
+
+If you have made changes to the C extension, those changes will be compiled and installed
+(to the lib/ directory) before the specs are run.
+
+You may also compile the C extension and run the specs separately, if you prefer:
+
+    bundle exec rake compile
+    bundle exec rspec spec/unit/
+
+### Footnotes
+
+  [1] You can build a local copy of sphinx without installing it on the system:
+
+    cd sphinx-2.0.4/
+    ./configure
+    make
+
+  The searchd binary will be found in /path/to/sphinx-2.0.4/src/searchd.
+
+## Future Plans
 
-I also intend to remove ruby-mysql from the dependencies, as it doesn't perfectly fit
-the needs of SphinxQL.  I will be implementing the limited subset of the MySQL protocol
-by hand (which is not as big a deal as it sounds).
+  * Integration with DataMapper and ActiveRecord (DataMapper first)
+  * Distributed index support (sharding writes between indexes)
+  * Make C extension optional and provide an implementation in pure-ruby
+  * N-dimensional faceted search (facets inside of facets)
+  * Query translation layer for Google-style AND/OR/NOT interpretation
+  * Fulltext query sanitization for unsafe user input (e.g. @missing field)
 
 ## Copyright and Licensing
 

data/Rakefile

@@ -1 +1,26 @@
 require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rake/extensiontask"
+
+Rake::ExtensionTask.new('oedipus') do |ext|
+  ext.lib_dir = File.join('lib', 'oedipus')
+end
+
+desc "Run the full RSpec suite (requires SEARCHD environment variable)"
+RSpec::Core::RakeTask.new('spec') do |t|
+  t.pattern     = 'spec/'
+end
+
+desc "Run the RSpec unit tests alone"
+RSpec::Core::RakeTask.new('spec:unit') do |t|
+  t.pattern = 'spec/unit/'
+end
+
+desc "Run the integration tests (requires SEARCHD environment variable)"
+RSpec::Core::RakeTask.new('spec:integration') do |t|
+  t.pattern = 'spec/integration/'
+end
+
+Rake::Task['spec'].prerequisites << :compile
+Rake::Task['spec:unit'].prerequisites << :compile
+Rake::Task['spec:integration'].prerequisites << :compile

data/ext/oedipus/extconf.rb

@@ -0,0 +1,72 @@
+# encoding: UTF-8
+require 'mkmf'
+
+# borrowed from https://github.com/brianmario/mysql2/master/ext/mysql2/extconf.rb
+
+def asplode lib
+  abort "-----\n#{lib} is missing.  please check your installation of mysql and try again.\n-----"
+end
+
+# borrowed from mysqlplus
+# http://github.com/oldmoe/mysqlplus/blob/master/ext/extconf.rb
+dirs = ENV['PATH'].split(File::PATH_SEPARATOR) + %w[
+  /opt
+  /opt/local
+  /opt/local/mysql
+  /opt/local/lib/mysql5
+  /usr
+  /usr/mysql
+  /usr/local
+  /usr/local/mysql
+  /usr/local/mysql-*
+  /usr/local/lib/mysql5
+].map{|dir| "#{dir}/bin" }
+
+GLOB = "{#{dirs.join(',')}}/{mysql_config,mysql_config5}"
+
+if RUBY_PLATFORM =~ /mswin|mingw/
+  inc, lib = dir_config('mysql')
+  exit 1 unless have_library("libmysql")
+elsif mc = (with_config('mysql-config') || Dir[GLOB].first) then
+  mc = Dir[GLOB].first if mc == true
+  cflags = `#{mc} --cflags`.chomp
+  exit 1 if $? != 0
+  libs = `#{mc} --libs_r`.chomp
+  if libs.empty?
+    libs = `#{mc} --libs`.chomp
+  end
+  exit 1 if $? != 0
+  $CPPFLAGS += ' ' + cflags
+  $libs = libs + " " + $libs
+else
+  inc, lib = dir_config('mysql', '/usr/local')
+  libs = ['m', 'z', 'socket', 'nsl', 'mygcc']
+  while not find_library('mysqlclient', 'mysql_query', lib, "#{lib}/mysql") do
+    exit 1 if libs.empty?
+    have_library(libs.shift)
+  end
+end
+
+if have_header('mysql.h') then
+  prefix = nil
+elsif have_header('mysql/mysql.h') then
+  prefix = 'mysql'
+else
+  asplode 'mysql.h'
+end
+
+%w{ errmsg.h mysqld_error.h }.each do |h|
+  header = [prefix, h].compact.join '/'
+  asplode h unless have_header h
+end
+
+# GCC specific flags
+if RbConfig::MAKEFILE_CONFIG['CC'] =~ /gcc/
+  $CFLAGS << ' -Wall -funroll-loops'
+
+  if hard_mysql_path = $libs[%r{-L(/[^ ]+)}, 1]
+    $LDFLAGS << " -Wl,-rpath,#{hard_mysql_path}"
+  end
+end
+
+create_makefile('oedipus/oedipus')

data/ext/oedipus/oedipus.c

@@ -0,0 +1,239 @@
+/*-- encoding: utf-8 --*/
+
+/*
+ * Oedipus Sphinx 2 Search.
+ * Copyright © 2012 Chris Corbyn.
+ *
+ * See LICENSE file for details.
+ */
+
+#include "oedipus.h"
+
+/* -- Public methods -- */
+
+static VALUE odp_new(VALUE klass, VALUE host, VALUE port) {
+  OdpMysql * conn;
+  VALUE      self;
+  VALUE      args[2];
+
+  conn = malloc(sizeof(OdpMysql));
+  conn->connected = 0;
+
+  self = Data_Wrap_Struct(klass, 0, odp_free, conn);
+
+  args[0] = host;
+  args[1] = port;
+
+  rb_obj_call_init(self, 2, args);
+
+  return self;
+}
+
+static VALUE odp_initialize(VALUE self, VALUE host, VALUE port) {
+  Check_Type(host, T_STRING);
+  Check_Type(port, T_FIXNUM);
+
+  rb_iv_set(self, "@host", host);
+  rb_iv_set(self, "@port", port);
+
+  odp_open(self);
+
+  return self;
+}
+
+static VALUE odp_open(VALUE self) {
+  OdpMysql * conn;
+
+  Data_Get_Struct(self, OdpMysql, conn);
+
+  if (conn->connected) {
+    return Qfalse;
+  }
+
+  if ((conn->ptr = mysql_init(NULL)) == NULL) {
+    odp_raise(self, "Unable to initialize mysql");
+  }
+
+  if (mysql_real_connect(conn->ptr,
+                         RSTRING_PTR(rb_iv_get(self, "@host")),
+                         "",
+                         "",
+                         NULL,
+                         NUM2UINT(rb_iv_get(self, "@port")),
+                         NULL,
+                         CLIENT_MULTI_STATEMENTS) == NULL) {
+    odp_raise(self, "Unable to connect to mysql");
+  }
+
+  conn->connected = 1;
+
+  return Qtrue;
+}
+
+static VALUE odp_close(VALUE self) {
+  OdpMysql * conn;
+
+  Data_Get_Struct(self, OdpMysql, conn);
+
+  if (!conn->connected) {
+    return Qfalse;
+  }
+
+  mysql_close(conn->ptr);
+  conn->connected = 0;
+
+  return Qtrue;
+}
+
+static VALUE odp_execute(VALUE self, VALUE sql) {
+  OdpMysql  * conn;
+
+  Check_Type(sql, T_STRING);
+
+  Data_Get_Struct(self, OdpMysql, conn);
+
+  if (!conn->connected) {
+    odp_raise(self, "Cannot execute query on a closed connection");
+  }
+
+  if (mysql_query(conn->ptr, RSTRING_PTR(sql))) {
+    odp_raise(self, "Failed to execute statement(s)");
+  }
+
+  return INT2NUM(mysql_affected_rows(conn->ptr));
+}
+
+static VALUE odp_query(VALUE self, VALUE sql) {
+  OdpMysql      * conn;
+  MYSQL_RES     * rs;
+  int             status;
+  int             num_fields;
+  MYSQL_ROW       row;
+  MYSQL_FIELD   * fields;
+  unsigned long * lengths;
+  int             i;
+  VALUE           rows;
+  VALUE           hash;
+  VALUE           results;
+
+  Check_Type(sql, T_STRING);
+
+  Data_Get_Struct(self, OdpMysql, conn);
+
+  if (!conn->connected) {
+    odp_raise(self, "Cannot execute query on a closed connection");
+  }
+
+  if (mysql_query(conn->ptr, RSTRING_PTR(sql))) {
+    odp_raise(self, "Failed to execute statement(s)");
+  }
+
+  results = rb_ary_new();
+
+  do {
+    if ((rs = mysql_store_result(conn->ptr)) != NULL) {
+      rb_ary_push(results, (rows = rb_ary_new()));
+
+      num_fields = mysql_num_fields(rs);
+      fields = mysql_fetch_fields(rs);
+
+      while ((row = mysql_fetch_row(rs))) {
+        lengths = mysql_fetch_lengths(rs);
+        rb_ary_push(rows, (hash = rb_hash_new()));
+        for (i = 0; i < num_fields; ++i) {
+          rb_hash_aset(hash,
+                       rb_str_new2(fields[i].name),
+                       odp_cast_value(fields[i], row[i], lengths[i]));
+        }
+      }
+
+      mysql_free_result(rs);
+    }
+
+    if ((status = mysql_next_result(conn->ptr)) > 0) {
+      odp_raise(self, "Query execution failed");
+    }
+  } while (status == 0);
+
+  return results;
+}
+
+/* -- Internal functions -- */
+
+static void odp_raise(VALUE self, const char *msg) {
+  OdpMysql * conn;
+
+  Data_Get_Struct(self, OdpMysql, conn);
+  rb_raise(rb_path2class("Oedipus::ConnectionError"),
+           "%s. Error %u: %s", msg, mysql_errno(conn->ptr), mysql_error(conn->ptr));
+}
+
+static void odp_free(OdpMysql *conn) {
+  if (conn->connected) {
+    mysql_close(conn->ptr);
+  }
+  free(conn);
+}
+
+static VALUE odp_cast_value(MYSQL_FIELD f, char * v, unsigned long len) {
+  short  s;
+  int    i;
+  long   l;
+  double d;
+
+  // FIXME: Add the DATETIME, TIMESTAMP, TIME, DATE and YEAR conversions
+  switch (f.type) {
+    case MYSQL_TYPE_NULL:
+      return Qnil;
+
+    case MYSQL_TYPE_TINY:
+    case MYSQL_TYPE_SHORT:
+      sscanf(v, "%hd", &s);
+      return INT2NUM(s);
+
+    case MYSQL_TYPE_LONG:
+      sscanf(v, "%d", &i);
+      return INT2NUM(i);
+
+    case MYSQL_TYPE_INT24:
+    case MYSQL_TYPE_LONGLONG:
+      sscanf(v, "%ld", &l);
+      return INT2NUM(l);
+
+    case MYSQL_TYPE_DECIMAL:
+    case MYSQL_TYPE_NEWDECIMAL:
+      rb_require("bigdecimal");
+      return rb_funcall(rb_path2class("BigDecimal"),
+                        rb_intern("new"),
+                        1,
+                        rb_str_new(v, len));
+
+    case MYSQL_TYPE_DOUBLE:
+    case MYSQL_TYPE_FLOAT:
+      sscanf(v, "%lf", &d);
+      return DBL2NUM(d);
+
+    case MYSQL_TYPE_STRING:
+    case MYSQL_TYPE_VAR_STRING:
+    case MYSQL_TYPE_BLOB:
+    case MYSQL_TYPE_SET:
+    case MYSQL_TYPE_ENUM:
+    default:
+      return rb_str_new(v, len);
+  }
+}
+
+/* -- Extension initialization -- */
+
+void Init_oedipus(void) {
+  VALUE mOedipus = rb_define_module("Oedipus");
+  VALUE cMysql   = rb_define_class_under(mOedipus, "Mysql", rb_cObject);
+
+  rb_define_method(cMysql, "initialize", odp_initialize, 2);
+  rb_define_method(cMysql, "open",       odp_open,       0);
+  rb_define_method(cMysql, "close",      odp_close,      0);
+  rb_define_method(cMysql, "execute",    odp_execute,    1);
+  rb_define_method(cMysql, "query",      odp_query,      1);
+
+  rb_define_singleton_method(cMysql, "new",  odp_new, 2);
+}