fast_trie 0.3.5

data/README.textile

@@ -0,0 +1,132 @@
+h1. Trie
+
+This is a trie for Ruby using libdatrie. It uses a dual-array system, meaning it has best-in-class memory usage and search time.
+
+
+h2. What is a trie?
+
+I suck at explaining things.  Wikipedia doesn't.  http://wikipedia.org/wiki/Trie.
+
+But in short a trie is a data structure that holds strings in a tree.  So if you inserted the words 'arc', 'ark', and 'ape' in a trie you could visualize it thusly:
+
+<pre>
+      p - e
+    /
+  a - r - c
+        \
+          k
+</pre>
+
+It's easy to see how this can have pretty neat implications for things like searching through lists of strings, sorting lists of strings, and things like spelling correction and autocompletion.
+
+
+h2. Tutorial
+
+Let's go through building a simple autocompleter using Trie.
+
+<pre><code>
+  Trie.new
+</code></pre>
+
+Anyway.  So we've created our blank trie.  Now, since we're creating an autocompleter, we'll need to add some words into it.  We do that simply with the add method.
+
+<pre><code>
+  words.each do |word|
+    trie.add word
+  end
+</code></pre>
+
+Or if you have some integer data to store along with the words, such as weights or scores of some kind, you'd do it like so...
+
+<pre><code>
+  words_and_weights do |word,weight|
+    trie.add word, weight
+  end
+</code></pre>
+
+Great, so we've populated our trie with some words. Let's make sure those words are really there.
+
+<pre><code>
+  trie.has_key?('widget')  #=> true
+
+  trie.get('widget')  #=> -1 or your value
+
+  trie.get('not-in-the-trie')  #=> nil
+</code></pre>
+
+If you didn't enter a value to go along with the word, calling <code>get</code> with it will return -1.
+
+Okay great, we have our populated trie, we've confirmed that the keys are in there.  Let's make an autocompleter!  For this we'll need to use the <code>children</code> method.  We'll do this as a simple Rails action, with the assumption you've initialized the trie into <code>TRIE</code>.
+
+<pre><code>
+  def autocomplete
+    children = TRIE.children(params[:prefix])
+
+    respond_to do |format|
+      format.js { render(:string => JSON.dump(children)) }
+      format.yaml { render(:string => YAML.dump(children)) }
+    end
+  end
+</code></pre>
+
+Yep, that's it.
+
+There are, of course, some more interesting and advanced ways to use a trie.  For instance, this snippet take a string, then walks down the trie, noting each word it finds along the way.
+
+<pre><code>
+  word = 'forestry'
+  node = trie.root
+
+  word.split('').each do |char|
+    break unless node.walk!(char)
+    if node.terminal?
+      puts "Found me a word: #{node.full_state}"
+    end
+  end
+</code></pre>
+
+By calling <code>root</code> on a Trie object, you get a TrieNode, pointed at the root of the trie.  You can then use this node to walk the trie and perceive things about each word.
+
+
+h2. Performance Characteristics
+
+Here are some quick benchmarks on my 2.4ghz Intel Core 2 Duo MacBook Pro:
+
+For keys that are 5 characters long:
+31,344 adds/second
+1,827,408 searches/second
+38,453 prefixes searches/second
+
+For keys that are 10 characters long:
+30,653 adds/second
+1,802,649 searches/second
+13,553 prefix searches/second
+
+For keys that are 20 characters long:
+30,488 adds/second
+1,851,461 searches/second
+5,855 prefix searches/second
+
+For keys that are 40 characters long:
+30,710 adds/second
+1,838,380 searches/second
+2,762 prefix searches/second
+
+
+There are a few takeaways from this. First, there is no strong correlation between length of keys and insert or retrieve time. They stay fairly constant as the length of keys increase. Secondly, doing prefix searches with this trie gets slower linearly with the length of the keys in the trie.
+
+This points to a limitation of this type of trie.  It is based on libdatrie, which is a dual-array trie.  When finding branches from a particular node, we must query all possible branches to determine whether or not they exist.  So for each node we do 255 of these queries.
+
+There may be some tricks to speed this up, but for now it is simply a limitation of this trie.
+
+Now, let's look at the effect of the size of the trie itself on query and insertion time.  For this test I inserted 100, 1000, 10000, 100000, and 1000000 words in the trie.  We measure the insertion and retrieval time in each.  The graph below shows the results.
+
+!http://codehallow.com/effect_of_size.png!
+
+So, keeping in mind that we're increasing by orders of magnitude, you can see that the insertion time does take a signifcant hit.  Retrieval also goes down but at a very gradual rate.  (It decreases by about 50% in total, despite the size increasing by 1,000,000%.)
+
+The reason the insertion times takes such a beating is due, again, to a limitation of the trie.  Storing a trie in the dual array setup that is used is excellent for memory usage and retrieval time.  Best in class, in fact. However, the more things are added into the trie the more complicated it gets to insert things.  It often requires shuffling large pieces of the arrays.  There may be room for optimization here, but ultimately insertion time will increase with the size of the trie.
+
+
+
+Copyright (c) 2008 Tyler McMullen. See LICENSE for details.

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+patch: 5
+major: 0
+minor: 3

data/ext/trie/Makefile

@@ -0,0 +1,149 @@
+
+SHELL = /bin/sh
+
+#### Start of system configuration section. ####
+
+srcdir = .
+topdir = /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin9.0
+hdrdir = $(topdir)
+VPATH = $(srcdir):$(topdir):$(hdrdir)
+prefix = $(DESTDIR)/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr
+exec_prefix = $(prefix)
+sitedir = $(DESTDIR)/Library/Ruby/Site
+rubylibdir = $(libdir)/ruby/$(ruby_version)
+docdir = $(datarootdir)/doc/$(PACKAGE)
+dvidir = $(docdir)
+datarootdir = $(prefix)/share
+archdir = $(rubylibdir)/$(arch)
+sbindir = $(exec_prefix)/sbin
+psdir = $(docdir)
+localedir = $(datarootdir)/locale
+htmldir = $(docdir)
+datadir = $(datarootdir)
+includedir = $(prefix)/include
+infodir = $(DESTDIR)/usr/share/info
+sysconfdir = $(prefix)/etc
+mandir = $(DESTDIR)/usr/share/man
+libdir = $(exec_prefix)/lib
+sharedstatedir = $(prefix)/com
+oldincludedir = $(DESTDIR)/usr/include
+pdfdir = $(docdir)
+sitearchdir = $(sitelibdir)/$(sitearch)
+bindir = $(exec_prefix)/bin
+localstatedir = $(prefix)/var
+sitelibdir = $(sitedir)/$(ruby_version)
+libexecdir = $(exec_prefix)/libexec
+
+CC = gcc
+LIBRUBY = $(LIBRUBY_SO)
+LIBRUBY_A = lib$(RUBY_SO_NAME)-static.a
+LIBRUBYARG_SHARED = -l$(RUBY_SO_NAME)
+LIBRUBYARG_STATIC = -l$(RUBY_SO_NAME)
+
+RUBY_EXTCONF_H = 
+CFLAGS   =  -fno-common -arch ppc -arch i386 -Os -pipe -fno-common 
+INCFLAGS = -I. -I$(topdir) -I$(hdrdir) -I$(srcdir)
+CPPFLAGS =  
+CXXFLAGS = $(CFLAGS) 
+DLDFLAGS = -L. -arch ppc -arch i386  
+LDSHARED = cc -arch ppc -arch i386 -pipe -bundle -undefined dynamic_lookup
+AR = ar
+EXEEXT = 
+
+RUBY_INSTALL_NAME = ruby
+RUBY_SO_NAME = ruby
+arch = universal-darwin9.0
+sitearch = universal-darwin9.0
+ruby_version = 1.8
+ruby = /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
+RUBY = $(ruby)
+RM = rm -f
+MAKEDIRS = mkdir -p
+INSTALL = /usr/bin/install -c
+INSTALL_PROG = $(INSTALL) -m 0755
+INSTALL_DATA = $(INSTALL) -m 644
+COPY = cp
+
+#### End of system configuration section. ####
+
+preload = 
+
+libpath = . $(libdir)
+LIBPATH =  -L"." -L"$(libdir)"
+DEFFILE = 
+
+CLEANFILES = mkmf.log
+DISTCLEANFILES = 
+
+extout = 
+extout_prefix = 
+target_prefix = 
+LOCAL_LIBS = 
+LIBS = $(LIBRUBYARG_SHARED)  -lpthread -ldl -lm  
+SRCS = darray.c fileutils.c tail.c trie-private.c trie.c
+OBJS = darray.o fileutils.o tail.o trie-private.o trie.o
+TARGET = trie
+DLLIB = $(TARGET).bundle
+EXTSTATIC = 
+STATIC_LIB = 
+
+RUBYCOMMONDIR = $(sitedir)$(target_prefix)
+RUBYLIBDIR    = $(sitelibdir)$(target_prefix)
+RUBYARCHDIR   = $(sitearchdir)$(target_prefix)
+
+TARGET_SO     = $(DLLIB)
+CLEANLIBS     = $(TARGET).bundle $(TARGET).il? $(TARGET).tds $(TARGET).map
+CLEANOBJS     = *.o *.a *.s[ol] *.pdb *.exp *.bak
+
+all:		$(DLLIB)
+static:		$(STATIC_LIB)
+
+clean:
+		@-$(RM) $(CLEANLIBS) $(CLEANOBJS) $(CLEANFILES)
+
+distclean:	clean
+		@-$(RM) Makefile $(RUBY_EXTCONF_H) conftest.* mkmf.log
+		@-$(RM) core ruby$(EXEEXT) *~ $(DISTCLEANFILES)
+
+realclean:	distclean
+install: install-so install-rb
+
+install-so: $(RUBYARCHDIR)
+install-so: $(RUBYARCHDIR)/$(DLLIB)
+$(RUBYARCHDIR)/$(DLLIB): $(DLLIB)
+	$(INSTALL_PROG) $(DLLIB) $(RUBYARCHDIR)
+install-rb: pre-install-rb install-rb-default
+install-rb-default: pre-install-rb-default
+pre-install-rb: Makefile
+pre-install-rb-default: Makefile
+$(RUBYARCHDIR):
+	$(MAKEDIRS) $@
+
+site-install: site-install-so site-install-rb
+site-install-so: install-so
+site-install-rb: install-rb
+
+.SUFFIXES: .c .m .cc .cxx .cpp .C .o
+
+.cc.o:
+	$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
+
+.cxx.o:
+	$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
+
+.cpp.o:
+	$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
+
+.C.o:
+	$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
+
+.c.o:
+	$(CC) $(INCFLAGS) $(CPPFLAGS) $(CFLAGS) -c $<
+
+$(DLLIB): $(OBJS)
+	@-$(RM) $@
+	$(LDSHARED) -o $@ $(OBJS) $(LIBPATH) $(DLDFLAGS) $(LOCAL_LIBS) $(LIBS)
+
+
+
+$(OBJS): ruby.h defines.h

data/ext/trie/darray.c

@@ -0,0 +1,673 @@
+/* -*- Mode: C; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- */
+/*
+ * darray.c - Double-array trie structure
+ * Created: 2006-08-13
+ * Author:  Theppitak Karoonboonyanan <thep@linux.thai.net>
+ */
+
+#include <string.h>
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "trie-private.h"
+#include "darray.h"
+#include "fileutils.h"
+
+/*----------------------------------*
+ *    INTERNAL TYPES DECLARATIONS   *
+ *----------------------------------*/
+
+typedef struct _Symbols Symbols;
+
+struct _Symbols {
+    short       num_symbols;
+    TrieChar    symbols[256];
+};
+
+static Symbols *    symbols_new ();
+static void         symbols_free (Symbols *syms);
+static void         symbols_add (Symbols *syms, TrieChar c);
+
+#define symbols_num(s)          ((s)->num_symbols)
+#define symbols_get(s,i)        ((s)->symbols[i])
+#define symbols_add_fast(s,c)   ((s)->symbols[(s)->num_symbols++] = c)
+
+/*-----------------------------------*
+ *    PRIVATE METHODS DECLARATIONS   *
+ *-----------------------------------*/
+
+#define da_get_free_list(d)      (1)
+
+static Bool         da_check_free_cell (DArray         *d,
+                                        TrieIndex       s);
+
+static Bool         da_has_children    (DArray         *d,
+                                        TrieIndex       s);
+
+static Symbols *    da_output_symbols  (const DArray   *d,
+                                        TrieIndex       s);
+
+static TrieChar *   da_get_state_key   (const DArray   *d,
+                                        TrieIndex       state);
+
+static TrieIndex    da_find_free_base  (DArray         *d,
+                                        const Symbols  *symbols);
+
+static Bool         da_fit_symbols     (DArray         *d,
+                                        TrieIndex       base,
+                                        const Symbols  *symbols);
+
+static void         da_relocate_base   (DArray         *d,
+                                        TrieIndex       s,
+                                        TrieIndex       new_base);
+
+static Bool         da_extend_pool     (DArray         *d,
+                                        TrieIndex       to_index);
+
+static void         da_alloc_cell      (DArray         *d,
+                                        TrieIndex       cell);
+
+static void         da_free_cell       (DArray         *d,
+                                        TrieIndex       cell);
+
+static Bool         da_enumerate_recursive (const DArray   *d,
+                                            TrieIndex       state,
+                                            DAEnumFunc      enum_func,
+                                            void           *user_data);
+
+/* ==================== BEGIN IMPLEMENTATION PART ====================  */
+
+/*------------------------------------*
+ *   INTERNAL TYPES IMPLEMENTATIONS   *
+ *------------------------------------*/
+
+static Symbols *
+symbols_new ()
+{
+    Symbols *syms;
+
+    syms = (Symbols *) malloc (sizeof (Symbols));
+
+    if (!syms)
+        return NULL;
+
+    syms->num_symbols = 0;
+
+    return syms;
+}
+
+static void
+symbols_free (Symbols *syms)
+{
+    free (syms);
+}
+
+static void
+symbols_add (Symbols *syms, TrieChar c)
+{
+    short lower, upper;
+
+    lower = 0;
+    upper = syms->num_symbols;
+    while (lower < upper) {
+        short middle;
+
+        middle = (lower + upper)/2;
+        if (c > syms->symbols[middle])
+            lower = middle + 1;
+        else if (c < syms->symbols[middle])
+            upper = middle;
+        else
+            return;
+    }
+    if (lower < syms->num_symbols) {
+        memmove (syms->symbols + lower + 1, syms->symbols + lower,
+                 syms->num_symbols - lower);
+    }
+    syms->symbols[lower] = c;
+    syms->num_symbols++;
+}
+
+/*------------------------------*
+ *    PRIVATE DATA DEFINITONS   *
+ *------------------------------*/
+
+typedef struct {
+    TrieIndex   base;
+    TrieIndex   check;
+} DACell;
+
+struct _DArray {
+    TrieIndex   num_cells;
+    DACell     *cells;
+};
+
+/*-----------------------------*
+ *    METHODS IMPLEMENTAIONS   *
+ *-----------------------------*/
+
+#define DA_SIGNATURE 0xDAFCDAFC
+
+/* DA Header:
+ * - Cell 0: SIGNATURE, number of cells
+ * - Cell 1: free circular-list pointers
+ * - Cell 2: root node
+ * - Cell 3: DA pool begin
+ */
+#define DA_POOL_BEGIN 3
+
+DArray *
+da_new ()
+{
+    DArray     *d;
+
+    d = (DArray *) malloc (sizeof (DArray));
+    if (!d)
+        return NULL;
+
+    d->num_cells = DA_POOL_BEGIN;
+    d->cells     = (DACell *) malloc (d->num_cells * sizeof (DACell));
+    if (!d->cells)
+        goto exit_da_created;
+    d->cells[0].base = DA_SIGNATURE;
+    d->cells[0].check = d->num_cells;
+    d->cells[1].base = -1;
+    d->cells[1].check = -1;
+    d->cells[2].base = DA_POOL_BEGIN;
+    d->cells[2].check = 0;
+
+    return d;
+
+exit_da_created:
+    free (d);
+    return NULL;
+}
+
+DArray *
+da_read (FILE *file)
+{
+    long        save_pos;
+    DArray     *d = NULL;
+    TrieIndex   n;
+
+    /* check signature */
+    save_pos = ftell (file);
+    if (!file_read_int32 (file, &n) || DA_SIGNATURE != (uint32) n) {
+        fseek (file, save_pos, SEEK_SET);
+        return NULL;
+    }
+
+    d = (DArray *) malloc (sizeof (DArray));
+    if (!d)
+        return NULL;
+
+    /* read number of cells */
+    file_read_int32 (file, &d->num_cells);
+    d->cells     = (DACell *) malloc (d->num_cells * sizeof (DACell));
+    if (!d->cells)
+        goto exit_da_created;
+    d->cells[0].base = DA_SIGNATURE;
+    d->cells[0].check= d->num_cells;
+    for (n = 1; n < d->num_cells; n++) {
+        file_read_int32 (file, &d->cells[n].base);
+        file_read_int32 (file, &d->cells[n].check);
+    }
+
+    return d;
+
+exit_da_created:
+    free (d);
+    return NULL;
+}
+
+void
+da_free (DArray *d)
+{
+    free (d->cells);
+    free (d);
+}
+
+int
+da_write (const DArray *d, FILE *file)
+{
+    TrieIndex   i;
+
+    for (i = 0; i < d->num_cells; i++) {
+        if (!file_write_int32 (file, d->cells[i].base) ||
+            !file_write_int32 (file, d->cells[i].check))
+        {
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+
+TrieIndex
+da_get_root (const DArray *d)
+{
+    /* can be calculated value for multi-index trie */
+    return 2;
+}
+
+
+TrieIndex
+da_get_base (const DArray *d, TrieIndex s)
+{
+    return (0 <= s && s < d->num_cells) ? d->cells[s].base : TRIE_INDEX_ERROR;
+}
+
+TrieIndex
+da_get_check (const DArray *d, TrieIndex s)
+{
+    return (0 <= s && s < d->num_cells) ? d->cells[s].check : TRIE_INDEX_ERROR;
+}
+
+
+void
+da_set_base (DArray *d, TrieIndex s, TrieIndex val)
+{
+    if (0 <= s && s < d->num_cells) {
+        d->cells[s].base = val;
+    }
+}
+
+void
+da_set_check (DArray *d, TrieIndex s, TrieIndex val)
+{
+    if (0 <= s && s < d->num_cells) {
+        d->cells[s].check = val;
+    }
+}
+
+Bool
+da_walk (const DArray *d, TrieIndex *s, TrieChar c)
+{
+    TrieIndex   next;
+
+    next = da_get_base (d, *s) + c;
+    if (da_get_check (d, next) == *s) {
+        *s = next;
+        return TRUE;
+    }
+    return FALSE;
+}
+
+TrieIndex
+da_insert_branch (DArray *d, TrieIndex s, TrieChar c)
+{
+    TrieIndex   base, next;
+
+    base = da_get_base (d, s);
+
+    if (base > 0) {
+        next = base + c;
+
+        /* if already there, do not actually insert */
+        if (da_get_check (d, next) == s)
+            return next;
+
+        /* if (base + c) > TRIE_INDEX_MAX which means 'next' is overflow,
+         * or cell [next] is not free, relocate to a free slot
+         */
+        if (base > TRIE_INDEX_MAX - c || !da_check_free_cell (d, next)) {
+            Symbols    *symbols;
+            TrieIndex   new_base;
+
+            /* relocate BASE[s] */
+            symbols = da_output_symbols (d, s);
+            symbols_add (symbols, c);
+            new_base = da_find_free_base (d, symbols);
+            symbols_free (symbols);
+
+            if (TRIE_INDEX_ERROR == new_base)
+                return TRIE_INDEX_ERROR;
+
+            da_relocate_base (d, s, new_base);
+            next = new_base + c;
+        }
+    } else {
+        Symbols    *symbols;
+        TrieIndex   new_base;
+
+        symbols = symbols_new ();
+        symbols_add (symbols, c);
+        new_base = da_find_free_base (d, symbols);
+        symbols_free (symbols);
+
+        if (TRIE_INDEX_ERROR == new_base)
+            return TRIE_INDEX_ERROR;
+
+        da_set_base (d, s, new_base);
+        next = new_base + c;
+    }
+    da_alloc_cell (d, next);
+    da_set_check (d, next, s);
+
+    return next;
+}
+
+static Bool
+da_check_free_cell (DArray         *d,
+                    TrieIndex       s)
+{
+    return da_extend_pool (d, s) && da_get_check (d, s) < 0;
+}
+
+static Bool
+da_has_children    (DArray         *d,
+                    TrieIndex       s)
+{
+    TrieIndex   base;
+    TrieIndex   c, max_c;
+
+    base = da_get_base (d, s);
+    if (TRIE_INDEX_ERROR == base || base < 0)
+        return FALSE;
+
+    max_c = MIN_VAL (TRIE_CHAR_MAX, TRIE_INDEX_MAX - base);
+    for (c = 0; c < max_c; c++) {
+        if (da_get_check (d, base + c) == s)
+            return TRUE;
+    }
+
+    return FALSE;
+}
+
+static Symbols *
+da_output_symbols  (const DArray   *d,
+                    TrieIndex       s)
+{
+    Symbols    *syms;
+    TrieIndex   base;
+    TrieIndex   c, max_c;
+
+    syms = symbols_new ();
+
+    base = da_get_base (d, s);
+    max_c = MIN_VAL (TRIE_CHAR_MAX, TRIE_INDEX_MAX - base);
+    for (c = 0; c < max_c; c++) {
+        if (da_get_check (d, base + c) == s)
+            symbols_add_fast (syms, (TrieChar) c);
+    }
+
+    return syms;
+}
+
+static TrieChar *
+da_get_state_key   (const DArray   *d,
+                    TrieIndex       state)
+{
+    TrieChar   *key;
+    int         key_size, key_length;
+    int         i;
+
+    key_size = 20;
+    key_length = 0;
+    key = (TrieChar *) malloc (key_size);
+
+    /* trace back to root */
+    while (da_get_root (d) != state) {
+        TrieIndex   parent;
+
+        if (key_length + 1 >= key_size) {
+            key_size += 20;
+            key = (TrieChar *) realloc (key, key_size);
+        }
+        parent = da_get_check (d, state);
+        key[key_length++] = (TrieChar) (state - da_get_base (d, parent));
+        state = parent;
+    }
+    key[key_length] = '\0';
+
+    /* reverse the string */
+    for (i = 0; i < --key_length; i++) {
+        TrieChar temp;
+
+        temp = key[i];
+        key[i] = key[key_length];
+        key[key_length] = temp;
+    }
+
+    return key;
+}
+
+static TrieIndex
+da_find_free_base  (DArray         *d,
+                    const Symbols  *symbols)
+{
+    TrieChar        first_sym;
+    TrieIndex       s;
+
+    /* find first free cell that is beyond the first symbol */
+    first_sym = symbols_get (symbols, 0);
+    s = -da_get_check (d, da_get_free_list (d));
+    while (s != da_get_free_list (d)
+           && s < (TrieIndex) first_sym + DA_POOL_BEGIN)
+    {
+        s = -da_get_check (d, s);
+    }
+    if (s == da_get_free_list (d)) {
+        for (s = first_sym + DA_POOL_BEGIN; ; ++s) {
+            if (!da_extend_pool (d, s))
+                return TRIE_INDEX_ERROR;
+            if (da_get_check (d, s) < 0)
+                break;
+        }
+    }
+
+    /* search for next free cell that fits the symbols set */
+    while (!da_fit_symbols (d, s - first_sym, symbols)) {
+        /* extend pool before getting exhausted */
+        if (-da_get_check (d, s) == da_get_free_list (d)) {
+            if (!da_extend_pool (d, d->num_cells))
+                return TRIE_INDEX_ERROR;
+        }
+
+        s = -da_get_check (d, s);
+    }
+
+    return s - first_sym;
+}
+
+static Bool
+da_fit_symbols     (DArray         *d,
+                    TrieIndex       base,
+                    const Symbols  *symbols)
+{
+    int         i;
+
+    for (i = 0; i < symbols_num (symbols); i++) {
+        TrieChar    sym = symbols_get (symbols, i);
+
+        /* if (base + sym) > TRIE_INDEX_MAX which means it's overflow,
+         * or cell [base + sym] is not free, the symbol is not fit.
+         */
+        if (base > TRIE_INDEX_MAX - sym || !da_check_free_cell (d, base + sym))
+            return FALSE;
+    }
+    return TRUE;
+}
+
+static void
+da_relocate_base   (DArray         *d,
+                    TrieIndex       s,
+                    TrieIndex       new_base)
+{
+    TrieIndex   old_base;
+    Symbols    *symbols;
+    int         i;
+
+    old_base = da_get_base (d, s);
+    symbols = da_output_symbols (d, s);
+
+    for (i = 0; i < symbols_num (symbols); i++) {
+        TrieIndex   old_next, new_next, old_next_base;
+
+        old_next = old_base + symbols_get (symbols, i);
+        new_next = new_base + symbols_get (symbols, i);
+        old_next_base = da_get_base (d, old_next);
+
+        /* allocate new next node and copy BASE value */
+        da_alloc_cell (d, new_next);
+        da_set_check (d, new_next, s);
+        da_set_base (d, new_next, old_next_base);
+
+        /* old_next node is now moved to new_next
+         * so, all cells belonging to old_next
+         * must be given to new_next
+         */
+        /* preventing the case of TAIL pointer */
+        if (old_next_base > 0) {
+            TrieIndex   c, max_c;
+
+            max_c = MIN_VAL (TRIE_CHAR_MAX, TRIE_INDEX_MAX - old_next_base);
+            for  (c = 0; c < max_c; c++) {
+                if (da_get_check (d, old_next_base + c) == old_next)
+                    da_set_check (d, old_next_base + c, new_next);
+            }
+        }
+
+        /* free old_next node */
+        da_free_cell (d, old_next);
+    }
+
+    symbols_free (symbols);
+
+    /* finally, make BASE[s] point to new_base */
+    da_set_base (d, s, new_base);
+}
+
+static Bool
+da_extend_pool     (DArray         *d,
+                    TrieIndex       to_index)
+{
+    TrieIndex   new_begin;
+    TrieIndex   i;
+    TrieIndex   free_tail;
+
+    if (to_index <= 0 || TRIE_INDEX_MAX <= to_index)
+        return FALSE;
+
+    if (to_index < d->num_cells)
+        return TRUE;
+
+    d->cells = (DACell *) realloc (d->cells, (to_index + 1) * sizeof (DACell));
+    new_begin = d->num_cells;
+    d->num_cells = to_index + 1;
+
+    /* initialize new free list */
+    for (i = new_begin; i < to_index; i++) {
+        da_set_check (d, i, -(i + 1));
+        da_set_base (d, i + 1, -i);
+    }
+
+    /* merge the new circular list to the old */
+    free_tail = -da_get_base (d, da_get_free_list (d));
+    da_set_check (d, free_tail, -new_begin);
+    da_set_base (d, new_begin, -free_tail);
+    da_set_check (d, to_index, -da_get_free_list (d));
+    da_set_base (d, da_get_free_list (d), -to_index);
+
+    /* update header cell */
+    d->cells[0].check = d->num_cells;
+
+    return TRUE;
+}
+
+void
+da_prune (DArray *d, TrieIndex s)
+{
+    da_prune_upto (d, da_get_root (d), s);
+}
+
+void
+da_prune_upto (DArray *d, TrieIndex p, TrieIndex s)
+{
+    while (p != s && !da_has_children (d, s)) {
+        TrieIndex   parent;
+
+        parent = da_get_check (d, s);
+        da_free_cell (d, s);
+        s = parent;
+    }
+}
+
+static void
+da_alloc_cell      (DArray         *d,
+                    TrieIndex       cell)
+{
+    TrieIndex   prev, next;
+
+    prev = -da_get_base (d, cell);
+    next = -da_get_check (d, cell);
+
+    /* remove the cell from free list */
+    da_set_check (d, prev, -next);
+    da_set_base (d, next, -prev);
+}
+
+static void
+da_free_cell       (DArray         *d,
+                    TrieIndex       cell)
+{
+    TrieIndex   i, prev;
+
+    /* find insertion point */
+    i = -da_get_check (d, da_get_free_list (d));
+    while (i != da_get_free_list (d) && i < cell)
+        i = -da_get_check (d, i);
+
+    prev = -da_get_base (d, i);
+
+    /* insert cell before i */
+    da_set_check (d, cell, -i);
+    da_set_base (d, cell, -prev);
+    da_set_check (d, prev, -cell);
+    da_set_base (d, i, -cell);
+}
+
+Bool
+da_enumerate (const DArray *d, DAEnumFunc enum_func, void *user_data)
+{
+    return da_enumerate_recursive (d, da_get_root (d), enum_func, user_data);
+}
+
+static Bool
+da_enumerate_recursive (const DArray   *d,
+                        TrieIndex       state,
+                        DAEnumFunc      enum_func,
+                        void           *user_data)
+{
+    Bool        ret;
+    TrieIndex   base;
+
+    base = da_get_base (d, state);
+
+    if (base < 0) {
+        TrieChar   *key;
+
+        key = da_get_state_key (d, state);
+        ret = (*enum_func) (key, state, user_data);
+        free (key);
+    } else {
+        Symbols *symbols;
+        int      i;
+
+        ret = TRUE;
+        symbols = da_output_symbols (d, state);
+        for (i = 0; ret && i < symbols_num (symbols); i++) {
+            ret = da_enumerate_recursive (d, base + symbols_get (symbols, i),
+                                          enum_func, user_data);
+        }
+
+        symbols_free (symbols);
+    }
+
+    return ret;
+}
+
+/*
+vi:ts=4:ai:expandtab
+*/