isomorfeus-ferret 0.13.6 → 0.13.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 524b384c2ab35457603726287160b443a50939473bab7852e5e6f5f886ded0a7
-  data.tar.gz: b304d0386bc0b14e68167bfcc1b62346f81c02f6eff25c3a6633bb0a1919083e
+  metadata.gz: 744efe9c78deef0c563e39ae29eb4e755f8afb493be0fe80416a55987db97ce7
+  data.tar.gz: d1f814f14b5a6ef18f612e545b54b0a113544f994c22901e7204ae191ebb644e
 SHA512:
-  metadata.gz: b378ce12aa56bb35e57eec2c8df64152a7a7bd5cb1345d4a7dc0264e236fddcea3ffbc41a7213401cf50ff163c3172169fbf93699a4a7f96505f300782bc9a92
-  data.tar.gz: 6954e297df285f7726bd925a6f4dda4300f06d059dc4b674a494e775b901f7ea9e529e62be8793049c6df20a17515c41741f63155fbdd661de4595694deb4c27
+  metadata.gz: 3fd11e3b3f0f04625c14b9b709d69ef6f1a1abdfdf5c50ad5e151a3fd87e300595e29639ac9455c66f8db5452d9569bb1585f6b2f9f57ac5c3103f2ff09bb8d1
+  data.tar.gz: 27ace1a41fcc7557490865e740dfbe9906ce999288bb31306efa9b02f519552f9092391ff64a3ededc6a2285a0e78533a80d43ad48e85abc97c1a154e0060a67

data/README.md

@@ -11,12 +11,12 @@ At the [Isomorfeus Framework Project](https://isomorfeus.com)
 
 ## About this project
 
-Isomorfeus-Ferret is a revived version of the original ferret gem created by Dave Balmain.
+Isomorfeus-Ferret is a revived version of the original ferret gem created by Dave Balmain, [https://github.com/dbalmain/ferret](https://github.com/dbalmain/ferret).
 During revival many things havbe been fixed, now all tests pass, no crashes and it
 successfully compiles and runs with rubys >3. Its no longer a goal to have
 a c library available, but instead the usage is meant as ruby gem with a c extension only.
 
-It should work on *nixes, *nuxes, *BSDs and also works on Windows.
+It works on *nixes, *nuxes, *BSDs and also works on Windows and RaspberryPi.
 
 ## Improvements and Changes in Version 0.13
 
@@ -63,14 +63,8 @@ fis.add_field(:compressed_field, :store => :yes, :compression => :brotli, :term_
 
 ### Performance
 
-The encoding support demands its toll, indexing performance dropped a bit in comparision to 0.12, but still thousands of docs per second, depending on machine/docs.
-On Windows the indexing performance is still terrible, but that may be resolved in a future project.
-
-Search performance is still excellent and multiple times faster than Lucene.
-
-Lucene achieves roughly double the indexing performance. This seems to be because of the different way strings and
-encodings are handled in Java. For example, the Java WhitespaceTokenizer code requires only one method call per character (check for whitespace), but for Ruby, to support all the different encodings, several method calls are required per character (retrieve character according to encoding, check character for whitespace).
-Ferret is internally using the standard Ruby string encoding methods.
+For version 0.13.7 the performance bottle neck has been identified and removed, ferret now delivers excellent indexing perfomance on all platforms, see numbers below.
+On Windows performance is still not as good as on Linux, but that is equally true for Lucene and because of how the Windows filesystem works.
 
 ## Documentation
 
@@ -105,43 +99,41 @@ Ensure your locale is set to C.UTF-8, because the internal c tests don't know ho
 
 A recent Java JDK must be installed to compile and run lucene benchmarks.
 
-Results on Linux:
-```
-Ferret 0.13.0:
-Indexing: 9.35 secs, Docs: 19043, 2035 docs/s
-Searching took: 0.3133133s for 8000 queries
-thats 25533 q/s
-Total found: 42000
-Index size: 28Mb
-
-Lucene 9.1.0:
-Indexing: 4.20 secs, Docs: 19043, 4538 docs/s
-Searching took: 1.64s for 8000 queries
-thats 4875 q/s
-Total found: 41000
-index size: 35Mb
-
-JVM 11.0.14.1 (Ubuntu)
-```
+Results, Ferret 0.13.7 vs. Lucene 9.1.0, WhitespaceAnalyzer,
+Linux Ubuntu 20.04, FreeBSD 13.0 and Windows 10 on old Intel Core i5 from 2015,
+LinuxPi on RaspberryPi 400:
+
+| OS      | Task       | Ferret          | Lucene*        |
+|---------|------------|-----------------|----------------|
+| Linux   | Indexing   |     4905 docs/s |    4785 docs/s |
+| FreeBSD | Indexing   |     4516 docs/s |       -        |
+| Windows | Indexing   |     2361 docs/s |    2395 docs/s |
+| LinuxPi | Indexing   |     1161 docs/s |     707 docs/s |
+| Linux   | Searching  | 25664 queries/s | 4708 queries/s |
+| FreeBSD | Searching  | 25073 queries/s |    -           |
+| Windows | Searching  |  3646 queries/s |  935 queries/s |
+| LinuxPi | Searching  |  5768 queries/s |  680 queries/s |
+|         | Index Size |           28 MB |          35 MB |
+
+*Lucene 9.1.0 on JVM 11.0.14.1 (Ubuntu)
 
 ### Storing Fields with Compression, Indexing and Retrieval
 - clone repo
 - bundle install
 - rake ferret_compression_benchmark
 
-Results on Linux, 0.13.0:
+Results on Linux, 0.13.7, on old Intel Core i5 from 2015:
 
 | Compression | Index & Store | Retrieve      | Index size |
 |-------------|---------------|---------------|------------|
-| none        |   2008 docs/s | 153853 docs/s |      43 MB |
-| brotli      |   1726 docs/s |  58315 docs/s |      36 MB |
-| bzip2       |   1438 docs/s |  15382 docs/s |      38 MB |
-| lz4         |   1932 docs/s | 127100 docs/s |      41 MB |
+| none        |   4866 docs/s | 153853 docs/s |      43 MB |
+| brotli      |   3539 docs/s |  58315 docs/s |      36 MB |
+| bzip2       |   2624 docs/s |  15382 docs/s |      38 MB |
+| lz4         |   4639 docs/s | 127100 docs/s |      41 MB |
 
 ## Future
 
 Lots of things to do:
-- Improve indexing performance on Windows (WriteFile is terribly slow, maybe use mapping, see libuv)
 - Bring documentation in order in a docs directory
 - Review code (especially for memory/stack issues, typical c issues)
 - Take care of ruby GVL and threading

data/ext/isomorfeus_ferret_ext/frb_analysis.c

@@ -458,7 +458,7 @@ static VALUE frb_ts_next(VALUE self) {
 
 static void frb_tf_mark(void *p) {
     FrtTokenStream *ts = (FrtTokenStream *)p;
-    if (TkFilt(ts)->sub_ts->rts)
+    if (TkFilt(ts)->sub_ts && TkFilt(ts)->sub_ts->rts)
         rb_gc_mark(TkFilt(ts)->sub_ts->rts);
 }
 
@@ -1331,9 +1331,10 @@ static void frb_h_mark_values_i(void *key, void *value, void *arg) {
 }
 
 static void frb_pfa_mark(void *p) {
-    if (PFA(p)->default_a->ranalyzer)
+    if (PFA(p)->default_a && PFA(p)->default_a->ranalyzer)
         rb_gc_mark(PFA(p)->default_a->ranalyzer);
-    frt_h_each(PFA(p)->dict, &frb_h_mark_values_i, NULL);
+    if (PFA(p)->dict)
+        frt_h_each(PFA(p)->dict, &frb_h_mark_values_i, NULL);
 }
 
 /*** PerFieldAnalyzer ***/
@@ -1370,6 +1371,8 @@ const rb_data_type_t frb_per_field_analyzer_t = {
 
 static VALUE frb_per_field_analyzer_alloc(VALUE rclass) {
     FrtAnalyzer *a = frt_per_field_analyzer_alloc();
+    PFA(a)->default_a = NULL;
+    PFA(a)->dict = NULL;
     return TypedData_Wrap_Struct(rclass, &frb_per_field_analyzer_t, a);
 }
 
@@ -1435,7 +1438,7 @@ static VALUE frb_pfa_analyzer_token_stream(VALUE self, VALUE rfield, VALUE rstri
 /*** RegExpAnalyzer ***/
 
 static void frb_re_analyzer_mark(void *p) {
-    if (((FrtAnalyzer *)p)->current_ts->rts)
+    if (((FrtAnalyzer *)p)->current_ts && ((FrtAnalyzer *)p)->current_ts->rts)
         rb_gc_mark(((FrtAnalyzer *)p)->current_ts->rts);
 }
 

data/ext/isomorfeus_ferret_ext/frb_index.c

@@ -2,7 +2,7 @@
 #include "isomorfeus_ferret.h"
 #include <ruby.h>
 
-#undef close
+// #undef close
 
 VALUE mIndex;
 
@@ -429,6 +429,7 @@ static VALUE frb_get_field_infos(FrtFieldInfos *fis) {
 
 static VALUE frb_fis_alloc(VALUE rclass) {
     FrtFieldInfos *fis = frt_fis_alloc();
+    fis->size = 0;
     return TypedData_Wrap_Struct(rclass, &frb_field_infos_t, fis);
 }
 
@@ -535,7 +536,7 @@ frb_fis_add_field(int argc, VALUE *argv, VALUE self)
 {
     FrtFieldInfos *fis = (FrtFieldInfos *)DATA_PTR(self);
     FrtFieldInfo *fi;
-    FrtStoreValue store = fis->store;
+    FrtStoreValue store_val = fis->store_val;
     FrtCompressionType compression = fis->compression;
     FrtIndexValue index = fis->index;
     FrtTermVectorValue term_vector = fis->term_vector;
@@ -544,9 +545,9 @@ frb_fis_add_field(int argc, VALUE *argv, VALUE self)
 
     rb_scan_args(argc, argv, "11", &rname, &roptions);
     if (argc > 1) {
-        frb_fi_get_params(roptions, &store, &compression, &index, &term_vector, &boost);
+        frb_fi_get_params(roptions, &store_val, &compression, &index, &term_vector, &boost);
     }
-    fi = frt_fi_new(frb_field(rname), store, compression, index, term_vector);
+    fi = frt_fi_new(frb_field(rname), store_val, compression, index, term_vector);
     fi->boost = boost;
     frt_fis_add_field(fis, fi);
     return self;
@@ -592,9 +593,7 @@ frb_fis_to_s(VALUE self)
  *
  *  Return the number of fields in the FieldInfos object.
  */
-static VALUE
-frb_fis_size(VALUE self)
-{
+static VALUE frb_fis_size(VALUE self) {
     FrtFieldInfos *fis = (FrtFieldInfos *)DATA_PTR(self);
     return INT2FIX(fis->size);
 }
@@ -609,21 +608,19 @@ frb_fis_size(VALUE self)
  *  existing index (or other files for that matter) will be deleted from the
  *  directory and overwritten by the new index.
  */
-static VALUE
-frb_fis_create_index(VALUE self, VALUE rdir)
-{
+static VALUE frb_fis_create_index(VALUE self, VALUE rdir) {
     FrtFieldInfos *fis = (FrtFieldInfos *)DATA_PTR(self);
     FrtStore *store = NULL;
     if (TYPE(rdir) == T_DATA) {
         store = DATA_PTR(rdir);
-        FRT_REF(store);
+        frt_index_create(store, fis);
     } else {
         StringValue(rdir);
         frb_create_dir(rdir);
         store = frt_open_fs_store(rs2s(rdir));
+        frt_index_create(store, fis);
+        frt_store_close(store);
     }
-    frt_index_create(store, fis);
-    frt_store_deref(store);
     return self;
 }
 
@@ -674,8 +671,7 @@ frb_fis_get_tk_fields(VALUE self)
  ****************************************************************************/
 
 static void frb_te_free(void *p) {
-    FrtTermEnum *te = (FrtTermEnum *)p;
-    te->close(te);
+    ((FrtTermEnum *)p)->close((FrtTermEnum *)p);
 }
 
 static size_t frb_te_size(const void *p) {
@@ -1344,6 +1340,8 @@ static VALUE frb_iw_alloc(VALUE rclass) {
     return TypedData_Wrap_Struct(rclass, &frb_index_writer_t, iw);
 }
 
+extern rb_data_type_t frb_store_t;
+
 static VALUE frb_iw_init(int argc, VALUE *argv, VALUE self) {
     VALUE roptions, rval;
     bool create = false;
@@ -1363,12 +1361,11 @@ static VALUE frb_iw_init(int argc, VALUE *argv, VALUE self) {
 
             if ((rval = rb_hash_aref(roptions, sym_dir)) != Qnil) {
                 // Check_Type(rval, T_DATA);
-                store = DATA_PTR(rval);
+                TypedData_Get_Struct(rval, FrtStore, &frb_store_t, store);
             } else if ((rval = rb_hash_aref(roptions, sym_path)) != Qnil) {
                 StringValue(rval);
                 frb_create_dir(rval);
                 store = frt_open_fs_store(rs2s(rval));
-                FRT_DEREF(store);
             }
             /* use_compound_file defaults to true */
             config.use_compound_file =
@@ -1393,7 +1390,6 @@ static VALUE frb_iw_init(int argc, VALUE *argv, VALUE self) {
         }
         if (NULL == store) {
             store = frt_open_ram_store(NULL);
-            FRT_DEREF(store);
         }
         if (!create && create_if_missing && !store->exists(store, "segments")) {
             create = true;
@@ -1411,7 +1407,7 @@ static VALUE frb_iw_init(int argc, VALUE *argv, VALUE self) {
         }
 
         TypedData_Get_Struct(self, FrtIndexWriter, &frb_index_writer_t, iw);
-        iw = frt_iw_open(iw, store, analyzer, &config);
+        frt_iw_open(iw, store, analyzer, &config);
     FRT_XCATCHALL
         ex_code = xcontext.excode;
         msg = xcontext.msg;
@@ -1598,9 +1594,9 @@ void frb_ir_free(void *p) {
 
 void frb_ir_mark(void *p) {
     FrtIndexReader *ir = (FrtIndexReader *)p;
-    FrtMultiReader *mr = (FrtMultiReader *)p;
 
     if (ir->type == FRT_MULTI_READER) {
+        FrtMultiReader *mr = (FrtMultiReader *)p;
         int i;
         for (i = 0; i < mr->r_cnt; i++) {
             if (mr->sub_readers[i]->rir)
@@ -1652,6 +1648,7 @@ static VALUE frb_iw_add_readers(VALUE self, VALUE rreaders) {
     while (i-- > 0) {
         FrtIndexReader *ir;
         TypedData_Get_Struct(RARRAY_PTR(rreaders)[i], FrtIndexReader, &frb_index_reader_t, ir);
+        FRT_REF(ir);
         irs[i] = ir;
     }
     frt_iw_add_readers(iw, irs, RARRAY_LEN(rreaders));
@@ -2157,7 +2154,6 @@ static VALUE frb_ir_init(VALUE self, VALUE rdir) {
     VALUE rfield_num_map = rb_hash_new();
     int ex_code = 0;
     const char *msg = NULL;
-
     FRT_TRY
         if (TYPE(rdir) == T_ARRAY) {
             VALUE rdirs = rdir;
@@ -2170,7 +2166,6 @@ static VALUE frb_ir_init(VALUE self, VALUE rdir) {
                     case T_DATA:
                         if (CLASS_OF(rdir) == cIndexReader) {
                             TypedData_Get_Struct(rdir, FrtIndexReader, &frb_index_reader_t, sub_readers[i]);
-                            FRT_REF(sub_readers[i]);
                             continue;
                         } else if (RTEST(rb_obj_is_kind_of(rdir, cDirectory))) {
                             store = DATA_PTR(rdir);
@@ -2185,7 +2180,6 @@ static VALUE frb_ir_init(VALUE self, VALUE rdir) {
                     case T_STRING:
                         frb_create_dir(rdir);
                         store = frt_open_fs_store(rs2s(rdir));
-                        FRT_DEREF(store);
                         break;
                     default:
                         FRT_RAISE(FRT_ARG_ERROR, "%s isn't a valid directory "
@@ -2195,6 +2189,7 @@ static VALUE frb_ir_init(VALUE self, VALUE rdir) {
                         break;
                 }
                 sub_readers[i] = frt_ir_open(NULL, store);
+                FRT_DEREF(sub_readers[i]);
             }
             TypedData_Get_Struct(self, FrtIndexReader, &frb_index_reader_t, ir);
             ir = frt_mr_open(ir, sub_readers, reader_cnt);
@@ -2206,7 +2201,6 @@ static VALUE frb_ir_init(VALUE self, VALUE rdir) {
                 case T_STRING:
                     frb_create_dir(rdir);
                     store = frt_open_fs_store(rs2s(rdir));
-                    FRT_DEREF(store);
                     break;
                 default:
                     FRT_RAISE(FRT_ARG_ERROR, "%s isn't a valid directory argument. "

data/ext/isomorfeus_ferret_ext/frb_qparser.c

@@ -34,7 +34,7 @@ static void frb_qp_free(void *p) {
 }
 
 static void frb_qp_mark(void *p) {
-    if (((FrtQParser *)p)->analyzer->ranalyzer)
+    if (((FrtQParser *)p)->analyzer && ((FrtQParser *)p)->analyzer->ranalyzer)
         rb_gc_mark(((FrtQParser *)p)->analyzer->ranalyzer);
 }
 
@@ -59,6 +59,7 @@ const rb_data_type_t frb_qp_t = {
 
 static VALUE frb_qp_alloc(VALUE rclass) {
     FrtQParser *qp = frt_qp_alloc();
+    qp->analyzer = NULL;
     return TypedData_Wrap_Struct(rclass, &frb_qp_t, qp);
 }
 

data/ext/isomorfeus_ferret_ext/frb_search.c

@@ -4,7 +4,7 @@
 #include "isomorfeus_ferret.h"
 #include <ruby.h>
 
-#undef close
+// #undef close
 
 VALUE mSearch;
 
@@ -713,7 +713,7 @@ static size_t frb_boolean_clause_t_size(const void *p) {
 }
 
 static void frb_bc_mark(void *p) {
-    if (((FrtBooleanClause *)p)->query->rquery)
+    if (((FrtBooleanClause *)p)->query && ((FrtBooleanClause *)p)->query->rquery)
         rb_gc_mark(((FrtBooleanClause *)p)->query->rquery);
 }
 
@@ -1837,11 +1837,11 @@ static size_t frb_filtered_query_size(const void *p) {
 }
 
 static void frb_fqq_mark(void *p) {
-    FrtFilteredQuery *fq = (FrtFilteredQuery *)p;
-    if (fq->query->rquery)
-        rb_gc_mark(fq->query->rquery);
-    if (fq->filter->rfilter)
-        rb_gc_mark(fq->filter->rfilter);
+    FrtFilteredQuery *fqq = (FrtFilteredQuery *)p;
+    if (fqq->query && fqq->query->rquery)
+        rb_gc_mark(fqq->query->rquery);
+    if (fqq->filter && fqq->filter->rfilter)
+        rb_gc_mark(fqq->filter->rfilter);
 }
 
 const rb_data_type_t frb_filtered_query_t = {
@@ -1860,6 +1860,8 @@ const rb_data_type_t frb_filtered_query_t = {
 
 static VALUE frb_fqq_alloc(VALUE rclass) {
     FrtQuery *fqq = frt_fq_alloc();
+    ((FrtFilteredQuery *)fqq)->query = NULL;
+    ((FrtFilteredQuery *)fqq)->filter = NULL;
     return TypedData_Wrap_Struct(rclass, &frb_filtered_query_t, fqq);
 }
 
@@ -2172,6 +2174,7 @@ const rb_data_type_t frb_span_near_query_t = {
 
 static VALUE frb_spannq_alloc(VALUE rclass) {
     FrtQuery *snq = frt_spannq_alloc();
+    ((FrtSpanNearQuery *)snq)->c_cnt = 0;
     return TypedData_Wrap_Struct(rclass, &frb_span_near_query_t, snq);
 }
 
@@ -2352,9 +2355,9 @@ static size_t frb_span_not_query_size(const void *p) {
 
 static void frb_spanxq_mark(void *p) {
     FrtSpanNotQuery *sxq = (FrtSpanNotQuery *)p;
-    if (sxq->inc->rquery)
+    if (sxq->inc && sxq->inc->rquery)
         rb_gc_mark(sxq->inc->rquery);
-    if (sxq->exc->rquery)
+    if (sxq->exc && sxq->exc->rquery)
         rb_gc_mark(sxq->exc->rquery);
 }
 
@@ -3557,10 +3560,12 @@ static size_t frb_index_searcher_size(const void *p) {
 
 static void frb_sea_mark(void *p) {
     FrtIndexSearcher *isea = (FrtIndexSearcher *)p;
-    if (isea->ir->rir)
-        rb_gc_mark(isea->ir->rir);
-    if (isea->ir->store->rstore)
-        rb_gc_mark(isea->ir->store->rstore);
+    if (isea->ir) {
+        if (isea->ir->rir)
+            rb_gc_mark(isea->ir->rir);
+        if (isea->ir->store && isea->ir->store->rstore)
+            rb_gc_mark(isea->ir->store->rstore);
+    }
 }
 
 const rb_data_type_t frb_index_searcher_t = {
@@ -3578,8 +3583,9 @@ const rb_data_type_t frb_index_searcher_t = {
 };
 
 static VALUE frb_sea_alloc(VALUE rclass) {
-    FrtSearcher *s = frt_isea_alloc();
-    return TypedData_Wrap_Struct(rclass, &frb_index_searcher_t, s);
+    FrtSearcher *sea = frt_isea_alloc();
+    ((FrtIndexSearcher *)sea)->ir = NULL;
+    return TypedData_Wrap_Struct(rclass, &frb_index_searcher_t, sea);
 }
 
 /*
@@ -3600,7 +3606,6 @@ static VALUE frb_sea_init(VALUE self, VALUE obj) {
         frb_create_dir(obj);
         store = frt_open_fs_store(rs2s(obj));
         ir = frt_ir_open(NULL, store);
-        FRT_DEREF(store);
         ir->rir = TypedData_Wrap_Struct(cIndexReader, &frb_index_reader_t, ir);
     } else {
         // Check_Type(obj, T_DATA);
@@ -3610,14 +3615,12 @@ static VALUE frb_sea_init(VALUE self, VALUE obj) {
             ir->rir = TypedData_Wrap_Struct(cIndexReader, &frb_index_reader_t, ir);
         } else if (rb_obj_is_kind_of(obj, cIndexReader) == Qtrue) {
             TypedData_Get_Struct(obj, FrtIndexReader, &frb_index_reader_t, ir);
-            ir->rir = obj;
         } else {
             rb_raise(rb_eArgError, "Unknown type for argument to IndexSearcher.new");
         }
     }
     TypedData_Get_Struct(self, FrtSearcher, &frb_index_searcher_t, sea);
     frt_isea_init(sea, ir);
-    ((FrtIndexSearcher *)sea)->close_ir = false;
     sea->rsea = self;
     return self;
 }
@@ -3665,6 +3668,7 @@ const rb_data_type_t frb_multi_searcher_t = {
 
 static VALUE frb_ms_alloc(VALUE rclass) {
     FrtSearcher *s = frt_msea_alloc();
+    ((FrtMultiSearcher *)s)->s_cnt = 0;
     return TypedData_Wrap_Struct(rclass, &frb_multi_searcher_t, s);
 }
 
@@ -3703,7 +3707,7 @@ static VALUE frb_ms_init(int argc, VALUE *argv, VALUE self) {
         }
     }
     TypedData_Get_Struct(self, FrtSearcher, &frb_multi_searcher_t, sea);
-    frt_msea_init(sea, searchers, top, false);
+    frt_msea_init(sea, searchers, top);
     sea->rsea = self;
     return self;
 }

data/ext/isomorfeus_ferret_ext/frb_store.c

@@ -2,7 +2,7 @@
 #include "isomorfeus_ferret.h"
 #include <time.h>
 
-#undef rename
+// #undef rename
 
 static ID id_ref_cnt;
 VALUE cLock;
@@ -34,7 +34,7 @@ void frb_lock_free(void *p) {
 
 void frb_lock_mark(void *p) {
     FrtLock *lock = (FrtLock *)p;
-    if (lock->store->rstore)
+    if (lock->store && lock->store->rstore)
         rb_gc_mark(lock->store->rstore);
 }
 
@@ -185,10 +185,35 @@ static VALUE frb_lock_release(VALUE self) {
  *
  ****************************************************************************/
 
+/*** FrbStore ****************************************************************/
+
+static size_t frb_store_size(const void *p) {
+    return sizeof(FrtStore);
+    (void)p;
+}
+
 void frb_dir_free(void *p) {
-    FrtStore *store = (FrtStore *)p;
-    frb_unwrap_locks(store);
-    frt_store_deref(store);
+    frb_unwrap_locks((FrtStore *)p);
+    frt_store_close((FrtStore *)p);
+}
+
+const rb_data_type_t frb_store_t = {
+    .wrap_struct_name = "FrbStore",
+    .function = {
+        .dmark = NULL,
+        .dfree = frb_dir_free,
+        .dsize = frb_store_size,
+        .dcompact = NULL,
+        .reserved = {0},
+    },
+    .parent = NULL,
+    .data = NULL,
+    .flags = RUBY_TYPED_FREE_IMMEDIATELY
+};
+
+static VALUE frb_store_alloc(VALUE rclass) {
+    FrtStore *st = frt_store_alloc();
+    return TypedData_Wrap_Struct(rclass, &frb_store_t, st);
 }
 
 /*
@@ -208,7 +233,7 @@ static VALUE frb_dir_close(VALUE self) {
         ((struct RData *)(self))->dmark = NULL;
         ((struct RData *)(self))->dfree = NULL;
         frb_unwrap_locks(store);
-        frt_store_deref(store);
+        frt_store_close(store);
     }
     return Qnil;
 }
@@ -308,32 +333,6 @@ static VALUE frb_dir_make_lock(VALUE self, VALUE rlock_name) {
     return rlock;
 }
 
-/*** FrbStore ****************************************************************/
-
-static size_t frb_store_size(const void *p) {
-    return sizeof(FrtStore);
-    (void)p;
-}
-
-const rb_data_type_t frb_store_t = {
-    .wrap_struct_name = "FrbStore",
-    .function = {
-        .dmark = NULL,
-        .dfree = frb_dir_free,
-        .dsize = frb_store_size,
-        .dcompact = NULL,
-        .reserved = {0},
-    },
-    .parent = NULL,
-    .data = NULL,
-    .flags = RUBY_TYPED_FREE_IMMEDIATELY
-};
-
-static VALUE frb_store_alloc(VALUE rclass) {
-    FrtStore *st = frt_store_alloc();
-    return TypedData_Wrap_Struct(rclass, &frb_store_t, st);
-}
-
 /****************************************************************************
  *
  * RAMDirectory Methods
@@ -403,19 +402,18 @@ static VALUE frb_fsdir_new(int argc, VALUE *argv, VALUE klass) {
         frb_create_dir(rpath);
     }
     if (!rb_funcall(rb_cFile, id_is_directory, 1, rpath)) {
-        rb_raise(rb_eIOError, "No directory <%s> found. Use :create => true"
-                 " to create one.", rs2s(rpath));
+        rb_raise(rb_eIOError, "No directory <%s> found. Use :create => true to create one.", rs2s(rpath));
     }
     store = frt_open_fs_store(rs2s(rpath));
     if (create) store->clear_all(store);
-    if ((self = store->rstore) == Qnil) {
+    self = store->rstore;
+    if (self == Qnil || DATA_PTR(self) == NULL) {
         self = TypedData_Wrap_Struct(klass, &frb_store_t, store);
         store->rstore = self;
         rb_ivar_set(self, id_ref_cnt, INT2FIX(0));
     } else {
         int ref_cnt = FIX2INT(rb_ivar_get(self, id_ref_cnt)) + 1;
         rb_ivar_set(self, id_ref_cnt, INT2FIX(ref_cnt));
-        FRT_DEREF(store);
     }
     return self;
 }

data/ext/isomorfeus_ferret_ext/frt_analysis.c

@@ -133,7 +133,7 @@ FrtToken *frt_tk_new(void) {
 /*****************************************************************************/
 
 void frt_ts_deref(FrtTokenStream *ts) {
-    if (--ts->ref_cnt <= 0)
+    if (FRT_DEREF(ts) == 0)
         ts->destroy_i(ts);
 }
 
@@ -1089,7 +1089,7 @@ FrtTokenStream *frt_stem_filter_new(FrtTokenStream *sub_ts, const char *algorith
 /*****************************************************************************/
 
 void frt_a_deref(FrtAnalyzer *a) {
-    if (--a->ref_cnt <= 0)
+    if (FRT_DEREF(a) == 0)
         a->destroy_i(a);
 }
 

data/ext/isomorfeus_ferret_ext/frt_analysis.h

@@ -33,15 +33,15 @@ typedef struct FrtTokenStream FrtTokenStream;
 struct FrtTokenStream {
     char           *t;             /* ptr used to scan text */
     char           *text;
-    int             length;
+    int            length;
     rb_encoding    *encoding;
     FrtToken       *(*next)(FrtTokenStream *ts);
     FrtTokenStream *(*reset)(FrtTokenStream *ts, char *text, rb_encoding *encoding);
     FrtTokenStream *(*clone_i)(FrtTokenStream *ts);
     void           (*destroy_i)(FrtTokenStream *ts);
-    int             ref_cnt;
-    VALUE           rts;
-    FrtToken        token;
+    _Atomic unsigned int    ref_cnt;
+    VALUE          rts;
+    FrtToken       token;
 };
 
 extern FrtTokenStream *frt_ts_new_i(size_t size);
@@ -184,7 +184,7 @@ typedef struct FrtAnalyzer {
     FrtTokenStream *current_ts;
     FrtTokenStream *(*get_ts)(struct FrtAnalyzer *a, ID field, char *text, rb_encoding *encoding);
     void           (*destroy_i)(struct FrtAnalyzer *a);
-    int            ref_cnt;
+    _Atomic unsigned int    ref_cnt;
     VALUE          ranalyzer;
 } FrtAnalyzer;
 

data/ext/isomorfeus_ferret_ext/frt_bitvector.c

@@ -19,7 +19,7 @@ FrtBitVector *frt_bv_new(void) {
 }
 
 void frt_bv_destroy(FrtBitVector *bv) {
-    if (--(bv->ref_cnt) == 0) {
+    if (FRT_DEREF(bv) == 0) {
         free(bv->bits);
         free(bv);
     }

data/ext/isomorfeus_ferret_ext/frt_bitvector.h

@@ -24,7 +24,7 @@ typedef struct FrtBitVector {
     int curr_bit;
 
     bool extends_as_ones : 1;
-    int ref_cnt;
+    _Atomic unsigned int ref_cnt;
     VALUE rbv;
 } FrtBitVector;