isomorfeus-ferret 0.13.11 → 0.14.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a5603738325c11384e3dcd2d60e8ce34effbbe7b64fdcc319315910fdca6cbae
-  data.tar.gz: f15cc75761f2b0152243bbdc3983d880636fa71d6a2f5fdc4a34443fad28250b
+  metadata.gz: 5818fce6d84b9bd4814be3bbed270127e05297dcf85adeebc495c8f334430d88
+  data.tar.gz: 77c9c3246c7777947084b47620d3aeeeb9eb76d7b0a17a4d30a37a38547a54da
 SHA512:
-  metadata.gz: 6ee5491226887aa837b0f71f053881c18be5dd460427aebf5501b17a3c863f9f28d4202848c82825ee110b568fb95a083e1defa4d1a316a6e7d9c35f41a328bc
-  data.tar.gz: 4e74d456a810c14b204e948c49885ef00710b2d79544ba5fc1c81c02bb1509459ad927935f9ef13fa73929d35919b18d4c5c56a4eb00790954da67f874fa591f
+  metadata.gz: 59632a0b46b9bd247da0f8b3908654a8027fbcef2aadc897f7681d25b03d4404191d037be323f666ef9bae679c72b135318aa853158e6bf0205b754ec3b2b18f
+  data.tar.gz: 2a037003347c6bca0900bf80410e83f43d397400f37e22f112e6ef6893a568dba29561b12594f803f3b28baee9f5f1ae67595c244d91b7dffa9d06e4e493c891

data/LICENSE

@@ -143,7 +143,7 @@ The following licenses apply to files, which are distributed within the repo
 but not distributed with the gem and not used at runtime:
 
 
-For the Reuter-21578 files in the misc/ferret_vs_lucene directory (corpus, etc.),
+For the Reuter-21578 files in the misc/ferret_vs_others directory (corpus, etc.),
 used for research for developing search engine technology:
 
    The copyright for the text of newswire articles and Reuters
@@ -156,7 +156,7 @@ Distribution 1.0", and inform your readers of the current location of
 the data set (see "Availability & Questions").
 
 
-Apache Lucene jars in  the misc/ferret_vs_lucene directory:
+Apache Lucene jars in  the misc/ferret_vs_others directory:
 
 
                                  Apache License

data/README.md

@@ -11,13 +11,29 @@ At the [Isomorfeus Framework Project](https://isomorfeus.com)
 
 ## About this project
 
-Isomorfeus-Ferret is a revived version of the original ferret gem created by Dave Balmain, [https://github.com/dbalmain/ferret](https://github.com/dbalmain/ferret).
+Isomorfeus-Ferret is a revived version of the original ferret gem created by Dave Balmain,
+[https://github.com/dbalmain/ferret](https://github.com/dbalmain/ferret).
 During revival many things havbe been fixed, now all tests pass, no crashes and it
 successfully compiles and runs with rubys >3. Its no longer a goal to have
 a c library available, but instead the usage is meant as ruby gem with a c extension only.
 
 It works on *nixes, *nuxes, *BSDs and also works on Windows and RaspberryPi.
 
+## Improvements and Changes in Version 0.14
+
+### Breaking
+
+- The API for LazyDocs has changed, they are read only now. LazyDoc#to_h may be used to create a hash, that may be changed and reindexed as doc.
+
+### Performance
+
+- LazyDoc is now truly lazy, fields are automatically retrieved. LazyDoc#load is no longer required, but may be used to preload all fields.
+- Index#each is now multiple times faster, depending on use case.
+
+### Other
+
+- The Index class now includes Enumerable
+
 ## Improvements and Changes in Version 0.13
 
 ### Breaking
@@ -53,7 +69,7 @@ Compression semantics have changed, now Brotli, BZip2 and LZ4 compression codecs
 - LZ4: fast compression, fast decrompression, low compression ratio
 
 To see performance and compression ratios `rake ferret_compression_bench` can be run from the cloned repo.
-It uses data and code within the misc/ferret_vs_lucene directory.
+It uses data and code within the misc/ferret_vs_others directory.
 
 To compress a stored field the :compression option can be used with one of: :no, :brotli, :bz2 or :lz4.
 Example:
@@ -80,7 +96,7 @@ https://github.com/isomorfeus/isomorfeus-ferret/blob/master/lib/isomorfeus/ferre
 The query language and parser are documented here:
 https://github.com/isomorfeus/isomorfeus-ferret/blob/master/ext/isomorfeus_ferret_ext/frb_qparser.c
 
-Examples can be found in the 'test' directory or in 'misc/ferret_vs_lucene'.
+Examples can be found in the 'test' directory or in 'misc/ferret_vs_others'.
 
 ## Running Specs
 
@@ -95,41 +111,46 @@ Ensure your locale is set to C.UTF-8, because the internal c tests don't know ho
 ### Indexing and Searching
 - clone repo
 - bundle install
-- rake ferret_vs_lucene
+- rake ferret_vs_others
 
 A recent Java JDK must be installed to compile and run lucene benchmarks.
 
-Results, Ferret 0.13.7 vs. Lucene 9.1.0, WhitespaceAnalyzer,
-Linux Ubuntu 20.04, FreeBSD 13.0 and Windows 10 on old Intel Core i5 from 2015,
+Results, Ferret 0.14.0 vs. Lucene 9.2.0, WhitespaceAnalyzer,
+Linux Ubuntu 22.04, FreeBSD 13.1 and Windows 10 on old Intel Core i5 from 2015,
 LinuxPi on RaspberryPi 400:
 
 | OS      | Task       | Ferret          | Lucene*        |
 |---------|------------|-----------------|----------------|
-| Linux   | Indexing   |     4905 docs/s |    4785 docs/s |
-| FreeBSD | Indexing   |     4516 docs/s |       -        |
-| Windows | Indexing   |     2361 docs/s |    2395 docs/s |
-| LinuxPi | Indexing   |     1161 docs/s |     707 docs/s |
-| Linux   | Searching  | 25664 queries/s | 4708 queries/s |
-| FreeBSD | Searching  | 25073 queries/s |    -           |
-| Windows | Searching  |  3646 queries/s |  935 queries/s |
-| LinuxPi | Searching  |  5768 queries/s |  680 queries/s |
+| Linux   | Indexing   |     5125 docs/s |    4959 docs/s |
+| FreeBSD | Indexing   |     4537 docs/s |    3831 docs/s |
+| Windows | Indexing   |     2488 docs/s |    2588 docs/s |
+| LinuxPi | Indexing   |     1200 docs/s |     755 docs/s |
+| Linux   | Searching  | 26610 queries/s | 7165 queries/s |
+| FreeBSD | Searching  | 24167 queries/s | 4288 queries/s |
+| Windows | Searching  |  3901 queries/s | 1033 queries/s |
+| LinuxPi | Searching  |  6194 queries/s |  785 queries/s |
 |         | Index Size |           28 MB |          35 MB |
 
-*Lucene 9.1.0 on JVM 11.0.14.1 (Ubuntu)
+* JVM Versions:
+OpenJDK Runtime Environment (build 18-ea+36-Ubuntu-1) (Linux)
+OpenJDK Runtime Environment (build 17.0.3+7-Raspbian-1deb11u1rpt1) (LinuxPi)
+OpenJDK Runtime Environment Temurin-18.0.1+10 (build 18.0.1+10) (Windows)
+OpenJDK Runtime Environment (build 17.0.2+8-1) (FreeBSD)
 
 ### Storing Fields with Compression, Indexing and Retrieval
+
 - clone repo
 - bundle install
 - rake ferret_compression_benchmark
 
-Results on Linux, 0.13.7, on old Intel Core i5 from 2015:
+Results on Linux, 0.14.0, on old Intel Core i5 from 2015:
 
-| Compression | Index & Store | Retrieve      | Index size |
-|-------------|---------------|---------------|------------|
-| none        |   4866 docs/s | 153853 docs/s |      43 MB |
-| brotli      |   3539 docs/s |  58315 docs/s |      36 MB |
-| bzip2       |   2624 docs/s |  15382 docs/s |      38 MB |
-| lz4         |   4639 docs/s | 127100 docs/s |      41 MB |
+| Compression | Index & Store | Retrieve Title | Index size |
+|-------------|---------------|----------------|------------|
+| none        |   4862 docs/s |  278827 docs/s |      43 MB |
+| brotli      |   3559 docs/s |  178170 docs/s |      36 MB |
+| bzip2       |   2628 docs/s |   81877 docs/s |      38 MB |
+| lz4         |   4648 docs/s |  232236 docs/s |      41 MB |
 
 ## Future
 

data/ext/isomorfeus_ferret_ext/frb_field_info.c

@@ -0,0 +1,539 @@
+#include "frt_index.h"
+#include "isomorfeus_ferret.h"
+
+VALUE cFieldInfo;
+
+static VALUE sym_store;
+static VALUE sym_index;
+static VALUE sym_compression;
+static VALUE sym_brotli;
+static VALUE sym_bz2;
+static VALUE sym_lz4;
+static VALUE sym_term_vector;
+static VALUE sym_omit_norms;
+static VALUE sym_untokenized;
+static VALUE sym_untokenized_omit_norms;
+static VALUE sym_with_offsets;
+static VALUE sym_with_positions;
+static VALUE sym_with_positions_offsets;
+
+extern VALUE sym_boost;
+
+void frb_fi_get_params(VALUE roptions, FrtStoreValue *store, FrtCompressionType *compression, FrtIndexValue *index, FrtTermVectorValue *term_vector, float *boost) {
+    VALUE v;
+    Check_Type(roptions, T_HASH);
+    v = rb_hash_aref(roptions, sym_boost);
+    if (Qnil != v) {
+        *boost = (float)NUM2DBL(v);
+    } else {
+        *boost = 1.0f;
+    }
+    v = rb_hash_aref(roptions, sym_store);
+    if (Qnil != v) Check_Type(v, T_SYMBOL);
+    if (v == sym_no || v == sym_false || v == Qfalse) {
+        *store = FRT_STORE_NO;
+    } else if (v == sym_yes || v == sym_true || v == Qtrue) {
+        *store = FRT_STORE_YES;
+    } else if (v == Qnil) {
+        /* leave as default */
+    } else {
+        rb_raise(rb_eArgError, ":%s isn't a valid argument for :store. Please choose from [:yes, :no]",
+                 rb_id2name(SYM2ID(v)));
+    }
+
+    v = rb_hash_aref(roptions, sym_compression);
+    if (Qnil != v) Check_Type(v, T_SYMBOL);
+    if (v == sym_no || v == sym_false || v == Qfalse) {
+        *compression = FRT_COMPRESSION_NONE;
+    } else if (v == sym_yes || v == sym_true || v == Qtrue || v == sym_brotli) {
+        *compression = FRT_COMPRESSION_BROTLI;
+    } else if (v == sym_bz2) {
+        *compression = FRT_COMPRESSION_BZ2;
+    } else if (v == sym_lz4) {
+        *compression = FRT_COMPRESSION_LZ4;
+    } else if (v == Qnil) {
+        /* leave as default */
+    } else {
+        rb_raise(rb_eArgError, ":%s isn't a valid argument for :compression. Please choose from [:yes, :no, :brotli, :bz2, :lz4]",
+                 rb_id2name(SYM2ID(v)));
+    }
+
+    v = rb_hash_aref(roptions, sym_index);
+    if (Qnil != v) Check_Type(v, T_SYMBOL);
+    if (v == sym_no || v == sym_false || v == Qfalse) {
+        *index = FRT_INDEX_NO;
+    } else if (v == sym_yes || v == sym_true || v == Qtrue) {
+        *index = FRT_INDEX_YES;
+    } else if (v == sym_untokenized) {
+        *index = FRT_INDEX_UNTOKENIZED;
+    } else if (v == sym_omit_norms) {
+        *index = FRT_INDEX_YES_OMIT_NORMS;
+    } else if (v == sym_untokenized_omit_norms) {
+        *index = FRT_INDEX_UNTOKENIZED_OMIT_NORMS;
+    } else if (v == Qnil) {
+        /* leave as default */
+    } else {
+        rb_raise(rb_eArgError, ":%s isn't a valid argument for :index. Please choose from [:no, :yes, :untokenized, "
+                 ":omit_norms, :untokenized_omit_norms]", rb_id2name(SYM2ID(v)));
+    }
+
+    v = rb_hash_aref(roptions, sym_term_vector);
+    if (Qnil != v) Check_Type(v, T_SYMBOL);
+    if (v == sym_no || v == sym_false || v == Qfalse) {
+        *term_vector = FRT_TERM_VECTOR_NO;
+    } else if (v == sym_yes || v == sym_true || v == Qtrue) {
+        *term_vector = FRT_TERM_VECTOR_YES;
+    } else if (v == sym_with_positions) {
+        *term_vector = FRT_TERM_VECTOR_WITH_POSITIONS;
+    } else if (v == sym_with_offsets) {
+        *term_vector = FRT_TERM_VECTOR_WITH_OFFSETS;
+    } else if (v == sym_with_positions_offsets) {
+        *term_vector = FRT_TERM_VECTOR_WITH_POSITIONS_OFFSETS;
+    } else if (v == Qnil) {
+        /* leave as default */
+        if (*index == FRT_INDEX_NO) *term_vector = FRT_TERM_VECTOR_NO;
+    } else {
+        rb_raise(rb_eArgError, ":%s isn't a valid argument for :term_vector. Please choose from [:no, :yes, "
+                 ":with_positions, :with_offsets, :with_positions_offsets]", rb_id2name(SYM2ID(v)));
+    }
+}
+
+static void frb_fi_free(void *p) {
+    frt_fi_deref((FrtFieldInfo *)p);
+}
+
+static size_t frb_fi_size(const void *p) {
+    return sizeof(FrtFieldInfo);
+    (void)p;
+}
+
+const rb_data_type_t frb_field_info_t = {
+    .wrap_struct_name = "FrbFieldInfo",
+    .function = {
+        .dmark = NULL,
+        .dfree = frb_fi_free,
+        .dsize = frb_fi_size,
+        .dcompact = NULL,
+        .reserved = {0},
+    },
+    .parent = NULL,
+    .data = NULL,
+    .flags = RUBY_TYPED_FREE_IMMEDIATELY
+};
+
+VALUE frb_get_field_info(FrtFieldInfo *fi) {
+    if (fi) {
+        if (fi->rfi == 0 || fi->rfi == Qnil) {
+            fi->rfi = TypedData_Wrap_Struct(cFieldInfo, &frb_field_info_t, fi);
+            FRT_REF(fi);
+        }
+        return fi->rfi;
+    }
+    return Qnil;
+}
+
+/*
+ *  call-seq:
+ *     FieldInfo.new(name, options = {}) -> field_info
+ *
+ *  Create a new FieldInfo object with the name +name+ and the properties
+ *  specified in +options+. The available options are [:store, :compression,
+ *  :index, :term_vector, :boost]. See the description of FieldInfo for more
+ *  information on these properties.
+ */
+static VALUE frb_fi_alloc(VALUE rclass) {
+    FrtFieldInfo *fi = frt_fi_alloc();
+    return TypedData_Wrap_Struct(rclass, &frb_field_info_t, fi);
+}
+
+static VALUE frb_fi_init(int argc, VALUE *argv, VALUE self) {
+    VALUE roptions, rname;
+    FrtFieldInfo *fi;
+    TypedData_Get_Struct(self, FrtFieldInfo, &frb_field_info_t, fi);
+    FrtStoreValue store = FRT_STORE_YES;
+    FrtCompressionType compression = FRT_COMPRESSION_NONE;
+    FrtIndexValue index = FRT_INDEX_YES;
+    FrtTermVectorValue term_vector = FRT_TERM_VECTOR_WITH_POSITIONS_OFFSETS;
+    float boost = 1.0f;
+
+    rb_scan_args(argc, argv, "11", &rname, &roptions);
+    if (argc > 1) {
+        frb_fi_get_params(roptions, &store, &compression, &index, &term_vector, &boost);
+    }
+    fi = frt_fi_init(fi, frb_field(rname), store, compression, index, term_vector);
+    fi->boost = boost;
+    fi->rfi = self;
+    return self;
+}
+
+/*
+ *  call-seq:
+ *     fi.name -> symbol
+ *
+ *  Return the name of the field
+ */
+static VALUE frb_fi_name(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return ID2SYM(fi->name);
+}
+
+/*
+ *  call-seq:
+ *     fi.stored? -> bool
+ *
+ *  Return true if the field is stored in the index.
+ */
+static VALUE frb_fi_is_stored(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_is_stored(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.compressed? -> bool
+ *
+ *  Return true if the field is stored in the index in compressed format.
+ */
+static VALUE frb_fi_is_compressed(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_is_compressed(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.indexed? -> bool
+ *
+ *  Return true if the field is indexed, ie searchable in the index.
+ */
+static VALUE frb_fi_is_indexed(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_is_indexed(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.tokenized? -> bool
+ *
+ *  Return true if the field is tokenized. Tokenizing is the process of
+ *  breaking the field up into tokens. That is "the quick brown fox" becomes:
+ *
+ *    ["the", "quick", "brown", "fox"]
+ *
+ *  A field can only be tokenized if it is indexed.
+ */
+static VALUE frb_fi_is_tokenized(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_is_tokenized(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.omit_norms? -> bool
+ *
+ *  Return true if the field omits the norm file. The norm file is the file
+ *  used to store the field boosts for an indexed field. If you do not boost
+ *  any fields, and you can live without scoring based on field length then
+ *  you can omit the norms file. This will give the index a slight performance
+ *  boost and it will use less memory, especially for indexes which have a
+ *  large number of documents.
+ */
+static VALUE frb_fi_omit_norms(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_omit_norms(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.store_term_vector? -> bool
+ *
+ *  Return true if the term-vectors are stored for this field.
+ */
+static VALUE frb_fi_store_term_vector(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_store_term_vector(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.store_positions? -> bool
+ *
+ *  Return true if positions are stored with the term-vectors for this field.
+ */
+static VALUE frb_fi_store_positions(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_store_positions(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.store_offsets? -> bool
+ *
+ *  Return true if offsets are stored with the term-vectors for this field.
+ */
+static VALUE frb_fi_store_offsets(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_store_offsets(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.has_norms? -> bool
+ *
+ *  Return true if this field has a norms file. This is the same as calling;
+ *
+ *    fi.indexed? and not fi.omit_norms?
+ */
+static VALUE frb_fi_has_norms(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return fi_has_norms(fi) ? Qtrue : Qfalse;
+}
+
+/*
+ *  call-seq:
+ *     fi.boost -> boost
+ *
+ *  Return the default boost for this field
+ */
+static VALUE frb_fi_boost(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    return rb_float_new((double)fi->boost);
+}
+
+/*
+ *  call-seq:
+ *     fi.to_s -> string
+ *
+ *  Return a string representation of the FieldInfo object.
+ */
+static VALUE frb_fi_to_s(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    char *fi_s = frt_fi_to_s(fi);
+    VALUE rfi_s = rb_str_new2(fi_s);
+    free(fi_s);
+    return rfi_s;
+}
+
+/*
+ *  call-seq:
+ *     fi.to_h -> Hssh
+ *
+ *  Return a Hash representation of the FieldInfo object.
+ */
+static VALUE frb_fi_to_h(VALUE self) {
+    FrtFieldInfo *fi = (FrtFieldInfo *)DATA_PTR(self);
+    VALUE hash = rb_hash_new();
+    VALUE val;
+    bool o;
+
+    // :index
+    if (!fi_is_indexed(fi)) val = sym_no;
+    else {
+      bool t = fi_is_tokenized(fi);
+      o = fi_omit_norms(fi);
+      if (!t && o) val = sym_untokenized_omit_norms;
+      else if (t && o) val = sym_omit_norms;
+      else if (!t && !o) val = sym_untokenized;
+      else val = sym_yes;
+    }
+    rb_hash_aset(hash, sym_index, val);
+
+    // :store
+    rb_hash_aset(hash, sym_store, fi_is_stored(fi) ? sym_yes : sym_no);
+
+    // :compress
+    if (!fi_is_compressed(fi)) val = sym_no;
+    else {
+      if (fi_is_compressed_brotli(fi)) val = sym_brotli;
+      else if (fi_is_compressed_bz2(fi)) val = sym_bz2;
+      else if (fi_is_compressed_lz4(fi)) val = sym_lz4;
+      else val = sym_yes;
+    }
+    rb_hash_aset(hash, sym_compression, val);
+
+    // :term_vector
+    if (!fi_store_term_vector(fi)) val = sym_no;
+    else {
+      bool p = fi_store_positions(fi);
+      o = fi_store_offsets(fi);
+      if (p && o) val = sym_with_positions_offsets;
+      else if (o) val = sym_with_offsets;
+      else if (p) val = sym_with_positions;
+      else val = sym_yes;
+    }
+    rb_hash_aset(hash, sym_term_vector, val);
+
+    // :boost
+    rb_hash_aset(hash, sym_boost, rb_float_new((double)fi->boost));
+
+    return hash;
+}
+
+/*
+ *  Document-class: Ferret::Index::FieldInfo
+ *
+ *  == Summary
+ *
+ *  The FieldInfo class is the field descriptor for the index. It specifies
+ *  whether a field is compressed or not or whether it should be indexed and
+ *  tokenized. Every field has a name which must be a symbol. There are three
+ *  properties that you can set, +:store+, +:index+ and +:term_vector+. You
+ *  can also set the default +:boost+ for a field as well.
+ *
+ *  == Properties
+ *
+ *  === :store
+ *
+ *  The +:store+ property allows you to specify how a field is stored. You can
+ *  leave a field unstored (+:no+), store it in it's original format (+:yes+)
+ *  or store it in compressed format (+:compressed+). By default the document
+ *  is stored in its original format. If the field is large and it is stored
+ *  elsewhere where it is easily accessible you might want to leave it
+ *  unstored. This will keep the index size a lot smaller and make the
+ *  indexing process a lot faster. For example, you should probably leave the
+ *  +:content+ field unstored when indexing all the documents in your
+ *  file-system.
+ *
+ *  === :index
+ *
+ *  The +:index+ property allows you to specify how a field is indexed. A
+ *  field must be indexed to be searchable. However, a field doesn't need to
+ *  be indexed to be store in the Ferret index. You may want to use the index
+ *  as a simple database and store things like images or MP3s in the index. By
+ *  default each field is indexed and tokenized (split into tokens) (+:yes+).
+ *  If you don't want to index the field use +:no+. If you want the field
+ *  indexed but not tokenized, use +:untokenized+. Do this for the fields you
+ *  wish to sort by. There are two other values for +:index+; +:omit_norms+
+ *  and +:untokenized_omit_norms+. These values correspond to +:yes+ and
+ *  +:untokenized+ respectively and are useful if you are not boosting any
+ *  fields and you'd like to speed up the index. The norms file is the file
+ *  which contains the boost values for each document for a particular field.
+ *
+ *  === :term_vector
+ *
+ *  See TermVector for a description of term-vectors. You can specify whether
+ *  or not you would like to store term-vectors. The available options are
+ *  +:no+, +:yes+, +:with_positions+, +:with_offsets+ and
+ *  +:with_positions_offsets+. Note that you need to store the positions to
+ *  associate offsets with individual terms in the term_vector.
+ *
+ *  == Property Table
+ *
+ *    Property       Value                     Description
+ *    ------------------------------------------------------------------------
+ *     :store       | :no                     | Don't store field
+ *                  |                         |
+ *                  | :yes (default)          | Store field in its original
+ *                  |                         | format. Use this value if you
+ *                  |                         | want to highlight matches.
+ *                  |                         | or print match excerpts a la
+ *                  |                         | Google search.
+ *     -------------|-------------------------|------------------------------
+ *     :compression | :no (default)           | Don't compress stored field
+ *                  |                         |
+ *                  | :brotli                 | Compress field using Brotli
+ *                  |                         |
+ *                  | :bz2                    | Compress field using BZip2
+ *                  |                         |
+ *                  | :lz4                    | Compress field using LZ4
+ *     -------------|-------------------------|------------------------------
+ *     :index       | :no                     | Do not make this field
+ *                  |                         | searchable.
+ *                  |                         |
+ *                  | :yes (default)          | Make this field searchable and
+ *                  |                         | tokenize its contents.
+ *                  |                         |
+ *                  | :untokenized            | Make this field searchable but
+ *                  |                         | do not tokenize its contents.
+ *                  |                         | use this value for fields you
+ *                  |                         | wish to sort by.
+ *                  |                         |
+ *                  | :omit_norms             | Same as :yes except omit the
+ *                  |                         | norms file. The norms file can
+ *                  |                         | be omitted if you don't boost
+ *                  |                         | any fields and you don't need
+ *                  |                         | scoring based on field length.
+ *                  |                         |
+ *                  | :untokenized_omit_norms | Same as :untokenized except omit
+ *                  |                         | the norms file. Norms files can
+ *                  |                         | be omitted if you don't boost
+ *                  |                         | any fields and you don't need
+ *                  |                         | scoring based on field length.
+ *                  |                         |
+ *     -------------|-------------------------|------------------------------
+ *     :term_vector | :no                     | Don't store term-vectors
+ *                  |                         |
+ *                  | :yes                    | Store term-vectors without
+ *                  |                         | storing positions or offsets.
+ *                  |                         |
+ *                  | :with_positions         | Store term-vectors with
+ *                  |                         | positions.
+ *                  |                         |
+ *                  | :with_offsets           | Store term-vectors with
+ *                  |                         | offsets.
+ *                  |                         |
+ *                  | :with_positions_offsets | Store term-vectors with
+ *                  | (default)               | positions and offsets.
+ *     -------------|-------------------------|------------------------------
+ *     :boost       | Float                   | The boost property is used to
+ *                  |                         | set the default boost for a
+ *                  |                         | field. This boost value will
+ *                  |                         | used for all instances of the
+ *                  |                         | field in the index unless
+ *                  |                         | otherwise specified when you
+ *                  |                         | create the field. All values
+ *                  |                         | should be positive.
+ *                  |                         |
+ *
+ *  == Examples
+ *
+ *    fi = FieldInfo.new(:title, :index => :untokenized, :term_vector => :no,
+ *                       :boost => 10.0)
+ *
+ *    fi = FieldInfo.new(:content)
+ *
+ *    fi = FieldInfo.new(:created_on, :index => :untokenized_omit_norms,
+ *                       :term_vector => :no)
+ *
+ *    fi = FieldInfo.new(:image, :store => :yes, :compression => :brotli, :index => :no,
+ *                       :term_vector => :no)
+ */
+void Init_FieldInfo(void) {
+    sym_store = ID2SYM(rb_intern("store"));
+    sym_index = ID2SYM(rb_intern("index"));
+    sym_term_vector = ID2SYM(rb_intern("term_vector"));
+
+    sym_brotli = ID2SYM(rb_intern("brotli"));
+    sym_bz2 = ID2SYM(rb_intern("bz2"));
+    sym_lz4 = ID2SYM(rb_intern("lz4"));
+    // sym_level = ID2SYM(rb_intern("level"));
+    sym_compression = ID2SYM(rb_intern("compression"));
+
+    sym_untokenized = ID2SYM(rb_intern("untokenized"));
+    sym_omit_norms = ID2SYM(rb_intern("omit_norms"));
+    sym_untokenized_omit_norms = ID2SYM(rb_intern("untokenized_omit_norms"));
+
+    sym_with_positions = ID2SYM(rb_intern("with_positions"));
+    sym_with_offsets = ID2SYM(rb_intern("with_offsets"));
+    sym_with_positions_offsets = ID2SYM(rb_intern("with_positions_offsets"));
+
+    cFieldInfo = rb_define_class_under(mIndex, "FieldInfo", rb_cObject);
+    rb_define_alloc_func(cFieldInfo, frb_fi_alloc);
+
+    rb_define_method(cFieldInfo, "initialize",  frb_fi_init, -1);
+    rb_define_method(cFieldInfo, "name",        frb_fi_name, 0);
+    rb_define_method(cFieldInfo, "stored?",     frb_fi_is_stored, 0);
+    rb_define_method(cFieldInfo, "compressed?", frb_fi_is_compressed, 0);
+    rb_define_method(cFieldInfo, "indexed?",    frb_fi_is_indexed, 0);
+    rb_define_method(cFieldInfo, "tokenized?",  frb_fi_is_tokenized, 0);
+    rb_define_method(cFieldInfo, "omit_norms?", frb_fi_omit_norms, 0);
+    rb_define_method(cFieldInfo, "store_term_vector?",
+                                                frb_fi_store_term_vector, 0);
+    rb_define_method(cFieldInfo, "store_positions?",
+                                                frb_fi_store_positions, 0);
+    rb_define_method(cFieldInfo, "store_offsets?",
+                                                frb_fi_store_offsets, 0);
+    rb_define_method(cFieldInfo, "has_norms?",  frb_fi_has_norms, 0);
+    rb_define_method(cFieldInfo, "boost",       frb_fi_boost, 0);
+    rb_define_method(cFieldInfo, "to_s",        frb_fi_to_s, 0);
+    rb_define_method(cFieldInfo, "to_h",        frb_fi_to_h, 0);
+}