RubyGems - rroonga - Versions diffs - 5.0.4-x64-mingw32 → 5.0.5-x64-mingw32 - Mend

rroonga 5.0.4-x64-mingw32 → 5.0.5-x64-mingw32

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (750) hide show

data/vendor/local/share/doc/groonga/en/html/reference/token_filters.html CHANGED Viewed

@@ -7,7 +7,7 @@
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-    <title>7.9. Token filters &mdash; Groonga v5.0.4-139-g6629adb documentation</title>
+    <title>7.9. Token filters &mdash; Groonga v5.0.6-226-gd7da7e7 documentation</title>
     <link rel="stylesheet" href="../_static/groonga.css" type="text/css" />
     <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
@@ -15,7 +15,7 @@
     <script type="text/javascript">
       var DOCUMENTATION_OPTIONS = {
         URL_ROOT:    '../',
-        VERSION:     '5.0.4-139-g6629adb',
+        VERSION:     '5.0.6-226-gd7da7e7',
         COLLAPSE_INDEX: false,
         FILE_SUFFIX: '.html',
         HAS_SOURCE:  true
@@ -25,12 +25,12 @@
     <script type="text/javascript" src="../_static/underscore.js"></script>
     <script type="text/javascript" src="../_static/doctools.js"></script>
     <link rel="shortcut icon" href="../_static/favicon.ico"/>
-    <link rel="top" title="Groonga v5.0.4-139-g6629adb documentation" href="../index.html" />
+    <link rel="top" title="Groonga v5.0.6-226-gd7da7e7 documentation" href="../index.html" />
     <link rel="up" title="7. Reference manual" href="../reference.html" />
     <link rel="next" title="7.10. Query expanders" href="query_expanders.html" />
     <link rel="prev" title="7.8. Tokenizers" href="tokenizers.html" />
   </head>
-  <body>
+  <body role="document">
 <div class="header">
   <h1 class="title">
     <a id="top-link" href="../index.html">
@@ -48,7 +48,7 @@
 </div>
-    <div class="related">
+    <div class="related" role="navigation" aria-label="related navigation">
       <h3>Navigation</h3>
       <ul>
         <li class="right" style="margin-right: 10px">
@@ -60,15 +60,15 @@
         <li class="right" >
           <a href="tokenizers.html" title="7.8. Tokenizers"
              accesskey="P">previous</a> |</li>
-        <li><a href="../index.html">Groonga v5.0.4-139-g6629adb documentation</a> &raquo;</li>
-          <li><a href="../reference.html" accesskey="U">7. Reference manual</a> &raquo;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v5.0.6-226-gd7da7e7 documentation</a> &raquo;</li>
+          <li class="nav-item nav-item-1"><a href="../reference.html" accesskey="U">7. Reference manual</a> &raquo;</li>
       </ul>
     </div>
     <div class="document">
       <div class="documentwrapper">
         <div class="bodywrapper">
-          <div class="body">
+          <div class="body" role="main">
   <div class="section" id="token-filters">
 <h1>7.9. Token filters<a class="headerlink" href="#token-filters" title="Permalink to this headline">¶</a></h1>
@@ -78,9 +78,9 @@
 <p>Token filter module can be added as a plugin.</p>
 <p>You can customize tokenized token by registering your token filters plugins to Groonga.</p>
 <p>A table can have zero or more token filters. You can attach token
-filters to a table by <a class="reference internal" href="commands/table_create.html#table-create-token-filters"><em>token_filters</em></a> option in
+filters to a table by <a class="reference internal" href="commands/table_create.html#table-create-token-filters"><span>token_filters</span></a> option in
 <a class="reference internal" href="commands/table_create.html"><em>table_create</em></a>.</p>
-<p>Here is an example <tt class="docutils literal"><span class="pre">table_create</span></tt> that uses <tt class="docutils literal"><span class="pre">TokenFilterStopWord</span></tt>
+<p>Here is an example <code class="docutils literal"><span class="pre">table_create</span></code> that uses <code class="docutils literal"><span class="pre">TokenFilterStopWord</span></code>
 token filter module:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>register token_filters/stop_word
@@ -97,17 +97,17 @@ table_create Terms TABLE_PAT_KEY ShortText \
 <h2>7.9.2. Available token filters<a class="headerlink" href="#available-token-filters" title="Permalink to this headline">¶</a></h2>
 <p>Here is the list of available token filters:</p>
 <ul class="simple">
-<li><tt class="docutils literal"><span class="pre">TokenFilterStopWord</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenFilterStem</span></tt></li>
+<li><code class="docutils literal"><span class="pre">TokenFilterStopWord</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenFilterStem</span></code></li>
 </ul>
 <div class="section" id="tokenfilterstopword">
-<span id="token-filter-stop-word"></span><h3>7.9.2.1. <tt class="docutils literal"><span class="pre">TokenFilterStopWord</span></tt><a class="headerlink" href="#tokenfilterstopword" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenFilterStopWord</span></tt> removes stop words from tokenized token
+<span id="token-filter-stop-word"></span><h3>7.9.2.1. <code class="docutils literal"><span class="pre">TokenFilterStopWord</span></code><a class="headerlink" href="#tokenfilterstopword" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenFilterStopWord</span></code> removes stop words from tokenized token
 in searching the documents.</p>
-<p><tt class="docutils literal"><span class="pre">TokenFilterStopWord</span></tt> can specify stop word after adding the
+<p><code class="docutils literal"><span class="pre">TokenFilterStopWord</span></code> can specify stop word after adding the
 documents because it removes token in searching the documents.</p>
-<p>The stop word is specified <tt class="docutils literal"><span class="pre">is_stop_word</span></tt> column on lexicon table.</p>
-<p>Here is an example that uses <tt class="docutils literal"><span class="pre">TokenFilterStopWord</span></tt> token filter:</p>
+<p>The stop word is specified <code class="docutils literal"><span class="pre">is_stop_word</span></code> column on lexicon table.</p>
+<p>Here is an example that uses <code class="docutils literal"><span class="pre">TokenFilterStopWord</span></code> token filter:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>register token_filters/stop_word
 # [[0, 1337566253.89858, 0.000355720520019531], true]
@@ -171,14 +171,14 @@ select Memos --match_columns content --query &quot;Hello and&quot;
 # ]
 </pre></div>
 </div>
-<p><tt class="docutils literal"><span class="pre">and</span></tt> token is marked as stop word in <tt class="docutils literal"><span class="pre">Terms</span></tt> table.</p>
-<p><tt class="docutils literal"><span class="pre">&quot;Hello&quot;</span></tt> that doesn't have <tt class="docutils literal"><span class="pre">and</span></tt> in content is matched. Because
-<tt class="docutils literal"><span class="pre">and</span></tt> is a stop word and <tt class="docutils literal"><span class="pre">and</span></tt> is removed from query.</p>
+<p><code class="docutils literal"><span class="pre">and</span></code> token is marked as stop word in <code class="docutils literal"><span class="pre">Terms</span></code> table.</p>
+<p><code class="docutils literal"><span class="pre">&quot;Hello&quot;</span></code> that doesn't have <code class="docutils literal"><span class="pre">and</span></code> in content is matched. Because
+<code class="docutils literal"><span class="pre">and</span></code> is a stop word and <code class="docutils literal"><span class="pre">and</span></code> is removed from query.</p>
 </div>
 <div class="section" id="tokenfilterstem">
-<span id="token-filter-stem"></span><h3>7.9.2.2. <tt class="docutils literal"><span class="pre">TokenFilterStem</span></tt><a class="headerlink" href="#tokenfilterstem" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenFilterStem</span></tt> stems tokenized token.</p>
-<p>Here is an example that uses <tt class="docutils literal"><span class="pre">TokenFilterStem</span></tt> token filter:</p>
+<span id="token-filter-stem"></span><h3>7.9.2.2. <code class="docutils literal"><span class="pre">TokenFilterStem</span></code><a class="headerlink" href="#tokenfilterstem" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenFilterStem</span></code> stems tokenized token.</p>
+<p>Here is an example that uses <code class="docutils literal"><span class="pre">TokenFilterStem</span></code> token filter:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>register token_filters/stem
 # [[0, 1337566253.89858, 0.000355720520019531], true]
@@ -239,9 +239,9 @@ select Memos --match_columns content --query &quot;develops&quot;
 # ]
 </pre></div>
 </div>
-<p>All of <tt class="docutils literal"><span class="pre">develop</span></tt>, <tt class="docutils literal"><span class="pre">developing</span></tt>, <tt class="docutils literal"><span class="pre">developed</span></tt> and <tt class="docutils literal"><span class="pre">develops</span></tt>
-tokens are stemmed as <tt class="docutils literal"><span class="pre">develop</span></tt>. So we can find <tt class="docutils literal"><span class="pre">develop</span></tt>,
-<tt class="docutils literal"><span class="pre">developing</span></tt> and <tt class="docutils literal"><span class="pre">developed</span></tt> by <tt class="docutils literal"><span class="pre">develops</span></tt> query.</p>
+<p>All of <code class="docutils literal"><span class="pre">develop</span></code>, <code class="docutils literal"><span class="pre">developing</span></code>, <code class="docutils literal"><span class="pre">developed</span></code> and <code class="docutils literal"><span class="pre">develops</span></code>
+tokens are stemmed as <code class="docutils literal"><span class="pre">develop</span></code>. So we can find <code class="docutils literal"><span class="pre">develop</span></code>,
+<code class="docutils literal"><span class="pre">developing</span></code> and <code class="docutils literal"><span class="pre">developed</span></code> by <code class="docutils literal"><span class="pre">develops</span></code> query.</p>
 </div>
 </div>
 <div class="section" id="see-also">
@@ -256,15 +256,15 @@ tokens are stemmed as <tt class="docutils literal"><span class="pre">develop</sp
           </div>
         </div>
       </div>
-      <div class="sphinxsidebar">
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
         <div class="sphinxsidebarwrapper">
   <h3><a href="../index.html">Table Of Contents</a></h3>
   <ul>
 <li><a class="reference internal" href="#">7.9. Token filters</a><ul>
 <li><a class="reference internal" href="#summary">7.9.1. Summary</a></li>
 <li><a class="reference internal" href="#available-token-filters">7.9.2. Available token filters</a><ul>
-<li><a class="reference internal" href="#tokenfilterstopword">7.9.2.1. <tt class="docutils literal"><span class="pre">TokenFilterStopWord</span></tt></a></li>
-<li><a class="reference internal" href="#tokenfilterstem">7.9.2.2. <tt class="docutils literal"><span class="pre">TokenFilterStem</span></tt></a></li>
+<li><a class="reference internal" href="#tokenfilterstopword">7.9.2.1. <code class="docutils literal"><span class="pre">TokenFilterStopWord</span></code></a></li>
+<li><a class="reference internal" href="#tokenfilterstem">7.9.2.2. <code class="docutils literal"><span class="pre">TokenFilterStem</span></code></a></li>
 </ul>
 </li>
 <li><a class="reference internal" href="#see-also">7.9.3. See also</a></li>
@@ -278,12 +278,14 @@ tokens are stemmed as <tt class="docutils literal"><span class="pre">develop</sp
   <h4>Next topic</h4>
   <p class="topless"><a href="query_expanders.html"
                         title="next chapter">7.10. Query expanders</a></p>
-  <h3>This Page</h3>
-  <ul class="this-page-menu">
-    <li><a href="../_sources/reference/token_filters.txt"
-           rel="nofollow">Show Source</a></li>
-  </ul>
-<div id="searchbox" style="display: none">
+  <div role="note" aria-label="source link">
+    <h3>This Page</h3>
+    <ul class="this-page-menu">
+      <li><a href="../_sources/reference/token_filters.txt"
+            rel="nofollow">Show Source</a></li>
+    </ul>
+   </div>
+<div id="searchbox" style="display: none" role="search">
   <h3>Quick search</h3>
     <form class="search" action="../search.html" method="get">
       <input type="text" name="q" />
@@ -300,7 +302,7 @@ tokens are stemmed as <tt class="docutils literal"><span class="pre">develop</sp
       </div>
       <div class="clearer"></div>
     </div>
-    <div class="related">
+    <div class="related" role="navigation" aria-label="related navigation">
       <h3>Navigation</h3>
       <ul>
         <li class="right" style="margin-right: 10px">
@@ -312,11 +314,11 @@ tokens are stemmed as <tt class="docutils literal"><span class="pre">develop</sp
         <li class="right" >
           <a href="tokenizers.html" title="7.8. Tokenizers"
              >previous</a> |</li>
-        <li><a href="../index.html">Groonga v5.0.4-139-g6629adb documentation</a> &raquo;</li>
-          <li><a href="../reference.html" >7. Reference manual</a> &raquo;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v5.0.6-226-gd7da7e7 documentation</a> &raquo;</li>
+          <li class="nav-item nav-item-1"><a href="../reference.html" >7. Reference manual</a> &raquo;</li>
       </ul>
     </div>
-    <div class="footer">
+    <div class="footer" role="contentinfo">
         &copy; Copyright 2009-2015, Brazil, Inc.
     </div>
   </body>

data/vendor/local/share/doc/groonga/en/html/reference/tokenizers.html CHANGED Viewed

@@ -7,7 +7,7 @@
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-    <title>7.8. Tokenizers &mdash; Groonga v5.0.4-139-g6629adb documentation</title>
+    <title>7.8. Tokenizers &mdash; Groonga v5.0.6-226-gd7da7e7 documentation</title>
     <link rel="stylesheet" href="../_static/groonga.css" type="text/css" />
     <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
@@ -15,7 +15,7 @@
     <script type="text/javascript">
       var DOCUMENTATION_OPTIONS = {
         URL_ROOT:    '../',
-        VERSION:     '5.0.4-139-g6629adb',
+        VERSION:     '5.0.6-226-gd7da7e7',
         COLLAPSE_INDEX: false,
         FILE_SUFFIX: '.html',
         HAS_SOURCE:  true
@@ -25,12 +25,12 @@
     <script type="text/javascript" src="../_static/underscore.js"></script>
     <script type="text/javascript" src="../_static/doctools.js"></script>
     <link rel="shortcut icon" href="../_static/favicon.ico"/>
-    <link rel="top" title="Groonga v5.0.4-139-g6629adb documentation" href="../index.html" />
+    <link rel="top" title="Groonga v5.0.6-226-gd7da7e7 documentation" href="../index.html" />
     <link rel="up" title="7. Reference manual" href="../reference.html" />
     <link rel="next" title="7.9. Token filters" href="token_filters.html" />
     <link rel="prev" title="7.7. Normalizers" href="normalizers.html" />
   </head>
-  <body>
+  <body role="document">
 <div class="header">
   <h1 class="title">
     <a id="top-link" href="../index.html">
@@ -48,7 +48,7 @@
 </div>
-    <div class="related">
+    <div class="related" role="navigation" aria-label="related navigation">
       <h3>Navigation</h3>
       <ul>
         <li class="right" style="margin-right: 10px">
@@ -60,15 +60,15 @@
         <li class="right" >
           <a href="normalizers.html" title="7.7. Normalizers"
              accesskey="P">previous</a> |</li>
-        <li><a href="../index.html">Groonga v5.0.4-139-g6629adb documentation</a> &raquo;</li>
-          <li><a href="../reference.html" accesskey="U">7. Reference manual</a> &raquo;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v5.0.6-226-gd7da7e7 documentation</a> &raquo;</li>
+          <li class="nav-item nav-item-1"><a href="../reference.html" accesskey="U">7. Reference manual</a> &raquo;</li>
       </ul>
     </div>
     <div class="document">
       <div class="documentwrapper">
         <div class="bodywrapper">
-          <div class="body">
+          <div class="body" role="main">
   <div class="section" id="tokenizers">
 <h1>7.8. Tokenizers<a class="headerlink" href="#tokenizers" title="Permalink to this headline">¶</a></h1>
@@ -79,15 +79,15 @@ the following cases:</p>
 <blockquote>
 <div><ul>
 <li><p class="first">Indexing text</p>
-<div class="figure align-center">
+<div class="figure align-center" id="id1">
 <a class="reference internal image-reference" href="../_images/used-when-indexing.png"><img alt="../_images/used-when-indexing.png" src="../_images/used-when-indexing.png" style="width: 80%;" /></a>
-<p class="caption">Tokenizer is used when indexing text.</p>
+<p class="caption"><span class="caption-text">Tokenizer is used when indexing text.</span></p>
 </div>
 </li>
 <li><p class="first">Searching by query</p>
-<div class="figure align-center">
+<div class="figure align-center" id="id2">
 <a class="reference internal image-reference" href="../_images/used-when-searching.png"><img alt="../_images/used-when-searching.png" src="../_images/used-when-searching.png" style="width: 80%;" /></a>
-<p class="caption">Tokenizer is used when searching by query.</p>
+<p class="caption"><span class="caption-text">Tokenizer is used when searching by query.</span></p>
 </div>
 </li>
 </ul>
@@ -95,12 +95,12 @@ the following cases:</p>
 <p>Tokenizer is an important module for full-text search. You can change
 trade-off between <a class="reference external" href="http://en.wikipedia.org/wiki/Precision_and_recall">precision and recall</a> by changing
 tokenizer.</p>
-<p>Normally, <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> is a suitable tokenizer. If you don't
+<p>Normally, <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> is a suitable tokenizer. If you don't
 know much about tokenizer, it's recommended that you choose
-<a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>.</p>
+<a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>.</p>
 <p>You can try a tokenizer by <a class="reference internal" href="commands/tokenize.html"><em>tokenize</em></a> and
 <a class="reference internal" href="commands/table_tokenize.html"><em>table_tokenize</em></a>. Here is an example to
-try <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> tokenizer by
+try <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> tokenizer by
 <a class="reference internal" href="commands/tokenize.html"><em>tokenize</em></a>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;Hello World&quot;
@@ -113,46 +113,57 @@ try <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;He&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;el&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ll&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;lo&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;o &quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot; W&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;Wo&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;or&quot;
 #     },
 #     {
 #       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;rl&quot;
 #     },
 #     {
 #       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ld&quot;
 #     },
 #     {
 #       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;d&quot;
 #     }
 #   ]
@@ -164,86 +175,86 @@ try <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>
 <h2>7.8.2. What is &quot;tokenize&quot;?<a class="headerlink" href="#what-is-tokenize" title="Permalink to this headline">¶</a></h2>
 <p>&quot;tokenize&quot; is the process that extracts zero or more tokens from a
 text. There are some &quot;tokenize&quot; methods.</p>
-<p>For example, <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> is tokenized to the following tokens by
+<p>For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
 bigram tokenize method:</p>
 <blockquote>
 <div><ul class="simple">
-<li><tt class="docutils literal"><span class="pre">He</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">el</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">ll</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">lo</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">o_</span></tt> (<tt class="docutils literal"><span class="pre">_</span></tt> means a white-space)</li>
-<li><tt class="docutils literal"><span class="pre">_W</span></tt> (<tt class="docutils literal"><span class="pre">_</span></tt> means a white-space)</li>
-<li><tt class="docutils literal"><span class="pre">Wo</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">or</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">rl</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">ld</span></tt></li>
+<li><code class="docutils literal"><span class="pre">He</span></code></li>
+<li><code class="docutils literal"><span class="pre">el</span></code></li>
+<li><code class="docutils literal"><span class="pre">ll</span></code></li>
+<li><code class="docutils literal"><span class="pre">lo</span></code></li>
+<li><code class="docutils literal"><span class="pre">o_</span></code> (<code class="docutils literal"><span class="pre">_</span></code> means a white-space)</li>
+<li><code class="docutils literal"><span class="pre">_W</span></code> (<code class="docutils literal"><span class="pre">_</span></code> means a white-space)</li>
+<li><code class="docutils literal"><span class="pre">Wo</span></code></li>
+<li><code class="docutils literal"><span class="pre">or</span></code></li>
+<li><code class="docutils literal"><span class="pre">rl</span></code></li>
+<li><code class="docutils literal"><span class="pre">ld</span></code></li>
 </ul>
 </div></blockquote>
-<p>In the above example, 10 tokens are extracted from one text <tt class="docutils literal"><span class="pre">Hello</span>
-<span class="pre">World</span></tt>.</p>
-<p>For example, <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> is tokenized to the following tokens by
+<p>In the above example, 10 tokens are extracted from one text <code class="docutils literal"><span class="pre">Hello</span>
+<span class="pre">World</span></code>.</p>
+<p>For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
 white-space-separate tokenize method:</p>
 <blockquote>
 <div><ul class="simple">
-<li><tt class="docutils literal"><span class="pre">Hello</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">World</span></tt></li>
+<li><code class="docutils literal"><span class="pre">Hello</span></code></li>
+<li><code class="docutils literal"><span class="pre">World</span></code></li>
 </ul>
 </div></blockquote>
-<p>In the above example, 2 tokens are extracted from one text <tt class="docutils literal"><span class="pre">Hello</span>
-<span class="pre">World</span></tt>.</p>
+<p>In the above example, 2 tokens are extracted from one text <code class="docutils literal"><span class="pre">Hello</span>
+<span class="pre">World</span></code>.</p>
 <p>Token is used as search key. You can find indexed documents only by
 tokens that are extracted by used tokenize method. For example, you
-can find <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> by <tt class="docutils literal"><span class="pre">ll</span></tt> with bigram tokenize method but you
-can't find <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> by <tt class="docutils literal"><span class="pre">ll</span></tt> with white-space-separate tokenize
+can find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal"><span class="pre">ll</span></code> with bigram tokenize method but you
+can't find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal"><span class="pre">ll</span></code> with white-space-separate tokenize
 method. Because white-space-separate tokenize method doesn't extract
-<tt class="docutils literal"><span class="pre">ll</span></tt> token. It just extracts <tt class="docutils literal"><span class="pre">Hello</span></tt> and <tt class="docutils literal"><span class="pre">World</span></tt> tokens.</p>
+<code class="docutils literal"><span class="pre">ll</span></code> token. It just extracts <code class="docutils literal"><span class="pre">Hello</span></code> and <code class="docutils literal"><span class="pre">World</span></code> tokens.</p>
 <p>In general, tokenize method that generates small tokens increases
 recall but decreases precision. Tokenize method that generates large
 tokens increases precision but decreases recall.</p>
-<p>For example, we can find <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> and <tt class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></tt> by <tt class="docutils literal"><span class="pre">or</span></tt> with
-bigram tokenize method. <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> is a noise for people who
+<p>For example, we can find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> and <code class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal"><span class="pre">or</span></code> with
+bigram tokenize method. <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is a noise for people who
 wants to search &quot;logical and&quot;. It means that precision is
 decreased. But recall is increased.</p>
-<p>We can find only <tt class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></tt> by <tt class="docutils literal"><span class="pre">or</span></tt> with white-space-separate
-tokenize method. Because <tt class="docutils literal"><span class="pre">World</span></tt> is tokenized to one token <tt class="docutils literal"><span class="pre">World</span></tt>
+<p>We can find only <code class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal"><span class="pre">or</span></code> with white-space-separate
+tokenize method. Because <code class="docutils literal"><span class="pre">World</span></code> is tokenized to one token <code class="docutils literal"><span class="pre">World</span></code>
 with white-space-separate tokenize method. It means that precision is
 increased for people who wants to search &quot;logical and&quot;. But recall is
-decreased because <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> that contains <tt class="docutils literal"><span class="pre">or</span></tt> isn't found.</p>
+decreased because <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> that contains <code class="docutils literal"><span class="pre">or</span></code> isn't found.</p>
 </div>
 <div class="section" id="built-in-tokenizsers">
 <h2>7.8.3. Built-in tokenizsers<a class="headerlink" href="#built-in-tokenizsers" title="Permalink to this headline">¶</a></h2>
 <p>Here is a list of built-in tokenizers:</p>
 <blockquote>
 <div><ul class="simple">
-<li><tt class="docutils literal"><span class="pre">TokenBigram</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitAlpha</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitAlphaDigit</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenUnigram</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenTrigram</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenDelimit</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenDelimitNull</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenMecab</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">TokenRegexp</span></tt></li>
+<li><code class="docutils literal"><span class="pre">TokenBigram</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitAlpha</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitAlphaDigit</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenUnigram</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenTrigram</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenDelimit</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenMecab</span></code></li>
+<li><code class="docutils literal"><span class="pre">TokenRegexp</span></code></li>
 </ul>
 </div></blockquote>
 <div class="section" id="tokenbigram">
-<span id="token-bigram"></span><h3>7.8.3.1. <tt class="docutils literal"><span class="pre">TokenBigram</span></tt><a class="headerlink" href="#tokenbigram" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigram</span></tt> is a bigram based tokenizer. It's recommended to use
+<span id="token-bigram"></span><h3>7.8.3.1. <code class="docutils literal"><span class="pre">TokenBigram</span></code><a class="headerlink" href="#tokenbigram" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> is a bigram based tokenizer. It's recommended to use
 this tokenizer for most cases.</p>
 <p>Bigram tokenize method tokenizes a text to two adjacent characters
-tokens. For example, <tt class="docutils literal"><span class="pre">Hello</span></tt> is tokenized to the following tokens:</p>
+tokens. For example, <code class="docutils literal"><span class="pre">Hello</span></code> is tokenized to the following tokens:</p>
 <blockquote>
 <div><ul class="simple">
-<li><tt class="docutils literal"><span class="pre">He</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">el</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">ll</span></tt></li>
-<li><tt class="docutils literal"><span class="pre">lo</span></tt></li>
+<li><code class="docutils literal"><span class="pre">He</span></code></li>
+<li><code class="docutils literal"><span class="pre">el</span></code></li>
+<li><code class="docutils literal"><span class="pre">ll</span></code></li>
+<li><code class="docutils literal"><span class="pre">lo</span></code></li>
 </ul>
 </div></blockquote>
 <p>Bigram tokenize method is good for recall because you can find all
@@ -252,15 +263,15 @@ texts by query consists of two or more characters.</p>
 character because one character token doesn't exist. But you can find
 all texts by query consists of one character in Groonga. Because
 Groonga find tokens that start with query by predictive search. For
-example, Groonga can find <tt class="docutils literal"><span class="pre">ll</span></tt> and <tt class="docutils literal"><span class="pre">lo</span></tt> tokens by <tt class="docutils literal"><span class="pre">l</span></tt> query.</p>
+example, Groonga can find <code class="docutils literal"><span class="pre">ll</span></code> and <code class="docutils literal"><span class="pre">lo</span></code> tokens by <code class="docutils literal"><span class="pre">l</span></code> query.</p>
 <p>Bigram tokenize method isn't good for precision because you can find
-texts that includes query in word. For example, you can find <tt class="docutils literal"><span class="pre">world</span></tt>
-by <tt class="docutils literal"><span class="pre">or</span></tt>. This is more sensitive for ASCII only languages rather than
-non-ASCII languages. <tt class="docutils literal"><span class="pre">TokenBigram</span></tt> has solution for this problem
+texts that includes query in word. For example, you can find <code class="docutils literal"><span class="pre">world</span></code>
+by <code class="docutils literal"><span class="pre">or</span></code>. This is more sensitive for ASCII only languages rather than
+non-ASCII languages. <code class="docutils literal"><span class="pre">TokenBigram</span></code> has solution for this problem
 described in the bellow.</p>
-<p><tt class="docutils literal"><span class="pre">TokenBigram</span></tt> behavior is different when it's worked with any
+<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> behavior is different when it's worked with any
 <a class="reference internal" href="normalizers.html"><em>Normalizers</em></a>.</p>
-<p>If no normalizer is used, <tt class="docutils literal"><span class="pre">TokenBigram</span></tt> uses pure bigram (all tokens
+<p>If no normalizer is used, <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses pure bigram (all tokens
 except the last token have two characters) tokenize method:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;Hello World&quot;
@@ -273,54 +284,65 @@ except the last token have two characters) tokenize method:</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;He&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;el&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ll&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;lo&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;o &quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot; W&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;Wo&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;or&quot;
 #     },
 #     {
 #       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;rl&quot;
 #     },
 #     {
 #       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ld&quot;
 #     },
 #     {
 #       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;d&quot;
 #     }
 #   ]
 # ]
 </pre></div>
 </div>
-<p>If normalizer is used, <tt class="docutils literal"><span class="pre">TokenBigram</span></tt> uses white-space-separate like
-tokenize method for ASCII characters. <tt class="docutils literal"><span class="pre">TokenBigram</span></tt> uses bigram
+<p>If normalizer is used, <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses white-space-separate like
+tokenize method for ASCII characters. <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses bigram
 tokenize method for non-ASCII characters.</p>
 <p>You may be confused with this combined behavior. But it's reasonable
 for most use cases such as English text (only ASCII characters) and
@@ -332,10 +354,10 @@ the case.</p>
 word separator. Bigram tokenize method is suitable for the case.</p>
 <p>Mixed tokenize method is suitable for mixed language case.</p>
 <p>If you want to use bigram tokenize method for ASCII character, see
-<tt class="docutils literal"><span class="pre">TokenBigramSplitXXX</span></tt> type tokenizers such as
-<a class="reference internal" href="#token-bigram-split-symbol-alpha"><em>TokenBigramSplitSymbolAlpha</em></a>.</p>
-<p>Let's confirm <tt class="docutils literal"><span class="pre">TokenBigram</span></tt> behavior by example.</p>
-<p><tt class="docutils literal"><span class="pre">TokenBigram</span></tt> uses one or more white-spaces as token delimiter for
+<code class="docutils literal"><span class="pre">TokenBigramSplitXXX</span></code> type tokenizers such as
+<a class="reference internal" href="#token-bigram-split-symbol-alpha"><span>TokenBigramSplitSymbolAlpha</span></a>.</p>
+<p>Let's confirm <code class="docutils literal"><span class="pre">TokenBigram</span></code> behavior by example.</p>
+<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> uses one or more white-spaces as token delimiter for
 ASCII characters:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;Hello World&quot; NormalizerAuto
@@ -348,23 +370,25 @@ ASCII characters:</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;hello&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;world&quot;
 #     }
 #   ]
 # ]
 </pre></div>
 </div>
-<p><tt class="docutils literal"><span class="pre">TokenBigram</span></tt> uses character type change as token delimiter for
+<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> uses character type change as token delimiter for
 ASCII characters. Character type is one of them:</p>
 <blockquote>
 <div><ul class="simple">
 <li>Alphabet</li>
 <li>Digit</li>
-<li>Symbol (such as <tt class="docutils literal"><span class="pre">(</span></tt>, <tt class="docutils literal"><span class="pre">)</span></tt> and <tt class="docutils literal"><span class="pre">!</span></tt>)</li>
+<li>Symbol (such as <code class="docutils literal"><span class="pre">(</span></code>, <code class="docutils literal"><span class="pre">)</span></code> and <code class="docutils literal"><span class="pre">!</span></code>)</li>
 <li>Hiragana</li>
 <li>Katakana</li>
 <li>Kanji</li>
@@ -374,8 +398,8 @@ ASCII characters. Character type is one of them:</p>
 <p>The following example shows two token delimiters:</p>
 <blockquote>
 <div><ul class="simple">
-<li>at between <tt class="docutils literal"><span class="pre">100</span></tt> (digits) and <tt class="docutils literal"><span class="pre">cents</span></tt> (alphabets)</li>
-<li>at between <tt class="docutils literal"><span class="pre">cents</span></tt> (alphabets) and <tt class="docutils literal"><span class="pre">!!!</span></tt> (symbols)</li>
+<li>at between <code class="docutils literal"><span class="pre">100</span></code> (digits) and <code class="docutils literal"><span class="pre">cents</span></code> (alphabets)</li>
+<li>at between <code class="docutils literal"><span class="pre">cents</span></code> (alphabets) and <code class="docutils literal"><span class="pre">!!!</span></code> (symbols)</li>
 </ul>
 </div></blockquote>
 <p>Execution example:</p>
@@ -389,21 +413,24 @@ ASCII characters. Character type is one of them:</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;100&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;cents&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!!&quot;
 #     }
 #   ]
 # ]
 </pre></div>
 </div>
-<p>Here is an example that <tt class="docutils literal"><span class="pre">TokenBigram</span></tt> uses bigram tokenize method
+<p>Here is an example that <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses bigram tokenize method
 for non-ASCII characters.</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;日本語の勉強&quot; NormalizerAuto
@@ -416,26 +443,32 @@ for non-ASCII characters.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語の&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;の勉&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;勉強&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;強&quot;
 #     }
 #   ]
@@ -444,9 +477,9 @@ for non-ASCII characters.</p>
 </div>
 </div>
 <div class="section" id="tokenbigramsplitsymbol">
-<span id="token-bigram-split-symbol"></span><h3>7.8.3.2. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></tt><a class="headerlink" href="#tokenbigramsplitsymbol" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></tt> is similar to <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The
-difference between them is symbol handling. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></tt>
+<span id="token-bigram-split-symbol"></span><h3>7.8.3.2. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code><a class="headerlink" href="#tokenbigramsplitsymbol" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code> is similar to <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The
+difference between them is symbol handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code>
 tokenizes symbols by bigram tokenize method:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigramSplitSymbol &quot;100cents!!!&quot; NormalizerAuto
@@ -459,22 +492,27 @@ tokenizes symbols by bigram tokenize method:</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;100&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;cents&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
@@ -483,10 +521,10 @@ tokenizes symbols by bigram tokenize method:</p>
 </div>
 </div>
 <div class="section" id="tokenbigramsplitsymbolalpha">
-<span id="token-bigram-split-symbol-alpha"></span><h3>7.8.3.3. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></tt><a class="headerlink" href="#tokenbigramsplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></tt> is similar to <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The
+<span id="token-bigram-split-symbol-alpha"></span><h3>7.8.3.3. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code><a class="headerlink" href="#tokenbigramsplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code> is similar to <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The
 difference between them is symbol and alphabet
-handling. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></tt> tokenizes symbols and
+handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code> tokenizes symbols and
 alphabets by bigram tokenize method:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigramSplitSymbolAlpha &quot;100cents!!!&quot; NormalizerAuto
@@ -499,38 +537,47 @@ alphabets by bigram tokenize method:</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;100&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ce&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;en&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;nt&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ts&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;s!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
@@ -539,10 +586,10 @@ alphabets by bigram tokenize method:</p>
 </div>
 </div>
 <div class="section" id="tokenbigramsplitsymbolalphadigit">
-<span id="token-bigram-split-symbol-alpha-digit"></span><h3>7.8.3.4. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></tt><a class="headerlink" href="#tokenbigramsplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></tt> is similar to
-<a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The difference between them is symbol, alphabet
-and digit handling. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></tt> tokenizes
+<span id="token-bigram-split-symbol-alpha-digit"></span><h3>7.8.3.4. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code><a class="headerlink" href="#tokenbigramsplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code> is similar to
+<a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The difference between them is symbol, alphabet
+and digit handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code> tokenizes
 symbols, alphabets and digits by bigram tokenize method. It means that
 all characters are tokenized by bigram tokenize method:</p>
 <p>Execution example:</p>
@@ -556,46 +603,57 @@ all characters are tokenized by bigram tokenize method:</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;10&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;00&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;0c&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ce&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;en&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;nt&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ts&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;s!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
@@ -604,13 +662,13 @@ all characters are tokenized by bigram tokenize method:</p>
 </div>
 </div>
 <div class="section" id="tokenbigramignoreblank">
-<span id="token-bigram-ignore-blank"></span><h3>7.8.3.5. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></tt><a class="headerlink" href="#tokenbigramignoreblank" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></tt> is similar to <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The
-difference between them is blank handling. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></tt>
+<span id="token-bigram-ignore-blank"></span><h3>7.8.3.5. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code><a class="headerlink" href="#tokenbigramignoreblank" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code> is similar to <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The
+difference between them is blank handling. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code>
 ignores white-spaces in continuous symbols and non-ASCII characters.</p>
-<p>You can find difference of them by <tt class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></tt> text because it
+<p>You can find difference of them by <code class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
 has symbols and non-ASCII characters.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> :</p>
+<p>Here is a result by <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> :</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;日 本 語 ! ! !&quot; NormalizerAuto
 # [
@@ -622,33 +680,39 @@ has symbols and non-ASCII characters.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
 # ]
 </pre></div>
 </div>
-<p>Here is a result by <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></tt>:</p>
+<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigramIgnoreBlank &quot;日 本 語 ! ! !&quot; NormalizerAuto
 # [
@@ -660,18 +724,22 @@ has symbols and non-ASCII characters.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!!&quot;
 #     }
 #   ]
@@ -680,22 +748,22 @@ has symbols and non-ASCII characters.</p>
 </div>
 </div>
 <div class="section" id="tokenbigramignoreblanksplitsymbol">
-<span id="token-bigram-ignore-blank-split-symbol"></span><h3>7.8.3.6. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></tt><a class="headerlink" href="#tokenbigramignoreblanksplitsymbol" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></tt> is similar to
-<a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The differences between them are the followings:</p>
+<span id="token-bigram-ignore-blank-split-symbol"></span><h3>7.8.3.6. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbol" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> is similar to
+<a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The differences between them are the followings:</p>
 <blockquote>
 <div><ul class="simple">
 <li>Blank handling</li>
 <li>Symbol handling</li>
 </ul>
 </div></blockquote>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></tt> ignores white-spaces in
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> ignores white-spaces in
 continuous symbols and non-ASCII characters.</p>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></tt> tokenizes symbols by bigram
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> tokenizes symbols by bigram
 tokenize method.</p>
-<p>You can find difference of them by <tt class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></tt> text because it
+<p>You can find difference of them by <code class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
 has symbols and non-ASCII characters.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> :</p>
+<p>Here is a result by <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> :</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;日 本 語 ! ! !&quot; NormalizerAuto
 # [
@@ -707,33 +775,39 @@ has symbols and non-ASCII characters.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
 # ]
 </pre></div>
 </div>
-<p>Here is a result by <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></tt>:</p>
+<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigramIgnoreBlankSplitSymbol &quot;日 本 語 ! ! !&quot; NormalizerAuto
 # [
@@ -745,26 +819,32 @@ has symbols and non-ASCII characters.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
@@ -773,22 +853,22 @@ has symbols and non-ASCII characters.</p>
 </div>
 </div>
 <div class="section" id="tokenbigramignoreblanksplitsymbolalpha">
-<span id="token-bigram-ignore-blank-split-symbol-alpha"></span><h3>7.8.3.7. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></tt><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></tt> is similar to
-<a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The differences between them are the followings:</p>
+<span id="token-bigram-ignore-blank-split-symbol-alpha"></span><h3>7.8.3.7. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> is similar to
+<a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The differences between them are the followings:</p>
 <blockquote>
 <div><ul class="simple">
 <li>Blank handling</li>
 <li>Symbol and alphabet handling</li>
 </ul>
 </div></blockquote>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></tt> ignores white-spaces in
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> ignores white-spaces in
 continuous symbols and non-ASCII characters.</p>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></tt> tokenizes symbols and
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> tokenizes symbols and
 alphabets by bigram tokenize method.</p>
-<p>You can find difference of them by <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></tt> text because it
+<p>You can find difference of them by <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
 has symbols and non-ASCII characters with white spaces and alphabets.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> :</p>
+<p>Here is a result by <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> :</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;Hello 日 本 語 ! ! !&quot; NormalizerAuto
 # [
@@ -800,37 +880,44 @@ has symbols and non-ASCII characters with white spaces and alphabets.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;hello&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
 # ]
 </pre></div>
 </div>
-<p>Here is a result by <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></tt>:</p>
+<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigramIgnoreBlankSplitSymbolAlpha &quot;Hello 日 本 語 ! ! !&quot; NormalizerAuto
 # [
@@ -842,46 +929,57 @@ has symbols and non-ASCII characters with white spaces and alphabets.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;he&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;el&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ll&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;lo&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;o日&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     }
 #   ]
@@ -890,24 +988,24 @@ has symbols and non-ASCII characters with white spaces and alphabets.</p>
 </div>
 </div>
 <div class="section" id="tokenbigramignoreblanksplitsymbolalphadigit">
-<span id="token-bigram-ignore-blank-split-symbol-alpha-digit"></span><h3>7.8.3.8. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></tt><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></tt> is similar to
-<a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The differences between them are the followings:</p>
+<span id="token-bigram-ignore-blank-split-symbol-alpha-digit"></span><h3>7.8.3.8. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> is similar to
+<a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The differences between them are the followings:</p>
 <blockquote>
 <div><ul class="simple">
 <li>Blank handling</li>
 <li>Symbol, alphabet and digit handling</li>
 </ul>
 </div></blockquote>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></tt> ignores white-spaces
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> ignores white-spaces
 in continuous symbols and non-ASCII characters.</p>
-<p><tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></tt> tokenizes symbols,
+<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> tokenizes symbols,
 alphabets and digits by bigram tokenize method. It means that all
 characters are tokenized by bigram tokenize method.</p>
-<p>You can find difference of them by <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">777</span></tt> text
+<p>You can find difference of them by <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">777</span></code> text
 because it has symbols and non-ASCII characters with white spaces,
 alphabets and digits.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> :</p>
+<p>Here is a result by <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> :</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigram &quot;Hello 日 本 語 ! ! ! 777&quot; NormalizerAuto
 # [
@@ -919,41 +1017,49 @@ alphabets and digits.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;hello&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;777&quot;
 #     }
 #   ]
 # ]
 </pre></div>
 </div>
-<p>Here is a result by <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></tt>:</p>
+<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenBigramIgnoreBlankSplitSymbolAlphaDigit &quot;Hello 日 本 語 ! ! ! 777&quot; NormalizerAuto
 # [
@@ -965,58 +1071,72 @@ alphabets and digits.</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;he&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;el&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ll&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;lo&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;o日&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;日本&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;本語&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;語!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!&quot;
 #     },
 #     {
 #       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!7&quot;
 #     },
 #     {
 #       &quot;position&quot;: 11,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;77&quot;
 #     },
 #     {
 #       &quot;position&quot;: 12,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;77&quot;
 #     },
 #     {
 #       &quot;position&quot;: 13,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;7&quot;
 #     }
 #   ]
@@ -1025,10 +1145,10 @@ alphabets and digits.</p>
 </div>
 </div>
 <div class="section" id="tokenunigram">
-<span id="token-unigram"></span><h3>7.8.3.9. <tt class="docutils literal"><span class="pre">TokenUnigram</span></tt><a class="headerlink" href="#tokenunigram" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenUnigram</span></tt> is similar to <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The differences
-between them is token unit. <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> uses 2 characters per
-token. <tt class="docutils literal"><span class="pre">TokenUnigram</span></tt> uses 1 character per token.</p>
+<span id="token-unigram"></span><h3>7.8.3.9. <code class="docutils literal"><span class="pre">TokenUnigram</span></code><a class="headerlink" href="#tokenunigram" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenUnigram</span></code> is similar to <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The differences
+between them is token unit. <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> uses 2 characters per
+token. <code class="docutils literal"><span class="pre">TokenUnigram</span></code> uses 1 character per token.</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenUnigram &quot;100cents!!!&quot; NormalizerAuto
 # [
@@ -1040,14 +1160,17 @@ token. <tt class="docutils literal"><span class="pre">TokenUnigram</span></tt> u
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;100&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;cents&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!!&quot;
 #     }
 #   ]
@@ -1056,10 +1179,10 @@ token. <tt class="docutils literal"><span class="pre">TokenUnigram</span></tt> u
 </div>
 </div>
 <div class="section" id="tokentrigram">
-<span id="token-trigram"></span><h3>7.8.3.10. <tt class="docutils literal"><span class="pre">TokenTrigram</span></tt><a class="headerlink" href="#tokentrigram" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenTrigram</span></tt> is similar to <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a>. The differences
-between them is token unit. <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> uses 2 characters per
-token. <tt class="docutils literal"><span class="pre">TokenTrigram</span></tt> uses 3 characters per token.</p>
+<span id="token-trigram"></span><h3>7.8.3.10. <code class="docutils literal"><span class="pre">TokenTrigram</span></code><a class="headerlink" href="#tokentrigram" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenTrigram</span></code> is similar to <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a>. The differences
+between them is token unit. <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> uses 2 characters per
+token. <code class="docutils literal"><span class="pre">TokenTrigram</span></code> uses 3 characters per token.</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenTrigram &quot;10000cents!!!!!&quot; NormalizerAuto
 # [
@@ -1071,14 +1194,17 @@ token. <tt class="docutils literal"><span class="pre">TokenTrigram</span></tt> u
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;10000&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;cents&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;!!!!!&quot;
 #     }
 #   ]
@@ -1087,14 +1213,14 @@ token. <tt class="docutils literal"><span class="pre">TokenTrigram</span></tt> u
 </div>
 </div>
 <div class="section" id="tokendelimit">
-<span id="token-delimit"></span><h3>7.8.3.11. <tt class="docutils literal"><span class="pre">TokenDelimit</span></tt><a class="headerlink" href="#tokendelimit" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenDelimit</span></tt> extracts token by splitting one or more space
-characters (<tt class="docutils literal"><span class="pre">U+0020</span></tt>). For example, <tt class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></tt> is tokenized to
-<tt class="docutils literal"><span class="pre">Hello</span></tt> and <tt class="docutils literal"><span class="pre">World</span></tt>.</p>
-<p><tt class="docutils literal"><span class="pre">TokenDelimit</span></tt> is suitable for tag text. You can extract <tt class="docutils literal"><span class="pre">groonga</span></tt>
-and <tt class="docutils literal"><span class="pre">full-text-search</span></tt> and <tt class="docutils literal"><span class="pre">http</span></tt> as tags from <tt class="docutils literal"><span class="pre">groonga</span>
-<span class="pre">full-text-search</span> <span class="pre">http</span></tt>.</p>
-<p>Here is an example of <tt class="docutils literal"><span class="pre">TokenDelimit</span></tt>:</p>
+<span id="token-delimit"></span><h3>7.8.3.11. <code class="docutils literal"><span class="pre">TokenDelimit</span></code><a class="headerlink" href="#tokendelimit" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenDelimit</span></code> extracts token by splitting one or more space
+characters (<code class="docutils literal"><span class="pre">U+0020</span></code>). For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to
+<code class="docutils literal"><span class="pre">Hello</span></code> and <code class="docutils literal"><span class="pre">World</span></code>.</p>
+<p><code class="docutils literal"><span class="pre">TokenDelimit</span></code> is suitable for tag text. You can extract <code class="docutils literal"><span class="pre">groonga</span></code>
+and <code class="docutils literal"><span class="pre">full-text-search</span></code> and <code class="docutils literal"><span class="pre">http</span></code> as tags from <code class="docutils literal"><span class="pre">groonga</span>
+<span class="pre">full-text-search</span> <span class="pre">http</span></code>.</p>
+<p>Here is an example of <code class="docutils literal"><span class="pre">TokenDelimit</span></code>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenDelimit &quot;Groonga full-text-search HTTP&quot; NormalizerAuto
 # [
@@ -1106,14 +1232,17 @@ and <tt class="docutils literal"><span class="pre">full-text-search</span></tt>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;groonga&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;full-text-search&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;http&quot;
 #     }
 #   ]
@@ -1122,13 +1251,13 @@ and <tt class="docutils literal"><span class="pre">full-text-search</span></tt>
 </div>
 </div>
 <div class="section" id="tokendelimitnull">
-<span id="token-delimit-null"></span><h3>7.8.3.12. <tt class="docutils literal"><span class="pre">TokenDelimitNull</span></tt><a class="headerlink" href="#tokendelimitnull" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenDelimitNull</span></tt> is similar to <a class="reference internal" href="#token-delimit"><em>TokenDelimit</em></a>. The
-difference between them is separator character. <a class="reference internal" href="#token-delimit"><em>TokenDelimit</em></a>
-uses space character (<tt class="docutils literal"><span class="pre">U+0020</span></tt>) but <tt class="docutils literal"><span class="pre">TokenDelimitNull</span></tt> uses NUL
-character (<tt class="docutils literal"><span class="pre">U+0000</span></tt>).</p>
-<p><tt class="docutils literal"><span class="pre">TokenDelimitNull</span></tt> is also suitable for tag text.</p>
-<p>Here is an example of <tt class="docutils literal"><span class="pre">TokenDelimitNull</span></tt>:</p>
+<span id="token-delimit-null"></span><h3>7.8.3.12. <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code><a class="headerlink" href="#tokendelimitnull" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> is similar to <a class="reference internal" href="#token-delimit"><span>TokenDelimit</span></a>. The
+difference between them is separator character. <a class="reference internal" href="#token-delimit"><span>TokenDelimit</span></a>
+uses space character (<code class="docutils literal"><span class="pre">U+0020</span></code>) but <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> uses NUL
+character (<code class="docutils literal"><span class="pre">U+0000</span></code>).</p>
+<p><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> is also suitable for tag text.</p>
+<p>Here is an example of <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenDelimitNull &quot;Groonga\u0000full-text-search\u0000HTTP&quot; NormalizerAuto
 # [
@@ -1140,6 +1269,7 @@ character (<tt class="docutils literal"><span class="pre">U+0000</span></tt>).</
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;groongau0000full-text-searchu0000http&quot;
 #     }
 #   ]
@@ -1148,23 +1278,23 @@ character (<tt class="docutils literal"><span class="pre">U+0000</span></tt>).</
 </div>
 </div>
 <div class="section" id="tokenmecab">
-<span id="token-mecab"></span><h3>7.8.3.13. <tt class="docutils literal"><span class="pre">TokenMecab</span></tt><a class="headerlink" href="#tokenmecab" title="Permalink to this headline">¶</a></h3>
-<p><tt class="docutils literal"><span class="pre">TokenMecab</span></tt> is a tokenizer based on <a class="reference external" href="http://mecab.sourceforge.net/">MeCab</a> part-of-speech and
+<span id="token-mecab"></span><h3>7.8.3.13. <code class="docutils literal"><span class="pre">TokenMecab</span></code><a class="headerlink" href="#tokenmecab" title="Permalink to this headline">¶</a></h3>
+<p><code class="docutils literal"><span class="pre">TokenMecab</span></code> is a tokenizer based on <a class="reference external" href="http://mecab.sourceforge.net/">MeCab</a> part-of-speech and
 morphological analyzer.</p>
 <p>MeCab doesn't depend on Japanese. You can use MeCab for other
 languages by creating dictionary for the languages. You can use <a class="reference external" href="http://osdn.jp/projects/naist-jdic/">NAIST
 Japanese Dictionary</a>
 for Japanese.</p>
-<p><tt class="docutils literal"><span class="pre">TokenMecab</span></tt> is good for precision rather than recall. You can find
-<tt class="docutils literal"><span class="pre">東京都</span></tt> and <tt class="docutils literal"><span class="pre">京都</span></tt> texts by <tt class="docutils literal"><span class="pre">京都</span></tt> query with
-<a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> but <tt class="docutils literal"><span class="pre">東京都</span></tt> isn't expected. You can find only
-<tt class="docutils literal"><span class="pre">京都</span></tt> text by <tt class="docutils literal"><span class="pre">京都</span></tt> query with <tt class="docutils literal"><span class="pre">TokenMecab</span></tt>.</p>
+<p><code class="docutils literal"><span class="pre">TokenMecab</span></code> is good for precision rather than recall. You can find
+<code class="docutils literal"><span class="pre">東京都</span></code> and <code class="docutils literal"><span class="pre">京都</span></code> texts by <code class="docutils literal"><span class="pre">京都</span></code> query with
+<a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> but <code class="docutils literal"><span class="pre">東京都</span></code> isn't expected. You can find only
+<code class="docutils literal"><span class="pre">京都</span></code> text by <code class="docutils literal"><span class="pre">京都</span></code> query with <code class="docutils literal"><span class="pre">TokenMecab</span></code>.</p>
 <p>If you want to support neologisms, you need to keep updating your
-MeCab dictionary. It needs maintain cost. (<a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> doesn't
-require dictionary maintenance because <a class="reference internal" href="#token-bigram"><em>TokenBigram</em></a> doesn't use
+MeCab dictionary. It needs maintain cost. (<a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> doesn't
+require dictionary maintenance because <a class="reference internal" href="#token-bigram"><span>TokenBigram</span></a> doesn't use
 dictionary.) <a class="reference external" href="https://github.com/neologd/mecab-ipadic-neologd">mecab-ipadic-NEologd : Neologism dictionary for MeCab</a> may help you.</p>
-<p>Here is an example of <tt class="docutils literal"><span class="pre">TokenMeCab</span></tt>. <tt class="docutils literal"><span class="pre">東京都</span></tt> is tokenized to <tt class="docutils literal"><span class="pre">東京</span></tt>
-and <tt class="docutils literal"><span class="pre">都</span></tt>. They don't include <tt class="docutils literal"><span class="pre">京都</span></tt>:</p>
+<p>Here is an example of <code class="docutils literal"><span class="pre">TokenMeCab</span></code>. <code class="docutils literal"><span class="pre">東京都</span></code> is tokenized to <code class="docutils literal"><span class="pre">東京</span></code>
+and <code class="docutils literal"><span class="pre">都</span></code>. They don't include <code class="docutils literal"><span class="pre">京都</span></code>:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenMecab &quot;東京都&quot;
 # [
@@ -1176,10 +1306,12 @@ and <tt class="docutils literal"><span class="pre">都</span></tt>. They don't i
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;東京&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;都&quot;
 #     }
 #   ]
@@ -1188,7 +1320,7 @@ and <tt class="docutils literal"><span class="pre">都</span></tt>. They don't i
 </div>
 </div>
 <div class="section" id="tokenregexp">
-<span id="token-regexp"></span><h3>7.8.3.14. <tt class="docutils literal"><span class="pre">TokenRegexp</span></tt><a class="headerlink" href="#tokenregexp" title="Permalink to this headline">¶</a></h3>
+<span id="token-regexp"></span><h3>7.8.3.14. <code class="docutils literal"><span class="pre">TokenRegexp</span></code><a class="headerlink" href="#tokenregexp" title="Permalink to this headline">¶</a></h3>
 <div class="versionadded">
 <p><span class="versionmodified">New in version 5.0.1.</span></p>
 </div>
@@ -1201,21 +1333,21 @@ and <tt class="docutils literal"><span class="pre">都</span></tt>. They don't i
 <p class="last">This tokenizer can be used only with UTF-8. You can't use this
 tokenizer with EUC-JP, Shift_JIS and so on.</p>
 </div>
-<p><tt class="docutils literal"><span class="pre">TokenRegexp</span></tt> is a tokenizer for supporting regular expression
+<p><code class="docutils literal"><span class="pre">TokenRegexp</span></code> is a tokenizer for supporting regular expression
 search by index.</p>
 <p>In general, regular expression search is evaluated as sequential
 search. But the following cases can be evaluated as index search:</p>
 <blockquote>
 <div><ul class="simple">
-<li>Literal only case such as <tt class="docutils literal"><span class="pre">hello</span></tt></li>
-<li>The beginning of text and literal case such as <tt class="docutils literal"><span class="pre">\A/home/alice</span></tt></li>
-<li>The end of text and literal case such as <tt class="docutils literal"><span class="pre">\.txt\z</span></tt></li>
+<li>Literal only case such as <code class="docutils literal"><span class="pre">hello</span></code></li>
+<li>The beginning of text and literal case such as <code class="docutils literal"><span class="pre">\A/home/alice</span></code></li>
+<li>The end of text and literal case such as <code class="docutils literal"><span class="pre">\.txt\z</span></code></li>
 </ul>
 </div></blockquote>
 <p>In most cases, index search is faster than sequential search.</p>
-<p><tt class="docutils literal"><span class="pre">TokenRegexp</span></tt> is based on bigram tokenize method. <tt class="docutils literal"><span class="pre">TokenRegexp</span></tt>
-adds the beginning of text mark (<tt class="docutils literal"><span class="pre">U+FFEF</span></tt>) at the begging of text
-and the end of text mark (<tt class="docutils literal"><span class="pre">U+FFF0</span></tt>) to the end of text when you
+<p><code class="docutils literal"><span class="pre">TokenRegexp</span></code> is based on bigram tokenize method. <code class="docutils literal"><span class="pre">TokenRegexp</span></code>
+adds the beginning of text mark (<code class="docutils literal"><span class="pre">U+FFEF</span></code>) at the begging of text
+and the end of text mark (<code class="docutils literal"><span class="pre">U+FFF0</span></code>) to the end of text when you
 index text:</p>
 <p>Execution example:</p>
 <div class="highlight-none"><div class="highlight"><pre>tokenize TokenRegexp &quot;/home/alice/test.txt&quot; NormalizerAuto --mode ADD
@@ -1228,194 +1360,112 @@ index text:</p>
 #   [
 #     {
 #       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;￯&quot;
 #     },
 #     {
 #       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;/h&quot;
 #     },
 #     {
 #       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ho&quot;
 #     },
 #     {
 #       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;om&quot;
 #     },
 #     {
 #       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;me&quot;
 #     },
 #     {
 #       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;e/&quot;
 #     },
 #     {
 #       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;/a&quot;
 #     },
 #     {
 #       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;al&quot;
 #     },
 #     {
 #       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;li&quot;
 #     },
 #     {
 #       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ic&quot;
 #     },
 #     {
 #       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;ce&quot;
 #     },
 #     {
 #       &quot;position&quot;: 11,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;e/&quot;
 #     },
 #     {
 #       &quot;position&quot;: 12,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;/t&quot;
 #     },
 #     {
 #       &quot;position&quot;: 13,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;te&quot;
 #     },
 #     {
 #       &quot;position&quot;: 14,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;es&quot;
 #     },
 #     {
 #       &quot;position&quot;: 15,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;st&quot;
 #     },
 #     {
 #       &quot;position&quot;: 16,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;t.&quot;
 #     },
 #     {
 #       &quot;position&quot;: 17,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;.t&quot;
 #     },
 #     {
 #       &quot;position&quot;: 18,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;tx&quot;
 #     },
 #     {
 #       &quot;position&quot;: 19,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;xt&quot;
 #     },
 #     {
 #       &quot;position&quot;: 20,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;t&quot;
 #     },
 #     {
 #       &quot;position&quot;: 21,
-#       &quot;value&quot;: &quot;￰&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>The beginning of text mark is used for the beginning of text search by
-<tt class="docutils literal"><span class="pre">\A</span></tt>. If you use <tt class="docutils literal"><span class="pre">TokenRegexp</span></tt> for tokenizing query,
-<tt class="docutils literal"><span class="pre">TokenRegexp</span></tt> adds the beginning of text mark (<tt class="docutils literal"><span class="pre">U+FFEF</span></tt>) as the
-first token. The beginning of text mark must be appeared at the first,
-you can get results of the beginning of text search.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre>tokenize TokenRegexp &quot;\\A/home/alice/&quot; NormalizerAuto --mode GET
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;value&quot;: &quot;￯&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;value&quot;: &quot;/h&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;value&quot;: &quot;ho&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;value&quot;: &quot;om&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;value&quot;: &quot;me&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;value&quot;: &quot;e/&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;value&quot;: &quot;/a&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;value&quot;: &quot;al&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;value&quot;: &quot;li&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;value&quot;: &quot;ic&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;value&quot;: &quot;ce&quot;
-#     },
-#     {
-#       &quot;position&quot;: 11,
-#       &quot;value&quot;: &quot;e/&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>The end of text mark is used for the end of text search by <tt class="docutils literal"><span class="pre">\z</span></tt>.
-If you use <tt class="docutils literal"><span class="pre">TokenRegexp</span></tt> for tokenizing query, <tt class="docutils literal"><span class="pre">TokenRegexp</span></tt> adds
-the end of text mark (<tt class="docutils literal"><span class="pre">U+FFF0</span></tt>) as the last token. The end of text
-mark must be appeared at the end, you can get results of the end of
-text search.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre>tokenize TokenRegexp &quot;\\.txt\\z&quot; NormalizerAuto --mode GET
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;value&quot;: &quot;\\.&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;value&quot;: &quot;.t&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;value&quot;: &quot;tx&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;value&quot;: &quot;xt&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
 #       &quot;value&quot;: &quot;￰&quot;
 #     }
 #   ]
@@ -1430,7 +1480,7 @@ text search.</p>
           </div>
         </div>
       </div>
-      <div class="sphinxsidebar">
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
         <div class="sphinxsidebarwrapper">
   <h3><a href="../index.html">Table Of Contents</a></h3>
   <ul>
@@ -1438,20 +1488,20 @@ text search.</p>
 <li><a class="reference internal" href="#summary">7.8.1. Summary</a></li>
 <li><a class="reference internal" href="#what-is-tokenize">7.8.2. What is &quot;tokenize&quot;?</a></li>
 <li><a class="reference internal" href="#built-in-tokenizsers">7.8.3. Built-in tokenizsers</a><ul>
-<li><a class="reference internal" href="#tokenbigram">7.8.3.1. <tt class="docutils literal"><span class="pre">TokenBigram</span></tt></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbol">7.8.3.2. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></tt></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbolalpha">7.8.3.3. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></tt></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbolalphadigit">7.8.3.4. <tt class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></tt></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblank">7.8.3.5. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></tt></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbol">7.8.3.6. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></tt></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalpha">7.8.3.7. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></tt></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalphadigit">7.8.3.8. <tt class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></tt></a></li>
-<li><a class="reference internal" href="#tokenunigram">7.8.3.9. <tt class="docutils literal"><span class="pre">TokenUnigram</span></tt></a></li>
-<li><a class="reference internal" href="#tokentrigram">7.8.3.10. <tt class="docutils literal"><span class="pre">TokenTrigram</span></tt></a></li>
-<li><a class="reference internal" href="#tokendelimit">7.8.3.11. <tt class="docutils literal"><span class="pre">TokenDelimit</span></tt></a></li>
-<li><a class="reference internal" href="#tokendelimitnull">7.8.3.12. <tt class="docutils literal"><span class="pre">TokenDelimitNull</span></tt></a></li>
-<li><a class="reference internal" href="#tokenmecab">7.8.3.13. <tt class="docutils literal"><span class="pre">TokenMecab</span></tt></a></li>
-<li><a class="reference internal" href="#tokenregexp">7.8.3.14. <tt class="docutils literal"><span class="pre">TokenRegexp</span></tt></a></li>
+<li><a class="reference internal" href="#tokenbigram">7.8.3.1. <code class="docutils literal"><span class="pre">TokenBigram</span></code></a></li>
+<li><a class="reference internal" href="#tokenbigramsplitsymbol">7.8.3.2. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code></a></li>
+<li><a class="reference internal" href="#tokenbigramsplitsymbolalpha">7.8.3.3. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></a></li>
+<li><a class="reference internal" href="#tokenbigramsplitsymbolalphadigit">7.8.3.4. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></a></li>
+<li><a class="reference internal" href="#tokenbigramignoreblank">7.8.3.5. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code></a></li>
+<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbol">7.8.3.6. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></a></li>
+<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalpha">7.8.3.7. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code></a></li>
+<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalphadigit">7.8.3.8. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code></a></li>
+<li><a class="reference internal" href="#tokenunigram">7.8.3.9. <code class="docutils literal"><span class="pre">TokenUnigram</span></code></a></li>
+<li><a class="reference internal" href="#tokentrigram">7.8.3.10. <code class="docutils literal"><span class="pre">TokenTrigram</span></code></a></li>
+<li><a class="reference internal" href="#tokendelimit">7.8.3.11. <code class="docutils literal"><span class="pre">TokenDelimit</span></code></a></li>
+<li><a class="reference internal" href="#tokendelimitnull">7.8.3.12. <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code></a></li>
+<li><a class="reference internal" href="#tokenmecab">7.8.3.13. <code class="docutils literal"><span class="pre">TokenMecab</span></code></a></li>
+<li><a class="reference internal" href="#tokenregexp">7.8.3.14. <code class="docutils literal"><span class="pre">TokenRegexp</span></code></a></li>
 </ul>
 </li>
 </ul>
@@ -1464,12 +1514,14 @@ text search.</p>
   <h4>Next topic</h4>
   <p class="topless"><a href="token_filters.html"
                         title="next chapter">7.9. Token filters</a></p>
-  <h3>This Page</h3>
-  <ul class="this-page-menu">
-    <li><a href="../_sources/reference/tokenizers.txt"
-           rel="nofollow">Show Source</a></li>
-  </ul>
-<div id="searchbox" style="display: none">
+  <div role="note" aria-label="source link">
+    <h3>This Page</h3>
+    <ul class="this-page-menu">
+      <li><a href="../_sources/reference/tokenizers.txt"
+            rel="nofollow">Show Source</a></li>
+    </ul>
+   </div>
+<div id="searchbox" style="display: none" role="search">
   <h3>Quick search</h3>
     <form class="search" action="../search.html" method="get">
       <input type="text" name="q" />
@@ -1486,7 +1538,7 @@ text search.</p>
       </div>
       <div class="clearer"></div>
     </div>
-    <div class="related">
+    <div class="related" role="navigation" aria-label="related navigation">
       <h3>Navigation</h3>
       <ul>
         <li class="right" style="margin-right: 10px">
@@ -1498,11 +1550,11 @@ text search.</p>
         <li class="right" >
           <a href="normalizers.html" title="7.7. Normalizers"
              >previous</a> |</li>
-        <li><a href="../index.html">Groonga v5.0.4-139-g6629adb documentation</a> &raquo;</li>
-          <li><a href="../reference.html" >7. Reference manual</a> &raquo;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v5.0.6-226-gd7da7e7 documentation</a> &raquo;</li>
+          <li class="nav-item nav-item-1"><a href="../reference.html" >7. Reference manual</a> &raquo;</li>
       </ul>
     </div>
-    <div class="footer">
+    <div class="footer" role="contentinfo">
         &copy; Copyright 2009-2015, Brazil, Inc.
     </div>
   </body>