RubyGems - rroonga - Versions diffs - 7.1.1-x64-mingw32 → 9.0.2-x64-mingw32 - Mend

rroonga 7.1.1-x64-mingw32 → 9.0.2-x64-mingw32

Files changed (980) hide show

data/vendor/local/share/doc/groonga/en/html/reference/tokenizer/summary.html ADDED Viewed

@@ -0,0 +1,259 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <title>7.8.1. Summary &#8212; Groonga v9.0.2 documentation</title>
+    <link rel="stylesheet" href="../../_static/groonga.css" type="text/css" />
+    <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
+    <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
+    <script type="text/javascript" src="../../_static/jquery.js"></script>
+    <script type="text/javascript" src="../../_static/underscore.js"></script>
+    <script type="text/javascript" src="../../_static/doctools.js"></script>
+    <script type="text/javascript" src="../../_static/language_data.js"></script>
+    <link rel="shortcut icon" href="../../_static/favicon.ico"/>
+    <link rel="index" title="Index" href="../../genindex.html" />
+    <link rel="search" title="Search" href="../../search.html" />
+    <link rel="next" title="7.8.2. TokenBigram" href="../tokenizers/token_bigram.html" />
+    <link rel="prev" title="7.8. Tokenizers" href="../tokenizers.html" />
+  </head><body>
+<div class="header">
+  <h1 class="title">
+    <a id="top-link" href="../../index.html">
+      <span class="project">groonga</span>
+      <span class="separator">-</span>
+      <span class="description">An open-source fulltext search engine and column store.</span>
+    </a>
+  </h1>
+  <div class="other-language-links">
+    <ul>
+      <li><a href="../../../../ja/html/reference/tokenizer/summary.html">日本語</a></li>
+    </ul>
+  </div>
+</div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../../genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="../tokenizers/token_bigram.html" title="7.8.2. TokenBigram"
+             accesskey="N">next</a> |</li>
+        <li class="right" >
+          <a href="../tokenizers.html" title="7.8. Tokenizers"
+             accesskey="P">previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
+          <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
+          <li class="nav-item nav-item-2"><a href="../tokenizers.html" accesskey="U">7.8. Tokenizers</a> &#187;</li>
+      </ul>
+    </div>
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+  <div class="section" id="summary">
+<h1>7.8.1. Summary<a class="headerlink" href="#summary" title="Permalink to this headline">¶</a></h1>
+<p>Groonga has tokenizer module that tokenizes text. It is used when
+the following cases:</p>
+<blockquote>
+<div><ul>
+<li><p>Indexing text</p>
+<div class="figure align-center" id="id1">
+<a class="reference internal image-reference" href="../../_images/used-when-indexing.png"><img alt="../../_images/used-when-indexing.png" src="../../_images/used-when-indexing.png" style="width: 80%;" /></a>
+<p class="caption"><span class="caption-text">Tokenizer is used when indexing text.</span><a class="headerlink" href="#id1" title="Permalink to this image">¶</a></p>
+</div>
+</li>
+<li><p>Searching by query</p>
+<div class="figure align-center" id="id2">
+<a class="reference internal image-reference" href="../../_images/used-when-searching.png"><img alt="../../_images/used-when-searching.png" src="../../_images/used-when-searching.png" style="width: 80%;" /></a>
+<p class="caption"><span class="caption-text">Tokenizer is used when searching by query.</span><a class="headerlink" href="#id2" title="Permalink to this image">¶</a></p>
+</div>
+</li>
+</ul>
+</div></blockquote>
+<p>Tokenizer is an important module for full-text search. You can change
+trade-off between <a class="reference external" href="http://en.wikipedia.org/wiki/Precision_and_recall">precision and recall</a> by changing
+tokenizer.</p>
+<p>Normally, <a class="reference internal" href="../tokenizers/token_bigram.html#token-bigram"><span class="std std-ref">TokenBigram</span></a> is a suitable tokenizer. If you don’t
+know much about tokenizer, it’s recommended that you choose
+<a class="reference internal" href="../tokenizers/token_bigram.html#token-bigram"><span class="std std-ref">TokenBigram</span></a>.</p>
+<p>You can try a tokenizer by <a class="reference internal" href="../commands/tokenize.html"><span class="doc">tokenize</span></a> and
+<a class="reference internal" href="../commands/table_tokenize.html"><span class="doc">table_tokenize</span></a>. Here is an example to
+try <a class="reference internal" href="../tokenizers/token_bigram.html#token-bigram"><span class="std std-ref">TokenBigram</span></a> tokenizer by
+<a class="reference internal" href="../commands/tokenize.html"><span class="doc">tokenize</span></a>:</p>
+<p>Execution example:</p>
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot;
+# [
+#   [
+#     0,
+#     1337566253.89858,
+#     0.000355720520019531
+#   ],
+#   [
+#     {
+#       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;He&quot;
+#     },
+#     {
+#       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;el&quot;
+#     },
+#     {
+#       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;ll&quot;
+#     },
+#     {
+#       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;lo&quot;
+#     },
+#     {
+#       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;o &quot;
+#     },
+#     {
+#       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot; W&quot;
+#     },
+#     {
+#       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;Wo&quot;
+#     },
+#     {
+#       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;or&quot;
+#     },
+#     {
+#       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;rl&quot;
+#     },
+#     {
+#       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;ld&quot;
+#     },
+#     {
+#       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;d&quot;
+#     }
+#   ]
+# ]
+</pre></div>
+</div>
+<p>“tokenize” is the process that extracts zero or more tokens from a
+text. There are some “tokenize” methods.</p>
+<p>For example, <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
+bigram tokenize method:</p>
+<blockquote>
+<div><ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">He</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">el</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">ll</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">lo</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">o_</span></code> (<code class="docutils literal notranslate"><span class="pre">_</span></code> means a white-space)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">_W</span></code> (<code class="docutils literal notranslate"><span class="pre">_</span></code> means a white-space)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">Wo</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">or</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">rl</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">ld</span></code></p></li>
+</ul>
+</div></blockquote>
+<p>In the above example, 10 tokens are extracted from one text <code class="docutils literal notranslate"><span class="pre">Hello</span>
+<span class="pre">World</span></code>.</p>
+<p>For example, <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
+white-space-separate tokenize method:</p>
+<blockquote>
+<div><ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">Hello</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">World</span></code></p></li>
+</ul>
+</div></blockquote>
+<p>In the above example, 2 tokens are extracted from one text <code class="docutils literal notranslate"><span class="pre">Hello</span>
+<span class="pre">World</span></code>.</p>
+<p>Token is used as search key. You can find indexed documents only by
+tokens that are extracted by used tokenize method. For example, you
+can find <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal notranslate"><span class="pre">ll</span></code> with bigram tokenize method but you
+can’t find <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal notranslate"><span class="pre">ll</span></code> with white-space-separate tokenize
+method. Because white-space-separate tokenize method doesn’t extract
+<code class="docutils literal notranslate"><span class="pre">ll</span></code> token. It just extracts <code class="docutils literal notranslate"><span class="pre">Hello</span></code> and <code class="docutils literal notranslate"><span class="pre">World</span></code> tokens.</p>
+<p>In general, tokenize method that generates small tokens increases
+recall but decreases precision. Tokenize method that generates large
+tokens increases precision but decreases recall.</p>
+<p>For example, we can find <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> and <code class="docutils literal notranslate"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal notranslate"><span class="pre">or</span></code> with
+bigram tokenize method. <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> is a noise for people who
+wants to search “logical and”. It means that precision is
+decreased. But recall is increased.</p>
+<p>We can find only <code class="docutils literal notranslate"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal notranslate"><span class="pre">or</span></code> with white-space-separate
+tokenize method. Because <code class="docutils literal notranslate"><span class="pre">World</span></code> is tokenized to one token <code class="docutils literal notranslate"><span class="pre">World</span></code>
+with white-space-separate tokenize method. It means that precision is
+increased for people who wants to search “logical and”. But recall is
+decreased because <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> that contains <code class="docutils literal notranslate"><span class="pre">or</span></code> isn’t found.</p>
+</div>
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+  <h4>Previous topic</h4>
+  <p class="topless"><a href="../tokenizers.html"
+                        title="previous chapter">7.8. Tokenizers</a></p>
+  <h4>Next topic</h4>
+  <p class="topless"><a href="../tokenizers/token_bigram.html"
+                        title="next chapter">7.8.2. <code class="docutils literal notranslate"><span class="pre">TokenBigram</span></code></a></p>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <div class="searchformwrapper">
+    <form class="search" action="../../search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+    </form>
+    </div>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../../genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="../tokenizers/token_bigram.html" title="7.8.2. TokenBigram"
+             >next</a> |</li>
+        <li class="right" >
+          <a href="../tokenizers.html" title="7.8. Tokenizers"
+             >previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
+          <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
+          <li class="nav-item nav-item-2"><a href="../tokenizers.html" >7.8. Tokenizers</a> &#187;</li>
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &#169; Copyright 2009-2019, Brazil, Inc.
+    </div>
+  </body>
+</html>

data/vendor/local/share/doc/groonga/en/html/reference/tokenizers.html CHANGED Viewed

@@ -1,34 +1,26 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
   <head>
-    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-    <title>7.8. Tokenizers &#8212; Groonga v7.1.0-73-g6d02cfa documentation</title>
+    <meta charset="utf-8" />
+    <title>7.8. Tokenizers &#8212; Groonga v9.0.2 documentation</title>
     <link rel="stylesheet" href="../_static/groonga.css" type="text/css" />
     <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-    <script type="text/javascript">
-      var DOCUMENTATION_OPTIONS = {
-        URL_ROOT:    '../',
-        VERSION:     '7.1.0-73-g6d02cfa',
-        COLLAPSE_INDEX: false,
-        FILE_SUFFIX: '.html',
-        HAS_SOURCE:  false,
-        SOURCELINK_SUFFIX: '.txt'
-      };
-    </script>
+    <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
     <script type="text/javascript" src="../_static/jquery.js"></script>
     <script type="text/javascript" src="../_static/underscore.js"></script>
     <script type="text/javascript" src="../_static/doctools.js"></script>
+    <script type="text/javascript" src="../_static/language_data.js"></script>
     <link rel="shortcut icon" href="../_static/favicon.ico"/>
     <link rel="index" title="Index" href="../genindex.html" />
     <link rel="search" title="Search" href="../search.html" />
-    <link rel="next" title="7.9. Token filters" href="token_filters.html" />
-    <link rel="prev" title="7.7. Normalizers" href="normalizers.html" />
-  </head>
-  <body>
+    <link rel="next" title="7.8.1. Summary" href="tokenizer/summary.html" />
+    <link rel="prev" title="7.7.2.3. NormalizerNFKC51" href="normalizers/normalizer_nfkc51.html" />
+  </head><body>
 <div class="header">
   <h1 class="title">
     <a id="top-link" href="../index.html">
@@ -53,12 +45,12 @@
           <a href="../genindex.html" title="General Index"
              accesskey="I">index</a></li>
         <li class="right" >
-          <a href="token_filters.html" title="7.9. Token filters"
+          <a href="tokenizer/summary.html" title="7.8.1. Summary"
              accesskey="N">next</a> |</li>
         <li class="right" >
-          <a href="normalizers.html" title="7.7. Normalizers"
+          <a href="normalizers/normalizer_nfkc51.html" title="7.7.2.3. NormalizerNFKC51"
              accesskey="P">previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../reference.html" accesskey="U">7. Reference manual</a> &#187;</li>
       </ul>
     </div>
@@ -70,1403 +62,24 @@
   <div class="section" id="tokenizers">
 <h1>7.8. Tokenizers<a class="headerlink" href="#tokenizers" title="Permalink to this headline">¶</a></h1>
-<div class="section" id="summary">
-<h2>7.8.1. Summary<a class="headerlink" href="#summary" title="Permalink to this headline">¶</a></h2>
-<p>Groonga has tokenizer module that tokenizes text. It is used when
-the following cases:</p>
-<blockquote>
-<div><ul>
-<li><p class="first">Indexing text</p>
-<div class="figure align-center" id="id1">
-<a class="reference internal image-reference" href="../_images/used-when-indexing.png"><img alt="../_images/used-when-indexing.png" src="../_images/used-when-indexing.png" style="width: 80%;" /></a>
-<p class="caption"><span class="caption-text">Tokenizer is used when indexing text.</span></p>
-</div>
-</li>
-<li><p class="first">Searching by query</p>
-<div class="figure align-center" id="id2">
-<a class="reference internal image-reference" href="../_images/used-when-searching.png"><img alt="../_images/used-when-searching.png" src="../_images/used-when-searching.png" style="width: 80%;" /></a>
-<p class="caption"><span class="caption-text">Tokenizer is used when searching by query.</span></p>
-</div>
-</li>
-</ul>
-</div></blockquote>
-<p>Tokenizer is an important module for full-text search. You can change
-trade-off between <a class="reference external" href="http://en.wikipedia.org/wiki/Precision_and_recall">precision and recall</a> by changing
-tokenizer.</p>
-<p>Normally, <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> is a suitable tokenizer. If you don't
-know much about tokenizer, it's recommended that you choose
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>.</p>
-<p>You can try a tokenizer by <a class="reference internal" href="commands/tokenize.html"><span class="doc">tokenize</span></a> and
-<a class="reference internal" href="commands/table_tokenize.html"><span class="doc">table_tokenize</span></a>. Here is an example to
-try <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> tokenizer by
-<a class="reference internal" href="commands/tokenize.html"><span class="doc">tokenize</span></a>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot;
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;He&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o &quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot; W&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;Wo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;or&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;rl&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ld&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;d&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="what-is-tokenize">
-<h2>7.8.2. What is &quot;tokenize&quot;?<a class="headerlink" href="#what-is-tokenize" title="Permalink to this headline">¶</a></h2>
-<p>&quot;tokenize&quot; is the process that extracts zero or more tokens from a
-text. There are some &quot;tokenize&quot; methods.</p>
-<p>For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
-bigram tokenize method:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">He</span></code></li>
-<li><code class="docutils literal"><span class="pre">el</span></code></li>
-<li><code class="docutils literal"><span class="pre">ll</span></code></li>
-<li><code class="docutils literal"><span class="pre">lo</span></code></li>
-<li><code class="docutils literal"><span class="pre">o_</span></code> (<code class="docutils literal"><span class="pre">_</span></code> means a white-space)</li>
-<li><code class="docutils literal"><span class="pre">_W</span></code> (<code class="docutils literal"><span class="pre">_</span></code> means a white-space)</li>
-<li><code class="docutils literal"><span class="pre">Wo</span></code></li>
-<li><code class="docutils literal"><span class="pre">or</span></code></li>
-<li><code class="docutils literal"><span class="pre">rl</span></code></li>
-<li><code class="docutils literal"><span class="pre">ld</span></code></li>
-</ul>
-</div></blockquote>
-<p>In the above example, 10 tokens are extracted from one text <code class="docutils literal"><span class="pre">Hello</span>
-<span class="pre">World</span></code>.</p>
-<p>For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
-white-space-separate tokenize method:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">Hello</span></code></li>
-<li><code class="docutils literal"><span class="pre">World</span></code></li>
-</ul>
-</div></blockquote>
-<p>In the above example, 2 tokens are extracted from one text <code class="docutils literal"><span class="pre">Hello</span>
-<span class="pre">World</span></code>.</p>
-<p>Token is used as search key. You can find indexed documents only by
-tokens that are extracted by used tokenize method. For example, you
-can find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal"><span class="pre">ll</span></code> with bigram tokenize method but you
-can't find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal"><span class="pre">ll</span></code> with white-space-separate tokenize
-method. Because white-space-separate tokenize method doesn't extract
-<code class="docutils literal"><span class="pre">ll</span></code> token. It just extracts <code class="docutils literal"><span class="pre">Hello</span></code> and <code class="docutils literal"><span class="pre">World</span></code> tokens.</p>
-<p>In general, tokenize method that generates small tokens increases
-recall but decreases precision. Tokenize method that generates large
-tokens increases precision but decreases recall.</p>
-<p>For example, we can find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> and <code class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal"><span class="pre">or</span></code> with
-bigram tokenize method. <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is a noise for people who
-wants to search &quot;logical and&quot;. It means that precision is
-decreased. But recall is increased.</p>
-<p>We can find only <code class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal"><span class="pre">or</span></code> with white-space-separate
-tokenize method. Because <code class="docutils literal"><span class="pre">World</span></code> is tokenized to one token <code class="docutils literal"><span class="pre">World</span></code>
-with white-space-separate tokenize method. It means that precision is
-increased for people who wants to search &quot;logical and&quot;. But recall is
-decreased because <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> that contains <code class="docutils literal"><span class="pre">or</span></code> isn't found.</p>
-</div>
-<div class="section" id="built-in-tokenizsers">
-<h2>7.8.3. Built-in tokenizsers<a class="headerlink" href="#built-in-tokenizsers" title="Permalink to this headline">¶</a></h2>
-<p>Here is a list of built-in tokenizers:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">TokenBigram</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenUnigram</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenTrigram</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenDelimit</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenMecab</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenRegexp</span></code></li>
-</ul>
-</div></blockquote>
-<div class="section" id="tokenbigram">
-<span id="token-bigram"></span><h3>7.8.3.1. <code class="docutils literal"><span class="pre">TokenBigram</span></code><a class="headerlink" href="#tokenbigram" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> is a bigram based tokenizer. It's recommended to use
-this tokenizer for most cases.</p>
-<p>Bigram tokenize method tokenizes a text to two adjacent characters
-tokens. For example, <code class="docutils literal"><span class="pre">Hello</span></code> is tokenized to the following tokens:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">He</span></code></li>
-<li><code class="docutils literal"><span class="pre">el</span></code></li>
-<li><code class="docutils literal"><span class="pre">ll</span></code></li>
-<li><code class="docutils literal"><span class="pre">lo</span></code></li>
-</ul>
-</div></blockquote>
-<p>Bigram tokenize method is good for recall because you can find all
-texts by query consists of two or more characters.</p>
-<p>In general, you can't find all texts by query consists of one
-character because one character token doesn't exist. But you can find
-all texts by query consists of one character in Groonga. Because
-Groonga find tokens that start with query by predictive search. For
-example, Groonga can find <code class="docutils literal"><span class="pre">ll</span></code> and <code class="docutils literal"><span class="pre">lo</span></code> tokens by <code class="docutils literal"><span class="pre">l</span></code> query.</p>
-<p>Bigram tokenize method isn't good for precision because you can find
-texts that includes query in word. For example, you can find <code class="docutils literal"><span class="pre">world</span></code>
-by <code class="docutils literal"><span class="pre">or</span></code>. This is more sensitive for ASCII only languages rather than
-non-ASCII languages. <code class="docutils literal"><span class="pre">TokenBigram</span></code> has solution for this problem
-described in the below.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> behavior is different when it's worked with any
-<a class="reference internal" href="normalizers.html"><span class="doc">Normalizers</span></a>.</p>
-<p>If no normalizer is used, <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses pure bigram (all tokens
-except the last token have two characters) tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot;
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;He&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o &quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot; W&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;Wo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;or&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;rl&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ld&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;d&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>If normalizer is used, <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses white-space-separate like
-tokenize method for ASCII characters. <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses bigram
-tokenize method for non-ASCII characters.</p>
-<p>You may be confused with this combined behavior. But it's reasonable
-for most use cases such as English text (only ASCII characters) and
-Japanese text (ASCII and non-ASCII characters are mixed).</p>
-<p>Most languages consists of only ASCII characters use white-space for
-word separator. White-space-separate tokenize method is suitable for
-the case.</p>
-<p>Languages consists of non-ASCII characters don't use white-space for
-word separator. Bigram tokenize method is suitable for the case.</p>
-<p>Mixed tokenize method is suitable for mixed language case.</p>
-<p>If you want to use bigram tokenize method for ASCII character, see
-<code class="docutils literal"><span class="pre">TokenBigramSplitXXX</span></code> type tokenizers such as
-<a class="reference internal" href="#token-bigram-split-symbol-alpha"><span class="std std-ref">TokenBigramSplitSymbolAlpha</span></a>.</p>
-<p>Let's confirm <code class="docutils literal"><span class="pre">TokenBigram</span></code> behavior by example.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> uses one or more white-spaces as token delimiter for
-ASCII characters:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;hello&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;world&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> uses character type change as token delimiter for
-ASCII characters. Character type is one of them:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Alphabet</li>
-<li>Digit</li>
-<li>Symbol (such as <code class="docutils literal"><span class="pre">(</span></code>, <code class="docutils literal"><span class="pre">)</span></code> and <code class="docutils literal"><span class="pre">!</span></code>)</li>
-<li>Hiragana</li>
-<li>Katakana</li>
-<li>Kanji</li>
-<li>Others</li>
-</ul>
-</div></blockquote>
-<p>The following example shows two token delimiters:</p>
-<blockquote>
-<div><ul class="simple">
-<li>at between <code class="docutils literal"><span class="pre">100</span></code> (digits) and <code class="docutils literal"><span class="pre">cents</span></code> (alphabets)</li>
-<li>at between <code class="docutils literal"><span class="pre">cents</span></code> (alphabets) and <code class="docutils literal"><span class="pre">!!!</span></code> (symbols)</li>
-</ul>
-</div></blockquote>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is an example that <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses bigram tokenize method
-for non-ASCII characters.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;日本語の勉強&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語の&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;の勉&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;勉強&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;強&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramsplitsymbol">
-<span id="token-bigram-split-symbol"></span><h3>7.8.3.2. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code><a class="headerlink" href="#tokenbigramsplitsymbol" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The
-difference between them is symbol handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code>
-tokenizes symbols by bigram tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbol &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramsplitsymbolalpha">
-<span id="token-bigram-split-symbol-alpha"></span><h3>7.8.3.3. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code><a class="headerlink" href="#tokenbigramsplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The
-difference between them is symbol and alphabet
-handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code> tokenizes symbols and
-alphabets by bigram tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbolAlpha &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ce&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;en&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;nt&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ts&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;s!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramsplitsymbolalphadigit">
-<span id="token-bigram-split-symbol-alpha-digit"></span><h3>7.8.3.4. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code><a class="headerlink" href="#tokenbigramsplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The difference between them is symbol, alphabet
-and digit handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code> tokenizes
-symbols, alphabets and digits by bigram tokenize method. It means that
-all characters are tokenized by bigram tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbolAlphaDigit &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;10&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;00&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;0c&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ce&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;en&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;nt&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ts&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;s!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblank">
-<span id="token-bigram-ignore-blank"></span><h3>7.8.3.5. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code><a class="headerlink" href="#tokenbigramignoreblank" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The
-difference between them is blank handling. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code>
-ignores white-spaces in continuous symbols and non-ASCII characters.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
-has symbols and non-ASCII characters.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlank &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblanksplitsymbol">
-<span id="token-bigram-ignore-blank-split-symbol"></span><h3>7.8.3.6. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbol" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences between them are the followings:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Blank handling</li>
-<li>Symbol handling</li>
-</ul>
-</div></blockquote>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> ignores white-spaces in
-continuous symbols and non-ASCII characters.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> tokenizes symbols by bigram
-tokenize method.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
-has symbols and non-ASCII characters.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlankSplitSymbol &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblanksplitsymbolalpha">
-<span id="token-bigram-ignore-blank-split-symbol-alpha"></span><h3>7.8.3.7. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences between them are the followings:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Blank handling</li>
-<li>Symbol and alphabet handling</li>
-</ul>
-</div></blockquote>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> ignores white-spaces in
-continuous symbols and non-ASCII characters.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> tokenizes symbols and
-alphabets by bigram tokenize method.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
-has symbols and non-ASCII characters with white spaces and alphabets.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello 日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;hello&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlankSplitSymbolAlpha &quot;Hello 日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;he&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblanksplitsymbolalphadigit">
-<span id="token-bigram-ignore-blank-split-symbol-alpha-digit"></span><h3>7.8.3.8. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences between them are the followings:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Blank handling</li>
-<li>Symbol, alphabet and digit handling</li>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="tokenizer/summary.html">7.8.1. Summary</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram.html">7.8.2. <code class="docutils literal notranslate"><span class="pre">TokenBigram</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank.html">7.8.3. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlank</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank_split_symbol.html">7.8.4. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank_split_symbol_alpha.html">7.8.5. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank_split_symbol_alpha_digit.html">7.8.6. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_split_symbol.html">7.8.7. <code class="docutils literal notranslate"><span class="pre">TokenBigramSplitSymbol</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_split_symbol_alpha.html">7.8.8. <code class="docutils literal notranslate"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_split_symbol_alpha_digit.html">7.8.9. <code class="docutils literal notranslate"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_delimit.html">7.8.10. <code class="docutils literal notranslate"><span class="pre">TokenDelimit</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_delimit_null.html">7.8.11. <code class="docutils literal notranslate"><span class="pre">TokenDelimitNull</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_mecab.html">7.8.12. <code class="docutils literal notranslate"><span class="pre">TokenMecab</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_regexp.html">7.8.13. <code class="docutils literal notranslate"><span class="pre">TokenRegexp</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_trigram.html">7.8.14. <code class="docutils literal notranslate"><span class="pre">TokenTrigram</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_unigram.html">7.8.15. <code class="docutils literal notranslate"><span class="pre">TokenUnigram</span></code></a></li>
 </ul>
-</div></blockquote>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> ignores white-spaces
-in continuous symbols and non-ASCII characters.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> tokenizes symbols,
-alphabets and digits by bigram tokenize method. It means that all
-characters are tokenized by bigram tokenize method.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">777</span></code> text
-because it has symbols and non-ASCII characters with white spaces,
-alphabets and digits.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello 日 本 語 ! ! ! 777&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;hello&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;777&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlankSplitSymbolAlphaDigit &quot;Hello 日 本 語 ! ! ! 777&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;he&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!7&quot;
-#     },
-#     {
-#       &quot;position&quot;: 11,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;77&quot;
-#     },
-#     {
-#       &quot;position&quot;: 12,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;77&quot;
-#     },
-#     {
-#       &quot;position&quot;: 13,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;7&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenunigram">
-<span id="token-unigram"></span><h3>7.8.3.9. <code class="docutils literal"><span class="pre">TokenUnigram</span></code><a class="headerlink" href="#tokenunigram" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenUnigram</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences
-between them is token unit. <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> uses 2 characters per
-token. <code class="docutils literal"><span class="pre">TokenUnigram</span></code> uses 1 character per token.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenUnigram &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokentrigram">
-<span id="token-trigram"></span><h3>7.8.3.10. <code class="docutils literal"><span class="pre">TokenTrigram</span></code><a class="headerlink" href="#tokentrigram" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenTrigram</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences
-between them is token unit. <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> uses 2 characters per
-token. <code class="docutils literal"><span class="pre">TokenTrigram</span></code> uses 3 characters per token.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenTrigram &quot;10000cents!!!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;10000&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokendelimit">
-<span id="token-delimit"></span><h3>7.8.3.11. <code class="docutils literal"><span class="pre">TokenDelimit</span></code><a class="headerlink" href="#tokendelimit" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenDelimit</span></code> extracts token by splitting one or more space
-characters (<code class="docutils literal"><span class="pre">U+0020</span></code>). For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to
-<code class="docutils literal"><span class="pre">Hello</span></code> and <code class="docutils literal"><span class="pre">World</span></code>.</p>
-<p><code class="docutils literal"><span class="pre">TokenDelimit</span></code> is suitable for tag text. You can extract <code class="docutils literal"><span class="pre">groonga</span></code>
-and <code class="docutils literal"><span class="pre">full-text-search</span></code> and <code class="docutils literal"><span class="pre">http</span></code> as tags from <code class="docutils literal"><span class="pre">groonga</span>
-<span class="pre">full-text-search</span> <span class="pre">http</span></code>.</p>
-<p>Here is an example of <code class="docutils literal"><span class="pre">TokenDelimit</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenDelimit &quot;Groonga full-text-search HTTP&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;groonga&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;full-text-search&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;http&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokendelimitnull">
-<span id="token-delimit-null"></span><h3>7.8.3.12. <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code><a class="headerlink" href="#tokendelimitnull" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> is similar to <a class="reference internal" href="#token-delimit"><span class="std std-ref">TokenDelimit</span></a>. The
-difference between them is separator character. <a class="reference internal" href="#token-delimit"><span class="std std-ref">TokenDelimit</span></a>
-uses space character (<code class="docutils literal"><span class="pre">U+0020</span></code>) but <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> uses NUL
-character (<code class="docutils literal"><span class="pre">U+0000</span></code>).</p>
-<p><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> is also suitable for tag text.</p>
-<p>Here is an example of <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenDelimitNull &quot;Groonga\u0000full-text-search\u0000HTTP&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;groongau0000full-text-searchu0000http&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenmecab">
-<span id="token-mecab"></span><h3>7.8.3.13. <code class="docutils literal"><span class="pre">TokenMecab</span></code><a class="headerlink" href="#tokenmecab" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenMecab</span></code> is a tokenizer based on <a class="reference external" href="https://taku910.github.io/mecab/">MeCab</a> part-of-speech and
-morphological analyzer.</p>
-<p>MeCab doesn't depend on Japanese. You can use MeCab for other
-languages by creating dictionary for the languages. You can use <a class="reference external" href="http://osdn.jp/projects/naist-jdic/">NAIST
-Japanese Dictionary</a>
-for Japanese.</p>
-<p><code class="docutils literal"><span class="pre">TokenMecab</span></code> is good for precision rather than recall. You can find
-<code class="docutils literal"><span class="pre">東京都</span></code> and <code class="docutils literal"><span class="pre">京都</span></code> texts by <code class="docutils literal"><span class="pre">京都</span></code> query with
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> but <code class="docutils literal"><span class="pre">東京都</span></code> isn't expected. You can find only
-<code class="docutils literal"><span class="pre">京都</span></code> text by <code class="docutils literal"><span class="pre">京都</span></code> query with <code class="docutils literal"><span class="pre">TokenMecab</span></code>.</p>
-<p>If you want to support neologisms, you need to keep updating your
-MeCab dictionary. It needs maintain cost. (<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> doesn't
-require dictionary maintenance because <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> doesn't use
-dictionary.) <a class="reference external" href="https://github.com/neologd/mecab-ipadic-neologd">mecab-ipadic-NEologd : Neologism dictionary for MeCab</a> may help you.</p>
-<p>Here is an example of <code class="docutils literal"><span class="pre">TokenMeCab</span></code>. <code class="docutils literal"><span class="pre">東京都</span></code> is tokenized to <code class="docutils literal"><span class="pre">東京</span></code>
-and <code class="docutils literal"><span class="pre">都</span></code>. They don't include <code class="docutils literal"><span class="pre">京都</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenMecab &quot;東京都&quot;
-# [
-#   [
-#     -22,
-#     1337566253.89858,
-#     0.000355720520019531,
-#     &quot;[tokenize] nonexistent tokenizer: &lt;TokenMecab&gt;&quot;,
-#     [
-#       [
-#         &quot;create_lexicon_for_tokenize&quot;,
-#         &quot;proc_tokenize.c&quot;,
-#         139
-#       ]
-#     ]
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenregexp">
-<span id="token-regexp"></span><h3>7.8.3.14. <code class="docutils literal"><span class="pre">TokenRegexp</span></code><a class="headerlink" href="#tokenregexp" title="Permalink to this headline">¶</a></h3>
-<div class="versionadded">
-<p><span class="versionmodified">New in version 5.0.1.</span></p>
-</div>
-<div class="admonition caution">
-<p class="first admonition-title">Caution</p>
-<p class="last">This tokenizer is experimental. Specification may be changed.</p>
-</div>
-<div class="admonition caution">
-<p class="first admonition-title">Caution</p>
-<p class="last">This tokenizer can be used only with UTF-8. You can't use this
-tokenizer with EUC-JP, Shift_JIS and so on.</p>
-</div>
-<p><code class="docutils literal"><span class="pre">TokenRegexp</span></code> is a tokenizer for supporting regular expression
-search by index.</p>
-<p>In general, regular expression search is evaluated as sequential
-search. But the following cases can be evaluated as index search:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Literal only case such as <code class="docutils literal"><span class="pre">hello</span></code></li>
-<li>The beginning of text and literal case such as <code class="docutils literal"><span class="pre">\A/home/alice</span></code></li>
-<li>The end of text and literal case such as <code class="docutils literal"><span class="pre">\.txt\z</span></code></li>
-</ul>
-</div></blockquote>
-<p>In most cases, index search is faster than sequential search.</p>
-<p><code class="docutils literal"><span class="pre">TokenRegexp</span></code> is based on bigram tokenize method. <code class="docutils literal"><span class="pre">TokenRegexp</span></code>
-adds the beginning of text mark (<code class="docutils literal"><span class="pre">U+FFEF</span></code>) at the begging of text
-and the end of text mark (<code class="docutils literal"><span class="pre">U+FFF0</span></code>) to the end of text when you
-index text:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenRegexp &quot;/home/alice/test.txt&quot; NormalizerAuto --mode ADD
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;￯&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;/h&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ho&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;om&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;me&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;e/&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;/a&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;al&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;li&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ic&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ce&quot;
-#     },
-#     {
-#       &quot;position&quot;: 11,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;e/&quot;
-#     },
-#     {
-#       &quot;position&quot;: 12,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;/t&quot;
-#     },
-#     {
-#       &quot;position&quot;: 13,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;te&quot;
-#     },
-#     {
-#       &quot;position&quot;: 14,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;es&quot;
-#     },
-#     {
-#       &quot;position&quot;: 15,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;st&quot;
-#     },
-#     {
-#       &quot;position&quot;: 16,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;t.&quot;
-#     },
-#     {
-#       &quot;position&quot;: 17,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;.t&quot;
-#     },
-#     {
-#       &quot;position&quot;: 18,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;tx&quot;
-#     },
-#     {
-#       &quot;position&quot;: 19,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;xt&quot;
-#     },
-#     {
-#       &quot;position&quot;: 20,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;t&quot;
-#     },
-#     {
-#       &quot;position&quot;: 21,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;￰&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
 </div>
 </div>
@@ -1476,46 +89,20 @@ index text:</p>
       </div>
       <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
         <div class="sphinxsidebarwrapper">
-  <h3><a href="../index.html">Table Of Contents</a></h3>
-  <ul>
-<li><a class="reference internal" href="#">7.8. Tokenizers</a><ul>
-<li><a class="reference internal" href="#summary">7.8.1. Summary</a></li>
-<li><a class="reference internal" href="#what-is-tokenize">7.8.2. What is &quot;tokenize&quot;?</a></li>
-<li><a class="reference internal" href="#built-in-tokenizsers">7.8.3. Built-in tokenizsers</a><ul>
-<li><a class="reference internal" href="#tokenbigram">7.8.3.1. <code class="docutils literal"><span class="pre">TokenBigram</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbol">7.8.3.2. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbolalpha">7.8.3.3. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbolalphadigit">7.8.3.4. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblank">7.8.3.5. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbol">7.8.3.6. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalpha">7.8.3.7. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalphadigit">7.8.3.8. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code></a></li>
-<li><a class="reference internal" href="#tokenunigram">7.8.3.9. <code class="docutils literal"><span class="pre">TokenUnigram</span></code></a></li>
-<li><a class="reference internal" href="#tokentrigram">7.8.3.10. <code class="docutils literal"><span class="pre">TokenTrigram</span></code></a></li>
-<li><a class="reference internal" href="#tokendelimit">7.8.3.11. <code class="docutils literal"><span class="pre">TokenDelimit</span></code></a></li>
-<li><a class="reference internal" href="#tokendelimitnull">7.8.3.12. <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code></a></li>
-<li><a class="reference internal" href="#tokenmecab">7.8.3.13. <code class="docutils literal"><span class="pre">TokenMecab</span></code></a></li>
-<li><a class="reference internal" href="#tokenregexp">7.8.3.14. <code class="docutils literal"><span class="pre">TokenRegexp</span></code></a></li>
-</ul>
-</li>
-</ul>
-</li>
-</ul>
   <h4>Previous topic</h4>
-  <p class="topless"><a href="normalizers.html"
-                        title="previous chapter">7.7. Normalizers</a></p>
+  <p class="topless"><a href="normalizers/normalizer_nfkc51.html"
+                        title="previous chapter">7.7.2.3. <code class="docutils literal notranslate"><span class="pre">NormalizerNFKC51</span></code></a></p>
   <h4>Next topic</h4>
-  <p class="topless"><a href="token_filters.html"
-                        title="next chapter">7.9. Token filters</a></p>
+  <p class="topless"><a href="tokenizer/summary.html"
+                        title="next chapter">7.8.1. Summary</a></p>
 <div id="searchbox" style="display: none" role="search">
   <h3>Quick search</h3>
+    <div class="searchformwrapper">
     <form class="search" action="../search.html" method="get">
-      <div><input type="text" name="q" /></div>
-      <div><input type="submit" value="Go" /></div>
-      <input type="hidden" name="check_keywords" value="yes" />
-      <input type="hidden" name="area" value="default" />
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
     </form>
+    </div>
 </div>
 <script type="text/javascript">$('#searchbox').show(0);</script>
         </div>
@@ -1529,17 +116,17 @@ index text:</p>
           <a href="../genindex.html" title="General Index"
              >index</a></li>
         <li class="right" >
-          <a href="token_filters.html" title="7.9. Token filters"
+          <a href="tokenizer/summary.html" title="7.8.1. Summary"
              >next</a> |</li>
         <li class="right" >
-          <a href="normalizers.html" title="7.7. Normalizers"
+          <a href="normalizers/normalizer_nfkc51.html" title="7.7.2.3. NormalizerNFKC51"
              >previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../reference.html" >7. Reference manual</a> &#187;</li>
       </ul>
     </div>
     <div class="footer" role="contentinfo">
-        &#169; Copyright 2009-2018, Brazil, Inc.
+        &#169; Copyright 2009-2019, Brazil, Inc.
     </div>
   </body>
 </html>