RubyGems - rroonga - Versions diffs - 7.1.1-x64-mingw32 → 9.0.2-x64-mingw32 - Mend

rroonga 7.1.1-x64-mingw32 → 9.0.2-x64-mingw32

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (980) hide show

data/vendor/local/share/doc/groonga/en/html/reference/tokenizer/summary.html ADDED Viewed

@@ -0,0 +1,259 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <title>7.8.1. Summary &#8212; Groonga v9.0.2 documentation</title>
+    <link rel="stylesheet" href="../../_static/groonga.css" type="text/css" />
+    <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
+    <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
+    <script type="text/javascript" src="../../_static/jquery.js"></script>
+    <script type="text/javascript" src="../../_static/underscore.js"></script>
+    <script type="text/javascript" src="../../_static/doctools.js"></script>
+    <script type="text/javascript" src="../../_static/language_data.js"></script>
+    <link rel="shortcut icon" href="../../_static/favicon.ico"/>
+    <link rel="index" title="Index" href="../../genindex.html" />
+    <link rel="search" title="Search" href="../../search.html" />
+    <link rel="next" title="7.8.2. TokenBigram" href="../tokenizers/token_bigram.html" />
+    <link rel="prev" title="7.8. Tokenizers" href="../tokenizers.html" />
+  </head><body>
+<div class="header">
+  <h1 class="title">
+    <a id="top-link" href="../../index.html">
+      <span class="project">groonga</span>
+      <span class="separator">-</span>
+      <span class="description">An open-source fulltext search engine and column store.</span>
+    </a>
+  </h1>
+  <div class="other-language-links">
+    <ul>
+      <li><a href="../../../../ja/html/reference/tokenizer/summary.html">日本語</a></li>
+    </ul>
+  </div>
+</div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../../genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="../tokenizers/token_bigram.html" title="7.8.2. TokenBigram"
+             accesskey="N">next</a> |</li>
+        <li class="right" >
+          <a href="../tokenizers.html" title="7.8. Tokenizers"
+             accesskey="P">previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
+          <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
+          <li class="nav-item nav-item-2"><a href="../tokenizers.html" accesskey="U">7.8. Tokenizers</a> &#187;</li>
+      </ul>
+    </div>
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+  <div class="section" id="summary">
+<h1>7.8.1. Summary<a class="headerlink" href="#summary" title="Permalink to this headline">¶</a></h1>
+<p>Groonga has tokenizer module that tokenizes text. It is used when
+the following cases:</p>
+<blockquote>
+<div><ul>
+<li><p>Indexing text</p>
+<div class="figure align-center" id="id1">
+<a class="reference internal image-reference" href="../../_images/used-when-indexing.png"><img alt="../../_images/used-when-indexing.png" src="../../_images/used-when-indexing.png" style="width: 80%;" /></a>
+<p class="caption"><span class="caption-text">Tokenizer is used when indexing text.</span><a class="headerlink" href="#id1" title="Permalink to this image">¶</a></p>
+</div>
+</li>
+<li><p>Searching by query</p>
+<div class="figure align-center" id="id2">
+<a class="reference internal image-reference" href="../../_images/used-when-searching.png"><img alt="../../_images/used-when-searching.png" src="../../_images/used-when-searching.png" style="width: 80%;" /></a>
+<p class="caption"><span class="caption-text">Tokenizer is used when searching by query.</span><a class="headerlink" href="#id2" title="Permalink to this image">¶</a></p>
+</div>
+</li>
+</ul>
+</div></blockquote>
+<p>Tokenizer is an important module for full-text search. You can change
+trade-off between <a class="reference external" href="http://en.wikipedia.org/wiki/Precision_and_recall">precision and recall</a> by changing
+tokenizer.</p>
+<p>Normally, <a class="reference internal" href="../tokenizers/token_bigram.html#token-bigram"><span class="std std-ref">TokenBigram</span></a> is a suitable tokenizer. If you don’t
+know much about tokenizer, it’s recommended that you choose
+<a class="reference internal" href="../tokenizers/token_bigram.html#token-bigram"><span class="std std-ref">TokenBigram</span></a>.</p>
+<p>You can try a tokenizer by <a class="reference internal" href="../commands/tokenize.html"><span class="doc">tokenize</span></a> and
+<a class="reference internal" href="../commands/table_tokenize.html"><span class="doc">table_tokenize</span></a>. Here is an example to
+try <a class="reference internal" href="../tokenizers/token_bigram.html#token-bigram"><span class="std std-ref">TokenBigram</span></a> tokenizer by
+<a class="reference internal" href="../commands/tokenize.html"><span class="doc">tokenize</span></a>:</p>
+<p>Execution example:</p>
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot;
+# [
+#   [
+#     0,
+#     1337566253.89858,
+#     0.000355720520019531
+#   ],
+#   [
+#     {
+#       &quot;position&quot;: 0,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;He&quot;
+#     },
+#     {
+#       &quot;position&quot;: 1,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;el&quot;
+#     },
+#     {
+#       &quot;position&quot;: 2,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;ll&quot;
+#     },
+#     {
+#       &quot;position&quot;: 3,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;lo&quot;
+#     },
+#     {
+#       &quot;position&quot;: 4,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;o &quot;
+#     },
+#     {
+#       &quot;position&quot;: 5,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot; W&quot;
+#     },
+#     {
+#       &quot;position&quot;: 6,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;Wo&quot;
+#     },
+#     {
+#       &quot;position&quot;: 7,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;or&quot;
+#     },
+#     {
+#       &quot;position&quot;: 8,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;rl&quot;
+#     },
+#     {
+#       &quot;position&quot;: 9,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;ld&quot;
+#     },
+#     {
+#       &quot;position&quot;: 10,
+#       &quot;force_prefix&quot;: false,
+#       &quot;value&quot;: &quot;d&quot;
+#     }
+#   ]
+# ]
+</pre></div>
+</div>
+<p>“tokenize” is the process that extracts zero or more tokens from a
+text. There are some “tokenize” methods.</p>
+<p>For example, <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
+bigram tokenize method:</p>
+<blockquote>
+<div><ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">He</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">el</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">ll</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">lo</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">o_</span></code> (<code class="docutils literal notranslate"><span class="pre">_</span></code> means a white-space)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">_W</span></code> (<code class="docutils literal notranslate"><span class="pre">_</span></code> means a white-space)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">Wo</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">or</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">rl</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">ld</span></code></p></li>
+</ul>
+</div></blockquote>
+<p>In the above example, 10 tokens are extracted from one text <code class="docutils literal notranslate"><span class="pre">Hello</span>
+<span class="pre">World</span></code>.</p>
+<p>For example, <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
+white-space-separate tokenize method:</p>
+<blockquote>
+<div><ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">Hello</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">World</span></code></p></li>
+</ul>
+</div></blockquote>
+<p>In the above example, 2 tokens are extracted from one text <code class="docutils literal notranslate"><span class="pre">Hello</span>
+<span class="pre">World</span></code>.</p>
+<p>Token is used as search key. You can find indexed documents only by
+tokens that are extracted by used tokenize method. For example, you
+can find <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal notranslate"><span class="pre">ll</span></code> with bigram tokenize method but you
+can’t find <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal notranslate"><span class="pre">ll</span></code> with white-space-separate tokenize
+method. Because white-space-separate tokenize method doesn’t extract
+<code class="docutils literal notranslate"><span class="pre">ll</span></code> token. It just extracts <code class="docutils literal notranslate"><span class="pre">Hello</span></code> and <code class="docutils literal notranslate"><span class="pre">World</span></code> tokens.</p>
+<p>In general, tokenize method that generates small tokens increases
+recall but decreases precision. Tokenize method that generates large
+tokens increases precision but decreases recall.</p>
+<p>For example, we can find <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> and <code class="docutils literal notranslate"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal notranslate"><span class="pre">or</span></code> with
+bigram tokenize method. <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> is a noise for people who
+wants to search “logical and”. It means that precision is
+decreased. But recall is increased.</p>
+<p>We can find only <code class="docutils literal notranslate"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal notranslate"><span class="pre">or</span></code> with white-space-separate
+tokenize method. Because <code class="docutils literal notranslate"><span class="pre">World</span></code> is tokenized to one token <code class="docutils literal notranslate"><span class="pre">World</span></code>
+with white-space-separate tokenize method. It means that precision is
+increased for people who wants to search “logical and”. But recall is
+decreased because <code class="docutils literal notranslate"><span class="pre">Hello</span> <span class="pre">World</span></code> that contains <code class="docutils literal notranslate"><span class="pre">or</span></code> isn’t found.</p>
+</div>
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+  <h4>Previous topic</h4>
+  <p class="topless"><a href="../tokenizers.html"
+                        title="previous chapter">7.8. Tokenizers</a></p>
+  <h4>Next topic</h4>
+  <p class="topless"><a href="../tokenizers/token_bigram.html"
+                        title="next chapter">7.8.2. <code class="docutils literal notranslate"><span class="pre">TokenBigram</span></code></a></p>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <div class="searchformwrapper">
+    <form class="search" action="../../search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+    </form>
+    </div>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../../genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="../tokenizers/token_bigram.html" title="7.8.2. TokenBigram"
+             >next</a> |</li>
+        <li class="right" >
+          <a href="../tokenizers.html" title="7.8. Tokenizers"
+             >previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
+          <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
+          <li class="nav-item nav-item-2"><a href="../tokenizers.html" >7.8. Tokenizers</a> &#187;</li>
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &#169; Copyright 2009-2019, Brazil, Inc.
+    </div>
+  </body>
+</html>

data/vendor/local/share/doc/groonga/en/html/reference/tokenizers.html CHANGED Viewed

@@ -1,34 +1,26 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
   <head>
-    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-    <title>7.8. Tokenizers &#8212; Groonga v7.1.0-73-g6d02cfa documentation</title>
+    <meta charset="utf-8" />
+    <title>7.8. Tokenizers &#8212; Groonga v9.0.2 documentation</title>
     <link rel="stylesheet" href="../_static/groonga.css" type="text/css" />
     <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-    <script type="text/javascript">
-      var DOCUMENTATION_OPTIONS = {
-        URL_ROOT:    '../',
-        VERSION:     '7.1.0-73-g6d02cfa',
-        COLLAPSE_INDEX: false,
-        FILE_SUFFIX: '.html',
-        HAS_SOURCE:  false,
-        SOURCELINK_SUFFIX: '.txt'
-      };
-    </script>
+    <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
     <script type="text/javascript" src="../_static/jquery.js"></script>
     <script type="text/javascript" src="../_static/underscore.js"></script>
     <script type="text/javascript" src="../_static/doctools.js"></script>
+    <script type="text/javascript" src="../_static/language_data.js"></script>
     <link rel="shortcut icon" href="../_static/favicon.ico"/>
     <link rel="index" title="Index" href="../genindex.html" />
     <link rel="search" title="Search" href="../search.html" />
-    <link rel="next" title="7.9. Token filters" href="token_filters.html" />
-    <link rel="prev" title="7.7. Normalizers" href="normalizers.html" />
-  </head>
-  <body>
+    <link rel="next" title="7.8.1. Summary" href="tokenizer/summary.html" />
+    <link rel="prev" title="7.7.2.3. NormalizerNFKC51" href="normalizers/normalizer_nfkc51.html" />
+  </head><body>
 <div class="header">
   <h1 class="title">
     <a id="top-link" href="../index.html">
@@ -53,12 +45,12 @@
           <a href="../genindex.html" title="General Index"
              accesskey="I">index</a></li>
         <li class="right" >
-          <a href="token_filters.html" title="7.9. Token filters"
+          <a href="tokenizer/summary.html" title="7.8.1. Summary"
              accesskey="N">next</a> |</li>
         <li class="right" >
-          <a href="normalizers.html" title="7.7. Normalizers"
+          <a href="normalizers/normalizer_nfkc51.html" title="7.7.2.3. NormalizerNFKC51"
              accesskey="P">previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../reference.html" accesskey="U">7. Reference manual</a> &#187;</li>
       </ul>
     </div>
@@ -70,1403 +62,24 @@
   <div class="section" id="tokenizers">
 <h1>7.8. Tokenizers<a class="headerlink" href="#tokenizers" title="Permalink to this headline">¶</a></h1>
-<div class="section" id="summary">
-<h2>7.8.1. Summary<a class="headerlink" href="#summary" title="Permalink to this headline">¶</a></h2>
-<p>Groonga has tokenizer module that tokenizes text. It is used when
-the following cases:</p>
-<blockquote>
-<div><ul>
-<li><p class="first">Indexing text</p>
-<div class="figure align-center" id="id1">
-<a class="reference internal image-reference" href="../_images/used-when-indexing.png"><img alt="../_images/used-when-indexing.png" src="../_images/used-when-indexing.png" style="width: 80%;" /></a>
-<p class="caption"><span class="caption-text">Tokenizer is used when indexing text.</span></p>
-</div>
-</li>
-<li><p class="first">Searching by query</p>
-<div class="figure align-center" id="id2">
-<a class="reference internal image-reference" href="../_images/used-when-searching.png"><img alt="../_images/used-when-searching.png" src="../_images/used-when-searching.png" style="width: 80%;" /></a>
-<p class="caption"><span class="caption-text">Tokenizer is used when searching by query.</span></p>
-</div>
-</li>
-</ul>
-</div></blockquote>
-<p>Tokenizer is an important module for full-text search. You can change
-trade-off between <a class="reference external" href="http://en.wikipedia.org/wiki/Precision_and_recall">precision and recall</a> by changing
-tokenizer.</p>
-<p>Normally, <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> is a suitable tokenizer. If you don't
-know much about tokenizer, it's recommended that you choose
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>.</p>
-<p>You can try a tokenizer by <a class="reference internal" href="commands/tokenize.html"><span class="doc">tokenize</span></a> and
-<a class="reference internal" href="commands/table_tokenize.html"><span class="doc">table_tokenize</span></a>. Here is an example to
-try <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> tokenizer by
-<a class="reference internal" href="commands/tokenize.html"><span class="doc">tokenize</span></a>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot;
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;He&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o &quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot; W&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;Wo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;or&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;rl&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ld&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;d&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="what-is-tokenize">
-<h2>7.8.2. What is &quot;tokenize&quot;?<a class="headerlink" href="#what-is-tokenize" title="Permalink to this headline">¶</a></h2>
-<p>&quot;tokenize&quot; is the process that extracts zero or more tokens from a
-text. There are some &quot;tokenize&quot; methods.</p>
-<p>For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
-bigram tokenize method:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">He</span></code></li>
-<li><code class="docutils literal"><span class="pre">el</span></code></li>
-<li><code class="docutils literal"><span class="pre">ll</span></code></li>
-<li><code class="docutils literal"><span class="pre">lo</span></code></li>
-<li><code class="docutils literal"><span class="pre">o_</span></code> (<code class="docutils literal"><span class="pre">_</span></code> means a white-space)</li>
-<li><code class="docutils literal"><span class="pre">_W</span></code> (<code class="docutils literal"><span class="pre">_</span></code> means a white-space)</li>
-<li><code class="docutils literal"><span class="pre">Wo</span></code></li>
-<li><code class="docutils literal"><span class="pre">or</span></code></li>
-<li><code class="docutils literal"><span class="pre">rl</span></code></li>
-<li><code class="docutils literal"><span class="pre">ld</span></code></li>
-</ul>
-</div></blockquote>
-<p>In the above example, 10 tokens are extracted from one text <code class="docutils literal"><span class="pre">Hello</span>
-<span class="pre">World</span></code>.</p>
-<p>For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to the following tokens by
-white-space-separate tokenize method:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">Hello</span></code></li>
-<li><code class="docutils literal"><span class="pre">World</span></code></li>
-</ul>
-</div></blockquote>
-<p>In the above example, 2 tokens are extracted from one text <code class="docutils literal"><span class="pre">Hello</span>
-<span class="pre">World</span></code>.</p>
-<p>Token is used as search key. You can find indexed documents only by
-tokens that are extracted by used tokenize method. For example, you
-can find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal"><span class="pre">ll</span></code> with bigram tokenize method but you
-can't find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> by <code class="docutils literal"><span class="pre">ll</span></code> with white-space-separate tokenize
-method. Because white-space-separate tokenize method doesn't extract
-<code class="docutils literal"><span class="pre">ll</span></code> token. It just extracts <code class="docutils literal"><span class="pre">Hello</span></code> and <code class="docutils literal"><span class="pre">World</span></code> tokens.</p>
-<p>In general, tokenize method that generates small tokens increases
-recall but decreases precision. Tokenize method that generates large
-tokens increases precision but decreases recall.</p>
-<p>For example, we can find <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> and <code class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal"><span class="pre">or</span></code> with
-bigram tokenize method. <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is a noise for people who
-wants to search &quot;logical and&quot;. It means that precision is
-decreased. But recall is increased.</p>
-<p>We can find only <code class="docutils literal"><span class="pre">A</span> <span class="pre">or</span> <span class="pre">B</span></code> by <code class="docutils literal"><span class="pre">or</span></code> with white-space-separate
-tokenize method. Because <code class="docutils literal"><span class="pre">World</span></code> is tokenized to one token <code class="docutils literal"><span class="pre">World</span></code>
-with white-space-separate tokenize method. It means that precision is
-increased for people who wants to search &quot;logical and&quot;. But recall is
-decreased because <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> that contains <code class="docutils literal"><span class="pre">or</span></code> isn't found.</p>
-</div>
-<div class="section" id="built-in-tokenizsers">
-<h2>7.8.3. Built-in tokenizsers<a class="headerlink" href="#built-in-tokenizsers" title="Permalink to this headline">¶</a></h2>
-<p>Here is a list of built-in tokenizers:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">TokenBigram</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenUnigram</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenTrigram</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenDelimit</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenMecab</span></code></li>
-<li><code class="docutils literal"><span class="pre">TokenRegexp</span></code></li>
-</ul>
-</div></blockquote>
-<div class="section" id="tokenbigram">
-<span id="token-bigram"></span><h3>7.8.3.1. <code class="docutils literal"><span class="pre">TokenBigram</span></code><a class="headerlink" href="#tokenbigram" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> is a bigram based tokenizer. It's recommended to use
-this tokenizer for most cases.</p>
-<p>Bigram tokenize method tokenizes a text to two adjacent characters
-tokens. For example, <code class="docutils literal"><span class="pre">Hello</span></code> is tokenized to the following tokens:</p>
-<blockquote>
-<div><ul class="simple">
-<li><code class="docutils literal"><span class="pre">He</span></code></li>
-<li><code class="docutils literal"><span class="pre">el</span></code></li>
-<li><code class="docutils literal"><span class="pre">ll</span></code></li>
-<li><code class="docutils literal"><span class="pre">lo</span></code></li>
-</ul>
-</div></blockquote>
-<p>Bigram tokenize method is good for recall because you can find all
-texts by query consists of two or more characters.</p>
-<p>In general, you can't find all texts by query consists of one
-character because one character token doesn't exist. But you can find
-all texts by query consists of one character in Groonga. Because
-Groonga find tokens that start with query by predictive search. For
-example, Groonga can find <code class="docutils literal"><span class="pre">ll</span></code> and <code class="docutils literal"><span class="pre">lo</span></code> tokens by <code class="docutils literal"><span class="pre">l</span></code> query.</p>
-<p>Bigram tokenize method isn't good for precision because you can find
-texts that includes query in word. For example, you can find <code class="docutils literal"><span class="pre">world</span></code>
-by <code class="docutils literal"><span class="pre">or</span></code>. This is more sensitive for ASCII only languages rather than
-non-ASCII languages. <code class="docutils literal"><span class="pre">TokenBigram</span></code> has solution for this problem
-described in the below.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> behavior is different when it's worked with any
-<a class="reference internal" href="normalizers.html"><span class="doc">Normalizers</span></a>.</p>
-<p>If no normalizer is used, <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses pure bigram (all tokens
-except the last token have two characters) tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot;
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;He&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o &quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot; W&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;Wo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;or&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;rl&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ld&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;d&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>If normalizer is used, <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses white-space-separate like
-tokenize method for ASCII characters. <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses bigram
-tokenize method for non-ASCII characters.</p>
-<p>You may be confused with this combined behavior. But it's reasonable
-for most use cases such as English text (only ASCII characters) and
-Japanese text (ASCII and non-ASCII characters are mixed).</p>
-<p>Most languages consists of only ASCII characters use white-space for
-word separator. White-space-separate tokenize method is suitable for
-the case.</p>
-<p>Languages consists of non-ASCII characters don't use white-space for
-word separator. Bigram tokenize method is suitable for the case.</p>
-<p>Mixed tokenize method is suitable for mixed language case.</p>
-<p>If you want to use bigram tokenize method for ASCII character, see
-<code class="docutils literal"><span class="pre">TokenBigramSplitXXX</span></code> type tokenizers such as
-<a class="reference internal" href="#token-bigram-split-symbol-alpha"><span class="std std-ref">TokenBigramSplitSymbolAlpha</span></a>.</p>
-<p>Let's confirm <code class="docutils literal"><span class="pre">TokenBigram</span></code> behavior by example.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> uses one or more white-spaces as token delimiter for
-ASCII characters:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello World&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;hello&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;world&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p><code class="docutils literal"><span class="pre">TokenBigram</span></code> uses character type change as token delimiter for
-ASCII characters. Character type is one of them:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Alphabet</li>
-<li>Digit</li>
-<li>Symbol (such as <code class="docutils literal"><span class="pre">(</span></code>, <code class="docutils literal"><span class="pre">)</span></code> and <code class="docutils literal"><span class="pre">!</span></code>)</li>
-<li>Hiragana</li>
-<li>Katakana</li>
-<li>Kanji</li>
-<li>Others</li>
-</ul>
-</div></blockquote>
-<p>The following example shows two token delimiters:</p>
-<blockquote>
-<div><ul class="simple">
-<li>at between <code class="docutils literal"><span class="pre">100</span></code> (digits) and <code class="docutils literal"><span class="pre">cents</span></code> (alphabets)</li>
-<li>at between <code class="docutils literal"><span class="pre">cents</span></code> (alphabets) and <code class="docutils literal"><span class="pre">!!!</span></code> (symbols)</li>
-</ul>
-</div></blockquote>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is an example that <code class="docutils literal"><span class="pre">TokenBigram</span></code> uses bigram tokenize method
-for non-ASCII characters.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;日本語の勉強&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語の&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;の勉&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;勉強&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;強&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramsplitsymbol">
-<span id="token-bigram-split-symbol"></span><h3>7.8.3.2. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code><a class="headerlink" href="#tokenbigramsplitsymbol" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The
-difference between them is symbol handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code>
-tokenizes symbols by bigram tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbol &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramsplitsymbolalpha">
-<span id="token-bigram-split-symbol-alpha"></span><h3>7.8.3.3. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code><a class="headerlink" href="#tokenbigramsplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The
-difference between them is symbol and alphabet
-handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code> tokenizes symbols and
-alphabets by bigram tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbolAlpha &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ce&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;en&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;nt&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ts&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;s!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramsplitsymbolalphadigit">
-<span id="token-bigram-split-symbol-alpha-digit"></span><h3>7.8.3.4. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code><a class="headerlink" href="#tokenbigramsplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The difference between them is symbol, alphabet
-and digit handling. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code> tokenizes
-symbols, alphabets and digits by bigram tokenize method. It means that
-all characters are tokenized by bigram tokenize method:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbolAlphaDigit &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;10&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;00&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;0c&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ce&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;en&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;nt&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ts&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;s!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblank">
-<span id="token-bigram-ignore-blank"></span><h3>7.8.3.5. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code><a class="headerlink" href="#tokenbigramignoreblank" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The
-difference between them is blank handling. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code>
-ignores white-spaces in continuous symbols and non-ASCII characters.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
-has symbols and non-ASCII characters.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlank &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblanksplitsymbol">
-<span id="token-bigram-ignore-blank-split-symbol"></span><h3>7.8.3.6. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbol" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences between them are the followings:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Blank handling</li>
-<li>Symbol handling</li>
-</ul>
-</div></blockquote>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> ignores white-spaces in
-continuous symbols and non-ASCII characters.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code> tokenizes symbols by bigram
-tokenize method.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
-has symbols and non-ASCII characters.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlankSplitSymbol &quot;日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblanksplitsymbolalpha">
-<span id="token-bigram-ignore-blank-split-symbol-alpha"></span><h3>7.8.3.7. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalpha" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences between them are the followings:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Blank handling</li>
-<li>Symbol and alphabet handling</li>
-</ul>
-</div></blockquote>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> ignores white-spaces in
-continuous symbols and non-ASCII characters.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code> tokenizes symbols and
-alphabets by bigram tokenize method.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span></code> text because it
-has symbols and non-ASCII characters with white spaces and alphabets.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello 日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;hello&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlankSplitSymbolAlpha &quot;Hello 日 本 語 ! ! !&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;he&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenbigramignoreblanksplitsymbolalphadigit">
-<span id="token-bigram-ignore-blank-split-symbol-alpha-digit"></span><h3>7.8.3.8. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code><a class="headerlink" href="#tokenbigramignoreblanksplitsymbolalphadigit" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> is similar to
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences between them are the followings:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Blank handling</li>
-<li>Symbol, alphabet and digit handling</li>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="tokenizer/summary.html">7.8.1. Summary</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram.html">7.8.2. <code class="docutils literal notranslate"><span class="pre">TokenBigram</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank.html">7.8.3. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlank</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank_split_symbol.html">7.8.4. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank_split_symbol_alpha.html">7.8.5. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_ignore_blank_split_symbol_alpha_digit.html">7.8.6. <code class="docutils literal notranslate"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_split_symbol.html">7.8.7. <code class="docutils literal notranslate"><span class="pre">TokenBigramSplitSymbol</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_split_symbol_alpha.html">7.8.8. <code class="docutils literal notranslate"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_bigram_split_symbol_alpha_digit.html">7.8.9. <code class="docutils literal notranslate"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_delimit.html">7.8.10. <code class="docutils literal notranslate"><span class="pre">TokenDelimit</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_delimit_null.html">7.8.11. <code class="docutils literal notranslate"><span class="pre">TokenDelimitNull</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_mecab.html">7.8.12. <code class="docutils literal notranslate"><span class="pre">TokenMecab</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_regexp.html">7.8.13. <code class="docutils literal notranslate"><span class="pre">TokenRegexp</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_trigram.html">7.8.14. <code class="docutils literal notranslate"><span class="pre">TokenTrigram</span></code></a></li>
+<li class="toctree-l1"><a class="reference internal" href="tokenizers/token_unigram.html">7.8.15. <code class="docutils literal notranslate"><span class="pre">TokenUnigram</span></code></a></li>
 </ul>
-</div></blockquote>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> ignores white-spaces
-in continuous symbols and non-ASCII characters.</p>
-<p><code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code> tokenizes symbols,
-alphabets and digits by bigram tokenize method. It means that all
-characters are tokenized by bigram tokenize method.</p>
-<p>You can find difference of them by <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">日</span> <span class="pre">本</span> <span class="pre">語</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">!</span> <span class="pre">777</span></code> text
-because it has symbols and non-ASCII characters with white spaces,
-alphabets and digits.</p>
-<p>Here is a result by <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> :</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Hello 日 本 語 ! ! ! 777&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;hello&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;777&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-<p>Here is a result by <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramIgnoreBlankSplitSymbolAlphaDigit &quot;Hello 日 本 語 ! ! ! 777&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;he&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;el&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ll&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;lo&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;o日&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;日本&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;本語&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;語!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!7&quot;
-#     },
-#     {
-#       &quot;position&quot;: 11,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;77&quot;
-#     },
-#     {
-#       &quot;position&quot;: 12,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;77&quot;
-#     },
-#     {
-#       &quot;position&quot;: 13,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;7&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenunigram">
-<span id="token-unigram"></span><h3>7.8.3.9. <code class="docutils literal"><span class="pre">TokenUnigram</span></code><a class="headerlink" href="#tokenunigram" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenUnigram</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences
-between them is token unit. <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> uses 2 characters per
-token. <code class="docutils literal"><span class="pre">TokenUnigram</span></code> uses 1 character per token.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenUnigram &quot;100cents!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;100&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokentrigram">
-<span id="token-trigram"></span><h3>7.8.3.10. <code class="docutils literal"><span class="pre">TokenTrigram</span></code><a class="headerlink" href="#tokentrigram" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenTrigram</span></code> is similar to <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a>. The differences
-between them is token unit. <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> uses 2 characters per
-token. <code class="docutils literal"><span class="pre">TokenTrigram</span></code> uses 3 characters per token.</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenTrigram &quot;10000cents!!!!!&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;10000&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;cents&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;!!!!!&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokendelimit">
-<span id="token-delimit"></span><h3>7.8.3.11. <code class="docutils literal"><span class="pre">TokenDelimit</span></code><a class="headerlink" href="#tokendelimit" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenDelimit</span></code> extracts token by splitting one or more space
-characters (<code class="docutils literal"><span class="pre">U+0020</span></code>). For example, <code class="docutils literal"><span class="pre">Hello</span> <span class="pre">World</span></code> is tokenized to
-<code class="docutils literal"><span class="pre">Hello</span></code> and <code class="docutils literal"><span class="pre">World</span></code>.</p>
-<p><code class="docutils literal"><span class="pre">TokenDelimit</span></code> is suitable for tag text. You can extract <code class="docutils literal"><span class="pre">groonga</span></code>
-and <code class="docutils literal"><span class="pre">full-text-search</span></code> and <code class="docutils literal"><span class="pre">http</span></code> as tags from <code class="docutils literal"><span class="pre">groonga</span>
-<span class="pre">full-text-search</span> <span class="pre">http</span></code>.</p>
-<p>Here is an example of <code class="docutils literal"><span class="pre">TokenDelimit</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenDelimit &quot;Groonga full-text-search HTTP&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;groonga&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;full-text-search&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;http&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokendelimitnull">
-<span id="token-delimit-null"></span><h3>7.8.3.12. <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code><a class="headerlink" href="#tokendelimitnull" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> is similar to <a class="reference internal" href="#token-delimit"><span class="std std-ref">TokenDelimit</span></a>. The
-difference between them is separator character. <a class="reference internal" href="#token-delimit"><span class="std std-ref">TokenDelimit</span></a>
-uses space character (<code class="docutils literal"><span class="pre">U+0020</span></code>) but <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> uses NUL
-character (<code class="docutils literal"><span class="pre">U+0000</span></code>).</p>
-<p><code class="docutils literal"><span class="pre">TokenDelimitNull</span></code> is also suitable for tag text.</p>
-<p>Here is an example of <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenDelimitNull &quot;Groonga\u0000full-text-search\u0000HTTP&quot; NormalizerAuto
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;groongau0000full-text-searchu0000http&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenmecab">
-<span id="token-mecab"></span><h3>7.8.3.13. <code class="docutils literal"><span class="pre">TokenMecab</span></code><a class="headerlink" href="#tokenmecab" title="Permalink to this headline">¶</a></h3>
-<p><code class="docutils literal"><span class="pre">TokenMecab</span></code> is a tokenizer based on <a class="reference external" href="https://taku910.github.io/mecab/">MeCab</a> part-of-speech and
-morphological analyzer.</p>
-<p>MeCab doesn't depend on Japanese. You can use MeCab for other
-languages by creating dictionary for the languages. You can use <a class="reference external" href="http://osdn.jp/projects/naist-jdic/">NAIST
-Japanese Dictionary</a>
-for Japanese.</p>
-<p><code class="docutils literal"><span class="pre">TokenMecab</span></code> is good for precision rather than recall. You can find
-<code class="docutils literal"><span class="pre">東京都</span></code> and <code class="docutils literal"><span class="pre">京都</span></code> texts by <code class="docutils literal"><span class="pre">京都</span></code> query with
-<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> but <code class="docutils literal"><span class="pre">東京都</span></code> isn't expected. You can find only
-<code class="docutils literal"><span class="pre">京都</span></code> text by <code class="docutils literal"><span class="pre">京都</span></code> query with <code class="docutils literal"><span class="pre">TokenMecab</span></code>.</p>
-<p>If you want to support neologisms, you need to keep updating your
-MeCab dictionary. It needs maintain cost. (<a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> doesn't
-require dictionary maintenance because <a class="reference internal" href="#token-bigram"><span class="std std-ref">TokenBigram</span></a> doesn't use
-dictionary.) <a class="reference external" href="https://github.com/neologd/mecab-ipadic-neologd">mecab-ipadic-NEologd : Neologism dictionary for MeCab</a> may help you.</p>
-<p>Here is an example of <code class="docutils literal"><span class="pre">TokenMeCab</span></code>. <code class="docutils literal"><span class="pre">東京都</span></code> is tokenized to <code class="docutils literal"><span class="pre">東京</span></code>
-and <code class="docutils literal"><span class="pre">都</span></code>. They don't include <code class="docutils literal"><span class="pre">京都</span></code>:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenMecab &quot;東京都&quot;
-# [
-#   [
-#     -22,
-#     1337566253.89858,
-#     0.000355720520019531,
-#     &quot;[tokenize] nonexistent tokenizer: &lt;TokenMecab&gt;&quot;,
-#     [
-#       [
-#         &quot;create_lexicon_for_tokenize&quot;,
-#         &quot;proc_tokenize.c&quot;,
-#         139
-#       ]
-#     ]
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
-<div class="section" id="tokenregexp">
-<span id="token-regexp"></span><h3>7.8.3.14. <code class="docutils literal"><span class="pre">TokenRegexp</span></code><a class="headerlink" href="#tokenregexp" title="Permalink to this headline">¶</a></h3>
-<div class="versionadded">
-<p><span class="versionmodified">New in version 5.0.1.</span></p>
-</div>
-<div class="admonition caution">
-<p class="first admonition-title">Caution</p>
-<p class="last">This tokenizer is experimental. Specification may be changed.</p>
-</div>
-<div class="admonition caution">
-<p class="first admonition-title">Caution</p>
-<p class="last">This tokenizer can be used only with UTF-8. You can't use this
-tokenizer with EUC-JP, Shift_JIS and so on.</p>
-</div>
-<p><code class="docutils literal"><span class="pre">TokenRegexp</span></code> is a tokenizer for supporting regular expression
-search by index.</p>
-<p>In general, regular expression search is evaluated as sequential
-search. But the following cases can be evaluated as index search:</p>
-<blockquote>
-<div><ul class="simple">
-<li>Literal only case such as <code class="docutils literal"><span class="pre">hello</span></code></li>
-<li>The beginning of text and literal case such as <code class="docutils literal"><span class="pre">\A/home/alice</span></code></li>
-<li>The end of text and literal case such as <code class="docutils literal"><span class="pre">\.txt\z</span></code></li>
-</ul>
-</div></blockquote>
-<p>In most cases, index search is faster than sequential search.</p>
-<p><code class="docutils literal"><span class="pre">TokenRegexp</span></code> is based on bigram tokenize method. <code class="docutils literal"><span class="pre">TokenRegexp</span></code>
-adds the beginning of text mark (<code class="docutils literal"><span class="pre">U+FFEF</span></code>) at the begging of text
-and the end of text mark (<code class="docutils literal"><span class="pre">U+FFF0</span></code>) to the end of text when you
-index text:</p>
-<p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenRegexp &quot;/home/alice/test.txt&quot; NormalizerAuto --mode ADD
-# [
-#   [
-#     0,
-#     1337566253.89858,
-#     0.000355720520019531
-#   ],
-#   [
-#     {
-#       &quot;position&quot;: 0,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;￯&quot;
-#     },
-#     {
-#       &quot;position&quot;: 1,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;/h&quot;
-#     },
-#     {
-#       &quot;position&quot;: 2,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ho&quot;
-#     },
-#     {
-#       &quot;position&quot;: 3,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;om&quot;
-#     },
-#     {
-#       &quot;position&quot;: 4,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;me&quot;
-#     },
-#     {
-#       &quot;position&quot;: 5,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;e/&quot;
-#     },
-#     {
-#       &quot;position&quot;: 6,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;/a&quot;
-#     },
-#     {
-#       &quot;position&quot;: 7,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;al&quot;
-#     },
-#     {
-#       &quot;position&quot;: 8,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;li&quot;
-#     },
-#     {
-#       &quot;position&quot;: 9,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ic&quot;
-#     },
-#     {
-#       &quot;position&quot;: 10,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;ce&quot;
-#     },
-#     {
-#       &quot;position&quot;: 11,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;e/&quot;
-#     },
-#     {
-#       &quot;position&quot;: 12,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;/t&quot;
-#     },
-#     {
-#       &quot;position&quot;: 13,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;te&quot;
-#     },
-#     {
-#       &quot;position&quot;: 14,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;es&quot;
-#     },
-#     {
-#       &quot;position&quot;: 15,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;st&quot;
-#     },
-#     {
-#       &quot;position&quot;: 16,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;t.&quot;
-#     },
-#     {
-#       &quot;position&quot;: 17,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;.t&quot;
-#     },
-#     {
-#       &quot;position&quot;: 18,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;tx&quot;
-#     },
-#     {
-#       &quot;position&quot;: 19,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;xt&quot;
-#     },
-#     {
-#       &quot;position&quot;: 20,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;t&quot;
-#     },
-#     {
-#       &quot;position&quot;: 21,
-#       &quot;force_prefix&quot;: false,
-#       &quot;value&quot;: &quot;￰&quot;
-#     }
-#   ]
-# ]
-</pre></div>
-</div>
-</div>
 </div>
 </div>
@@ -1476,46 +89,20 @@ index text:</p>
       </div>
       <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
         <div class="sphinxsidebarwrapper">
-  <h3><a href="../index.html">Table Of Contents</a></h3>
-  <ul>
-<li><a class="reference internal" href="#">7.8. Tokenizers</a><ul>
-<li><a class="reference internal" href="#summary">7.8.1. Summary</a></li>
-<li><a class="reference internal" href="#what-is-tokenize">7.8.2. What is &quot;tokenize&quot;?</a></li>
-<li><a class="reference internal" href="#built-in-tokenizsers">7.8.3. Built-in tokenizsers</a><ul>
-<li><a class="reference internal" href="#tokenbigram">7.8.3.1. <code class="docutils literal"><span class="pre">TokenBigram</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbol">7.8.3.2. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbol</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbolalpha">7.8.3.3. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramsplitsymbolalphadigit">7.8.3.4. <code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlphaDigit</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblank">7.8.3.5. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlank</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbol">7.8.3.6. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbol</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalpha">7.8.3.7. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlpha</span></code></a></li>
-<li><a class="reference internal" href="#tokenbigramignoreblanksplitsymbolalphadigit">7.8.3.8. <code class="docutils literal"><span class="pre">TokenBigramIgnoreBlankSplitSymbolAlphaDigit</span></code></a></li>
-<li><a class="reference internal" href="#tokenunigram">7.8.3.9. <code class="docutils literal"><span class="pre">TokenUnigram</span></code></a></li>
-<li><a class="reference internal" href="#tokentrigram">7.8.3.10. <code class="docutils literal"><span class="pre">TokenTrigram</span></code></a></li>
-<li><a class="reference internal" href="#tokendelimit">7.8.3.11. <code class="docutils literal"><span class="pre">TokenDelimit</span></code></a></li>
-<li><a class="reference internal" href="#tokendelimitnull">7.8.3.12. <code class="docutils literal"><span class="pre">TokenDelimitNull</span></code></a></li>
-<li><a class="reference internal" href="#tokenmecab">7.8.3.13. <code class="docutils literal"><span class="pre">TokenMecab</span></code></a></li>
-<li><a class="reference internal" href="#tokenregexp">7.8.3.14. <code class="docutils literal"><span class="pre">TokenRegexp</span></code></a></li>
-</ul>
-</li>
-</ul>
-</li>
-</ul>
   <h4>Previous topic</h4>
-  <p class="topless"><a href="normalizers.html"
-                        title="previous chapter">7.7. Normalizers</a></p>
+  <p class="topless"><a href="normalizers/normalizer_nfkc51.html"
+                        title="previous chapter">7.7.2.3. <code class="docutils literal notranslate"><span class="pre">NormalizerNFKC51</span></code></a></p>
   <h4>Next topic</h4>
-  <p class="topless"><a href="token_filters.html"
-                        title="next chapter">7.9. Token filters</a></p>
+  <p class="topless"><a href="tokenizer/summary.html"
+                        title="next chapter">7.8.1. Summary</a></p>
 <div id="searchbox" style="display: none" role="search">
   <h3>Quick search</h3>
+    <div class="searchformwrapper">
     <form class="search" action="../search.html" method="get">
-      <div><input type="text" name="q" /></div>
-      <div><input type="submit" value="Go" /></div>
-      <input type="hidden" name="check_keywords" value="yes" />
-      <input type="hidden" name="area" value="default" />
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
     </form>
+    </div>
 </div>
 <script type="text/javascript">$('#searchbox').show(0);</script>
         </div>
@@ -1529,17 +116,17 @@ index text:</p>
           <a href="../genindex.html" title="General Index"
              >index</a></li>
         <li class="right" >
-          <a href="token_filters.html" title="7.9. Token filters"
+          <a href="tokenizer/summary.html" title="7.8.1. Summary"
              >next</a> |</li>
         <li class="right" >
-          <a href="normalizers.html" title="7.7. Normalizers"
+          <a href="normalizers/normalizer_nfkc51.html" title="7.7.2.3. NormalizerNFKC51"
              >previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../reference.html" >7. Reference manual</a> &#187;</li>
       </ul>
     </div>
     <div class="footer" role="contentinfo">
-        &#169; Copyright 2009-2018, Brazil, Inc.
+        &#169; Copyright 2009-2019, Brazil, Inc.
     </div>
   </body>
 </html>