RubyGems - rroonga - Versions diffs - 7.1.1-x86-mingw32 → 9.0.2-x86-mingw32 - Mend

rroonga 7.1.1-x86-mingw32 → 9.0.2-x86-mingw32

Files changed (980) hide show

data/vendor/local/share/doc/groonga/en/html/reference/commands/tokenize.html CHANGED Viewed

@@ -1,34 +1,26 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
   <head>
-    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-    <title>7.3.65. tokenize &#8212; Groonga v7.1.0-73-g6d02cfa documentation</title>
+    <meta charset="utf-8" />
+    <title>7.3.65. tokenize &#8212; Groonga v9.0.2 documentation</title>
     <link rel="stylesheet" href="../../_static/groonga.css" type="text/css" />
     <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
-    <script type="text/javascript">
-      var DOCUMENTATION_OPTIONS = {
-        URL_ROOT:    '../../',
-        VERSION:     '7.1.0-73-g6d02cfa',
-        COLLAPSE_INDEX: false,
-        FILE_SUFFIX: '.html',
-        HAS_SOURCE:  false,
-        SOURCELINK_SUFFIX: '.txt'
-      };
-    </script>
+    <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
     <script type="text/javascript" src="../../_static/jquery.js"></script>
     <script type="text/javascript" src="../../_static/underscore.js"></script>
     <script type="text/javascript" src="../../_static/doctools.js"></script>
+    <script type="text/javascript" src="../../_static/language_data.js"></script>
     <link rel="shortcut icon" href="../../_static/favicon.ico"/>
     <link rel="index" title="Index" href="../../genindex.html" />
     <link rel="search" title="Search" href="../../search.html" />
     <link rel="next" title="7.3.66. tokenizer_list" href="tokenizer_list.html" />
     <link rel="prev" title="7.3.64. thread_limit" href="thread_limit.html" />
-  </head>
-  <body>
+  </head><body>
 <div class="header">
   <h1 class="title">
     <a id="top-link" href="../../index.html">
@@ -58,7 +50,7 @@
         <li class="right" >
           <a href="thread_limit.html" title="7.3.64. thread_limit"
              accesskey="P">previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
           <li class="nav-item nav-item-2"><a href="../command.html" accesskey="U">7.3. Command</a> &#187;</li>
       </ul>
@@ -70,18 +62,18 @@
           <div class="body" role="main">
   <div class="section" id="tokenize">
-<h1>7.3.65. <code class="docutils literal"><span class="pre">tokenize</span></code><a class="headerlink" href="#tokenize" title="Permalink to this headline">¶</a></h1>
+<h1>7.3.65. <code class="docutils literal notranslate"><span class="pre">tokenize</span></code><a class="headerlink" href="#tokenize" title="Permalink to this headline">¶</a></h1>
 <div class="section" id="summary">
 <h2>7.3.65.1. Summary<a class="headerlink" href="#summary" title="Permalink to this headline">¶</a></h2>
-<p><code class="docutils literal"><span class="pre">tokenize</span></code> command tokenizes text by the specified tokenizer.
+<p><code class="docutils literal notranslate"><span class="pre">tokenize</span></code> command tokenizes text by the specified tokenizer.
 It is useful to debug tokenization.</p>
 </div>
 <div class="section" id="syntax">
 <h2>7.3.65.2. Syntax<a class="headerlink" href="#syntax" title="Permalink to this headline">¶</a></h2>
 <p>This command takes many parameters.</p>
-<p><code class="docutils literal"><span class="pre">tokenizer</span></code> and <code class="docutils literal"><span class="pre">string</span></code> are required parameters. Others are
+<p><code class="docutils literal notranslate"><span class="pre">tokenizer</span></code> and <code class="docutils literal notranslate"><span class="pre">string</span></code> are required parameters. Others are
 optional:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize tokenizer
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize tokenizer
          string
          [normalizer=null]
          [flags=NONE]
@@ -94,7 +86,7 @@ optional:</p>
 <h2>7.3.65.3. Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2>
 <p>Here is a simple example.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot;
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot;
 # [
 #   [
 #     0,
@@ -181,25 +173,25 @@ optional:</p>
 # ]
 </pre></div>
 </div>
-<p>It has only required parameters. <code class="docutils literal"><span class="pre">tokenizer</span></code> is <code class="docutils literal"><span class="pre">TokenBigram</span></code> and
-<code class="docutils literal"><span class="pre">string</span></code> is <code class="docutils literal"><span class="pre">&quot;Fulltext</span> <span class="pre">Search&quot;</span></code>. It returns tokens that is
-generated by tokenizing <code class="docutils literal"><span class="pre">&quot;Fulltext</span> <span class="pre">Search&quot;</span></code> with <code class="docutils literal"><span class="pre">TokenBigram</span></code>
-tokenizer. It doesn't normalize <code class="docutils literal"><span class="pre">&quot;Fulltext</span> <span class="pre">Search&quot;</span></code>.</p>
+<p>It has only required parameters. <code class="docutils literal notranslate"><span class="pre">tokenizer</span></code> is <code class="docutils literal notranslate"><span class="pre">TokenBigram</span></code> and
+<code class="docutils literal notranslate"><span class="pre">string</span></code> is <code class="docutils literal notranslate"><span class="pre">&quot;Fulltext</span> <span class="pre">Search&quot;</span></code>. It returns tokens that is
+generated by tokenizing <code class="docutils literal notranslate"><span class="pre">&quot;Fulltext</span> <span class="pre">Search&quot;</span></code> with <code class="docutils literal notranslate"><span class="pre">TokenBigram</span></code>
+tokenizer. It doesn’t normalize <code class="docutils literal notranslate"><span class="pre">&quot;Fulltext</span> <span class="pre">Search&quot;</span></code>.</p>
 </div>
 <div class="section" id="parameters">
 <h2>7.3.65.4. Parameters<a class="headerlink" href="#parameters" title="Permalink to this headline">¶</a></h2>
 <p>This section describes all parameters. Parameters are categorized.</p>
 <div class="section" id="required-parameters">
 <h3>7.3.65.4.1. Required parameters<a class="headerlink" href="#required-parameters" title="Permalink to this headline">¶</a></h3>
-<p>There are required parameters, <code class="docutils literal"><span class="pre">tokenizer</span></code> and <code class="docutils literal"><span class="pre">string</span></code>.</p>
+<p>There are required parameters, <code class="docutils literal notranslate"><span class="pre">tokenizer</span></code> and <code class="docutils literal notranslate"><span class="pre">string</span></code>.</p>
 <div class="section" id="tokenizer">
-<span id="tokenize-tokenizer"></span><h4>7.3.65.4.1.1. <code class="docutils literal"><span class="pre">tokenizer</span></code><a class="headerlink" href="#tokenizer" title="Permalink to this headline">¶</a></h4>
-<p>Specifies the tokenizer name. <code class="docutils literal"><span class="pre">tokenize</span></code> command uses the
-tokenizer that is named <code class="docutils literal"><span class="pre">tokenizer</span></code>.</p>
+<span id="tokenize-tokenizer"></span><h4>7.3.65.4.1.1. <code class="docutils literal notranslate"><span class="pre">tokenizer</span></code><a class="headerlink" href="#tokenizer" title="Permalink to this headline">¶</a></h4>
+<p>Specifies the tokenizer name. <code class="docutils literal notranslate"><span class="pre">tokenize</span></code> command uses the
+tokenizer that is named <code class="docutils literal notranslate"><span class="pre">tokenizer</span></code>.</p>
 <p>See <a class="reference internal" href="../tokenizers.html"><span class="doc">Tokenizers</span></a> about built-in tokenizers.</p>
-<p>Here is an example to use built-in <code class="docutils literal"><span class="pre">TokenTrigram</span></code> tokenizer.</p>
+<p>Here is an example to use built-in <code class="docutils literal notranslate"><span class="pre">TokenTrigram</span></code> tokenizer.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenTrigram &quot;Fulltext Search&quot;
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenTrigram &quot;Fulltext Search&quot;
 # [
 #   [
 #     0,
@@ -289,16 +281,16 @@ tokenizer that is named <code class="docutils literal"><span class="pre">tokeniz
 <p>If you want to use other tokenizers, you need to register additional
 tokenizer plugin by <a class="reference internal" href="register.html"><span class="doc">register</span></a> command. For example, you can use
 <a class="reference external" href="http://www.phontron.com/kytea/">KyTea</a> based tokenizer by
-registering <code class="docutils literal"><span class="pre">tokenizers/kytea</span></code>.</p>
+registering <code class="docutils literal notranslate"><span class="pre">tokenizers/kytea</span></code>.</p>
 </div>
 <div class="section" id="string">
-<span id="tokenize-string"></span><h4>7.3.65.4.1.2. <code class="docutils literal"><span class="pre">string</span></code><a class="headerlink" href="#string" title="Permalink to this headline">¶</a></h4>
+<span id="tokenize-string"></span><h4>7.3.65.4.1.2. <code class="docutils literal notranslate"><span class="pre">string</span></code><a class="headerlink" href="#string" title="Permalink to this headline">¶</a></h4>
 <p>Specifies any string which you want to tokenize.</p>
-<p>If you want to include spaces in <code class="docutils literal"><span class="pre">string</span></code>, you need to quote
-<code class="docutils literal"><span class="pre">string</span></code> by single quotation (<code class="docutils literal"><span class="pre">'</span></code>) or double quotation (<code class="docutils literal"><span class="pre">&quot;</span></code>).</p>
-<p>Here is an example to use spaces in <code class="docutils literal"><span class="pre">string</span></code>.</p>
+<p>If you want to include spaces in <code class="docutils literal notranslate"><span class="pre">string</span></code>, you need to quote
+<code class="docutils literal notranslate"><span class="pre">string</span></code> by single quotation (<code class="docutils literal notranslate"><span class="pre">'</span></code>) or double quotation (<code class="docutils literal notranslate"><span class="pre">&quot;</span></code>).</p>
+<p>Here is an example to use spaces in <code class="docutils literal notranslate"><span class="pre">string</span></code>.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Groonga is a fast fulltext earch engine!&quot;
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Groonga is a fast fulltext earch engine!&quot;
 # [
 #   [
 #     0,
@@ -516,16 +508,16 @@ registering <code class="docutils literal"><span class="pre">tokenizers/kytea</s
 <h3>7.3.65.4.2. Optional parameters<a class="headerlink" href="#optional-parameters" title="Permalink to this headline">¶</a></h3>
 <p>There are optional parameters.</p>
 <div class="section" id="normalizer">
-<span id="tokenize-normalizer"></span><h4>7.3.65.4.2.1. <code class="docutils literal"><span class="pre">normalizer</span></code><a class="headerlink" href="#normalizer" title="Permalink to this headline">¶</a></h4>
-<p>Specifies the normalizer name. <code class="docutils literal"><span class="pre">tokenize</span></code> command uses the
-normalizer that is named <code class="docutils literal"><span class="pre">normalizer</span></code>. Normalizer is important for
-N-gram family tokenizers such as <code class="docutils literal"><span class="pre">TokenBigram</span></code>.</p>
+<span id="tokenize-normalizer"></span><h4>7.3.65.4.2.1. <code class="docutils literal notranslate"><span class="pre">normalizer</span></code><a class="headerlink" href="#normalizer" title="Permalink to this headline">¶</a></h4>
+<p>Specifies the normalizer name. <code class="docutils literal notranslate"><span class="pre">tokenize</span></code> command uses the
+normalizer that is named <code class="docutils literal notranslate"><span class="pre">normalizer</span></code>. Normalizer is important for
+N-gram family tokenizers such as <code class="docutils literal notranslate"><span class="pre">TokenBigram</span></code>.</p>
 <p>Normalizer detects character type for each character while
 normalizing. N-gram family tokenizers use character types while
 tokenizing.</p>
-<p>Here is an example that doesn't use normalizer.</p>
+<p>Here is an example that doesn’t use normalizer.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot;
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot;
 # [
 #   [
 #     0,
@@ -612,11 +604,11 @@ tokenizing.</p>
 # ]
 </pre></div>
 </div>
-<p>All alphabets are tokenized by two characters. For example, <code class="docutils literal"><span class="pre">Fu</span></code> is
+<p>All alphabets are tokenized by two characters. For example, <code class="docutils literal notranslate"><span class="pre">Fu</span></code> is
 a token.</p>
 <p>Here is an example that uses normalizer.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot; NormalizerAuto
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot; NormalizerAuto
 # [
 #   [
 #     0,
@@ -639,11 +631,11 @@ a token.</p>
 </pre></div>
 </div>
 <p>Continuous alphabets are tokenized as one token. For example,
-<code class="docutils literal"><span class="pre">fulltext</span></code> is a token.</p>
+<code class="docutils literal notranslate"><span class="pre">fulltext</span></code> is a token.</p>
 <p>If you want to tokenize by two characters with noramlizer, use
-<code class="docutils literal"><span class="pre">TokenBigramSplitSymbolAlpha</span></code>.</p>
+<code class="docutils literal notranslate"><span class="pre">TokenBigramSplitSymbolAlpha</span></code>.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbolAlpha &quot;Fulltext Search&quot; NormalizerAuto
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigramSplitSymbolAlpha &quot;Fulltext Search&quot; NormalizerAuto
 # [
 #   [
 #     0,
@@ -726,37 +718,37 @@ a token.</p>
 </pre></div>
 </div>
 <p>All alphabets are tokenized by two characters. And they are normalized
-to lower case characters. For example, <code class="docutils literal"><span class="pre">fu</span></code> is a token.</p>
+to lower case characters. For example, <code class="docutils literal notranslate"><span class="pre">fu</span></code> is a token.</p>
 </div>
 <div class="section" id="flags">
-<span id="tokenize-flags"></span><h4>7.3.65.4.2.2. <code class="docutils literal"><span class="pre">flags</span></code><a class="headerlink" href="#flags" title="Permalink to this headline">¶</a></h4>
+<span id="tokenize-flags"></span><h4>7.3.65.4.2.2. <code class="docutils literal notranslate"><span class="pre">flags</span></code><a class="headerlink" href="#flags" title="Permalink to this headline">¶</a></h4>
 <p>Specifies a tokenization customize options. You can specify
-multiple options separated by &quot;<code class="docutils literal"><span class="pre">|</span></code>&quot;. For example,
-<code class="docutils literal"><span class="pre">NONE|ENABLE_TOKENIZED_DELIMITER</span></code>.</p>
+multiple options separated by “<code class="docutils literal notranslate"><span class="pre">|</span></code>”. For example,
+<code class="docutils literal notranslate"><span class="pre">NONE|ENABLE_TOKENIZED_DELIMITER</span></code>.</p>
 <p>Here are available flags.</p>
-<table border="1" class="docutils">
+<table class="docutils align-center">
 <colgroup>
-<col width="50%" />
-<col width="50%" />
+<col style="width: 50%" />
+<col style="width: 50%" />
 </colgroup>
-<thead valign="bottom">
-<tr class="row-odd"><th class="head">Flag</th>
-<th class="head">Description</th>
+<thead>
+<tr class="row-odd"><th class="head"><p>Flag</p></th>
+<th class="head"><p>Description</p></th>
 </tr>
 </thead>
-<tbody valign="top">
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">NONE</span></code></td>
-<td>Just ignored.</td>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">NONE</span></code></p></td>
+<td><p>Just ignored.</p></td>
 </tr>
-<tr class="row-odd"><td><code class="docutils literal"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code></td>
-<td>Enables tokenized delimiter. See <a class="reference internal" href="../tokenizers.html"><span class="doc">Tokenizers</span></a> about
-tokenized delimiter details.</td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code></p></td>
+<td><p>Enables tokenized delimiter. See <a class="reference internal" href="../tokenizers.html"><span class="doc">Tokenizers</span></a> about
+tokenized delimiter details.</p></td>
 </tr>
 </tbody>
 </table>
-<p>Here is an example that uses <code class="docutils literal"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code>.</p>
+<p>Here is an example that uses <code class="docutils literal notranslate"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code>.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenDelimit &quot;Fulltext Seacrch&quot; NormalizerAuto ENABLE_TOKENIZED_DELIMITER
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenDelimit &quot;Fulltext Seacrch&quot; NormalizerAuto ENABLE_TOKENIZED_DELIMITER
 # [
 #   [
 #     0,
@@ -783,26 +775,26 @@ tokenized delimiter details.</td>
 # ]
 </pre></div>
 </div>
-<p><code class="docutils literal"><span class="pre">TokenDelimit</span></code> tokenizer is one of tokenized delimiter supported
-tokenizer. <code class="docutils literal"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code> enables tokenized delimiter.
+<p><code class="docutils literal notranslate"><span class="pre">TokenDelimit</span></code> tokenizer is one of tokenized delimiter supported
+tokenizer. <code class="docutils literal notranslate"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code> enables tokenized delimiter.
 Tokenized delimiter is special character that indicates token
-border. It is <code class="docutils literal"><span class="pre">U+FFFE</span></code>. The character is not assigned any
+border. It is <code class="docutils literal notranslate"><span class="pre">U+FFFE</span></code>. The character is not assigned any
 character. It means that the character is not appeared in normal
 string. So the character is good character for this puropose. If
-<code class="docutils literal"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code> is enabled, the target string is
+<code class="docutils literal notranslate"><span class="pre">ENABLE_TOKENIZED_DELIMITER</span></code> is enabled, the target string is
 treated as already tokenized string. Tokenizer just tokenizes by
 tokenized delimiter.</p>
 </div>
 <div class="section" id="mode">
-<span id="tokenize-mode"></span><h4>7.3.65.4.2.3. <code class="docutils literal"><span class="pre">mode</span></code><a class="headerlink" href="#mode" title="Permalink to this headline">¶</a></h4>
-<p>Specifies a tokenize mode. If the mode is specified <code class="docutils literal"><span class="pre">ADD</span></code>, the text
+<span id="tokenize-mode"></span><h4>7.3.65.4.2.3. <code class="docutils literal notranslate"><span class="pre">mode</span></code><a class="headerlink" href="#mode" title="Permalink to this headline">¶</a></h4>
+<p>Specifies a tokenize mode. If the mode is specified <code class="docutils literal notranslate"><span class="pre">ADD</span></code>, the text
 is tokenized by the rule that adding a document. If the mode is specified
-<code class="docutils literal"><span class="pre">GET</span></code>, the text is tokenized by the rule that searching a document. If
-the mode is omitted, the text is tokenized by the <code class="docutils literal"><span class="pre">ADD</span></code> mode.</p>
-<p>The default mode is <code class="docutils literal"><span class="pre">ADD</span></code>.</p>
-<p>Here is an example to the <code class="docutils literal"><span class="pre">ADD</span></code> mode.</p>
+<code class="docutils literal notranslate"><span class="pre">GET</span></code>, the text is tokenized by the rule that searching a document. If
+the mode is omitted, the text is tokenized by the <code class="docutils literal notranslate"><span class="pre">ADD</span></code> mode.</p>
+<p>The default mode is <code class="docutils literal notranslate"><span class="pre">ADD</span></code>.</p>
+<p>Here is an example to the <code class="docutils literal notranslate"><span class="pre">ADD</span></code> mode.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot; --mode ADD
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot; --mode ADD
 # [
 #   [
 #     0,
@@ -890,9 +882,9 @@ the mode is omitted, the text is tokenized by the <code class="docutils literal"
 </pre></div>
 </div>
 <p>The last alphabet is tokenized by one character.</p>
-<p>Here is an example to the <code class="docutils literal"><span class="pre">GET</span></code> mode.</p>
+<p>Here is an example to the <code class="docutils literal notranslate"><span class="pre">GET</span></code> mode.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot; --mode GET
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenize TokenBigram &quot;Fulltext Search&quot; --mode GET
 # [
 #   [
 #     0,
@@ -977,44 +969,45 @@ the mode is omitted, the text is tokenized by the <code class="docutils literal"
 <p>The last alphabet is tokenized by two characters.</p>
 </div>
 <div class="section" id="token-filters">
-<span id="tokenize-token-filters"></span><h4>7.3.65.4.2.4. <code class="docutils literal"><span class="pre">token_filters</span></code><a class="headerlink" href="#token-filters" title="Permalink to this headline">¶</a></h4>
-<p>Specifies the token filter names. <code class="docutils literal"><span class="pre">tokenize</span></code> command uses the
-tokenizer that is named <code class="docutils literal"><span class="pre">token_filters</span></code>.</p>
+<span id="tokenize-token-filters"></span><h4>7.3.65.4.2.4. <code class="docutils literal notranslate"><span class="pre">token_filters</span></code><a class="headerlink" href="#token-filters" title="Permalink to this headline">¶</a></h4>
+<p>Specifies the token filter names. <code class="docutils literal notranslate"><span class="pre">tokenize</span></code> command uses the
+tokenizer that is named <code class="docutils literal notranslate"><span class="pre">token_filters</span></code>.</p>
 <p>See <a class="reference internal" href="../token_filters.html"><span class="doc">Token filters</span></a> about token filters.</p>
 </div>
 </div>
 </div>
 <div class="section" id="return-value">
 <span id="tokenize-return-value"></span><h2>7.3.65.5. Return value<a class="headerlink" href="#return-value" title="Permalink to this headline">¶</a></h2>
-<p><code class="docutils literal"><span class="pre">tokenize</span></code> command returns tokenized tokens. Each token has some
+<p><code class="docutils literal notranslate"><span class="pre">tokenize</span></code> command returns tokenized tokens. Each token has some
 attributes except token itself. The attributes will be increased in
 the feature:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>[HEADER, tokens]
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>[HEADER, tokens]
 </pre></div>
 </div>
-<p><code class="docutils literal"><span class="pre">HEADER</span></code></p>
+<p><code class="docutils literal notranslate"><span class="pre">HEADER</span></code></p>
 <blockquote>
-<div>See <a class="reference internal" href="../command/output_format.html"><span class="doc">Output format</span></a> about <code class="docutils literal"><span class="pre">HEADER</span></code>.</div></blockquote>
-<p><code class="docutils literal"><span class="pre">tokens</span></code></p>
+<div><p>See <a class="reference internal" href="../command/output_format.html"><span class="doc">Output format</span></a> about <code class="docutils literal notranslate"><span class="pre">HEADER</span></code>.</p>
+</div></blockquote>
+<p><code class="docutils literal notranslate"><span class="pre">tokens</span></code></p>
 <blockquote>
-<div><p><code class="docutils literal"><span class="pre">tokens</span></code> is an array of token. Token is an object that has the following
+<div><p><code class="docutils literal notranslate"><span class="pre">tokens</span></code> is an array of token. Token is an object that has the following
 attributes.</p>
-<table border="1" class="docutils">
+<table class="docutils align-center">
 <colgroup>
-<col width="50%" />
-<col width="50%" />
+<col style="width: 50%" />
+<col style="width: 50%" />
 </colgroup>
-<thead valign="bottom">
-<tr class="row-odd"><th class="head">Name</th>
-<th class="head">Description</th>
+<thead>
+<tr class="row-odd"><th class="head"><p>Name</p></th>
+<th class="head"><p>Description</p></th>
 </tr>
 </thead>
-<tbody valign="top">
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">value</span></code></td>
-<td>Token itself.</td>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">value</span></code></p></td>
+<td><p>Token itself.</p></td>
 </tr>
-<tr class="row-odd"><td><code class="docutils literal"><span class="pre">position</span></code></td>
-<td>The N-th token.</td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">position</span></code></p></td>
+<td><p>The N-th token.</p></td>
 </tr>
 </tbody>
 </table>
@@ -1023,7 +1016,7 @@ attributes.</p>
 <div class="section" id="see-also">
 <h2>7.3.65.6. See also<a class="headerlink" href="#see-also" title="Permalink to this headline">¶</a></h2>
 <ul class="simple">
-<li><a class="reference internal" href="../tokenizers.html"><span class="doc">Tokenizers</span></a></li>
+<li><p><a class="reference internal" href="../tokenizers.html"><span class="doc">Tokenizers</span></a></p></li>
 </ul>
 </div>
 </div>
@@ -1034,23 +1027,23 @@ attributes.</p>
       </div>
       <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
         <div class="sphinxsidebarwrapper">
-  <h3><a href="../../index.html">Table Of Contents</a></h3>
+  <h3><a href="../../index.html">Table of Contents</a></h3>
   <ul>
-<li><a class="reference internal" href="#">7.3.65. <code class="docutils literal"><span class="pre">tokenize</span></code></a><ul>
+<li><a class="reference internal" href="#">7.3.65. <code class="docutils literal notranslate"><span class="pre">tokenize</span></code></a><ul>
 <li><a class="reference internal" href="#summary">7.3.65.1. Summary</a></li>
 <li><a class="reference internal" href="#syntax">7.3.65.2. Syntax</a></li>
 <li><a class="reference internal" href="#usage">7.3.65.3. Usage</a></li>
 <li><a class="reference internal" href="#parameters">7.3.65.4. Parameters</a><ul>
 <li><a class="reference internal" href="#required-parameters">7.3.65.4.1. Required parameters</a><ul>
-<li><a class="reference internal" href="#tokenizer">7.3.65.4.1.1. <code class="docutils literal"><span class="pre">tokenizer</span></code></a></li>
-<li><a class="reference internal" href="#string">7.3.65.4.1.2. <code class="docutils literal"><span class="pre">string</span></code></a></li>
+<li><a class="reference internal" href="#tokenizer">7.3.65.4.1.1. <code class="docutils literal notranslate"><span class="pre">tokenizer</span></code></a></li>
+<li><a class="reference internal" href="#string">7.3.65.4.1.2. <code class="docutils literal notranslate"><span class="pre">string</span></code></a></li>
 </ul>
 </li>
 <li><a class="reference internal" href="#optional-parameters">7.3.65.4.2. Optional parameters</a><ul>
-<li><a class="reference internal" href="#normalizer">7.3.65.4.2.1. <code class="docutils literal"><span class="pre">normalizer</span></code></a></li>
-<li><a class="reference internal" href="#flags">7.3.65.4.2.2. <code class="docutils literal"><span class="pre">flags</span></code></a></li>
-<li><a class="reference internal" href="#mode">7.3.65.4.2.3. <code class="docutils literal"><span class="pre">mode</span></code></a></li>
-<li><a class="reference internal" href="#token-filters">7.3.65.4.2.4. <code class="docutils literal"><span class="pre">token_filters</span></code></a></li>
+<li><a class="reference internal" href="#normalizer">7.3.65.4.2.1. <code class="docutils literal notranslate"><span class="pre">normalizer</span></code></a></li>
+<li><a class="reference internal" href="#flags">7.3.65.4.2.2. <code class="docutils literal notranslate"><span class="pre">flags</span></code></a></li>
+<li><a class="reference internal" href="#mode">7.3.65.4.2.3. <code class="docutils literal notranslate"><span class="pre">mode</span></code></a></li>
+<li><a class="reference internal" href="#token-filters">7.3.65.4.2.4. <code class="docutils literal notranslate"><span class="pre">token_filters</span></code></a></li>
 </ul>
 </li>
 </ul>
@@ -1063,18 +1056,18 @@ attributes.</p>
   <h4>Previous topic</h4>
   <p class="topless"><a href="thread_limit.html"
-                        title="previous chapter">7.3.64. <code class="docutils literal"><span class="pre">thread_limit</span></code></a></p>
+                        title="previous chapter">7.3.64. <code class="docutils literal notranslate"><span class="pre">thread_limit</span></code></a></p>
   <h4>Next topic</h4>
   <p class="topless"><a href="tokenizer_list.html"
-                        title="next chapter">7.3.66. <code class="docutils literal"><span class="pre">tokenizer_list</span></code></a></p>
+                        title="next chapter">7.3.66. <code class="docutils literal notranslate"><span class="pre">tokenizer_list</span></code></a></p>
 <div id="searchbox" style="display: none" role="search">
   <h3>Quick search</h3>
+    <div class="searchformwrapper">
     <form class="search" action="../../search.html" method="get">
-      <div><input type="text" name="q" /></div>
-      <div><input type="submit" value="Go" /></div>
-      <input type="hidden" name="check_keywords" value="yes" />
-      <input type="hidden" name="area" value="default" />
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
     </form>
+    </div>
 </div>
 <script type="text/javascript">$('#searchbox').show(0);</script>
         </div>
@@ -1093,13 +1086,13 @@ attributes.</p>
         <li class="right" >
           <a href="thread_limit.html" title="7.3.64. thread_limit"
              >previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
           <li class="nav-item nav-item-2"><a href="../command.html" >7.3. Command</a> &#187;</li>
       </ul>
     </div>
     <div class="footer" role="contentinfo">
-        &#169; Copyright 2009-2018, Brazil, Inc.
+        &#169; Copyright 2009-2019, Brazil, Inc.
     </div>
   </body>
 </html>

data/vendor/local/share/doc/groonga/en/html/reference/commands/tokenizer_list.html CHANGED Viewed

@@ -1,34 +1,26 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
   <head>
-    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-    <title>7.3.66. tokenizer_list &#8212; Groonga v7.1.0-73-g6d02cfa documentation</title>
+    <meta charset="utf-8" />
+    <title>7.3.66. tokenizer_list &#8212; Groonga v9.0.2 documentation</title>
     <link rel="stylesheet" href="../../_static/groonga.css" type="text/css" />
     <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
-    <script type="text/javascript">
-      var DOCUMENTATION_OPTIONS = {
-        URL_ROOT:    '../../',
-        VERSION:     '7.1.0-73-g6d02cfa',
-        COLLAPSE_INDEX: false,
-        FILE_SUFFIX: '.html',
-        HAS_SOURCE:  false,
-        SOURCELINK_SUFFIX: '.txt'
-      };
-    </script>
+    <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
     <script type="text/javascript" src="../../_static/jquery.js"></script>
     <script type="text/javascript" src="../../_static/underscore.js"></script>
     <script type="text/javascript" src="../../_static/doctools.js"></script>
+    <script type="text/javascript" src="../../_static/language_data.js"></script>
     <link rel="shortcut icon" href="../../_static/favicon.ico"/>
     <link rel="index" title="Index" href="../../genindex.html" />
     <link rel="search" title="Search" href="../../search.html" />
     <link rel="next" title="7.3.67. truncate" href="truncate.html" />
     <link rel="prev" title="7.3.65. tokenize" href="tokenize.html" />
-  </head>
-  <body>
+  </head><body>
 <div class="header">
   <h1 class="title">
     <a id="top-link" href="../../index.html">
@@ -58,7 +50,7 @@
         <li class="right" >
           <a href="tokenize.html" title="7.3.65. tokenize"
              accesskey="P">previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
           <li class="nav-item nav-item-2"><a href="../command.html" accesskey="U">7.3. Command</a> &#187;</li>
       </ul>
@@ -70,15 +62,15 @@
           <div class="body" role="main">
   <div class="section" id="tokenizer-list">
-<h1>7.3.66. <code class="docutils literal"><span class="pre">tokenizer_list</span></code><a class="headerlink" href="#tokenizer-list" title="Permalink to this headline">¶</a></h1>
+<h1>7.3.66. <code class="docutils literal notranslate"><span class="pre">tokenizer_list</span></code><a class="headerlink" href="#tokenizer-list" title="Permalink to this headline">¶</a></h1>
 <div class="section" id="summary">
 <h2>7.3.66.1. Summary<a class="headerlink" href="#summary" title="Permalink to this headline">¶</a></h2>
-<p><code class="docutils literal"><span class="pre">tokenizer_list</span></code> command lists tokenizers in a database.</p>
+<p><code class="docutils literal notranslate"><span class="pre">tokenizer_list</span></code> command lists tokenizers in a database.</p>
 </div>
 <div class="section" id="syntax">
 <h2>7.3.66.2. Syntax<a class="headerlink" href="#syntax" title="Permalink to this headline">¶</a></h2>
 <p>This command takes no parameters:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenizer_list
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenizer_list
 </pre></div>
 </div>
 </div>
@@ -86,7 +78,7 @@
 <h2>7.3.66.3. Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2>
 <p>Here is a simple example.</p>
 <p>Execution example:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>tokenizer_list
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>tokenizer_list
 # [
 #   [
 #     0,
@@ -141,31 +133,32 @@
 </div>
 <div class="section" id="return-value">
 <h2>7.3.66.4. Return value<a class="headerlink" href="#return-value" title="Permalink to this headline">¶</a></h2>
-<p><code class="docutils literal"><span class="pre">tokenizer_list</span></code> command returns tokenizers. Each tokenizers has an attribute
+<p><code class="docutils literal notranslate"><span class="pre">tokenizer_list</span></code> command returns tokenizers. Each tokenizers has an attribute
 that contains the name. The attribute will be increased in the feature:</p>
-<div class="highlight-none"><div class="highlight"><pre><span></span>[HEADER, tokenizers]
+<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>[HEADER, tokenizers]
 </pre></div>
 </div>
-<p><code class="docutils literal"><span class="pre">HEADER</span></code></p>
+<p><code class="docutils literal notranslate"><span class="pre">HEADER</span></code></p>
 <blockquote>
-<div>See <a class="reference internal" href="../command/output_format.html"><span class="doc">Output format</span></a> about <code class="docutils literal"><span class="pre">HEADER</span></code>.</div></blockquote>
-<p><code class="docutils literal"><span class="pre">tokenizers</span></code></p>
+<div><p>See <a class="reference internal" href="../command/output_format.html"><span class="doc">Output format</span></a> about <code class="docutils literal notranslate"><span class="pre">HEADER</span></code>.</p>
+</div></blockquote>
+<p><code class="docutils literal notranslate"><span class="pre">tokenizers</span></code></p>
 <blockquote>
-<div><p><code class="docutils literal"><span class="pre">tokenizers</span></code> is an array of tokenizer. Tokenizer is an object that has the following
+<div><p><code class="docutils literal notranslate"><span class="pre">tokenizers</span></code> is an array of tokenizer. Tokenizer is an object that has the following
 attributes.</p>
-<table border="1" class="docutils">
+<table class="docutils align-center">
 <colgroup>
-<col width="50%" />
-<col width="50%" />
+<col style="width: 50%" />
+<col style="width: 50%" />
 </colgroup>
-<thead valign="bottom">
-<tr class="row-odd"><th class="head">Name</th>
-<th class="head">Description</th>
+<thead>
+<tr class="row-odd"><th class="head"><p>Name</p></th>
+<th class="head"><p>Description</p></th>
 </tr>
 </thead>
-<tbody valign="top">
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">name</span></code></td>
-<td>Tokenizer name.</td>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">name</span></code></p></td>
+<td><p>Tokenizer name.</p></td>
 </tr>
 </tbody>
 </table>
@@ -174,8 +167,8 @@ attributes.</p>
 <div class="section" id="see-also">
 <h2>7.3.66.5. See also<a class="headerlink" href="#see-also" title="Permalink to this headline">¶</a></h2>
 <ul class="simple">
-<li><a class="reference internal" href="../tokenizers.html"><span class="doc">Tokenizers</span></a></li>
-<li><a class="reference internal" href="tokenize.html"><span class="doc">tokenize</span></a></li>
+<li><p><a class="reference internal" href="../tokenizers.html"><span class="doc">Tokenizers</span></a></p></li>
+<li><p><a class="reference internal" href="tokenize.html"><span class="doc">tokenize</span></a></p></li>
 </ul>
 </div>
 </div>
@@ -186,9 +179,9 @@ attributes.</p>
       </div>
       <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
         <div class="sphinxsidebarwrapper">
-  <h3><a href="../../index.html">Table Of Contents</a></h3>
+  <h3><a href="../../index.html">Table of Contents</a></h3>
   <ul>
-<li><a class="reference internal" href="#">7.3.66. <code class="docutils literal"><span class="pre">tokenizer_list</span></code></a><ul>
+<li><a class="reference internal" href="#">7.3.66. <code class="docutils literal notranslate"><span class="pre">tokenizer_list</span></code></a><ul>
 <li><a class="reference internal" href="#summary">7.3.66.1. Summary</a></li>
 <li><a class="reference internal" href="#syntax">7.3.66.2. Syntax</a></li>
 <li><a class="reference internal" href="#usage">7.3.66.3. Usage</a></li>
@@ -200,18 +193,18 @@ attributes.</p>
   <h4>Previous topic</h4>
   <p class="topless"><a href="tokenize.html"
-                        title="previous chapter">7.3.65. <code class="docutils literal"><span class="pre">tokenize</span></code></a></p>
+                        title="previous chapter">7.3.65. <code class="docutils literal notranslate"><span class="pre">tokenize</span></code></a></p>
   <h4>Next topic</h4>
   <p class="topless"><a href="truncate.html"
-                        title="next chapter">7.3.67. <code class="docutils literal"><span class="pre">truncate</span></code></a></p>
+                        title="next chapter">7.3.67. <code class="docutils literal notranslate"><span class="pre">truncate</span></code></a></p>
 <div id="searchbox" style="display: none" role="search">
   <h3>Quick search</h3>
+    <div class="searchformwrapper">
     <form class="search" action="../../search.html" method="get">
-      <div><input type="text" name="q" /></div>
-      <div><input type="submit" value="Go" /></div>
-      <input type="hidden" name="check_keywords" value="yes" />
-      <input type="hidden" name="area" value="default" />
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
     </form>
+    </div>
 </div>
 <script type="text/javascript">$('#searchbox').show(0);</script>
         </div>
@@ -230,13 +223,13 @@ attributes.</p>
         <li class="right" >
           <a href="tokenize.html" title="7.3.65. tokenize"
              >previous</a> |</li>
-        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v7.1.0-73-g6d02cfa documentation</a> &#187;</li>
+        <li class="nav-item nav-item-0"><a href="../../index.html">Groonga v9.0.2 documentation</a> &#187;</li>
           <li class="nav-item nav-item-1"><a href="../../reference.html" >7. Reference manual</a> &#187;</li>
           <li class="nav-item nav-item-2"><a href="../command.html" >7.3. Command</a> &#187;</li>
       </ul>
     </div>
     <div class="footer" role="contentinfo">
-        &#169; Copyright 2009-2018, Brazil, Inc.
+        &#169; Copyright 2009-2019, Brazil, Inc.
     </div>
   </body>
 </html>