rroonga 5.0.0-x86-mingw32 → 5.0.1-x86-mingw32

data/vendor/local/share/doc/groonga/en/html/_sources/reference/scorers/scorer_tf_at_most.txt

@@ -0,0 +1,22 @@
+.. -*- rst -*-
+
+.. highlightlang:: none
+
+.. groonga-command
+.. database: scorer_tf_at_most
+
+``scorer_tf_at_most``
+=====================
+
+.. note::
+
+   This scorer is an experimental feature.
+
+.. versionadded:: 5.0.1
+
+Summary
+-------
+
+``scorer_tf_at_most`` is a scorer based on TF (term frequency).
+
+TODO

data/vendor/local/share/doc/groonga/en/html/_sources/reference/scorers/scorer_tf_idf.txt

@@ -0,0 +1,110 @@
+.. -*- rst -*-
+
+.. highlightlang:: none
+
+.. groonga-command
+.. database: scorer_tf_idf
+
+``scorer_tf_idf``
+=================
+
+.. note::
+
+   This scorer is an experimental feature.
+
+.. versionadded:: 5.0.1
+
+Summary
+-------
+
+``scorer_tf_idf`` is a scorer based of `TF-IDF
+<http://en.wikipedia.org/wiki/Tf%E2%80%93idf>`_ (term
+frequency-inverse document frequency) score function.
+
+To put it simply, TF (term frequency) divided by DF (document
+frequency) is TF-IDF. "TF" means that "the number of occurrences is
+more important". "TF divided by DF" means that "the number of
+occurrences of important term is more important".
+
+The default score function in Groonga is TF (term frequency). It
+doesn't care about term importance but is fast.
+
+TF-IDF cares about term importance but is slower than TF.
+
+TF-IDF will compute more suitable score rather than TF for many cases.
+But it's not perfect.
+
+If document contains many same keywords such as "They are keyword,
+keyword, keyword ... and keyword". It increases score by TF and
+TF-IDF. Search engine spammer may use the technique. But TF-IDF
+doesn't guard from the technique.
+
+`Okapi BM25 <http://en.wikipedia.org/wiki/Okapi_BM25>`_ can solve the
+case. But it's more slower than TF-IDF and not implemented yet in
+Groonga.
+
+.. include:: ../scoring_note.rst
+
+Usage
+-----
+
+This section describes how to use this scorer.
+
+Here are a schema definition and sample data to show usage.
+
+Sample schema:
+
+.. groonga-command
+.. include:: ../example/reference/scorers/scorer_tf_idf/usage_setup_schema.log
+.. table_create Logs TABLE_NO_KEY
+.. column_create Logs message COLUMN_SCALAR Text
+..
+.. table_create Terms TABLE_PAT_KEY ShortText \
+..   --default_tokenizer TokenBigram \
+..   --normalizer NormalizerAuto
+.. column_create Terms message_index COLUMN_INDEX|WITH_POSITION Logs message
+
+Sample data:
+
+.. groonga-command
+.. include:: ../example/reference/scorers/scorer_tf_idf/usage_setup_data.log
+.. load --table Logs
+.. [
+.. {"message": "Error"},
+.. {"message": "Warning"},
+.. {"message": "Warning Warning"},
+.. {"message": "Warning Warning Warning"},
+.. {"message": "Info"},
+.. {"message": "Info Info"},
+.. {"message": "Info Info Info"},
+.. {"message": "Info Info Info Info"},
+.. {"message": "Notice"},
+.. {"message": "Notice Notice"},
+.. {"message": "Notice Notice Notice"},
+.. {"message": "Notice Notice Notice Notice"},
+.. {"message": "Notice Notice Notice Notice Notice"}
+.. ]
+
+You specify ``scorer_tf_idf`` in :ref:`select-match-columns` like the
+following:
+
+.. groonga-command
+.. include:: ../example/reference/scorers/scorer_tf_idf/usage_no_weight.log
+.. select Logs \
+..   --match_columns "scorer_tf_idf(message)" \
+..   --query "Error OR Info" \
+..   --output_columns "message, _score" \
+..   --sortby "-_score"
+
+Both the score of ``Info Info Info`` and the score of ``Error`` are
+``2`` even ``Info Info Info`` includes three ``Info`` terms. Because
+``Error`` is more important term rather than ``Info``. The number of
+documents that include ``Info`` is ``4``. The number of documents that
+include ``Error`` is ``1``. Term that is included in less documents
+means that the term is more characteristic term. Characteristic term
+is important term.
+
+See also
+--------
+
+* :doc:`../scorer`

data/vendor/local/share/doc/groonga/en/html/_sources/reference/scoring_note.txt

@@ -0,0 +1,13 @@
+.. _note:
+
+   You don't need to resolve scoring only by score function. Score
+   function is highly depends on search query. You may be able to use
+   metadata of matched record.
+
+   For example, Google uses `PageRank
+   <http://en.wikipedia.org/wiki/PageRank>`_ for scoring. You may be
+   able to use data type ("title" data are important rather than
+   "memo" data), tag, geolocation and so on.
+
+   Please stop to think about only score function for scoring.
+

data/vendor/local/share/doc/groonga/en/html/_sources/reference/tables.txt

@@ -69,6 +69,8 @@ prefix is omitted in the table.)
 | search       |            |              |             |             |
 +--------------+------------+--------------+-------------+-------------+
 
+.. _table-no-key:
+
 ``TABLE_NO_KEY``
 ^^^^^^^^^^^^^^^^
 
@@ -79,6 +81,8 @@ You cannot use ``TABLE_NO_KEY`` for lexicon for fulltext search
 because lexicon stores tokens as key. ``TABLE_NO_KEY`` is useful for
 no key records such as log.
 
+.. _table-hash-key:
+
 ``TABLE_HASH_KEY``
 ^^^^^^^^^^^^^^^^^^
 
@@ -88,6 +92,8 @@ functions such as common prefix search and predictive search.
 ``TABLE_HASH_KEY`` is useful for index for exact search such as tag
 search.
 
+.. _table-pat-key:
+
 ``TABLE_PAT_KEY``
 ^^^^^^^^^^^^^^^^^
 
@@ -96,6 +102,8 @@ search.
 ``TABLE_PAT_KEY`` is useful for lexicon for fulltext search and
 index for range search.
 
+.. _table-dat-key:
+
 ``TABLE_DAT_KEY``
 ^^^^^^^^^^^^^^^^^
 

data/vendor/local/share/doc/groonga/en/html/_sources/reference/tokenizers.txt

@@ -2,23 +2,537 @@
 
 .. highlightlang:: none
 
+.. groonga-command
+.. database: tokenizers
+
 Tokenizers
 ==========
 
-TODO: Write me.
-
-Here are the list of built-in tokenizers:
-
-* TokenBigram
-* TokenBigramSplitSymbol
-* TokenBigramSplitSymbolAlpha
-* TokenBigramSplitSymbolAlphaDigit
-* TokenBigramIgnoreBlank
-* TokenBigramIgnoreBlankSplitSymbol
-* TokenBigramIgnoreBlankSplitAlpha
-* TokenBigramIgnoreBlankSplitAlphaDigit
-* TokenDelimit
-* TokenDelimitNull
-* TokenTrigram
-* TokenUnigram
+Summary
+-------
+
+Groonga has tokenizer module that tokenizes text. It is used when
+the following cases:
+
+  * Indexing text
+
+    .. figure:: /images/reference/tokenizers/used-when-indexing.png
+       :align: center
+       :width: 80%
+
+       Tokenizer is used when indexing text.
+
+  * Searching by query
+
+    .. figure:: /images/reference/tokenizers/used-when-searching.png
+       :align: center
+       :width: 80%
+
+       Tokenizer is used when searching by query.
+
+Tokenizer is an important module for full-text search. You can change
+trade-off between `precision and recall
+<http://en.wikipedia.org/wiki/Precision_and_recall>`_ by changing
+tokenizer.
+
+Normally, :ref:`token-bigram` is a suitable tokenizer. If you don't
+know much about tokenizer, it's recommended that you choose
+:ref:`token-bigram`.
+
+You can try a tokenizer by :doc:`/reference/commands/tokenize` and
+:doc:`/reference/commands/table_tokenize`. Here is an example to
+try :ref:`token-bigram` tokenizer by
+:doc:`/reference/commands/tokenize`:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/tokenize-example.log
+.. tokenize TokenBigram "Hello World"
+
+What is "tokenize"?
+-------------------
+
+"tokenize" is the process that extracts zero or more tokens from a
+text. There are some "tokenize" methods.
+
+For example, ``Hello World`` is tokenized to the following tokens by
+bigram tokenize method:
+
+  * ``He``
+  * ``el``
+  * ``ll``
+  * ``lo``
+  * ``o_`` (``_`` means a white-space)
+  * ``_W`` (``_`` means a white-space)
+  * ``Wo``
+  * ``or``
+  * ``rl``
+  * ``ld``
+
+In the above example, 10 tokens are extracted from one text ``Hello
+World``.
+
+For example, ``Hello World`` is tokenized to the following tokens by
+white-space-separate tokenize method:
+
+  * ``Hello``
+  * ``World``
+
+In the above example, 2 tokens are extracted from one text ``Hello
+World``.
+
+Token is used as search key. You can find indexed documents only by
+tokens that are extracted by used tokenize method. For example, you
+can find ``Hello World`` by ``ll`` with bigram tokenize method but you
+can't find ``Hello World`` by ``ll`` with white-space-separate tokenize
+method. Because white-space-separate tokenize method doesn't extract
+``ll`` token. It just extracts ``Hello`` and ``World`` tokens.
+
+In general, tokenize method that generates small tokens increases
+recall but decreases precision. Tokenize method that generates large
+tokens increases precision but decreases recall.
+
+For example, we can find ``Hello World`` and ``A or B`` by ``or`` with
+bigram tokenize method. ``Hello World`` is a noise for people who
+wants to search "logical and". It means that precision is
+decreased. But recall is increased.
+
+We can find only ``A or B`` by ``or`` with white-space-separate
+tokenize method. Because ``World`` is tokenized to one token ``World``
+with white-space-separate tokenize method. It means that precision is
+increased for people who wants to search "logical and". But recall is
+decreased because ``Hello World`` that contains ``or`` isn't found.
+
+Built-in tokenizsers
+--------------------
+
+Here is a list of built-in tokenizers:
+
+  * ``TokenBigram``
+  * ``TokenBigramSplitSymbol``
+  * ``TokenBigramSplitSymbolAlpha``
+  * ``TokenBigramSplitSymbolAlphaDigit``
+  * ``TokenBigramIgnoreBlank``
+  * ``TokenBigramIgnoreBlankSplitSymbol``
+  * ``TokenBigramIgnoreBlankSplitAlpha``
+  * ``TokenBigramIgnoreBlankSplitAlphaDigit``
+  * ``TokenUnigram``
+  * ``TokenTrigram``
+  * ``TokenDelimit``
+  * ``TokenDelimitNull``
+  * ``TokenMecab``
+  * ``TokenRegexp``
+
+.. _token-bigram:
+
+``TokenBigram``
+^^^^^^^^^^^^^^^
+
+``TokenBigram`` is a bigram based tokenizer. It's recommended to use
+this tokenizer for most cases.
+
+Bigram tokenize method tokenizes a text to two adjacent characters
+tokens. For example, ``Hello`` is tokenized to the following tokens:
+
+  * ``He``
+  * ``el``
+  * ``ll``
+  * ``lo``
+
+Bigram tokenize method is good for recall because you can find all
+texts by query consists of two or more characters.
+
+In general, you can't find all texts by query consists of one
+character because one character token doesn't exist. But you can find
+all texts by query consists of one character in Groonga. Because
+Groonga find tokens that start with query by predictive search. For
+example, Groonga can find ``ll`` and ``lo`` tokens by ``l`` query.
+
+Bigram tokenize method isn't good for precision because you can find
+texts that includes query in word. For example, you can find ``world``
+by ``or``. This is more sensitive for ASCII only languages rather than
+non-ASCII languages. ``TokenBigram`` has solution for this problem
+described in the bellow.
+
+``TokenBigram`` behavior is different when it's worked with any
+:doc:`/reference/normalizers`.
+
+If no normalizer is used, ``TokenBigram`` uses pure bigram (all tokens
+except the last token have two characters) tokenize method:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-no-normalizer.log
+.. tokenize TokenBigram "Hello World"
+
+If normalizer is used, ``TokenBigram`` uses white-space-separate like
+tokenize method for ASCII characters. ``TokenBigram`` uses bigram
+tokenize method for non-ASCII characters.
+
+You may be confused with this combined behavior. But it's reasonable
+for most use cases such as English text (only ASCII characters) and
+Japanese text (ASCII and non-ASCII characters are mixed).
+
+Most languages consists of only ASCII characters use white-space for
+word separator. White-space-separate tokenize method is suitable for
+the case.
+
+Languages consists of non-ASCII characters don't use white-space for
+word separator. Bigram tokenize method is suitable for the case.
+
+Mixed tokenize method is suitable for mixed language case.
+
+If you want to use bigram tokenize method for ASCII character, see
+``TokenBigramSplitXXX`` type tokenizers such as
+:ref:`token-bigram-split-symbol-alpha`.
+
+Let's confirm ``TokenBigram`` behavior by example.
+
+``TokenBigram`` uses one or more white-spaces as token delimiter for
+ASCII characters:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-ascii-and-white-space-with-normalizer.log
+.. tokenize TokenBigram "Hello World" NormalizerAuto
+
+``TokenBigram`` uses character type change as token delimiter for
+ASCII characters. Character type is one of them:
+
+  * Alphabet
+  * Digit
+  * Symbol (such as ``(``, ``)`` and ``!``)
+  * Hiragana
+  * Katakana
+  * Kanji
+  * Others
+
+The following example shows two token delimiters:
+
+  * at between ``100`` (digits) and ``cents`` (alphabets)
+  * at between ``cents`` (alphabets) and ``!!!`` (symbols)
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-ascii-and-character-type-change-with-normalizer.log
+.. tokenize TokenBigram "100cents!!!" NormalizerAuto
+
+Here is an example that ``TokenBigram`` uses bigram tokenize method
+for non-ASCII characters.
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-non-ascii-with-normalizer.log
+.. tokenize TokenBigram "日本語の勉強" NormalizerAuto
+
+.. _token-bigram-split-symbol:
+
+``TokenBigramSplitSymbol``
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``TokenBigramSplitSymbol`` is similar to :ref:`token-bigram`. The
+difference between them is symbol handling. ``TokenBigramSplitSymbol``
+tokenizes symbols by bigram tokenize method:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-split-symbol-with-normalizer.log
+.. tokenize TokenBigramSplitSymbol "100cents!!!" NormalizerAuto
+
+.. _token-bigram-split-symbol-alpha:
+
+``TokenBigramSplitSymbolAlpha``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``TokenBigramSplitSymbolAlpha`` is similar to :ref:`token-bigram`. The
+difference between them is symbol and alphabet
+handling. ``TokenBigramSplitSymbolAlpha`` tokenizes symbols and
+alphabets by bigram tokenize method:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-split-symbol-alpha-with-normalizer.log
+.. tokenize TokenBigramSplitSymbolAlpha "100cents!!!" NormalizerAuto
+
+.. _token-bigram-split-symbol-alpha-digit:
+
+``TokenBigramSplitSymbolAlphaDigit``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``TokenBigramSplitSymbolAlphaDigit`` is similar to
+:ref:`token-bigram`. The difference between them is symbol, alphabet
+and digit handling. ``TokenBigramSplitSymbolAlphaDigit`` tokenizes
+symbols, alphabets and digits by bigram tokenize method. It means that
+all characters are tokenized by bigram tokenize method:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-split-symbol-alpha-digit-with-normalizer.log
+.. tokenize TokenBigramSplitSymbolAlphaDigit "100cents!!!" NormalizerAuto
+
+.. _token-bigram-ignore-blank:
+
+``TokenBigramIgnoreBlank``
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``TokenBigramIgnoreBlank`` is similar to :ref:`token-bigram`. The
+difference between them is blank handling. ``TokenBigramIgnoreBlank``
+ignores white-spaces in continuous symbols and non-ASCII characters.
+
+You can find difference of them by ``日 本 語 ! ! !`` text because it
+has symbols and non-ASCII characters.
+
+Here is a result by :ref:`token-bigram` :
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-with-white-spaces.log
+.. tokenize TokenBigram "日 本 語 ! ! !" NormalizerAuto
+
+Here is a result by ``TokenBigramIgnoreBlank``:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-ignore-blank-with-white-spaces.log
+.. tokenize TokenBigramIgnoreBlank "日 本 語 ! ! !" NormalizerAuto
+
+.. _token-bigram-ignore-blank-split-symbol:
+
+``TokenBigramIgnoreBlankSplitSymbol``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``TokenBigramIgnoreBlankSplitSymbol`` is similar to
+:ref:`token-bigram`. The differences between them are the followings:
+
+  * Blank handling
+  * Symbol handling
+
+``TokenBigramIgnoreBlankSplitSymbol`` ignores white-spaces in
+continuous symbols and non-ASCII characters.
+
+``TokenBigramIgnoreBlankSplitSymbol`` tokenizes symbols by bigram
+tokenize method.
+
+You can find difference of them by ``日 本 語 ! ! !`` text because it
+has symbols and non-ASCII characters.
+
+Here is a result by :ref:`token-bigram` :
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-with-white-spaces-and-symbol.log
+.. tokenize TokenBigram "日 本 語 ! ! !" NormalizerAuto
+
+Here is a result by ``TokenBigramIgnoreBlankSplitSymbol``:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-ignore-blank-split-symbol-with-white-spaces-and-symbol.log
+.. tokenize TokenBigramIgnoreBlankSplitSymbol "日 本 語 ! ! !" NormalizerAuto
+
+.. _token-bigram-ignore-blank-split-symbol-alpha:
+
+``TokenBigramIgnoreBlankSplitSymbolAlpha``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``TokenBigramIgnoreBlankSplitSymbolAlpha`` is similar to
+:ref:`token-bigram`. The differences between them are the followings:
+
+  * Blank handling
+  * Symbol and alphabet handling
+
+``TokenBigramIgnoreBlankSplitSymbolAlpha`` ignores white-spaces in
+continuous symbols and non-ASCII characters.
+
+``TokenBigramIgnoreBlankSplitSymbolAlpha`` tokenizes symbols and
+alphabets by bigram tokenize method.
+
+You can find difference of them by ``Hello 日 本 語 ! ! !`` text because it
+has symbols and non-ASCII characters with white spaces and alphabets.
+
+Here is a result by :ref:`token-bigram` :
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-with-white-spaces-and-symbol-and-alphabet.log
+.. tokenize TokenBigram "Hello 日 本 語 ! ! !" NormalizerAuto
+
+Here is a result by ``TokenBigramIgnoreBlankSplitSymbolAlpha``:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-ignore-blank-split-symbol-with-white-spaces-and-symbol-and-alphabet.log
+.. tokenize TokenBigramIgnoreBlankSplitSymbolAlpha "Hello 日 本 語 ! ! !" NormalizerAuto
+
+.. _token-bigram-ignore-blank-split-symbol-alpha-digit:
+
+``TokenBigramIgnoreBlankSplitSymbolAlphaDigit``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``TokenBigramIgnoreBlankSplitSymbolAlphaDigit`` is similar to
+:ref:`token-bigram`. The differences between them are the followings:
+
+  * Blank handling
+  * Symbol, alphabet and digit handling
+
+``TokenBigramIgnoreBlankSplitSymbolAlphaDigit`` ignores white-spaces
+in continuous symbols and non-ASCII characters.
+
+``TokenBigramIgnoreBlankSplitSymbolAlphaDigit`` tokenizes symbols,
+alphabets and digits by bigram tokenize method. It means that all
+characters are tokenized by bigram tokenize method.
+
+You can find difference of them by ``Hello 日 本 語 ! ! ! 777`` text
+because it has symbols and non-ASCII characters with white spaces,
+alphabets and digits.
+
+Here is a result by :ref:`token-bigram` :
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-with-white-spaces-and-symbol-and-alphabet-and-digit.log
+.. tokenize TokenBigram "Hello 日 本 語 ! ! ! 777" NormalizerAuto
+
+Here is a result by ``TokenBigramIgnoreBlankSplitSymbolAlphaDigit``:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-bigram-ignore-blank-split-symbol-with-white-spaces-and-symbol-and-alphabet-digit.log
+.. tokenize TokenBigramIgnoreBlankSplitSymbolAlphaDigit "Hello 日 本 語 ! ! ! 777" NormalizerAuto
+
+.. _token-unigram:
+
+``TokenUnigram``
+^^^^^^^^^^^^^^^^
+
+``TokenUnigram`` is similar to :ref:`token-bigram`. The differences
+between them is token unit. :ref:`token-bigram` uses 2 characters per
+token. ``TokenUnigram`` uses 1 character per token.
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-unigram.log
+.. tokenize TokenUnigram "100cents!!!" NormalizerAuto
+
+.. _token-trigram:
+
+``TokenTrigram``
+^^^^^^^^^^^^^^^^
+
+``TokenTrigram`` is similar to :ref:`token-bigram`. The differences
+between them is token unit. :ref:`token-bigram` uses 2 characters per
+token. ``TokenTrigram`` uses 3 characters per token.
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-trigram.log
+.. tokenize TokenTrigram "10000cents!!!!!" NormalizerAuto
+
+.. _token-delimit:
+
+``TokenDelimit``
+^^^^^^^^^^^^^^^^
+
+``TokenDelimit`` extracts token by splitting one or more space
+characters (``U+0020``). For example, ``Hello World`` is tokenized to
+``Hello`` and ``World``.
+
+``TokenDelimit`` is suitable for tag text. You can extract ``groonga``
+and ``full-text-search`` and ``http`` as tags from ``groonga
+full-text-search http``.
+
+Here is an example of ``TokenDelimit``:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-delimit.log
+.. tokenize TokenDelimit "Groonga full-text-search HTTP" NormalizerAuto
+
+.. _token-delimit-null:
+
+``TokenDelimitNull``
+^^^^^^^^^^^^^^^^^^^^
+
+``TokenDelimitNull`` is similar to :ref:`token-delimit`. The
+difference between them is separator character. :ref:`token-delimit`
+uses space character (``U+0020``) but ``TokenDelimitNull`` uses NUL
+character (``U+0000``).
+
+``TokenDelimitNull`` is also suitable for tag text.
+
+Here is an example of ``TokenDelimitNull``:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-delimit-null.log
+.. tokenize TokenDelimitNull "Groonga\u0000full-text-search\u0000HTTP" NormalizerAuto
+
+.. _token-mecab:
+
+``TokenMecab``
+^^^^^^^^^^^^^^
+
+``TokenMecab`` is a tokenizer based on `MeCab
+<http://mecab.sourceforge.net/>`_ part-of-speech and
+morphological analyzer.
+
+MeCab doesn't depend on Japanese. You can use MeCab for other
+languages by creating dictionary for the languages. You can use `NAIST
+Japanese Dictionary <http://sourceforge.jp/projects/naist-jdic/>`_
+for Japanese.
+
+``TokenMecab`` is good for precision rather than recall. You can find
+``東京都`` and ``京都`` texts by ``京都`` query with
+:ref:`token-bigram` but ``東京都`` isn't expected. You can find only
+``京都`` text by ``京都`` query with ``TokenMecab``.
+
+If you want to support neologisms, you need to keep updating your
+MeCab dictionary. It needs maintain cost. (:ref:`token-bigram` doesn't
+require dictionary maintenance because :ref:`token-bigram` doesn't use
+dictionary.) `mecab-ipadic-NEologd : Neologism dictionary for MeCab
+<https://github.com/neologd/mecab-ipadic-neologd>`_ may help you.
+
+Here is an example of ``TokenMeCab``. ``東京都`` is tokenized to ``東京``
+and ``都``. They don't include ``京都``:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-mecab.log
+.. tokenize TokenMecab "東京都"
+
+.. _token-regexp:
+
+``TokenRegexp``
+^^^^^^^^^^^^^^^
+
+.. versionadded:: 5.0.1
+
+.. caution::
+
+   This tokenizer is experimental. Specification may be changed.
+
+.. caution::
+
+   This tokenizer can be used only with UTF-8. You can't use this
+   tokenizer with EUC-JP, Shift_JIS and so on.
+
+``TokenRegexp`` is a tokenizer for supporting regular expression
+search by index.
+
+In general, regular expression search is evaluated as sequential
+search. But the following cases can be evaluated as index search:
+
+  * Literal only case such as ``hello``
+  * The beginning of text and literal case such as ``\A/home/alice``
+  * The end of text and literal case such as ``\.txt\z``
+
+In most cases, index search is faster than sequential search.
+
+``TokenRegexp`` is based on bigram tokenize method. ``TokenRegexp``
+adds the beginning of text mark (``U+FFEF``) at the begging of text
+and the end of text mark (``U+FFF0``) to the end of text when you
+index text:
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-regexp-add.log
+.. tokenize TokenRegexp "/home/alice/test.txt" NormalizerAuto --mode ADD
+
+The beginning of text mark is used for the beginning of text search by
+``\A``. If you use ``TokenRegexp`` for tokenizing query,
+``TokenRegexp`` adds the beginning of text mark (``U+FFEF``) as the
+first token. The beginning of text mark must be appeared at the first,
+you can get results of the beginning of text search.
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-regexp-get-beginning-of-text.log
+.. tokenize TokenRegexp "\\A/home/alice/" NormalizerAuto --mode GET
+
+The end of text mark is used for the end of text search by ``\z``.
+If you use ``TokenRegexp`` for tokenizing query, ``TokenRegexp`` adds
+the end of text mark (``U+FFF0``) as the last token. The end of text
+mark must be appeared at the end, you can get results of the end of
+text search.
+
+.. groonga-command
+.. include:: ../example/reference/tokenizers/token-regexp-get-end-of-text.log
+.. tokenize TokenRegexp "\\.txt\\z" NormalizerAuto --mode GET