rabbit-slide-kou-jpmug-db-study-1 2018.1.30.0

data/memo.md

@@ -0,0 +1,304 @@
+# メモ
+
+## ベンチマーク
+
+### データ
+
+シカゴの犯罪データ。651万レコード。
+
+https://catalog.data.gov/dataset/crimes-2001-to-present-398a4
+
+### バージョン
+
+MariaDBは10.3.2。Mroongaは7.10。
+
+`innodb_buffer_pool_size`は512M。
+
+### インデックス作成時間
+
+Mroongaのテーブルの`block`に全文検索インデックスを作成。7秒。
+
+```text
+MariaDB> create fulltext index block_index on crimes (block);
+Query OK, 0 rows affected (6.989 sec)
+Records: 0  Duplicates: 0  Warnings: 0
+```
+
+Mroongaのテーブルの`description`に全文検索インデックスを作成。6秒。
+
+```text
+MariaDB> create fulltext index description_index on crimes (description);
+Query OK, 0 rows affected (5.873 sec)
+Records: 0  Duplicates: 0  Warnings: 0
+```
+
+InnoDBのテーブルの`block`に全文検索インデックスを作成。35秒。
+
+```text
+MariaDB> create fulltext index block_index on crimes2 (block);
+Query OK, 0 rows affected (35.382 sec)
+Records: 0  Duplicates: 0  Warnings: 0
+```
+
+InnoDBのテーブルの`description`に全文検索インデックスを作成。1分50秒。
+
+```text
+MariaDB> create fulltext index description_index on crimes2 (description);
+Query OK, 0 rows affected, 1 warning (1 min 49.443 sec)
+Records: 0  Duplicates: 0  Warnings: 1
+```
+
+### 全文検索だけ
+
+半分くらい（300万件くらい）ヒットするケース。Mroongaは1秒。
+
+```text
+MariaDB > select count(*) from crimes where match(block) against('+ave' in boolean mode);
++----------+
+| count(*) |
++----------+
+|  3126521 |
++----------+
+1 row in set (1.066 sec)
+```
+
+半分くらい（300万件くらい）ヒットするケース。Groongaは0.3秒。
+
+```text
+MariaDB > select mroonga_command('select --table crimes --query "block:@ave" --limit 0 --output_columns _key') as response;
++----------------------------------+
+| response                         |
++----------------------------------+
+| [[[3126521],[["_key","Int32"]]]] |
++----------------------------------+
+1 row in set (0.305 sec)
+```
+
+半分くらい（300万件くらい）ヒットするケース。InnoDBは18秒。
+
+```text
+MariaDB> select count(*) from crimes2 where match(block) against('+ave' in boolean mode);
++----------+
+| count(*) |
++----------+
+|  3126521 |
++----------+
+1 row in set (17.563 sec)
+```
+
+4万件くらいヒットするケース。Mroongaは0.02秒。
+
+```text
+MariaDB> select count(*) from crimes where match(block) against('+milwaukee' in boolean mode);
++----------+
+| count(*) |
++----------+
+|    38781 |
++----------+
+1 row in set (0.022 sec)
+```
+
+4万件くらいヒットするケース。Groongaは0.01秒。
+
+```text
+MariaDB> select mroonga_command('select --table crimes --query "block:@milwaukee" --limit 0 --output_columns _key') as response;
++--------------------------------+
+| response                       |
++--------------------------------+
+| [[[38781],[["_key","Int32"]]]] |
++--------------------------------+
+1 row in set (0.010 sec)
+```
+
+4万件ヒットするケース。InnoDBは0.2秒。
+
+```text
+MariaDB> select count(*) from crimes2 where match(block) against('+milwaukee' in boolean mode);
++----------+
+| count(*) |
++----------+
+|    38781 |
++----------+
+1 row in set (0.189 sec)
+```
+
+### 通常の検索だけ
+
+数値の等価条件1つで、26万件ヒットするケース。Mroongaの通常バージョンは1.3秒。
+
+```text
+MariaDB> select count(*) from crimes where year = 2017;
++----------+
+| count(*) |
++----------+
+|   265156 |
++----------+
+1 row in set (1.285 sec)
+```
+
+数値の等価条件1つで、26万件ヒットするケース。Mroongaの最適化ONバージョンは0.4秒。
+
+```text
+MariaDB> set mroonga_condition_push_down_type = all;
+Query OK, 0 rows affected (0.000 sec)
+
+MariaDB> select count(*) from crimes where year = 2017;
++----------+
+| count(*) |
++----------+
+|   265156 |
++----------+
+1 row in set (0.395 sec)
+```
+
+数値の等価条件1つで、26万件ヒットするケース。Groongaは0.4秒。
+
+```text
+MariaDB> select mroonga_command('select --table crimes --filter "year == 2017" --limit 0 --output_columns _key') as response;
++---------------------------------+
+| response                        |
++---------------------------------+
+| [[[265156],[["_key","Int32"]]]] |
++---------------------------------+
+1 row in set (0.361 sec)
+```
+
+数値の等価条件1つで、26万件ヒットするケース。InnoDBは1.3秒。
+
+```text
+MariaDB> select count(*) from crimes2 where year = 2017;
++----------+
+| count(*) |
++----------+
+|   265156 |
++----------+
+1 row in set (1.304 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つで、7000件ヒットするケース。Mroongaの通常バージョンは2.3秒。
+
+```text
+MariaDB> set mroonga_condition_push_down_type = default;
+Query OK, 0 rows affected (0.000 sec)
+
+MariaDB> select count(*) from crimes where year = 2017 and domestic = true and arrest = true;
++----------+
+| count(*) |
++----------+
+|     7148 |
++----------+
+1 row in set (2.262 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つで、7000件ヒットするケース。Mroongaの最適化ONバージョンは0.4秒。
+
+```text
+MariaDB> set mroonga_condition_push_down_type = all;
+Query OK, 0 rows affected (0.000 sec)
+
+MariaDB> select count(*) from crimes where year = 2017 and domestic = true and arrest = true;
++----------+
+| count(*) |
++----------+
+|     7148 |
++----------+
+1 row in set (0.365 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つで、7000件ヒットするケース。Groongaは0.4秒。
+
+```text
+MariaDB> select mroonga_command('select --table crimes --filter "year == 2017 && domestic == true && arrest == true" --limit 0 --output_columns _key') as response;
++-------------------------------+
+| response                      |
++-------------------------------+
+| [[[7148],[["_key","Int32"]]]] |
++-------------------------------+
+1 row in set (0.365 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つで、7000件ヒットするケース。InnoDBは1.6秒。
+
+```text
+MariaDB> select count(*) from crimes2 where year = 2017 and domestic = true and arrest = true;
++----------+
+| count(*) |
++----------+
+|     7148 |
++----------+
+1 row in set (1.582 sec)
+```
+
+### 全文検索と通常の検索
+
+数値の等価条件1つと真偽値の等価条件2つと全文検索（300万件くらいヒット）で、4000件ヒットするケース。Mroongaは0.4秒。（このときは常に最適化が効く。）
+
+```text
+MariaDB> select count(*) from crimes where year = 2017 and domestic = true and arrest = true and match(block) against('+ave' in boolean mode);
++----------+
+| count(*) |
++----------+
+|     3982 |
++----------+
+1 row in set (0.440 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つと全文検索（300万件くらいヒット）で、4000件ヒットするケース。Groongaは0.4秒。
+
+```text
+MariaDB> select mroonga_command('select --table crimes --filter "block @ \'ave\' && year == 2017 && domestic == true && arrest == true" --limit 0 --output_columns _key') as response;
++-------------------------------+
+| response                      |
++-------------------------------+
+| [[[3982],[["_key","Int32"]]]] |
++-------------------------------+
+1 row in set (0.436 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つと全文検索（300万件くらいヒット）で、4000件ヒットするケース。InnoDBは18秒。
+
+```text
+MariaDB> select count(*) from crimes2 where year = 2017 and domestic = true and arrest = true and match(block) against('+ave' in boolean mode);
++----------+
+| count(*) |
++----------+
+|     3982 |
++----------+
+1 row in set (17.697 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つと全文検索（4万件くらいヒット）で、10件ヒットするケース。Mroongaは0.01秒。（このときは常に最適化が効く。）
+
+```text
+MariaDB> select count(*) from crimes where year = 2017 and domestic = true and arrest = true and match(block) against('+milwaukee' in boolean mode);
++----------+
+| count(*) |
++----------+
+|       10 |
++----------+
+1 row in set (0.010 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つと全文検索（4万件くらいヒット）で、10件ヒットするケース。Groongaは0.01秒。
+
+```text
+MariaDB> select mroonga_command('select --table crimes --filter "block @ \'milwaukee\' && year == 2017 && domestic == true && arrest == true" --limit 0 --output_columns _key') as response;
++-----------------------------+
+| response                    |
++-----------------------------+
+| [[[10],[["_key","Int32"]]]] |
++-----------------------------+
+1 row in set (0.010 sec)
+```
+
+数値の等価条件1つと真偽値の等価条件2つと全文検索（4万件くらいヒット）で、10件ヒットするケース。InnoDBは0.2秒。
+
+```text
+MariaDB> select count(*) from crimes2 where year = 2017 and domestic = true and arrest = true and match(block) against('+milwaukee' in boolean mode);
++----------+
+| count(*) |
++----------+
+|       10 |
++----------+
+1 row in set (0.197 sec)
+```

data/theme.rb

@@ -0,0 +1,5 @@
+@title_slide_title_font_size = @large_font_size
+
+@groonga_product = "mroonga"
+
+include_theme("groonga")

metadata

@@ -0,0 +1,90 @@
+--- !ruby/object:Gem::Specification
+name: rabbit-slide-kou-jpmug-db-study-1
+version: !ruby/object:Gem::Version
+  version: 2018.1.30.0
+platform: ruby
+authors:
+- 須藤功平
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-01-30 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rabbit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.2
+- !ruby/object:Gem::Dependency
+  name: rabbit-theme-groonga
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: MariaDBには超高速な全文検索を実現するMroongaストレージエンジンがバンドルされていることを知っていますか？Mroongaを使うと日本語だけでなくアジア圏の言語も含むすべての言語をサポートした超高速な全文検索システムを簡単に作ることができます。どれだけ簡単に作ることができるか紹介します。
+email:
+- kou@clear-code.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".rabbit"
+- README.rd
+- Rakefile
+- config.yaml
+- images/chupa-text-web-ui-extract-metadata.png
+- images/chupa-text-web-ui-extract-text-and-screenshot.png
+- images/chupa-text-web-ui-form.png
+- images/column-store.svg
+- images/count-star.svg
+- images/not-access-to-needless-columns.svg
+- images/per-column-processing.svg
+- images/php-document-search-search.png
+- images/php-document-search.png
+- mariadb-mroonga-all-languages-supported-super-fast-full-text-search-system.rab
+- memo.md
+- pdf/jpmug-db-study-1-mariadb-mroonga-all-languages-supported-super-fast-full-text-search-system.pdf
+- theme.rb
+homepage: http://slide.rabbit-shocker.org/authors/kou/jpmug-db-study-1/
+licenses:
+- CC-BY-SA-4.0
+- CC-BY-3.0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.2.2
+signing_key: 
+specification_version: 4
+summary: MariaDBとMroongaで作る全言語対応超高速全文検索システム
+test_files: []