gmail_search_syntax 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bb674a944e51bd81d0690d2d972c0b63bae352a30469d5d1719b959d78316b4f
-  data.tar.gz: 03fc2f44531f2d610e6dc189ab7104bf753cb3fff325de5f6e388b4529e7c9d3
+  metadata.gz: 60c2512ef5571bfb496971d202b28148946bdc60355f96d2048151cfb12956d4
+  data.tar.gz: 6aac7ff084afb11ff543fc87c6d78dd8efd702ae69ca5a9c6a63348239743134
 SHA512:
-  metadata.gz: 13394002e2a5546876608249caeb8ed3379e6cad5bab7aaf324830f5ca885c9838838d3416eb6e4b6f5a17cc728e453dc586648572db769b7e5ec9b6fa758244
-  data.tar.gz: f22df8edc76415087dc990be7c895ec2af13545c7238c573cf129bbf8152adceded0f82d1d62dadebdc71f00bcb37edd477ec28661c365efb549dcc6e9688551
+  metadata.gz: e2ff3554682a4a877d289aa11970bc77c2cde13bc532e485b693e52fe408482372079ab8119e10ace3234097410128b4ea48dddb24a317cc2894f32ed0f34f7d
+  data.tar.gz: 64dd03a936bcfae224b8ee54ee9d462901daaf7fcc9edecc86e760494ec11fb9c0154d5f7c8203b01772d8e423851d2df79ac03707ffaa50d70bcb8eef57f53a

data/IMPLEMENTATION_NOTES.md

@@ -0,0 +1,174 @@
+# Implementation Notes: StringToken vs Substring Nodes
+
+## Overview
+
+This implementation distinguishes between **word boundary matching** (unquoted text) and **substring matching** (quoted text) in the Gmail search syntax parser.
+
+## Changes Made
+
+### 1. Renamed and New AST Nodes
+
+- **Renamed** `Text` to `StringToken` for clarity - represents unquoted text tokens
+- **Added** `Substring` node to the AST (`lib/gmail_search_syntax/ast.rb`) that represents quoted strings.
+
+```ruby
+class Substring < Node
+  attr_reader :value
+  
+  def initialize(value)
+    @value = value
+  end
+  
+  def inspect
+    "#<Substring #{@value.inspect}>"
+  end
+end
+```
+
+### 2. Parser Updates
+
+Modified the parser (`lib/gmail_search_syntax/parser.rb`) to create:
+- `StringToken` nodes for unquoted text
+- `Substring` nodes for quoted strings (`:quoted_string` tokens)
+
+### 3. SQL Visitor Updates
+
+Updated the SQL visitor (`lib/gmail_search_syntax/sql_visitor.rb`) with two different behaviors:
+
+#### StringToken Node (Word Boundary Matching)
+```ruby
+def visit_string_token(node)
+  # Matches complete words only
+  # Uses: = exact, LIKE "value %", LIKE "% value", LIKE "% value %"
+end
+```
+
+SQL Pattern:
+```sql
+(m0.subject = ? OR m0.subject LIKE ? OR m0.subject LIKE ? OR m0.subject LIKE ?)
+OR
+(m0.body = ? OR m0.body LIKE ? OR m0.body LIKE ? OR m0.body LIKE ?)
+```
+
+Parameters: `["meeting", "meeting %", "% meeting", "% meeting %", ...]`
+
+**Matches:** "meeting tomorrow", "the meeting", "just meeting"  
+**Does NOT match:** "meetings", "premeeting", "meetingroom"
+
+#### Substring Node (Partial Matching)
+```ruby
+def visit_substring(node)
+  # Matches anywhere in the text
+  # Uses: LIKE "%value%"
+end
+```
+
+SQL Pattern:
+```sql
+(m0.subject LIKE ? OR m0.body LIKE ?)
+```
+
+Parameters: `["%meeting%", "%meeting%"]`
+
+**Matches:** "meeting", "meetings", "premeeting", "meetingroom"
+
+## Examples
+
+### Unquoted (Word Boundary)
+```ruby
+GmailSearchSyntax.parse!("meeting")
+# => #<StringToken "meeting">
+# SQL: ... WHERE m0.subject = ? OR m0.subject LIKE ? OR ...
+```
+
+### Quoted (Substring)
+```ruby
+GmailSearchSyntax.parse!('"meeting"')
+# => #<Substring "meeting">
+# SQL: ... WHERE m0.subject LIKE ? OR m0.body LIKE ?
+```
+
+### Combined
+```ruby
+GmailSearchSyntax.parse!('urgent "q1 report"')
+# => #<And #<StringToken "urgent"> AND #<Substring "q1 report">>
+```
+
+## Rationale
+
+This implementation provides:
+
+1. **More precise searching** - Unquoted text matches complete words/tokens, avoiding false positives from partial matches
+2. **Flexible substring search** - Quoted text still allows finding substrings when needed
+3. **Gmail-like behavior** - Aligns with user expectations from Gmail's search syntax
+4. **SQL efficiency** - Word boundary matching is more specific than substring matching
+
+## Escape Sequences
+
+Both `StringToken` and `Substring` nodes support escape sequences in **both quoted and unquoted tokens**:
+
+### Supported Escapes
+
+- `\"` - Literal double quote
+- `\\` - Literal backslash
+- Other escape sequences (e.g., `\n`, `\t`) are preserved as-is (backslash + character)
+
+### Examples
+
+**Quoted Strings (Substring nodes):**
+```ruby
+# Escaped quotes in quoted string
+'"She said \\"hello\\" to me"'
+# => #<Substring 'She said "hello" to me'>
+
+# Escaped backslashes in quoted string
+'"path\\\\to\\\\file"'
+# => #<Substring 'path\\to\\file'>
+
+# In operator values with quoted strings
+'subject:"Meeting: \\"Q1 Review\\""'
+# => #<Operator subject: 'Meeting: "Q1 Review"'>
+```
+
+**Unquoted Tokens (StringToken nodes):**
+```ruby
+# Escaped quotes in unquoted token
+'meeting\\"room'
+# => #<StringToken 'meeting"room'>
+
+# Escaped backslashes in unquoted token
+'path\\\\to\\\\file'
+# => #<StringToken 'path\\to\\file'>
+
+# In operator values with unquoted tokens
+'subject:test\\"value'
+# => #<Operator subject: 'test"value'>
+```
+
+This allows you to include literal quotes and backslashes in any token, whether quoted or unquoted.
+
+## Testing
+
+All tests pass with comprehensive coverage:
+- Basic functionality tests
+- Escape sequence tests in tokenizer
+- Integration tests for parsing with escaped quotes
+- SQL generation tests with escaped quotes
+
+New tests added:
+- `test_quoted_text_search_uses_substring` in `test/sql_visitor_test.rb`
+- `test_tokenize_quoted_string_with_escaped_quote` in `test/tokenizer_test.rb`
+- `test_tokenize_quoted_string_with_escaped_backslash` in `test/tokenizer_test.rb`
+- `test_tokenize_word_with_escaped_quote` in `test/tokenizer_test.rb`
+- `test_tokenize_word_with_escaped_backslash` in `test/tokenizer_test.rb`
+- `test_quoted_string_with_escaped_quotes` in `test/gmail_search_syntax_test.rb`
+- `test_unquoted_text_with_escaped_quote` in `test/gmail_search_syntax_test.rb`
+- `test_unquoted_text_with_escaped_backslash` in `test/gmail_search_syntax_test.rb`
+- `test_subject_with_escaped_quotes` in `test/sql_visitor_test.rb`
+- `test_unquoted_token_with_escaped_quote` in `test/sql_visitor_test.rb`
+- `test_operator_with_unquoted_escaped_quote` in `test/sql_visitor_test.rb`
+
+Run demos:
+- `bundle exec ruby examples/text_vs_substring_demo.rb`
+- `bundle exec ruby examples/escaped_quotes_demo.rb`
+

data/README.md

@@ -39,11 +39,11 @@ GmailSearchSyntax.parse!("from:amy OR from:bob")
 
 # Negation
 GmailSearchSyntax.parse!("dinner -movie")
-# => #<And #<Text "dinner"> AND #<Not #<Text "movie">>>
+# => #<And #<StringToken "dinner"> AND #<Not #<StringToken "movie">>>
 
 # Proximity search
 GmailSearchSyntax.parse!("holiday AROUND 10 vacation")
-# => #<Around #<Text "holiday"> AROUND 10 #<Text "vacation">>
+# => #<Around #<StringToken "holiday"> AROUND 10 #<StringToken "vacation">>
 
 # Complex query with OR inside operator values
 GmailSearchSyntax.parse!("from:{alice@ bob@} subject:urgent")

data/examples/escaped_quotes_demo.rb

@@ -0,0 +1,152 @@
+#!/usr/bin/env ruby
+
+require_relative "../lib/gmail_search_syntax"
+
+puts "=" * 80
+puts "Escaped Quotes Demo"
+puts "=" * 80
+puts
+
+# Example 1: Escaped quotes in a substring search
+puts "1. Substring with escaped quotes"
+puts "-" * 40
+query1 = '"She said \\"hello\\" to me"'
+puts "Input: #{query1}"
+ast1 = GmailSearchSyntax.parse!(query1)
+puts "AST: #{ast1.inspect}"
+puts "Value: #{ast1.value.inspect}"
+
+visitor1 = GmailSearchSyntax::SQLiteVisitor.new
+visitor1.visit(ast1)
+sql1, params1 = visitor1.to_query.to_sql
+
+puts "\nSQL:\n#{sql1}"
+puts "\nParams: #{params1.inspect}"
+puts
+
+# Example 2: Escaped quotes in subject operator
+puts "2. Subject with escaped quotes"
+puts "-" * 40
+query2 = 'subject:"Meeting: \\"Q1 Review\\""'
+puts "Input: #{query2}"
+ast2 = GmailSearchSyntax.parse!(query2)
+puts "AST: #{ast2.inspect}"
+puts "Operator: #{ast2.name}"
+puts "Value: #{ast2.value.inspect}"
+
+visitor2 = GmailSearchSyntax::SQLiteVisitor.new
+visitor2.visit(ast2)
+sql2, params2 = visitor2.to_query.to_sql
+
+puts "\nSQL:\n#{sql2}"
+puts "\nParams: #{params2.inspect}"
+puts
+
+# Example 3: Escaped backslashes
+puts "3. Escaped backslashes"
+puts "-" * 40
+query3 = '"path\\\\to\\\\file"'
+puts "Input: #{query3}"
+ast3 = GmailSearchSyntax.parse!(query3)
+puts "AST: #{ast3.inspect}"
+puts "Value: #{ast3.value.inspect}"
+puts
+
+# Example 4: Mixed escapes
+puts "4. Mixed escapes (quotes and backslashes)"
+puts "-" * 40
+query4 = '"He said: \\"C:\\\\Users\\\\file.txt\\""'
+puts "Input: #{query4}"
+ast4 = GmailSearchSyntax.parse!(query4)
+puts "AST: #{ast4.inspect}"
+puts "Value: #{ast4.value.inspect}"
+
+visitor4 = GmailSearchSyntax::SQLiteVisitor.new
+visitor4.visit(ast4)
+sql4, params4 = visitor4.to_query.to_sql
+
+puts "\nSQL:\n#{sql4}"
+puts "\nParams: #{params4.inspect}"
+puts
+
+# Example 5: Complex query with escaped quotes
+puts "5. Complex query with escaped quotes"
+puts "-" * 40
+query5 = 'from:boss subject:"\\"Important\\" Meeting" has:attachment'
+puts "Input: #{query5}"
+ast5 = GmailSearchSyntax.parse!(query5)
+puts "AST: #{ast5.inspect}"
+
+visitor5 = GmailSearchSyntax::SQLiteVisitor.new
+visitor5.visit(ast5)
+sql5, params5 = visitor5.to_query.to_sql
+
+puts "\nSQL:\n#{sql5}"
+puts "\nParams: #{params5.inspect}"
+puts
+
+# Example 6: Escaped quotes in unquoted tokens
+puts "6. Unquoted token with escaped quote"
+puts "-" * 40
+query6 = 'meeting\\"room'
+puts "Input: #{query6}"
+ast6 = GmailSearchSyntax.parse!(query6)
+puts "AST: #{ast6.inspect}"
+puts "Value: #{ast6.value.inspect}"
+
+visitor6 = GmailSearchSyntax::SQLiteVisitor.new
+visitor6.visit(ast6)
+sql6, params6 = visitor6.to_query.to_sql
+
+puts "\nSQL:\n#{sql6}"
+puts "\nParams: #{params6.inspect}"
+puts "\nNote: Unquoted tokens use word boundary matching (not substring)"
+puts
+
+# Example 7: Escaped quotes in operator with unquoted value
+puts "7. Operator with unquoted escaped quote"
+puts "-" * 40
+query7 = 'subject:test\\"value'
+puts "Input: #{query7}"
+ast7 = GmailSearchSyntax.parse!(query7)
+puts "AST: #{ast7.inspect}"
+puts "Operator: #{ast7.name}"
+puts "Value: #{ast7.value.inspect}"
+
+visitor7 = GmailSearchSyntax::SQLiteVisitor.new
+visitor7.visit(ast7)
+sql7, params7 = visitor7.to_query.to_sql
+
+puts "\nSQL:\n#{sql7}"
+puts "\nParams: #{params7.inspect}"
+puts
+
+# Example 8: Escaped backslash in unquoted token
+puts "8. Unquoted token with escaped backslash"
+puts "-" * 40
+query8 = 'path\\\\to\\\\file'
+puts "Input: #{query8}"
+ast8 = GmailSearchSyntax.parse!(query8)
+puts "AST: #{ast8.inspect}"
+puts "Value: #{ast8.value.inspect}"
+puts
+
+puts "=" * 80
+puts "Summary"
+puts "=" * 80
+puts "Escape sequences work in BOTH quoted and unquoted tokens:"
+puts
+puts "Quoted strings (Substring nodes):"
+puts "  - Use substring matching (LIKE %value%)"
+puts "  - \"She said \\\"hello\\\"\" → 'She said \"hello\"'"
+puts
+puts "Unquoted tokens (StringToken nodes):"
+puts "  - Use word boundary matching (= or LIKE with boundaries)"
+puts "  - meeting\\\"room → 'meeting\"room'"
+puts "  - path\\\\to\\\\file → 'path\\to\\file'"
+puts
+puts "Supported escapes:"
+puts "  \\\" → literal double quote"
+puts "  \\\\ → literal backslash"
+puts "  Other (\\n, \\t, etc.) → preserved as-is"
+puts "=" * 80

data/examples/text_vs_substring_demo.rb

@@ -0,0 +1,93 @@
+#!/usr/bin/env ruby
+
+require_relative "../lib/gmail_search_syntax"
+
+puts "=" * 80
+puts "StringToken vs Substring Demo"
+puts "=" * 80
+puts
+
+# Example 1: Unquoted text (StringToken node) - word boundary matching
+puts "1. Unquoted text: meeting"
+puts "-" * 40
+query1 = "meeting"
+ast1 = GmailSearchSyntax.parse!(query1)
+puts "AST: #{ast1.inspect}"
+puts "Node type: #{ast1.class.name}"
+
+visitor1 = GmailSearchSyntax::SQLiteVisitor.new
+visitor1.visit(ast1)
+sql1, params1 = visitor1.to_query.to_sql
+
+puts "\nSQL:\n#{sql1}"
+puts "\nParams: #{params1.inspect}"
+puts "\nExplanation:"
+puts "  - Matches 'meeting' as a complete word"
+puts "  - Will match: 'meeting tomorrow', 'the meeting', 'just a meeting here'"
+puts "  - Will NOT match: 'meetings', 'premeeting', 'meetingroom'"
+puts
+
+# Example 2: Quoted text (Substring node) - substring matching
+puts "2. Quoted text: \"meeting\""
+puts "-" * 40
+query2 = '"meeting"'
+ast2 = GmailSearchSyntax.parse!(query2)
+puts "AST: #{ast2.inspect}"
+puts "Node type: #{ast2.class.name}"
+
+visitor2 = GmailSearchSyntax::SQLiteVisitor.new
+visitor2.visit(ast2)
+sql2, params2 = visitor2.to_query.to_sql
+
+puts "\nSQL:\n#{sql2}"
+puts "\nParams: #{params2.inspect}"
+puts "\nExplanation:"
+puts "  - Matches 'meeting' as a substring anywhere"
+puts "  - Will match: 'meeting', 'meetings', 'premeeting', 'meetingroom'"
+puts "  - This is useful for partial matching"
+puts
+
+# Example 3: Multi-word quoted phrase
+puts "3. Quoted phrase: \"quarterly review\""
+puts "-" * 40
+query3 = '"quarterly review"'
+ast3 = GmailSearchSyntax.parse!(query3)
+puts "AST: #{ast3.inspect}"
+puts "Node type: #{ast3.class.name}"
+
+visitor3 = GmailSearchSyntax::SQLiteVisitor.new
+visitor3.visit(ast3)
+sql3, params3 = visitor3.to_query.to_sql
+
+puts "\nSQL:\n#{sql3}"
+puts "\nParams: #{params3.inspect}"
+puts "\nExplanation:"
+puts "  - Matches 'quarterly review' as a substring"
+puts "  - Will match: 'quarterly review meeting', 'the quarterly review is done'"
+puts
+
+# Example 4: Combined usage
+puts "4. Combined: urgent \"q1 report\""
+puts "-" * 40
+query4 = 'urgent "q1 report"'
+ast4 = GmailSearchSyntax.parse!(query4)
+puts "AST: #{ast4.inspect}"
+
+visitor4 = GmailSearchSyntax::SQLiteVisitor.new
+visitor4.visit(ast4)
+sql4, params4 = visitor4.to_query.to_sql
+
+puts "\nSQL:\n#{sql4}"
+puts "\nParams: #{params4.inspect}"
+puts "\nExplanation:"
+puts "  - 'urgent' uses word boundary matching (complete word)"
+puts "  - '\"q1 report\"' uses substring matching (partial match)"
+puts "  - Both conditions must be satisfied (AND)"
+puts
+
+puts "=" * 80
+puts "Summary"
+puts "=" * 80
+puts "StringToken node (unquoted):  Word boundary matching - finds complete words"
+puts "Substring node (quoted):      Substring matching - finds partial matches"
+puts "=" * 80

data/lib/gmail_search_syntax/ast.rb

@@ -23,7 +23,7 @@ module GmailSearchSyntax
       end
     end
 
-    class Text < Node
+    class StringToken < Node
       attr_reader :value
 
       def initialize(value)
@@ -31,7 +31,19 @@ module GmailSearchSyntax
       end
 
       def inspect
-        "#<Text #{@value.inspect}>"
+        "#<StringToken #{@value.inspect}>"
+      end
+    end
+
+    class Substring < Node
+      attr_reader :value
+
+      def initialize(value)
+        @value = value
+      end
+
+      def inspect
+        "#<Substring #{@value.inspect}>"
       end
     end
 

data/lib/gmail_search_syntax/parser.rb

@@ -129,11 +129,11 @@ module GmailSearchSyntax
       when :quoted_string
         value = current_token.value
         advance
-        AST::Text.new(value)
+        AST::Substring.new(value)
       when :email, :number, :date, :relative_time
         value = current_token.value
         advance
-        AST::Text.new(value)
+        AST::StringToken.new(value)
       else
         advance
         nil
@@ -183,7 +183,7 @@ module GmailSearchSyntax
       end
 
       advance
-      AST::Text.new(word)
+      AST::StringToken.new(word)
     end
 
     def parse_operator_value

data/lib/gmail_search_syntax/sql_visitor.rb

@@ -46,8 +46,10 @@ module GmailSearchSyntax
       case node
       when AST::Operator
         visit_operator(node)
-      when AST::Text
-        visit_text(node)
+      when AST::StringToken
+        visit_string_token(node)
+      when AST::Substring
+        visit_substring(node)
       when AST::And
         visit_and(node)
       when AST::Or
@@ -325,11 +327,26 @@ module GmailSearchSyntax
       @query.add_condition("m0.rfc822_message_id = ?")
     end
 
-    def visit_text(node)
-      @query.add_param(node.value)
-      @query.add_condition("(m0.subject LIKE ? OR m0.body LIKE ?)")
+    def visit_string_token(node)
+      # Word boundary matching - the value should appear as a complete word/token
+      # We use LIKE with word boundaries: spaces, start/end of string
+      value = node.value
+      @query.add_param(value)
+      @query.add_param("#{value} %")
+      @query.add_param("% #{value}")
+      @query.add_param("% #{value} %")
+      @query.add_condition("((m0.subject = ? OR m0.subject LIKE ? OR m0.subject LIKE ? OR m0.subject LIKE ?) OR (m0.body = ? OR m0.body LIKE ? OR m0.body LIKE ? OR m0.body LIKE ?))")
+      @query.add_param(value)
+      @query.add_param("#{value} %")
+      @query.add_param("% #{value}")
+      @query.add_param("% #{value} %")
+    end
+
+    def visit_substring(node)
+      # Substring matching - the value can appear anywhere in the text
       @query.add_param("%#{node.value}%")
       @query.add_param("%#{node.value}%")
+      @query.add_condition("(m0.subject LIKE ? OR m0.body LIKE ?)")
     end
 
     def visit_and(node)

data/lib/gmail_search_syntax/tokenizer.rb

@@ -103,20 +103,37 @@ module GmailSearchSyntax
     end
 
     def read_quoted_string
-      advance
+      advance # Skip opening quote
 
       value = ""
-      while @position < @input.length && current_char != '"'
-        if current_char == "\\"
+      while @position < @input.length
+        char = current_char
+
+        if char == "\\"
+          # Handle escape sequences
           advance
-          value += current_char if @position < @input.length
+          if @position < @input.length
+            next_char = current_char
+            value += case next_char
+            when '"', "\\"
+              # Escaped quote or backslash - add the literal character
+              next_char
+            else
+              # Other escapes - keep the backslash and the character
+              "\\" + next_char
+            end
+            advance
+          end
+        elsif char == '"'
+          # Unescaped quote - end of string
+          break
         else
-          value += current_char
+          value += char
+          advance
         end
-        advance
       end
 
-      advance if @position < @input.length
+      advance if @position < @input.length && current_char == '"' # Skip closing quote
 
       add_token(:quoted_string, value)
     end
@@ -128,8 +145,26 @@ module GmailSearchSyntax
         char = current_char
         break if /[\s():{}]/.match?(char)
         break if char == "-"
-        value += char
-        advance
+
+        if char == "\\"
+          # Handle escape sequences in unquoted tokens
+          advance
+          if @position < @input.length
+            next_char = current_char
+            value += case next_char
+            when '"', "\\"
+              # Escaped quote or backslash - add the literal character
+              next_char
+            else
+              # Other escapes - keep the backslash and the character
+              "\\" + next_char
+            end
+            advance
+          end
+        else
+          value += char
+          advance
+        end
       end
 
       return if value.empty?

data/lib/gmail_search_syntax/version.rb

@@ -1,3 +1,3 @@
 module GmailSearchSyntax
-  VERSION = "0.1.0"
+  VERSION = "0.1.1"
 end

data/test/gmail_search_syntax_test.rb

@@ -4,7 +4,7 @@ class GmailSearchSyntaxTest < Minitest::Test
   include GmailSearchSyntax::AST
 
   def test_version
-    assert_equal "0.1.0", GmailSearchSyntax::VERSION
+    assert GmailSearchSyntax::VERSION
   end
 
   def test_simple_from_operator
@@ -129,11 +129,11 @@ class GmailSearchSyntaxTest < Minitest::Test
     assert_instance_of And, ast
 
     assert_equal 2, ast.operands.length
-    assert_instance_of Text, ast.operands[0]
+    assert_instance_of StringToken, ast.operands[0]
     assert_equal "dinner", ast.operands[0].value
 
     assert_instance_of Not, ast.operands[1]
-    assert_instance_of Text, ast.operands[1].child
+    assert_instance_of StringToken, ast.operands[1].child
     assert_equal "movie", ast.operands[1].child.value
   end
 
@@ -141,17 +141,17 @@ class GmailSearchSyntaxTest < Minitest::Test
     ast = GmailSearchSyntax.parse!("holiday AROUND 10 vacation")
     assert_instance_of Around, ast
 
-    assert_instance_of Text, ast.left
+    assert_instance_of StringToken, ast.left
     assert_equal "holiday", ast.left.value
     assert_equal 10, ast.distance
 
-    assert_instance_of Text, ast.right
+    assert_instance_of StringToken, ast.right
     assert_equal "vacation", ast.right.value
   end
 
   def test_around_with_quoted_string
     ast = GmailSearchSyntax.parse!('"secret AROUND 25 birthday"')
-    assert_instance_of Text, ast
+    assert_instance_of Substring, ast
     assert_equal "secret AROUND 25 birthday", ast.value
   end
 
@@ -192,7 +192,7 @@ class GmailSearchSyntaxTest < Minitest::Test
 
   def test_quoted_exact_phrase
     ast = GmailSearchSyntax.parse!('"dinner and movie tonight"')
-    assert_instance_of Text, ast
+    assert_instance_of Substring, ast
     assert_equal "dinner and movie tonight", ast.value
   end
 
@@ -203,9 +203,9 @@ class GmailSearchSyntaxTest < Minitest::Test
 
     assert_instance_of And, ast.value
     assert_equal 2, ast.value.operands.length
-    assert_instance_of Text, ast.value.operands[0]
+    assert_instance_of StringToken, ast.value.operands[0]
     assert_equal "dinner", ast.value.operands[0].value
-    assert_instance_of Text, ast.value.operands[1]
+    assert_instance_of StringToken, ast.value.operands[1]
     assert_equal "movie", ast.value.operands[1].value
   end
 
@@ -218,7 +218,7 @@ class GmailSearchSyntaxTest < Minitest::Test
     assert_equal "in", ast.operands[0].name
     assert_equal "anywhere", ast.operands[0].value
 
-    assert_instance_of Text, ast.operands[1]
+    assert_instance_of StringToken, ast.operands[1]
     assert_equal "movie", ast.operands[1].value
   end
 
@@ -327,7 +327,7 @@ class GmailSearchSyntaxTest < Minitest::Test
 
   def test_plain_text_search
     ast = GmailSearchSyntax.parse!("meeting")
-    assert_instance_of Text, ast
+    assert_instance_of StringToken, ast
     assert_equal "meeting", ast.value
   end
 
@@ -336,10 +336,10 @@ class GmailSearchSyntaxTest < Minitest::Test
     assert_instance_of And, ast
 
     assert_equal 2, ast.operands.length
-    assert_instance_of Text, ast.operands[0]
+    assert_instance_of StringToken, ast.operands[0]
     assert_equal "project", ast.operands[0].value
 
-    assert_instance_of Text, ast.operands[1]
+    assert_instance_of StringToken, ast.operands[1]
     assert_equal "report", ast.operands[1].value
   end
 
@@ -417,7 +417,7 @@ class GmailSearchSyntaxTest < Minitest::Test
 
   def test_quoted_string_with_operators_inside
     ast = GmailSearchSyntax.parse!('"from:amy to:bob"')
-    assert_instance_of Text, ast
+    assert_instance_of Substring, ast
     assert_equal "from:amy to:bob", ast.value
   end
 
@@ -460,7 +460,7 @@ class GmailSearchSyntaxTest < Minitest::Test
 
   def test_parentheses_with_single_term
     ast = GmailSearchSyntax.parse!("(meeting)")
-    assert_instance_of Text, ast
+    assert_instance_of StringToken, ast
     assert_equal "meeting", ast.value
   end
 
@@ -471,11 +471,11 @@ class GmailSearchSyntaxTest < Minitest::Test
 
     assert_instance_of And, ast.value
     assert_equal 3, ast.value.operands.length
-    assert_instance_of Text, ast.value.operands[0]
+    assert_instance_of StringToken, ast.value.operands[0]
     assert_equal "project", ast.value.operands[0].value
-    assert_instance_of Text, ast.value.operands[1]
+    assert_instance_of StringToken, ast.value.operands[1]
     assert_equal "status", ast.value.operands[1].value
-    assert_instance_of Text, ast.value.operands[2]
+    assert_instance_of StringToken, ast.value.operands[2]
     assert_equal "update", ast.value.operands[2].value
   end
 
@@ -484,10 +484,10 @@ class GmailSearchSyntaxTest < Minitest::Test
     assert_instance_of And, ast
 
     assert_equal 2, ast.operands.length
-    assert_instance_of Text, ast.operands[0]
+    assert_instance_of StringToken, ast.operands[0]
     assert_equal "meeting", ast.operands[0].value
 
-    assert_instance_of Text, ast.operands[1]
+    assert_instance_of StringToken, ast.operands[1]
     assert_equal "project", ast.operands[1].value
   end
 
@@ -505,9 +505,9 @@ class GmailSearchSyntaxTest < Minitest::Test
 
     assert_instance_of Or, ast.value
     assert_equal 2, ast.value.operands.length
-    assert_instance_of Text, ast.value.operands[0]
+    assert_instance_of StringToken, ast.value.operands[0]
     assert_equal "mischa@", ast.value.operands[0].value
-    assert_instance_of Text, ast.value.operands[1]
+    assert_instance_of StringToken, ast.value.operands[1]
     assert_equal "julik@", ast.value.operands[1].value
   end
 
@@ -518,9 +518,9 @@ class GmailSearchSyntaxTest < Minitest::Test
 
     assert_instance_of Or, ast.value
     assert_equal 2, ast.value.operands.length
-    assert_instance_of Text, ast.value.operands[0]
+    assert_instance_of StringToken, ast.value.operands[0]
     assert_equal "amy@example.com", ast.value.operands[0].value
-    assert_instance_of Text, ast.value.operands[1]
+    assert_instance_of StringToken, ast.value.operands[1]
     assert_equal "bob@example.com", ast.value.operands[1].value
   end
 
@@ -543,9 +543,9 @@ class GmailSearchSyntaxTest < Minitest::Test
 
     assert_instance_of And, ast.value
     assert_equal 2, ast.value.operands.length
-    assert_instance_of Text, ast.value.operands[0]
+    assert_instance_of StringToken, ast.value.operands[0]
     assert_equal "urgent", ast.value.operands[0].value
-    assert_instance_of Text, ast.value.operands[1]
+    assert_instance_of StringToken, ast.value.operands[1]
     assert_equal "meeting", ast.value.operands[1].value
   end
 
@@ -570,7 +570,7 @@ class GmailSearchSyntaxTest < Minitest::Test
 
     assert_instance_of And, ast.value
     assert_equal 2, ast.value.operands.length
-    assert_instance_of Text, ast.value.operands[0]
+    assert_instance_of StringToken, ast.value.operands[0]
     assert_equal "meeting", ast.value.operands[0].value
     assert_instance_of Not, ast.value.operands[1]
     assert_equal "cancelled", ast.value.operands[1].child.value
@@ -604,7 +604,7 @@ class GmailSearchSyntaxTest < Minitest::Test
     assert_equal 2, ast.value.operands[0].operands.length
     assert_equal "urgent", ast.value.operands[0].operands[0].value
     assert_equal "important", ast.value.operands[0].operands[1].value
-    assert_instance_of Text, ast.value.operands[1]
+    assert_instance_of StringToken, ast.value.operands[1]
     assert_equal "meeting", ast.value.operands[1].value
   end
 
@@ -615,9 +615,9 @@ class GmailSearchSyntaxTest < Minitest::Test
 
     assert_instance_of Or, ast.value
     assert_equal 2, ast.value.operands.length
-    assert_instance_of Text, ast.value.operands[0]
+    assert_instance_of StringToken, ast.value.operands[0]
     assert_equal "mischa@", ast.value.operands[0].value
-    assert_instance_of Text, ast.value.operands[1]
+    assert_instance_of StringToken, ast.value.operands[1]
     assert_equal "marc@", ast.value.operands[1].value
   end
 
@@ -688,4 +688,50 @@ class GmailSearchSyntaxTest < Minitest::Test
     assert_equal "subject", ast.operands[1].name
     assert_instance_of And, ast.operands[1].value
   end
+
+  def test_quoted_string_with_escaped_quotes
+    ast = GmailSearchSyntax.parse!('"She said \\"hello\\" to me"')
+    assert_instance_of Substring, ast
+    assert_equal 'She said "hello" to me', ast.value
+  end
+
+  def test_quoted_string_with_escaped_backslash
+    ast = GmailSearchSyntax.parse!('"path\\\\to\\\\file"')
+    assert_instance_of Substring, ast
+    assert_equal 'path\\to\\file', ast.value
+  end
+
+  def test_subject_with_escaped_quotes
+    ast = GmailSearchSyntax.parse!('subject:"Meeting: \\"Q1 Review\\""')
+    assert_instance_of Operator, ast
+    assert_equal "subject", ast.name
+    assert_equal 'Meeting: "Q1 Review"', ast.value
+  end
+
+  def test_unquoted_text_with_escaped_quote
+    ast = GmailSearchSyntax.parse!('meeting\\"room')
+    assert_instance_of StringToken, ast
+    assert_equal 'meeting"room', ast.value
+  end
+
+  def test_unquoted_text_with_escaped_backslash
+    ast = GmailSearchSyntax.parse!('path\\\\to\\\\file')
+    assert_instance_of StringToken, ast
+    assert_equal 'path\\to\\file', ast.value
+  end
+
+  def test_operator_with_unquoted_escaped_quote
+    ast = GmailSearchSyntax.parse!('subject:test\\"value')
+    assert_instance_of Operator, ast
+    assert_equal "subject", ast.name
+    assert_equal 'test"value', ast.value
+  end
+
+  def test_multiple_tokens_with_escapes
+    ast = GmailSearchSyntax.parse!('meeting\\"room project\\\\plan')
+    assert_instance_of And, ast
+    assert_equal 2, ast.operands.length
+    assert_equal 'meeting"room', ast.operands[0].value
+    assert_equal 'project\\plan', ast.operands[1].value
+  end
 end

data/test/sql_visitor_test.rb

@@ -244,9 +244,21 @@ class SqlVisitorTest < Minitest::Test
   def test_plain_text_search
     sql, params = parse_and_visit("meeting")
 
+    # Text nodes now use word boundary matching
+    assert_includes sql, "m0.subject = ?"
     assert_includes sql, "m0.subject LIKE ?"
+    assert_includes sql, "m0.body = ?"
     assert_includes sql, "m0.body LIKE ?"
-    assert_equal ["meeting", "%meeting%", "%meeting%"], params
+    assert_equal ["meeting", "meeting %", "% meeting", "% meeting %", "meeting", "meeting %", "% meeting", "% meeting %"], params
+  end
+
+  def test_quoted_text_search_uses_substring
+    sql, params = parse_and_visit('"meeting"')
+
+    # Quoted strings create Substring nodes which use LIKE %value%
+    assert_includes sql, "m0.subject LIKE ?"
+    assert_includes sql, "m0.body LIKE ?"
+    assert_equal ["%meeting%", "%meeting%"], params
   end
 
   def test_complex_query
@@ -343,4 +355,35 @@ class SqlVisitorTest < Minitest::Test
     join_count = sql.scan("INNER JOIN message_addresses").length
     assert_equal 2, join_count
   end
+
+  def test_quoted_string_with_escaped_quotes
+    sql, params = parse_and_visit('"She said \\"hello\\" to me"')
+
+    assert_includes sql, "m0.subject LIKE ?"
+    assert_includes sql, "m0.body LIKE ?"
+    assert_equal ['%She said "hello" to me%', '%She said "hello" to me%'], params
+  end
+
+  def test_subject_with_escaped_quotes
+    sql, params = parse_and_visit('subject:"Meeting: \\"Q1 Review\\""')
+
+    assert_includes sql, "m0.subject LIKE ?"
+    assert_equal ['%Meeting: "Q1 Review"%'], params
+  end
+
+  def test_unquoted_token_with_escaped_quote
+    sql, params = parse_and_visit('meeting\\"room')
+
+    # Unquoted tokens use word boundary matching
+    assert_includes sql, "m0.subject = ?"
+    assert_includes sql, "m0.body = ?"
+    assert_equal ['meeting"room', 'meeting"room %', '% meeting"room', '% meeting"room %', 'meeting"room', 'meeting"room %', '% meeting"room', '% meeting"room %'], params
+  end
+
+  def test_operator_with_unquoted_escaped_quote
+    sql, params = parse_and_visit('subject:test\\"value')
+
+    assert_includes sql, "m0.subject LIKE ?"
+    assert_equal ['%test"value%'], params
+  end
 end

data/test/tokenizer_test.rb

@@ -182,4 +182,58 @@ class TokenizerTest < Minitest::Test
     assert_equal "report", word_tokens[1].value
     assert_equal "meeting", word_tokens[2].value
   end
+
+  def test_tokenize_quoted_string_with_escaped_quote
+    tokens = tokenize('"She said \\"hello\\" to me"')
+    assert_equal 2, tokens.length
+    assert_equal :quoted_string, tokens[0].type
+    assert_equal 'She said "hello" to me', tokens[0].value
+    assert_equal :eof, tokens[1].type
+  end
+
+  def test_tokenize_quoted_string_with_escaped_backslash
+    tokens = tokenize('"path\\\\to\\\\file"')
+    assert_equal 2, tokens.length
+    assert_equal :quoted_string, tokens[0].type
+    assert_equal 'path\\to\\file', tokens[0].value
+  end
+
+  def test_tokenize_quoted_string_with_multiple_escapes
+    tokens = tokenize('"test \\"nested\\" and \\\\ slash"')
+    assert_equal 2, tokens.length
+    assert_equal :quoted_string, tokens[0].type
+    assert_equal 'test "nested" and \\ slash', tokens[0].value
+  end
+
+  def test_tokenize_word_with_escaped_quote
+    tokens = tokenize('meeting\\"room')
+    assert_equal 2, tokens.length
+    assert_equal :word, tokens[0].type
+    assert_equal 'meeting"room', tokens[0].value
+  end
+
+  def test_tokenize_word_with_escaped_backslash
+    tokens = tokenize('path\\\\to')
+    assert_equal 2, tokens.length
+    assert_equal :word, tokens[0].type
+    assert_equal 'path\\to', tokens[0].value
+  end
+
+  def test_tokenize_multiple_words_with_escapes
+    tokens = tokenize('meeting\\"room another\\\\word')
+    word_tokens = tokens.select { |t| t.type == :word }
+    assert_equal 2, word_tokens.length
+    assert_equal 'meeting"room', word_tokens[0].value
+    assert_equal 'another\\word', word_tokens[1].value
+  end
+
+  def test_tokenize_operator_value_with_escaped_quote
+    tokens = tokenize('subject:test\\"value')
+    assert_equal 4, tokens.length
+    assert_equal :word, tokens[0].type
+    assert_equal "subject", tokens[0].value
+    assert_equal :colon, tokens[1].type
+    assert_equal :word, tokens[2].type
+    assert_equal 'test"value', tokens[2].value
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: gmail_search_syntax
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - me@julik.nl
@@ -70,14 +70,17 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ARCHITECTURE.md
+- IMPLEMENTATION_NOTES.md
 - README.md
 - Rakefile
 - SCHEMA.md
 - examples/alias_collision_fix.rb
 - examples/demo.rb
+- examples/escaped_quotes_demo.rb
 - examples/gmail_message_id_demo.rb
 - examples/postgres_vs_sqlite.rb
 - examples/sql_query.rb
+- examples/text_vs_substring_demo.rb
 - lib/GMAIL_SEARCH_OPERATORS.md
 - lib/gmail_search_syntax.rb
 - lib/gmail_search_syntax/ast.rb