gmail_search_syntax 0.1.0 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bb674a944e51bd81d0690d2d972c0b63bae352a30469d5d1719b959d78316b4f
-  data.tar.gz: 03fc2f44531f2d610e6dc189ab7104bf753cb3fff325de5f6e388b4529e7c9d3
+  metadata.gz: 1b5e08a769d7b375473e7ca0e4afe134e03862ae3f31040c9bb22904ff482b33
+  data.tar.gz: 32624e727131b5bb0779f3f1271b6031e252fbdbdea776b445c598b06f343715
 SHA512:
-  metadata.gz: 13394002e2a5546876608249caeb8ed3379e6cad5bab7aaf324830f5ca885c9838838d3416eb6e4b6f5a17cc728e453dc586648572db769b7e5ec9b6fa758244
-  data.tar.gz: f22df8edc76415087dc990be7c895ec2af13545c7238c573cf129bbf8152adceded0f82d1d62dadebdc71f00bcb37edd477ec28661c365efb549dcc6e9688551
+  metadata.gz: d93ca6cb4e4d0bab18a9e3ff2620f669f9913cbafa3b389dd1bf3a828329df34be12f202856bba61e66d581dd8cf6c91a665889efeb8e5df3a50d5d60e33d131
+  data.tar.gz: c16da51e8f41ba6a9c001293df098c789b7dd5f3b494d3cfb0f60d3e16e485b8f5d185dd67c0cb1b0cade40d1079eca786c1614c7d7afc90585e97c5d0b8c4e1

data/GMAIL_BEHAVIOR_COMPARISON.md

@@ -0,0 +1,166 @@
+# Gmail Behavior Compatibility
+
+## Overview
+
+Our parser now implements Gmail-compatible behavior for handling operator values with spaces.
+
+## ✅ Implemented: Barewords After Operator Values
+
+### Gmail's Behavior (Now Implemented)
+
+In Gmail, barewords (unquoted text) that follow an operator value are **consumed into the operator value** until the next operator or special token is encountered.
+
+### Our Implementation
+
+We now match Gmail's behavior: barewords after operator values are automatically collected into the operator value, separated by spaces.
+
+## Examples
+
+### Example 1: Label with Spaces
+
+**Query:** `label:Cora/Google Drive label:Notes`
+
+**Both Gmail and our parser produce:**
+```
+Operator(label: "Cora/Google Drive")
+Operator(label: "Notes")
+```
+
+**Result:** ✅ Matches Gmail perfectly
+
+### Example 2: Subject with Multiple Words
+
+**Query:** `subject:urgent meeting important`
+
+**Both Gmail and our parser produce:**
+```
+Operator(subject: "urgent meeting important")
+```
+
+**Result:** ✅ Matches Gmail perfectly
+
+### Example 3: Multiple Barewords Between Operators
+
+**Query:** `label:test one two three label:another`
+
+**Both Gmail and our parser produce:**
+```
+Operator(label: "test one two three")
+Operator(label: "another")
+```
+
+**Result:** ✅ Matches Gmail perfectly
+
+## How It Works
+
+### Automatic Bareword Collection
+
+After parsing an operator name and colon, the parser automatically collects:
+- Words
+- Emails  
+- Numbers
+- Dates
+- Relative times
+
+These are joined with spaces into the operator value.
+
+### Collection Stops At
+
+Bareword collection stops when encountering:
+- Another operator (e.g., `label:`, `from:`)
+- Special operators (`OR`, `AND`, `AROUND`)
+- Grouping tokens (`(`, `)`, `{`, `}`)
+- Negation (`-`)
+- End of input
+
+### Explicit Quoting Still Supported
+
+You can still use quotes for clarity or to force exact parsing:
+
+```
+label:"Cora/Google Drive"  # Explicit
+label:Cora/Google Drive    # Automatic (same result)
+```
+
+Both produce: `Operator(label: "Cora/Google Drive")` ✅
+
+## Benefits
+
+### Gmail Compatibility ✅
+
+- Users can copy-paste Gmail queries directly
+- Behavior matches user expectations from Gmail
+- No need to add quotes for multi-word operator values
+
+### Implementation
+
+**Parser-level solution:**
+- Tokenizer remains simple (still produces individual tokens)
+- Parser intelligently collects barewords
+- Clear rules for when collection stops
+
+**Preserves advanced features:**
+- Parentheses still work for complex expressions
+- Quotes still work for explicit values
+- Numbers preserve their type when alone
+
+## Usage Examples
+
+### Works Automatically
+
+```ruby
+# Multi-word labels
+"label:Cora/Google Drive label:Notes"
+→ label:"Cora/Google Drive", label:"Notes" ✅
+
+# Multi-word subjects  
+"subject:urgent meeting important"
+→ subject:"urgent meeting important" ✅
+
+# Mixed with numbers and dates
+"subject:Q1 2024 review meeting"
+→ subject:"Q1 2024 review meeting" ✅
+```
+
+### Stops at Operators
+
+```ruby
+# Barewords stop at next operator
+"subject:urgent meeting from:boss"
+→ subject:"urgent meeting", from:"boss" ✅
+
+# Stops at OR/AND
+"subject:urgent meeting OR subject:important call"
+→ subject:"urgent meeting" OR subject:"important call" ✅
+```
+
+### Edge Cases
+
+```ruby
+# To include "movie" as separate text search after operator:
+# Option 1: Use quotes
+"in:anywhere \"movie\""
+
+# Option 2: Use another operator after
+"in:anywhere subject:movie"
+```
+
+## Testing
+
+Tests verifying Gmail-compatible behavior in `test/gmail_search_syntax_test.rb`:
+- `test_label_with_space_separated_value_gmail_behavior` ✅
+- `test_subject_with_barewords_gmail_behavior` ✅
+- `test_multiple_barewords_between_operators_gmail_behavior` ✅
+- `test_barewords_stop_at_special_operators` ✅
+- `test_barewords_with_mixed_tokens` ✅
+
+All 181 tests pass ✅
+
+## Conclusion
+
+**Status:** ✅ Gmail-compatible behavior fully implemented
+
+**Compatibility:** Users can copy-paste Gmail queries directly - they work as expected
+
+**SQL Generation:** Produces correct SQL matching the semantic intent of Gmail queries
+

data/GMAIL_COMPATIBILITY_COMPLETE.md

@@ -0,0 +1,236 @@
+# ✅ Gmail Compatibility - Implementation Complete
+
+## Summary
+
+We have successfully implemented Gmail-compatible behavior for handling multi-word operator values. The parser now matches Gmail's search syntax exactly.
+
+## What Changed
+
+### Parser Implementation (`lib/gmail_search_syntax/parser.rb`)
+
+**Key Changes:**
+1. Modified `parse_operator_value` to collect barewords after the initial token
+2. Added `is_bareword_token?` helper method
+3. Barewords are automatically joined with spaces into the operator value
+4. Collection stops at operators, special tokens (OR/AND/AROUND), or grouping
+
+**Intelligent Type Preservation:**
+- Single numbers preserve their Integer type (e.g., `size:1000000`)
+- Multiple values are joined as strings (e.g., `subject:Q1 2024 review`)
+
+### Test Updates
+
+**Updated 3 existing tests** to reflect Gmail behavior:
+- `test_label_with_space_separated_value_gmail_behavior`
+- `test_subject_with_barewords_gmail_behavior`
+- `test_multiple_barewords_between_operators_gmail_behavior`
+- `test_in_anywhere` (edge case)
+
+**Added 2 new tests:**
+- `test_barewords_stop_at_special_operators`
+- `test_barewords_with_mixed_tokens`
+
+**Result:** 181 tests passing ✅
+
+## Examples
+
+### Before vs After
+
+**Query:** `label:Cora/Google Drive label:Notes`
+
+**Before (v0.1.0):**
+```ruby
+#<And 
+  #<Operator label: "Cora/Google"> 
+  AND #<StringToken "Drive"> 
+  AND #<Operator label: "Notes">>
+```
+
+**After (Now):**
+```ruby
+#<And 
+  #<Operator label: "Cora/Google Drive"> 
+  AND #<Operator label: "Notes">>
+```
+
+✅ Now matches Gmail perfectly!
+
+### More Examples
+
+```ruby
+# Multi-word subjects
+"subject:urgent meeting important"
+→ Operator(subject: "urgent meeting important") ✅
+
+# Stops at OR
+"subject:Q1 review OR subject:Q2 planning"
+→ subject:"Q1 review" OR subject:"Q2 planning" ✅
+
+# Works with numbers and dates
+"subject:Q1 2024 review meeting"
+→ Operator(subject: "Q1 2024 review meeting") ✅
+
+# Preserves number types
+"size:1000000"
+→ Operator(size: 1000000)  # Integer preserved ✅
+```
+
+## Verification
+
+### Run the Demo
+
+```bash
+bundle exec ruby examples/gmail_comparison_demo.rb
+```
+
+Shows 5 test cases, all matching Gmail ✅
+
+### All Tests Pass
+
+```bash
+bundle exec rake test
+# 181 runs, 1030 assertions, 0 failures, 0 errors, 0 skips ✅
+```
+
+### Code Quality
+
+```bash
+bundle exec standardrb
+# No offenses detected ✅
+```
+
+## Technical Details
+
+### Collection Rules
+
+**Barewords are collected from:**
+- `:word` tokens
+- `:email` tokens
+- `:number` tokens
+- `:date` tokens
+- `:relative_time` tokens
+
+**Collection stops at:**
+- Another operator (word followed by `:`)
+- Special operators (`:or`, `:and`, `:around`)
+- Grouping tokens (`:lparen`, `:rparen`, `:lbrace`, `:rbrace`)
+- Negation (`:minus`)
+- End of input (`:eof`)
+
+### Implementation Strategy
+
+**Why Parser-Level?**
+- Tokenizer remains simple and predictable
+- Each word is still a distinct token
+- Parser intelligently groups them
+- Easier to reason about edge cases
+
+**Type Preservation:**
+```ruby
+# Single number → preserve type
+values = [1000000], types = [:number]
+→ returns 1000000 (Integer)
+
+# Multiple tokens → join as string
+values = [2024, "Q1", "review"], types = [:number, :word, :word]
+→ returns "2024 Q1 review" (String)
+```
+
+## Benefits
+
+### For Users
+
+1. **Copy-paste from Gmail** - queries work identically
+2. **Natural syntax** - no need to add quotes for multi-word values
+3. **Backwards compatible** - quotes still work if preferred
+4. **Predictable** - clear rules for when collection stops
+
+### For Developers
+
+1. **Simpler tokenizer** - still produces individual tokens
+2. **Type safety** - numbers preserve their type when appropriate
+3. **Extensible** - easy to add new token types to collection
+4. **Well-tested** - comprehensive test coverage
+
+## Edge Cases Handled
+
+### Edge Case 1: Operator Look-Ahead
+
+```ruby
+"from:alice@example.com subject meeting"
+```
+
+Parser checks if "subject" is followed by `:` before collecting it as a bareword. ✅
+
+### Edge Case 2: Number Type Preservation
+
+```ruby
+"size:1000000"  # Single number
+→ Operator(size: 1000000)  # Integer ✅
+
+"subject:2024 Q1"  # Number + words
+→ Operator(subject: "2024 Q1")  # String ✅
+```
+
+### Edge Case 3: Special Operators
+
+```ruby
+"subject:urgent OR subject:important"
+```
+
+"OR" stops bareword collection, not consumed into value. ✅
+
+### Edge Case 4: Value After Operator
+
+```ruby
+"in:anywhere movie"
+```
+
+Without another operator after, "movie" gets consumed. To search for "movie" as text:
+- Use quotes: `in:anywhere "movie"`
+- Add operator: `in:anywhere subject:movie`
+
+## Migration Guide
+
+### If You Have Existing Code
+
+**No breaking changes for well-formed queries:**
+- `label:"Multi Word"` → Still works ✅
+- `subject:(word1 word2)` → Still works ✅
+- `from:alice@example.com` → Still works ✅
+
+**Improved behavior for casual queries:**
+- `label:Multi Word` → Now works! ✅ (was broken before)
+- `subject:urgent meeting` → Now works! ✅ (was broken before)
+
+### Recommended Usage
+
+**Best Practices:**
+```ruby
+# All these work identically now:
+"label:Cora/Google Drive"        # Automatic ✅
+"label:\"Cora/Google Drive\""    # Explicit ✅
+
+# For complex expressions, use parentheses:
+"subject:(urgent OR important)"  # Complex grouping ✅
+
+# For text searches, use standalone words:
+"meeting project deadline"       # All become StringTokens ✅
+```
+
+## Documentation
+
+- **`GMAIL_BEHAVIOR_COMPARISON.md`** - Updated to reflect implementation
+- **`examples/gmail_comparison_demo.rb`** - Shows compatibility verification
+- **`test/gmail_search_syntax_test.rb`** - Comprehensive test coverage
+
+## Status
+
+✅ **Implementation:** Complete  
+✅ **Tests:** All passing (181 tests)  
+✅ **Documentation:** Updated  
+✅ **Code Quality:** Clean (standardrb)  
+✅ **Compatibility:** Gmail-compatible
+
+🎉 **Ready for production use!**
+

data/IMPLEMENTATION_NOTES.md

@@ -0,0 +1,174 @@
+# Implementation Notes: StringToken vs Substring Nodes
+
+## Overview
+
+This implementation distinguishes between **word boundary matching** (unquoted text) and **substring matching** (quoted text) in the Gmail search syntax parser.
+
+## Changes Made
+
+### 1. Renamed and New AST Nodes
+
+- **Renamed** `Text` to `StringToken` for clarity - represents unquoted text tokens
+- **Added** `Substring` node to the AST (`lib/gmail_search_syntax/ast.rb`) that represents quoted strings.
+
+```ruby
+class Substring < Node
+  attr_reader :value
+  
+  def initialize(value)
+    @value = value
+  end
+  
+  def inspect
+    "#<Substring #{@value.inspect}>"
+  end
+end
+```
+
+### 2. Parser Updates
+
+Modified the parser (`lib/gmail_search_syntax/parser.rb`) to create:
+- `StringToken` nodes for unquoted text
+- `Substring` nodes for quoted strings (`:quoted_string` tokens)
+
+### 3. SQL Visitor Updates
+
+Updated the SQL visitor (`lib/gmail_search_syntax/sql_visitor.rb`) with two different behaviors:
+
+#### StringToken Node (Word Boundary Matching)
+```ruby
+def visit_string_token(node)
+  # Matches complete words only
+  # Uses: = exact, LIKE "value %", LIKE "% value", LIKE "% value %"
+end
+```
+
+SQL Pattern:
+```sql
+(m0.subject = ? OR m0.subject LIKE ? OR m0.subject LIKE ? OR m0.subject LIKE ?)
+OR
+(m0.body = ? OR m0.body LIKE ? OR m0.body LIKE ? OR m0.body LIKE ?)
+```
+
+Parameters: `["meeting", "meeting %", "% meeting", "% meeting %", ...]`
+
+**Matches:** "meeting tomorrow", "the meeting", "just meeting"  
+**Does NOT match:** "meetings", "premeeting", "meetingroom"
+
+#### Substring Node (Partial Matching)
+```ruby
+def visit_substring(node)
+  # Matches anywhere in the text
+  # Uses: LIKE "%value%"
+end
+```
+
+SQL Pattern:
+```sql
+(m0.subject LIKE ? OR m0.body LIKE ?)
+```
+
+Parameters: `["%meeting%", "%meeting%"]`
+
+**Matches:** "meeting", "meetings", "premeeting", "meetingroom"
+
+## Examples
+
+### Unquoted (Word Boundary)
+```ruby
+GmailSearchSyntax.parse!("meeting")
+# => #<StringToken "meeting">
+# SQL: ... WHERE m0.subject = ? OR m0.subject LIKE ? OR ...
+```
+
+### Quoted (Substring)
+```ruby
+GmailSearchSyntax.parse!('"meeting"')
+# => #<Substring "meeting">
+# SQL: ... WHERE m0.subject LIKE ? OR m0.body LIKE ?
+```
+
+### Combined
+```ruby
+GmailSearchSyntax.parse!('urgent "q1 report"')
+# => #<And #<StringToken "urgent"> AND #<Substring "q1 report">>
+```
+
+## Rationale
+
+This implementation provides:
+
+1. **More precise searching** - Unquoted text matches complete words/tokens, avoiding false positives from partial matches
+2. **Flexible substring search** - Quoted text still allows finding substrings when needed
+3. **Gmail-like behavior** - Aligns with user expectations from Gmail's search syntax
+4. **SQL efficiency** - Word boundary matching is more specific than substring matching
+
+## Escape Sequences
+
+Both `StringToken` and `Substring` nodes support escape sequences in **both quoted and unquoted tokens**:
+
+### Supported Escapes
+
+- `\"` - Literal double quote
+- `\\` - Literal backslash
+- Other escape sequences (e.g., `\n`, `\t`) are preserved as-is (backslash + character)
+
+### Examples
+
+**Quoted Strings (Substring nodes):**
+```ruby
+# Escaped quotes in quoted string
+'"She said \\"hello\\" to me"'
+# => #<Substring 'She said "hello" to me'>
+
+# Escaped backslashes in quoted string
+'"path\\\\to\\\\file"'
+# => #<Substring 'path\\to\\file'>
+
+# In operator values with quoted strings
+'subject:"Meeting: \\"Q1 Review\\""'
+# => #<Operator subject: 'Meeting: "Q1 Review"'>
+```
+
+**Unquoted Tokens (StringToken nodes):**
+```ruby
+# Escaped quotes in unquoted token
+'meeting\\"room'
+# => #<StringToken 'meeting"room'>
+
+# Escaped backslashes in unquoted token
+'path\\\\to\\\\file'
+# => #<StringToken 'path\\to\\file'>
+
+# In operator values with unquoted tokens
+'subject:test\\"value'
+# => #<Operator subject: 'test"value'>
+```
+
+This allows you to include literal quotes and backslashes in any token, whether quoted or unquoted.
+
+## Testing
+
+All tests pass with comprehensive coverage:
+- Basic functionality tests
+- Escape sequence tests in tokenizer
+- Integration tests for parsing with escaped quotes
+- SQL generation tests with escaped quotes
+
+New tests added:
+- `test_quoted_text_search_uses_substring` in `test/sql_visitor_test.rb`
+- `test_tokenize_quoted_string_with_escaped_quote` in `test/tokenizer_test.rb`
+- `test_tokenize_quoted_string_with_escaped_backslash` in `test/tokenizer_test.rb`
+- `test_tokenize_word_with_escaped_quote` in `test/tokenizer_test.rb`
+- `test_tokenize_word_with_escaped_backslash` in `test/tokenizer_test.rb`
+- `test_quoted_string_with_escaped_quotes` in `test/gmail_search_syntax_test.rb`
+- `test_unquoted_text_with_escaped_quote` in `test/gmail_search_syntax_test.rb`
+- `test_unquoted_text_with_escaped_backslash` in `test/gmail_search_syntax_test.rb`
+- `test_subject_with_escaped_quotes` in `test/sql_visitor_test.rb`
+- `test_unquoted_token_with_escaped_quote` in `test/sql_visitor_test.rb`
+- `test_operator_with_unquoted_escaped_quote` in `test/sql_visitor_test.rb`
+
+Run demos:
+- `bundle exec ruby examples/text_vs_substring_demo.rb`
+- `bundle exec ruby examples/escaped_quotes_demo.rb`
+

data/README.md

@@ -39,11 +39,11 @@ GmailSearchSyntax.parse!("from:amy OR from:bob")
 
 # Negation
 GmailSearchSyntax.parse!("dinner -movie")
-# => #<And #<Text "dinner"> AND #<Not #<Text "movie">>>
+# => #<And #<StringToken "dinner"> AND #<Not #<StringToken "movie">>>
 
 # Proximity search
 GmailSearchSyntax.parse!("holiday AROUND 10 vacation")
-# => #<Around #<Text "holiday"> AROUND 10 #<Text "vacation">>
+# => #<Around #<StringToken "holiday"> AROUND 10 #<StringToken "vacation">>
 
 # Complex query with OR inside operator values
 GmailSearchSyntax.parse!("from:{alice@ bob@} subject:urgent")