gmail_search_syntax 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: bb674a944e51bd81d0690d2d972c0b63bae352a30469d5d1719b959d78316b4f
+  data.tar.gz: 03fc2f44531f2d610e6dc189ab7104bf753cb3fff325de5f6e388b4529e7c9d3
+SHA512:
+  metadata.gz: 13394002e2a5546876608249caeb8ed3379e6cad5bab7aaf324830f5ca885c9838838d3416eb6e4b6f5a17cc728e453dc586648572db769b7e5ec9b6fa758244
+  data.tar.gz: f22df8edc76415087dc990be7c895ec2af13545c7238c573cf129bbf8152adceded0f82d1d62dadebdc71f00bcb37edd477ec28661c365efb549dcc6e9688551

data/ARCHITECTURE.md

@@ -0,0 +1,338 @@
+# Gmail Search Syntax Parser Architecture
+
+A rigorous parser for Gmail's search syntax as documented at [Gmail Help](https://support.google.com/mail/answer/7190).
+
+## Architecture Overview
+
+The parser is built in three stages:
+
+1. **Tokenization** - Breaking the input string into tokens
+2. **Parsing** - Building an Abstract Syntax Tree (AST) from tokens
+3. **AST** - Minimal tree structure representing the search query
+
+## Tokenization
+
+The `Tokenizer` class scans the input character by character and produces a stream of tokens.
+
+### Token Types
+
+- **Keywords**: `word` - Operator names (from, to, subject, etc.)
+- **Punctuation**: `colon`, `lparen`, `rparen`, `lbrace`, `rbrace`, `minus`, `plus`
+- **Logical Operators**: `or`, `and`, `around`
+- **Values**: `email`, `number`, `date`, `relative_time`, `quoted_string`
+- **End**: `eof`
+
+### Key Features
+
+- Recognizes email addresses (contains `@`)
+- Parses quoted strings with escape sequences
+- Identifies dates (`YYYY/MM/DD` or `MM/DD/YYYY`)
+- Recognizes relative times (`1y`, `2d`, `3m`)
+- Handles logical operators (`OR`, `AND`, `AROUND`)
+- Properly tokenizes negation (`-`)
+
+### Example
+
+```ruby
+Input:  'from:amy@example.com subject:"meeting notes"'
+Tokens: [word("from"), colon, email("amy@example.com"), 
+         word("subject"), colon, quoted_string("meeting notes"), eof]
+```
+
+## Parsing
+
+The `Parser` class implements a recursive descent parser that builds an AST from the token stream.
+
+### Operator Precedence (highest to lowest)
+
+1. Unary operators (`-`, `+`)
+2. Primary expressions (operators, text, grouping)
+3. `AROUND` (proximity search)
+4. Implicit `AND` (adjacency)
+5. Explicit `AND`
+6. `OR`
+
+### Grammar
+
+```
+expression       → or_expression
+or_expression    → and_expression ( "OR" and_expression )*
+and_expression   → around_expr ( around_expr )* [ "AND" around_expr ]*
+around_expr      → unary_expr [ "AROUND" NUMBER? unary_expr ]
+unary_expr       → "-" primary | "+" primary | primary
+primary          → "(" expression ")" 
+                 | "{" or_list "}"
+                 | operator ":" value
+                 | quoted_string
+                 | word
+```
+
+### Key Features
+
+- **Implicit AND**: Adjacent terms are combined with AND
+- **Braces as OR**: `{a b c}` is equivalent to `a OR b OR c`
+- **Negation**: `-term` creates a NOT node
+- **Grouping**: Parentheses override precedence
+- **Operator values**: Can be words, emails, numbers, dates, or even grouped expressions
+
+## AST Structure
+
+The AST is a minimal tree representation with the following node types:
+
+### Node Types
+
+#### `Operator`
+Represents a search operator (from, to, subject, etc.)
+```ruby
+name: String        # "from", "to", "subject", etc.
+value: String|Node  # Value or nested expression
+```
+
+#### `Text`
+Plain text search term.
+```ruby
+value: String
+```
+
+#### `And`
+Logical AND combination with 2 or more operands.
+```ruby
+operands: [Node]  # Array of 2+ nodes
+```
+
+#### `Or`
+Logical OR combination with 2 or more operands.
+```ruby
+operands: [Node]  # Array of 2+ nodes
+```
+
+#### `Not`
+Negation (exclusion).
+```ruby
+child: Node
+```
+
+#### `Group`
+Parenthesized grouping.
+```ruby
+children: [Node]
+```
+
+#### `Around`
+Proximity search.
+```ruby
+left: Node
+distance: Integer  # Default: 5
+right: Node
+```
+
+## Supported Operators
+
+Based on Gmail's official documentation:
+
+### Email Routing
+- `from:`, `to:`, `cc:`, `bcc:`, `deliveredto:`
+
+### Metadata
+- `subject:`, `label:`, `category:`, `list:`
+
+### Dates & Times
+- `after:`, `before:`, `older:`, `newer:`, `older_than:`, `newer_than:`
+
+### Attachments
+- `has:`, `filename:`
+
+### Status & Location
+- `is:`, `in:`
+
+### Size
+- `size:`, `larger:`, `smaller:`
+
+### Advanced
+- `rfc822msgid:`
+
+## Examples
+
+### Simple Query
+```ruby
+Input: "from:amy@example.com"
+AST:   Operator("from", "amy@example.com")
+```
+
+### Logical OR
+```ruby
+Input: "from:amy OR from:bob"
+AST:   Or([
+         Operator("from", "amy"),
+         Operator("from", "bob")
+       ])
+```
+
+### Multiple OR
+```ruby
+Input: "{from:a from:b from:c}"
+AST:   Or([
+         Operator("from", "a"),
+         Operator("from", "b"),
+         Operator("from", "c")
+       ])
+```
+
+### Implicit AND
+```ruby
+Input: "subject:meeting has:attachment"
+AST:   And([
+         Operator("subject", "meeting"),
+         Operator("has", "attachment")
+       ])
+```
+
+### Multiple AND
+```ruby
+Input: "from:boss subject:urgent has:attachment"
+AST:   And([
+         Operator("from", "boss"),
+         Operator("subject", "urgent"),
+         Operator("has", "attachment")
+       ])
+```
+
+### Negation
+```ruby
+Input: "dinner -movie"
+AST:   And([
+         Text("dinner"),
+         Not(Text("movie"))
+       ])
+```
+
+### Proximity Search
+```ruby
+Input: "holiday AROUND 10 vacation"
+AST:   Around(
+         Text("holiday"),
+         10,
+         Text("vacation")
+       )
+```
+
+### Complex Query
+```ruby
+Input: "(from:boss OR from:manager) subject:urgent -label:spam"
+AST:   And([
+         Or([
+           Operator("from", "boss"),
+           Operator("from", "manager")
+         ]),
+         Operator("subject", "urgent"),
+         Not(Operator("label", "spam"))
+       ])
+```
+
+### Grouped Operator Values
+```ruby
+Input: "subject:(dinner movie)"
+AST:   Operator("subject",
+         And([
+           Text("dinner"),
+           Text("movie")
+         ])
+       )
+```
+
+### OR/AND Inside Operator Values
+
+**Using Parentheses with OR:**
+```ruby
+Input: "from:(mischa@ OR julik@)"
+AST:   Operator("from",
+         Or([
+           Text("mischa@"),
+           Text("julik@")
+         ])
+       )
+```
+
+**Using Curly Braces (implicit OR):**
+```ruby
+Input: "from:{mischa@ marc@}"
+AST:   Operator("from",
+         Or([
+           Text("mischa@"),
+           Text("marc@")
+         ])
+       )
+```
+
+**Combined with Other Conditions:**
+```ruby
+Input: "from:(alice@ OR bob@) subject:meeting"
+AST:   And([
+         Operator("from",
+           Or([
+             Text("alice@"),
+             Text("bob@")
+           ])
+         ),
+         Operator("subject", "meeting")
+       ])
+```
+
+## Testing
+
+The parser includes comprehensive test coverage:
+
+- **84 tests** across two test suites
+- **460 assertions** verifying behavior
+- Tests for all operators from Gmail documentation
+- Edge case handling (empty queries, nested groups, etc.)
+- Separate tokenizer tests with strict order verification
+- Tests for complex expressions inside operator values:
+  - OR/AND with parentheses
+  - OR with curly braces (implicit OR)
+  - Mixed parentheses and curly braces
+
+Run tests:
+```bash
+bundle exec rake test
+```
+
+## Usage
+
+```ruby
+require 'gmail_search_syntax'
+
+ast = GmailSearchSyntax.parse!("from:manager subject:meeting")
+# => #<And #<Operator from: "manager"> AND #<Operator subject: "meeting">>
+
+# Access AST nodes
+ast.operands[0].name   # => "from"
+ast.operands[0].value  # => "manager"
+ast.operands[1].name   # => "subject"
+ast.operands[1].value  # => "meeting"
+
+# Empty queries raise an error
+GmailSearchSyntax.parse!("")
+# => raises GmailSearchSyntax::EmptyQueryError: "Query cannot be empty"
+```
+
+## Design Decisions
+
+1. **Minimal AST**: No redundant nodes; single-child nodes are collapsed
+2. **Multi-operand AND/OR**: Support 2+ operands instead of binary left/right structure
+3. **Strict tokenization**: Emails, dates, and numbers are recognized at tokenization
+4. **Operator precedence**: Matches Gmail's actual behavior
+5. **Implicit AND**: Adjacent terms combine naturally
+6. **Value flexibility**: Operator values can be expressions (for grouping)
+7. **Fail-fast on empty**: `parse!` raises `EmptyQueryError` for empty/whitespace-only input
+
+## Extensions
+
+The parser is designed to be extended:
+
+1. Add semantic validation (valid operator names, date formats)
+2. Convert AST to other query formats (SQL, Elasticsearch)
+3. Add query optimization (flatten nested ANDs/ORs)
+4. Pretty-print queries
+5. Query builder API
+

data/README.md

@@ -0,0 +1,129 @@
+# gmail_search_syntax
+
+Parser for Gmail's search syntax. Converts Gmail search queries into an Abstract Syntax Tree.
+
+Based on the official Gmail search operators documentation:  
+https://support.google.com/mail/answer/7190
+
+> [!TIP]
+> This gem was created for [Cora,](https://cora.computer/) your personal e-mail assistant. 
+> Send them some love for allowing me to share it.
+
+## Installation
+
+```ruby
+gem 'gmail_search_syntax'
+```
+
+## Usage
+
+```ruby
+require 'gmail_search_syntax'
+
+ast = GmailSearchSyntax.parse!("from:boss subject:meeting")
+# => #<And #<Operator from: "boss"> AND #<Operator subject: "meeting">>
+```
+
+Afterwards you can do all sorts of interesting things with this, for example - transform your AST nodes into Elastic or SQL queries, or execute them bottom-op just from arrays in memory.
+
+### Examples
+
+```ruby
+# Simple operator
+GmailSearchSyntax.parse!("from:amy@example.com")
+# => #<Operator from: "amy@example.com">
+
+# Logical OR
+GmailSearchSyntax.parse!("from:amy OR from:bob")
+# => #<Or #<Operator from: "amy"> OR #<Operator from: "bob">>
+
+# Negation
+GmailSearchSyntax.parse!("dinner -movie")
+# => #<And #<Text "dinner"> AND #<Not #<Text "movie">>>
+
+# Proximity search
+GmailSearchSyntax.parse!("holiday AROUND 10 vacation")
+# => #<Around #<Text "holiday"> AROUND 10 #<Text "vacation">>
+
+# Complex query with OR inside operator values
+GmailSearchSyntax.parse!("from:{alice@ bob@} subject:urgent")
+# => #<And #<Operator from: #<Or ...>> AND #<Operator subject: "urgent">>
+
+# Empty queries raise an error
+GmailSearchSyntax.parse!("")
+# => raises GmailSearchSyntax::EmptyQueryError
+```
+
+### Converting to SQL
+
+The gem includes a SQLite visitor that can convert Gmail queries to SQL. Here's a complex example:
+
+```ruby
+require 'gmail_search_syntax'
+
+# A complex Gmail query with multiple operators
+query = '(from:manager OR from:boss) subject:"quarterly review" has:attachment -label:archived after:2024/01/01 larger:5M'
+
+ast = GmailSearchSyntax.parse!(query)
+visitor = GmailSearchSyntax::SQLiteVisitor.new(current_user_email: "user@example.com")
+visitor.visit(ast)
+
+sql, params = visitor.to_query.to_sql
+```
+
+This generates the following SQL:
+
+```sql
+SELECT DISTINCT m0.id 
+FROM messages AS m0 
+INNER JOIN message_addresses AS ma1 ON m0.id = ma1.message_id 
+INNER JOIN message_addresses AS ma3 ON m0.id = ma3.message_id 
+INNER JOIN message_labels AS ml ON m0.id = ml.message_id 
+INNER JOIN labels AS l ON ml.label_id = l.id 
+WHERE ((((ma1.address_type = ? OR ma1.address_type = ? OR ma1.address_type = ?) 
+         AND ma1.email_address = ?) 
+        OR ((ma3.address_type = ? OR ma3.address_type = ? OR ma3.address_type = ?) 
+            AND ma3.email_address = ?)) 
+       AND m0.subject LIKE ? 
+       AND m0.has_attachment = 1 
+       AND NOT l.name = ? 
+       AND m0.internal_date > ? 
+       AND m0.size_bytes > ?)
+```
+
+With parameters: `["from", "cc", "bcc", "manager", "from", "cc", "bcc", "boss", "%quarterly review%", "archived", "2024-01-01", 5242880]`
+
+A similar visitor is provided for PostgreSQL.
+
+## Supported Operators
+
+Email routing: `from:`, `to:`, `cc:`, `bcc:`, `deliveredto:`  
+Metadata: `subject:`, `label:`, `category:`, `list:`  
+Dates: `after:`, `before:`, `older:`, `newer:`, `older_than:`, `newer_than:`  
+Attachments: `has:`, `filename:`  
+Status: `is:`, `in:`  
+Size: `size:`, `larger:`, `smaller:`
+
+## Features
+
+- Handles operator precedence (negation, AROUND, implicit AND, explicit AND, OR)
+- Supports grouping with parentheses and braces
+- Recognizes emails, dates, quoted strings, and numbers
+- Minimal AST structure
+
+There is also a converter from the operators to SQL queries against an embedded SQLite database. This is meant more as an example than a fully-featured store, but it shows what's possible.
+
+## Testing
+
+```bash
+bundle exec rake test
+```
+
+## License
+
+MIT
+
+## Legal Notes
+
+Gmail is a trademark of Google LLC.
+

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+require "standard/rake"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task default: [:test, :standard]

data/SCHEMA.md

@@ -0,0 +1,223 @@
+# Database Schema for Gmail Search
+
+## Overview
+
+This document describes the SQLite database schema designed to support Gmail search syntax queries. The schema is optimized for the search operators defined in `lib/GMAIL_SEARCH_OPERATORS.md`.
+
+## Core Tables
+
+### messages
+Primary table storing email message metadata.
+
+```sql
+CREATE TABLE messages (
+    id TEXT PRIMARY KEY,
+    rfc822_message_id TEXT,
+    subject TEXT,
+    body TEXT,
+    internal_date DATETIME,
+    size_bytes INTEGER,
+    
+    is_important BOOLEAN DEFAULT 0,
+    is_starred BOOLEAN DEFAULT 0,
+    is_unread BOOLEAN DEFAULT 0,
+    is_read BOOLEAN DEFAULT 0,
+    is_muted BOOLEAN DEFAULT 0,
+    
+    in_inbox BOOLEAN DEFAULT 1,
+    in_archive BOOLEAN DEFAULT 0,
+    in_snoozed BOOLEAN DEFAULT 0,
+    in_spam BOOLEAN DEFAULT 0,
+    in_trash BOOLEAN DEFAULT 0,
+    
+    has_attachment BOOLEAN DEFAULT 0,
+    has_youtube BOOLEAN DEFAULT 0,
+    has_drive BOOLEAN DEFAULT 0,
+    has_document BOOLEAN DEFAULT 0,
+    has_spreadsheet BOOLEAN DEFAULT 0,
+    has_presentation BOOLEAN DEFAULT 0,
+    
+    has_yellow_star BOOLEAN DEFAULT 0,
+    has_orange_star BOOLEAN DEFAULT 0,
+    has_red_star BOOLEAN DEFAULT 0,
+    has_purple_star BOOLEAN DEFAULT 0,
+    has_blue_star BOOLEAN DEFAULT 0,
+    has_green_star BOOLEAN DEFAULT 0,
+    has_red_bang BOOLEAN DEFAULT 0,
+    has_orange_guillemet BOOLEAN DEFAULT 0,
+    has_yellow_bang BOOLEAN DEFAULT 0,
+    has_green_check BOOLEAN DEFAULT 0,
+    has_blue_info BOOLEAN DEFAULT 0,
+    has_purple_question BOOLEAN DEFAULT 0,
+    
+    category TEXT,
+    mailing_list TEXT,
+    
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### message_addresses
+Stores email addresses associated with messages (from, to, cc, bcc, delivered_to).
+
+```sql
+CREATE TABLE message_addresses (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    message_id TEXT NOT NULL,
+    address_type TEXT NOT NULL,
+    email_address TEXT NOT NULL,
+    display_name TEXT,
+    
+    FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE CASCADE
+);
+```
+
+**Note:** The `from:` and `to:` operators search across `from`, `cc`, and `bcc` address types per Gmail specification.
+
+### labels
+Label definitions with external string IDs.
+
+```sql
+CREATE TABLE labels (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    is_system_label BOOLEAN DEFAULT 0,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### message_labels
+Many-to-many relationship between messages and labels.
+
+```sql
+CREATE TABLE message_labels (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    message_id TEXT NOT NULL,
+    label_id TEXT NOT NULL,
+    
+    FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE CASCADE,
+    FOREIGN KEY (label_id) REFERENCES labels(id) ON DELETE CASCADE,
+    UNIQUE(message_id, label_id)
+);
+```
+
+### attachments
+File attachments associated with messages.
+
+```sql
+CREATE TABLE attachments (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    message_id TEXT NOT NULL,
+    filename TEXT NOT NULL,
+    content_type TEXT,
+    size_bytes INTEGER,
+    
+    FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE CASCADE
+);
+```
+
+## String Matching Requirements
+
+### Prefix/Suffix Matching
+Required for:
+- **Email addresses** (from:, to:, cc:, bcc:, deliveredto:)
+  - `from:marc@` → prefix match → `WHERE email_address LIKE 'marc@%'`
+  - `from:@example.com` → suffix match → `WHERE email_address LIKE '%@example.com'`
+  - `from:marc@example.com` → exact match → `WHERE email_address = 'marc@example.com'`
+
+- **Mailing lists** (list:)
+  - Same pattern as email addresses
+
+- **Filenames** (filename:)
+  - `filename:pdf` → extension match → `WHERE filename LIKE '%.pdf'`
+  - `filename:homework` → prefix match → `WHERE filename LIKE 'homework%'`
+
+### Exact Match Only
+- RFC822 message IDs
+- Boolean/enum fields (is:, has:, in:, category:, label:)
+
+## SQL Visitor Usage
+
+The library provides two SQL visitor implementations for different database backends:
+
+### SQLiteVisitor
+
+Converts parsed AST into SQLite-compatible SQL queries:
+
+```ruby
+ast = GmailSearchSyntax.parse!("from:amy@example.com newer_than:7d")
+visitor = GmailSearchSyntax::SQLiteVisitor.new(current_user_email: "me@example.com")
+visitor.visit(ast)
+
+sql, params = visitor.to_query.to_sql
+# sql: "SELECT DISTINCT m.id FROM messages m ... WHERE ... m.internal_date > datetime('now', ?)"
+# params: ["from", "cc", "bcc", "amy@example.com", "-7 days"]
+```
+
+### PostgresVisitor
+
+Converts parsed AST into PostgreSQL-compatible SQL queries:
+
+```ruby
+ast = GmailSearchSyntax.parse!("from:amy@example.com newer_than:7d")
+visitor = GmailSearchSyntax::PostgresVisitor.new(current_user_email: "me@example.com")
+visitor.visit(ast)
+
+sql, params = visitor.to_query.to_sql
+# sql: "SELECT DISTINCT m.id FROM messages m ... WHERE ... m.internal_date > (NOW() - ?::interval)"
+# params: ["from", "cc", "bcc", "amy@example.com", "7 days"]
+```
+
+**Note**: `SqlVisitor` is an alias for `SQLiteVisitor` for backward compatibility.
+
+### Database-Specific Differences
+
+The main difference between the visitors is in relative date handling:
+
+| Feature | SQLite | PostgreSQL |
+|---------|--------|------------|
+| `older_than:7d` | `datetime('now', '-7 days')` | `NOW() - '7 days'::interval` |
+| `newer_than:3m` | `datetime('now', '-3 months')` | `NOW() - '3 months'::interval` |
+| Parameter format | `"-7 days"` (negative) | `"7 days"` (positive with cast) |
+
+All other query generation is identical between the two visitors.
+
+### Features
+
+- **Parameterized queries**: All user input is bound via `?` placeholders
+- **Automatic table joins**: Joins required tables based on operators
+- **Nested conditions**: Properly handles AND/OR/NOT with parentheses
+- **Special operators**:
+  - `from:me` / `to:me` → uses `current_user_email`
+  - `in:anywhere` → no location filter
+  - `AROUND` → generates `(1 = 0)` no-op condition
+- **Date handling**:
+  - Converts dates from `YYYY/MM/DD` to `YYYY-MM-DD`
+  - Parses relative times (`1y`, `2d`, `3m`) to database-specific datetime functions
+- **Size parsing**: Converts `10M`, `1G` to bytes
+
+### Query Object
+
+The `Query` class accumulates SQL components:
+
+```ruby
+query = visitor.to_query
+
+query.conditions  # Array of WHERE conditions
+query.joins       # Hash of JOIN clauses
+query.params      # Array of bound parameters
+
+sql, params = query.to_sql
+# Returns: [sql_string, parameters_array]
+```
+
+## Fuzzy Matching Limitations
+
+The current implementation does **not** support:
+- **AROUND operator** (proximity search) - generates no-op `(1 = 0)` condition
+- Full-text search with word distance calculations
+- Stemming or phonetic matching
+- Levenshtein distance / typo tolerance
+
+These features require additional implementation, potentially using SQLite FTS5 extensions.
+