heathrow 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. checksums.yaml +7 -0
  2. data/.gitignore +58 -0
  3. data/README.md +205 -0
  4. data/bin/heathrow +42 -0
  5. data/bin/heathrowd +283 -0
  6. data/docs/ARCHITECTURE.md +1172 -0
  7. data/docs/DATABASE_SCHEMA.md +685 -0
  8. data/docs/DEVELOPMENT_WORKFLOW.md +867 -0
  9. data/docs/DISCORD_SETUP.md +142 -0
  10. data/docs/GMAIL_OAUTH_SETUP.md +120 -0
  11. data/docs/PLUGIN_SYSTEM.md +1370 -0
  12. data/docs/PROJECT_PLAN.md +1022 -0
  13. data/docs/README.md +417 -0
  14. data/docs/REDDIT_SETUP.md +174 -0
  15. data/docs/REPLY_FORWARD.md +182 -0
  16. data/docs/WHATSAPP_TELEGRAM_SETUP.md +306 -0
  17. data/heathrow.gemspec +34 -0
  18. data/heathrowd.service +21 -0
  19. data/img/heathrow.svg +95 -0
  20. data/img/rss_threaded.png +0 -0
  21. data/img/sources.png +0 -0
  22. data/lib/heathrow/address_book.rb +42 -0
  23. data/lib/heathrow/config.rb +332 -0
  24. data/lib/heathrow/database.rb +731 -0
  25. data/lib/heathrow/database_new.rb +392 -0
  26. data/lib/heathrow/event_bus.rb +175 -0
  27. data/lib/heathrow/logger.rb +122 -0
  28. data/lib/heathrow/message.rb +176 -0
  29. data/lib/heathrow/message_composer.rb +399 -0
  30. data/lib/heathrow/message_organizer.rb +774 -0
  31. data/lib/heathrow/migrations/001_initial_schema.rb +248 -0
  32. data/lib/heathrow/notmuch.rb +45 -0
  33. data/lib/heathrow/oauth2_smtp.rb +254 -0
  34. data/lib/heathrow/plugin/base.rb +212 -0
  35. data/lib/heathrow/plugin_manager.rb +141 -0
  36. data/lib/heathrow/poller.rb +93 -0
  37. data/lib/heathrow/smtp_sender.rb +204 -0
  38. data/lib/heathrow/source.rb +39 -0
  39. data/lib/heathrow/sources/base.rb +74 -0
  40. data/lib/heathrow/sources/discord.rb +357 -0
  41. data/lib/heathrow/sources/gmail.rb +294 -0
  42. data/lib/heathrow/sources/imap.rb +198 -0
  43. data/lib/heathrow/sources/instagram.rb +307 -0
  44. data/lib/heathrow/sources/instagram_fetch.py +101 -0
  45. data/lib/heathrow/sources/instagram_send.py +55 -0
  46. data/lib/heathrow/sources/instagram_send_marionette.py +104 -0
  47. data/lib/heathrow/sources/maildir.rb +606 -0
  48. data/lib/heathrow/sources/messenger.rb +212 -0
  49. data/lib/heathrow/sources/messenger_fetch.js +297 -0
  50. data/lib/heathrow/sources/messenger_fetch_marionette.py +138 -0
  51. data/lib/heathrow/sources/messenger_send.js +32 -0
  52. data/lib/heathrow/sources/messenger_send.py +100 -0
  53. data/lib/heathrow/sources/reddit.rb +461 -0
  54. data/lib/heathrow/sources/rss.rb +299 -0
  55. data/lib/heathrow/sources/slack.rb +375 -0
  56. data/lib/heathrow/sources/source_manager.rb +328 -0
  57. data/lib/heathrow/sources/telegram.rb +498 -0
  58. data/lib/heathrow/sources/webpage.rb +207 -0
  59. data/lib/heathrow/sources/weechat.rb +479 -0
  60. data/lib/heathrow/sources/whatsapp.rb +474 -0
  61. data/lib/heathrow/ui/application.rb +8098 -0
  62. data/lib/heathrow/ui/navigation.rb +8 -0
  63. data/lib/heathrow/ui/panes.rb +8 -0
  64. data/lib/heathrow/ui/source_wizard.rb +567 -0
  65. data/lib/heathrow/ui/threaded_view.rb +780 -0
  66. data/lib/heathrow/ui/views.rb +8 -0
  67. data/lib/heathrow/version.rb +3 -0
  68. data/lib/heathrow/wizards/discord_wizard.rb +193 -0
  69. data/lib/heathrow/wizards/slack_wizard.rb +140 -0
  70. data/lib/heathrow.rb +55 -0
  71. metadata +147 -0
@@ -0,0 +1,1370 @@
1
+ # Heathrow Plugin System
2
+
3
+ **Philosophy:** Every communication platform is a self-contained plugin with zero coupling to other plugins.
4
+
5
+ ---
6
+
7
+ ## Table of Contents
8
+
9
+ 1. [Architecture](#architecture)
10
+ 2. [Plugin Lifecycle](#plugin-lifecycle)
11
+ 3. [Plugin Interface](#plugin-interface)
12
+ 4. [Plugin Discovery](#plugin-discovery)
13
+ 5. [Error Handling](#error-handling)
14
+ 6. [Testing Plugins](#testing-plugins)
15
+ 7. [Example Plugins](#example-plugins)
16
+ 8. [Plugin Development Guide](#plugin-development-guide)
17
+
18
+ ---
19
+
20
+ ## Architecture
21
+
22
+ ### Core Principles
23
+
24
+ 1. **Zero Coupling** - Plugins cannot interact with each other directly
25
+ 2. **Fail Independently** - Plugin crash must not crash core or other plugins
26
+ 3. **Hot Reload** - Plugins can be loaded/unloaded without restart
27
+ 4. **Version Compatibility** - Old plugins work with new core (within major version)
28
+ 5. **Sandboxed** - Plugins have limited access to system resources
29
+
30
+ ### Plugin Isolation
31
+
32
+ ```
33
+ ┌─────────────────────────────────────────────┐
34
+ │ Core Application │
35
+ │ ┌────────────────────────────────────┐ │
36
+ │ │ Plugin Manager │ │
37
+ │ │ ┌──────────────────────────────┐ │ │
38
+ │ │ │ Error Boundary (rescue) │ │ │
39
+ │ │ │ ┌────────────────────────┐ │ │ │
40
+ │ │ │ │ Plugin Instance │ │ │ │
41
+ │ │ │ │ - fetch_messages │ │ │ │
42
+ │ │ │ │ - send_message │ │ │ │
43
+ │ │ │ └────────────────────────┘ │ │ │
44
+ │ │ └──────────────────────────────┘ │ │
45
+ │ └────────────────────────────────────┘ │
46
+ └─────────────────────────────────────────────┘
47
+ ```
48
+
49
+ **If plugin crashes:**
50
+ 1. Error caught by boundary
51
+ 2. Error logged
52
+ 3. Plugin marked as failed
53
+ 4. Other plugins continue working
54
+ 5. UI shows error notification
55
+ 6. User can retry or disable plugin
56
+
57
+ ---
58
+
59
+ ## Plugin Lifecycle
60
+
61
+ ### States
62
+
63
+ ```
64
+ ┌──────────┐
65
+ │ Unloaded │
66
+ └────┬─────┘
67
+ │ load_plugin(name)
68
+
69
+ ┌──────────┐ start() fails ┌────────┐
70
+ │ Loaded │────────────────>│ Failed │
71
+ └────┬─────┘ └────┬───┘
72
+ │ start() succeeds │
73
+ ▼ │ retry
74
+ ┌──────────┐ │
75
+ │ Running │<─────────────────────┘
76
+ └────┬─────┘
77
+ │ stop() or error
78
+
79
+ ┌──────────┐
80
+ │ Stopped │
81
+ └────┬─────┘
82
+ │ unload_plugin(name)
83
+
84
+ ┌──────────┐
85
+ │ Unloaded │
86
+ └──────────┘
87
+ ```
88
+
89
+ ### Lifecycle Methods
90
+
91
+ ```ruby
92
+ class MyPlugin < Heathrow::Plugin::Base
93
+ # Called once when plugin is loaded
94
+ def initialize(config, event_bus, db)
95
+ super
96
+ @connection = nil
97
+ end
98
+
99
+ # Called when plugin is started (connects to service)
100
+ def start
101
+ @connection = connect_to_service
102
+ log(:info, "Connected successfully")
103
+ end
104
+
105
+ # Called when plugin is stopped (cleanup)
106
+ def stop
107
+ @connection&.close
108
+ log(:info, "Disconnected")
109
+ end
110
+
111
+ # Main work methods
112
+ def fetch_messages
113
+ # ...
114
+ end
115
+
116
+ def send_message(message, target)
117
+ # ...
118
+ end
119
+ end
120
+ ```
121
+
122
+ ---
123
+
124
+ ## Plugin Interface
125
+
126
+ ### Base Class: `Heathrow::Plugin::Base`
127
+
128
+ **File:** `lib/heathrow/plugin/base.rb`
129
+
130
+ ```ruby
131
+ module Heathrow
132
+ module Plugin
133
+ class Base
134
+ attr_reader :name, :config, :capabilities, :state
135
+
136
+ # Initialize plugin (do not connect to service here)
137
+ # @param config [Hash] Plugin-specific configuration
138
+ # @param event_bus [Heathrow::EventBus] For emitting events
139
+ # @param db [Heathrow::Database] For storing data
140
+ def initialize(config, event_bus, db)
141
+ @config = config
142
+ @event_bus = event_bus
143
+ @db = db
144
+ @state = :stopped
145
+ @name = self.class.name.split('::').last.downcase
146
+ end
147
+
148
+ # Start the plugin (connect to service)
149
+ # @raise [PluginError] If connection fails
150
+ def start
151
+ @state = :running
152
+ end
153
+
154
+ # Stop the plugin (disconnect, cleanup)
155
+ def stop
156
+ @state = :stopped
157
+ end
158
+
159
+ # Fetch new messages from the service
160
+ # @param since [Integer, nil] Unix timestamp to fetch from
161
+ # @return [Array<Heathrow::Message>] Array of messages
162
+ def fetch_messages(since: nil)
163
+ raise NotImplementedError, "#{self.class} must implement #fetch_messages"
164
+ end
165
+
166
+ # Send a message through this service
167
+ # @param message [Heathrow::Message] Message to send
168
+ # @param target [String, Hash] Target identifier (email, channel, etc.)
169
+ # @return [Boolean] True if sent successfully
170
+ # @raise [PluginError] If send fails
171
+ def send_message(message, target)
172
+ raise NotImplementedError, "#{self.class} must implement #send_message"
173
+ end
174
+
175
+ # Get plugin capabilities
176
+ # @return [Array<Symbol>] Capability symbols
177
+ def capabilities
178
+ []
179
+ end
180
+
181
+ # Get current plugin status
182
+ # @return [Hash] Status information
183
+ def status
184
+ {
185
+ state: @state,
186
+ name: @name,
187
+ connected: @state == :running,
188
+ last_sync: @last_sync,
189
+ last_error: @last_error
190
+ }
191
+ end
192
+
193
+ # Get plugin configuration schema (for UI)
194
+ # @return [Hash] JSON Schema for config
195
+ def self.config_schema
196
+ {}
197
+ end
198
+
199
+ # Get plugin metadata
200
+ # @return [Hash] Plugin information
201
+ def self.metadata
202
+ {
203
+ name: name.split('::').last,
204
+ description: "A Heathrow plugin",
205
+ version: "1.0.0",
206
+ author: "Unknown",
207
+ two_way: false
208
+ }
209
+ end
210
+
211
+ protected
212
+
213
+ # Log a message
214
+ # @param level [Symbol] :debug, :info, :warn, :error
215
+ # @param message [String] Log message
216
+ def log(level, message)
217
+ @event_bus.publish(:log, {
218
+ level: level,
219
+ component: @name,
220
+ message: message,
221
+ timestamp: Time.now.to_i
222
+ })
223
+ end
224
+
225
+ # Emit an event
226
+ # @param type [Symbol] Event type
227
+ # @param data [Hash] Event data
228
+ def emit_event(type, data)
229
+ @event_bus.publish(type, data.merge(plugin: @name))
230
+ end
231
+
232
+ # Store encrypted credential
233
+ # @param key [String] Credential key
234
+ # @param value [String] Credential value
235
+ def store_credential(key, value)
236
+ # Encrypt and store in database
237
+ encrypted = Heathrow::Crypto.encrypt(value, encryption_key)
238
+ @db.exec(
239
+ "UPDATE sources SET config = json_set(config, '$.credentials.#{key}', ?) WHERE plugin_type = ?",
240
+ [encrypted, @name]
241
+ )
242
+ end
243
+
244
+ # Retrieve encrypted credential
245
+ # @param key [String] Credential key
246
+ # @return [String, nil] Decrypted credential
247
+ def retrieve_credential(key)
248
+ result = @db.query(
249
+ "SELECT json_extract(config, '$.credentials.#{key}') FROM sources WHERE plugin_type = ?",
250
+ [@name]
251
+ ).first&.first
252
+
253
+ return nil unless result
254
+ Heathrow::Crypto.decrypt(result, encryption_key)
255
+ end
256
+
257
+ # Get encryption key from keychain
258
+ # @return [String] Encryption key
259
+ def encryption_key
260
+ Heathrow::Keychain.get_or_create("heathrow.encryption_key")
261
+ end
262
+
263
+ # Normalize external message to Heathrow::Message format
264
+ # @param external_msg [Object] Platform-specific message object
265
+ # @return [Heathrow::Message] Normalized message
266
+ def normalize_message(external_msg)
267
+ raise NotImplementedError, "#{self.class} must implement #normalize_message"
268
+ end
269
+ end
270
+ end
271
+ end
272
+ ```
273
+
274
+ ### Required Methods
275
+
276
+ Plugins **must** implement:
277
+
278
+ 1. `fetch_messages(since: nil)` - Return array of `Heathrow::Message` objects
279
+ 2. `capabilities` - Return array of capability symbols
280
+
281
+ Plugins **should** implement (if two-way):
282
+
283
+ 3. `send_message(message, target)` - Send a message
284
+
285
+ ### Optional Methods
286
+
287
+ Plugins **may** implement:
288
+
289
+ 1. `start` - Custom connection logic
290
+ 2. `stop` - Custom cleanup logic
291
+ 3. `normalize_message(external)` - Convert platform format to Heathrow format
292
+ 4. `search(query)` - Server-side search
293
+ 5. `mark_read(message_id)` - Mark message as read on server
294
+ 6. `delete_message(message_id)` - Delete message on server
295
+ 7. `get_thread(thread_id)` - Fetch entire thread
296
+
297
+ ---
298
+
299
+ ## Plugin Discovery
300
+
301
+ ### Automatic Discovery
302
+
303
+ Plugins are automatically discovered from:
304
+
305
+ 1. `lib/heathrow/plugins/*.rb` (built-in)
306
+ 2. `~/.heathrow/plugins/*.rb` (user-installed)
307
+ 3. Gem plugins: `heathrow-plugin-*` (community)
308
+
309
+ ### Registration
310
+
311
+ ```ruby
312
+ # lib/heathrow/plugin/registry.rb
313
+ module Heathrow
314
+ module Plugin
315
+ class Registry
316
+ @plugins = {}
317
+
318
+ # Auto-register on class definition
319
+ def self.register(plugin_class)
320
+ name = plugin_class.name.split('::').last.downcase
321
+ @plugins[name] = plugin_class
322
+ end
323
+
324
+ def self.get(name)
325
+ @plugins[name]
326
+ end
327
+
328
+ def self.all
329
+ @plugins.values
330
+ end
331
+ end
332
+
333
+ class Base
334
+ # Automatically register when subclassed
335
+ def self.inherited(subclass)
336
+ Registry.register(subclass)
337
+ end
338
+ end
339
+ end
340
+ end
341
+ ```
342
+
343
+ ### Plugin Loading
344
+
345
+ ```ruby
346
+ # lib/heathrow/plugin_manager.rb
347
+ module Heathrow
348
+ class PluginManager
349
+ def initialize(event_bus, config, db)
350
+ @event_bus = event_bus
351
+ @config = config
352
+ @db = db
353
+ @plugins = {}
354
+ end
355
+
356
+ def load_all
357
+ # Load built-in plugins
358
+ Dir["lib/heathrow/plugins/*.rb"].each { |f| require f }
359
+
360
+ # Load user plugins
361
+ user_plugin_dir = File.expand_path("~/.heathrow/plugins")
362
+ Dir["#{user_plugin_dir}/*.rb"].each { |f| require f } if Dir.exist?(user_plugin_dir)
363
+
364
+ # Load gem plugins
365
+ Gem.find_files("heathrow/plugin/*.rb").each { |f| require f }
366
+
367
+ # Instantiate configured plugins
368
+ sources = @db.query("SELECT * FROM sources WHERE enabled = 1")
369
+ sources.each do |source|
370
+ load_plugin(source['plugin_type'], source['id'])
371
+ end
372
+ end
373
+
374
+ def load_plugin(plugin_type, source_id)
375
+ plugin_class = Plugin::Registry.get(plugin_type)
376
+ raise "Unknown plugin: #{plugin_type}" unless plugin_class
377
+
378
+ # Get source configuration
379
+ source = @db.query("SELECT * FROM sources WHERE id = ?", [source_id]).first
380
+ config = JSON.parse(source['config'])
381
+
382
+ # Instantiate plugin with error boundary
383
+ plugin = plugin_class.new(config, @event_bus, @db)
384
+ plugin.start
385
+
386
+ @plugins[source_id] = plugin
387
+ @event_bus.publish(:plugin_loaded, {plugin: plugin_type, source_id: source_id})
388
+
389
+ rescue => e
390
+ @event_bus.publish(:plugin_error, {
391
+ plugin: plugin_type,
392
+ source_id: source_id,
393
+ error: e.message,
394
+ backtrace: e.backtrace
395
+ })
396
+ nil
397
+ end
398
+
399
+ def get_plugin(source_id)
400
+ @plugins[source_id]
401
+ end
402
+
403
+ def unload_plugin(source_id)
404
+ plugin = @plugins.delete(source_id)
405
+ return unless plugin
406
+
407
+ plugin.stop rescue nil # Ignore errors during shutdown
408
+ @event_bus.publish(:plugin_unloaded, {source_id: source_id})
409
+ end
410
+
411
+ def reload_plugin(source_id)
412
+ unload_plugin(source_id)
413
+
414
+ # Get plugin type for this source
415
+ source = @db.query("SELECT plugin_type FROM sources WHERE id = ?", [source_id]).first
416
+ load_plugin(source['plugin_type'], source_id)
417
+ end
418
+ end
419
+ end
420
+ ```
421
+
422
+ ---
423
+
424
+ ## Error Handling
425
+
426
+ ### Error Boundary Pattern
427
+
428
+ ```ruby
429
+ # Every plugin method is wrapped in error boundary
430
+ def safe_fetch_messages(plugin, since: nil)
431
+ plugin.fetch_messages(since: since)
432
+ rescue => e
433
+ log_plugin_error(plugin, e)
434
+ [] # Return empty array, don't crash
435
+ end
436
+
437
+ def log_plugin_error(plugin, error)
438
+ @db.exec(
439
+ "UPDATE sources SET last_error = ?, updated_at = ? WHERE id = ?",
440
+ [error.message, Time.now.to_i, plugin.source_id]
441
+ )
442
+
443
+ @event_bus.publish(:plugin_error, {
444
+ plugin: plugin.name,
445
+ error: error.message,
446
+ backtrace: error.backtrace.first(5)
447
+ })
448
+ end
449
+ ```
450
+
451
+ ### Plugin-Specific Errors
452
+
453
+ ```ruby
454
+ # lib/heathrow/plugin/errors.rb
455
+ module Heathrow
456
+ module Plugin
457
+ class Error < StandardError; end
458
+ class ConnectionError < Error; end
459
+ class AuthenticationError < Error; end
460
+ class RateLimitError < Error; end
461
+ class NotFoundError < Error; end
462
+ class ValidationError < Error; end
463
+ end
464
+ end
465
+ ```
466
+
467
+ ### Retry Strategy
468
+
469
+ ```ruby
470
+ def fetch_with_retry(plugin, max_retries: 3)
471
+ retries = 0
472
+
473
+ begin
474
+ plugin.fetch_messages
475
+ rescue Plugin::RateLimitError => e
476
+ # Wait and retry
477
+ sleep_time = 2 ** retries
478
+ log(:warn, "Rate limited, waiting #{sleep_time}s")
479
+ sleep(sleep_time)
480
+ retries += 1
481
+ retry if retries < max_retries
482
+
483
+ raise
484
+ rescue Plugin::ConnectionError => e
485
+ # Temporary network issue, retry
486
+ retries += 1
487
+ retry if retries < max_retries
488
+
489
+ raise
490
+ rescue Plugin::AuthenticationError => e
491
+ # Don't retry, needs user intervention
492
+ raise
493
+ end
494
+ end
495
+ ```
496
+
497
+ ---
498
+
499
+ ## Testing Plugins
500
+
501
+ ### Unit Testing
502
+
503
+ ```ruby
504
+ # test/plugins/test_gmail.rb
505
+ require 'minitest/autorun'
506
+ require_relative '../test_helper'
507
+
508
+ class TestGmailPlugin < Minitest::Test
509
+ def setup
510
+ @config = {
511
+ 'email' => 'test@example.com',
512
+ 'credentials' => 'mock_token'
513
+ }
514
+ @event_bus = MockEventBus.new
515
+ @db = MockDatabase.new
516
+
517
+ @plugin = Heathrow::Plugin::Gmail.new(@config, @event_bus, @db)
518
+ end
519
+
520
+ def test_capabilities
521
+ assert_includes @plugin.capabilities, :read
522
+ assert_includes @plugin.capabilities, :write
523
+ assert_includes @plugin.capabilities, :attachments
524
+ end
525
+
526
+ def test_fetch_messages
527
+ # Mock Gmail API response
528
+ stub_gmail_api do
529
+ messages = @plugin.fetch_messages
530
+ assert_instance_of Array, messages
531
+ assert messages.all? { |m| m.is_a?(Heathrow::Message) }
532
+ end
533
+ end
534
+
535
+ def test_send_message
536
+ message = Heathrow::Message.new(
537
+ recipients: ['recipient@example.com'],
538
+ subject: 'Test',
539
+ content: 'Test message'
540
+ )
541
+
542
+ stub_gmail_api do
543
+ assert @plugin.send_message(message, nil)
544
+ end
545
+ end
546
+
547
+ def test_error_handling
548
+ # Simulate network error
549
+ stub_gmail_api_error(Net::ReadTimeout) do
550
+ assert_raises(Heathrow::Plugin::ConnectionError) do
551
+ @plugin.fetch_messages
552
+ end
553
+ end
554
+ end
555
+ end
556
+ ```
557
+
558
+ ### Integration Testing
559
+
560
+ ```ruby
561
+ # test/integration/test_gmail_live.rb
562
+ # These tests require real credentials and are skipped in CI
563
+
564
+ class TestGmailLive < Minitest::Test
565
+ def setup
566
+ skip unless ENV['HEATHROW_GMAIL_TEST_TOKEN']
567
+
568
+ @config = {
569
+ 'email' => ENV['HEATHROW_GMAIL_TEST_EMAIL'],
570
+ 'credentials' => ENV['HEATHROW_GMAIL_TEST_TOKEN']
571
+ }
572
+ # ... setup
573
+ end
574
+
575
+ def test_real_connection
576
+ @plugin.start
577
+ assert_equal :running, @plugin.status[:state]
578
+ end
579
+
580
+ def test_real_fetch
581
+ messages = @plugin.fetch_messages
582
+ # Just verify it doesn't crash, actual count varies
583
+ assert_instance_of Array, messages
584
+ end
585
+ end
586
+ ```
587
+
588
+ ### Mock Helpers
589
+
590
+ ```ruby
591
+ # test/mocks.rb
592
+ class MockEventBus
593
+ def initialize
594
+ @events = []
595
+ end
596
+
597
+ def publish(type, data)
598
+ @events << {type: type, data: data}
599
+ end
600
+
601
+ def events_of_type(type)
602
+ @events.select { |e| e[:type] == type }
603
+ end
604
+ end
605
+
606
+ class MockDatabase
607
+ def initialize
608
+ @data = {}
609
+ end
610
+
611
+ def query(sql, params = [])
612
+ # Simple mock implementation
613
+ []
614
+ end
615
+
616
+ def exec(sql, params = [])
617
+ # No-op
618
+ end
619
+ end
620
+ ```
621
+
622
+ ---
623
+
624
+ ## Example Plugins
625
+
626
+ ### 1. Gmail Plugin (Two-way)
627
+
628
+ **File:** `lib/heathrow/plugins/gmail.rb`
629
+
630
+ ```ruby
631
+ require 'google/apis/gmail_v1'
632
+ require 'googleauth'
633
+
634
+ module Heathrow
635
+ module Plugin
636
+ class Gmail < Base
637
+ def self.metadata
638
+ {
639
+ name: "Gmail",
640
+ description: "Gmail email integration via Google API",
641
+ version: "1.0.0",
642
+ author: "Heathrow Team",
643
+ two_way: true
644
+ }
645
+ end
646
+
647
+ def self.config_schema
648
+ {
649
+ type: "object",
650
+ required: ["email"],
651
+ properties: {
652
+ email: {
653
+ type: "string",
654
+ format: "email",
655
+ description: "Gmail address"
656
+ },
657
+ sync_labels: {
658
+ type: "array",
659
+ items: {type: "string"},
660
+ default: ["INBOX"],
661
+ description: "Labels to sync"
662
+ },
663
+ sync_days: {
664
+ type: "integer",
665
+ default: 30,
666
+ description: "Days of history to sync"
667
+ }
668
+ }
669
+ }
670
+ end
671
+
672
+ def initialize(config, event_bus, db)
673
+ super
674
+ @service = nil
675
+ end
676
+
677
+ def start
678
+ super
679
+
680
+ # Initialize Gmail API service
681
+ @service = Google::Apis::GmailV1::GmailService.new
682
+ @service.authorization = get_authorization
683
+
684
+ # Test connection
685
+ @service.get_user_profile('me')
686
+
687
+ log(:info, "Connected to Gmail: #{@config['email']}")
688
+ rescue Google::Apis::AuthorizationError => e
689
+ raise Plugin::AuthenticationError, "Gmail auth failed: #{e.message}"
690
+ rescue => e
691
+ raise Plugin::ConnectionError, "Gmail connection failed: #{e.message}"
692
+ end
693
+
694
+ def stop
695
+ super
696
+ @service = nil
697
+ end
698
+
699
+ def capabilities
700
+ [:read, :write, :attachments, :threads, :search]
701
+ end
702
+
703
+ def fetch_messages(since: nil)
704
+ query = build_query(since)
705
+
706
+ message_ids = fetch_message_ids(query)
707
+ messages = message_ids.map { |id| fetch_message_detail(id) }
708
+
709
+ messages.compact.map { |m| normalize_message(m) }
710
+
711
+ rescue Google::Apis::RateLimitError => e
712
+ raise Plugin::RateLimitError, "Gmail rate limit: #{e.message}"
713
+ rescue => e
714
+ log(:error, "Fetch failed: #{e.message}")
715
+ raise Plugin::Error, e.message
716
+ end
717
+
718
+ def send_message(message, target = nil)
719
+ raw_message = build_raw_message(message)
720
+
721
+ @service.send_user_message(
722
+ 'me',
723
+ Google::Apis::GmailV1::Message.new(raw: raw_message)
724
+ )
725
+
726
+ emit_event(:message_sent, {
727
+ recipients: message.recipients,
728
+ subject: message.subject
729
+ })
730
+
731
+ true
732
+ rescue => e
733
+ log(:error, "Send failed: #{e.message}")
734
+ raise Plugin::Error, e.message
735
+ end
736
+
737
+ def search(query)
738
+ # Implement Gmail search syntax
739
+ fetch_messages(query: query)
740
+ end
741
+
742
+ private
743
+
744
+ def get_authorization
745
+ token = retrieve_credential('oauth_token')
746
+ refresh_token = retrieve_credential('refresh_token')
747
+
748
+ # Create OAuth2 credentials
749
+ credentials = Google::Auth::UserRefreshCredentials.new(
750
+ client_id: ENV['GMAIL_CLIENT_ID'],
751
+ client_secret: ENV['GMAIL_CLIENT_SECRET'],
752
+ refresh_token: refresh_token,
753
+ access_token: token
754
+ )
755
+
756
+ # Refresh if needed
757
+ credentials.refresh! if credentials.expired?
758
+
759
+ # Store new token
760
+ store_credential('oauth_token', credentials.access_token) if credentials.access_token != token
761
+
762
+ credentials
763
+ end
764
+
765
+ def build_query(since)
766
+ parts = []
767
+
768
+ # Sync specific labels
769
+ labels = @config['sync_labels'] || ['INBOX']
770
+ parts << "label:(#{labels.join(' OR ')})"
771
+
772
+ # Since timestamp
773
+ if since
774
+ date = Time.at(since).strftime('%Y/%m/%d')
775
+ parts << "after:#{date}"
776
+ else
777
+ # Default: last N days
778
+ days = @config['sync_days'] || 30
779
+ parts << "newer_than:#{days}d"
780
+ end
781
+
782
+ parts.join(' ')
783
+ end
784
+
785
+ def fetch_message_ids(query)
786
+ ids = []
787
+ page_token = nil
788
+
789
+ loop do
790
+ result = @service.list_user_messages('me', q: query, page_token: page_token)
791
+ ids.concat(result.messages.map(&:id)) if result.messages
792
+
793
+ page_token = result.next_page_token
794
+ break unless page_token
795
+ end
796
+
797
+ ids
798
+ end
799
+
800
+ def fetch_message_detail(message_id)
801
+ @service.get_user_message('me', message_id, format: 'full')
802
+ end
803
+
804
+ def normalize_message(gmail_msg)
805
+ headers = headers_to_hash(gmail_msg.payload.headers)
806
+
807
+ Heathrow::Message.new(
808
+ external_id: gmail_msg.id,
809
+ thread_id: gmail_msg.thread_id,
810
+ sender: headers['from'],
811
+ recipients: parse_recipients(headers['to']),
812
+ subject: headers['subject'],
813
+ content: extract_body(gmail_msg.payload),
814
+ html_content: extract_html(gmail_msg.payload),
815
+ timestamp: gmail_msg.internal_date / 1000,
816
+ received_at: Time.now.to_i,
817
+ labels: gmail_msg.label_ids,
818
+ metadata: {
819
+ gmail_message_id: headers['message-id'],
820
+ gmail_thread_id: gmail_msg.thread_id,
821
+ gmail_labels: gmail_msg.label_ids
822
+ }.to_json
823
+ )
824
+ end
825
+
826
+ def headers_to_hash(headers)
827
+ headers.each_with_object({}) do |h, hash|
828
+ hash[h.name.downcase] = h.value
829
+ end
830
+ end
831
+
832
+ def parse_recipients(to_header)
833
+ return [] unless to_header
834
+ to_header.split(',').map(&:strip)
835
+ end
836
+
837
+ def extract_body(payload)
838
+ if payload.parts
839
+ text_part = payload.parts.find { |p| p.mime_type == 'text/plain' }
840
+ return decode_body(text_part.body.data) if text_part
841
+ end
842
+
843
+ decode_body(payload.body.data) if payload.body&.data
844
+ end
845
+
846
+ def extract_html(payload)
847
+ if payload.parts
848
+ html_part = payload.parts.find { |p| p.mime_type == 'text/html' }
849
+ return decode_body(html_part.body.data) if html_part
850
+ end
851
+
852
+ nil
853
+ end
854
+
855
+ def decode_body(data)
856
+ return nil unless data
857
+ Base64.urlsafe_decode64(data)
858
+ end
859
+
860
+ def build_raw_message(message)
861
+ # Build RFC 2822 email
862
+ mail = <<~EMAIL
863
+ From: #{@config['email']}
864
+ To: #{message.recipients.join(', ')}
865
+ Subject: #{message.subject}
866
+ Content-Type: text/plain; charset=UTF-8
867
+
868
+ #{message.content}
869
+ EMAIL
870
+
871
+ Base64.urlsafe_encode64(mail, padding: false)
872
+ end
873
+ end
874
+ end
875
+ end
876
+ ```
877
+
878
+ ### 2. RSS Plugin (One-way)
879
+
880
+ **File:** `lib/heathrow/plugins/rss.rb`
881
+
882
+ ```ruby
883
+ require 'rss'
884
+ require 'open-uri'
885
+
886
+ module Heathrow
887
+ module Plugin
888
+ class Rss < Base
889
+ def self.metadata
890
+ {
891
+ name: "RSS",
892
+ description: "RSS/Atom feed reader",
893
+ version: "1.0.0",
894
+ author: "Heathrow Team",
895
+ two_way: false
896
+ }
897
+ end
898
+
899
+ def self.config_schema
900
+ {
901
+ type: "object",
902
+ required: ["feed_url"],
903
+ properties: {
904
+ feed_url: {
905
+ type: "string",
906
+ format: "uri",
907
+ description: "RSS feed URL"
908
+ },
909
+ update_interval: {
910
+ type: "integer",
911
+ default: 3600,
912
+ description: "Update interval in seconds"
913
+ }
914
+ }
915
+ }
916
+ end
917
+
918
+ def capabilities
919
+ [:read]
920
+ end
921
+
922
+ def fetch_messages(since: nil)
923
+ feed = fetch_feed
924
+ items = feed.items
925
+
926
+ # Filter by timestamp if provided
927
+ items = items.select { |i| i.pubDate.to_i > since } if since
928
+
929
+ items.map { |item| normalize_message(item) }
930
+
931
+ rescue => e
932
+ log(:error, "Feed fetch failed: #{e.message}")
933
+ []
934
+ end
935
+
936
+ private
937
+
938
+ def fetch_feed
939
+ URI.open(@config['feed_url']) do |rss|
940
+ RSS::Parser.parse(rss)
941
+ end
942
+ end
943
+
944
+ def normalize_message(item)
945
+ Heathrow::Message.new(
946
+ external_id: item.guid&.content || item.link,
947
+ sender: @config['feed_url'],
948
+ recipients: [],
949
+ subject: item.title,
950
+ content: strip_html(item.description || item.content_encoded),
951
+ html_content: item.description || item.content_encoded,
952
+ timestamp: item.pubDate.to_i,
953
+ received_at: Time.now.to_i,
954
+ metadata: {
955
+ rss_link: item.link,
956
+ rss_categories: item.categories.map(&:content)
957
+ }.to_json
958
+ )
959
+ end
960
+
961
+ def strip_html(html)
962
+ return '' unless html
963
+ html.gsub(/<[^>]+>/, '').strip
964
+ end
965
+ end
966
+ end
967
+ end
968
+ ```
969
+
970
+ ### 3. Slack Plugin (Two-way, Real-time)
971
+
972
+ **File:** `lib/heathrow/plugins/slack.rb`
973
+
974
+ ```ruby
975
+ require 'slack-ruby-client'
976
+
977
+ module Heathrow
978
+ module Plugin
979
+ class Slack < Base
980
+ def self.metadata
981
+ {
982
+ name: "Slack",
983
+ description: "Slack workspace integration",
984
+ version: "1.0.0",
985
+ author: "Heathrow Team",
986
+ two_way: true
987
+ }
988
+ end
989
+
990
+ def capabilities
991
+ [:read, :write, :real_time, :threads, :reactions]
992
+ end
993
+
994
+ def start
995
+ super
996
+
997
+ # Initialize Slack client
998
+ ::Slack.configure do |config|
999
+ config.token = retrieve_credential('token')
1000
+ end
1001
+
1002
+ @client = ::Slack::Web::Client.new
1003
+ @rtm_client = ::Slack::RealTime::Client.new
1004
+
1005
+ # Test connection
1006
+ auth = @client.auth_test
1007
+ log(:info, "Connected to Slack: #{auth.team}")
1008
+
1009
+ # Start real-time client
1010
+ start_rtm
1011
+
1012
+ rescue ::Slack::Web::Api::Errors::SlackError => e
1013
+ raise Plugin::AuthenticationError, "Slack auth failed: #{e.message}"
1014
+ end
1015
+
1016
+ def stop
1017
+ super
1018
+ @rtm_client&.stop!
1019
+ end
1020
+
1021
+ def fetch_messages(since: nil)
1022
+ messages = []
1023
+
1024
+ # Fetch from configured channels
1025
+ channels = @config['sync_channels'] || []
1026
+ channels.each do |channel_name|
1027
+ channel = find_channel(channel_name)
1028
+ next unless channel
1029
+
1030
+ history = @client.conversations_history(
1031
+ channel: channel['id'],
1032
+ oldest: since || (Time.now.to_i - 86400) # Default: last 24h
1033
+ )
1034
+
1035
+ messages.concat(history.messages.map { |m| normalize_message(m, channel) })
1036
+ end
1037
+
1038
+ messages
1039
+ end
1040
+
1041
+ def send_message(message, target)
1042
+ channel = find_channel(target)
1043
+ raise Plugin::NotFoundError, "Channel not found: #{target}" unless channel
1044
+
1045
+ @client.chat_postMessage(
1046
+ channel: channel['id'],
1047
+ text: message.content,
1048
+ thread_ts: message.metadata&.dig('slack_thread_ts')
1049
+ )
1050
+
1051
+ true
1052
+ end
1053
+
1054
+ private
1055
+
1056
+ def start_rtm
1057
+ @rtm_client.on :message do |data|
1058
+ next if data.subtype # Skip edited, deleted, etc.
1059
+
1060
+ channel = find_channel_by_id(data.channel)
1061
+ message = normalize_message(data, channel)
1062
+
1063
+ emit_event(:message_received, {message: message})
1064
+ end
1065
+
1066
+ Thread.new { @rtm_client.start! }
1067
+ end
1068
+
1069
+ def find_channel(name)
1070
+ @channels_cache ||= @client.conversations_list.channels
1071
+ @channels_cache.find { |c| c.name == name }
1072
+ end
1073
+
1074
+ def find_channel_by_id(id)
1075
+ @channels_cache ||= @client.conversations_list.channels
1076
+ @channels_cache.find { |c| c.id == id }
1077
+ end
1078
+
1079
+ def normalize_message(slack_msg, channel)
1080
+ Heathrow::Message.new(
1081
+ external_id: slack_msg.ts,
1082
+ thread_id: slack_msg.thread_ts || slack_msg.ts,
1083
+ sender: get_user_name(slack_msg.user),
1084
+ recipients: [channel['name']],
1085
+ subject: "#{channel['name']} - Slack",
1086
+ content: slack_msg.text,
1087
+ timestamp: slack_msg.ts.to_f.to_i,
1088
+ received_at: Time.now.to_i,
1089
+ metadata: {
1090
+ slack_channel: channel['name'],
1091
+ slack_channel_id: channel['id'],
1092
+ slack_ts: slack_msg.ts,
1093
+ slack_thread_ts: slack_msg.thread_ts
1094
+ }.to_json
1095
+ )
1096
+ end
1097
+
1098
+ def get_user_name(user_id)
1099
+ @users_cache ||= {}
1100
+ @users_cache[user_id] ||= begin
1101
+ user = @client.users_info(user: user_id).user
1102
+ user.profile.display_name.empty? ? user.name : user.profile.display_name
1103
+ end
1104
+ end
1105
+ end
1106
+ end
1107
+ end
1108
+ ```
1109
+
1110
+ ---
1111
+
1112
+ ## Plugin Development Guide
1113
+
1114
+ ### Step-by-Step: Creating a New Plugin
1115
+
1116
+ #### 1. Create Plugin File
1117
+
1118
+ ```bash
1119
+ touch lib/heathrow/plugins/myservice.rb
1120
+ ```
1121
+
1122
+ #### 2. Define Plugin Class
1123
+
1124
+ ```ruby
1125
+ module Heathrow
1126
+ module Plugin
1127
+ class Myservice < Base
1128
+ def self.metadata
1129
+ {
1130
+ name: "MyService",
1131
+ description: "Integration with MyService platform",
1132
+ version: "1.0.0",
1133
+ author: "Your Name",
1134
+ two_way: true # or false for read-only
1135
+ }
1136
+ end
1137
+
1138
+ def capabilities
1139
+ [:read, :write] # Adjust based on what you implement
1140
+ end
1141
+
1142
+ # Implement required methods...
1143
+ end
1144
+ end
1145
+ end
1146
+ ```
1147
+
1148
+ #### 3. Implement Required Methods
1149
+
1150
+ ```ruby
1151
+ def fetch_messages(since: nil)
1152
+ # 1. Connect to service API
1153
+ # 2. Fetch messages since timestamp
1154
+ # 3. Convert to Heathrow::Message objects
1155
+ # 4. Return array
1156
+ []
1157
+ end
1158
+
1159
+ def send_message(message, target)
1160
+ # 1. Connect to service API
1161
+ # 2. Send message to target
1162
+ # 3. Return true on success
1163
+ # 4. Raise Plugin::Error on failure
1164
+ true
1165
+ end
1166
+ ```
1167
+
1168
+ #### 4. Add Configuration Schema
1169
+
1170
+ ```ruby
1171
+ def self.config_schema
1172
+ {
1173
+ type: "object",
1174
+ required: ["api_key"],
1175
+ properties: {
1176
+ api_key: {
1177
+ type: "string",
1178
+ description: "MyService API key"
1179
+ },
1180
+ sync_interval: {
1181
+ type: "integer",
1182
+ default: 300,
1183
+ description: "Sync interval in seconds"
1184
+ }
1185
+ }
1186
+ }
1187
+ end
1188
+ ```
1189
+
1190
+ #### 5. Test Your Plugin
1191
+
1192
+ ```ruby
1193
+ # test/plugins/test_myservice.rb
1194
+ class TestMyservice < Minitest::Test
1195
+ def test_fetch_messages
1196
+ # Mock API, test fetch logic
1197
+ end
1198
+
1199
+ def test_send_message
1200
+ # Mock API, test send logic
1201
+ end
1202
+ end
1203
+ ```
1204
+
1205
+ #### 6. Document Your Plugin
1206
+
1207
+ ```ruby
1208
+ # Add to docs/PLUGINS.md
1209
+ ## MyService Plugin
1210
+
1211
+ **Type:** Two-way
1212
+ **Capabilities:** read, write
1213
+
1214
+ ### Configuration
1215
+
1216
+ - `api_key` (required): Your MyService API key
1217
+ - `sync_interval` (optional): Sync interval in seconds (default: 300)
1218
+
1219
+ ### Setup
1220
+
1221
+ 1. Get API key from https://myservice.com/api
1222
+ 2. Add source in Heathrow configuration
1223
+ 3. Test connection
1224
+ ```
1225
+
1226
+ ---
1227
+
1228
+ ## Best Practices
1229
+
1230
+ ### 1. Error Handling
1231
+
1232
+ Always use specific error types:
1233
+
1234
+ ```ruby
1235
+ def fetch_messages(since: nil)
1236
+ response = api_call
1237
+
1238
+ case response.code
1239
+ when 401
1240
+ raise Plugin::AuthenticationError, "Invalid credentials"
1241
+ when 429
1242
+ raise Plugin::RateLimitError, "Rate limit exceeded"
1243
+ when 404
1244
+ raise Plugin::NotFoundError, "Resource not found"
1245
+ when 500..599
1246
+ raise Plugin::ConnectionError, "Server error: #{response.code}"
1247
+ end
1248
+
1249
+ # Process response...
1250
+ rescue Timeout::Error
1251
+ raise Plugin::ConnectionError, "Request timeout"
1252
+ rescue JSON::ParserError => e
1253
+ raise Plugin::Error, "Invalid response format: #{e.message}"
1254
+ end
1255
+ ```
1256
+
1257
+ ### 2. Credentials
1258
+
1259
+ Never log or expose credentials:
1260
+
1261
+ ```ruby
1262
+ def start
1263
+ api_key = retrieve_credential('api_key')
1264
+
1265
+ # Good
1266
+ log(:info, "Connecting to service...")
1267
+
1268
+ # Bad - leaks credential
1269
+ # log(:info, "Using API key: #{api_key}")
1270
+
1271
+ connect(api_key)
1272
+ end
1273
+ ```
1274
+
1275
+ ### 3. Rate Limiting
1276
+
1277
+ Respect API rate limits:
1278
+
1279
+ ```ruby
1280
+ def fetch_messages(since: nil)
1281
+ # Implement backoff
1282
+ sleep(rate_limit_delay) if rate_limited?
1283
+
1284
+ # Batch requests
1285
+ message_ids.each_slice(100) do |batch|
1286
+ fetch_batch(batch)
1287
+ end
1288
+ end
1289
+ ```
1290
+
1291
+ ### 4. Caching
1292
+
1293
+ Cache expensive operations:
1294
+
1295
+ ```ruby
1296
+ def get_user_name(user_id)
1297
+ @user_cache ||= {}
1298
+ @user_cache[user_id] ||= fetch_user_from_api(user_id).name
1299
+ end
1300
+ ```
1301
+
1302
+ ### 5. Logging
1303
+
1304
+ Log important events:
1305
+
1306
+ ```ruby
1307
+ def fetch_messages(since: nil)
1308
+ log(:debug, "Fetching messages since #{since || 'beginning'}")
1309
+
1310
+ messages = do_fetch
1311
+
1312
+ log(:info, "Fetched #{messages.count} messages")
1313
+
1314
+ messages
1315
+ rescue => e
1316
+ log(:error, "Fetch failed: #{e.message}")
1317
+ raise
1318
+ end
1319
+ ```
1320
+
1321
+ ---
1322
+
1323
+ ## Plugin Testing Checklist
1324
+
1325
+ Before submitting a plugin, test:
1326
+
1327
+ - [ ] Connection/authentication
1328
+ - [ ] Fetch messages (empty, small, large batches)
1329
+ - [ ] Send message
1330
+ - [ ] Handle network errors gracefully
1331
+ - [ ] Handle rate limits
1332
+ - [ ] Handle authentication errors
1333
+ - [ ] Plugin can be stopped and restarted
1334
+ - [ ] No memory leaks on long runs
1335
+ - [ ] Credentials are encrypted
1336
+ - [ ] No credentials in logs
1337
+ - [ ] Works with proxy (if applicable)
1338
+ - [ ] Handles malformed API responses
1339
+ - [ ] Thread-safe (if using real-time)
1340
+
1341
+ ---
1342
+
1343
+ ## Community Plugins
1344
+
1345
+ To publish a community plugin:
1346
+
1347
+ 1. Create gem: `heathrow-plugin-myservice`
1348
+ 2. Include plugin class in `lib/heathrow/plugin/myservice.rb`
1349
+ 3. Add README with setup instructions
1350
+ 4. Publish to RubyGems
1351
+ 5. Submit PR to add to official plugin directory
1352
+
1353
+ **Plugin Gem Template:**
1354
+
1355
+ ```
1356
+ heathrow-plugin-myservice/
1357
+ ├── lib/
1358
+ │ └── heathrow/
1359
+ │ └── plugin/
1360
+ │ └── myservice.rb
1361
+ ├── test/
1362
+ │ └── test_myservice.rb
1363
+ ├── README.md
1364
+ ├── LICENSE
1365
+ └── heathrow-plugin-myservice.gemspec
1366
+ ```
1367
+
1368
+ ---
1369
+
1370
+ This plugin system ensures every integration is isolated, testable, and can fail independently without bringing down Heathrow.