@chronicle.app/imessage 0.3.0 → 0.5.0

package/README.md

@@ -1,9 +1,13 @@
 # @chronicle.app/imessage
 
-Read `chat.db` with `ImessageExtractor`, then emit MessageAction/Message records with `ImessageTransformer`. `input` defaults to the macOS Messages database; use a consistent backup for exported data. The connection is read-only via built-in `node:sqlite`. Date windows are exclusive, records newest-first, and `limit: 0` means unlimited.
+Read `chat.db` with `ImessageExtractor`, then emit MessageAction/Message records with `ImessageTransformer`. Sending a message brings it into being, so the `Message` is the action's `result`, and the action is keyed by it. `input` defaults to the macOS Messages database; use a consistent backup for exported data. Reading it needs Full Disk Access for the app that runs Chronicle (Terminal, iTerm, your editor): System Settings → Privacy & Security → Full Disk Access. Without it the run stops with that hint, not SQLite's `unable to open database file`. The connection is read-only via built-in `node:sqlite`. Date windows are exclusive, records newest-first, and `limit: 0` means unlimited.
 
-Automatic iCloud account and AddressBook enrichment comes from `@chronicle.app/icloud`. `includeContactNames: false` skips contacts. An explicit `account` object (or null) lets callers describe exported data without host account lookup. Transformer config can inject `lookupContact` for exported contacts. Missing permissions/account/contact data fall back to source-local identity and unlabelled participants. When no identifier for the account owner is available, the owner is left out of the agent and recipients.
+A message in a group chat is part of the group's `Channel` (`isPartOf`), keyed in the `imessage` namespace by Apple's ID for the group: `original_group_id`, the ID it started with, `sameAs` `group_id` when Messages replaced it. `groupChatChannel` in `@chronicle.app/apple` builds it, and the Apple Photos plugin's Shared with You keys the same group with it. Each device keeps its own `chat_identifier` for a chat, so that isn't the key. The Channel's `name` is the group's name when it has one, and its `member` edges are the chat's participants and you, from the first message a run reads in the chat. chat.db's roster lists only the others, but every group chat in it is one you're in, or were. You're left out when there's no identifier for you, as you are from a message's sender and recipients. A group message's recipients are still each other participant and you. A one-to-one chat has no `Channel`.
 
-Attachments are local file references with MIME metadata, not copied bytes. Missing files are skipped. Attributed-body decoding retains the existing heuristic; it is not a complete typedstream decoder. Raw records preserve the source blob, and large integer timestamps are decimal strings so serialization is safe.
+Automatic iCloud account and AddressBook enrichment comes from `@chronicle.app/apple`. `includeContactNames: false` skips contacts. An explicit `account` object (or null) lets callers describe exported data without host account lookup. Transformer config can inject `lookupContact` for exported contacts. Missing permissions/account/contact data fall back to source-local identity and unlabelled participants. When no identifier for the account owner is available, the owner is left out of the agent and recipients.
+
+Messages keeps system events, such as a group being renamed or someone joining it, as rows beside the messages, with no text and often no sender. They aren't messages, so they're left out, and a survey doesn't count them. Attachments are local file references with MIME metadata, not copied bytes. Missing files are skipped. So are files Messages keeps hidden, such as a link preview's images and icon, which no one sent: the link is in the message's text. Attributed-body decoding retains the existing heuristic; it is not a complete typedstream decoder. Raw records preserve the source blob, and large integer timestamps are decimal strings so serialization is safe.
+
+The plugin declares a `deepLinks` template for `Message` (`messages://open?message-guid=<GUID>`), so a host can open the message in Messages. SMS messages are linked the same way, since they are in `chat.db` too. Attachments are not linked, because an attachment's GUID is not a message GUID.
 
 Tests use synthetic Messages and AddressBook databases, account command fixtures, and temporary attachment files. They do not read personal messages or contacts.

package/dist/connectors/ImessageExtractor.d.ts

@@ -1,4 +1,4 @@
-import { Record } from '@chronicle.app/etl';
+import { Record, type Survey, type SurveyOptions } from '@chronicle.app/etl';
 import { SqliteExtractor } from '@chronicle.app/etl-sqlite';
 import ImessageTransformer from './ImessageTransformer.js';
 export declare class ImessageExtractor extends SqliteExtractor<typeof ImessageExtractor> {
@@ -8,11 +8,43 @@ export declare class ImessageExtractor extends SqliteExtractor<typeof ImessageEx
     static delivery: "local";
     static strategy: string;
     static default: boolean;
+    static databases: {
+        tables: {
+            message: string[];
+            handle: string[];
+            chat: string[];
+            chat_message_join: string[];
+            chat_handle_join: string[];
+            message_attachment_join: string[];
+            attachment: string[];
+        };
+    }[];
     static schema: any;
     static defaultTransformer: typeof ImessageTransformer;
     extract(): AsyncGenerator<Record>;
+    /** The messages a run reads, one per chat they're in, and when, from three aggregates. */
+    survey(options?: SurveyOptions): Promise<Survey>;
+    /**
+     * Messages within since and until. chat.db stores message.date as nanoseconds
+     * since the Apple epoch. A system event, such as a group being renamed or
+     * someone joining it, is a row too, with no text and often no sender, so
+     * only rows that are messages (`item_type` 0) are read.
+     */
+    private messageFilter;
     private queryMessages;
     private loadChats;
+    /**
+     * The group chats, by chat, with their names and Apple's IDs for them. An
+     * older Messages without a column has none of what it holds.
+     */
+    private loadGroups;
+    /** A column of `chat` the schema marks optional, or NULL when this Messages has none. */
+    private chatColumn;
+    /**
+     * The attachments a run reads: files Messages shows. It keeps others hidden,
+     * such as the images and icon of a link preview, which no one sent.
+     */
+    private attachmentFilter;
     private loadAttachments;
     private processAttributedBody;
 }

package/dist/connectors/ImessageExtractor.js

@@ -1,5 +1,5 @@
-import { SqliteExtractor, iosToUnixTimestamp, timeRangeConditions, APPLE_EPOCH_OFFSET_SECONDS, } from '@chronicle.app/etl-sqlite';
-import { getICloudAccount } from '@chronicle.app/icloud';
+import { SqliteExtractor, countsByMonth, iosToUnixTimestamp, monthSql, timeRangeConditions, APPLE_EPOCH_OFFSET_SECONDS, } from '@chronicle.app/etl-sqlite';
+import { GROUP_CHAT_STYLE, getICloudAccount } from '@chronicle.app/apple';
 import ImessageTransformer from './ImessageTransformer.js';
 import { z } from 'zod';
 export class ImessageExtractor extends SqliteExtractor {
@@ -9,6 +9,40 @@ export class ImessageExtractor extends SqliteExtractor {
     static delivery = 'local';
     static strategy = 'app-db';
     static default = true;
+    // What extract() and survey() read, which a chat.db must have.
+    static databases = [
+        {
+            tables: {
+                message: [
+                    'ROWID',
+                    'guid',
+                    'text',
+                    'service',
+                    'date',
+                    'is_from_me',
+                    'handle_id',
+                    'attributedBody',
+                    'item_type?',
+                ],
+                handle: ['ROWID', 'id', 'service'],
+                chat: ['ROWID', 'style', 'display_name?', 'group_id?', 'original_group_id?'],
+                chat_message_join: ['chat_id', 'message_id'],
+                chat_handle_join: ['chat_id', 'handle_id'],
+                message_attachment_join: ['message_id', 'attachment_id'],
+                attachment: [
+                    'ROWID',
+                    'guid',
+                    'filename',
+                    'mime_type',
+                    'uti',
+                    'total_bytes',
+                    'transfer_name',
+                    'is_outgoing',
+                    'hide_attachment?',
+                ],
+            },
+        },
+    ];
     static schema = SqliteExtractor.schema.omit({ input: true }).extend({
         input: z
             .string()
@@ -32,11 +66,20 @@ export class ImessageExtractor extends SqliteExtractor {
     static defaultTransformer = ImessageTransformer;
     async *extract() {
         const chats = this.loadChats();
+        const groups = this.loadGroups();
         const rows = this.queryMessages();
         const attachments = this.loadAttachments();
         // Get real iCloud account info automatically
         const realIcloudAccount = this.config.account === undefined ? await getICloudAccount() : this.config.account;
+        // The group chats whose roster a message has carried, so each goes on its
+        // Channel once, on the first message seen in it, not on every message.
+        const rostered = new Set();
         for (const row of rows) {
+            const chatId = row.chat_id;
+            const group = groups[chatId];
+            const groupRoster = group !== undefined && !rostered.has(chatId);
+            if (groupRoster)
+                rostered.add(chatId);
             const attributedBody = this.processAttributedBody(row.attributedBody);
             const timestamp = iosToUnixTimestamp(Number.parseInt(row.date, 10));
             const participants = chats[row.chat_id] || [];
@@ -52,11 +95,62 @@ export class ImessageExtractor extends SqliteExtractor {
                 handleIdText: row.handle_id_text,
                 participants,
                 attachments: messageAttachments,
+                group,
+                groupRoster,
                 myIcloudAccount: realIcloudAccount,
                 includeContactNames: this.config.includeContactNames,
             });
         }
     }
+    /** The messages a run reads, one per chat they're in, and when, from three aggregates. */
+    async survey(options = {}) {
+        const { conditions, values } = this.messageFilter();
+        const inRange = conditions.map(condition => ` AND ${condition}`).join('');
+        // A message's chats, as the run joins them, and only those in range.
+        const read = `SELECT 1 FROM message AS m INNER JOIN chat_message_join AS cm ON m.ROWID = cm.message_id`;
+        const messages = this.db.prepare(`SELECT COUNT(*) AS n, MIN(m.date) AS first, MAX(m.date) AS last
+      FROM message AS m INNER JOIN chat_message_join AS cm ON m.ROWID = cm.message_id
+      WHERE 1${inRange}`);
+        messages.setReadBigInts(true);
+        const { n, first, last } = messages.get(...values);
+        const { chats } = this.db.prepare(`SELECT COUNT(*) AS chats FROM chat WHERE EXISTS (${read} WHERE cm.chat_id = chat.ROWID${inRange})`).get(...values);
+        const { bytes } = this.db.prepare(`SELECT COALESCE(SUM(a.total_bytes), 0) AS bytes
+      FROM message_attachment_join AS maj INNER JOIN attachment AS a ON maj.attachment_id = a.ROWID
+      WHERE ${this.attachmentFilter()} AND EXISTS (${read} WHERE m.ROWID = maj.message_id${inRange})`).get(...values);
+        const at = (date) => date === null ? undefined : new Date(iosToUnixTimestamp(Number(date)));
+        const count = Number(n);
+        const month = monthSql(`m.date / 1000000000 + ${APPLE_EPOCH_OFFSET_SECONDS}`);
+        const byMonth = options.byMonth &&
+            countsByMonth(this.db.prepare(`SELECT ${month} AS month, COUNT(*) AS n
+          FROM message AS m INNER JOIN chat_message_join AS cm ON m.ROWID = cm.message_id
+          WHERE 1${inRange} GROUP BY month`).all(...values));
+        return {
+            messages: {
+                count,
+                ...(count > 0 && { earliest: at(first), latest: at(last) }),
+                ...(byMonth && { byMonth }),
+                measures: [
+                    { key: 'conversations', label: 'conversations', value: chats },
+                    { key: 'attachmentBytes', label: 'attachments', value: bytes, unit: 'bytes' },
+                ],
+            },
+        };
+    }
+    /**
+     * Messages within since and until. chat.db stores message.date as nanoseconds
+     * since the Apple epoch. A system event, such as a group being renamed or
+     * someone joining it, is a row too, with no text and often no sender, so
+     * only rows that are messages (`item_type` 0) are read.
+     */
+    messageFilter() {
+        const filter = timeRangeConditions('m.date', this.config, {
+            convert: date => (BigInt(date.getTime()) * 1000000n -
+                BigInt(APPLE_EPOCH_OFFSET_SECONDS) * 1000000000n).toString(),
+        });
+        if (this.has('message.item_type'))
+            filter.conditions.push('m.item_type = 0');
+        return filter;
+    }
     queryMessages() {
         let sql = `
       SELECT
@@ -67,11 +161,7 @@ export class ImessageExtractor extends SqliteExtractor {
         message AS m
         LEFT OUTER JOIN handle AS h ON m.handle_id = h.ROWID
         INNER JOIN chat_message_join AS cm ON m.ROWID = cm.message_id`;
-        // chat.db stores message.date as nanoseconds since the Apple epoch
-        const { conditions, values } = timeRangeConditions('m.date', this.config, {
-            convert: date => (BigInt(date.getTime()) * 1000000n -
-                BigInt(APPLE_EPOCH_OFFSET_SECONDS) * 1000000000n).toString(),
-        });
+        const { conditions, values } = this.messageFilter();
         if (conditions.length > 0) {
             sql += ` WHERE ${conditions.join(' AND ')}`;
         }
@@ -118,6 +208,39 @@ export class ImessageExtractor extends SqliteExtractor {
         }
         return chats;
     }
+    /**
+     * The group chats, by chat, with their names and Apple's IDs for them. An
+     * older Messages without a column has none of what it holds.
+     */
+    loadGroups() {
+        const rows = this.db.prepare(`SELECT ROWID AS chat_id, ${this.chatColumn('display_name')} AS name,
+        ${this.chatColumn('group_id')} AS group_id,
+        ${this.chatColumn('original_group_id')} AS original_group_id
+      FROM chat WHERE style = ?`).all(GROUP_CHAT_STYLE);
+        const groups = {};
+        for (const row of rows) {
+            groups[row.chat_id] = {
+                name: row.name?.trim() || null,
+                groupId: row.group_id,
+                originalGroupId: row.original_group_id,
+            };
+        }
+        return groups;
+    }
+    /** A column of `chat` the schema marks optional, or NULL when this Messages has none. */
+    chatColumn(name) {
+        return this.has(`chat.${name}`) ? name : 'NULL';
+    }
+    /**
+     * The attachments a run reads: files Messages shows. It keeps others hidden,
+     * such as the images and icon of a link preview, which no one sent.
+     */
+    attachmentFilter() {
+        const shown = this.has('attachment.hide_attachment')
+            ? ' AND COALESCE(a.hide_attachment, 0) = 0'
+            : '';
+        return `a.filename IS NOT NULL${shown}`;
+    }
     loadAttachments() {
         const sql = `
       SELECT 
@@ -132,7 +255,7 @@ export class ImessageExtractor extends SqliteExtractor {
         a.is_outgoing
       FROM message_attachment_join AS maj
       INNER JOIN attachment AS a ON maj.attachment_id = a.ROWID
-      WHERE a.filename IS NOT NULL
+      WHERE ${this.attachmentFilter()}
     `;
         const stmt = this.db.prepare(sql);
         const results = stmt.all();

package/dist/connectors/ImessageTransformer.d.ts

@@ -5,6 +5,14 @@ export default class ImessageTransformer extends ChronicleTransformer {
     transform(record: Record): Promise<ActionAndChildren[]>;
     private buildMessageAction;
     private buildMessage;
+    /**
+     * The group chat a message is in, keyed by Apple's ID for the group, as
+     * `groupChatChannel` keys it for every plugin. Its members are the chat's
+     * roster and you: chat.db's roster lists only the others, but every group
+     * chat in it is one you're in. They go on it from the first message the
+     * extractor saw in the chat, as WhatsApp does, rather than from every message.
+     */
+    private buildChannel;
     private buildParticipants;
     /**
      * One chat participant as an Agent in the service's identity namespace

package/dist/connectors/ImessageTransformer.js

@@ -1,5 +1,5 @@
 import { ChronicleTransformer, normalizePhoneNumber, createMediaObject, } from '@chronicle.app/etl';
-import { lookupContact, buildICloudPersonSchema } from '@chronicle.app/icloud';
+import { lookupContact, buildICloudPersonSchema, groupChatChannel, } from '@chronicle.app/apple';
 import * as fs from 'node:fs';
 import * as path from 'node:path';
 export default class ImessageTransformer extends ChronicleTransformer {
@@ -20,14 +20,15 @@ export default class ImessageTransformer extends ChronicleTransformer {
             '@type': 'MessageAction',
             timestamp: new Date(record.context.timestamp),
             source: service,
-            sourceId: record.context.guid, // Use GUID instead of ROWID
-            '@key': ['source', 'sourceId', '@type'],
-            object: this.buildMessage(record, recipients),
+            // Keyed by the message it sent.
+            '@key': ['@type', 'source', 'result.sourceId'],
+            // Sending brings the message into being, so it's the action's result.
+            result: this.buildMessage(record, recipients, this.buildChannel(record, participants, myIdentity)),
             ...(agent && { agent }),
         };
         return [obj];
     }
-    buildMessage(record, recipients) {
+    buildMessage(record, recipients, channel) {
         const messageSource = this.getServiceSource(record.context.service);
         const message = {
             '@type': 'Message',
@@ -40,6 +41,10 @@ export default class ImessageTransformer extends ChronicleTransformer {
         if (recipients.length > 0) {
             message.recipient = recipients;
         }
+        // A group chat's message is part of its Channel. A one-to-one chat has none.
+        if (channel) {
+            message.isPartOf = [channel];
+        }
         // Add attachments using contains property (following Bluesky pattern)
         const attachments = record.context.attachments || [];
         if (attachments.length > 0) {
@@ -47,6 +52,25 @@ export default class ImessageTransformer extends ChronicleTransformer {
         }
         return message;
     }
+    /**
+     * The group chat a message is in, keyed by Apple's ID for the group, as
+     * `groupChatChannel` keys it for every plugin. Its members are the chat's
+     * roster and you: chat.db's roster lists only the others, but every group
+     * chat in it is one you're in. They go on it from the first message the
+     * extractor saw in the chat, as WhatsApp does, rather than from every message.
+     */
+    buildChannel(record, participants, myIdentity) {
+        const { group, groupRoster } = record.context;
+        const channel = group ? groupChatChannel(group) : null;
+        if (!channel)
+            return null;
+        if (group.name)
+            channel.name = group.name;
+        const members = [...participants, ...(myIdentity ? [myIdentity] : [])];
+        if (groupRoster && members.length > 0)
+            channel.member = members;
+        return channel;
+    }
     buildParticipants(record) {
         const agentSource = this.getAgentSource(record.context.service);
         const chatParticipants = record.context.participants || [];
@@ -163,6 +187,9 @@ export default class ImessageTransformer extends ChronicleTransformer {
         const { handleIdText } = record.context;
         const agent = participants.find(p => p.handle === handleIdText);
         if (!agent) {
+            // A sender whose handle is gone from chat.db is unknown.
+            if (!handleIdText)
+                return { agent: undefined, recipients: myIdentity ? [myIdentity] : [] };
             // The sender is not among the chat's participants: build them from the
             // message's own handle, the same way as any other participant.
             const agentSource = this.getAgentSource(record.context.service);
@@ -204,7 +231,7 @@ export default class ImessageTransformer extends ChronicleTransformer {
                 const filePath = this.resolveAttachmentPath(attachment.filename);
                 // Check if file exists
                 if (!fs.existsSync(filePath)) {
-                    console.warn(`Attachment file not found: ${filePath}`);
+                    this.logger.warn('Attachment file not found', { file: filePath });
                     continue;
                 }
                 // MIME drives the media object @type (ImageObject vs DocumentObject…).
@@ -228,7 +255,10 @@ export default class ImessageTransformer extends ChronicleTransformer {
                 mediaObjects.push(mediaObject);
             }
             catch (error) {
-                console.warn(`Failed to process attachment ${attachment.filename}:`, error);
+                this.logger.warn('Failed to process attachment', {
+                    file: attachment.filename,
+                    error: String(error),
+                });
             }
         }
         return mediaObjects;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@chronicle.app/imessage",
-  "version": "0.3.0",
+  "version": "0.5.0",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
   "files": [
@@ -19,9 +19,9 @@
   "type": "module",
   "description": "Chronicle imessage extraction plugin",
   "devDependencies": {
-    "@chronicle.app/etl": "0.3.0",
-    "@chronicle.app/schema": "0.3.0",
-    "@chronicle.app/tsconfig": "0.3.0",
+    "@chronicle.app/etl": "0.5.0",
+    "@chronicle.app/schema": "0.5.0",
+    "@chronicle.app/tsconfig": "0.5.0",
     "@types/node": "^22.10.1",
     "typescript": "^5.7.2"
   },
@@ -36,8 +36,8 @@
     "registry": "https://registry.npmjs.org/"
   },
   "dependencies": {
-    "@chronicle.app/etl-sqlite": "0.3.0",
-    "@chronicle.app/icloud": "0.3.0",
+    "@chronicle.app/etl-sqlite": "0.5.0",
+    "@chronicle.app/apple": "0.5.0",
     "zod": "^3.24.1"
   },
   "peerDependencies": {
@@ -63,7 +63,64 @@
             "recordTypes": [
               "messages"
             ],
-            "default": true
+            "default": true,
+            "locations": [
+              {
+                "path": "~/Library/Messages/chat.db"
+              }
+            ],
+            "databases": [
+              {
+                "tables": {
+                  "message": [
+                    "ROWID",
+                    "guid",
+                    "text",
+                    "service",
+                    "date",
+                    "is_from_me",
+                    "handle_id",
+                    "attributedBody",
+                    "item_type?"
+                  ],
+                  "handle": [
+                    "ROWID",
+                    "id",
+                    "service"
+                  ],
+                  "chat": [
+                    "ROWID",
+                    "style",
+                    "display_name?",
+                    "group_id?",
+                    "original_group_id?"
+                  ],
+                  "chat_message_join": [
+                    "chat_id",
+                    "message_id"
+                  ],
+                  "chat_handle_join": [
+                    "chat_id",
+                    "handle_id"
+                  ],
+                  "message_attachment_join": [
+                    "message_id",
+                    "attachment_id"
+                  ],
+                  "attachment": [
+                    "ROWID",
+                    "guid",
+                    "filename",
+                    "mime_type",
+                    "uti",
+                    "total_bytes",
+                    "transfer_name",
+                    "is_outgoing",
+                    "hide_attachment?"
+                  ]
+                }
+              }
+            ]
           }
         },
         "platforms": [
@@ -73,6 +130,11 @@
           "full-disk-access"
         ]
       }
+    },
+    "deepLinks": {
+      "imessage": {
+        "Message": "messages://open?message-guid={sourceId}"
+      }
     }
   }
 }