@chronicle.app/gmail 0.0.0-stage → 0.4.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Andrew Louis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,64 @@
-# Temporary Holding Version
+# @chronicle.app/gmail
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Your Gmail: messages, the threads Gmail groups them in, and their labels. Read live through the [Gmail API](https://developers.google.com/workspace/gmail/api/reference/rest), or from the mbox in a [Google Takeout](https://takeout.google.com).
+
+## Usage
+
+```sh
+chronicle auth login google
+chronicle extract gmail
+chronicle extract gmail --sent --since 2025-01-01
+chronicle extract gmail --label Work,Starred
+chronicle extract gmail --query "from:boss@example.com has:attachment"
+chronicle extract gmail --strategy takeout --input "All mail Including Spam and Trash.mbox" --sent
+```
+
+Sign in first: see [@chronicle.app/google](../../core/google/README.md), which walks you through making your own Google client. `--account` picks one of several signed-in accounts.
+
+The API reads newest first. A Takeout reads in the order the mbox has, and needs no sign-in.
+
+### What to extract
+
+- `--sent`: only mail you sent.
+- `--label Work,Starred`: only mail with all of these labels, yours or Gmail's own (Inbox, Starred, Important). Spam and Trash are left out unless you name them: `--label Spam`.
+- `--query "<search>"`: any Gmail search, as in Gmail's search box: `from:`, `to:`, `has:attachment`, `-in:sent`. It needs the API.
+- `--since` and `--until` narrow by when the mail was sent, with either.
+
+## Schema
+
+Messages are built by [@chronicle.app/email-core](../../core/email/README.md), the same way the email plugin builds them from an mbox, with Gmail's thread and labels added.
+
+| Gmail thing       | Chronicle node  | Key                                                                                               | Properties                                                                                                                                                                                                         |
+| ----------------- | --------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Sending a message | `MessageAction` | `@type`, `sourceId` (the Message-ID)                                                              | `agent` (the sender), `object` (the `Message`), `timestamp` (the Date header)                                                                                                                                      |
+| A message         | `Message`       | `@type`, `sourceId` (the Message-ID)                                                              | `inAccount` (the account at the mailbox's address, from the API), `name` (subject), `body` (what was written, as text or Markdown), `author`, `recipient`, `inReplyTo`, `isPartOf` (its `Thread`), `tags` (labels) |
+| A conversation    | `Thread`        | `@type`, `source`, `inAccount` (the Google account, from the API), `sourceId` (Gmail's thread ID) |                                                                                                                                                                                                                    |
+| A person          | `Agent`         | `@type`, `source` (`email`), `handle` (their address, lowercased)                                 | `alternateName` (each display name the mail gave it); you are `sameAs` `@me`, and your Google account from the API                                                                                                 |
+
+`SHAPES.md` sketches the output, generated from the tests.
+
+### Decisions
+
+**A message is its Message-ID, wherever it's read.** A message and the sending of it are keyed by its Message-ID, without the angle brackets around it, in the `email` namespace, the protocol's own, so the same message from the API, a Takeout, another mailbox, or an mbox is one node. The message is `inAccount` the account at the mailbox's address, an Agent keyed by that address in the `email` namespace, as any mail source of the same mailbox can say, so "messages in this account" is one query; a message in several of your accounts is in each. Gmail's own message ID isn't kept: it's only unique within the account (it's the time the message arrived there, and a counter), and the Message-ID already identifies the message. A thread is Gmail's: the Google account issued its ID, which is only unique within it, so the thread is `inAccount` the Google account (an Agent keyed by the address in the `google-account` namespace) and keyed within it. A person is their lowercased address in the `email` namespace, the identity other plugins link an address to, so a person seen in Gmail, any other mail, or a calendar is one node. A message without a Message-ID is identified by its sender, date, and subject.
+
+**Threads link to Gmail, in their account.** The plugin declares a `deepLinks` template for its `Thread` nodes, `https://mail.google.com/mail/?authuser=<address>#all/<thread ID>`, so a host can open the conversation in Gmail. `authuser` picks the signed-in Google account by its address, so with several signed in it opens in the right one. A thread from a Takeout has no account to fill it with.
+
+**Threads and labels come from Gmail either way.** The API gives a message's thread ID and label IDs; a Takeout writes the same thread ID in decimal (`X-GM-THRID`) and the labels by name (`X-Gmail-Labels`). Both give the same thread ID and the same label names, as Gmail shows them. Read state (Unread, Opened) changes as you read, so it isn't kept as a label.
+
+**A Takeout names no account.** The mbox doesn't say which account it was exported from. A message's `Delivered-To` can be an alias, and mail you sent says only the address you sent it from, which can be a send-as address, so neither is the account. So a Takeout's messages are in no account, its threads are keyed by Gmail's thread ID alone, and you aren't linked to a Google account or to your contacts. Its messages are the API's messages, but its threads aren't the API's threads.
+
+**A body is what was written, not what was sent.** It's plain text or Markdown, never HTML: the text part, or the HTML part converted to Markdown. Quoted history, signatures, newsletter footers, images, and tracking parameters are left out; see [@chronicle.app/email-core](../../core/email/README.md).
+
+**Your contacts can link people.** With `--link-contacts` (and `chronicle auth login google --add contacts`), a person on one of your contacts is `sameAs` your contact: the entry in your Google account's contacts, keyed by its ID there and named as you saved it, which is `sameAs` the contact's other addresses and phone numbers, so the person who emails you is the person who texts or calls you. A Takeout doesn't: it can't say whose contacts they'd be.
+
+**A draft isn't a message.** Drafts are left out, from the API and a Takeout alike: they were never sent.
+
+**A display name is what a message called someone.** The name in a `From`, `To`, or `Cc` header is chosen by the sender or their address book, and a group or notification address carries a different one in nearly every message. So it's an `alternateName` of the address, never its `name`, and an address gathers one for each name it was shown with. A person's `name` comes from a source that keeps a record of them: the name you saved for a contact is that contact's `name`.
+
+**Mail in Sent is yours.** Its sender is `sameAs` `@me`, whatever address it came from. Gmail also says the mailbox's own address (the API's profile, or the `Delivered-To` header in a Takeout), so you are `@me` as a recipient too.
+
+**You are your Google account.** Wherever the API has you as `@me`, you are also `sameAs` the Google account the mailbox is: an Agent keyed by the mailbox's address in the `google-account` namespace, the node Google Calendar and Chrome link you to. A Takeout names no account, so it doesn't link you to one.
+
+## Tests
+
+The tests run against a fake Gmail API on 127.0.0.1 and a synthetic Takeout mbox in a temporary directory, both built from the same made-up messages, and pass the access token directly. They never read stored credentials or reach Google.

package/dist/connectors/GmailApiExtractor.d.ts

@@ -0,0 +1,76 @@
+import { Extractor, Record } from '@chronicle.app/etl';
+import { z } from 'zod';
+import type { GmailRecord } from '../types.js';
+import GmailTransformer from './GmailTransformer.js';
+/**
+ * Your Gmail, newest first, through the Gmail API. Each message is fetched
+ * whole and parsed like any email, with Gmail's thread and labels.
+ */
+export declare class GmailApiExtractor extends Extractor<typeof GmailApiExtractor> {
+    static source: string;
+    static description: string;
+    static delivery: "api";
+    static strategy: string;
+    static recordTypes: string[];
+    static default: boolean;
+    static newestFirst: boolean;
+    static defaultTransformer: typeof GmailTransformer;
+    /** Swapped for a fake server in tests. */
+    static apiBaseURL: string;
+    /**
+     * The least time a batch takes. Fetching a message costs 5 of the 6,000
+     * units Gmail allows a user each minute, so a batch of 10 every half second
+     * stays inside it however long a run goes. Tests set it to 0.
+     */
+    static batchIntervalMs: number;
+    static schema: z.ZodObject<{
+        since: z.ZodOptional<z.ZodDate>;
+        until: z.ZodOptional<z.ZodDate>;
+        limit: z.ZodOptional<z.ZodNumber>;
+    } & {
+        input: z.ZodOptional<z.ZodString>;
+        sent: z.ZodOptional<z.ZodBoolean>;
+        label: z.ZodOptional<z.ZodString>;
+        query: z.ZodOptional<z.ZodString>;
+        linkContacts: z.ZodOptional<z.ZodBoolean>;
+        account: z.ZodOptional<z.ZodString>;
+        accessToken: z.ZodOptional<z.ZodString>;
+    }, "strip", z.ZodTypeAny, {
+        account?: string | undefined;
+        input?: string | undefined;
+        since?: Date | undefined;
+        until?: Date | undefined;
+        limit?: number | undefined;
+        sent?: boolean | undefined;
+        label?: string | undefined;
+        query?: string | undefined;
+        linkContacts?: boolean | undefined;
+        accessToken?: string | undefined;
+    }, {
+        account?: string | undefined;
+        input?: string | undefined;
+        since?: Date | undefined;
+        until?: Date | undefined;
+        limit?: number | undefined;
+        sent?: boolean | undefined;
+        label?: string | undefined;
+        query?: string | undefined;
+        linkContacts?: boolean | undefined;
+        accessToken?: string | undefined;
+    }>;
+    private api;
+    private contacts;
+    private labelNames;
+    private owner;
+    keyOf(record: Record): string | null;
+    occurredAt(record: Record): Date | undefined;
+    private get options();
+    setup(): Promise<void>;
+    determineCount(): Promise<number | null>;
+    extract(): AsyncGenerator<Record>;
+    /** The label IDs every message must have: `--label`, and Sent for `--sent`. */
+    private labelIds;
+    private recordOf;
+    recordToString(data?: GmailRecord): string;
+}
+//# sourceMappingURL=GmailApiExtractor.d.ts.map

package/dist/connectors/GmailApiExtractor.js

@@ -0,0 +1,186 @@
+import { Extractor } from '@chronicle.app/etl';
+import { ContactDirectory, GoogleApi, contactOptions, googleAccountOptions, } from '@chronicle.app/google';
+import { identityOf, parseMessage } from '@chronicle.app/email-core';
+import { EXIT_CODES, ExtractorError } from '@chronicle.app/logging';
+import { z } from 'zod';
+import { filterOptions, gmailSearch, includesSpamTrash, narrows, wantedLabels, } from '../filters.js';
+import { labelId, labelName } from '../labels.js';
+import GmailTransformer from './GmailTransformer.js';
+/** Messages fetched at once. */
+const CONCURRENCY = 10;
+/**
+ * Your Gmail, newest first, through the Gmail API. Each message is fetched
+ * whole and parsed like any email, with Gmail's thread and labels.
+ */
+export class GmailApiExtractor extends Extractor {
+    static source = 'gmail';
+    static description = 'Messages in your Gmail';
+    static delivery = 'api';
+    static strategy = 'api';
+    static recordTypes = ['messages'];
+    static default = true;
+    static newestFirst = true;
+    static defaultTransformer = GmailTransformer;
+    /** Swapped for a fake server in tests. */
+    static apiBaseURL = 'https://gmail.googleapis.com/gmail/v1';
+    /**
+     * The least time a batch takes. Fetching a message costs 5 of the 6,000
+     * units Gmail allows a user each minute, so a batch of 10 every half second
+     * stays inside it however long a run goes. Tests set it to 0.
+     */
+    static batchIntervalMs = 500;
+    static schema = Extractor.schema.extend({
+        ...googleAccountOptions,
+        ...contactOptions,
+        ...filterOptions,
+        // A Takeout's; listed here too, since a source's flags are shared.
+        input: z.string().optional().describe('Path to the Gmail mbox from Google Takeout'),
+    });
+    api;
+    contacts = ContactDirectory.empty;
+    labelNames = new Map();
+    owner = null;
+    keyOf(record) {
+        return identityOf(record.data.mail);
+    }
+    occurredAt(record) {
+        const { mail, gmail } = record.data;
+        const at = gmail.receivedAt ?? mail.date;
+        return at ? new Date(at) : undefined;
+    }
+    get options() {
+        return this.config;
+    }
+    async setup() {
+        await super.setup();
+        const { account, accessToken } = this.options;
+        this.api = new GoogleApi({
+            service: 'gmail',
+            baseURL: GmailApiExtractor.apiBaseURL,
+            account,
+            accessToken,
+        });
+        await this.api.initialize();
+        const [profile, labels] = await Promise.all([
+            this.api.get('/users/me/profile'),
+            this.api.get('/users/me/labels'),
+        ]);
+        this.owner = profile.emailAddress ?? null;
+        for (const label of labels.labels ?? [])
+            this.labelNames.set(label.id, label.name);
+        const { directory, missing } = await ContactDirectory.load({
+            account,
+            accessToken,
+            linkContacts: this.options.linkContacts,
+        });
+        this.contacts = directory;
+        if (missing) {
+            this.hint('Your contacts aren’t linked to these people', {
+                action: 'Run `chronicle auth login google --add contacts` to link them.',
+            });
+        }
+    }
+    async determineCount() {
+        // Only the whole mailbox has a count without listing it.
+        if (narrows(this.options))
+            return null;
+        const profile = await this.api.get('/users/me/profile');
+        const total = profile.messagesTotal ?? null;
+        const limit = this.getEffectiveLimit();
+        return total !== null && limit !== null ? Math.min(total, limit) : total;
+    }
+    async *extract() {
+        const ids = this.api.pages('/users/me/messages', {
+            q: gmailSearch(this.options),
+            labelIds: this.labelIds(),
+            includeSpamTrash: includesSpamTrash(this.options),
+            maxResults: 500,
+        }, 'messages');
+        let count = 0;
+        let unidentified = 0;
+        let batch = [];
+        // Fetched a batch at a time, and yielded in the order Gmail listed them.
+        const flush = async function* () {
+            const started = Date.now();
+            // A message deleted between the listing and the fetch is just gone.
+            const messages = await Promise.all(batch.map(id => this.api.get(`/users/me/messages/${id}`, { format: 'raw' }).catch(error => {
+                if (error?.code === 'not-found')
+                    return null;
+                throw error;
+            })));
+            const wait = GmailApiExtractor.batchIntervalMs - (Date.now() - started);
+            if (wait > 0)
+                await new Promise(resolve => setTimeout(resolve, wait));
+            batch = [];
+            for (const message of messages) {
+                // A draft hasn't been sent: it isn't a message yet.
+                if (!message || message.labelIds?.includes('DRAFT'))
+                    continue;
+                const record = await this.recordOf(message);
+                if (!record) {
+                    unidentified++;
+                    continue;
+                }
+                yield this.createRecord(record, { recordType: 'messages' });
+                if (this.shouldStopExtracting(++count))
+                    return true;
+            }
+            return false;
+        }.bind(this);
+        try {
+            for await (const { id } of ids) {
+                batch.push(id);
+                if (batch.length < CONCURRENCY)
+                    continue;
+                if (yield* flush())
+                    return;
+            }
+            if (batch.length > 0)
+                yield* flush();
+        }
+        finally {
+            if (unidentified > 0) {
+                this.logger.warn(`Skipped ${unidentified} message(s) with no Message-ID and no From + Date to key on`);
+            }
+        }
+    }
+    /** The label IDs every message must have: `--label`, and Sent for `--sent`. */
+    labelIds() {
+        const ids = wantedLabels(this.options).map(name => {
+            const id = labelId(name, this.labelNames);
+            if (!id) {
+                throw new ExtractorError(`Gmail has no label "${name}"`, {
+                    code: 'unknown-label',
+                    exitCode: EXIT_CODES.usage,
+                    hint: `Your labels are ${[...this.labelNames.values()].sort().join(', ')}.`,
+                });
+            }
+            return id;
+        });
+        return ids.length > 0 ? ids : undefined;
+    }
+    async recordOf(message) {
+        const mail = await parseMessage(Buffer.from(message.raw, 'base64url'));
+        // Nothing real to identify it by: a made-up id would mint a new message each run.
+        if (identityOf(mail) === null)
+            return null;
+        return {
+            mail,
+            gmail: {
+                id: message.id,
+                threadId: message.threadId,
+                labels: (message.labelIds ?? [])
+                    .map(id => labelName(id, this.labelNames))
+                    .filter((name) => name !== undefined),
+                receivedAt: new Date(Number(message.internalDate)).toISOString(),
+                // The profile's address is the account the API read, and yours.
+                owner: this.owner,
+                account: this.owner,
+            },
+            contacts: this.contacts.linksFor([mail.from, ...mail.to, ...mail.cc, ...mail.bcc].flatMap(person => person ? [person.address] : [])),
+        };
+    }
+    recordToString(data) {
+        return data?.mail?.subject || 'gmail.messages';
+    }
+}

package/dist/connectors/GmailTakeoutExtractor.d.ts

@@ -0,0 +1,52 @@
+import { Extractor, Record } from '@chronicle.app/etl';
+import { z } from 'zod';
+import type { GmailRecord } from '../types.js';
+import GmailTransformer from './GmailTransformer.js';
+/**
+ * Your Gmail from a Google Takeout: the mbox it exports (`All mail Including
+ * Spam and Trash.mbox`). Each message keeps the thread and labels Gmail wrote
+ * into its `X-GM-THRID` and `X-Gmail-Labels` headers, so it becomes the same
+ * record the API makes.
+ */
+export declare class GmailTakeoutExtractor extends Extractor<typeof GmailTakeoutExtractor> {
+    static source: string;
+    static description: string;
+    static delivery: "export";
+    static strategy: string;
+    static recordTypes: string[];
+    static defaultTransformer: typeof GmailTransformer;
+    static schema: z.ZodObject<{
+        since: z.ZodOptional<z.ZodDate>;
+        until: z.ZodOptional<z.ZodDate>;
+        limit: z.ZodOptional<z.ZodNumber>;
+    } & {
+        input: z.ZodOptional<z.ZodString>;
+        sent: z.ZodOptional<z.ZodBoolean>;
+        label: z.ZodOptional<z.ZodString>;
+        query: z.ZodOptional<z.ZodString>;
+    }, "strip", z.ZodTypeAny, {
+        input?: string | undefined;
+        since?: Date | undefined;
+        until?: Date | undefined;
+        limit?: number | undefined;
+        sent?: boolean | undefined;
+        label?: string | undefined;
+        query?: string | undefined;
+    }, {
+        input?: string | undefined;
+        since?: Date | undefined;
+        until?: Date | undefined;
+        limit?: number | undefined;
+        sent?: boolean | undefined;
+        label?: string | undefined;
+        query?: string | undefined;
+    }>;
+    keyOf(record: Record): string | null;
+    occurredAt(record: Record): Date | undefined;
+    private get options();
+    private get input();
+    determineCount(): Promise<number | null>;
+    extract(): AsyncGenerator<Record>;
+    recordToString(data?: GmailRecord): string;
+}
+//# sourceMappingURL=GmailTakeoutExtractor.d.ts.map

package/dist/connectors/GmailTakeoutExtractor.js

@@ -0,0 +1,118 @@
+import { Extractor, InputNotFound } from '@chronicle.app/etl';
+import { countMbox, identityOf, parseMessage, readMbox } from '@chronicle.app/email-core';
+import { EXIT_CODES, ExtractorError } from '@chronicle.app/logging';
+import { z } from 'zod';
+import { filterOptions, matches, narrows } from '../filters.js';
+import { hexId, takeoutLabels } from '../labels.js';
+import GmailTransformer from './GmailTransformer.js';
+/**
+ * Your Gmail from a Google Takeout: the mbox it exports (`All mail Including
+ * Spam and Trash.mbox`). Each message keeps the thread and labels Gmail wrote
+ * into its `X-GM-THRID` and `X-Gmail-Labels` headers, so it becomes the same
+ * record the API makes.
+ */
+export class GmailTakeoutExtractor extends Extractor {
+    static source = 'gmail';
+    static description = 'Messages in a Google Takeout of your Gmail';
+    static delivery = 'export';
+    static strategy = 'takeout';
+    static recordTypes = ['messages'];
+    static defaultTransformer = GmailTransformer;
+    // `input` is checked when it's read rather than by the schema: a source's
+    // flags are shared, so a schema-required flag here would also gate the API.
+    static schema = Extractor.schema.extend({
+        ...filterOptions,
+        input: z.string().optional().describe('Path to the Gmail mbox from Google Takeout'),
+    });
+    keyOf(record) {
+        return identityOf(record.data.mail);
+    }
+    occurredAt(record) {
+        const { date } = record.data.mail;
+        return date ? new Date(date) : undefined;
+    }
+    get options() {
+        return this.config;
+    }
+    get input() {
+        if (this.options.query) {
+            throw new ExtractorError('A Gmail search needs the API', {
+                code: 'unsupported-flag',
+                exitCode: EXIT_CODES.usage,
+                hint: 'Use `--label` or `--sent` with a Takeout instead.',
+            });
+        }
+        if (!this.options.input) {
+            throw new InputNotFound('No Takeout mbox given', {
+                hint: 'Pass the Gmail mbox from your Takeout with `--input <path>`.',
+            });
+        }
+        return this.options.input;
+    }
+    async determineCount() {
+        if (narrows(this.options))
+            return null;
+        try {
+            const total = await countMbox(this.input);
+            const limit = this.getEffectiveLimit();
+            return limit === null ? total : Math.min(total, limit);
+        }
+        catch {
+            return null;
+        }
+    }
+    async *extract() {
+        let count = 0;
+        let unidentified = 0;
+        let undated = 0;
+        for await (const { raw, envelopeDate } of readMbox(this.input)) {
+            const mail = await parseMessage(raw);
+            // No Date header: when its `From ` line says Gmail received it.
+            mail.date ??= envelopeDate;
+            if (!mail.date) {
+                undated++;
+                continue;
+            }
+            if (identityOf(mail) === null) {
+                unidentified++;
+                continue;
+            }
+            const labels = takeoutLabels(mail.headers['x-gmail-labels']);
+            // A draft hasn't been sent: it isn't a message yet.
+            if (labels.includes('Drafts') || !matches(mail, labels, this.options))
+                continue;
+            const record = {
+                mail,
+                gmail: {
+                    id: hexId(mail.headers['x-gm-msgid']),
+                    threadId: hexId(mail.headers['x-gm-thrid']),
+                    labels,
+                    receivedAt: envelopeDate,
+                    // Gmail writes the mailbox's own address on every message it
+                    // delivers; a message you sent has none, but it's from you.
+                    owner: addressIn(mail.headers['delivered-to']) ??
+                        (labels.includes('Sent') ? (mail.from?.address.toLowerCase() ?? null) : null),
+                    // Those addresses can be aliases or send-as addresses, not the
+                    // account, and nothing in the mbox names the account it came from.
+                    account: null,
+                },
+            };
+            yield this.createRecord(record, { recordType: 'messages' });
+            if (this.shouldStopExtracting(++count))
+                break;
+        }
+        if (undated > 0) {
+            this.logger.warn(`Skipped ${undated} message(s) with no date to place them by`);
+        }
+        if (unidentified > 0) {
+            this.logger.warn(`Skipped ${unidentified} message(s) with no Message-ID and no From + Date to key on`);
+        }
+    }
+    recordToString(data) {
+        return data?.mail?.subject || 'gmail.messages';
+    }
+}
+/** The address in a header like `Delivered-To: me@example.com`. */
+function addressIn(header) {
+    return header?.match(/[^\s<>,"]+@[^\s<>,"]+/)?.[0]?.toLowerCase() ?? null;
+}

package/dist/connectors/GmailTransformer.d.ts

@@ -0,0 +1,16 @@
+import { ChronicleTransformer, Record } from '@chronicle.app/etl';
+import { ActionAndChildren } from '@chronicle.app/schema';
+/**
+ * A Gmail message is keyed by its Message-ID, so copies in other mailboxes
+ * and mbox files are one message. When the extractor names the account (the
+ * API does, a Takeout doesn't), the message is `inAccount` its address, as
+ * any mail source of the same mailbox says, and you are `sameAs` the Google
+ * account at that address. It `isPartOf` its `Thread`, whose ID the Google
+ * account issued, so the thread is `inAccount` it and keyed within it. Its
+ * labels are its `tags`. Mail in Sent is yours. A person on one of your
+ * contacts is `sameAs` the contact's other addresses and phone numbers.
+ */
+export default class GmailTransformer extends ChronicleTransformer {
+    transform(record: Record): Promise<ActionAndChildren[]>;
+}
+//# sourceMappingURL=GmailTransformer.d.ts.map

package/dist/connectors/GmailTransformer.js

@@ -0,0 +1,52 @@
+import { ChronicleTransformer } from '@chronicle.app/etl';
+import { contactIdentities, googleAccount } from '@chronicle.app/google';
+import { emailAccount, messageAction } from '@chronicle.app/email-core';
+/**
+ * A Gmail message is keyed by its Message-ID, so copies in other mailboxes
+ * and mbox files are one message. When the extractor names the account (the
+ * API does, a Takeout doesn't), the message is `inAccount` its address, as
+ * any mail source of the same mailbox says, and you are `sameAs` the Google
+ * account at that address. It `isPartOf` its `Thread`, whose ID the Google
+ * account issued, so the thread is `inAccount` it and keyed within it. Its
+ * labels are its `tags`. Mail in Sent is yours. A person on one of your
+ * contacts is `sameAs` the contact's other addresses and phone numbers.
+ */
+export default class GmailTransformer extends ChronicleTransformer {
+    async transform(record) {
+        if (record.extraction.recordType !== 'messages')
+            return [];
+        const { mail, gmail, contacts } = record.data;
+        // The account at the mailbox's address, which any mail source can name,
+        // and the Google account, which issues Gmail's thread IDs: they're only
+        // unique within it, so they're keyed in it.
+        const account = gmail.account ? emailAccount(gmail.account) : undefined;
+        const google = gmail.account ? googleAccount(gmail.account) : undefined;
+        const thread = gmail.threadId
+            ? {
+                '@type': 'Thread',
+                source: 'gmail',
+                sourceId: gmail.threadId,
+                ...(google
+                    ? {
+                        '@key': ['@type', 'source', 'inAccount[*].handle', 'sourceId'],
+                        inAccount: [google],
+                    }
+                    : { '@key': ['@type', 'source', 'sourceId'] }),
+            }
+            : undefined;
+        return [
+            messageAction(mail, {
+                me: gmail.owner ? [gmail.owner] : [],
+                sentByMe: gmail.labels.includes('Sent'),
+                ...(gmail.receivedAt && { receivedAt: new Date(gmail.receivedAt) }),
+                ...(thread && { thread }),
+                tags: gmail.labels,
+                ...(account && { account }),
+                // You are the Google account at the mailbox's address.
+                meIdentities: google ? [google] : [],
+                // Your contacts link a person to their other addresses and numbers.
+                identitiesOf: address => contactIdentities(address, contacts?.[address.toLowerCase()], gmail.account),
+            }),
+        ];
+    }
+}

package/dist/filters.d.ts

@@ -0,0 +1,32 @@
+import type { MailMessage } from '@chronicle.app/email-core';
+import { z } from 'zod';
+/**
+ * What to extract. `--sent` and `--label` work with the API and a Takeout
+ * alike; anything else is a Gmail search, which needs the API.
+ */
+export declare const filterOptions: {
+    sent: z.ZodOptional<z.ZodBoolean>;
+    label: z.ZodOptional<z.ZodString>;
+    query: z.ZodOptional<z.ZodString>;
+};
+export interface Filters {
+    sent?: boolean;
+    label?: string;
+    query?: string;
+    since?: Date;
+    until?: Date;
+}
+/** Whether a run asks for anything less than the whole mailbox. */
+export declare const narrows: (filters: Filters) => boolean;
+/** The label names every message must have: `--label`, and Sent for `--sent`. */
+export declare function wantedLabels(filters: Filters): string[];
+/** Spam and Trash are read only when a run names them. */
+export declare const includesSpamTrash: (filters: Filters) => boolean;
+/**
+ * The Gmail search for a run: its `--query`, and the window (`after:` and
+ * `before:` take epoch seconds). Labels go by ID instead, in `labelIds`.
+ */
+export declare function gmailSearch(filters: Filters): string | undefined;
+/** Whether a message read from a Takeout passes the filters, given its labels. */
+export declare function matches(mail: MailMessage, labels: string[], filters: Filters): boolean;
+//# sourceMappingURL=filters.d.ts.map

package/dist/filters.js

@@ -0,0 +1,54 @@
+import { z } from 'zod';
+/**
+ * What to extract. `--sent` and `--label` work with the API and a Takeout
+ * alike; anything else is a Gmail search, which needs the API.
+ */
+export const filterOptions = {
+    sent: z.boolean().optional().describe('Only mail you sent'),
+    label: z
+        .string()
+        .optional()
+        .describe('Only mail with all of these labels, comma-separated, like "Work" or "Starred". Spam and Trash are left out unless named.'),
+    query: z
+        .string()
+        .optional()
+        .describe('A Gmail search, as in Gmail’s search box, like "from:boss has:attachment"'),
+};
+/** Whether a run asks for anything less than the whole mailbox. */
+export const narrows = (filters) => Boolean(filters.sent || filters.label || filters.query || filters.since || filters.until);
+/** The label names every message must have: `--label`, and Sent for `--sent`. */
+export function wantedLabels(filters) {
+    const labels = (filters.label ?? '')
+        .split(',')
+        .map(label => label.trim())
+        .filter(Boolean);
+    return filters.sent ? [...labels, 'Sent'] : labels;
+}
+/** Spam and Trash are read only when a run names them. */
+export const includesSpamTrash = (filters) => wantedLabels(filters).some(label => /^(spam|trash)$/i.test(label));
+/**
+ * The Gmail search for a run: its `--query`, and the window (`after:` and
+ * `before:` take epoch seconds). Labels go by ID instead, in `labelIds`.
+ */
+export function gmailSearch(filters) {
+    const terms = [
+        filters.query,
+        filters.since && `after:${Math.floor(filters.since.getTime() / 1000)}`,
+        filters.until && `before:${Math.floor(filters.until.getTime() / 1000)}`,
+    ].filter(Boolean);
+    return terms.length > 0 ? terms.join(' ') : undefined;
+}
+/** Whether a message read from a Takeout passes the filters, given its labels. */
+export function matches(mail, labels, filters) {
+    const has = new Set(labels.map(label => label.toLowerCase()));
+    if (!includesSpamTrash(filters) && (has.has('spam') || has.has('trash')))
+        return false;
+    if (!wantedLabels(filters).every(label => has.has(label.toLowerCase())))
+        return false;
+    const sent = mail.date ? new Date(mail.date) : undefined;
+    if (filters.since && (!sent || sent < filters.since))
+        return false;
+    if (filters.until && (!sent || sent >= filters.until))
+        return false;
+    return true;
+}

package/dist/index.d.ts

@@ -0,0 +1,5 @@
+import '@chronicle.app/google';
+export { GmailApiExtractor } from './connectors/GmailApiExtractor.js';
+export { GmailTakeoutExtractor } from './connectors/GmailTakeoutExtractor.js';
+export { default as GmailTransformer } from './connectors/GmailTransformer.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.js

@@ -0,0 +1,6 @@
+// Registers the Google sign-in, so `chronicle auth login google` finds it.
+import '@chronicle.app/google';
+// Export extractors for plugin scanning
+export { GmailApiExtractor } from './connectors/GmailApiExtractor.js';
+export { GmailTakeoutExtractor } from './connectors/GmailTakeoutExtractor.js';
+export { default as GmailTransformer } from './connectors/GmailTransformer.js';

package/dist/labels.d.ts

@@ -0,0 +1,15 @@
+/** A label's name from its API ID: Gmail's own by its shown name, yours from `names`. */
+export declare function labelName(id: string, names: Map<string, string>): string | undefined;
+/** A label's API ID from its name, either kind, ignoring case. */
+export declare function labelId(name: string, names: Map<string, string>): string | undefined;
+/**
+ * The labels in a Takeout message's `X-Gmail-Labels` header: comma-separated,
+ * a name with a comma in quotes. Read state is left out.
+ */
+export declare function takeoutLabels(header: string | undefined): string[];
+/**
+ * Gmail's thread and message IDs: the API writes them in hex, a Takeout's
+ * `X-GM-THRID` and `X-GM-MSGID` in decimal. Hex either way, so they match.
+ */
+export declare function hexId(decimal: string | undefined): string | null;
+//# sourceMappingURL=labels.d.ts.map

package/dist/labels.js

@@ -0,0 +1,58 @@
+/**
+ * Gmail's own labels by the names Gmail shows, the same names a Takeout
+ * writes in `X-Gmail-Labels`, so a message has the same labels either way.
+ * Read state (Unread, Opened) changes as you read, so it isn't a label here.
+ */
+const SYSTEM_LABELS = {
+    INBOX: 'Inbox',
+    SENT: 'Sent',
+    DRAFT: 'Drafts',
+    STARRED: 'Starred',
+    IMPORTANT: 'Important',
+    SPAM: 'Spam',
+    TRASH: 'Trash',
+    CHAT: 'Chat',
+    CATEGORY_PERSONAL: 'Category Personal',
+    CATEGORY_SOCIAL: 'Category Social',
+    CATEGORY_PROMOTIONS: 'Category Promotions',
+    CATEGORY_UPDATES: 'Category Updates',
+    CATEGORY_FORUMS: 'Category Forums',
+};
+const READ_STATE = new Set(['UNREAD', 'Unread', 'Opened']);
+/** A label's name from its API ID: Gmail's own by its shown name, yours from `names`. */
+export function labelName(id, names) {
+    if (READ_STATE.has(id))
+        return undefined;
+    return SYSTEM_LABELS[id] ?? names.get(id);
+}
+/** A label's API ID from its name, either kind, ignoring case. */
+export function labelId(name, names) {
+    const wanted = name.toLowerCase();
+    for (const [id, shown] of Object.entries(SYSTEM_LABELS)) {
+        if (shown.toLowerCase() === wanted || id.toLowerCase() === wanted)
+            return id;
+    }
+    for (const [id, shown] of names)
+        if (shown.toLowerCase() === wanted)
+            return id;
+    return undefined;
+}
+/**
+ * The labels in a Takeout message's `X-Gmail-Labels` header: comma-separated,
+ * a name with a comma in quotes. Read state is left out.
+ */
+export function takeoutLabels(header) {
+    const labels = (header ?? '').match(/"[^"]*"|[^,]+/g) ?? [];
+    return labels
+        .map(label => label.trim().replaceAll(/^"|"$/g, ''))
+        .filter(label => label && !READ_STATE.has(label));
+}
+/**
+ * Gmail's thread and message IDs: the API writes them in hex, a Takeout's
+ * `X-GM-THRID` and `X-GM-MSGID` in decimal. Hex either way, so they match.
+ */
+export function hexId(decimal) {
+    if (!decimal || !/^\d+$/.test(decimal.trim()))
+        return null;
+    return BigInt(decimal.trim()).toString(16);
+}

package/dist/types.d.ts

@@ -0,0 +1,31 @@
+import type { ContactLinks } from '@chronicle.app/google';
+import type { MailMessage } from '@chronicle.app/email-core';
+/** One extracted record: the message, and what Gmail knows about it. */
+export interface GmailRecord {
+    mail: MailMessage;
+    gmail: {
+        /** Gmail's message ID, in hex. */
+        id: string | null;
+        /** Gmail's thread ID, in hex: the API's and a Takeout's agree. */
+        threadId: string | null;
+        /** Label names, as Gmail shows them. */
+        labels: string[];
+        /** When Gmail received it: the API's internal date, or a Takeout's `From ` line. */
+        receivedAt: string | null;
+        /**
+         * Your address on the message: the API's profile, or a Takeout's
+         * Delivered-To, or the sender of mail in Sent.
+         */
+        owner: string | null;
+        /**
+         * The Google account the mailbox is, by its address: the API's profile.
+         * A Takeout doesn't say which account it came from, so it has none.
+         */
+        account: string | null;
+    };
+    /** What your contacts link each address on it to (API only). */
+    contacts?: {
+        [address: string]: ContactLinks;
+    };
+}
+//# sourceMappingURL=types.d.ts.map

package/dist/types.js

@@ -0,0 +1 @@
+export {};

package/package.json

@@ -1,6 +1,90 @@
 {
   "name": "@chronicle.app/gmail",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.4.0",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "files": [
+    "dist/**/*.js",
+    "dist/**/*.d.ts"
+  ],
+  "scripts": {
+    "build": "tsc -b tsconfig.lib.json",
+    "typecheck": "tsc -p tsconfig.json",
+    "lint": "eslint src --ext .ts,.js",
+    "test": "node --test src/*.test.js",
+    "shapes": "node src/shapes.test.js --update"
+  },
+  "keywords": [
+    "chronicle",
+    "google",
+    "gmail",
+    "email"
+  ],
+  "author": "Andrew Louis",
+  "license": "MIT",
+  "type": "module",
+  "description": "Chronicle plugin for your Gmail: messages, threads, and labels",
+  "dependencies": {
+    "@chronicle.app/google": "0.4.0",
+    "@chronicle.app/logging": "0.4.0",
+    "@chronicle.app/email-core": "0.4.0",
+    "zod": "^3.24.1"
+  },
+  "devDependencies": {
+    "@chronicle.app/etl": "0.4.0",
+    "@chronicle.app/schema": "0.4.0",
+    "@chronicle.app/tsconfig": "0.4.0",
+    "@types/node": "^22.10.1",
+    "typescript": "^5.7.2"
+  },
+  "homepage": "https://chronicle.app",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/chronicle-app/chronicle.git",
+    "directory": "plugins/gmail"
+  },
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/"
+  },
+  "peerDependencies": {
+    "@chronicle.app/etl": ">=0.1.0 <1.0.0",
+    "@chronicle.app/schema": ">=0.1.0 <1.0.0"
+  },
+  "engines": {
+    "node": ">=22.13.0"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "chronicle": {
+    "plugin": true,
+    "sources": {
+      "gmail": {
+        "strategies": {
+          "api": {
+            "delivery": "api",
+            "recordTypes": [
+              "messages"
+            ],
+            "default": true
+          },
+          "takeout": {
+            "delivery": "export",
+            "recordTypes": [
+              "messages"
+            ]
+          }
+        }
+      }
+    },
+    "deepLinks": {
+      "gmail": {
+        "Thread": "https://mail.google.com/mail/?authuser={inAccount[*].handle}#all/{sourceId}"
+      }
+    }
+  }
+}