@chronicle.app/chrome 0.0.0-stage → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +12 -2
- package/dist/connectors/ChromeExtractor.d.ts +63 -0
- package/dist/connectors/ChromeExtractor.js +267 -0
- package/dist/connectors/ChromeTransformer.d.ts +23 -0
- package/dist/connectors/ChromeTransformer.js +77 -0
- package/dist/connectors/profile.d.ts +14 -0
- package/dist/connectors/profile.js +34 -0
- package/dist/index.d.ts +3 -0
- package/dist/index.js +3 -0
- package/package.json +76 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Andrew Louis
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,13 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @chronicle.app/chrome
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Read Chrome's `History` database with `ChromeExtractor`, then emit one `ViewAction` per page visit with `ChromeTransformer`. `input` defaults to the `History` file of the `profile` setting (`Default` unless set) in Chrome's profile directory on macOS, Linux, or Windows. The extractor reads a temporary copy of the database, read-only via built-in `node:sqlite` and removed when it finishes, so Chrome can keep its own open while it runs. A copy taken while Chrome was saving is taken again. On Linux, the profile directory follows `CHROME_CONFIG_HOME` or `XDG_CONFIG_HOME` when set.
|
|
4
|
+
|
|
5
|
+
Records are visits, newest first. Redirect hops, pages loaded in frames, the extra visits Chrome adds to a search engine's keyword URL, pages that aren't on the web (`chrome://` and extension pages, files), visits imported from other browsers, and visits an extension added are left out. Visits synced from your other devices are included. `since`/`until` are exclusive visit-time bounds, and `limit: 0` means unlimited.
|
|
6
|
+
|
|
7
|
+
Each `ViewAction` is keyed by the page's URL and its timestamp, to the millisecond. Chrome's visit id is only unique within one `History` file, but a synced visit keeps its time, so the same visit read on two devices is one node, and two pages opened in the same millisecond are two. The visited page is an `Entity` keyed by its URL and named after the page title, or the URL when there is none. Chrome keeps one title per URL, not per visit, so every visit to a page carries its latest title.
|
|
8
|
+
|
|
9
|
+
The agent is the Google account the profile belongs to, read from its `Preferences`: the one signed in now, or else the last one that synced, which Chrome keeps after sign-out. It is the account's email address, lowercased, as an `email` Agent named by the account's name, `sameAs` the Google account, an Agent keyed by the same address in the `google-account` namespace: the nodes Gmail and Google Calendar make for you. `Preferences` lists every Google account signed in on the web too, so the profile's own is the one with its Gaia id; the id isn't recorded. A profile never signed in, or an account without an address, has no agent. The account is read when the run starts and applies to every visit in the file: History doesn't record who was signed in at each visit.
|
|
10
|
+
|
|
11
|
+
A visit reached by following a link or submitting a form also becomes a `NavigateAction` at the same time: its `object` is the page you came from and its `target` the page you landed on, both keyed by URL, and it's keyed as the visit is, by its `target`'s URL and its timestamp. The page you came from is the one before in the same tab, or the page in the tab that opened it. Chrome also keeps a referrer as a URL when it isn't a visit, but for a link to another site browsers cut it to the site's home page, so it isn't used. For a redirected visit, it is where the redirect chain began. For a followed link, the page you came from also `references` the URL you followed, before any redirects. Typed URLs, bookmarks, reloads, and going back or forward have no `NavigateAction`, nor do steps from a page that isn't on the web or from the same page.
|
|
12
|
+
|
|
13
|
+
[SHAPES.md](SHAPES.md) shows what each record type becomes; run `npm run shapes` to update it after changing the transformer. Tests use a synthetic history database and `Preferences` file and never read the host's browsing history.
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
import { Record } from '@chronicle.app/etl';
|
|
2
|
+
import { SqliteExtractor } from '@chronicle.app/etl-sqlite';
|
|
3
|
+
import ChromeTransformer from './ChromeTransformer.js';
|
|
4
|
+
/** A link followed or a form submitted, from one page to the visit's. */
|
|
5
|
+
export interface ChromeNavigation {
|
|
6
|
+
from_url: string;
|
|
7
|
+
/** The URL followed: the redirect chain's first, before any redirects. */
|
|
8
|
+
followed_url: string;
|
|
9
|
+
form_submit: boolean;
|
|
10
|
+
}
|
|
11
|
+
export declare class ChromeExtractor extends SqliteExtractor<typeof ChromeExtractor> {
|
|
12
|
+
static source: string;
|
|
13
|
+
static delivery: "local";
|
|
14
|
+
static strategy: string;
|
|
15
|
+
static description: string;
|
|
16
|
+
static default: boolean;
|
|
17
|
+
static recordTypes: string[];
|
|
18
|
+
static newestFirst: boolean;
|
|
19
|
+
/** Where Chrome keeps its profiles, per platform. */
|
|
20
|
+
static userDataDirs: {
|
|
21
|
+
[P in NodeJS.Platform]?: () => string;
|
|
22
|
+
};
|
|
23
|
+
static schema: any;
|
|
24
|
+
static defaultTransformer: typeof ChromeTransformer;
|
|
25
|
+
/** Chrome saves its history every few seconds: tries at copying it, and the wait between. */
|
|
26
|
+
static copyAttempts: number;
|
|
27
|
+
static copyRetryMs: number;
|
|
28
|
+
private account;
|
|
29
|
+
private referrerColumns;
|
|
30
|
+
private visitById;
|
|
31
|
+
private copyDir;
|
|
32
|
+
/** visits.id — Chrome's own visit id, AUTOINCREMENT so never reused. */
|
|
33
|
+
keyOf(record: Record): string | null;
|
|
34
|
+
setup(): Promise<void>;
|
|
35
|
+
/**
|
|
36
|
+
* Copy History into `dir`, to read it without Chrome's lock, and without
|
|
37
|
+
* holding the file Chrome writes, which could keep a Chrome that starts
|
|
38
|
+
* mid-run from opening it. A copy is whole when Chrome wasn't saving and
|
|
39
|
+
* the file didn't change while it was copied. A copy that caught a save is
|
|
40
|
+
* taken again.
|
|
41
|
+
*/
|
|
42
|
+
private copyHistory;
|
|
43
|
+
teardown(): Promise<void>;
|
|
44
|
+
private userDataDir;
|
|
45
|
+
private visitFilter;
|
|
46
|
+
private static readonly FROM;
|
|
47
|
+
extract(): AsyncGenerator<Record>;
|
|
48
|
+
private processRow;
|
|
49
|
+
/**
|
|
50
|
+
* How the visit was reached by following a link or submitting a form: from
|
|
51
|
+
* the page before it in its tab, else the page in the tab that opened it.
|
|
52
|
+
* Read from the first visit of its redirect chain, as Chrome's own history
|
|
53
|
+
* features do. Null for a typed URL, a bookmark, a reload, going back or
|
|
54
|
+
* forward, or a page reached from itself or from a page that isn't on the
|
|
55
|
+
* web. Chrome also keeps a referrer that isn't a visit as a URL, but for a
|
|
56
|
+
* link to another site browsers cut it to that site's home page, which isn't
|
|
57
|
+
* the page you came from.
|
|
58
|
+
*/
|
|
59
|
+
private navigation;
|
|
60
|
+
private visit;
|
|
61
|
+
determineCount(): Promise<number | null>;
|
|
62
|
+
}
|
|
63
|
+
//# sourceMappingURL=ChromeExtractor.d.ts.map
|
|
@@ -0,0 +1,267 @@
|
|
|
1
|
+
import { constants as fsConstants, copyFileSync, mkdtempSync, rmSync, statSync } from 'node:fs';
|
|
2
|
+
import { homedir, tmpdir } from 'node:os';
|
|
3
|
+
import { dirname, join } from 'node:path';
|
|
4
|
+
import { EXIT_CODES, ExtractorError, delay } from '@chronicle.app/etl';
|
|
5
|
+
import { SqliteExtractor, allRows, chromeToUnixMsSql, getRow, iterateRows, timeRangeConditions, unixMsToChromeTimestamp, } from '@chronicle.app/etl-sqlite';
|
|
6
|
+
import { z } from 'zod';
|
|
7
|
+
import ChromeTransformer from './ChromeTransformer.js';
|
|
8
|
+
import { readChromeAccount } from './profile.js';
|
|
9
|
+
// visits.transition: the low byte is the core type, the high bits qualifiers.
|
|
10
|
+
// CHAIN_END marks the visit a redirect chain landed on (a visit with no
|
|
11
|
+
// redirects is both start and end), so redirect hops drop out.
|
|
12
|
+
const CHAIN_END = 0x20_00_00_00;
|
|
13
|
+
const CORE_MASK = 0xff;
|
|
14
|
+
// Core types that are not a page you visited: pages loaded inside a frame,
|
|
15
|
+
// automatically or by a click there (3, 4), and the extra visit Chrome adds to
|
|
16
|
+
// a search engine's keyword URL when you search from the address bar (10).
|
|
17
|
+
const SKIPPED_TYPES = [3, 4, 10];
|
|
18
|
+
// visit_source.source for visits you made: synced from another device (0) or
|
|
19
|
+
// browsed here (1, which Chrome usually leaves unrecorded). Not ones an
|
|
20
|
+
// extension added (2), or imported from another browser or device migration
|
|
21
|
+
// (Firefox, IE, Safari, OS migration): another browser's history belongs to
|
|
22
|
+
// that browser's source.
|
|
23
|
+
const BROWSED_SOURCES = [0, 1];
|
|
24
|
+
// Web pages only: not browser or extension pages, files, or blobs.
|
|
25
|
+
const WEB_SCHEMES = ['http://', 'https://'];
|
|
26
|
+
// A redirect chain's first visit: its transition says how the navigation
|
|
27
|
+
// began, and its from_visit is the page it began on.
|
|
28
|
+
const CHAIN_START = 0x10_00_00_00;
|
|
29
|
+
const MAX_REDIRECTS = 20;
|
|
30
|
+
// Core types for following something on a page: a link (0) or a form (7).
|
|
31
|
+
const LINK = 0;
|
|
32
|
+
const FORM_SUBMIT = 7;
|
|
33
|
+
// Qualifiers for going back or forward (0x01000000) and for a URL entered in
|
|
34
|
+
// the address bar (0x02000000): not a step from the page before.
|
|
35
|
+
const NOT_FOLLOWED = 0x03_00_00_00;
|
|
36
|
+
const visitTime = chromeToUnixMsSql('visits.visit_time');
|
|
37
|
+
export class ChromeExtractor extends SqliteExtractor {
|
|
38
|
+
static source = 'chrome';
|
|
39
|
+
static delivery = 'local';
|
|
40
|
+
static strategy = 'app-db';
|
|
41
|
+
static description = 'Browsing history';
|
|
42
|
+
static default = true;
|
|
43
|
+
static recordTypes = ['views'];
|
|
44
|
+
// extract() orders by visits.visit_time desc, which with keyOf opts this
|
|
45
|
+
// extractor into the frontier cursor (incremental catch-up import).
|
|
46
|
+
static newestFirst = true;
|
|
47
|
+
/** Where Chrome keeps its profiles, per platform. */
|
|
48
|
+
static userDataDirs = {
|
|
49
|
+
darwin: () => join(homedir(), 'Library/Application Support/Google/Chrome'),
|
|
50
|
+
// Chrome's own override for ~/.config, then the XDG one.
|
|
51
|
+
linux: () => join(process.env.CHROME_CONFIG_HOME || process.env.XDG_CONFIG_HOME || join(homedir(), '.config'), 'google-chrome'),
|
|
52
|
+
win32: () => join(process.env.LOCALAPPDATA ?? join(homedir(), 'AppData/Local'), 'Google/Chrome/User Data'),
|
|
53
|
+
};
|
|
54
|
+
static schema = SqliteExtractor.schema.omit({ input: true }).extend({
|
|
55
|
+
input: z
|
|
56
|
+
.string()
|
|
57
|
+
.optional()
|
|
58
|
+
.describe("Path to a Chrome History file (defaults to the profile's)"),
|
|
59
|
+
profile: z
|
|
60
|
+
.string()
|
|
61
|
+
.default('Default')
|
|
62
|
+
.describe('Chrome profile directory name, such as Default or "Profile 1"'),
|
|
63
|
+
});
|
|
64
|
+
static defaultTransformer = ChromeTransformer;
|
|
65
|
+
/** Chrome saves its history every few seconds: tries at copying it, and the wait between. */
|
|
66
|
+
static copyAttempts = 5;
|
|
67
|
+
static copyRetryMs = 200;
|
|
68
|
+
account = null;
|
|
69
|
+
referrerColumns = '';
|
|
70
|
+
visitById = null;
|
|
71
|
+
copyDir = null;
|
|
72
|
+
/** visits.id — Chrome's own visit id, AUTOINCREMENT so never reused. */
|
|
73
|
+
keyOf(record) {
|
|
74
|
+
const visitId = record.data.visit_id;
|
|
75
|
+
return visitId === null || visitId === undefined ? null : String(visitId);
|
|
76
|
+
}
|
|
77
|
+
async setup() {
|
|
78
|
+
const cfg = this.config;
|
|
79
|
+
cfg.input ??= join(this.userDataDir(), cfg.profile, 'History');
|
|
80
|
+
// Says whether History is there and readable; the run reads a copy.
|
|
81
|
+
await super.setup();
|
|
82
|
+
this.db.close();
|
|
83
|
+
this.db = null;
|
|
84
|
+
this.copyDir = mkdtempSync(join(tmpdir(), 'chronicle-chrome-'));
|
|
85
|
+
try {
|
|
86
|
+
this.db = this.openDatabase(await this.copyHistory(cfg.input, this.copyDir));
|
|
87
|
+
this.account = readChromeAccount(join(dirname(cfg.input), 'Preferences'));
|
|
88
|
+
// History files from before 2022 have no opener.
|
|
89
|
+
const columns = new Set(allRows(this.db.prepare('PRAGMA table_info(visits)')).map(c => c.name));
|
|
90
|
+
this.referrerColumns = [
|
|
91
|
+
'visits.transition',
|
|
92
|
+
`visits.transition & ${CORE_MASK} AS core_type`,
|
|
93
|
+
`(visits.transition & ${CHAIN_START}) != 0 AS chain_start`,
|
|
94
|
+
`(visits.transition & ${NOT_FOLLOWED}) != 0 AS not_followed`,
|
|
95
|
+
'visits.from_visit',
|
|
96
|
+
columns.has('opener_visit') ? 'visits.opener_visit' : '0 AS opener_visit',
|
|
97
|
+
].join(', ');
|
|
98
|
+
this.visitById = this.db.prepare(`
|
|
99
|
+
SELECT urls.url, ${this.referrerColumns}
|
|
100
|
+
FROM visits JOIN urls ON urls.id = visits.url
|
|
101
|
+
WHERE visits.id = ?`);
|
|
102
|
+
}
|
|
103
|
+
catch (error) {
|
|
104
|
+
await this.teardown();
|
|
105
|
+
throw error;
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
/**
|
|
109
|
+
* Copy History into `dir`, to read it without Chrome's lock, and without
|
|
110
|
+
* holding the file Chrome writes, which could keep a Chrome that starts
|
|
111
|
+
* mid-run from opening it. A copy is whole when Chrome wasn't saving and
|
|
112
|
+
* the file didn't change while it was copied. A copy that caught a save is
|
|
113
|
+
* taken again.
|
|
114
|
+
*/
|
|
115
|
+
async copyHistory(history, dir) {
|
|
116
|
+
const cls = this.constructor;
|
|
117
|
+
const copy = join(dir, 'History');
|
|
118
|
+
for (let attempt = 1;; attempt++) {
|
|
119
|
+
const before = savedState(history);
|
|
120
|
+
copyFileSync(history, copy, fsConstants.COPYFILE_FICLONE);
|
|
121
|
+
if (before !== null && before === savedState(history))
|
|
122
|
+
return copy;
|
|
123
|
+
if (attempt >= cls.copyAttempts) {
|
|
124
|
+
throw new ExtractorError('Chrome kept saving its history', {
|
|
125
|
+
code: 'input-busy',
|
|
126
|
+
exitCode: EXIT_CODES.transient,
|
|
127
|
+
hint: `Run \`chronicle extract ${cls.source}\` again in a moment.`,
|
|
128
|
+
fields: { path: history },
|
|
129
|
+
});
|
|
130
|
+
}
|
|
131
|
+
await delay(cls.copyRetryMs);
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
async teardown() {
|
|
135
|
+
await super.teardown();
|
|
136
|
+
if (this.copyDir) {
|
|
137
|
+
rmSync(this.copyDir, { recursive: true, force: true });
|
|
138
|
+
this.copyDir = null;
|
|
139
|
+
}
|
|
140
|
+
}
|
|
141
|
+
userDataDir() {
|
|
142
|
+
const cls = this.constructor;
|
|
143
|
+
const dir = cls.userDataDirs[process.platform];
|
|
144
|
+
if (!dir)
|
|
145
|
+
throw new Error(`No default ${cls.source} profile on ${process.platform}; pass input`);
|
|
146
|
+
return dir();
|
|
147
|
+
}
|
|
148
|
+
// Visits that landed on a web page, browsed or synced, within since/until.
|
|
149
|
+
visitFilter() {
|
|
150
|
+
const range = timeRangeConditions('visits.visit_time', this.config, {
|
|
151
|
+
convert: date => unixMsToChromeTimestamp(date.getTime()),
|
|
152
|
+
});
|
|
153
|
+
const conditions = [
|
|
154
|
+
`(visits.transition & ${CHAIN_END}) != 0`,
|
|
155
|
+
`(visits.transition & ${CORE_MASK}) NOT IN (${SKIPPED_TYPES.join(', ')})`,
|
|
156
|
+
`(visit_source.source IS NULL OR visit_source.source IN (${BROWSED_SOURCES.join(', ')}))`,
|
|
157
|
+
`(${WEB_SCHEMES.map(() => `urls.url LIKE ? || '%'`).join(' OR ')})`,
|
|
158
|
+
...range.conditions,
|
|
159
|
+
];
|
|
160
|
+
return {
|
|
161
|
+
where: conditions.join(' AND '),
|
|
162
|
+
values: [...WEB_SCHEMES, ...range.values],
|
|
163
|
+
};
|
|
164
|
+
}
|
|
165
|
+
static FROM = `
|
|
166
|
+
FROM visits
|
|
167
|
+
JOIN urls ON urls.id = visits.url
|
|
168
|
+
LEFT JOIN visit_source ON visit_source.id = visits.id`;
|
|
169
|
+
async *extract() {
|
|
170
|
+
if (!this.db) {
|
|
171
|
+
throw new Error('Database not initialized');
|
|
172
|
+
}
|
|
173
|
+
// 0 is the "no limit" convention (null here); the CLI's --limit default is the safety net
|
|
174
|
+
const limit = this.getEffectiveLimit();
|
|
175
|
+
const { where, values } = this.visitFilter();
|
|
176
|
+
let sql = `
|
|
177
|
+
SELECT visits.id AS visit_id,
|
|
178
|
+
urls.url,
|
|
179
|
+
urls.title,
|
|
180
|
+
${visitTime} AS unix_ms,
|
|
181
|
+
${this.referrerColumns}
|
|
182
|
+
${ChromeExtractor.FROM}
|
|
183
|
+
WHERE ${where}
|
|
184
|
+
ORDER BY visits.visit_time DESC`;
|
|
185
|
+
if (limit !== null) {
|
|
186
|
+
sql += ` LIMIT ?`;
|
|
187
|
+
values.push(limit);
|
|
188
|
+
}
|
|
189
|
+
const { account } = this;
|
|
190
|
+
for (const row of iterateRows(this.db.prepare(sql), ...values)) {
|
|
191
|
+
yield this.createRecord(this.processRow(row), { source: 'chrome', account });
|
|
192
|
+
}
|
|
193
|
+
}
|
|
194
|
+
processRow(row) {
|
|
195
|
+
return {
|
|
196
|
+
visit_id: row.visit_id,
|
|
197
|
+
url: row.url,
|
|
198
|
+
title: row.title || null,
|
|
199
|
+
unix_ms: row.unix_ms,
|
|
200
|
+
transition: row.transition,
|
|
201
|
+
navigation: this.navigation(row),
|
|
202
|
+
};
|
|
203
|
+
}
|
|
204
|
+
/**
|
|
205
|
+
* How the visit was reached by following a link or submitting a form: from
|
|
206
|
+
* the page before it in its tab, else the page in the tab that opened it.
|
|
207
|
+
* Read from the first visit of its redirect chain, as Chrome's own history
|
|
208
|
+
* features do. Null for a typed URL, a bookmark, a reload, going back or
|
|
209
|
+
* forward, or a page reached from itself or from a page that isn't on the
|
|
210
|
+
* web. Chrome also keeps a referrer that isn't a visit as a URL, but for a
|
|
211
|
+
* link to another site browsers cut it to that site's home page, which isn't
|
|
212
|
+
* the page you came from.
|
|
213
|
+
*/
|
|
214
|
+
navigation(row) {
|
|
215
|
+
let start = row;
|
|
216
|
+
for (let hops = 0; !start.chain_start; hops++) {
|
|
217
|
+
const previous = hops < MAX_REDIRECTS ? this.visit(start.from_visit) : undefined;
|
|
218
|
+
if (!previous)
|
|
219
|
+
return null;
|
|
220
|
+
start = previous;
|
|
221
|
+
}
|
|
222
|
+
if (start.core_type !== LINK && start.core_type !== FORM_SUBMIT)
|
|
223
|
+
return null;
|
|
224
|
+
if (start.not_followed)
|
|
225
|
+
return null;
|
|
226
|
+
const from = this.visit(start.from_visit)?.url ?? this.visit(start.opener_visit)?.url;
|
|
227
|
+
if (!from || from === row.url || !WEB_SCHEMES.some(scheme => from.startsWith(scheme))) {
|
|
228
|
+
return null;
|
|
229
|
+
}
|
|
230
|
+
return {
|
|
231
|
+
from_url: from,
|
|
232
|
+
followed_url: start.url,
|
|
233
|
+
form_submit: start.core_type === FORM_SUBMIT,
|
|
234
|
+
};
|
|
235
|
+
}
|
|
236
|
+
visit(id) {
|
|
237
|
+
return id ? getRow(this.visitById, id) : undefined;
|
|
238
|
+
}
|
|
239
|
+
async determineCount() {
|
|
240
|
+
if (!this.db)
|
|
241
|
+
return null;
|
|
242
|
+
try {
|
|
243
|
+
// Same filters as the main query, counted
|
|
244
|
+
const { where, values } = this.visitFilter();
|
|
245
|
+
const row = this.db
|
|
246
|
+
.prepare(`SELECT COUNT(*) AS n ${ChromeExtractor.FROM} WHERE ${where}`)
|
|
247
|
+
.get(...values);
|
|
248
|
+
return row.n;
|
|
249
|
+
}
|
|
250
|
+
catch {
|
|
251
|
+
return null;
|
|
252
|
+
}
|
|
253
|
+
}
|
|
254
|
+
}
|
|
255
|
+
/**
|
|
256
|
+
* History's size and modification time while it holds only saved changes,
|
|
257
|
+
* or null while Chrome saves. Chrome keeps a transaction open between saves,
|
|
258
|
+
* and SQLite writes each change to the journal before the file, so the file
|
|
259
|
+
* is whole while its journal is empty or was written after it.
|
|
260
|
+
*/
|
|
261
|
+
function savedState(history) {
|
|
262
|
+
const file = statSync(history);
|
|
263
|
+
const journal = statSync(`${history}-journal`, { throwIfNoEntry: false });
|
|
264
|
+
if (journal && journal.size > 0 && journal.mtimeMs <= file.mtimeMs)
|
|
265
|
+
return null;
|
|
266
|
+
return `${file.size}:${file.mtimeMs}`;
|
|
267
|
+
}
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
import { ChronicleTransformer, Record } from '@chronicle.app/etl';
|
|
2
|
+
import { ActionAndChildren } from '@chronicle.app/schema';
|
|
3
|
+
export default class ChromeTransformer extends ChronicleTransformer {
|
|
4
|
+
transform(record: Record): Promise<ActionAndChildren[]>;
|
|
5
|
+
private buildViewAction;
|
|
6
|
+
/**
|
|
7
|
+
* The step from the page you followed a link or submitted a form on to the
|
|
8
|
+
* page you landed on, at the same moment as the visit. Both pages are the
|
|
9
|
+
* bare URL nodes the visits name. A followed link is also evidence that its
|
|
10
|
+
* page links to the URL followed, before any redirects, so that page
|
|
11
|
+
* `references` it. A form's address is not a link on the page.
|
|
12
|
+
*/
|
|
13
|
+
private buildNavigateAction;
|
|
14
|
+
/**
|
|
15
|
+
* The profile's Google account as you: its address, as Gmail and Google
|
|
16
|
+
* Calendar key you, `sameAs` the account, keyed by the same address in the
|
|
17
|
+
* `google-account` namespace they link you to.
|
|
18
|
+
*/
|
|
19
|
+
private buildUser;
|
|
20
|
+
private buildWebPage;
|
|
21
|
+
private pageByUrl;
|
|
22
|
+
}
|
|
23
|
+
//# sourceMappingURL=ChromeTransformer.d.ts.map
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
import { ChronicleTransformer, selfAgent } from '@chronicle.app/etl';
|
|
2
|
+
import { googleAccount } from '@chronicle.app/google';
|
|
3
|
+
export default class ChromeTransformer extends ChronicleTransformer {
|
|
4
|
+
async transform(record) {
|
|
5
|
+
const actions = [];
|
|
6
|
+
if (record.extraction.recordType === 'views') {
|
|
7
|
+
const user = this.buildUser(record.context.account);
|
|
8
|
+
actions.push(this.buildViewAction(record, user));
|
|
9
|
+
const navigation = record.data.navigation;
|
|
10
|
+
if (navigation)
|
|
11
|
+
actions.push(this.buildNavigateAction(record, navigation, user));
|
|
12
|
+
}
|
|
13
|
+
return actions;
|
|
14
|
+
}
|
|
15
|
+
buildViewAction(record, user) {
|
|
16
|
+
return {
|
|
17
|
+
'@type': 'ViewAction',
|
|
18
|
+
timestamp: new Date(record.data.unix_ms),
|
|
19
|
+
// Chrome's visit id is only unique within one History file, but a
|
|
20
|
+
// synced visit keeps its time, so the page and time are the visit on
|
|
21
|
+
// every device. Two pages can open in the same millisecond.
|
|
22
|
+
'@key': ['@type', 'source', 'object.url', 'timestamp'],
|
|
23
|
+
source: 'chrome',
|
|
24
|
+
// A profile never signed in can't identify the viewer, so omit them.
|
|
25
|
+
...(user && { agent: user }),
|
|
26
|
+
object: this.buildWebPage(record),
|
|
27
|
+
};
|
|
28
|
+
}
|
|
29
|
+
/**
|
|
30
|
+
* The step from the page you followed a link or submitted a form on to the
|
|
31
|
+
* page you landed on, at the same moment as the visit. Both pages are the
|
|
32
|
+
* bare URL nodes the visits name. A followed link is also evidence that its
|
|
33
|
+
* page links to the URL followed, before any redirects, so that page
|
|
34
|
+
* `references` it. A form's address is not a link on the page.
|
|
35
|
+
*/
|
|
36
|
+
buildNavigateAction(record, navigation, user) {
|
|
37
|
+
const followed = this.pageByUrl(navigation.followed_url);
|
|
38
|
+
return {
|
|
39
|
+
'@type': 'NavigateAction',
|
|
40
|
+
timestamp: new Date(record.data.unix_ms),
|
|
41
|
+
// Keyed as the visit is: the page landed on, and when.
|
|
42
|
+
'@key': ['@type', 'source', 'target.url', 'timestamp'],
|
|
43
|
+
source: 'chrome',
|
|
44
|
+
...(user && { agent: user }),
|
|
45
|
+
object: {
|
|
46
|
+
...this.pageByUrl(navigation.from_url),
|
|
47
|
+
...(!navigation.form_submit && { references: [followed] }),
|
|
48
|
+
},
|
|
49
|
+
target: this.pageByUrl(record.data.url),
|
|
50
|
+
};
|
|
51
|
+
}
|
|
52
|
+
/**
|
|
53
|
+
* The profile's Google account as you: its address, as Gmail and Google
|
|
54
|
+
* Calendar key you, `sameAs` the account, keyed by the same address in the
|
|
55
|
+
* `google-account` namespace they link you to.
|
|
56
|
+
*/
|
|
57
|
+
buildUser(account) {
|
|
58
|
+
if (!account)
|
|
59
|
+
return null;
|
|
60
|
+
return selfAgent({
|
|
61
|
+
type: 'Agent',
|
|
62
|
+
source: 'email',
|
|
63
|
+
handle: account.email,
|
|
64
|
+
...(account.name && { name: account.name }),
|
|
65
|
+
sameAs: [googleAccount(account.email)],
|
|
66
|
+
});
|
|
67
|
+
}
|
|
68
|
+
buildWebPage(record) {
|
|
69
|
+
return {
|
|
70
|
+
...this.pageByUrl(record.data.url),
|
|
71
|
+
name: record.data.title || record.data.url,
|
|
72
|
+
};
|
|
73
|
+
}
|
|
74
|
+
pageByUrl(url) {
|
|
75
|
+
return { '@type': 'Entity', '@key': ['url'], url };
|
|
76
|
+
}
|
|
77
|
+
}
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
/** The Google account a Chrome profile is signed in to, by its address. */
|
|
2
|
+
export interface ChromeAccount {
|
|
3
|
+
email: string;
|
|
4
|
+
/** The account's name, as Google shows it. */
|
|
5
|
+
name?: string;
|
|
6
|
+
}
|
|
7
|
+
/**
|
|
8
|
+
* Read the profile's account from its `Preferences` JSON: the one signed in
|
|
9
|
+
* now, else the last one that synced, which Chrome keeps after sign-out. A
|
|
10
|
+
* profile never signed in, an account without an address, or a missing or
|
|
11
|
+
* unreadable file has none: visits are still readable, with no agent.
|
|
12
|
+
*/
|
|
13
|
+
export declare function readChromeAccount(preferencesPath: string): ChromeAccount | null;
|
|
14
|
+
//# sourceMappingURL=profile.d.ts.map
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
import { readFileSync } from 'node:fs';
|
|
2
|
+
/**
|
|
3
|
+
* Read the profile's account from its `Preferences` JSON: the one signed in
|
|
4
|
+
* now, else the last one that synced, which Chrome keeps after sign-out. A
|
|
5
|
+
* profile never signed in, an account without an address, or a missing or
|
|
6
|
+
* unreadable file has none: visits are still readable, with no agent.
|
|
7
|
+
*/
|
|
8
|
+
export function readChromeAccount(preferencesPath) {
|
|
9
|
+
let prefs;
|
|
10
|
+
try {
|
|
11
|
+
prefs = JSON.parse(readFileSync(preferencesPath, 'utf8'));
|
|
12
|
+
}
|
|
13
|
+
catch {
|
|
14
|
+
return null;
|
|
15
|
+
}
|
|
16
|
+
return readAccount(prefs);
|
|
17
|
+
}
|
|
18
|
+
// `sync.gaia_id` is the account signed in now, cleared at sign-out;
|
|
19
|
+
// `google.services.last_gaia_id` is the last one that synced. `account_info`
|
|
20
|
+
// lists every Google account signed in on the web as well, so the profile's
|
|
21
|
+
// own is the entry with its Gaia id. The address is lowercased, as mail keys
|
|
22
|
+
// it.
|
|
23
|
+
function readAccount(prefs) {
|
|
24
|
+
const gaiaId = [prefs?.sync?.gaia_id, prefs?.google?.services?.last_gaia_id].find((id) => typeof id === 'string' && id !== '');
|
|
25
|
+
if (!gaiaId)
|
|
26
|
+
return null;
|
|
27
|
+
const info = Array.isArray(prefs?.account_info)
|
|
28
|
+
? prefs.account_info.find((a) => a?.gaia === gaiaId)
|
|
29
|
+
: undefined;
|
|
30
|
+
if (typeof info?.email !== 'string' || info.email === '')
|
|
31
|
+
return null;
|
|
32
|
+
const name = typeof info.full_name === 'string' && info.full_name !== '' ? info.full_name : null;
|
|
33
|
+
return { email: info.email.toLowerCase(), ...(name && { name }) };
|
|
34
|
+
}
|
package/dist/index.d.ts
ADDED
package/dist/index.js
ADDED
package/package.json
CHANGED
|
@@ -1,6 +1,78 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@chronicle.app/chrome",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.4.0",
|
|
4
|
+
"main": "./dist/index.js",
|
|
5
|
+
"types": "./dist/index.d.ts",
|
|
6
|
+
"files": [
|
|
7
|
+
"dist/**/*.js",
|
|
8
|
+
"dist/**/*.d.ts"
|
|
9
|
+
],
|
|
10
|
+
"scripts": {
|
|
11
|
+
"build": "tsc -b tsconfig.lib.json",
|
|
12
|
+
"typecheck": "tsc -p tsconfig.json",
|
|
13
|
+
"lint": "eslint src --ext .ts,.js",
|
|
14
|
+
"test": "node --test src/*.test.js",
|
|
15
|
+
"shapes": "node src/shapes.test.js --update"
|
|
16
|
+
},
|
|
17
|
+
"keywords": [],
|
|
18
|
+
"author": "Andrew Louis",
|
|
19
|
+
"license": "MIT",
|
|
20
|
+
"type": "module",
|
|
21
|
+
"description": "Chronicle Chrome browsing history plugin",
|
|
22
|
+
"devDependencies": {
|
|
23
|
+
"@chronicle.app/etl": "0.4.0",
|
|
24
|
+
"@chronicle.app/schema": "0.4.0",
|
|
25
|
+
"@chronicle.app/tsconfig": "0.4.0",
|
|
26
|
+
"@types/node": "^22.10.1",
|
|
27
|
+
"typescript": "^5.7.2"
|
|
28
|
+
},
|
|
29
|
+
"homepage": "https://chronicle.app",
|
|
30
|
+
"repository": {
|
|
31
|
+
"type": "git",
|
|
32
|
+
"url": "https://github.com/chronicle-app/chronicle.git",
|
|
33
|
+
"directory": "plugins/chrome"
|
|
34
|
+
},
|
|
35
|
+
"publishConfig": {
|
|
36
|
+
"access": "public",
|
|
37
|
+
"registry": "https://registry.npmjs.org/"
|
|
38
|
+
},
|
|
39
|
+
"dependencies": {
|
|
40
|
+
"@chronicle.app/etl-sqlite": "0.4.0",
|
|
41
|
+
"@chronicle.app/google": "0.4.0",
|
|
42
|
+
"zod": "^3.24.1"
|
|
43
|
+
},
|
|
44
|
+
"peerDependencies": {
|
|
45
|
+
"@chronicle.app/etl": ">=0.1.0 <1.0.0",
|
|
46
|
+
"@chronicle.app/schema": ">=0.1.0 <1.0.0"
|
|
47
|
+
},
|
|
48
|
+
"engines": {
|
|
49
|
+
"node": ">=22.13.0"
|
|
50
|
+
},
|
|
51
|
+
"exports": {
|
|
52
|
+
".": {
|
|
53
|
+
"types": "./dist/index.d.ts",
|
|
54
|
+
"import": "./dist/index.js"
|
|
55
|
+
}
|
|
56
|
+
},
|
|
57
|
+
"chronicle": {
|
|
58
|
+
"plugin": true,
|
|
59
|
+
"sources": {
|
|
60
|
+
"chrome": {
|
|
61
|
+
"strategies": {
|
|
62
|
+
"app-db": {
|
|
63
|
+
"delivery": "local",
|
|
64
|
+
"recordTypes": [
|
|
65
|
+
"views"
|
|
66
|
+
],
|
|
67
|
+
"default": true
|
|
68
|
+
}
|
|
69
|
+
},
|
|
70
|
+
"platforms": [
|
|
71
|
+
"darwin",
|
|
72
|
+
"linux",
|
|
73
|
+
"win32"
|
|
74
|
+
]
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
}
|