birdclaw 0.3.0 → 0.4.0

package/CHANGELOG.md

@@ -2,6 +2,25 @@
 
 ## Unreleased
 
+## 0.4.0 - 2026-05-09
+
+### Added
+
+- Preserve richer X profile metadata during `bird`/`xurl` profile hydration, including profile URLs, profile URL entities, locations, verification type, raw profile JSON, and X affiliation/highlighted-label metadata.
+- Add first-class `profile_affiliations` storage, backup/export/import support, and `whois --json` `profileEvidence` so identity lookups can explain whether a match came from bio text, profile URLs, affiliation badges, DM context, or expanded links.
+- Add profile-change snapshots for hydrated profiles, preserving prior bio/profile URL/location/verification/affiliation states so identity searches can surface current and previous affiliation evidence.
+- Add first-class bio entity extraction for profile bios and profile URLs, including `@handle`, domain, and company-phrase evidence used by fuzzy identity searches such as `whois "blacksmith guy"`.
+- Add a derived `identity_search_index` and `whois` filters for affiliation-oriented identity lookups: `--affiliation`, `--current-affiliation`, and `--exclude-domain-only`.
+
+### Changed
+
+- Use Node's native `node:sqlite` runtime instead of `better-sqlite3`, removing the native npm dependency while preserving the existing synchronous SQLite API surface.
+- Allow Node 26.x in the package engine range and update install docs for the native SQLite runtime.
+- Improve DM `whois` ranking with Sweetistics-style profile evidence scoring: profile URLs and affiliation badges now boost relevant candidates, while cached profile and URL lookups still avoid repeated API/network work.
+- Resolve synthetic X highlighted-label organization badges into real local organization profile ids when `bird` can hydrate the org handle.
+- Rank current affiliation and bio identity evidence above plain profile domains in `whois`, group human output into ambiguity buckets, and explain "why this person?" with the strongest typed evidence first.
+- Use `bird profiles --json` for batch profile hydration when available, falling back to single-profile `bird user --profile-only --json`.
+
 ## 0.3.0 - 2026-05-05
 
 ### Added

package/README.md

@@ -17,7 +17,7 @@ Status: WIP. Real and usable. Not done. Expect schema churn, transport gaps, and
 
 ### Local data + storage
 
-- one shared SQLite DB for multiple accounts
+- one shared SQLite DB for multiple accounts, with canonical tweets/profiles and account-scoped timeline/collection edges
 - FTS5 search over tweets and DMs
 - archive autodiscovery on macOS
 - archive import for tweets, likes, profiles, and full DMs
@@ -25,6 +25,7 @@ Status: WIP. Real and usable. Not done. Expect schema churn, transport gaps, and
 - live likes and bookmarks sync through `xurl` or `bird`
 - Git-friendly text backups with yearly tweet shards and per-conversation DM shards
 - profile hydration from live Twitter metadata
+- profile-change history, affiliation badge edges, and extracted bio entities for local identity lookups
 - local avatar cache
 - local media cache root under `~/.birdclaw`
 
@@ -112,7 +113,7 @@ export BIRDCLAW_HOME=/path/to/custom/root
 
 ## Requirements
 
-- Node `25.8.1`
+- Node `25.8.1` or Node 26.x
 - `pnpm`
 - macOS recommended for Spotlight archive discovery
 - `xurl` optional for live reads / writes
@@ -259,7 +260,7 @@ Notes:
 - `actions.transport` accepts `auto`, `bird`, or `xurl`
 - `bird` mode uses your local `bird` CLI and caches its mentions output into birdclaw's canonical store
 - filters still work in `xurl` mode; filtered payloads are rebuilt from the local canonical store after sync
-- `sync likes`, `sync bookmarks`, `sync timeline`, and `sync mention-threads` store live results in the same local timeline table, so `search tweets`, `search tweets --liked`, and `search tweets --bookmarked` work across archive and live data
+- `sync likes`, `sync bookmarks`, `sync timeline`, and `sync mention-threads` store live results in the canonical local store; per-account home/mention/like/bookmark membership is kept as edges so shared tweets do not clobber account ownership
 
 ### Research bookmarks and threads
 
@@ -275,11 +276,33 @@ birdclaw research --account acct_primary --out ~/research/codex.md
 ```bash
 pnpm cli search dms "prototype" --json
 pnpm cli search dms "layout" --min-followers 1000 --min-influence-score 120 --sort influence --json
+pnpm cli search dms "blacksmith" --context 4 --resolve-profiles --expand-urls --no-xurl-fallback --json
+pnpm cli whois "blacksmith guy" --context 4 --no-xurl-fallback --json
+pnpm cli whois "github guy" --current-affiliation github --exclude-domain-only --no-xurl-fallback
+pnpm cli whois "blacksmith" --tweets --context 4 --no-xurl-fallback --json
 pnpm cli dms sync --limit 50 --refresh --json
 pnpm cli dms list --refresh --limit 10 --json
 pnpm cli dms list --unreplied --min-followers 500 --min-influence-score 90 --sort influence --json
 ```
 
+`--resolve-profiles` fills archive-imported numeric DM profiles through the local
+cache first, then `bird`, then `xurl` unless `--no-xurl-fallback` is set.
+Resolved profiles keep bio, location, profile URL, verification type, structured
+URL entities, raw profile JSON, and any X affiliation badge metadata Birdclaw can
+see. When a highlighted-label badge only gives a synthetic label plus handle,
+Birdclaw tries to hydrate that org handle through `bird` and rewrites the edge to
+the real local organization profile id. Profile changes are snapshotted over
+time, and bios are indexed for `@handles`, domains, and company phrases.
+`whois` uses that profile context plus DM context and cached URL expansion to
+return typed evidence such as `profile_bio`, `profile_url`, `profile_bio_url`,
+`affiliation`, `bio_handle`, `bio_domain`, `bio_company`, `profile_history`,
+`dm_context`, and `expanded_url`. It keeps a derived `identity_search_index`
+for fast local profile-evidence lookups, ranks current affiliation and bio
+identity evidence above plain domains, and groups human output into likely
+affiliated, ecosystem, link-only, and DM-context buckets. Use
+`--affiliation`, `--current-affiliation`, and `--exclude-domain-only` when you
+want "GitHub people" rather than anyone with a `github.com` link.
+
 ### AI inbox
 
 ```bash
@@ -377,8 +400,13 @@ Layout:
 manifest.json
 data/accounts.jsonl
 data/profiles.jsonl
+data/profile_affiliations.jsonl
+data/profile_snapshots.jsonl
+data/profile_bio_entities.jsonl
 data/tweets/YYYY.jsonl
 data/tweets/unknown.jsonl
+data/timeline_edges/home.jsonl
+data/timeline_edges/mention.jsonl
 data/collections/likes.jsonl
 data/collections/bookmarks.jsonl
 data/dms/conversations.jsonl
@@ -387,7 +415,7 @@ data/moderation/blocks.jsonl
 data/moderation/mutes.jsonl
 ```
 
-Tweets are sharded by year for human browsing and yearly analysis. Collection-only tweets whose real creation date is unknown go into `data/tweets/unknown.jsonl` instead of pretending they belong to 1970. DMs are sharded by year with `conversation_id` in each row; this keeps Git fast while preserving conversation membership. Likes and bookmarks are stored as collection edges in `data/collections` and mirrored into the timeline rows for current query compatibility.
+Tweets are sharded by year for human browsing and yearly analysis. Collection-only tweets whose real creation date is unknown go into `data/tweets/unknown.jsonl` instead of pretending they belong to 1970. Timeline membership is stored in `data/timeline_edges`; likes and bookmarks are stored as account-scoped collection edges in `data/collections`. DMs are sharded by year with `conversation_id` in each row; this keeps Git fast while preserving conversation membership.
 
 Use `backup sync` when the target is a private Git repo. It pulls first, merge-imports the remote backup into local SQLite, exports the local union back into text shards, commits, and pushes.
 
@@ -477,11 +505,11 @@ pnpm cli auth status --json
 ## Testing
 
 ```bash
-fnm exec --using 25.8.1 pnpm check
-fnm exec --using 25.8.1 pnpm test
-fnm exec --using 25.8.1 pnpm coverage
-fnm exec --using 25.8.1 pnpm build
-fnm exec --using 25.8.1 pnpm e2e
+pnpm check
+pnpm test
+pnpm coverage
+pnpm build
+pnpm e2e
 ```
 
 Current bar:

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "birdclaw",
-	"version": "0.3.0",
+	"version": "0.4.0",
 	"description": "Local Twitter memory in SQLite for archives, DMs, likes, bookmarks, and moderation",
 	"homepage": "https://github.com/steipete/birdclaw#readme",
 	"license": "MIT",
@@ -46,7 +46,8 @@
 		"format:check": "oxfmt --check src scripts playwright vite.config.ts vitest.config.ts playwright.config.ts",
 		"lint": "oxlint --import-plugin --node-plugin --vitest-plugin --deny-warnings -A require-mock-type-parameters -A no-control-regex src scripts playwright vite.config.ts vitest.config.ts playwright.config.ts",
 		"check": "pnpm run format:check && pnpm run lint",
-		"cli": "tsx src/cli.ts"
+		"cli": "tsx src/cli.ts",
+		"docs:site": "node scripts/build-docs-site.mjs"
 	},
 	"dependencies": {
 		"@steipete/sweet-cookie": "^0.2.0",
@@ -59,7 +60,6 @@
 		"@tanstack/react-start": "1.167.61",
 		"@tanstack/router-plugin": "^1.167.32",
 		"@vitejs/plugin-react": "^6.0.1",
-		"better-sqlite3": "^12.9.0",
 		"commander": "^14.0.3",
 		"kysely": "^0.28.17",
 		"lucide-react": "^1.14.0",
@@ -76,7 +76,6 @@
 		"@testing-library/dom": "^10.4.1",
 		"@testing-library/jest-dom": "^6.9.1",
 		"@testing-library/react": "^16.3.2",
-		"@types/better-sqlite3": "^7.6.13",
 		"@types/node": "^25.6.0",
 		"@types/react": "^19.2.14",
 		"@types/react-dom": "^19.2.3",
@@ -89,12 +88,11 @@
 		"vitest": "^4.1.5"
 	},
 	"engines": {
-		"node": ">=25.8.1 <26"
+		"node": ">=25.8.1 <27"
 	},
 	"packageManager": "pnpm@10.31.0",
 	"pnpm": {
 		"onlyBuiltDependencies": [
-			"better-sqlite3",
 			"esbuild",
 			"lightningcss"
 		]

package/src/cli.ts

@@ -36,6 +36,7 @@ import {
 	exportMentionsViaCachedXurl,
 } from "#/lib/mentions-live";
 import { hydrateProfilesFromX } from "#/lib/profile-hydration";
+import { resolveProfilesForIds } from "#/lib/profile-resolver";
 import { inspectProfileReplies } from "#/lib/profile-replies";
 import { runResearchMode } from "#/lib/research";
 import {
@@ -51,6 +52,8 @@ import {
 	type TimelineCollectionMode,
 } from "#/lib/timeline-collections-live";
 import { syncHomeTimeline } from "#/lib/timeline-live";
+import { expandUrlsFromTexts } from "#/lib/url-expansion";
+import { formatWhois, runWhois } from "#/lib/whois";
 
 const program = new Command();
 const packageRoot = dirname(dirname(fileURLToPath(import.meta.url)));
@@ -101,6 +104,56 @@ function resolveActionOptions(options: { transport?: string }) {
 	};
 }
 
+async function enrichDmItems(
+	query: Parameters<typeof listDmConversations>[0],
+	options: {
+		resolveProfiles?: boolean;
+		expandUrls?: boolean;
+		refreshProfileCache?: boolean;
+		refreshUrlCache?: boolean;
+		xurlFallback?: boolean;
+	},
+) {
+	let items = listDmConversations(query);
+	const profileResolution = options.resolveProfiles
+		? await resolveProfilesForIds(
+				items.map((item) => item.participant.id),
+				{
+					refresh: options.refreshProfileCache,
+					xurlFallback: options.xurlFallback ?? true,
+				},
+			)
+		: undefined;
+	if (profileResolution) {
+		items = listDmConversations(query);
+	}
+
+	const urlExpansions = options.expandUrls
+		? await expandUrlsFromTexts(
+				items.flatMap((item) => [
+					item.lastMessagePreview,
+					item.searchSnippet ?? "",
+					...(item.matches ?? []).flatMap((match) => [
+						...match.before.map((message) => message.text),
+						match.message.text,
+						...match.after.map((message) => message.text),
+					]),
+				]),
+				{ refresh: options.refreshUrlCache },
+			)
+		: undefined;
+
+	if (!profileResolution && !urlExpansions) {
+		return items;
+	}
+
+	return {
+		items,
+		...(profileResolution ? { profileResolution } : {}),
+		...(urlExpansions ? { urlExpansions } : {}),
+	};
+}
+
 async function autoUpdateBeforeRead() {
 	const result = await maybeAutoUpdateBackup();
 	if (!result.ok) {
@@ -251,17 +304,37 @@ searchCommand
 	.option("--min-influence-score <n>", "Minimum derived influence score")
 	.option("--max-influence-score <n>", "Maximum derived influence score")
 	.option("--sort <mode>", "recent or influence", "recent")
+	.option(
+		"--context <n>",
+		"Include N messages before and after each match",
+		"0",
+	)
+	.option(
+		"--resolve-profiles",
+		"Resolve placeholder DM profiles through cache/bird/xurl",
+	)
+	.option("--expand-urls", "Expand URLs through the persistent URL cache")
+	.option("--refresh-profile-cache", "Bypass profile lookup cache")
+	.option("--refresh-url-cache", "Bypass URL expansion cache")
+	.option(
+		"--no-xurl-fallback",
+		"Do not fall back to xurl after bird profile lookup",
+	)
 	.option("--replied", "Only replied threads")
 	.option("--unreplied", "Only unreplied threads")
 	.option("--limit <n>", "Limit results", "20")
 	.action(async (query, options) => {
 		await autoUpdateBeforeRead();
+		const context = parseNonNegativeIntegerOption(options.context, "--context");
+		if (context === undefined) {
+			return;
+		}
 		const replyFilter = options.replied
 			? "replied"
 			: options.unreplied
 				? "unreplied"
 				: "all";
-		const items = listDmConversations({
+		const dmQuery = {
 			search: query,
 			participant: options.participant,
 			minFollowers: options.minFollowers
@@ -278,11 +351,74 @@ searchCommand
 				: undefined,
 			sort: options.sort === "influence" ? "influence" : "recent",
 			replyFilter,
+			context,
 			limit: Number(options.limit),
+		} as const;
+		const items = await enrichDmItems(dmQuery, {
+			resolveProfiles: Boolean(options.resolveProfiles),
+			expandUrls: Boolean(options.expandUrls),
+			refreshProfileCache: Boolean(options.refreshProfileCache),
+			refreshUrlCache: Boolean(options.refreshUrlCache),
+			xurlFallback: options.xurlFallback,
 		});
 		print(items, program.opts().json ?? false);
 	});
 
+program
+	.command("whois <query>")
+	.description("Identify likely people or orgs from local DMs and tweets")
+	.option("--account <accountId>", "Account id")
+	.option("--no-dms", "Do not search DMs")
+	.option("--tweets", "Include local tweet search evidence")
+	.option("--no-resolve-profiles", "Do not resolve placeholder profiles")
+	.option("--no-expand-urls", "Do not expand URLs")
+	.option("--refresh-profile-cache", "Bypass profile lookup cache")
+	.option("--refresh-url-cache", "Bypass URL expansion cache")
+	.option(
+		"--no-xurl-fallback",
+		"Do not fall back to xurl after bird profile lookup",
+	)
+	.option(
+		"--affiliation <query>",
+		"Require affiliation, bio, or history evidence",
+	)
+	.option(
+		"--current-affiliation <query>",
+		"Require an active affiliation badge",
+	)
+	.option(
+		"--exclude-domain-only",
+		"Drop candidates that only match domains/URLs",
+	)
+	.option("--context <n>", "DM messages before and after each match", "4")
+	.option("--limit <n>", "Limit candidates", "10")
+	.action(async (query, options) => {
+		await autoUpdateBeforeRead();
+		const context = parseNonNegativeIntegerOption(options.context, "--context");
+		if (context === undefined) {
+			return;
+		}
+		const result = await runWhois(query, {
+			account: options.account,
+			dms: options.dms,
+			tweets: Boolean(options.tweets),
+			resolveProfiles: options.resolveProfiles,
+			expandUrls: options.expandUrls,
+			refreshProfileCache: Boolean(options.refreshProfileCache),
+			refreshUrlCache: Boolean(options.refreshUrlCache),
+			xurlFallback: options.xurlFallback,
+			affiliation: options.affiliation,
+			currentAffiliation: options.currentAffiliation,
+			excludeDomainOnly: Boolean(options.excludeDomainOnly),
+			context,
+			limit: Number(options.limit),
+		});
+		print(
+			program.opts().json ? result : formatWhois(result),
+			program.opts().json ?? false,
+		);
+	});
+
 program
 	.command("research [query]")
 	.description("Build a markdown research brief from bookmarked threads")
@@ -547,6 +683,17 @@ dmsCommand
 	.option("--min-influence-score <n>", "Minimum derived influence score")
 	.option("--max-influence-score <n>", "Maximum derived influence score")
 	.option("--sort <mode>", "recent or influence", "recent")
+	.option(
+		"--resolve-profiles",
+		"Resolve placeholder DM profiles through cache/bird/xurl",
+	)
+	.option("--expand-urls", "Expand URLs through the persistent URL cache")
+	.option("--refresh-profile-cache", "Bypass profile lookup cache")
+	.option("--refresh-url-cache", "Bypass URL expansion cache")
+	.option(
+		"--no-xurl-fallback",
+		"Do not fall back to xurl after bird profile lookup",
+	)
 	.option("--replied", "Only replied threads")
 	.option("--unreplied", "Only unreplied threads")
 	.option("--limit <n>", "Limit results", "20")
@@ -567,25 +714,34 @@ dmsCommand
 		} else {
 			await autoUpdateBeforeRead();
 		}
-		const items = listDmConversations({
-			account: options.account,
-			participant: options.participant,
-			minFollowers: options.minFollowers
-				? Number(options.minFollowers)
-				: undefined,
-			maxFollowers: options.maxFollowers
-				? Number(options.maxFollowers)
-				: undefined,
-			minInfluenceScore: options.minInfluenceScore
-				? Number(options.minInfluenceScore)
-				: undefined,
-			maxInfluenceScore: options.maxInfluenceScore
-				? Number(options.maxInfluenceScore)
-				: undefined,
-			sort: options.sort === "influence" ? "influence" : "recent",
-			replyFilter,
-			limit: Number(options.limit),
-		});
+		const items = await enrichDmItems(
+			{
+				account: options.account,
+				participant: options.participant,
+				minFollowers: options.minFollowers
+					? Number(options.minFollowers)
+					: undefined,
+				maxFollowers: options.maxFollowers
+					? Number(options.maxFollowers)
+					: undefined,
+				minInfluenceScore: options.minInfluenceScore
+					? Number(options.minInfluenceScore)
+					: undefined,
+				maxInfluenceScore: options.maxInfluenceScore
+					? Number(options.maxInfluenceScore)
+					: undefined,
+				sort: options.sort === "influence" ? "influence" : "recent",
+				replyFilter,
+				limit: Number(options.limit),
+			},
+			{
+				resolveProfiles: Boolean(options.resolveProfiles),
+				expandUrls: Boolean(options.expandUrls),
+				refreshProfileCache: Boolean(options.refreshProfileCache),
+				refreshUrlCache: Boolean(options.refreshUrlCache),
+				xurlFallback: options.xurlFallback,
+			},
+		);
 		print(items, program.opts().json ?? false);
 	});
 

package/src/lib/archive-import.ts

@@ -286,6 +286,7 @@ function clearImportedData() {
 	db.exec(`
     delete from ai_scores;
     delete from tweet_actions;
+    delete from tweet_account_edges;
     delete from tweet_collections;
     delete from dm_fts;
     delete from tweets_fts;
@@ -776,6 +777,16 @@ export async function importArchive(
 	const insertTweetFts = db.prepare(
 		"insert into tweets_fts (tweet_id, text) values (?, ?)",
 	);
+	const insertTimelineEdge = db.prepare(`
+    insert into tweet_account_edges (
+      account_id, tweet_id, kind, first_seen_at, last_seen_at, seen_count, source,
+      raw_json, updated_at
+    ) values (?, ?, ?, ?, ?, 1, 'archive', '{}', ?)
+    on conflict(account_id, tweet_id, kind) do update set
+      first_seen_at = min(tweet_account_edges.first_seen_at, excluded.first_seen_at),
+      last_seen_at = max(tweet_account_edges.last_seen_at, excluded.last_seen_at),
+      updated_at = max(tweet_account_edges.updated_at, excluded.updated_at)
+  `);
 	const insertCollection = db.prepare(`
     insert into tweet_collections (
       account_id, tweet_id, kind, collected_at, source, raw_json, updated_at
@@ -842,6 +853,16 @@ export async function importArchive(
 				tweet.mediaJson,
 				tweet.quotedTweetId,
 			);
+			if (tweet.kind === "home") {
+				insertTimelineEdge.run(
+					"acct_primary",
+					tweet.id,
+					tweet.kind,
+					tweet.createdAt,
+					tweet.createdAt,
+					new Date().toISOString(),
+				);
+			}
 			insertTweetFts.run(tweet.id, tweet.text);
 		}
 
@@ -905,8 +926,11 @@ export async function importArchive(
 }
 
 export const __test__ = {
+	normalizeArchivePath,
 	extractArchiveJson,
 	parseArchiveArray,
+	getFirstEntry,
+	getMatchingEntries,
 	parseTwitterDate,
 	asRecord,
 	asArray,
@@ -915,6 +939,7 @@ export const __test__ = {
 	getTweetMediaCount,
 	extractTweetEntities,
 	extractTweetMedia,
+	extractCollectionTweet,
 	buildAccountPayload,
 	inferProfileFromDirectory,
 };