linkedin-automation-cli 1.0.0

package/docs/superpowers/plans/2026-03-15-company-profile-extraction.md

@@ -0,0 +1,591 @@
+# Company Profile Extraction Implementation Plan
+
+> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add `linkedin-cli company info <url>` command to extract structured data from LinkedIn company profile pages.
+
+**Architecture:** Playwright-based extraction with multi-layer selector fallbacks. No auth required - company pages are public with dismissible authwall popup. Outputs JSON to stdout.
+
+**Tech Stack:** TypeScript, Playwright, Commander.js (existing)
+
+---
+
+## File Structure
+
+| File | Action | Purpose |
+|------|--------|---------|
+| `src/types/index.ts` | Modify | Add `CompanyProfile` interface |
+| `src/linkedin/selectors.ts` | Modify | Add `company` selector category |
+| `src/linkedin/company-extractor.ts` | Create | `CompanyExtractor` class with extraction logic |
+| `src/cli/company.ts` | Create | Command registration (`company info`) |
+| `src/cli/index.ts` | Modify | Export and register `registerCompanyCommands` |
+| `src/linkedin/company-extractor.test.ts` | Create | Unit tests for extractor |
+
+---
+
+## Chunk 1: Types and Selectors
+
+### Task 1: Add CompanyProfile Interface
+
+**Files:**
+- Modify: `src/types/index.ts`
+
+- [ ] **Step 1: Add CompanyProfile interface to types/index.ts**
+
+Add after the `AuditLog` interface (around line 82):
+
+```typescript
+// Company profile types
+export interface CompanyProfile {
+  name: string;
+  linkedin_url: string;
+  website: string | null;
+  industry: string | null;
+  company_size: string | null;
+  headquarters: string | null;
+  founded: string | null;
+  specialties: string[] | null;
+  type: string | null;
+  follower_count: number | null;
+}
+```
+
+- [ ] **Step 2: Verify TypeScript compiles**
+
+Run: `npx tsc --noEmit`
+Expected: No errors
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add src/types/index.ts
+git commit -m "feat: add CompanyProfile interface for company extraction"
+```
+
+---
+
+### Task 2: Add Company Selectors
+
+**Files:**
+- Modify: `src/linkedin/selectors.ts`
+
+- [ ] **Step 1: Add company selector category to SELECTORS**
+
+Add after the `connection` object (before the closing brace, around line 135):
+
+```typescript
+  // Company profile extraction
+  company: {
+    name: [
+      'h1.text-heading-xlarge',
+      '.org-top-card-primary-content h1',
+      'h1.org-top-card-summary__title',
+    ],
+    website: [
+      'a[data-testid="website-link"]',
+      '.org-about-us-module__website a',
+      'dd a[href^="http"]:not([href*="linkedin"])',
+    ],
+    industry: [
+      'dt:has-text("Industry") + dd',
+      '.org-about-company-module__dl dt:has-text("Industry") + dd',
+    ],
+    company_size: [
+      'dt:has-text("Company size") + dd',
+      'dt:has-text("Employees") + dd',
+      '.org-about-company-module__dl dt:has-text("Company size") + dd',
+    ],
+    headquarters: [
+      'dt:has-text("Headquarters") + dd',
+      '.org-about-company-module__dl dt:has-text("Headquarters") + dd',
+    ],
+    founded: [
+      'dt:has-text("Founded") + dd',
+      '.org-about-company-module__dl dt:has-text("Founded") + dd',
+    ],
+    specialties: [
+      'dt:has-text("Specialties") + dd',
+      '.org-about-company-module__dl dt:has-text("Specialties") + dd',
+    ],
+    type: [
+      'dt:has-text("Company type") + dd',
+      'dt:has-text("Type") + dd',
+      '.org-about-company-module__dl dt:has-text("Company type") + dd',
+    ],
+    follower_count: [
+      '.org-top-card-primary-content__followers-count',
+      '.org-top-card-module__followers-count',
+      'span:has-text("followers")',
+      '.org-top-card-summary__followers',
+    ],
+    // Authwall dismiss button
+    dismissAuthwall: [
+      'button[aria-label="Dismiss"]',
+      'button[aria-label="Close"]',
+      '.artdeco-modal__dismiss',
+      'button.artdeco-modal__dismiss',
+    ],
+  },
+```
+
+- [ ] **Step 2: Verify TypeScript compiles**
+
+Run: `npx tsc --noEmit`
+Expected: No errors
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add src/linkedin/selectors.ts
+git commit -m "feat: add company selectors for profile extraction"
+```
+
+---
+
+## Chunk 2: Company Extractor
+
+### Task 3: Create CompanyExtractor Class
+
+**Files:**
+- Create: `src/linkedin/company-extractor.ts`
+- Create: `src/linkedin/company-extractor.test.ts`
+
+- [ ] **Step 1: Write the failing test for URL validation**
+
+Create `src/linkedin/company-extractor.test.ts`:
+
+```typescript
+import { describe, it, expect } from 'vitest';
+import { isValidCompanyUrl } from './company-extractor';
+
+describe('isValidCompanyUrl', () => {
+  it('should accept valid HTTPS company URLs', () => {
+    expect(isValidCompanyUrl('https://www.linkedin.com/company/openai/')).toBe(true);
+    expect(isValidCompanyUrl('https://www.linkedin.com/company/microsoft')).toBe(true);
+    expect(isValidCompanyUrl('https://www.linkedin.com/company/123company/')).toBe(true);
+  });
+
+  it('should reject invalid URLs', () => {
+    expect(isValidCompanyUrl('http://www.linkedin.com/company/openai/')).toBe(false);
+    expect(isValidCompanyUrl('https://linkedin.com/company/openai/')).toBe(false);
+    expect(isValidCompanyUrl('https://www.linkedin.com/in/openai/')).toBe(false);
+    expect(isValidCompanyUrl('not-a-url')).toBe(false);
+    expect(isValidCompanyUrl('')).toBe(false);
+  });
+});
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `npx vitest run src/linkedin/company-extractor.test.ts`
+Expected: FAIL - `isValidCompanyUrl` not defined
+
+- [ ] **Step 3: Create CompanyExtractor with URL validation**
+
+Create `src/linkedin/company-extractor.ts`:
+
+```typescript
+/**
+ * Company Profile Extractor
+ *
+ * Extracts structured data from LinkedIn company profile pages.
+ * No authentication required - company pages are public.
+ */
+
+import type { Page } from 'playwright';
+import type { CompanyProfile } from '../types';
+import { SELECTORS } from './selectors';
+
+/** URL pattern for valid LinkedIn company pages */
+const COMPANY_URL_PATTERN = /^https:\/\/www\.linkedin\.com\/company\/[^\/]+\/?$/;
+
+/**
+ * Validate if a URL is a valid LinkedIn company URL
+ */
+export function isValidCompanyUrl(url: string): boolean {
+  return COMPANY_URL_PATTERN.test(url);
+}
+
+/**
+ * Parse follower count string to number
+ * Handles formats like "2.5M followers", "12K", "1,234"
+ */
+export function parseFollowerCount(text: string | null): number | null {
+  if (!text) return null;
+
+  // Extract numeric part with optional K/M suffix
+  const match = text.match(/([\d,.]+)\s*([KMkm]?)/);
+  if (!match) return null;
+
+  let num = parseFloat(match[1].replace(/,/g, ''));
+  const suffix = match[2].toUpperCase();
+
+  if (suffix === 'K') num *= 1000;
+  else if (suffix === 'M') num *= 1000000;
+
+  return Math.round(num);
+}
+
+/**
+ * Parse specialties string to array
+ * Splits by comma and trims whitespace
+ */
+export function parseSpecialties(text: string | null): string[] | null {
+  if (!text) return null;
+
+  const specialties = text
+    .split(',')
+    .map((s) => s.trim())
+    .filter((s) => s.length > 0);
+
+  return specialties.length > 0 ? specialties : null;
+}
+
+/**
+ * Extract a field using multiple selector fallbacks
+ */
+async function extractField(
+  page: Page,
+  selectors: readonly string[]
+): Promise<string | null> {
+  for (const selector of selectors) {
+    try {
+      const element = page.locator(selector).first();
+      const text = await element.textContent({ timeout: 2000 });
+      if (text?.trim()) {
+        return text.trim();
+      }
+    } catch {
+      // Try next selector
+    }
+  }
+  return null;
+}
+
+/**
+ * Extract href from a link element using selector fallbacks
+ */
+async function extractHref(
+  page: Page,
+  selectors: readonly string[]
+): Promise<string | null> {
+  for (const selector of selectors) {
+    try {
+      const element = page.locator(selector).first();
+      const href = await element.getAttribute('href', { timeout: 2000 });
+      if (href?.trim()) {
+        return href.trim();
+      }
+    } catch {
+      // Try next selector
+    }
+  }
+  return null;
+}
+
+/**
+ * Dismiss authwall popup if present
+ */
+async function dismissAuthwall(page: Page): Promise<void> {
+  const selectors = SELECTORS.company.dismissAuthwall;
+
+  for (const selector of selectors) {
+    try {
+      const btn = page.locator(selector).first();
+      if (await btn.isVisible({ timeout: 1000 })) {
+        await btn.click();
+        await page.waitForTimeout(500);
+        return;
+      }
+    } catch {
+      // Try next selector
+    }
+  }
+}
+
+/**
+ * Company Profile Extractor
+ *
+ * Extracts structured data from LinkedIn company profile pages.
+ */
+export class CompanyExtractor {
+  constructor(private page: Page) {}
+
+  /**
+   * Extract company profile data from a LinkedIn company URL
+   */
+  async extract(url: string): Promise<CompanyProfile> {
+    // Navigate to company page
+    await this.page.goto(url, {
+      waitUntil: 'domcontentloaded',
+      timeout: 30000,
+    });
+
+    // Wait for page to load
+    await this.page.waitForTimeout(3000);
+
+    // Dismiss authwall popup if present
+    await dismissAuthwall(this.page);
+
+    // Extract all fields
+    const name = await extractField(this.page, SELECTORS.company.name);
+    const website = await extractHref(this.page, SELECTORS.company.website);
+    const industry = await extractField(this.page, SELECTORS.company.industry);
+    const company_size = await extractField(this.page, SELECTORS.company.company_size);
+    const headquarters = await extractField(this.page, SELECTORS.company.headquarters);
+    const founded = await extractField(this.page, SELECTORS.company.founded);
+    const specialtiesRaw = await extractField(this.page, SELECTORS.company.specialties);
+    const type = await extractField(this.page, SELECTORS.company.type);
+    const followerRaw = await extractField(this.page, SELECTORS.company.follower_count);
+
+    // Build profile object
+    const profile: CompanyProfile = {
+      name: name || '',
+      linkedin_url: url,
+      website,
+      industry,
+      company_size,
+      headquarters,
+      founded,
+      specialties: parseSpecialties(specialtiesRaw),
+      type,
+      follower_count: parseFollowerCount(followerRaw),
+    };
+
+    return profile;
+  }
+}
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `npx vitest run src/linkedin/company-extractor.test.ts`
+Expected: PASS
+
+- [ ] **Step 5: Write tests for parseFollowerCount**
+
+Add to `src/linkedin/company-extractor.test.ts`:
+
+```typescript
+import { parseFollowerCount, parseSpecialties } from './company-extractor';
+
+describe('parseFollowerCount', () => {
+  it('should parse K suffix', () => {
+    expect(parseFollowerCount('12K followers')).toBe(12000);
+    expect(parseFollowerCount('5k')).toBe(5000);
+  });
+
+  it('should parse M suffix', () => {
+    expect(parseFollowerCount('2.5M followers')).toBe(2500000);
+    expect(parseFollowerCount('1.2m')).toBe(1200000);
+  });
+
+  it('should parse plain numbers with commas', () => {
+    expect(parseFollowerCount('1,234 followers')).toBe(1234);
+    expect(parseFollowerCount('500+ followers')).toBe(500);
+  });
+
+  it('should return null for invalid input', () => {
+    expect(parseFollowerCount(null)).toBe(null);
+    expect(parseFollowerCount('')).toBe(null);
+    expect(parseFollowerCount('no numbers')).toBe(null);
+  });
+});
+
+describe('parseSpecialties', () => {
+  it('should split by comma and trim', () => {
+    expect(parseSpecialties('AI, ML, Research')).toEqual(['AI', 'ML', 'Research']);
+    expect(parseSpecialties('Artificial Intelligence')).toEqual(['Artificial Intelligence']);
+  });
+
+  it('should handle whitespace', () => {
+    expect(parseSpecialties('  AI  ,  ML  ')).toEqual(['AI', 'ML']);
+  });
+
+  it('should return null for empty input', () => {
+    expect(parseSpecialties(null)).toBe(null);
+    expect(parseSpecialties('')).toBe(null);
+    expect(parseSpecialties('   ')).toBe(null);
+  });
+});
+```
+
+- [ ] **Step 6: Run all tests**
+
+Run: `npx vitest run src/linkedin/company-extractor.test.ts`
+Expected: All tests PASS
+
+**Note:** Integration tests for `CompanyExtractor.extract()` that mock Playwright pages are out of scope for this implementation. The extraction logic will be validated via manual testing (Task 5).
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/linkedin/company-extractor.ts src/linkedin/company-extractor.test.ts
+git commit -m "feat: add CompanyExtractor class with parsing utilities"
+```
+
+---
+
+## Chunk 3: CLI Command
+
+### Task 4: Create Company Command
+
+**Files:**
+- Create: `src/cli/company.ts`
+- Modify: `src/cli/index.ts`
+
+- [ ] **Step 1: Create company.ts command file**
+
+Create `src/cli/company.ts`:
+
+```typescript
+import { Command } from 'commander';
+import chalk from 'chalk';
+import { BrowserController } from '../core/browser';
+import { CompanyExtractor, isValidCompanyUrl } from '../linkedin/company-extractor';
+import { getConfig } from '../core/config';
+import { getAuditLogger } from '../core/audit';
+
+export function registerCompanyCommands(program: Command): void {
+  const company = program
+    .command('company')
+    .description('Extract information from LinkedIn company profiles');
+
+  company
+    .command('info')
+    .description('Extract company profile information')
+    .argument('<url>', 'LinkedIn company URL (e.g., https://www.linkedin.com/company/openai/)')
+    .option('--headless', 'Run browser in headless mode', false)
+    .option('--debug', 'Enable debug mode with screenshots and HTML snapshots', false)
+    .action(async (url, options) => {
+      const config = getConfig();
+      const logger = getAuditLogger();
+      let browser: BrowserController | null = null;
+
+      try {
+        // Validate URL
+        if (!isValidCompanyUrl(url)) {
+          console.error(chalk.red('✗ Invalid company URL'));
+          console.log(chalk.gray('Expected format: https://www.linkedin.com/company/<name>/'));
+          process.exit(1);
+        }
+
+        console.error(chalk.gray(`Extracting company info from ${url}...`));
+
+        // Launch browser
+        // Note: --debug flag enables screenshots and HTML snapshots via BrowserController
+        browser = new BrowserController({
+          headless: options.headless !== undefined ? options.headless : config.getValue('headless'),
+          debug: options.debug,
+        });
+
+        await browser.launch();
+        const page = browser.getPage();
+
+        if (!page) {
+          throw new Error('Failed to get page from browser');
+        }
+
+        // Extract company profile
+        const extractor = new CompanyExtractor(page);
+        const profile = await extractor.extract(url);
+
+        // Log successful extraction
+        const fieldsExtracted = Object.keys(profile).filter(k => profile[k as keyof typeof profile] !== null);
+        logger.log('company.info', { url, fieldsExtracted }, true);
+
+        // Output JSON to stdout
+        console.log(JSON.stringify(profile, null, 2));
+
+        await browser.close();
+      } catch (error) {
+        if (browser) {
+          await browser.close();
+        }
+
+        // Log failed extraction
+        logger.log('company.info', { url, error: error instanceof Error ? error.message : String(error) }, false);
+
+        console.error(
+          chalk.red('✗ Failed to extract company data:'),
+          error instanceof Error ? error.message : error
+        );
+        process.exit(1);
+      }
+    });
+}
+```
+
+- [ ] **Step 2: Verify TypeScript compiles**
+
+Run: `npx tsc --noEmit`
+Expected: No errors
+
+- [ ] **Step 3: Register command in CLI index**
+
+Read `src/cli/index.ts` and add the import and registration.
+
+First, check what's in the file:
+```bash
+cat src/cli/index.ts
+```
+
+Then add:
+1. Import: `import { registerCompanyCommands } from './company';`
+2. Call: `registerCompanyCommands(program);` (after other register calls)
+
+- [ ] **Step 4: Build the CLI**
+
+Run: `npm run build`
+Expected: Build succeeds
+
+- [ ] **Step 5: Test the command manually**
+
+Run: `node dist/cli/index.js company info https://www.linkedin.com/company/openai/`
+Expected: JSON output with company data
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/cli/company.ts src/cli/index.ts
+git commit -m "feat: add company info CLI command"
+```
+
+---
+
+### Task 5: Final Integration Test
+
+**Files:**
+- None (manual testing)
+
+- [ ] **Step 1: Test with OpenAI company page**
+
+Run: `node dist/cli/index.js company info https://www.linkedin.com/company/openai/`
+Expected: JSON with OpenAI data
+
+- [ ] **Step 2: Test with Microsoft company page**
+
+Run: `node dist/cli/index.js company info https://www.linkedin.com/company/microsoft/`
+Expected: JSON with Microsoft data
+
+- [ ] **Step 3: Test invalid URL**
+
+Run: `node dist/cli/index.js company info https://www.linkedin.com/in/openai/`
+Expected: Error message "Invalid company URL"
+
+- [ ] **Step 4: Test headless mode**
+
+Run: `node dist/cli/index.js company info https://www.linkedin.com/company/openai/ --headless`
+Expected: JSON output (browser runs headless)
+
+---
+
+## Summary
+
+| Task | Description | Files |
+|------|-------------|-------|
+| 1 | Add CompanyProfile interface | `src/types/index.ts` |
+| 2 | Add company selectors | `src/linkedin/selectors.ts` |
+| 3 | Create CompanyExtractor class | `src/linkedin/company-extractor.ts`, `src/linkedin/company-extractor.test.ts` |
+| 4 | Create company CLI command | `src/cli/company.ts`, `src/cli/index.ts` |
+| 5 | Integration testing | Manual |