eggi-ai-db-schema-2 12.54.2

package/CHANGELOG.md

@@ -0,0 +1,750 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [12.13.0] - 2025-11-18
+
+### Added
+
+- **Organizations Array in Mapping Jobs Response**: Added `organizations` array to `OnDemandMappingJobMinimal` interface
+  - Each mapping job now includes which organizations it was exported to
+  - Supports multiple organizations per mapping job
+  - Includes organization `id`, `name`, and `workspaceType` (INDIVIDUALS or BUSINESS)
+  - Enables frontend to display "Exported to Personal Workspace" or "Exported to [Business Name]"
+
+### Changed
+
+- **Mapping Jobs Query**: Updated `getOnDemandMappingJobsMinimal()` to join with `organization_assignment_jobs` and `organizations` tables
+  - Returns all unique organizations a mapping job was exported to
+  - Handles multiple organization assignments per mapping job
+
+## [12.12.0] - 2025-11-18
+
+### Added
+
+- **Unlimited Quota Support**: Added support for unlimited quotas using `null` instead of `-1`
+  - `quota_limit` column now allows `NULL` values (migration `0047_make_quota_limit_nullable_for_unlimited.sql`)
+  - `null` quotaLimit indicates unlimited (PRO_VERSION with active subscription, not trialing)
+  - Usage is still tracked even for unlimited quotas
+
+### Changed
+
+- **Quota System**: Updated quota operations to always return both daily and monthly quotas for all organizations
+  - `getAllQuotasForUser()` now ensures both daily and monthly quotas are returned for every organization
+  - Creates default quota records if they don't exist in the database
+  - `getQuotaLimit()` returns `null` for PRO_VERSION with active subscription status
+  - `checkAndIncrementQuota()` handles `null` quota limits correctly (always allows and increments)
+  - `getQuotaStatus()` returns `null` for remainingQuota and quotaLimit when unlimited
+
+- **UTC-Based Period Calculations**: Fixed timezone issues in quota period calculations
+  - `getDailyPeriod()` and `getMonthlyPeriod()` now use UTC methods (`getUTCFullYear()`, `getUTCMonth()`, `getUTCDate()`, `Date.UTC()`)
+  - Ensures consistent date boundaries regardless of server timezone
+  - Prevents quota period mismatches when server is in different timezone than UTC
+
+### Migration
+
+- **Migration File**: `0047_make_quota_limit_nullable_for_unlimited.sql`
+  - Makes `quota_limit` column nullable in `monitoring.mapping_job_quotas` table
+  - Updates column comment to reflect that NULL means unlimited
+
+## [Unreleased]
+
+### Added
+
+- **Ghost Genius Provider Support**: Added support for Ghost Genius LinkedIn profiles alongside Forager
+  - New module: `ghost-genius-linkedin-operations.ts` with `findLinkedInAccountByAcoa()` and `findOrCreateLinkedInAccountByAcoa()`
+  - New function: `storeCompleteLinkedInProfileByAcoa()` for Ghost Genius profile storage
+  - Database migration: `0031_make_forager_id_optional_for_ghost_genius.sql` makes `forager_id` nullable
+  - Updated constraints to allow `forager_id` to be null while ensuring at least one identifier exists
+  - Separate cache tables and operations for Forager and Ghost Genius providers
+
+### Changed
+
+- **Database Schema**: `linkedin.accounts.forager_id` is now nullable to support Ghost Genius profiles
+- **LinkedIn Operations**: Separated Forager and Ghost Genius operations into distinct modules for better modularity
+  - `forager-linkedin-operations.ts`: Forager-specific operations (uses `forager_id`)
+  - `ghost-genius-linkedin-operations.ts`: Ghost Genius-specific operations (uses ACoA identifier)
+- **Type Exports**: Updated exports to include Ghost Genius operations while maintaining backward compatibility
+
+### Migration
+
+- **Migration File**: `0031_make_forager_id_optional_for_ghost_genius.sql`
+  - Makes `forager_id` column nullable
+  - Updates unique constraint to allow multiple NULL values
+  - Updates identifier check constraint to allow `forager_id` to be NULL
+
+## [11.14.7] - 2025-10-20
+
+- chore: minor docs touch to trigger CI publish
+
+## [11.12.0] - 2025-10-18
+
+### Added
+
+- **Organization Assignment Jobs Table**: New `monitoring.organization_assignment_jobs` table for tracking organization assignment job lifecycle
+  - `id`: Serial primary key
+  - `mapping_job_id`: Foreign key to mapping jobs (cascade delete)
+  - `organization_id`: Foreign key to organizations (cascade delete)
+  - `created_at`: Timestamp when job was created
+  - `started_at`: Timestamp when job started processing (defaults to created_at)
+  - `completed_at`: Timestamp when job completed (nullable)
+  - `metadata`: JSONB column for job statistics (relationships_removed, relationships_added, contributor_role_action, error_details)
+  - Indexes on mapping_job_id, organization_id, created_at, completed_at for query performance
+  - Comments on table and columns for documentation
+
+- **Organization Assignment Job Operations**: New utility functions in `organization-assignment-job-operations.ts`
+  - `createOrganizationAssignmentJob()`: Create new job record when assignment starts
+  - `completeOrganizationAssignmentJob()`: Mark job as completed with statistics
+  - `updateOrganizationAssignmentJobMetadata()`: Update job metadata during processing
+  - `getOrganizationAssignmentJobsForMappingJob()`: Query jobs by mapping job ID
+  - `getOrganizationAssignmentJobsForOrganization()`: Query jobs by organization ID
+  - `getPendingOrganizationAssignmentJobs()`: Get incomplete jobs
+  - `getOrganizationAssignmentJobById()`: Get specific job by ID
+  - `getLatestOrganizationAssignmentJob()`: Get most recent job for mapping+org combo
+  - `validateOrganizationAssignmentJobCompletion()`: Check if job is complete
+  - `deleteOrganizationAssignmentJob()`: Delete job record
+
+- **TypeScript Types**: Added `OrganizationAssignmentJob` and `NewOrganizationAssignmentJob` types
+- **Relations**: Added relations between organization_assignment_jobs ↔ mapping_jobs and organizations
+- **Migration**: Manual migration file `0021_add_organization_assignment_jobs_table.sql`
+
+### Purpose
+
+This table provides observability for the `organization_assignments_handler` Lambda (formerly `mapping_job_completion_handler`). Each record tracks:
+
+- When an assignment job runs
+- Which organization is being updated
+- How many relationships were removed (old assignments cleared)
+- How many relationships were added (new assignments created)
+- Whether the contributor was created, promoted, or unchanged
+- Any errors that occurred during processing
+
+This enables monitoring, debugging, and auditing of the organization assignment process.
+
+## [8.13.0] - 2025-10-06
+
+### Added
+
+- **Waitlist Table**: New `public.waitlist` table for capturing email signups
+  - `id`: Serial primary key
+  - `email`: VARCHAR(255) with regex validation (email format check)
+  - `linkedin_public_identifier`: VARCHAR(255) for LinkedIn profile identifier (optional)
+  - `created_at`: Timestamp with timezone, auto-set to current time
+  - Unique constraint on email to prevent duplicates
+  - Email format validation using PostgreSQL regex check constraint
+  - Index on email column for fast lookups
+
+## [8.7.0] - 2025-10-02
+
+### 🚀 MAJOR: LinkedIn Schema Migration Complete
+
+**Major Enhancement**: Successfully migrated `public.relationship_scores` to `linkedin.relationships` with zero data loss and full backward compatibility.
+
+#### Added
+
+- **New Schema Structure**: `linkedin.relationships` table with proper LinkedIn-scoped relationships
+- **Enhanced Type Safety**: Updated TypeScript types (`LinkedinRelationship`, `NewLinkedinRelationship`)
+- **Improved Schema Organization**: Relationships properly scoped to LinkedIn platform
+- **Comprehensive Documentation**: Complete migration guide and field migration documentation
+
+#### Changed
+
+- **BREAKING**: `public.relationship_scores` table migrated to `linkedin.relationships`
+- **Field Names**: Updated relationship table to use `linkedin_account_id_a` and `linkedin_account_id_b`
+- **Schema References**: All Drizzle schema definitions updated to use `linkedinRelationships`
+- **Sync Scripts**: All production-to-dev sync operations updated for new schema
+- **Documentation**: Updated all API examples and schema documentation
+
+#### Migration Details
+
+- **Data Migration**: Successfully migrated **45,847 relationship records** with zero data loss
+- **Performance**: Optimized migration completed in ~2 minutes vs 20+ minutes initial approach
+- **Foreign Keys**: All constraints properly updated and maintained
+- **Backward Compatibility**: API function names preserved (e.g., `getTopRelationshipScoresForMappingJob`)
+
+#### Technical Changes
+
+- **Schema Layer**: Updated `schema.ts` with `linkedinRelationships` table definition
+- **Relations**: Updated all Drizzle relations to use correct table references
+- **Imports**: All imports updated from `relationshipScores` to `linkedinRelationships`
+- **Sync Operations**: Updated all prod-to-dev sync scripts to use `linkedin.relationships`
+- **Validation**: Updated all validation and utility scripts
+
+#### Files Updated
+
+- ✅ **Schema Definition**: `src/lib/schema.ts` with new table structure
+- ✅ **Sync Scripts**: All scripts in `scripts/sync-prod-to-dev/` updated
+- ✅ **Documentation**: `README.md`, `docs/API.md`, `docs/FIELD_MIGRATION_GUIDE.md`
+- ✅ **Migration Guide**: Complete `LINKEDIN_SCHEMA_MIGRATION.md` with step-by-step process
+- ✅ **Lambda Functions**: All application code maintains API compatibility
+
+#### Verification
+
+- ✅ **Zero Data Loss**: All 45,847 records successfully migrated
+- ✅ **Foreign Key Integrity**: All relationships maintained and working
+- ✅ **API Compatibility**: Existing consumers unaffected by changes
+- ✅ **Performance**: Optimized schema structure with proper indexing
+- ✅ **Documentation**: Complete guides for migration and field changes
+
+**Migration Status**: ✅ **COMPLETE SUCCESS** - Production ready with full backward compatibility
+
+## [6.5.4] - 2025-09-24
+
+### Fixed
+
+- **CRITICAL FIX**: Fixed `getAllOnDemandMappingJobsForUser()` pagination and data retrieval
+  - **ISSUE**: Complex LEFT JOINs were filtering out jobs without complete target profile data
+  - **BEFORE**: Only returned 4 jobs out of 22 total (jobs with complete LinkedIn profile data only)
+  - **AFTER**: Returns ALL 22 jobs with proper pagination support
+  - **SOLUTION**: Refactored to two-step approach:
+    1. Get all on-demand mapping jobs (guaranteed to return all jobs)
+    2. Enrich each job with target profile data (if available)
+  - **IMPACT**: Chrome extension now shows all mapping jobs instead of just the ones with complete profiles
+  - **PERFORMANCE**: More reliable than complex JOINs, handles missing target data gracefully
+  - **PAGINATION**: Fixed limit/offset to work correctly with all jobs
+
+### Changed
+
+- **QUERY ARCHITECTURE**: Improved robustness of on-demand mapping job queries
+  - Jobs without target LinkedIn profiles now display "Unknown User" until data is available
+  - Eliminated data loss from complex JOIN conditions
+  - Better error handling for target profile data fetching
+  - Maintains backwards compatibility with existing API responses
+
+### Technical Details
+
+- Modified `getAllOnDemandMappingJobsForUser()` in `utils/on-demand-mapping-operations.ts`
+- Separated base job fetching from target profile enrichment
+- Added proper TypeScript typing for nullable target profile fields
+- Improved error handling with try/catch for individual target data queries
+
+## [4.25.0] - 2025-09-22
+
+### Changed
+
+- **RELATIONSHIP FILTERING**: Updated relationship queries to use latest mapping job filtering
+  - Modified `findRelationshipsByLinkedInIdentifier()` to filter by highest `mapping_job_id` instead of `model_version`
+  - Updated `findRelationshipsByLinkedInIdentifierPaginated()` to use same filtering logic
+  - Ensures that for each person, only relationships from the most recent mapping job are returned
+  - Uses PostgreSQL `DISTINCT ON` with `mapping_job_id DESC` ordering for optimal performance
+
+### Added
+
+- **NEW FUNCTIONS**: Added convenience functions for latest relationship filtering
+  - `findLatestRelationshipsByLinkedInIdentifier()` - Explicit function for latest mapping job filtering
+  - `getRelationshipVersionStats()` - Provides insights into relationship data versioning per mapping job
+- **ENHANCED SELECTION**: Added `mappingJobId` to query selections for better debugging and tracking
+
+## [4.24.0] - 2025-09-10
+
+### Added
+
+- **VALIDATION**: ACoA identifier validation for `internal_identifier_regular` column
+  - Added `validateAcoIdentifier()` function with comprehensive validation rules
+  - Added `InvalidAcoIdentifierError` custom error class for clear error messages
+  - Validates identifiers must start with "ACoA", be exactly 39 characters, and contain only valid characters
+
+### Changed
+
+- **DATABASE CONSTRAINT**: Updated `internal_identifier_regular` column length from 255 to exactly 39 characters
+- **SCHEMA MIGRATION**: Added `chk_internal_identifier_regular_length` constraint to enforce exact 39-character length
+- **QUERY VALIDATION**: Added validation to all functions that set `internal_identifier_regular`:
+  - `buildLinkedInAccountValues()` validates when mapping identifiers
+  - `upsertLinkedInAccount()` validates incoming ACoA identifiers
+  - `createUserAndLinkedInAccountForLinkedIn()` validates both request and provider IDs
+
+### Fixed
+
+- **DATA CLEANUP**: Removed 2 duplicate LinkedIn account records with 40-character trailing slashes
+- **MIGRATION COMPATIBILITY**: Resolved "value too long for type character varying(39)" migration errors
+
+### Developer Tools
+
+- Added `scripts/check-internal-identifier-regular-issues.ts` for data validation
+- Added `scripts/remove-duplicate-trailing-slash-profiles.ts` for cleanup
+- Added `scripts/test-aco-identifier-validation.ts` with comprehensive test suite
+
+### Migration Notes
+
+This version includes database schema changes. The migration:
+
+1. Changes `internal_identifier_regular` column from `varchar(255)` to `varchar(39)`
+2. Adds check constraint to ensure exactly 39 characters when not null
+3. All existing data has been validated and cleaned to meet the new requirements
+
+## [4.23.3] - 2025-09-10
+
+### BREAKING CHANGES
+
+- **REMOVED**: Complete cache_db support and infrastructure
+- **REMOVED**: All cache-related files, schemas, and configurations
+- **REMOVED**: `cache-db.ts`, `cache-schema.ts`, `drizzle-cache.config.ts`
+- **REMOVED**: `drizzle-cache/` directory with all cache migrations
+- **REMOVED**: Cache-related npm scripts (`cache:*` commands)
+- **REMOVED**: Cache database documentation (`CACHE_DATABASE.md`)
+- **SIMPLIFIED**: `aws-config.ts` to only support main database connections
+- **SIMPLIFIED**: Environment configuration to remove cache database references
+
+### Migration Guide
+
+If you were using cache database functionality:
+
+1. Migrate any critical data from `cache_db` to `eggi_main`
+2. Update any code that imported cache-related exports
+3. Remove cache-related environment variables from your configuration
+4. Update database connection logic to use only main database
+
+### Database Architecture
+
+The RDS schema now focuses exclusively on the main `eggi_main` database with proper cascade delete chains for:
+
+- On-demand mapping jobs → Mapping jobs → Relationship scores + LinkedIn account mappings
+- Simplified database access patterns for better maintainability
+
+## [4.22.15] - 2025-09-09
+
+### Changed
+
+- **BREAKING**: Updated `checkLinkedInProfileAvailability` to check ALL completed mapping jobs instead of just on-demand jobs
+- Function now joins `mapping_jobs_linkedin_accounts` with `mapping_jobs` table to verify completion status
+- Only considers mapping jobs with `completedAt` timestamp (fully processed mappings)
+- Improved accuracy of "mapped_before" detection across all mapping job types
+- Enhanced 24-hour deduplication logic based on actual completion time
+
+### Fixed
+
+- Fixed issue where availability check only looked at on-demand mapping jobs, missing other types of completed mappings
+- Corrected logic to properly detect when LinkedIn profiles have been successfully mapped by any service
+
+## [4.22.13] - 2025-09-09
+
+### Final Production Release
+
+- ✅ **Profile Skills Verification**: Confirmed Lambda correctly creates profile-level skills from Ghost Genius top-level skills array
+- ✅ **End-to-End Testing**: All field extraction working perfectly (profile, education, work experience skills)
+- ✅ **Database Constraints**: All regex and unique constraints verified and working
+- ✅ **Identifier Routing**: ACoA identifiers correctly stored in `internal_identifier_regular` column
+- ✅ **Code Cleanup**: Removed temporary debugging scripts, production-ready codebase
+- ✅ **Complete Field Mapping**: All required fields for Supabase sync properly extracted and stored
+- 🚀 **Ready for Lambda Update**: Package ready for deployment to `get_linkedin_profile` Lambda function
+
+### Technical Details
+
+- **Profile Skills**: 20+ skills correctly extracted from Ghost Genius response
+- **Education Skills**: Properly linked to education entries with correct foreign keys
+- **Work Experience Skills**: Correctly associated with work experience entries
+- **Database Integrity**: All constraints enforced at database level
+- **Migration System**: Fully synchronized with Drizzle migrations
+
+## [4.22.12] - 2025-09-09
+
+### Added
+
+- **Unique Constraint** - Added unique constraint on `internal_identifier_regular` to prevent duplicate ACoA identifiers across all LinkedIn accounts
+- Comprehensive constraint testing suite for both regex and unique constraints
+- Duplicate detection and cleanup utilities
+
+### Fixed
+
+- **Data Cleanup** - Resolved existing duplicate ACoA identifier (John Ciannello profile) by removing the newer duplicate record
+- **Migration System** - Fixed drizzle migration numbering conflicts and ensured proper constraint application through migration 31
+- **Database Integrity** - All LinkedIn identifier constraints now properly enforced at database level
+
+### Technical
+
+- Enhanced constraint validation with proper error handling
+- Improved data consistency and prevented race condition duplicates
+- Streamlined migration process with comprehensive testing
+
+## [4.22.11] - 2025-09-09
+
+### Added
+
+- **Database Regex Constraints** - Added check constraints to enforce LinkedIn identifier format rules:
+  - `internal_identifier` must start with 'ACwA' (for Unipile/authenticated accounts)
+  - `internal_identifier_regular` must start with 'ACoA' (for Guest/anonymous accounts)
+- Comprehensive test suite to verify constraint enforcement with invalid data rejection
+- Migration sync verification tools and scripts
+
+### Fixed
+
+- **Migration System** - Fixed migration 0 tracking issue and ensured database is fully in sync with migration files (32/32)
+- **Data Integrity** - All existing data validated as compliant with new constraints
+- **Infrastructure** - Lambda rebuilt with latest rds-schema to use new constraints
+
+### Technical
+
+- Added WhatsApp to `linkedin_account_type` enum
+- Enhanced database schema validation and error prevention
+- Improved identifier routing reliability and data consistency
+
+## [4.22.10] - 2025-09-08
+
+### Fixed
+
+- Verified exports for `createUserAndLinkedInAccountForLinkedIn` and `storeCompleteLinkedInProfile` functions
+- Ensured proper function availability for Lambda consumption
+
+## [4.22.9] - 2025-09-08
+
+### Fixed
+
+- **CRITICAL FIX: ACoA identifier routing** - Fixed issue where ACoA identifiers were incorrectly stored in `internal_identifier` instead of `internal_identifier_regular`
+- Removed legacy `handleLinkedInProfileFetch` function and updated Lambda to use proper functions
+- Fixed Ghost Genius identifier routing in `createUserAndLinkedInAccountForLinkedIn` to treat ACoA identifiers as provider responses
+- Updated function signatures to match current implementation
+
+### Removed
+
+- Removed all legacy compatibility code for cleaner implementation
+- Removed deprecated `handleLinkedInProfileFetch` function
+
+### Updated
+
+- Updated Lambda to use `storeCompleteLinkedInProfile` and `createUserAndLinkedInAccountForLinkedIn` directly
+- Enhanced identifier routing logic to ensure ACoA identifiers from Ghost Genius go to `internal_identifier_regular`
+
+## [4.22.6] - 2025-01-09
+
+### 🔧 LINKEDIN PROFILE DATA EXTRACTION FIXES
+
+**Critical Fix**: Resolved missing field extraction and skills insertion issues in LinkedIn profile processing.
+
+#### Fixed
+
+- **Education Field Extraction**: Fixed missing fields in Ghost Genius to LinkedInProfileData transformation
+  - ✅ `description`: Now properly extracted from Ghost Genius education data
+  - ✅ `activities_and_societies`: Fixed mapping from Ghost Genius `activites_and_societies` (handles API typo)
+  - ✅ `school_linkedin_url`: Now correctly maps from `school.url` field
+  - ✅ `school_id`: Properly extracted and validated (excludes URL-formatted IDs)
+  - ✅ `grade`: Successfully extracted from Ghost Genius education data
+
+- **Work Experience Field Extraction**: Enhanced work experience data mapping
+  - ✅ `company_linkedin_url`: Now correctly extracted from `company.url` field
+  - ✅ `employment_type`: Properly mapped from Ghost Genius work experience data
+
+- **Skills Insertion Bug**: Fixed critical `onConflictDoNothing()` misalignment issue
+  - **Education Skills**: Fixed index misalignment between `profileData.education` and `insertedEducation` arrays
+  - **Work Experience Skills**: Fixed same index misalignment for work experience skills
+  - **Solution**: Query existing records after insertion and match by identifying fields (school+degree, company+position)
+
+- **Type Safety**: Updated TypeScript interfaces to prevent compilation errors
+  - Added `activites_and_societies` field to Ghost Genius `Education` interface
+  - Added `school_linkedin_url` field to `LinkedInProfileData.education` interface (backward compatible with legacy `url` field)
+
+#### Added
+
+- **Comprehensive Test Suite**: Created `test-linkedin-profile-fetch.ts` script for end-to-end validation
+- **Test Data**: Added `ghost-genius-test-data.ts` with real LinkedIn profile data for testing
+- **NPM Script**: Added `test:profile-fetch` for easy test execution
+
+#### Technical Details
+
+- **Skills Association Logic**: Replaced unreliable array index mapping with database record matching
+  - Education skills now match by `school` and `degree` fields
+  - Work experience skills now match by `company` and `position` fields
+- **Database Operations**: Enhanced logging and error handling for profile storage operations
+- **Field Validation**: Improved handling of Ghost Genius API response variations and edge cases
+
+#### Verification
+
+- ✅ All education fields properly extracted and stored
+- ✅ All work experience fields properly extracted and stored
+- ✅ Education skills correctly associated with education records
+- ✅ Work experience skills correctly associated with work experience records
+- ✅ End-to-end test confirms complete data pipeline functionality
+
+## [4.12.0] - 2025-01-24
+
+### 🎛️ QUERY PARAMETER CONTROL FOR COMPLEMENTARY IDENTIFIERS
+
+**Major Enhancement**: Added `complementary_ids` query parameter for fine-grained control over identifier returns.
+
+#### Added
+
+- **Query Parameter**: `complementary_ids=true/false` for GET and POST endpoints
+- **Database Response Enhancement**: Return both `internal_identifier_regular` and `internal_identifier` fields
+- **Lambda Decision Logic**: Business logic moved to Lambda layer for better separation of concerns
+
+#### Behavior
+
+- **`complementary_ids=true`** + DB lookup → return complementary identifier when available
+  - ACoA* request → return ACwA* identifier (if stored)
+  - ACwA* request → return ACoA* identifier (if stored)
+- **`complementary_ids=false`** + DB lookup → return original stored identifier
+- **GET fresh** (cache miss) → parameter ignored, return exact Unipile response
+- **Missing parameter** → defaults to `false` (original behavior)
+
+#### Architecture
+
+- **Database Layer**: Returns both identifiers, lets Lambda decide
+- **Lambda Layer**: Contains business logic for complementary identifier selection
+- **Clean Response**: Single `provider_id` field, no internal identifier fields exposed
+
+## [4.11.8] - 2025-01-24
+
+### 🔄 COMPLEMENTARY IDENTIFIER LOGIC FOR DB-BASED RESPONSES
+
+**Critical Fix**: Corrected complementary identifier logic to return complementary IDs for DB-based responses.
+
+#### Fixed
+
+- **`getOptimalProviderIdForResponse`**: Now returns **complementary** identifiers instead of preferred ones
+  - **ACoA\*** requests → return complementary `internalIdentifier` (ACwA\*) when available
+  - **ACwA\*** requests → return complementary `internalIdentifierRegular` (ACoA\*) when available
+- **Lambda POST responses**: After storing data, use `getComprehensiveLinkedinProfile` for consistent complementary logic
+- **Cache hit responses**: Properly return complementary identifiers from stored data
+
+#### Behavior
+
+- **Cache Miss (GET fresh)**: Returns exact Unipile response (no complementary logic)
+- **Cache Hit (GET cached)**: Returns complementary identifier when available
+- **POST requests**: Store data then return complementary identifier when available
+
+#### Expected Results
+
+- Request `ACwAADSH2jABexaQDAFqIGS1V8YQzzS3N0OMMlY` → Returns `ACoAADSH2jAB6ogAhfdoxnmNoYpnkHiiMhsBK-0`
+- Request `ACoAADSH2jAB6ogAhfdoxnmNoYpnkHiiMhsBK-0` → Returns `ACwAADSH2jABexaQDAFqIGS1V8YQzzS3N0OMMlY`
+
+## [4.11.7] - 2025-01-24
+
+### 🎯 INTELLIGENT PROVIDER_ID SELECTION FOR CACHE RESPONSES
+
+**Enhancement**: Updated `getComprehensiveLinkedinProfile` to intelligently choose provider_id based on request type.
+
+#### Added
+
+- **`getOptimalProviderIdForResponse`**: Smart provider_id selection logic
+  - **ACoA\*** requests → prefer returning `internalIdentifierRegular` (ACoA\* stored), fallback to `internalIdentifier`
+  - **ACwA\*** requests → prefer returning `internalIdentifier` (ACwA\* stored), fallback to `internalIdentifierRegular`
+  - **Other requests** → return whichever identifier is available
+
+#### Fixed
+
+- **Cache Response Provider ID**: Now returns the most appropriate identifier based on request type
+- **Missing Field**: Added `internalIdentifierRegular` to comprehensive profile query selection
+- **Request-Response Matching**: ACwA\* requests now get back the stored ACwA\* identifier when available, ACoA\* requests get back ACoA\* identifiers
+
+#### Impact
+
+- **✅ Proper Complementary Returns**: Cache hits now return the expected identifier format based on request type
+- **✅ Fallback Logic**: When preferred identifier isn't available, falls back to the complementary one
+- **✅ API Consistency**: Maintains consistent provider_id format expectations for different request types
+
+## [4.11.6] - 2025-01-24
+
+### 🔍 COMPLEMENTARY IDENTIFIER CACHE LOOKUP FIX
+
+**Critical Fix**: Updated search functions to enable cache hits for complementary identifiers.
+
+#### Fixed
+
+- **`findLinkedInAccountByLinkedInIdentifier`**: Now searches **ALL** internal identifier fields instead of just the expected field
+  - ACwA* requests can now find profiles stored with ACoA* identifiers
+  - ACoA* requests can now find profiles stored with ACwA* identifiers
+- **`findUserByLinkedInIdentifier`**: Applied same comprehensive search logic for consistency
+- **Cache Hit Logic**: Fixed issue where complementary identifiers wouldn't match existing cached profiles
+
+#### Impact
+
+- **✅ Solves Cache Miss Issue**: When requesting `ACwAADSH2jABexaQDAFqIGS1V8YQzzS3N0OMMlY` but database has `ACoAADSH2jAB6ogAhfdoxnmNoYpnkHiiMhsBK-0` stored - now finds the cached profile
+- **✅ Proper Complementary Returns**: Cache hits now return the actual stored identifiers instead of echoing request identifiers
+- **✅ Consistent Search Logic**: All LinkedIn identifier search functions now use the same comprehensive approach
+
+## [4.11.5] - 2025-01-24
+
+### 🎯 IDENTIFIER ROUTING & COMPLEMENTARY ID STORAGE
+
+**Enhancement**: Improved identifier mapping and complementary identifier storage for LinkedIn profile operations.
+
+#### Changed
+
+- **Identifier Routing Logic**: Updated `buildLinkedInAccountValues` to properly handle complementary identifiers
+  - **ACoA\*** identifiers (Sales Navigator) → stored in `internal_identifier_regular`
+  - **ACwA\*** identifiers (Regular LinkedIn) → stored in `internal_identifier`
+- **Complementary ID Storage**: Enhanced storage logic to preserve both request and response identifiers
+  - ACwA request → ACoA response: Both identifiers properly stored in their respective columns
+  - ACoA request → ACwA response: Both identifiers properly stored in their respective columns
+- **Search Logic**: Updated `findLinkedInAccountByLinkedInIdentifier` to match new identifier routing
+  - ACoA\* identifiers → search in `internal_identifier_regular` field
+  - ACwA\* identifiers → search in `internal_identifier` field
+- **Enhanced Logging**: Improved debugging information for identifier mapping decisions
+
+#### Benefits
+
+- **Data Integrity**: Ensures both complementary identifiers are always stored for future cross-reference
+- **Routing Accuracy**: Proper account type routing based on identifier prefixes
+- **Better Debugging**: Enhanced logging for identifier mapping troubleshooting
+
+## [4.1.0] - 2024-12-31
+
+### 🔐 USER-SPECIFIC SECRETS ARCHITECTURE
+
+**Evolution**: Migrated from database-specific to user-specific secrets for better security and access control.
+
+#### Changed
+
+- **BREAKING**: `MAIN_DB_SECRET_ARN` → `DB_ADMIN_PROXY_MAIN_SECRET_ARN` and `DB_ADMIN_DIRECT_MAIN_SECRET_ARN` (specific connection secrets)
+- **BREAKING**: `CACHE_DB_SECRET_ARN` removed (admin user has access to both databases)
+- **Secret Architecture**: Now organized by database users and connection types:
+  - `DB_ADMIN_PROXY_MAIN_SECRET_ARN`: Admin user via RDS Proxy for production
+  - `DB_ADMIN_DIRECT_MAIN_SECRET_ARN`: Admin user via direct connection for development
+  - `DB_MANTAS_SECRET_ARN`: Limited user with access to `cache_db` only
+- **Infrastructure**: Removed deprecated `main_db` and `cache_db` secrets from AWS
+- **Connection Logic**: Same smart connection selection (direct vs proxy)
+
+#### Migration Guide
+
+**From v4.0.0 to v4.1.0:**
+
+1. **Update environment variables**:
+   - Replace `MAIN_DB_SECRET_ARN` with `DB_ADMIN_PROXY_MAIN_SECRET_ARN` (for production)
+   - Add `DB_ADMIN_DIRECT_MAIN_SECRET_ARN` (for development, optional)
+   - Remove `CACHE_DB_SECRET_ARN` (admin user covers both databases)
+
+2. **Use environment variables directly**:
+   - Set `DB_ADMIN_PROXY_MAIN_SECRET_ARN` environment variable
+   - Use `terraform output user_secrets_summary` to get secret ARNs
+
+3. **Architecture benefits**:
+   - Better security through user-based access control
+   - Simplified secret management
+   - Granular permissions per database user
+
+## [4.0.0] - 2024-12-31
+
+### 🚀 BREAKING CHANGES - Smart Database Secret Architecture
+
+**Major architectural change**: Replaced fragmented secrets with database-centric smart secrets for better organization and security.
+
+#### Added
+
+- **Smart Database Secrets**: Self-contained secrets per database with both direct and proxy connection info
+- **Automatic Connection Selection**: Environment variable `USE_DIRECT_CONNECTION` to choose connection type
+- **Database-Specific Secrets**: `MAIN_DB_SECRET_ARN` and `CACHE_DB_SECRET_ARN` replace old secret structure
+- **Connection Type Auto-Detection**: Prefer proxy for applications, direct for admin operations
+
+#### Changed
+
+- **BREAKING**: `RDS_DIRECT_SECRET_ARN` → `MAIN_DB_SECRET_ARN`
+- **BREAKING**: `RDS_PROXY_SECRET_ARN` → `MAIN_DB_SECRET_ARN` (smart secret contains both)
+- **Secret Structure**: Each database secret now contains:
+  ```json
+  {
+    "username": "eggi_admin",
+    "password": "...",
+    "database": "eggi_main|cache_db",
+    "port": 5432,
+    "direct_connection": { ... },
+    "proxy_connection": { ... },
+    "usage_guide": { ... }
+  }
+  ```
+
+#### Improved
+
+- **Simplified Configuration**: One secret per database instead of multiple secrets
+- **Better Documentation**: Self-documenting secrets with usage guides
+- **Consistent Credentials**: Same username/password works for both connection types
+- **Enhanced Security**: Clear separation between databases while maintaining same credentials
+
+#### Migration Guide
+
+1. **Environment Variables**:
+   - Replace `RDS_DIRECT_SECRET_ARN` with `MAIN_DB_SECRET_ARN`
+   - Replace `RDS_PROXY_SECRET_ARN` with `MAIN_DB_SECRET_ARN`
+   - Add `CACHE_DB_SECRET_ARN` for cache database access
+   - Optional: Set `USE_DIRECT_CONNECTION=true` for admin operations
+
+2. **Secret Structure**:
+   - Terraform will create new smart secrets automatically
+   - Old secrets can be safely removed after migration
+   - Same credentials work for both connection types
+
+3. **Code Changes**:
+   - No changes needed if using `getDatabaseUrl()` or `getCredentials()`
+   - Package automatically handles smart secret destructuring
+
+## [3.2.5] - 2024-12-30
+
+### Fixed
+
+- Fixed duplicate key constraint violations in LinkedIn profile skills processing
+- Added proper UPSERT logic with `.onConflictDoNothing()` in `findOrCreateSkills()`
+- Improved error handling for skills processing operations
+
+### Changed
+
+- Updated skills processing to handle idempotent operations correctly
+- Enhanced logging for skills processing workflow
+
+## [3.2.4] - 2024-12-29
+
+### Added
+
+- Comprehensive LinkedIn profile processing utilities
+- Work experience skills processing with normalized architecture
+- Enhanced contact information processing
+
+### Fixed
+
+- Improved error handling in profile processing workflows
+- Better transaction management for batch operations
+
+## [3.2.3] - 2024-12-28
+
+### Added
+
+- Cache database support for conversations and messages
+- Multi-database configuration support
+- Enhanced connection management utilities
+
+### Improved
+
+- Better database connection pooling
+- Optimized query performance for large datasets
+
+   - Same credentials work for both connection types
+
+3. **Code Changes**:
+   - No changes needed if using `getDatabaseUrl()` or `getCredentials()`
+   - Package automatically handles smart secret destructuring
+
+## [3.2.5] - 2024-12-30
+
+### Fixed
+
+- Fixed duplicate key constraint violations in LinkedIn profile skills processing
+- Added proper UPSERT logic with `.onConflictDoNothing()` in `findOrCreateSkills()`
+- Improved error handling for skills processing operations
+
+### Changed
+
+- Updated skills processing to handle idempotent operations correctly
+- Enhanced logging for skills processing workflow
+
+## [3.2.4] - 2024-12-29
+
+### Added
+
+- Comprehensive LinkedIn profile processing utilities
+- Work experience skills processing with normalized architecture
+- Enhanced contact information processing
+
+### Fixed
+
+- Improved error handling in profile processing workflows
+- Better transaction management for batch operations
+
+## [3.2.3] - 2024-12-28
+
+### Added
+
+- Cache database support for conversations and messages
+- Multi-database configuration support
+- Enhanced connection management utilities
+
+### Improved
+
+- Better database connection pooling
+- Optimized query performance for large datasets