mbuzz 0.8.2 → 0.9.0

data/lib/specs/v0.7.0_deterministic_sessions.md

@@ -1,647 +0,0 @@
-# mbuzz SDK v0.7.0 - Deterministic Session IDs
-
-> **⚠️ SUPERSEDED**: This spec has been replaced by server-side session resolution.
-> See: `multibuzz/lib/specs/1_visitor_session_tracking_spec.md`
->
-> **What changed**: Instead of SDKs generating deterministic session IDs using time-buckets,
-> the server now handles all session resolution using IP + User-Agent fingerprinting with
-> a true 30-minute sliding window. SDKs no longer manage session cookies.
-
-**Status**: SUPERSEDED (2026-01-09)
-**Last Updated**: 2025-12-29
-**Breaking Change**: No (backward compatible)
-**Affects**: All SDKs (Ruby, Python, PHP, Node)
-
----
-
-## Problem Statement
-
-### The Race Condition
-
-When a page loads, multiple concurrent HTTP requests can hit the server before the first response sets cookies. Each request generates a new random session ID, creating duplicate sessions:
-
-```
-User clicks link → Page starts loading
-
-Request 1 (HTML) ──────────────────▶ No cookies
-                                     Server: session_id = random_1
-
-Request 2 (Turbo/fetch) ───────────▶ No cookies (response 1 not back yet)
-                                     Server: session_id = random_2
-
-Request 3 (XHR) ───────────────────▶ No cookies
-                                     Server: session_id = random_3
-
-Result: 3 different sessions created for the same page load!
-```
-
-### Production Evidence
-
-From Pet Resorts Australia (visitor #139):
-- 65 timestamps with multiple sessions created at the exact same second
-- 5 sessions created within 35ms, all with different session_ids
-- Same visitor_id preserved (2-year cookie works)
-- 94.8% of sessions have 0 page views (phantom sessions)
-
-### Impact on Attribution
-
-1. **Inflated "Direct" channel**: Internal navigations create new sessions classified as "direct"
-2. **Broken Last Touch**: Real acquisition channels get overwritten by phantom "direct" sessions
-3. **Skewed Metrics**: Average visits per conversion is 35.1 instead of ~1-2
-
----
-
-## Solution: Deterministic Session ID Generation
-
-### Core Concept
-
-Instead of generating random session IDs, generate **deterministic** IDs based on:
-1. Visitor ID (from cookie - has 2-year expiry)
-2. Time bucket (30-minute windows matching session timeout)
-3. Request fingerprint (IP + User-Agent for new visitors)
-
-All concurrent requests will generate the **same session ID**.
-
-### Algorithm
-
-```
-IF visitor_id cookie exists:
-    session_id = SHA256(visitor_id + time_bucket)[0:64]
-ELSE:
-    fingerprint = SHA256(client_ip + user_agent)[0:32]
-    session_id = SHA256(fingerprint + time_bucket)[0:64]
-
-WHERE:
-    time_bucket = floor(unix_timestamp / SESSION_TIMEOUT_SECONDS)
-    SESSION_TIMEOUT_SECONDS = 1800 (30 minutes)
-```
-
-### Why This Works
-
-| Scenario | Visitor Cookie | Result |
-|----------|----------------|--------|
-| Returning visitor, expired session | ✅ Exists | All concurrent requests get same session_id |
-| New visitor, first page load | ❌ Missing | All concurrent requests from same IP+UA get same session_id |
-| Active session | ✅ Exists + Session cookie | Cookie used directly (no generation needed) |
-
-### Security Considerations
-
-1. **Session IDs remain unpredictable**: SHA256 hash is not reversible
-2. **No PII in session ID**: Only hashed values stored
-3. **Time bucket prevents replay**: Old session IDs naturally expire
-4. **IP+UA fingerprint is ephemeral**: Not stored, only used for generation
-
----
-
-## Implementation Specification
-
-### Constants
-
-```
-SESSION_TIMEOUT_SECONDS = 1800        # 30 minutes
-SESSION_ID_LENGTH = 64                # Characters (256 bits as hex)
-FINGERPRINT_LENGTH = 32               # Characters for IP+UA hash
-HASH_ALGORITHM = "SHA256"
-```
-
-### New Module: `SessionIdGenerator`
-
-Each SDK must implement a `SessionIdGenerator` with these methods:
-
-#### `generate_deterministic(visitor_id, timestamp)`
-
-Generate session ID for **returning visitors** (have visitor cookie).
-
-```
-Input:
-  - visitor_id: string (64 hex chars)
-  - timestamp: unix timestamp (seconds)
-
-Output:
-  - session_id: string (64 hex chars)
-
-Algorithm:
-  time_bucket = floor(timestamp / SESSION_TIMEOUT_SECONDS)
-  raw = visitor_id + "_" + string(time_bucket)
-  session_id = SHA256(raw).hexdigest()[0:64]
-```
-
-#### `generate_from_fingerprint(client_ip, user_agent, timestamp)`
-
-Generate session ID for **new visitors** (no visitor cookie).
-
-```
-Input:
-  - client_ip: string (e.g., "192.168.1.1" or "2001:db8::1")
-  - user_agent: string (browser user agent)
-  - timestamp: unix timestamp (seconds)
-
-Output:
-  - session_id: string (64 hex chars)
-
-Algorithm:
-  fingerprint = SHA256(client_ip + "|" + user_agent).hexdigest()[0:32]
-  time_bucket = floor(timestamp / SESSION_TIMEOUT_SECONDS)
-  raw = fingerprint + "_" + string(time_bucket)
-  session_id = SHA256(raw).hexdigest()[0:64]
-```
-
-#### `generate_random()`
-
-Fallback for edge cases (no IP available, etc.).
-
-```
-Output:
-  - session_id: string (64 hex chars, cryptographically random)
-```
-
----
-
-## SDK Implementation Details
-
-### Ruby (mbuzz-ruby)
-
-**File**: `lib/mbuzz/session/id_generator.rb`
-
-```ruby
-# frozen_string_literal: true
-
-require "digest"
-require "securerandom"
-
-module Mbuzz
-  module Session
-    class IdGenerator
-      SESSION_TIMEOUT_SECONDS = 1800
-      SESSION_ID_LENGTH = 64
-      FINGERPRINT_LENGTH = 32
-
-      class << self
-        def generate_deterministic(visitor_id:, timestamp: Time.now.to_i)
-          time_bucket = timestamp / SESSION_TIMEOUT_SECONDS
-          raw = "#{visitor_id}_#{time_bucket}"
-          Digest::SHA256.hexdigest(raw)[0, SESSION_ID_LENGTH]
-        end
-
-        def generate_from_fingerprint(client_ip:, user_agent:, timestamp: Time.now.to_i)
-          fingerprint = Digest::SHA256.hexdigest("#{client_ip}|#{user_agent}")[0, FINGERPRINT_LENGTH]
-          time_bucket = timestamp / SESSION_TIMEOUT_SECONDS
-          raw = "#{fingerprint}_#{time_bucket}"
-          Digest::SHA256.hexdigest(raw)[0, SESSION_ID_LENGTH]
-        end
-
-        def generate_random
-          SecureRandom.hex(32)
-        end
-      end
-    end
-  end
-end
-```
-
-**File**: `lib/mbuzz/middleware/tracking.rb` (updated)
-
-```ruby
-def build_request_context(request)
-  visitor_id = visitor_id_from_cookie(request)
-  is_new_visitor = visitor_id.nil?
-  visitor_id ||= Visitor::Identifier.generate
-
-  session_id = session_id_from_cookie(request) || generate_session_id(
-    visitor_id: is_new_visitor ? nil : visitor_id,
-    request: request
-  )
-
-  # ... rest unchanged
-end
-
-def generate_session_id(visitor_id:, request:)
-  if visitor_id
-    Session::IdGenerator.generate_deterministic(visitor_id: visitor_id)
-  else
-    Session::IdGenerator.generate_from_fingerprint(
-      client_ip: client_ip(request),
-      user_agent: user_agent(request)
-    )
-  end
-end
-
-def client_ip(request)
-  request.env["HTTP_X_FORWARDED_FOR"]&.split(",")&.first&.strip ||
-    request.env["HTTP_X_REAL_IP"] ||
-    request.ip ||
-    "unknown"
-end
-
-def user_agent(request)
-  request.user_agent || "unknown"
-end
-```
-
----
-
-### Node.js (mbuzz-node)
-
-**File**: `src/utils/sessionId.ts`
-
-```typescript
-import { createHash, randomBytes } from 'node:crypto';
-
-const SESSION_TIMEOUT_SECONDS = 1800;
-const SESSION_ID_LENGTH = 64;
-const FINGERPRINT_LENGTH = 32;
-
-export function generateDeterministic(
-  visitorId: string,
-  timestamp: number = Math.floor(Date.now() / 1000)
-): string {
-  const timeBucket = Math.floor(timestamp / SESSION_TIMEOUT_SECONDS);
-  const raw = `${visitorId}_${timeBucket}`;
-  return createHash('sha256').update(raw).digest('hex').slice(0, SESSION_ID_LENGTH);
-}
-
-export function generateFromFingerprint(
-  clientIp: string,
-  userAgent: string,
-  timestamp: number = Math.floor(Date.now() / 1000)
-): string {
-  const fingerprint = createHash('sha256')
-    .update(`${clientIp}|${userAgent}`)
-    .digest('hex')
-    .slice(0, FINGERPRINT_LENGTH);
-  const timeBucket = Math.floor(timestamp / SESSION_TIMEOUT_SECONDS);
-  const raw = `${fingerprint}_${timeBucket}`;
-  return createHash('sha256').update(raw).digest('hex').slice(0, SESSION_ID_LENGTH);
-}
-
-export function generateRandom(): string {
-  return randomBytes(32).toString('hex');
-}
-```
-
-**File**: `src/middleware/express.ts` (updated)
-
-```typescript
-import { generateDeterministic, generateFromFingerprint, generateRandom } from '../utils/sessionId';
-
-const getClientIp = (req: Request): string => {
-  const forwarded = req.headers['x-forwarded-for'];
-  if (typeof forwarded === 'string') {
-    return forwarded.split(',')[0].trim();
-  }
-  return req.ip || req.socket.remoteAddress || 'unknown';
-};
-
-const getUserAgent = (req: Request): string => {
-  return req.headers['user-agent'] || 'unknown';
-};
-
-const getSessionId = (req: Request, visitorId: string | null): { id: string; isNew: boolean } => {
-  const existing = req.cookies?.[SESSION_COOKIE];
-  if (existing) {
-    return { id: existing, isNew: false };
-  }
-
-  const id = visitorId
-    ? generateDeterministic(visitorId)
-    : generateFromFingerprint(getClientIp(req), getUserAgent(req));
-
-  return { id, isNew: true };
-};
-```
-
----
-
-### Python (mbuzz-python)
-
-**File**: `src/mbuzz/utils/session_id.py`
-
-```python
-"""Deterministic session ID generation."""
-
-import hashlib
-import secrets
-import time
-
-SESSION_TIMEOUT_SECONDS = 1800
-SESSION_ID_LENGTH = 64
-FINGERPRINT_LENGTH = 32
-
-
-def generate_deterministic(visitor_id: str, timestamp: int | None = None) -> str:
-    """Generate session ID for returning visitors."""
-    if timestamp is None:
-        timestamp = int(time.time())
-    time_bucket = timestamp // SESSION_TIMEOUT_SECONDS
-    raw = f"{visitor_id}_{time_bucket}"
-    return hashlib.sha256(raw.encode()).hexdigest()[:SESSION_ID_LENGTH]
-
-
-def generate_from_fingerprint(
-    client_ip: str,
-    user_agent: str,
-    timestamp: int | None = None
-) -> str:
-    """Generate session ID for new visitors using IP+UA fingerprint."""
-    if timestamp is None:
-        timestamp = int(time.time())
-    fingerprint = hashlib.sha256(
-        f"{client_ip}|{user_agent}".encode()
-    ).hexdigest()[:FINGERPRINT_LENGTH]
-    time_bucket = timestamp // SESSION_TIMEOUT_SECONDS
-    raw = f"{fingerprint}_{time_bucket}"
-    return hashlib.sha256(raw.encode()).hexdigest()[:SESSION_ID_LENGTH]
-
-
-def generate_random() -> str:
-    """Generate random session ID (fallback)."""
-    return secrets.token_hex(32)
-```
-
-**File**: `src/mbuzz/middleware/flask.py` (updated)
-
-```python
-from flask import request
-from ..utils.session_id import generate_deterministic, generate_from_fingerprint
-
-
-def _get_client_ip() -> str:
-    """Get client IP from request headers."""
-    forwarded = request.headers.get('X-Forwarded-For', '')
-    if forwarded:
-        return forwarded.split(',')[0].strip()
-    return request.remote_addr or 'unknown'
-
-
-def _get_user_agent() -> str:
-    """Get user agent from request."""
-    return request.headers.get('User-Agent', 'unknown')
-
-
-def _get_or_create_session_id(visitor_id: str | None) -> str:
-    """Get session ID from cookie or generate deterministic one."""
-    existing = request.cookies.get(SESSION_COOKIE)
-    if existing:
-        return existing
-
-    if visitor_id:
-        return generate_deterministic(visitor_id)
-    else:
-        return generate_from_fingerprint(_get_client_ip(), _get_user_agent())
-```
-
----
-
-### PHP (mbuzz-php)
-
-**File**: `src/Mbuzz/SessionIdGenerator.php`
-
-```php
-<?php
-
-declare(strict_types=1);
-
-namespace Mbuzz;
-
-final class SessionIdGenerator
-{
-    private const SESSION_TIMEOUT_SECONDS = 1800;
-    private const SESSION_ID_LENGTH = 64;
-    private const FINGERPRINT_LENGTH = 32;
-
-    /**
-     * Generate session ID for returning visitors (have visitor cookie).
-     */
-    public static function generateDeterministic(
-        string $visitorId,
-        ?int $timestamp = null
-    ): string {
-        $timestamp = $timestamp ?? time();
-        $timeBucket = intdiv($timestamp, self::SESSION_TIMEOUT_SECONDS);
-        $raw = "{$visitorId}_{$timeBucket}";
-        return substr(hash('sha256', $raw), 0, self::SESSION_ID_LENGTH);
-    }
-
-    /**
-     * Generate session ID for new visitors using IP+UA fingerprint.
-     */
-    public static function generateFromFingerprint(
-        string $clientIp,
-        string $userAgent,
-        ?int $timestamp = null
-    ): string {
-        $timestamp = $timestamp ?? time();
-        $fingerprint = substr(
-            hash('sha256', "{$clientIp}|{$userAgent}"),
-            0,
-            self::FINGERPRINT_LENGTH
-        );
-        $timeBucket = intdiv($timestamp, self::SESSION_TIMEOUT_SECONDS);
-        $raw = "{$fingerprint}_{$timeBucket}";
-        return substr(hash('sha256', $raw), 0, self::SESSION_ID_LENGTH);
-    }
-
-    /**
-     * Generate random session ID (fallback).
-     */
-    public static function generateRandom(): string
-    {
-        return bin2hex(random_bytes(32));
-    }
-}
-```
-
-**File**: `src/Mbuzz/Context.php` (updated to use new generator)
-
-```php
-<?php
-
-// In the session ID resolution logic:
-
-private function resolveSessionId(?string $visitorId): string
-{
-    $existing = $this->cookieManager->getSessionId();
-    if ($existing !== null) {
-        return $existing;
-    }
-
-    if ($visitorId !== null) {
-        return SessionIdGenerator::generateDeterministic($visitorId);
-    }
-
-    return SessionIdGenerator::generateFromFingerprint(
-        $this->getClientIp(),
-        $this->getUserAgent()
-    );
-}
-
-private function getClientIp(): string
-{
-    return $_SERVER['HTTP_X_FORWARDED_FOR']
-        ? explode(',', $_SERVER['HTTP_X_FORWARDED_FOR'])[0]
-        : ($_SERVER['HTTP_X_REAL_IP'] ?? $_SERVER['REMOTE_ADDR'] ?? 'unknown');
-}
-
-private function getUserAgent(): string
-{
-    return $_SERVER['HTTP_USER_AGENT'] ?? 'unknown';
-}
-```
-
----
-
-## Server-Side Handling (multibuzz API)
-
-The API should handle idempotent session creation:
-
-### Sessions Endpoint Update
-
-```ruby
-# app/services/sessions/creation_service.rb
-
-def run
-  return existing_session_result if session_exists?
-
-  # Create new session...
-end
-
-def session_exists?
-  # Check if session with this session_id already exists
-  @existing_session = account.sessions.find_by(
-    session_id: session_id,
-    visitor: visitor
-  )
-end
-
-def existing_session_result
-  success_result(
-    visitor_id: visitor.visitor_id,
-    session_id: @existing_session.session_id,
-    channel: @existing_session.channel,
-    existing: true
-  )
-end
-```
-
-This ensures:
-1. Multiple requests with same deterministic session_id don't create duplicates
-2. First request's data (UTM, referrer) is preserved
-3. Subsequent requests are no-ops
-
----
-
-## Migration Path
-
-### Backward Compatibility
-
-- Random session IDs still work (cookie-based sessions unaffected)
-- No changes to cookie format or names
-- No changes to API endpoints
-- Existing sessions continue to function
-
-### Rollout Strategy
-
-1. **Phase 1**: Deploy API changes (idempotent session creation)
-2. **Phase 2**: Release SDK updates (v0.7.0 for all SDKs)
-3. **Phase 3**: Monitor metrics for reduced duplicate sessions
-
-### Version Matrix
-
-| SDK | Current | After |
-|-----|---------|-------|
-| mbuzz-ruby | 0.6.x | 0.7.0 |
-| mbuzz-node | 0.6.x | 0.7.0 |
-| mbuzz-python | 0.6.x | 0.7.0 |
-| mbuzz-php | 0.6.x | 0.7.0 |
-
----
-
-## Testing Requirements
-
-### Unit Tests
-
-Each SDK must test:
-
-1. **Deterministic generation is consistent**
-   ```
-   generate_deterministic("visitor_abc", 1735500000) == generate_deterministic("visitor_abc", 1735500000)
-   ```
-
-2. **Different visitors get different IDs**
-   ```
-   generate_deterministic("visitor_a", t) != generate_deterministic("visitor_b", t)
-   ```
-
-3. **Time bucket boundaries work correctly**
-   ```
-   # Same 30-minute window = same ID
-   generate_deterministic("v", 1735500000) == generate_deterministic("v", 1735500001)
-
-   # Different window = different ID
-   generate_deterministic("v", 1735500000) != generate_deterministic("v", 1735501800)
-   ```
-
-4. **Fingerprint generation is consistent**
-   ```
-   generate_from_fingerprint("1.2.3.4", "Mozilla/5.0", t) ==
-   generate_from_fingerprint("1.2.3.4", "Mozilla/5.0", t)
-   ```
-
-5. **Different fingerprints get different IDs**
-   ```
-   generate_from_fingerprint("1.2.3.4", "UA1", t) !=
-   generate_from_fingerprint("1.2.3.4", "UA2", t)
-   ```
-
-### Integration Tests
-
-1. Concurrent requests from same visitor get same session
-2. Session cookie is set correctly on first response
-3. Subsequent requests use cookie (not regenerated)
-
----
-
-## Metrics to Monitor
-
-After deployment, track:
-
-1. **Sessions per visitor ratio** - Should decrease toward 1.0
-2. **Duplicate session timestamps** - Should approach 0
-3. **"Direct" channel percentage** - Should decrease if inflated
-4. **Average visits per conversion** - Should normalize
-
-### Success Criteria
-
-| Metric | Before | Target |
-|--------|--------|--------|
-| Sessions per new visitor | 2.0+ | < 1.2 |
-| Timestamps with duplicates | 65+ | < 5 |
-| Empty sessions (0 page views) | 94.8% | < 10% |
-
----
-
-## Open Questions
-
-1. **IPv6 handling**: Should we normalize IPv6 addresses before hashing?
-2. **Proxy detection**: Should we try multiple headers (X-Real-IP, CF-Connecting-IP)?
-3. **Bot detection**: Should known bot user agents bypass deterministic generation?
-
----
-
-## Appendix: Hash Examples
-
-```
-# Returning visitor
-visitor_id = "a1b2c3d4e5f6..."
-timestamp = 1735500000
-time_bucket = 1735500000 / 1800 = 964166
-raw = "a1b2c3d4e5f6..._964166"
-session_id = SHA256(raw)[0:64] = "7f8e9d0c1b2a..."
-
-# New visitor
-client_ip = "203.0.113.42"
-user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)..."
-fingerprint = SHA256("203.0.113.42|Mozilla/5.0...")[0:32] = "abc123..."
-raw = "abc123..._964166"
-session_id = SHA256(raw)[0:64] = "9e8d7c6b5a4..."
-```