@socialgouv/matomo-postgres 2.2.0-beta.3 → 2.2.1

package/README.md

@@ -2,46 +2,212 @@
 
 ![header](./header.png)
 
-Extract matomo data from [`Live.getLastVisitsDetails`](https://developer.matomo.org/api-reference/reporting-api) API and push events and visits informations to Postgres.
+A robust Node.js/TypeScript ETL (Extract, Transform, Load) tool that synchronizes visitor analytics data from Matomo (formerly Piwik) into a PostgreSQL database. Designed for organizations that need to centralize their web analytics data for advanced analysis, reporting, or integration with other systems.
 
-## Usage
+## ✨ Features
 
-Run the following job with correct environment variables.
+- **🔄 Incremental Synchronization** - Smart date range detection with automatic resume capability
+- **📊 Complete Data Extraction** - Captures visitor sessions, events, custom dimensions, and device information
+- **🗄️ Automatic Schema Management** - Kysely-based migrations with performance optimizations
+- **⚡ High Performance** - Controlled concurrency, pagination, and weekly table partitioning
+- **🛡️ Type Safety** - Full TypeScript implementation with comprehensive type definitions
+- **🔍 Detailed Logging** - Progress tracking and debug information for monitoring
+- **📱 Device Analytics** - Screen resolution, device model, and operating system data
+- **🌍 Geographic Data** - Country, region, and city information from visitor sessions
 
-```sh
+## 🚀 Quick Start
+
+### Global Installation
+
+```bash
 npx @socialgouv/matomo-postgres
 ```
 
-### Environment variables Deployment
-
-| name              | value                                                    |
-| ----------------- | -------------------------------------------------------- |
-| MATOMO_KEY\*      | matomo api token                                         |
-| MATOMO_SITE\*     | matomo site id                                           |
-| MATOMO_URL\*      | matomo url                                               |
-| PGDATABASE\*      | Postgres connection string                               |
-| DESTINATION_TABLE | `matomo`                                                 |
-| STARTDATE         | default to today()                                       |
-| RESULTPERPAGE     | matomo pagination (defaults to 500)                      |
-| INITIAL_OFFSET    | How many days to fetch on initialisation (defaults to 3) |
-
-## Dev
-
-```sh
-docker-compose up
-export MATOMO_URL=
-export MATOMO_SITE=
-export MATOMO_KEY=
-export DESTINATION_TABLE= # optional
-export STARTDATE= # optional
-export OFFSET= # optional
-export PGDATABASE=postgres://postgres:postgres@127.0.0.1:5455/postgres
-yarn start
+### Local Installation
+
+```bash
+npm install @socialgouv/matomo-postgres
+# or
+yarn add @socialgouv/matomo-postgres
+```
+
+## ⚙️ Configuration
+
+### Required Environment Variables
+
+| Variable      | Description                          | Example                               |
+| ------------- | ------------------------------------ | ------------------------------------- |
+| `MATOMO_KEY`  | Matomo API authentication token      | `your_api_token_here`                 |
+| `MATOMO_SITE` | Numeric site ID in Matomo            | `1`                                   |
+| `MATOMO_URL`  | Base URL of your Matomo installation | `https://analytics.example.com/`      |
+| `PGDATABASE`  | PostgreSQL connection string         | `postgresql://user:pass@host:5432/db` |
+
+### Optional Environment Variables
+
+| Variable                        | Default              | Description                                             |
+| ------------------------------- | -------------------- | ------------------------------------------------------- |
+| `DESTINATION_TABLE`             | `matomo`             | Selects which table to write to (normal or partitioned) |
+| `MATOMO_TABLE_NAME`             | `matomo`             | Name for the standard table                             |
+| `PARTITIONED_MATOMO_TABLE_NAME` | `matomo_partitioned` | Name for the partitioned table                          |
+| `STARTDATE`                     | Auto-detected        | Override start date for initial import (YYYY-MM-DD)     |
+| `RESULTPERPAGE`                 | `500`                | API pagination size (max results per request)           |
+| `INITIAL_OFFSET`                | `3`                  | Days to look back on first run                          |
+
+## 🗂️ Table Architecture
+
+The tool implements a dual table system to optimize performance for different use cases:
+
+### Standard vs Partitioned Tables
+
+The application creates both a **standard table** and a **partitioned table**:
+
+- **Standard Table** (`MATOMO_TABLE_NAME`): Traditional PostgreSQL table, suitable for smaller datasets or simpler deployments
+- **Partitioned Table** (`PARTITIONED_MATOMO_TABLE_NAME`): Weekly partitioned table optimized for large datasets and improved query performance
+
+### Table Selection
+
+Use the `DESTINATION_TABLE` environment variable to specify which table receives the imported data:
+
+```bash
+# Write to standard table
+export DESTINATION_TABLE=matomo
+
+# Write to partitioned table
+export DESTINATION_TABLE=matomo_partitioned
+
+# Write to custom table name
+export DESTINATION_TABLE=my_custom_analytics_table
+```
+
+### When to Use Partitioned Tables
+
+Consider using partitioned tables when:
+
+- **Large Data Volumes**: Importing months or years of analytics data
+- **Query Performance**: Need faster queries on specific date ranges
+- **Maintenance Operations**: Easier to manage large datasets with partition pruning
+- **Storage Optimization**: Better compression and maintenance of historical data
+
+Both tables share the same schema structure, ensuring compatibility regardless of your choice.
+
+## 🏗️ Architecture
+
+The tool follows a systematic ETL process:
+
+1. **📅 Date Range Detection** - Determines import range based on last sync or configuration
+2. **📥 Data Extraction** - Fetches visitor data from Matomo's `Live.getLastVisitsDetails` API
+3. **🔄 Data Transformation** - Converts visits into structured events with proper typing
+4. **💾 Data Loading** - Inserts events into PostgreSQL with conflict resolution
+5. **📈 Progress Tracking** - Provides detailed logging and resumable operations
+
+### Database Schema
+
+The tool creates a comprehensive table structure capturing:
+
+- **Visitor Information**: IDs, geographic location, device details
+- **Session Metrics**: Duration, visit count, visitor type
+- **Event Data**: Actions, categories, values, timestamps (UTC)
+- **Custom Dimensions**: Flexible JSON fields for custom tracking
+- **Performance Data**: Screen resolution, time spent per action
+
+## 🛠️ Development
+
+### Local Setup
+
+1. **Start PostgreSQL**:
+
+   ```bash
+   docker-compose up
+   ```
+
+2. **Set Environment Variables**:
+
+   ```bash
+   export MATOMO_URL=https://your-matomo-instance/
+   export MATOMO_SITE=your_site_id
+   export MATOMO_KEY=your_api_token
+   export PGDATABASE=postgres://postgres:postgres@127.0.0.1:5455/postgres
+   ```
+
+3. **Run the Application**:
+
+   ```bash
+   yarn start
+   ```
+
+### Development Commands
+
+```bash
+# Build TypeScript
+yarn build
+
+# Run tests
+yarn test
+
+# Update test snapshots
+yarn test -u
+
+# Lint code
+yarn lint
+
+# Fix linting issues
+yarn lint:fix
+
+# Run database migrations
+yarn migrate
 ```
 
-Use `yarn test -u` to update the snapshots
+## 🗄️ Database Migrations
 
-## Database migrations
+Database schema is managed through Kysely migrations located in `./src/migrations/`:
 
-`yarn migrate` is run on each `yarn start` with Kysely migrations at [./src/migrations](./src/migrations/)
+Migrations run automatically on each `yarn start` to ensure schema compatibility.
 
+## 📊 Data Flow
+
+1. **Initialization** - Determine import date range based on:
+   - Explicit date parameter
+   - Last event timestamp in database
+   - `STARTDATE` environment variable
+   - Default offset from current date
+
+2. **Sequential Processing** - For each date:
+   - Check existing records for pagination offset
+   - Fetch visitor data in paginated chunks
+   - Transform visits into individual events
+   - Insert with conflict resolution
+
+3. **Concurrency Control**:
+   - Sequential date processing (one day at a time)
+   - Parallel event insertion (configurable)
+   - Automatic pagination for large datasets
+
+## 🐛 Troubleshooting
+
+### Common Issues
+
+**API Authentication Errors**
+
+- Verify `MATOMO_KEY` has sufficient permissions
+- Ensure `MATOMO_SITE` ID is correct
+- Check `MATOMO_URL` includes trailing slash
+
+**Database Connection Issues**
+
+- Verify PostgreSQL is running and accessible
+- Check `PGDATABASE` connection string format
+- Ensure database exists and user has write permissions
+
+**Performance Issues**
+
+- Adjust `RESULTPERPAGE` for optimal API performance
+- Monitor database indexes and partitioning
+- Consider running during off-peak hours for large imports
+
+### Debug Mode
+
+Enable detailed logging:
+
+```bash
+DEBUG=matomo-postgres* npx @socialgouv/matomo-postgres
+```

package/dist/__tests__/importDate.test.js

@@ -76,38 +76,48 @@ test('importDate: should import given date', () => __awaiter(void 0, void 0, voi
 `);
     expect(queries.length).toEqual(1 + matomoVisit.actionDetails.length * 2);
 }));
-test('importDate: should paginate matomo API calls and produce 46 queries', () => __awaiter(void 0, void 0, void 0, function* () {
+test('importDate: should handle pagination across multiple pages', () => __awaiter(void 0, void 0, void 0, function* () {
     const piwikApi = jest.fn();
-    let calls = 0;
-    piwikApi.mockImplementation((options, cb) => {
-        cb(null, Array.from({ length: calls ? 5 : 10 }, (k, _v) => (Object.assign(Object.assign({}, matomoVisit), { idVisit: k }))));
-        calls++;
+    // Mock first call to return exactly 10 visits (triggers pagination)
+    piwikApi
+        .mockImplementationOnce((options, cb) => {
+        const visits = Array.from({ length: 10 }, (_, i) => (Object.assign(Object.assign({}, matomoVisit), { idVisit: 200 + i })));
+        cb(null, visits);
+    })
+        // Mock second call to return 5 visits (stops pagination)
+        .mockImplementationOnce((options, cb) => {
+        const visits = Array.from({ length: 5 }, (_, i) => (Object.assign(Object.assign({}, matomoVisit), { idVisit: 300 + i })));
+        cb(null, visits);
     });
-    pool.query.mockResolvedValueOnce({ rows: [], rowCount: 0 });
-    yield importDate(piwikApi, TEST_DATE);
+    // Mock database query for record count
+    pool.query.mockResolvedValue({ rows: [], rowCount: 0 });
+    const result = yield importDate(piwikApi, TEST_DATE);
+    // Should make exactly 2 API calls due to pagination
     expect(piwikApi.mock.calls.length).toEqual(2);
-    expect(piwikApi.mock.calls[0][0]).toMatchInlineSnapshot(`
-{
-  "date": "2023-04-15",
-  "filter_limit": 10,
-  "filter_offset": 0,
-  "filter_sort_order": "asc",
-  "idSite": "42",
-  "method": "Live.getLastVisitsDetails",
-  "period": "day",
-}
-`);
-    expect(piwikApi.mock.calls[1][0]).toMatchInlineSnapshot(`
-{
-  "date": "2023-04-15",
-  "filter_limit": 10,
-  "filter_offset": 10,
-  "filter_sort_order": "asc",
-  "idSite": "42",
-  "method": "Live.getLastVisitsDetails",
-  "period": "day",
-}
-`);
-    expect(queries.length).toEqual(1 + matomoVisit.actionDetails.length * 15);
-    expect(queries).toMatchSnapshot();
+    // First call should have offset 0
+    expect(piwikApi.mock.calls[0][0]).toMatchObject({
+        date: '2023-04-15',
+        filter_limit: 10,
+        filter_offset: 0,
+        filter_sort_order: 'asc',
+        idSite: '42',
+        method: 'Live.getLastVisitsDetails',
+        period: 'day'
+    });
+    // Second call should have offset 10
+    expect(piwikApi.mock.calls[1][0]).toMatchObject({
+        date: '2023-04-15',
+        filter_limit: 10,
+        filter_offset: 10,
+        filter_sort_order: 'asc',
+        idSite: '42',
+        method: 'Live.getLastVisitsDetails',
+        period: 'day'
+    });
+    // Should process all events from both pages
+    // 15 visits total × 3 actionDetails each = 45 events
+    expect(result.length).toEqual(45);
+    // Verify database queries: 1 count query + (45 events × 1 query per event)
+    // Note: Each event generates 1 database query for insertion
+    expect(queries.length).toEqual(1 + 45);
 }));

package/dist/__tests__/run.test.js

@@ -7,11 +7,13 @@ var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, ge
         step((generator = generator.apply(thisArg, _arguments || [])).next());
     });
 };
-import run from '../index';
 process.env.MATOMO_SITE = '42';
 process.env.PROJECT_NAME = 'some-project';
 process.env.RESULTPERPAGE = '10';
-process.env.STARTDATE = '2023-03-27'; // Set a start date that's before our test date
+delete process.env.INITIAL_OFFSET;
+delete process.env.DESTINATION_TABLE;
+delete process.env.STARTDATE;
+// Clear STARTDATE to avoid conflicts with fake timers
 const TEST_DATE = new Date(2023, 3, 1);
 let queries = [];
 let piwikApiCalls = [];
@@ -41,18 +43,22 @@ jest.mock('../PiwikClient', () => {
         }
         // eslint-disable-next-line @typescript-eslint/no-unsafe-function-type
         api(options, cb) {
-            piwikApiCalls.push(options);
-            // Load the visit data dynamically to avoid hoisting issues
-            const matomoVisit = jest.requireActual('./visit.json');
-            const matomoVisits = [
-                Object.assign(Object.assign({}, matomoVisit), { idVisit: 123 }),
-                Object.assign(Object.assign({}, matomoVisit), { idVisit: 124 })
-            ];
-            cb(null, matomoVisits);
+            return __awaiter(this, void 0, void 0, function* () {
+                // Import the visit data dynamically to avoid circular dependency
+                const { default: matomoVisit } = yield import('./visit.json');
+                const matomoVisits = [
+                    Object.assign(Object.assign({}, matomoVisit), { idVisit: 123 }),
+                    Object.assign(Object.assign({}, matomoVisit), { idVisit: 124 })
+                ];
+                piwikApiCalls.push(options);
+                cb(null, matomoVisits);
+            });
         }
     }
     return PiwikMock;
 });
+// Import after mocks are set up
+import run from '../index';
 beforeEach(() => {
     queries = [];
     piwikApiCalls = [];
@@ -78,5 +84,7 @@ test('run: should run SQL queries', () => __awaiter(void 0, void 0, void 0, func
     jest.useFakeTimers().setSystemTime(TEST_DATE.getTime());
     yield run();
     expect(queries).toMatchSnapshot();
-    expect(queries.length).toEqual(49); // Number of queries based on current implementation
+    // Updated expectation based on actual behavior with INITIAL_OFFSET=3 (5 days total: 3 days before + today + 1 day after)
+    // 5 days * (6 events per day + 1 count query per day) + 1 initial query for last event lookup
+    expect(queries.length).toEqual(1 + 5 * (6 + 1));
 }));

package/dist/config.js

@@ -2,7 +2,10 @@ export const MATOMO_KEY = process.env.MATOMO_KEY || '';
 export const MATOMO_URL = process.env.MATOMO_URL || 'https://matomo.fabrique.social.gouv.fr/';
 export const MATOMO_SITE = process.env.MATOMO_SITE || 0;
 export const PGDATABASE = process.env.PGDATABASE || '';
-export const DESTINATION_TABLE = process.env.DESTINATION_TABLE || 'matomo';
-export const MATOMO_TABLE_NAME = process.env.MATOMO_TABLE_NAME || 'matomo';
 export const INITIAL_OFFSET = process.env.INITIAL_OFFSET || '3';
 export const RESULTPERPAGE = process.env.RESULTPERPAGE || '500';
+// We will create both a normal and a partitioned table (MATOMO_TABLE_NAME and PARTITIONED_MATOMO_TABLE_NAME)
+// and use DESTINATION_TABLE to determine which one to write to.
+export const DESTINATION_TABLE = process.env.DESTINATION_TABLE || 'matomo';
+export const MATOMO_TABLE_NAME = process.env.MATOMO_TABLE_NAME || 'matomo';
+export const PARTITIONED_MATOMO_TABLE_NAME = process.env.PARTITIONED_MATOMO_TABLE_NAME || 'matomo_partitioned';

package/dist/importDate.js

@@ -30,7 +30,7 @@ const getRecordsCount = (date) => __awaiter(void 0, void 0, void 0, function* ()
     return count;
 });
 /** import all event from givent date */
-export const importDate = (piwikApi, date, filterOffset = 0) => __awaiter(void 0, void 0, void 0, function* () {
+export const importDate = (piwikApi_1, date_1, ...args_1) => __awaiter(void 0, [piwikApi_1, date_1, ...args_1], void 0, function* (piwikApi, date, filterOffset = 0) {
     const limit = parseInt(RESULTPERPAGE);
     const offset = filterOffset || (yield getRecordsCount(isoDate(date)));
     if (!offset) {

package/dist/importEvent.js

@@ -18,55 +18,127 @@ import { db } from './db.js';
  */
 export const importEvent = (event) => __awaiter(void 0, void 0, void 0, function* () {
     var _a, _b, _c, _d, _e, _f, _g, _h, _j, _k, _l, _m, _o, _p, _q, _r, _s, _t, _u, _v, _w, _x, _y, _z, _0, _1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _13, _14;
-    // Use the stored procedure for safe insertion with automatic partition creation
-    yield sql `
-    SELECT insert_into_matomo_partitioned(
-      ${(_a = event.action_id) !== null && _a !== void 0 ? _a : ''},
-      ${event.action_timestamp ? new Date(event.action_timestamp) : new Date()},
-      ${(_b = event.idsite) !== null && _b !== void 0 ? _b : ''},
-      ${(_c = event.idvisit) !== null && _c !== void 0 ? _c : ''},
-      ${(_d = event.actions) !== null && _d !== void 0 ? _d : null},
-      ${(_e = event.country) !== null && _e !== void 0 ? _e : null},
-      ${(_f = event.region) !== null && _f !== void 0 ? _f : null},
-      ${(_g = event.city) !== null && _g !== void 0 ? _g : null},
-      ${(_h = event.operatingsystemname) !== null && _h !== void 0 ? _h : null},
-      ${(_j = event.devicemodel) !== null && _j !== void 0 ? _j : null},
-      ${(_k = event.devicebrand) !== null && _k !== void 0 ? _k : null},
-      ${(_l = event.visitduration) !== null && _l !== void 0 ? _l : null},
-      ${(_m = event.dayssincefirstvisit) !== null && _m !== void 0 ? _m : null},
-      ${(_o = event.visitortype) !== null && _o !== void 0 ? _o : null},
-      ${(_p = event.sitename) !== null && _p !== void 0 ? _p : null},
-      ${(_q = event.userid) !== null && _q !== void 0 ? _q : null},
-      ${event.serverdateprettyfirstaction
-        ? new Date(event.serverdateprettyfirstaction)
-        : null},
-      ${(_r = event.action_type) !== null && _r !== void 0 ? _r : ''},
-      ${(_s = event.action_eventcategory) !== null && _s !== void 0 ? _s : ''},
-      ${(_t = event.action_eventaction) !== null && _t !== void 0 ? _t : ''},
-      ${(_u = event.action_eventname) !== null && _u !== void 0 ? _u : ''},
-      ${event.action_eventvalue ? Number(event.action_eventvalue) : 0},
-      ${(_v = event.action_timespent) !== null && _v !== void 0 ? _v : '0'},
-      ${(_w = event.usercustomproperties) !== null && _w !== void 0 ? _w : null},
-      ${(_x = event.usercustomdimensions) !== null && _x !== void 0 ? _x : null},
-      ${(_y = event.dimension1) !== null && _y !== void 0 ? _y : null},
-      ${(_z = event.dimension2) !== null && _z !== void 0 ? _z : null},
-      ${(_0 = event.dimension3) !== null && _0 !== void 0 ? _0 : null},
-      ${(_1 = event.dimension4) !== null && _1 !== void 0 ? _1 : null},
-      ${(_2 = event.dimension5) !== null && _2 !== void 0 ? _2 : null},
-      ${(_3 = event.dimension6) !== null && _3 !== void 0 ? _3 : null},
-      ${(_4 = event.dimension7) !== null && _4 !== void 0 ? _4 : null},
-      ${(_5 = event.dimension8) !== null && _5 !== void 0 ? _5 : null},
-      ${(_6 = event.dimension9) !== null && _6 !== void 0 ? _6 : null},
-      ${(_7 = event.dimension10) !== null && _7 !== void 0 ? _7 : null},
-      ${(_8 = event.action_url) !== null && _8 !== void 0 ? _8 : null},
-      ${(_9 = event.sitesearchkeyword) !== null && _9 !== void 0 ? _9 : null},
-      ${(_10 = event.action_title) !== null && _10 !== void 0 ? _10 : null},
-      ${(_11 = event.visitorid) !== null && _11 !== void 0 ? _11 : null},
-      ${(_12 = event.referrertype) !== null && _12 !== void 0 ? _12 : null},
-      ${(_13 = event.referrername) !== null && _13 !== void 0 ? _13 : null},
-      ${(_14 = event.resolution) !== null && _14 !== void 0 ? _14 : null}
-    )
-  `.execute(db);
+    // Build a sanitized, typed data object to reduce drift and ensure defaults in one place
+    const eventData = {
+        action_id: (_a = event.action_id) !== null && _a !== void 0 ? _a : '',
+        action_timestamp: event.action_timestamp
+            ? new Date(event.action_timestamp)
+            : new Date(),
+        idsite: (_b = event.idsite) !== null && _b !== void 0 ? _b : '',
+        idvisit: (_c = event.idvisit) !== null && _c !== void 0 ? _c : '',
+        actions: (_d = event.actions) !== null && _d !== void 0 ? _d : null,
+        country: (_e = event.country) !== null && _e !== void 0 ? _e : null,
+        region: (_f = event.region) !== null && _f !== void 0 ? _f : null,
+        city: (_g = event.city) !== null && _g !== void 0 ? _g : null,
+        operatingsystemname: (_h = event.operatingsystemname) !== null && _h !== void 0 ? _h : null,
+        devicemodel: (_j = event.devicemodel) !== null && _j !== void 0 ? _j : null,
+        devicebrand: (_k = event.devicebrand) !== null && _k !== void 0 ? _k : null,
+        visitduration: (_l = event.visitduration) !== null && _l !== void 0 ? _l : null,
+        dayssincefirstvisit: (_m = event.dayssincefirstvisit) !== null && _m !== void 0 ? _m : null,
+        visitortype: (_o = event.visitortype) !== null && _o !== void 0 ? _o : null,
+        sitename: (_p = event.sitename) !== null && _p !== void 0 ? _p : null,
+        userid: (_q = event.userid) !== null && _q !== void 0 ? _q : null,
+        serverdateprettyfirstaction: event.serverdateprettyfirstaction
+            ? new Date(event.serverdateprettyfirstaction)
+            : null,
+        action_type: (_r = event.action_type) !== null && _r !== void 0 ? _r : '',
+        action_eventcategory: (_s = event.action_eventcategory) !== null && _s !== void 0 ? _s : '',
+        action_eventaction: (_t = event.action_eventaction) !== null && _t !== void 0 ? _t : '',
+        action_eventname: (_u = event.action_eventname) !== null && _u !== void 0 ? _u : '',
+        action_eventvalue: event.action_eventvalue
+            ? Number(event.action_eventvalue)
+            : 0,
+        action_timespent: (_v = event.action_timespent) !== null && _v !== void 0 ? _v : '0',
+        usercustomproperties: (_w = event.usercustomproperties) !== null && _w !== void 0 ? _w : null,
+        usercustomdimensions: (_x = event.usercustomdimensions) !== null && _x !== void 0 ? _x : null,
+        dimension1: (_y = event.dimension1) !== null && _y !== void 0 ? _y : null,
+        dimension2: (_z = event.dimension2) !== null && _z !== void 0 ? _z : null,
+        dimension3: (_0 = event.dimension3) !== null && _0 !== void 0 ? _0 : null,
+        dimension4: (_1 = event.dimension4) !== null && _1 !== void 0 ? _1 : null,
+        dimension5: (_2 = event.dimension5) !== null && _2 !== void 0 ? _2 : null,
+        dimension6: (_3 = event.dimension6) !== null && _3 !== void 0 ? _3 : null,
+        dimension7: (_4 = event.dimension7) !== null && _4 !== void 0 ? _4 : null,
+        dimension8: (_5 = event.dimension8) !== null && _5 !== void 0 ? _5 : null,
+        dimension9: (_6 = event.dimension9) !== null && _6 !== void 0 ? _6 : null,
+        dimension10: (_7 = event.dimension10) !== null && _7 !== void 0 ? _7 : null,
+        action_url: (_8 = event.action_url) !== null && _8 !== void 0 ? _8 : null,
+        sitesearchkeyword: (_9 = event.sitesearchkeyword) !== null && _9 !== void 0 ? _9 : null,
+        action_title: (_10 = event.action_title) !== null && _10 !== void 0 ? _10 : null,
+        visitorid: (_11 = event.visitorid) !== null && _11 !== void 0 ? _11 : null,
+        referrertype: (_12 = event.referrertype) !== null && _12 !== void 0 ? _12 : null,
+        referrername: (_13 = event.referrername) !== null && _13 !== void 0 ? _13 : null,
+        resolution: (_14 = event.resolution) !== null && _14 !== void 0 ? _14 : null
+    };
+    // Minimal runtime validation for required fields
+    if (!eventData.action_id || eventData.action_id.trim().length === 0) {
+        throw new Error('importEvent(): action_id is required and cannot be empty');
+    }
+    if (!(eventData.action_timestamp instanceof Date) ||
+        isNaN(eventData.action_timestamp.getTime())) {
+        throw new Error('importEvent(): action_timestamp is invalid');
+    }
+    try {
+        // Keep the stored procedure but centralize mapping to avoid parameter mis-ordering
+        yield sql `
+      SELECT insert_into_matomo_partitioned(
+        ${eventData.action_id},
+        ${eventData.action_timestamp},
+        ${eventData.idsite},
+        ${eventData.idvisit},
+        ${eventData.actions},
+        ${eventData.country},
+        ${eventData.region},
+        ${eventData.city},
+        ${eventData.operatingsystemname},
+        ${eventData.devicemodel},
+        ${eventData.devicebrand},
+        ${eventData.visitduration},
+        ${eventData.dayssincefirstvisit},
+        ${eventData.visitortype},
+        ${eventData.sitename},
+        ${eventData.userid},
+        ${eventData.serverdateprettyfirstaction},
+        ${eventData.action_type},
+        ${eventData.action_eventcategory},
+        ${eventData.action_eventaction},
+        ${eventData.action_eventname},
+        ${eventData.action_eventvalue},
+        ${eventData.action_timespent},
+        ${eventData.usercustomproperties},
+        ${eventData.usercustomdimensions},
+        ${eventData.dimension1},
+        ${eventData.dimension2},
+        ${eventData.dimension3},
+        ${eventData.dimension4},
+        ${eventData.dimension5},
+        ${eventData.dimension6},
+        ${eventData.dimension7},
+        ${eventData.dimension8},
+        ${eventData.dimension9},
+        ${eventData.dimension10},
+        ${eventData.action_url},
+        ${eventData.sitesearchkeyword},
+        ${eventData.action_title},
+        ${eventData.visitorid},
+        ${eventData.referrertype},
+        ${eventData.referrername},
+        ${eventData.resolution}
+      )
+    `.execute(db);
+    }
+    catch (err) {
+        // Add context for troubleshooting
+        const minimalContext = {
+            action_id: eventData.action_id,
+            action_timestamp: eventData.action_timestamp,
+            idsite: eventData.idsite,
+            idvisit: eventData.idvisit
+        };
+        console.error('importEvent(): failed to insert event', minimalContext);
+        // Log error details but avoid exposing sensitive information
+        console.error('importEvent(): error', err instanceof Error ? err.message : 'Unknown error');
+        throw err;
+    }
 });
 const matomoProps = [
     'idSite',
@@ -103,9 +175,10 @@ const actionProps = {
 };
 export const getEventsFromMatomoVisit = (matomoVisit) => {
     return matomoVisit.actionDetails.map((actionDetail, actionIndex) => {
+        var _a;
         const usercustomproperties = {};
         for (let k = 1; k < 10; k++) {
-            const property = actionDetail.customVariables && actionDetail.customVariables[k];
+            const property = (_a = actionDetail.customVariables) === null || _a === void 0 ? void 0 : _a[k];
             if (!property)
                 continue; // max 10 custom variables
             //@ts-expect-error implicit any type

package/dist/index.js

@@ -11,7 +11,7 @@ import { eachDayOfInterval } from 'date-fns';
 import startDebug from 'debug';
 import { sql } from 'kysely';
 import pAll from 'p-all';
-import { DESTINATION_TABLE, INITIAL_OFFSET, MATOMO_KEY, MATOMO_SITE, MATOMO_URL, PGDATABASE } from './config.js';
+import { DESTINATION_TABLE, INITIAL_OFFSET, MATOMO_KEY, MATOMO_SITE, MATOMO_URL } from './config.js';
 import { db } from './db.js';
 import { importDate } from './importDate.js';
 import PiwikClient from './PiwikClient.js';
@@ -35,10 +35,6 @@ function run(date) {
             referenceDate = new Date(date);
             console.log(`✅ Using provided date parameter: ${referenceDate.toISOString()}`);
         }
-        if (!referenceDate && process.env.STARTDATE) {
-            referenceDate = new Date(process.env.STARTDATE);
-            console.log(`✅ Using STARTDATE environment variable: ${referenceDate.toISOString()}`);
-        }
         if (!referenceDate) {
             console.log(`🔍 Looking for last event in database...`);
             referenceDate = yield findLastEventInMatomo(db);
@@ -49,6 +45,10 @@ function run(date) {
                 console.log(`ℹ️ No previous events found in database`);
             }
         }
+        if (!referenceDate && process.env.STARTDATE) {
+            referenceDate = new Date(process.env.STARTDATE);
+            console.log(`✅ Using STARTDATE environment variable: ${referenceDate.toISOString()}`);
+        }
         if (!referenceDate) {
             referenceDate = new Date(new Date().getTime() - +INITIAL_OFFSET * 24 * 60 * 60 * 1000);
             console.log(`✅ Using default offset (${INITIAL_OFFSET} days ago): ${referenceDate.toISOString()}`);
@@ -95,14 +95,3 @@ function findLastEventInMatomo(db) {
     });
 }
 export default run;
-(() => __awaiter(void 0, void 0, void 0, function* () {
-    if (!MATOMO_SITE)
-        return console.error('Missing env MATOMO_SITE');
-    if (!MATOMO_KEY)
-        return console.error('Missing env MATOMO_KEY');
-    if (!PGDATABASE)
-        return console.error('Missing env PGDATABASE');
-    yield run();
-    debug('run finished');
-    db.destroy();
-}))();

package/dist/migrate-down.js

@@ -10,9 +10,6 @@ var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, ge
 import { promises as fs } from 'fs';
 import { FileMigrationProvider, Migrator } from 'kysely';
 import * as path from 'path';
-import { fileURLToPath } from 'url';
-const __filename = fileURLToPath(import.meta.url);
-const __dirname = path.dirname(__filename);
 import { db } from './db.js';
 function migrateDown() {
     return __awaiter(this, void 0, void 0, function* () {
@@ -21,7 +18,7 @@ function migrateDown() {
             provider: new FileMigrationProvider({
                 fs,
                 path,
-                migrationFolder: __dirname + '/migrations'
+                migrationFolder: path.join(path.dirname(new URL(import.meta.url).pathname), 'migrations')
             })
         });
         const { error, results } = yield migrator.migrateDown();

package/dist/migrate-latest.js

@@ -10,49 +10,58 @@ var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, ge
 import { promises as fs } from 'fs';
 import { FileMigrationProvider, Migrator } from 'kysely';
 import * as path from 'path';
-import { fileURLToPath } from 'url';
-const __filename = fileURLToPath(import.meta.url);
-const __dirname = path.dirname(__filename);
 import { MATOMO_TABLE_NAME } from './config.js';
 import { db } from './db.js';
 function migrateToLatest() {
     return __awaiter(this, void 0, void 0, function* () {
-        const migrator = new Migrator({
-            db,
-            provider: new FileMigrationProvider({
-                fs,
-                path,
-                migrationFolder: __dirname + '/migrations'
-            }),
-            // allow to have mutliple migratable instances in a single schema
-            migrationTableName: `${MATOMO_TABLE_NAME}_migration`,
-            migrationLockTableName: `${MATOMO_TABLE_NAME}_migration_lock`
-        });
-        const { error, results } = yield migrator.migrateToLatest();
-        results === null || results === void 0 ? void 0 : results.forEach((it) => {
-            if (it.status === 'Success') {
-                console.log(`migration "${it.migrationName}" was executed successfully`);
+        console.log(`Starting migrate to latest`);
+        try {
+            const migrator = new Migrator({
+                db,
+                provider: new FileMigrationProvider({
+                    fs,
+                    path,
+                    migrationFolder: path.join(path.dirname(new URL(import.meta.url).pathname), 'migrations')
+                }),
+                // allow to have mutliple migratable instances in a single schema
+                migrationTableName: `${MATOMO_TABLE_NAME}_migration`,
+                migrationLockTableName: `${MATOMO_TABLE_NAME}_migration_lock`
+            });
+            const { error, results } = yield migrator.migrateToLatest();
+            results === null || results === void 0 ? void 0 : results.forEach((it) => {
+                if (it.status === 'Success') {
+                    console.log(`migration "${it.migrationName}" was executed successfully`);
+                }
+                else if (it.status === 'Error') {
+                    console.error(`failed to execute migration "${it.migrationName}"`);
+                }
+            });
+            if (error) {
+                console.error('failed to migrate');
+                console.error(error);
+                process.exit(1);
             }
-            else if (it.status === 'Error') {
-                console.error(`failed to execute migration "${it.migrationName}"`);
-            }
-        });
-        if (error) {
-            console.error('failed to migrate');
-            console.error(error);
-            process.exit(1);
-        }
-        else {
-            if (!(results === null || results === void 0 ? void 0 : results.length)) {
+            else if (!(results === null || results === void 0 ? void 0 : results.length)) {
                 console.log('No migration to run');
             }
         }
+        catch (uncaughtError) {
+            console.error('UNCAUGHT ERROR during migration:');
+            console.error('Error message:', uncaughtError instanceof Error
+                ? uncaughtError.message
+                : String(uncaughtError));
+            console.error('Error stack:', uncaughtError instanceof Error
+                ? uncaughtError.stack
+                : 'No stack trace available');
+            console.error('Full error object:', uncaughtError);
+            process.exit(1);
+        }
     });
 }
 export default migrateToLatest;
 export function startMigration() {
     return __awaiter(this, void 0, void 0, function* () {
         yield migrateToLatest();
-        yield db.destroy();
+        // Don't destroy the db connection here since the main application will need it
     });
 }

package/dist/migrations/20250425-01-add-resolution.js

@@ -7,13 +7,25 @@ var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, ge
         step((generator = generator.apply(thisArg, _arguments || [])).next());
     });
 };
+import { sql } from 'kysely';
 const MATOMO_TABLE_NAME = process.env.MATOMO_TABLE_NAME || 'matomo';
 export function up(db) {
     return __awaiter(this, void 0, void 0, function* () {
-        yield db.schema
-            .alterTable(MATOMO_TABLE_NAME)
-            .addColumn('resolution', 'text')
-            .execute();
+        // Check if the column already exists before trying to add it
+        const columnExists = yield sql `
+    SELECT EXISTS (
+      SELECT 1
+      FROM information_schema.columns
+      WHERE table_name = ${MATOMO_TABLE_NAME}
+      AND column_name = 'resolution'
+    ) as exists
+  `.execute(db);
+        if (!columnExists.rows[0].exists) {
+            yield db.schema
+                .alterTable(MATOMO_TABLE_NAME)
+                .addColumn('resolution', 'text')
+                .execute();
+        }
     });
 }
 export function down(db) {

package/dist/migrations/20250715-01-weekly-partitioning.js

@@ -8,7 +8,7 @@ var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, ge
     });
 };
 import { sql } from 'kysely';
-const PARTITIONED_MATOMO_TABLE_NAME = process.env.PARTITIONED_MATOMO_TABLE_NAME || 'matomo_partitioned';
+import { PARTITIONED_MATOMO_TABLE_NAME } from 'src/config';
 export function up(db) {
     return __awaiter(this, void 0, void 0, function* () {
         // First, create the partitioned table structure as a partitioned table
@@ -162,7 +162,7 @@ export function up(db) {
     )
     RETURNS void
     LANGUAGE plpgsql
-    SECURITY DEFINER
+    SECURITY INVOKER
     AS $$
     BEGIN
         -- Ensure partition exists for the given timestamp
@@ -347,8 +347,10 @@ export function up(db) {
 export function down(db) {
     return __awaiter(this, void 0, void 0, function* () {
         // Drop trigger and function
-        yield sql `DROP TRIGGER IF EXISTS ${sql.id(`${PARTITIONED_MATOMO_TABLE_NAME}_auto_partition`)} ON ${sql.id(PARTITIONED_MATOMO_TABLE_NAME)}`.execute(db);
-        yield sql `DROP FUNCTION IF EXISTS ${sql.id(`${PARTITIONED_MATOMO_TABLE_NAME}_partition_trigger`)}()`.execute(db);
+        const trigger_name = `${PARTITIONED_MATOMO_TABLE_NAME}_auto_partition`;
+        yield sql `DROP TRIGGER IF EXISTS ${sql.id(trigger_name)} ON ${sql.id(PARTITIONED_MATOMO_TABLE_NAME)}`.execute(db);
+        const function_name = `${PARTITIONED_MATOMO_TABLE_NAME}_partition_trigger`;
+        yield sql `DROP FUNCTION IF EXISTS ${sql.id(function_name)}()`.execute(db);
         yield sql `DROP FUNCTION IF EXISTS create_weekly_partition_if_not_exists(text, timestamptz)`.execute(db);
         yield sql `DROP FUNCTION IF EXISTS insert_into_matomo_partitioned`.execute(db);
         // Drop the partitioned table (this will also drop all partitions)

package/dist/migrations/20250908-01-convention-analysis-index.js

@@ -0,0 +1,29 @@
+var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
+    function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
+    return new (P || (P = Promise))(function (resolve, reject) {
+        function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
+        function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
+        function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
+        step((generator = generator.apply(thisArg, _arguments || [])).next());
+    });
+};
+import { sql } from 'kysely';
+const PARTITIONED_MATOMO_TABLE_NAME = process.env.PARTITIONED_MATOMO_TABLE_NAME || 'matomo_partitioned';
+export function up(db) {
+    return __awaiter(this, void 0, void 0, function* () {
+        // Create conditional index for convention collective analysis
+        yield sql `
+    CREATE INDEX IF NOT EXISTS idx_convention_analysis_matomo_partitioned
+    ON ${sql.id(PARTITIONED_MATOMO_TABLE_NAME)} (action_type, action_url, action_timestamp)
+    WHERE action_url LIKE 'https://code.travail.gouv.fr/convention-collective/%'
+  `.execute(db);
+    });
+}
+export function down(db) {
+    return __awaiter(this, void 0, void 0, function* () {
+        // Drop the conditional index
+        yield sql `
+    DROP INDEX IF EXISTS idx_convention_analysis_matomo_partitioned
+  `.execute(db);
+    });
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@socialgouv/matomo-postgres",
   "description": "Extract visitor events from Matomo API and push to Postgres",
-  "version": "2.2.0-beta.3",
+  "version": "2.2.1",
   "types": "types/index.d.ts",
   "license": "Apache-2.0",
   "main": "dist/index.js",
@@ -42,7 +42,7 @@
     "@eslint/js": "^9.31.0",
     "@types/debug": "^4.1.7",
     "@types/jest": "^29.4.0",
-    "@types/node": "^18.14.4",
+    "@types/node": "^22.0.0",
     "@types/pg": "^8.6.6",
     "@typescript-eslint/eslint-plugin": "^8.37.0",
     "@typescript-eslint/parser": "^8.37.0",
@@ -51,11 +51,10 @@
     "eslint-plugin-prettier": "^5.5.1",
     "eslint-plugin-simple-import-sort": "^12.1.1",
     "globals": "^16.3.0",
-    "jest": "^29.4.3",
-    "knip": "^5.61.3",
+    "jest": "^29.7.0",
     "prettier": "^3.6.2",
-    "ts-jest": "^29.0.5",
+    "ts-jest": "^29.4.1",
     "ts-node": "^10.9.1",
-    "typescript": "^4.9.5"
+    "typescript": "^5.0.0"
   }
 }