@bytebrand/export-cars 1.26.38 → 1.26.40

package/docs/criteo/RUNBOOK.md

@@ -0,0 +1,86 @@
+# Criteo catalog feed — runbook (ADV-319)
+
+Status 06.11.2026, `@bytebrand/export-cars` 1.26.40. DEV only; PROD events `4.0` / `4.1` are disabled
+and no Criteo account, contract or importer test exists yet. Nothing here allows a live upload.
+
+## What the job does
+
+`4.0 export criteo - part 1` (`exportType=criteo`) reads the base export snapshot
+(`resources/cars-export/cars.<n>.json`) and writes the legacy Criteo RSS/XML file
+`resources/criteo-export/api_export_cars_criteo.xml`. The chained `4.1 Export criteo - part 2` uploads that
+file. Format, enums and field order are the legacy ones (decision 20.10.2026: restore as before).
+
+Per car, in this order (first match wins, counted in `excluded`):
+
+| Reason | Rule |
+|---|---|
+| `no_id` | empty `_id` |
+| `duplicate_id` | `_id` already seen in this run (first one is kept) |
+| `filtered` | shared partner filter: `metaData.imagesCount > 0` and the Cronicle rule params |
+| `reserved_or_sold` | `offer.offerAvailability` is reserved or sold (Criteo 16.10: only marketable stock) |
+| `no_valid_price` | `price.currentSalesPrice` missing, not a number or ≤ 0 |
+| `no_marketing_image` | no generated `criteo-<id>.jpg`; no noimage placeholder is ever sent |
+
+A car that went offline is not in the snapshot, so it is missing from the next full file; Criteo's next
+full import removes it. `published-ids.json` keeps the ids of the last published file, and the report
+lists `ids.added` / `ids.removed` / `ids.kept` (+ up to 20 removed ids) for each run.
+
+Item fields: `installment` and `custom_label_0` only when the car has a real rate (months and amount > 0);
+`energy_efficiency_class` only for a known WLTP class.
+
+## Fail-closed guard
+
+Shared `FeedGuard` (same as meinestadt / AutoUncle / OpenAI). The new XML is written to
+`api_export_cars_criteo.xml.tmp` and renamed over the published file only after all checks passed; the
+previous file is kept as `.prev`. A blocked run keeps the published file, moves the temp file to
+`.rejected`, and **fails the job** (Cronicle code 1), so `4.1` does not upload anything.
+
+| Code | Meaning | Action |
+|---|---|---|
+| `source_empty` | no snapshot chunk files (base export failed / wiped) | fix the base export, re-run |
+| `source_partial` | chunk files with gaps | re-run the base export |
+| `source_stale` | newest chunk older than `feedGuardMaxSourceAgeHours` (26) | base export did not run |
+| `source_too_small` | snapshot below `feedGuardMinSourceItems` (1000) | check the base export |
+| `published_too_small` | fewer items than `feedGuardMinPublishedItems` (500), e.g. all reserved/sold | check stock / images |
+| `count_drop` | source or items dropped by more than `feedGuardMaxDropPercent` (20 %) vs last good | if the stock really dropped: re-run once with `feedGuardAcceptDrop=true` |
+| `locked` | another run holds `resources/criteo-state/run.lock` (stale after 6 h) | wait / check the other job |
+
+Stale fallback: each report carries `lastGoodAgeHours`; after `feedGuardMaxStaleHours` (48) without a
+successful run, `stale: true` and the log line `criteo feed STALE`. The job itself already failed on every
+blocked run, which is the alert. Whether Criteo should then get an empty file (to stop advertising
+old stock) is an open partner question, not built.
+
+All thresholds are Cronicle params of the `4.0` event (`feedGuardMinSourceItems`,
+`feedGuardMinPublishedItems`, `feedGuardMaxDropPercent`, `feedGuardMaxSourceAgeHours`,
+`feedGuardMaxStaleHours`, `feedGuardAcceptDrop`).
+
+## Files
+
+| Path (relative to `/opt/cronicle`) | Content |
+|---|---|
+| `resources/criteo-export/api_export_cars_criteo.xml` | published file (what `4.1` uploads) |
+| `resources/criteo-export/api_export_cars_criteo.xml.prev` | file before the last publish |
+| `resources/criteo-export/api_export_cars_criteo.xml.rejected` | output of the last blocked run |
+| `resources/criteo-state/last-good.json` | counts + time of the last publish |
+| `resources/criteo-state/last-run-report.json` | report of the last run (also printed as `criteo feed report: {...}`) |
+| `resources/criteo-state/published-ids.json` | ids of the published file |
+
+Report metrics: source chunk age (`source.ageHours`), run time (`durationMs`), source cars, items,
+exclusions by reason, items with a rate, items without energy class, id changes, last good age.
+Criteo's accepted / rejected records come from the Criteo importer, not from this job.
+
+## Rollback
+
+1. Code: `sudo npm --prefix /opt/cronicle install @bytebrand/export-cars@<previous> --prefer-online`
+   (DEV before this release: 1.26.39). Only the export-cars package changes; back up
+   `/opt/cronicle/package.json` and `package-lock.json` first.
+2. File: `cp resources/criteo-export/api_export_cars_criteo.xml.prev resources/criteo-export/api_export_cars_criteo.xml`.
+3. State: `resources/criteo-state/` can be deleted; the next run then has no last good counts (no drop check
+   once) and reports all ids as added.
+
+## Tests
+
+`test/criteo-mapping.test.ts` (item mapping) and `test/criteo-feed-run.test.ts`: 20 anonymized DEV cars
+(`test/fixtures/criteo-20-cases.json`, synthetic ids, mapping fields only) plus sold, reserved, offline,
+no image, price change, invalid price, duplicate id, missing id, source outage, partial / stale snapshot,
+count drop + acceptDrop, empty stock, lock, stale last good, atomic publish. Run: `npm test`.

package/package.json

@@ -1,65 +1,65 @@
-{
-  "name": "@bytebrand/export-cars",
-  "version": "1.26.38",
-  "description": "",
-  "author": "bytebrand",
-  "license": "UNLICENSED",
-  "scripts": {
-    "prebuild": "rimraf dist",
-    "build": "nest build",
-    "prepublishOnly": "npm run build",
-    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
-    "lint": "eslint -c .eslintrc.js --ext .ts src/",
-    "start:dev": "nest start --watch",
-    "test": "ts-node --transpile-only test/google-feed-envkv.test.ts && ts-node --transpile-only test/google-feed-item-row.test.ts && ts-node --transpile-only test/publication-gate.test.ts && ts-node --transpile-only test/images-regression.test.ts && ts-node --transpile-only test/clean-images.test.ts && ts-node --transpile-only test/image-url-validator.test.ts && ts-node --transpile-only test/openai-image-fields.test.ts && ts-node --transpile-only test/openai-ads-mapping.test.ts && ts-node --transpile-only test/openai-ads-feed-run.test.ts && ts-node --transpile-only test/openai-ads-finance.test.ts && ts-node --transpile-only test/openai-ads-price-cap.test.ts && ts-node --transpile-only test/meinstadt-mapping.test.ts && ts-node --transpile-only test/meinstadt-feed-run.test.ts && ts-node --transpile-only test/autouncle-mapping.test.ts && ts-node --transpile-only test/autouncle-feed-run.test.ts && ts-node --transpile-only test/criteo-images.test.ts"
-  },
-  "dependencies": {
-    "@bytebrand/autode-request": "^1.19.2",
-    "@bytebrand/bigquery-logger": "^2.1.2",
-    "@bytebrand/car-basic-data-scope": "^9.11.24",
-    "@bytebrand/car-description": "^1.17.20",
-    "@bytebrand/car-schema": "^3.14.15",
-    "@bytebrand/chunkable": "^5.19.8",
-    "@bytebrand/cronicle-helper": "^1.21.8",
-    "@bytebrand/fe-financing": "^5.14.19",
-    "@bytebrand/i18n-dictionaries": "^1.5.0",
-    "@bytebrand/import-helper": "^0.45.0",
-    "@bytebrand/schema-tools": "^3.18.8",
-    "@bytebrand/zipper": "^2.2.0",
-    "@nestjs/cache-manager": "^2.1.0",
-    "@nestjs/common": "^10.2.7",
-    "@nestjs/core": "^10.2.7",
-    "adm-zip": "0.5.5",
-    "axios": "^1.5.1",
-    "bottleneck": "^2.19.5",
-    "cache-manager": "^5.2.4",
-    "csv-write-stream": "^2.0.0",
-    "google-spreadsheet": "^3.3.0",
-    "lodash": "^4.17.21",
-    "moment": "^2.29.4",
-    "nest-commander": "~3.15.0",
-    "reflect-metadata": "^0.1.13",
-    "vin-generator": "^1.0.1"
-  },
-  "devDependencies": {
-    "@nestjs/cli": "^10.1.18",
-    "@types/google-spreadsheet": "^3.3.0",
-    "@types/inquirer": "^8.2.4",
-    "@types/lodash": "^4.14.186",
-    "@types/node": "^16.18.58",
-    "@typescript-eslint/eslint-plugin": "^6.8.0",
-    "@typescript-eslint/parser": "^6.8.0",
-    "eslint": "^8.0.1",
-    "eslint-config-prettier": "^8.3.0",
-    "eslint-plugin-prettier": "^4.0.0",
-    "gts": "^3.1.1",
-    "prettier": "^2.3.2",
-    "ts-node": "^10.0.0",
-    "tsconfig-paths": "4.1.0",
-    "typescript": "^5.2.2"
-  },
-  "engines": {
-    "node": ">=14.0.0"
-  },
-  "gitHead": "e827f35865032821bf3a081f03a026d3d3f8a540"
-}
+{
+  "name": "@bytebrand/export-cars",
+  "version": "1.26.40",
+  "description": "",
+  "author": "bytebrand",
+  "license": "UNLICENSED",
+  "scripts": {
+    "prebuild": "rimraf dist",
+    "build": "nest build",
+    "prepublishOnly": "npm run build",
+    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
+    "lint": "eslint -c .eslintrc.js --ext .ts src/",
+    "start:dev": "nest start --watch",
+    "test": "ts-node --transpile-only test/google-feed-envkv.test.ts && ts-node --transpile-only test/google-feed-item-row.test.ts && ts-node --transpile-only test/publication-gate.test.ts && ts-node --transpile-only test/images-regression.test.ts && ts-node --transpile-only test/clean-images.test.ts && ts-node --transpile-only test/image-url-validator.test.ts && ts-node --transpile-only test/openai-image-fields.test.ts && ts-node --transpile-only test/openai-ads-mapping.test.ts && ts-node --transpile-only test/openai-ads-feed-run.test.ts && ts-node --transpile-only test/openai-ads-finance.test.ts && ts-node --transpile-only test/openai-ads-price-cap.test.ts && ts-node --transpile-only test/meinstadt-mapping.test.ts && ts-node --transpile-only test/meinstadt-feed-run.test.ts && ts-node --transpile-only test/autouncle-mapping.test.ts && ts-node --transpile-only test/autouncle-feed-run.test.ts && ts-node --transpile-only test/criteo-images.test.ts && ts-node --transpile-only test/criteo-mapping.test.ts && ts-node --transpile-only test/criteo-feed-run.test.ts"
+  },
+  "dependencies": {
+    "@bytebrand/autode-request": "^1.19.2",
+    "@bytebrand/bigquery-logger": "^2.1.2",
+    "@bytebrand/car-basic-data-scope": "^9.11.24",
+    "@bytebrand/car-description": "^1.17.20",
+    "@bytebrand/car-schema": "^3.14.15",
+    "@bytebrand/chunkable": "^5.19.8",
+    "@bytebrand/cronicle-helper": "^1.21.8",
+    "@bytebrand/fe-financing": "^5.14.19",
+    "@bytebrand/i18n-dictionaries": "^1.5.0",
+    "@bytebrand/import-helper": "^0.45.0",
+    "@bytebrand/schema-tools": "^3.18.8",
+    "@bytebrand/zipper": "^2.2.0",
+    "@nestjs/cache-manager": "^2.1.0",
+    "@nestjs/common": "^10.2.7",
+    "@nestjs/core": "^10.2.7",
+    "adm-zip": "0.5.5",
+    "axios": "^1.5.1",
+    "bottleneck": "^2.19.5",
+    "cache-manager": "^5.2.4",
+    "csv-write-stream": "^2.0.0",
+    "google-spreadsheet": "^3.3.0",
+    "lodash": "^4.17.21",
+    "moment": "^2.29.4",
+    "nest-commander": "~3.15.0",
+    "reflect-metadata": "^0.1.13",
+    "vin-generator": "^1.0.1"
+  },
+  "devDependencies": {
+    "@nestjs/cli": "^10.1.18",
+    "@types/google-spreadsheet": "^3.3.0",
+    "@types/inquirer": "^8.2.4",
+    "@types/lodash": "^4.14.186",
+    "@types/node": "^16.18.58",
+    "@typescript-eslint/eslint-plugin": "^6.8.0",
+    "@typescript-eslint/parser": "^6.8.0",
+    "eslint": "^8.0.1",
+    "eslint-config-prettier": "^8.3.0",
+    "eslint-plugin-prettier": "^4.0.0",
+    "gts": "^3.1.1",
+    "prettier": "^2.3.2",
+    "ts-node": "^10.0.0",
+    "tsconfig-paths": "4.1.0",
+    "typescript": "^5.2.2"
+  },
+  "engines": {
+    "node": ">=14.0.0"
+  },
+  "gitHead": "e827f35865032821bf3a081f03a026d3d3f8a540"
+}

package/src/configs/criteo.config.ts

@@ -1,17 +1,54 @@
-import { ExportType, FileType, ImageType } from 'src/enums';
-import { Params } from '../interfaces/params.interface';
-import { Config } from './config';
-
-export class CriteoConfig extends Config {
-  constructor(params: Params) {
-    super(params);
-    this.campaign = ExportType.CRITEO;
-    this.pathesConfig.localConvertedDirectory = `resources/${ExportType.CRITEO}-export`;
-    this.imageType = ImageType.marketing;
-    this.marketingImageConfigId = this.environmentConfig[params.environment].criteoMarketingImagesConfigId;
-    this.outputFiles = [
-      `${this.pathesConfig.localConvertedDirectory}/api_export_cars_${ExportType.CRITEO}.${FileType.XML}`,
-    ];
-    this.isCriteo = true;
-  }
-}
+import { toNumber } from 'lodash';
+
+import { ExportType, FileType, ImageType } from 'src/enums';
+import { Params } from '../interfaces/params.interface';
+import { FeedGuardOptions } from '../services/feed-guard';
+import { Config } from './config';
+
+const text = (value: unknown): string => (typeof value === 'string' ? value.trim() : '');
+
+const positiveNumber = (value: unknown, fallback: number): number => {
+  const parsed = toNumber(text(value));
+  return text(value) && Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
+};
+
+// Same guard defaults as the meinestadt / OpenAI feeds; a job can override them via the
+// feedGuard* Cronicle params.
+export const CRITEO_DEFAULTS = {
+  minSourceItems: 1000,
+  minPublishedItems: 500,
+  maxDropPercent: 20,
+  maxSourceAgeHours: 26,
+  maxStaleHours: 48,
+};
+
+/**
+ * Criteo catalog (ADV-319): legacy RSS/XML file at the same path as before, which the chained
+ * `4.1 Export criteo - part 2` step uploads. Guard state lives in resources/criteo-state.
+ */
+export class CriteoConfig extends Config {
+  stateDir: string;
+  feedGuard: FeedGuardOptions;
+
+  constructor(params: Params) {
+    super(params);
+    this.campaign = ExportType.CRITEO;
+    this.pathesConfig.localConvertedDirectory = `resources/${ExportType.CRITEO}-export`;
+    this.stateDir = `resources/${ExportType.CRITEO}-state`;
+    this.imageType = ImageType.marketing;
+    this.marketingImageConfigId = this.environmentConfig[params.environment].criteoMarketingImagesConfigId;
+    this.feedGuard = {
+      stateDir: this.stateDir,
+      minSourceItems: positiveNumber(params.feedGuardMinSourceItems, CRITEO_DEFAULTS.minSourceItems),
+      minPublishedItems: positiveNumber(params.feedGuardMinPublishedItems, CRITEO_DEFAULTS.minPublishedItems),
+      maxDropPercent: positiveNumber(params.feedGuardMaxDropPercent, CRITEO_DEFAULTS.maxDropPercent),
+      maxSourceAgeHours: positiveNumber(params.feedGuardMaxSourceAgeHours, CRITEO_DEFAULTS.maxSourceAgeHours),
+      maxStaleHours: positiveNumber(params.feedGuardMaxStaleHours, CRITEO_DEFAULTS.maxStaleHours),
+      acceptDrop: text(params.feedGuardAcceptDrop).toLowerCase() === 'true',
+    };
+    this.outputFiles = [
+      `${this.pathesConfig.localConvertedDirectory}/api_export_cars_${ExportType.CRITEO}.${FileType.XML}`,
+    ];
+    this.isCriteo = true;
+  }
+}

package/src/exporters/criteo.exporter.ts

@@ -1,51 +1,63 @@
-import { Injectable } from '@nestjs/common';
-import { ModuleRef } from '@nestjs/core';
-
-import { CronicleUtils } from '@bytebrand/cronicle-helper';
-import { FsChunkInterface } from '@bytebrand/chunkable';
-
-import { CarService, ConfigService, LoaderService, RulesService } from '../services';
-import { GoogleXmlBuilder } from '../formats/google-xml.builder';
-import { Exporter } from '../interfaces/exporter.interface';
-import { CriteoTransformer } from 'src/transformers/criteo.transformer';
-
-@Injectable()
-export class CriteoExporter implements Exporter {
-	constructor(
-		private carService: CarService,
-		private configService: ConfigService,
-		private rulesService: RulesService,
-		private loaderService: LoaderService,
-		private moduleRef: ModuleRef,
-	) { }
-
-	async export(): Promise<void> {
-		const [fileName] = this.configService.config.outputFiles;
-		await CronicleUtils.ensureDir(this.configService.config.pathesConfig.localConvertedDirectory);
-		await CronicleUtils.emptyDir(this.configService.config.pathesConfig.localConvertedDirectory);
-
-		const googleXmlBuilder = new GoogleXmlBuilder(fileName);
-		await googleXmlBuilder.buildStart(this.configService.config.domain);
-
-		const criteoTransformer = await this.moduleRef.create(CriteoTransformer);
-		let withoutImage = 0;
-
-		await this.loaderService.process(async (chunk: FsChunkInterface) => {
-			chunk.data = chunk.data.filter(car => this.carService.filterCar(car, this.rulesService.rules));
-
-			for (const data of chunk.data) {
-				const car = await criteoTransformer.transform(data);
-				if (car) {
-					await googleXmlBuilder.addItem(car);
-				} else {
-					withoutImage += 1;
-				}
-				this.carService.counter.countCar(data.images);
-			}
-		});
-
-		await googleXmlBuilder.buildEnd();
-		console.log(this.carService.counter.getCountedCars());
-		console.log(`criteo: ${withoutImage} cars skipped without a marketing image`);
-	}
-}
+import { Injectable } from '@nestjs/common';
+import { ModuleRef } from '@nestjs/core';
+
+import { FsChunkInterface } from '@bytebrand/chunkable';
+import { LoggerService } from '@bytebrand/cronicle-helper';
+
+import { CarService, ConfigService, LoaderService, RulesService } from '../services';
+import { FeedGuard } from '../services/feed-guard';
+import { CriteoFeedRun, CriteoSourceCar } from '../services/criteo-feed-run';
+import { Exporter } from '../interfaces/exporter.interface';
+import { CriteoConfig } from '../configs/criteo.config';
+import { CriteoTransformer } from 'src/transformers/criteo.transformer';
+
+/**
+ * ExportType.CRITEO (ADV-319). Wires the Cronicle/Nest services into CriteoFeedRun, which holds the
+ * logic. The output directory is no longer emptied: the new XML replaces the old one only after the
+ * fail-closed guard passed.
+ */
+@Injectable()
+export class CriteoExporter implements Exporter {
+	constructor(
+		private carService: CarService,
+		private configService: ConfigService,
+		private rulesService: RulesService,
+		private loaderService: LoaderService,
+		private moduleRef: ModuleRef,
+	) { }
+
+	async export(): Promise<void> {
+		const config = this.configService.config as CriteoConfig;
+		const [xmlFile] = config.outputFiles;
+		const transformer = await this.moduleRef.create(CriteoTransformer);
+
+		const run = new CriteoFeedRun(
+			{
+				forEachChunk: fn => this.loaderService.process((chunk: FsChunkInterface) => fn(chunk.data as CriteoSourceCar[])),
+				filterCar: source => this.carService.filterCar(source, this.rulesService.rules),
+				transform: source => transformer.transform(source),
+				guard: new FeedGuard(config.feedGuard),
+			},
+			{
+				runId: config.runId,
+				domain: config.domain,
+				sourceDirectory: config.pathesConfig.localDirectory,
+				sourceBaseFileName: config.pathesConfig.baseFileName,
+				xmlFile,
+				stateDir: config.stateDir,
+			},
+		);
+
+		try {
+			const report = await run.run();
+			if (report.stale) {
+				console.log(`criteo feed STALE: last good file ${report.lastGoodAgeHours?.toFixed(1)} h old`);
+			}
+		} catch (error) {
+			// main.ts would report a rejected run as {"complete":1,"code":0}; fail the job so the chained
+			// upload step does not run (report + lock already handled by the run).
+			LoggerService.exitError(`criteo feed not published: ${error?.message ?? error}`);
+			process.exit(1);
+		}
+	}
+}

package/src/interfaces/export-item.interface.ts

@@ -152,10 +152,10 @@ export interface CriteoXmlItem {
   brand: any;
   price: string;
   sale_price: string;
-  energy_efficiency_class: string;
+  energy_efficiency_class?: string;
   color: string;
-  installment: { paybackPeriod: any; monthlyInstallment: any };
-  custom_label_0: string;
+  installment?: { paybackPeriod: number; monthlyInstallment: number };
+  custom_label_0?: string;
 }
 
 export interface GoogleXmlItem {