@bytebrand/export-cars 1.26.45 → 1.26.47

package/docs/marktde/AUDIT.md

@@ -13,7 +13,7 @@ letters are code evidence, not a contract.
 | Row format | `MobileTransformer` + Google sheet `147qz9sT…` tab `mobile_yescar` (165 rules, ACF-4370 reused the YesCar / mobile.de CSV). `;` separated, every cell quoted, no header row, UTF-8, 517–528 columns per row |
 | Files | `api_export_cars_marktde.csv` + `.zip` (was `resources/marktde-export/`) |
 | DEV Cronicle | `9c.0 Export markt.de - part 1` `ekpy4pqg21u` (rule `metaData.ownerId:xo1ag9JooQebXhnfSclw3NhZyG63`) → `9c.1 … part 2` `ekyvib2rm06` (plain `ftp -pinv 185.15.211.90`, uploads `develop_marktde.zip/.csv`, credentials in the script) → `9d.0 facebook shop`. Triggered by the chain after `9b.1 Export Focus`. No run in the job history |
-| DEV runtime defects found | (a) the Google Sheets API key in `9c.0` is **expired** — the mapping sheet cannot load, every row-building run fails; (b) the filtered dealer has 0 cars in the DEV snapshot; (c) the host has no `zip` binary — the legacy run wrote the CSV, failed the ZIP and still reported `code 0`; (d) `9c.1` uploads names the exporter never writes; (e) 185.15.211.90 is the legacy Jelastic FTP (port 21 closed, see ADV-319 4.1) |
+| DEV runtime defects found | (a) the Google Sheets API key in `9c.0` is **expired** — the mapping sheet could not load (fixed in 1.26.47: public CSV, no key); (b) the filtered dealer has 0 cars in the DEV snapshot; (c) the host has no `zip` binary — the legacy run wrote the CSV, failed the ZIP and still reported `code 0`; (d) `9c.1` uploads names the exporter never writes; (e) 185.15.211.90 is the legacy Jelastic FTP (port 21 closed, see ADV-319 4.1) |
 | PROD | not checked (DEV-only task). Old contract / account belong to the former auto.de Media GmbH — not transferable (Daniel, 16.10) |
 
 ## 2. Source → target matrix (columns the feed relies on)

package/docs/marktde/RUNBOOK.md

@@ -4,7 +4,7 @@
 
 | Param | Default | Meaning |
 |---|---|---|
-| `apikey` | — | Google Sheets API key for the mapping sheet `mobile_yescar`. **DEV 9c.0: expired (12.11.2026)** |
+| `apikey` | — | no longer used for the mapping (since 1.26.47 the tab `mobile_yescar` is read from the sheet's public CSV, no key); the sheet must stay readable by everyone with the link. The DEV 9c.0 key is expired (12.11.2026) |
 | `String` / `Number` / `Date` | — | Cronicle filter rules (DEV 9c.0: one dealer, 0 cars today) |
 | `marktdeDryRun` | dry-run | only the exact text `false` makes a live run |
 | `marktdeMaxImages` | no cut | cut the image list (no markt.de limit known) |
@@ -42,7 +42,7 @@ the API with `chain: ""` (see below) or from the UI with the chain removed for t
 D=/home/deploy/export-cars-<ver>-dryrun/<name>   # job.json (chmod 600) + credentials + resources/cars-export symlinks
 NODE_PATH=<pkg>/dist:/opt/cronicle/node_modules node <pkg>/dist/main.js < job.json > run.log
 ```
-While the event's Sheets key is expired: `node harness.js <pkg> < job.json` — the harness runs the same
+Before 1.26.47 (Sheets API key expired): `node harness.js <pkg> < job.json` — the harness runs the same
 `main.js` but loads the mapping rules from the public CSV of the same sheet tab (gviz) with the same
 post-processing as `MappingService`.
 
@@ -50,7 +50,7 @@ post-processing as `MappingService`.
 
 1. Signed contract / account for auto.de GmbH, current markt.de feed spec + test importer
    (`PARTNER-QUESTIONS.md`); written acceptance of a test import of the 20-row sample.
-2. New Sheets API key in the 9c.0 params (or move the mapping out of Google Sheets).
+2. Keep the mapping sheet shared with everyone who has the link (1.26.47 reads its public CSV; a private sheet fails the run).
 3. Dealer scope (the rule in 9c.0) and `feedGuardMinPublishedItems` from a PROD dry-run.
 4. Replace `9c.1`: transport and file names from the spec, credentials out of the script; never the
    legacy Jelastic host. Only then `marktdeDryRun=false`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bytebrand/export-cars",
-  "version": "1.26.45",
+  "version": "1.26.47",
   "description": "",
   "author": "bytebrand",
   "license": "UNLICENSED",
@@ -11,7 +11,7 @@
     "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
     "lint": "eslint -c .eslintrc.js --ext .ts src/",
     "start:dev": "nest start --watch",
-    "test": "ts-node --transpile-only test/google-feed-envkv.test.ts && ts-node --transpile-only test/google-feed-item-row.test.ts && ts-node --transpile-only test/publication-gate.test.ts && ts-node --transpile-only test/images-regression.test.ts && ts-node --transpile-only test/clean-images.test.ts && ts-node --transpile-only test/image-url-validator.test.ts && ts-node --transpile-only test/openai-image-fields.test.ts && ts-node --transpile-only test/openai-ads-mapping.test.ts && ts-node --transpile-only test/openai-ads-feed-run.test.ts && ts-node --transpile-only test/openai-ads-finance.test.ts && ts-node --transpile-only test/openai-ads-price-cap.test.ts && ts-node --transpile-only test/meinstadt-mapping.test.ts && ts-node --transpile-only test/meinstadt-feed-run.test.ts && ts-node --transpile-only test/autouncle-mapping.test.ts && ts-node --transpile-only test/autouncle-feed-run.test.ts && ts-node --transpile-only test/marktde-mapping.test.ts && ts-node --transpile-only test/marktde-feed-run.test.ts && ts-node --transpile-only test/criteo-images.test.ts && ts-node --transpile-only test/criteo-mapping.test.ts && ts-node --transpile-only test/criteo-feed-run.test.ts"
+    "test": "ts-node --transpile-only test/google-feed-envkv.test.ts && ts-node --transpile-only test/google-feed-item-row.test.ts && ts-node --transpile-only test/publication-gate.test.ts && ts-node --transpile-only test/images-regression.test.ts && ts-node --transpile-only test/clean-images.test.ts && ts-node --transpile-only test/image-url-validator.test.ts && ts-node --transpile-only test/openai-image-fields.test.ts && ts-node --transpile-only test/openai-ads-mapping.test.ts && ts-node --transpile-only test/openai-ads-feed-run.test.ts && ts-node --transpile-only test/openai-ads-finance.test.ts && ts-node --transpile-only test/openai-ads-price-cap.test.ts && ts-node --transpile-only test/meinstadt-mapping.test.ts && ts-node --transpile-only test/meinstadt-feed-run.test.ts && ts-node --transpile-only test/autouncle-mapping.test.ts && ts-node --transpile-only test/autouncle-feed-run.test.ts && ts-node --transpile-only test/mapping-csv.test.ts && ts-node --transpile-only test/marktde-mapping.test.ts && ts-node --transpile-only test/marktde-feed-run.test.ts && ts-node --transpile-only test/criteo-images.test.ts && ts-node --transpile-only test/criteo-mapping.test.ts && ts-node --transpile-only test/criteo-feed-run.test.ts"
   },
   "dependencies": {
     "@bytebrand/autode-request": "^1.19.2",

package/src/constants/fb.constants.ts

@@ -28,7 +28,7 @@ export const TERM_QUALIFIER_MONTHS = 'months';
 
 export const OFFER_DISCLAIMER = 'Für Ihre Finanzierungswünsche stellen wir eine Bonitätsanfrage. Bonität ' +
   'vorausgesetzt, ist dies ein repräsentatives Berechnungsbeispiel gem. der Angaben, welches 2/3 aller Kunden, ' +
-  'im Sinne des § 17a Abs. 4 PAngV, erhalten. Dieses freibleibende Angebot der Santander Consumer Bank AG, ' +
+  'im Sinne des § 17a Abs. 4 PAngV, erhalten. Dieses freibleibende Angebot der Openbank Deutschland AG, ' +
   'Santander-Platz 1, 41061 Mönchengladbach wird vermittelt durch die auto.de GmbH, Max-Planck-Str. 19, 06796 ' +
   'Sandersdorf-Brehna, die als ungebundener Vermittler nicht beratend tätig ist. Irrtümer vorbehalten. Preise ' +
   'ggf. inkl. MwSt.';

package/src/services/mapping.service.ts

@@ -1,11 +1,22 @@
 import { Injectable } from '@nestjs/common';
 import axios from 'axios';
+import { CronicleUtils } from '@bytebrand/cronicle-helper';
 import { MappingRules } from '../interfaces/mapping-rules.interface';
 import { GoogleSpreadsheet, GoogleSpreadsheetRow } from 'google-spreadsheet';
 import { ConfigService } from './config.service';
 import { DealersMapping, MetaPushPriorityMapping } from '../interfaces/dealers-mapping.interface';
 import { parseCsvToObjects } from '../utils';
 
+// Public CSV of one tab of a Google sheet, addressed by the tab title — no API key. The sheet must stay
+// readable by everyone with the link. (Checked 13.11.2026: gviz CSV = the raw export of the tabs
+// mobile_yescar and mobile, cell for cell; gviz only adds trailing empty columns.)
+export const mappingSheetCsvUrl = (spreadsheetId: string, title: string): string =>
+  `https://docs.google.com/spreadsheets/d/${spreadsheetId}/gviz/tq?tqx=out:csv&sheet=${encodeURIComponent(title)}`;
+
+// Columns every mapping tab read by getRules() has; missing ones mean the response is not the tab
+// (e.g. Google's sign-in page once the sheet is no longer shared).
+const REQUIRED_MAPPING_COLUMNS = ['importField', 'importFieldType', 'autodeField', 'autodeFieldType'];
+
 @Injectable()
 export class MappingService {
   constructor(private configService: ConfigService) {}
@@ -13,24 +24,54 @@ export class MappingService {
   private dealers: any;
   private metaPushPriorities: MetaPushPriorityMapping[];
 
+  /**
+   * Mapping rules of the export type's tab (mappingSpreadsheetTitle) of the mapping sheet. Read from the
+   * sheet's public CSV instead of the Google Sheets API: the API needed the event's `apikey`, and an
+   * expired key (DEV 9c.0, found 12.11.2026) silently broke every feed on the sheet. Fails the run
+   * when the tab cannot be read or yields no rule — never an export without a mapping.
+   */
   async getRules(): Promise<MappingRules[]> {
     if (!this.rules) {
-      const spreadsheet = new GoogleSpreadsheet(this.configService.config.mappingSpreadsheetId);
-      spreadsheet.useApiKey(this.configService.config.apikey);
-      await spreadsheet.loadInfo();
-      const rows = await spreadsheet.sheetsByTitle[this.configService.config.mappingSpreadsheetTitle].getRows();
-      const mappingRules = this.rowsToRules(rows) as MappingRules[];
+      const { mappingSpreadsheetId, mappingSpreadsheetTitle } = this.configService.config;
+      const url = mappingSheetCsvUrl(mappingSpreadsheetId, mappingSpreadsheetTitle);
+      const { data } = await CronicleUtils.retry(
+        () => axios.get<string>(url, { responseType: 'text', timeout: 30_000 }),
+        3,
+        5_000,
+        true,
+      );
+      this.rules = this.rulesFromCsv(data, mappingSpreadsheetTitle);
+      console.log(`mapping: ${this.rules.length} rules from the public CSV of tab ${mappingSpreadsheetTitle}`);
+    }
+    return this.rules;
+  }
 
-      mappingRules.map((rule, _i, selfRules) => {
-        this.removeEmptyKeys(rule);
-        if (rule.importFieldType && rule.importFieldType.includes('selector')) {
-          this.groupSelectors(rule, selfRules);
-        }
-      });
+  // CSV text of a mapping tab -> rules, with the same post-processing the Sheets API rows had.
+  rulesFromCsv(csv: string, title: string): MappingRules[] {
+    const rows = parseCsvToObjects<Record<string, string>>(typeof csv === 'string' ? csv : '');
+    const columns = rows.length ? Object.keys(rows[0]) : [];
+    const missing = REQUIRED_MAPPING_COLUMNS.filter(column => !columns.includes(column));
+    if (missing.length) {
+      throw new Error(`mapping sheet tab "${title}" not readable as CSV (missing columns ${missing.join(', ')}) — is the sheet still shared with everyone who has the link?`);
+    }
+    // unnamed columns (notes next to the table) are not part of a rule
+    const mappingRules = rows.map((row) => {
+      delete row[''];
+      return row;
+    }) as unknown as MappingRules[];
+
+    mappingRules.forEach((rule, _i, selfRules) => {
+      this.removeEmptyKeys(rule);
+      if (rule.importFieldType && rule.importFieldType.includes('selector')) {
+        this.groupSelectors(rule, selfRules);
+      }
+    });
 
-      this.rules = this.filterResultedRows(mappingRules);
+    const rules = this.filterResultedRows(mappingRules);
+    if (!rules.length) {
+      throw new Error(`mapping sheet tab "${title}" has no rule`);
     }
-    return this.rules;
+    return rules;
   }
 
   async getDealersMapping(): Promise<DealersMapping[]> {

package/test/mapping-csv.test.ts

@@ -0,0 +1,65 @@
+// Run: npm test (ts-node, no test framework needed)
+// Mapping rules from the public CSV of the mapping sheet tab (no Sheets API key) — pure, no I/O.
+import * as assert from 'node:assert/strict';
+
+import { MappingService, mappingSheetCsvUrl } from '../src/services/mapping.service';
+
+const service = new MappingService({ config: {} } as any);
+
+// Excerpt of the tab mobile_yescar as gviz returns it (13.11.2026): quoted cells, an unnamed column,
+// trailing empty columns, selector rows that group onto the first row of their importField.
+const CSV = [
+  '"importField","importFieldType","autodeField","autodeFieldType","importFieldSelector","autodeFieldSelector","","custom","",""',
+  '"[\'0\']","string","persistentId","string","","","","yes","",""',
+  '"[\'2\']","categoryOrClass_selector","mainData.category","selector","cabrio/roadster","selector_category_convertible","note","","",""',
+  '"[\'2\']","categoryOrClass_selector","mainData.category","selector","Kombi, Van","selector_category_combi","","","",""',
+  '"[\'3\']","string","mainData.make","string","","","","","",""',
+  '"[\'110\']","selector","driveSuspension.gearbox","selector","1","selector_gearbox_manualShift","","","",""',
+  '"[\'110\']","selector","driveSuspension.gearbox","selector","3","selector_gearbox_automatic","","","",""',
+  '"[\'999\']","string","","string","","","","","",""',
+  '',
+].join('\n');
+
+const tests: Array<[string, () => void]> = [
+  ['url: gviz CSV of the tab by title, title URL-encoded, no key', () => {
+    assert.equal(
+      mappingSheetCsvUrl('147qz9sTgs1wi13ua8YKa0IgfxAn1X_qH6LkGnAnzeyY', 'mobile_yescar'),
+      'https://docs.google.com/spreadsheets/d/147qz9sTgs1wi13ua8YKa0IgfxAn1X_qH6LkGnAnzeyY/gviz/tq?tqx=out:csv&sheet=mobile_yescar',
+    );
+    assert.ok(mappingSheetCsvUrl('x', 'a b&c').endsWith('sheet=a%20b%26c'));
+  }],
+
+  ['rules: same post-processing as the Sheets API rows (empty keys dropped, selectors grouped, rows without autodeField out)', () => {
+    const rules = service.rulesFromCsv(CSV, 'mobile_yescar') as any[];
+    assert.deepEqual(rules.map(r => r.importField), ["['0']", "['2']", "['3']", "['110']"]);
+    assert.deepEqual(rules[0], { importField: "['0']", importFieldType: 'string', autodeField: 'persistentId', autodeFieldType: 'string', custom: 'yes' });
+    assert.deepEqual(rules[1].selectors, ['cabrio/roadster:selector_category_convertible', 'Kombi, Van:selector_category_combi']);
+    assert.equal(rules[1].importFieldSelector, undefined);
+    assert.equal(rules[1][''], undefined, 'unnamed column is not part of the rule');
+    assert.deepEqual(rules[3].selectors, ['1:selector_gearbox_manualShift', '3:selector_gearbox_automatic']);
+    assert.deepEqual(Object.keys(rules[2]).sort(), ['autodeField', 'autodeFieldType', 'importField', 'importFieldType']);
+  }],
+
+  ['not the tab (sign-in page / other table) -> error, never an empty mapping', () => {
+    assert.throws(() => service.rulesFromCsv('<!DOCTYPE html><html><body>Sign in</body></html>', 'mobile_yescar'), /not readable as CSV.*shared/);
+    assert.throws(() => service.rulesFromCsv('', 'mobile_yescar'), /not readable as CSV/);
+    assert.throws(() => service.rulesFromCsv('"a","b"\n"1","2"\n', 'mobile_yescar'), /missing columns importField/);
+  }],
+
+  ['a tab with the columns but no usable rule -> error', () => {
+    const header = '"importField","importFieldType","autodeField","autodeFieldType"';
+    assert.throws(() => service.rulesFromCsv(`${header}\n"[\'1\']","string","",""\n`, 'mobile'), /has no rule/);
+  }],
+];
+
+let failed = 0;
+for (const [name, fn] of tests) {
+  try {
+    fn();
+    console.log(`ok   ${name}`);
+  } catch (e) {
+    failed += 1;
+    console.error(`FAIL ${name}\n     ${e.message}`);
+  }
+}
+if (failed) process.exit(1);