@bytebrand/export-cars 1.26.40 → 1.26.42
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/configs/autouncle.config.js +22 -0
- package/dist/configs/autouncle.config.js.map +1 -1
- package/dist/exporters/autouncle.exporter.js +10 -0
- package/dist/exporters/autouncle.exporter.js.map +1 -1
- package/dist/services/autouncle-feed-run.d.ts +6 -0
- package/dist/services/autouncle-feed-run.js +11 -1
- package/dist/services/autouncle-feed-run.js.map +1 -1
- package/dist/transformers/autouncle.transformer.d.ts +4 -0
- package/dist/transformers/autouncle.transformer.js +10 -5
- package/dist/transformers/autouncle.transformer.js.map +1 -1
- package/dist/transformers/google-feed-envkv.d.ts +2 -0
- package/dist/transformers/google-feed-envkv.js +9 -1
- package/dist/transformers/google-feed-envkv.js.map +1 -1
- package/dist/tsconfig.build.tsbuildinfo +1 -1
- package/docs/autouncle/AUDIT.md +103 -97
- package/docs/autouncle/PARTNER-QUESTIONS.md +3 -1
- package/docs/autouncle/RUNBOOK.md +2 -0
- package/docs/criteo/RUNBOOK.md +86 -86
- package/package.json +1 -1
- package/src/configs/autouncle.config.ts +26 -1
- package/src/configs/criteo.config.ts +54 -54
- package/src/exporters/autouncle.exporter.ts +12 -0
- package/src/exporters/criteo.exporter.ts +63 -63
- package/src/services/autouncle-feed-run.ts +19 -2
- package/src/services/criteo-feed-run.ts +253 -253
- package/src/transformers/autouncle.transformer.ts +20 -4
- package/src/transformers/google-feed-envkv.ts +10 -0
- package/test/autouncle-feed-run.test.ts +25 -2
- package/test/autouncle-mapping.test.ts +41 -2
- package/test/criteo-feed-run.test.ts +365 -365
- package/test/criteo-mapping.test.ts +122 -122
- package/test/fixtures/criteo-20-cases.json +581 -581
package/docs/autouncle/AUDIT.md
CHANGED
|
@@ -1,97 +1,103 @@
|
|
|
1
|
-
# AutoUncle inventory feed — audit (ADV-322 / ADV-323)
|
|
2
|
-
|
|
3
|
-
Status 13.10.2026, base `origin/deploy_develop` 3a025095, `@bytebrand/export-cars` 1.26.31.
|
|
4
|
-
Historical documents (Confluence "autouncle [paused]", "2.7.2 Export Plugins", ACF-3604/3650/3717/3725/6172/6190)
|
|
5
|
-
are evidence only. No current AutoUncle inventory schema is available; the public AutoUncle B2B
|
|
6
|
-
"automotive API" is a dealer/DMS **valuation** API and is not used as a schema here.
|
|
7
|
-
|
|
8
|
-
## 1. Is the export live? — No
|
|
9
|
-
|
|
10
|
-
| Where | Event | State | Last run | Output |
|
|
11
|
-
|---|---|---|---|---|
|
|
12
|
-
| PROD Cronicle (`auto-de-prod-cronicle-01`, export-cars 1.26.18) | `3.0 Export to autouncle - part 1` `ek61rkico08` | **disabled**, no timing, nothing chains into it | one manual run 07.07.2026, aborted by user | `resources/autouncle-export/api_export_cars_autouncle.json` = 8 bytes (`{"ads":[`) |
|
|
13
|
-
| | `3.1 Export to autouncle - part 2` `ek61ro0yv09` | **disabled**; shell `cp` of the JSON to `htdocs/autouncle/export_cars_autouncle.json` | never ran | `htdocs/autouncle/` does not exist |
|
|
14
|
-
| DEV Cronicle (`auto-de-dev-cronicle-01`, 1.26.31) | `3.0 Export to Autouncle part1` `ek60tym7m0b` | enabled, no timing; chained from `2.1 Export facebook - part 2` | 10.09.2026, code 0 | 467 KB JSON of 10.09 (298 ads) |
|
|
15
|
-
| | `3.1 Export to Autouncle part2` `ek60u1gsd0c` | enabled, chains to `ek87441xa04` | 10.09.2026, **code 1** every run (`cp: … htdocs/autouncle/… No such file or directory`) | — the rest of the DEV chain (criteo, meinestadt, ooyyo, …) stops here |
|
|
16
|
-
|
|
17
|
-
PROD 3.0 params: `environment=production`, `String` = allow-list of 19 dealer `metaData.ownerId`s, `Number` =
|
|
18
|
-
`metaData.imagesCount:2`. DEV 3.0: no rules. Both events were last modified 2023 (DEV) / 01.04.2026 (PROD,
|
|
19
|
-
migration). Events, keys and scripts were read with `storage-cli list_get`, secrets redacted.
|
|
20
|
-
|
|
21
|
-
Hosting: `https://scheduler.auto.de/autouncle/export_cars_autouncle.json` (historic) — no answer (timeout);
|
|
22
|
-
`https://cronicle.auto.de/autouncle/…` and `http://cronicle.autode-dev.de/autouncle/…` — 404. Nothing is
|
|
23
|
-
hosted for AutoUncle today, on any environment.
|
|
24
|
-
|
|
25
|
-
## 2. Code path
|
|
26
|
-
|
|
27
|
-
`ExportType.AUTOUNCLE = 'autouncle'` → `AutouncleConfig` + `AutouncleExporter` (`export.schema.ts`).
|
|
28
|
-
|
|
29
|
-
Before (unchanged since the monorepo import 3046a15d, CO2 class ADV-187 b46e7e83):
|
|
30
|
-
`emptyDir(resources/autouncle-export)` → for every car of the base export snapshot: shared `filterCar`
|
|
31
|
-
(imagesCount > 0 + Cronicle rules) → UPA dealer profile (skip if none) → `AutouncleTransformer` →
|
|
32
|
-
`AutouncleJsonBuilder` appends `{"ads":[…]}` directly to the published file.
|
|
33
|
-
|
|
34
|
-
After (this branch): `AutouncleFeedRun` (`services/autouncle-feed-run.ts`) — lock → source snapshot check
|
|
35
|
-
(shared ADV-335 `FeedGuard`) → per car: duplicate id, `filterCar`, car-level exclusions, dealer address (UPA,
|
|
36
|
-
cached per dealer), clean images (ADV-336 `getCleanImages` + `ImageUrlValidator` + `selectOpenAiImageFields`)
|
|
37
|
-
→ pure `mapAutouncleItem` → temp file → count check → atomic rename (`.prev` kept) → report.
|
|
38
|
-
Dry-run is the default and writes `resources/autouncle-dryrun/`, which no Cronicle step reads.
|
|
39
|
-
|
|
40
|
-
## 3. Field mapping (`ads[]`, keys/order/spellings unchanged)
|
|
41
|
-
|
|
42
|
-
| Key | Source | Before | Now | Open |
|
|
43
|
-
|---|---|---|---|---|
|
|
44
|
-
| back_link | CBD `_id` | `https://www.<domain>/search/vehicle/<_id>?source=classifieds&medium=cpc&campaign=autouncle` |
|
|
45
|
-
| make / model_name / model_variant | mainData.make / model / subModel | raw | same | `SONSTIGE` placeholder models are exported (data) → Q9 |
|
|
46
|
-
| fuel / body / gear_type | consumption.fuel / mainData.category / driveSuspension.gearbox | fixed maps; an unmapped or null selector made the key **disappear** from JSON | unmapped/null → `Other` / `other` | enum values → Q6 |
|
|
47
|
-
| doors | mainData.doors | string | same | |
|
|
48
|
-
| price | price.currentSalesPrice | `EUR <value>`; missing price → `"EUR "` and the car was still sent | same format; no valid positive price → car not sent (`no_valid_price`). Never a rate, never purchasePrice | currency format → Q6 |
|
|
49
|
-
| year | mainData.firstRegistration (Unix s) | null → `moment(0)` → **1970** (143 of 1044 DEV cars) | missing/0/null/invalid → `""`; outside 1900..next year → `""` | null vs "" → Q6 |
|
|
50
|
-
| status | same | `Neuwagen` only when firstRegistration === 0 | `Neuwagen` when no valid firstRegistration (same rule, null included) | 3 of 143 DEV cars are `condition=used` without registration → Q7 |
|
|
51
|
-
| mileage | mainData.mileage | `<n>km`, 0 → `1km` | same | |
|
|
52
|
-
| leasing / has_4wd | — | always `""` | same | → Q10 |
|
|
53
|
-
| color | exterior.exteriorColor, German label | same | same | |
|
|
54
|
-
| engine_size | engineData.cubicCapacity | `ccm` with no number when missing (189/1044) | `""` when missing | |
|
|
55
|
-
| power | engineData.powerKW | kW, no unit | same | unit → Q6 |
|
|
56
|
-
| seller_type / seller_name | — | `Dealer` / `auto.de` | same | marketplace vs dealer identity → Q3 |
|
|
57
|
-
| seller_id | metaData.ownerId | **dealer** id already (not a vehicle id) | same; contract test with two cars of one dealer | |
|
|
58
|
-
| seller_region / city / street_address / postcode | UPA businessAddress, else mainAddress; region from the postal code table | if the table did not know the place, street/city/zip were **blanked** (664/1044 DEV cars) | dealer's own values kept, region `Deutschland` | |
|
|
59
|
-
| description | offer.offerDescription | raw | same; 192/1002 empty on DEV, some contain literal `\\` line breaks (dealer data) | → Q10 |
|
|
60
|
-
| images | Imbo images | default profile: `/large` (format negotiated, can be WebP), padded with **3 noimage placeholders** below 17 photos (2820 placeholder URLs on DEV) | `[{url}]`, ADV-336 clean profile, explicit `large-cached.jpeg`, every URL checked (200 + image/jpeg|png); no valid 3/4 hero → car not sent | limits → ADV-323 gate |
|
|
61
|
-
| fuel_economy / co2_emission | consumption.consumptionCombined / environmentEmissions.co2 | raw |
|
|
62
|
-
| co2_effiency_class
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
1
|
+
# AutoUncle inventory feed — audit (ADV-322 / ADV-323)
|
|
2
|
+
|
|
3
|
+
Status 13.10.2026, base `origin/deploy_develop` 3a025095, `@bytebrand/export-cars` 1.26.31.
|
|
4
|
+
Historical documents (Confluence "autouncle [paused]", "2.7.2 Export Plugins", ACF-3604/3650/3717/3725/6172/6190)
|
|
5
|
+
are evidence only. No current AutoUncle inventory schema is available; the public AutoUncle B2B
|
|
6
|
+
"automotive API" is a dealer/DMS **valuation** API and is not used as a schema here.
|
|
7
|
+
|
|
8
|
+
## 1. Is the export live? — No
|
|
9
|
+
|
|
10
|
+
| Where | Event | State | Last run | Output |
|
|
11
|
+
|---|---|---|---|---|
|
|
12
|
+
| PROD Cronicle (`auto-de-prod-cronicle-01`, export-cars 1.26.18) | `3.0 Export to autouncle - part 1` `ek61rkico08` | **disabled**, no timing, nothing chains into it | one manual run 07.07.2026, aborted by user | `resources/autouncle-export/api_export_cars_autouncle.json` = 8 bytes (`{"ads":[`) |
|
|
13
|
+
| | `3.1 Export to autouncle - part 2` `ek61ro0yv09` | **disabled**; shell `cp` of the JSON to `htdocs/autouncle/export_cars_autouncle.json` | never ran | `htdocs/autouncle/` does not exist |
|
|
14
|
+
| DEV Cronicle (`auto-de-dev-cronicle-01`, 1.26.31) | `3.0 Export to Autouncle part1` `ek60tym7m0b` | enabled, no timing; chained from `2.1 Export facebook - part 2` | 10.09.2026, code 0 | 467 KB JSON of 10.09 (298 ads) |
|
|
15
|
+
| | `3.1 Export to Autouncle part2` `ek60u1gsd0c` | enabled, chains to `ek87441xa04` | 10.09.2026, **code 1** every run (`cp: … htdocs/autouncle/… No such file or directory`) | — the rest of the DEV chain (criteo, meinestadt, ooyyo, …) stops here |
|
|
16
|
+
|
|
17
|
+
PROD 3.0 params: `environment=production`, `String` = allow-list of 19 dealer `metaData.ownerId`s, `Number` =
|
|
18
|
+
`metaData.imagesCount:2`. DEV 3.0: no rules. Both events were last modified 2023 (DEV) / 01.04.2026 (PROD,
|
|
19
|
+
migration). Events, keys and scripts were read with `storage-cli list_get`, secrets redacted.
|
|
20
|
+
|
|
21
|
+
Hosting: `https://scheduler.auto.de/autouncle/export_cars_autouncle.json` (historic) — no answer (timeout);
|
|
22
|
+
`https://cronicle.auto.de/autouncle/…` and `http://cronicle.autode-dev.de/autouncle/…` — 404. Nothing is
|
|
23
|
+
hosted for AutoUncle today, on any environment.
|
|
24
|
+
|
|
25
|
+
## 2. Code path
|
|
26
|
+
|
|
27
|
+
`ExportType.AUTOUNCLE = 'autouncle'` → `AutouncleConfig` + `AutouncleExporter` (`export.schema.ts`).
|
|
28
|
+
|
|
29
|
+
Before (unchanged since the monorepo import 3046a15d, CO2 class ADV-187 b46e7e83):
|
|
30
|
+
`emptyDir(resources/autouncle-export)` → for every car of the base export snapshot: shared `filterCar`
|
|
31
|
+
(imagesCount > 0 + Cronicle rules) → UPA dealer profile (skip if none) → `AutouncleTransformer` →
|
|
32
|
+
`AutouncleJsonBuilder` appends `{"ads":[…]}` directly to the published file.
|
|
33
|
+
|
|
34
|
+
After (this branch): `AutouncleFeedRun` (`services/autouncle-feed-run.ts`) — lock → source snapshot check
|
|
35
|
+
(shared ADV-335 `FeedGuard`) → per car: duplicate id, `filterCar`, car-level exclusions, dealer address (UPA,
|
|
36
|
+
cached per dealer), clean images (ADV-336 `getCleanImages` + `ImageUrlValidator` + `selectOpenAiImageFields`)
|
|
37
|
+
→ pure `mapAutouncleItem` → temp file → count check → atomic rename (`.prev` kept) → report.
|
|
38
|
+
Dry-run is the default and writes `resources/autouncle-dryrun/`, which no Cronicle step reads.
|
|
39
|
+
|
|
40
|
+
## 3. Field mapping (`ads[]`, keys/order/spellings unchanged)
|
|
41
|
+
|
|
42
|
+
| Key | Source | Before | Now | Open |
|
|
43
|
+
|---|---|---|---|---|
|
|
44
|
+
| back_link | CBD `_id` | `https://www.<domain>/search/vehicle/<_id>?source=classifieds&medium=cpc&campaign=autouncle` | stable across runs; since 1.26.41 (ADV-324) `?source=autouncle&medium=listing&campaign=listing&utm_source=autouncle&utm_medium=listing&utm_campaign=listing` | pending AutoUncle approval → Q8 |
|
|
45
|
+
| make / model_name / model_variant | mainData.make / model / subModel | raw | same | `SONSTIGE` placeholder models are exported (data) → Q9 |
|
|
46
|
+
| fuel / body / gear_type | consumption.fuel / mainData.category / driveSuspension.gearbox | fixed maps; an unmapped or null selector made the key **disappear** from JSON | unmapped/null → `Other` / `other` | enum values → Q6 |
|
|
47
|
+
| doors | mainData.doors | string | same | |
|
|
48
|
+
| price | price.currentSalesPrice | `EUR <value>`; missing price → `"EUR "` and the car was still sent | same format; no valid positive price → car not sent (`no_valid_price`). Never a rate, never purchasePrice | currency format → Q6 |
|
|
49
|
+
| year | mainData.firstRegistration (Unix s) | null → `moment(0)` → **1970** (143 of 1044 DEV cars) | missing/0/null/invalid → `""`; outside 1900..next year → `""` | null vs "" → Q6 |
|
|
50
|
+
| status | same | `Neuwagen` only when firstRegistration === 0 | `Neuwagen` when no valid firstRegistration (same rule, null included) | 3 of 143 DEV cars are `condition=used` without registration → Q7 |
|
|
51
|
+
| mileage | mainData.mileage | `<n>km`, 0 → `1km` | same | |
|
|
52
|
+
| leasing / has_4wd | — | always `""` | same | → Q10 |
|
|
53
|
+
| color | exterior.exteriorColor, German label | same | same | |
|
|
54
|
+
| engine_size | engineData.cubicCapacity | `ccm` with no number when missing (189/1044) | `""` when missing | |
|
|
55
|
+
| power | engineData.powerKW | kW, no unit | same | unit → Q6 |
|
|
56
|
+
| seller_type / seller_name | — | `Dealer` / `auto.de` | same | marketplace vs dealer identity → Q3 |
|
|
57
|
+
| seller_id | metaData.ownerId | **dealer** id already (not a vehicle id) | same; contract test with two cars of one dealer | |
|
|
58
|
+
| seller_region / city / street_address / postcode | UPA businessAddress, else mainAddress; region from the postal code table | if the table did not know the place, street/city/zip were **blanked** (664/1044 DEV cars) | dealer's own values kept, region `Deutschland` | |
|
|
59
|
+
| description | offer.offerDescription | raw | same; 192/1002 empty on DEV, some contain literal `\\` line breaks (dealer data) | → Q10 |
|
|
60
|
+
| images | Imbo images | default profile: `/large` (format negotiated, can be WebP), padded with **3 noimage placeholders** below 17 photos (2820 placeholder URLs on DEV) | `[{url}]`, ADV-336 clean profile, explicit `large-cached.jpeg`, every URL checked (200 + image/jpeg|png); no valid 3/4 hero → car not sent | limits → ADV-323 gate |
|
|
61
|
+
| fuel_economy / co2_emission | consumption.consumptionCombined / environmentEmissions.co2 | raw legacy fields; a BEV sent its kWh (CBD mirrors it into consumptionCombined) as fuel economy | since 1.26.41: the Google feed's EnVKV picks as plain numbers (WLTP first, legacy only as fallback; confirmed plug-in weighted); BEV: fuel `""`, CO₂ `0`; estimated values withheld by the publication gate (enforce) | units → Q6 |
|
|
62
|
+
| co2_effiency_class | environmentEmissions.wltpEnergyEfficiencyClass (ADV-187) | raw | same; `""` when the CO₂ value is withheld (the class is derived from CO₂) | → Q6 |
|
|
63
|
+
| euro_emission_class / enviromental_sticker | environmentEmissions.* | raw selector keys | same (no schema to translate to) | → Q6 |
|
|
64
|
+
|
|
65
|
+
Car-level exclusions (new, counted in the report): `duplicate_id`, `filtered`, `no_id`, `no_dealer`,
|
|
66
|
+
`reserved_or_sold` (offer.offerAvailability reserved/sold), `no_valid_price`, `no_valid_image`.
|
|
67
|
+
|
|
68
|
+
## 4. Fixes on this branch
|
|
69
|
+
|
|
70
|
+
1. Builder wrote the first car twice (`{"ads":[A,A,B…`) — fixed, test.
|
|
71
|
+
2. Year 1970 for a null first registration (ACF-3717/3725 class of bug) — fixed, tests.
|
|
72
|
+
3. Price must be a valid positive end-customer price, otherwise the car is not sent.
|
|
73
|
+
4. Reserved / sold cars are not sent (absence = removal; partner semantics unknown → Q5).
|
|
74
|
+
5. Dealer address no longer blanked when the postal code table misses the place.
|
|
75
|
+
6. `engine_size` without a value is `""`, not `ccm`; unmapped selectors never drop a key.
|
|
76
|
+
7. Images: clean profile + validator, no placeholders, no WebP risk, hero first (ADV-323).
|
|
77
|
+
8. Fail-closed: no `emptyDir`; empty / partial / stale snapshot, empty result or a drop > 20 % against the
|
|
78
|
+
last good run → run blocked (exit 1, `3.1` copy does not run), last good file stays. Lock per state dir.
|
|
79
|
+
9. Dry-run default; live only with `autouncleDryRun=false`.
|
|
80
|
+
10. (1.26.41, 10.11) EnVKV: WLTP picks + EnVKV C8 publication gate (same params as the Google job). A live run
|
|
81
|
+
refuses to start without `publicationGate=enforce`. The run report has `envkv.estimated` for the two sent
|
|
82
|
+
attributes (fuel_consumption, co2_emissions; 1.26.42). DEV dry-run 10.11 (enforce): 1731 ads, estimated fuel 884 /
|
|
83
|
+
CO₂ 848 sent empty, 128 BEV with fuel "" and CO₂ 0, WLTP picks changed 338 fuel / 403 CO₂ values.
|
|
84
|
+
11. (1.26.41, 10.11) `back_link` parameters per ADV-324 (partner in `source`, identical utm pair).
|
|
85
|
+
|
|
86
|
+
## 5. Evidence
|
|
87
|
+
|
|
88
|
+
- `npm test`: 138 ok (116 existing + 14 mapping + 8 run).
|
|
89
|
+
- Other feeds byte-identical: `origin/deploy_develop` and this branch compiled with the same `node_modules`;
|
|
90
|
+
every emitted `.js` outside the AutoUncle files is identical (only `params.interface.d.ts` gains three
|
|
91
|
+
optional fields). Covers Google, FB/Meta, Criteo, meinestadt, OOYYO, OpenAI and the rest.
|
|
92
|
+
- DEV dry-run 13.10.2026 (local, DEV snapshot 12.10 15:00 CEST, DEV `3.0` params): 2234 source cars →
|
|
93
|
+
1002 ads, 370 dealers, excluded `filtered` 1190 / `no_valid_image` 42, images 7259/7259 ok, 0 placeholders,
|
|
94
|
+
0 × 1970, 0 duplicate links. Old exporter on the same snapshot: 1045 ads (first duplicated), 143 × 1970,
|
|
95
|
+
2820 placeholder URLs. 20-car sample in `autouncle-sample/` (worktree, not committed).
|
|
96
|
+
|
|
97
|
+
## 6. Not done / not decided
|
|
98
|
+
|
|
99
|
+
- No partner schema, no test import, no activation, no Cronicle change, no version bump / publish.
|
|
100
|
+
- `minPublishedItems` default is 1 (PROD exports an allow-list of 19 dealers; real size unknown) — set per job
|
|
101
|
+
before activation.
|
|
102
|
+
- `3.1` copy target `htdocs/autouncle/` does not exist on either Cronicle; hosting/transport is a partner question (Q4).
|
|
103
|
+
- Portrait / small images: the validator checks status + MIME only, no pixel size — waits for partner limits.
|
|
@@ -33,7 +33,9 @@ ingestion contract and is not used as one.
|
|
|
33
33
|
registration date "new"?
|
|
34
34
|
8. **Unique key and links.** Which field is the unique ad key on your side — `back_link`, or should we add a
|
|
35
35
|
stable vehicle id? Which UTM parameters do you want on `back_link` (historic:
|
|
36
|
-
`source=classifieds&medium=cpc&campaign=autouncle
|
|
36
|
+
`source=classifieds&medium=cpc&campaign=autouncle`; proposed and on DEV since 1.26.41:
|
|
37
|
+
`source=autouncle&medium=listing&campaign=listing` plus the same values as `utm_*`)? Do you need a click id
|
|
38
|
+
passed through?
|
|
37
39
|
9. How do you treat ads whose model is unknown (our placeholder "SONSTIGE")? Exclude or send?
|
|
38
40
|
10. Optional fields: do you use `leasing`, `has_4wd`, `description` (German text), and do you support
|
|
39
41
|
structured financing offers (rate, duration, deposit, balloon, mileage, APR with legal disclosures) instead
|
|
@@ -15,6 +15,8 @@ Not activated. Nothing below has been done on any Cronicle host.
|
|
|
15
15
|
| `feedGuardMaxSourceAgeHours` | 26 | newest base export chunk must be younger |
|
|
16
16
|
| `feedGuardMaxStaleHours` | 48 | last good file reported STALE after this |
|
|
17
17
|
| `feedGuardAcceptDrop` | off | `true` for ONE run after a real stock drop |
|
|
18
|
+
| `publicationGate` | off | `audit` counts estimated EnVKV values, `enforce` sends them empty. **A live run needs `enforce`** (config error otherwise). enforce fails the run if the enrichment log cannot be read |
|
|
19
|
+
| `publicationGateBqProject` / `…Dataset` / `…Table` | — | CBD enrichment log table, same values as the Google job; required for audit / enforce |
|
|
18
20
|
|
|
19
21
|
`String` / `Number` rules stay as they are (PROD: dealer allow-list + `metaData.imagesCount:2`).
|
|
20
22
|
|
package/docs/criteo/RUNBOOK.md
CHANGED
|
@@ -1,86 +1,86 @@
|
|
|
1
|
-
# Criteo catalog feed — runbook (ADV-319)
|
|
2
|
-
|
|
3
|
-
Status 06.11.2026, `@bytebrand/export-cars` 1.26.40. DEV only; PROD events `4.0` / `4.1` are disabled
|
|
4
|
-
and no Criteo account, contract or importer test exists yet. Nothing here allows a live upload.
|
|
5
|
-
|
|
6
|
-
## What the job does
|
|
7
|
-
|
|
8
|
-
`4.0 export criteo - part 1` (`exportType=criteo`) reads the base export snapshot
|
|
9
|
-
(`resources/cars-export/cars.<n>.json`) and writes the legacy Criteo RSS/XML file
|
|
10
|
-
`resources/criteo-export/api_export_cars_criteo.xml`. The chained `4.1 Export criteo - part 2` uploads that
|
|
11
|
-
file. Format, enums and field order are the legacy ones (decision 20.10.2026: restore as before).
|
|
12
|
-
|
|
13
|
-
Per car, in this order (first match wins, counted in `excluded`):
|
|
14
|
-
|
|
15
|
-
| Reason | Rule |
|
|
16
|
-
|---|---|
|
|
17
|
-
| `no_id` | empty `_id` |
|
|
18
|
-
| `duplicate_id` | `_id` already seen in this run (first one is kept) |
|
|
19
|
-
| `filtered` | shared partner filter: `metaData.imagesCount > 0` and the Cronicle rule params |
|
|
20
|
-
| `reserved_or_sold` | `offer.offerAvailability` is reserved or sold (Criteo 16.10: only marketable stock) |
|
|
21
|
-
| `no_valid_price` | `price.currentSalesPrice` missing, not a number or ≤ 0 |
|
|
22
|
-
| `no_marketing_image` | no generated `criteo-<id>.jpg`; no noimage placeholder is ever sent |
|
|
23
|
-
|
|
24
|
-
A car that went offline is not in the snapshot, so it is missing from the next full file; Criteo's next
|
|
25
|
-
full import removes it. `published-ids.json` keeps the ids of the last published file, and the report
|
|
26
|
-
lists `ids.added` / `ids.removed` / `ids.kept` (+ up to 20 removed ids) for each run.
|
|
27
|
-
|
|
28
|
-
Item fields: `installment` and `custom_label_0` only when the car has a real rate (months and amount > 0);
|
|
29
|
-
`energy_efficiency_class` only for a known WLTP class.
|
|
30
|
-
|
|
31
|
-
## Fail-closed guard
|
|
32
|
-
|
|
33
|
-
Shared `FeedGuard` (same as meinestadt / AutoUncle / OpenAI). The new XML is written to
|
|
34
|
-
`api_export_cars_criteo.xml.tmp` and renamed over the published file only after all checks passed; the
|
|
35
|
-
previous file is kept as `.prev`. A blocked run keeps the published file, moves the temp file to
|
|
36
|
-
`.rejected`, and **fails the job** (Cronicle code 1), so `4.1` does not upload anything.
|
|
37
|
-
|
|
38
|
-
| Code | Meaning | Action |
|
|
39
|
-
|---|---|---|
|
|
40
|
-
| `source_empty` | no snapshot chunk files (base export failed / wiped) | fix the base export, re-run |
|
|
41
|
-
| `source_partial` | chunk files with gaps | re-run the base export |
|
|
42
|
-
| `source_stale` | newest chunk older than `feedGuardMaxSourceAgeHours` (26) | base export did not run |
|
|
43
|
-
| `source_too_small` | snapshot below `feedGuardMinSourceItems` (1000) | check the base export |
|
|
44
|
-
| `published_too_small` | fewer items than `feedGuardMinPublishedItems` (500), e.g. all reserved/sold | check stock / images |
|
|
45
|
-
| `count_drop` | source or items dropped by more than `feedGuardMaxDropPercent` (20 %) vs last good | if the stock really dropped: re-run once with `feedGuardAcceptDrop=true` |
|
|
46
|
-
| `locked` | another run holds `resources/criteo-state/run.lock` (stale after 6 h) | wait / check the other job |
|
|
47
|
-
|
|
48
|
-
Stale fallback: each report carries `lastGoodAgeHours`; after `feedGuardMaxStaleHours` (48) without a
|
|
49
|
-
successful run, `stale: true` and the log line `criteo feed STALE`. The job itself already failed on every
|
|
50
|
-
blocked run, which is the alert. Whether Criteo should then get an empty file (to stop advertising
|
|
51
|
-
old stock) is an open partner question, not built.
|
|
52
|
-
|
|
53
|
-
All thresholds are Cronicle params of the `4.0` event (`feedGuardMinSourceItems`,
|
|
54
|
-
`feedGuardMinPublishedItems`, `feedGuardMaxDropPercent`, `feedGuardMaxSourceAgeHours`,
|
|
55
|
-
`feedGuardMaxStaleHours`, `feedGuardAcceptDrop`).
|
|
56
|
-
|
|
57
|
-
## Files
|
|
58
|
-
|
|
59
|
-
| Path (relative to `/opt/cronicle`) | Content |
|
|
60
|
-
|---|---|
|
|
61
|
-
| `resources/criteo-export/api_export_cars_criteo.xml` | published file (what `4.1` uploads) |
|
|
62
|
-
| `resources/criteo-export/api_export_cars_criteo.xml.prev` | file before the last publish |
|
|
63
|
-
| `resources/criteo-export/api_export_cars_criteo.xml.rejected` | output of the last blocked run |
|
|
64
|
-
| `resources/criteo-state/last-good.json` | counts + time of the last publish |
|
|
65
|
-
| `resources/criteo-state/last-run-report.json` | report of the last run (also printed as `criteo feed report: {...}`) |
|
|
66
|
-
| `resources/criteo-state/published-ids.json` | ids of the published file |
|
|
67
|
-
|
|
68
|
-
Report metrics: source chunk age (`source.ageHours`), run time (`durationMs`), source cars, items,
|
|
69
|
-
exclusions by reason, items with a rate, items without energy class, id changes, last good age.
|
|
70
|
-
Criteo's accepted / rejected records come from the Criteo importer, not from this job.
|
|
71
|
-
|
|
72
|
-
## Rollback
|
|
73
|
-
|
|
74
|
-
1. Code: `sudo npm --prefix /opt/cronicle install @bytebrand/export-cars@<previous> --prefer-online`
|
|
75
|
-
(DEV before this release: 1.26.39). Only the export-cars package changes; back up
|
|
76
|
-
`/opt/cronicle/package.json` and `package-lock.json` first.
|
|
77
|
-
2. File: `cp resources/criteo-export/api_export_cars_criteo.xml.prev resources/criteo-export/api_export_cars_criteo.xml`.
|
|
78
|
-
3. State: `resources/criteo-state/` can be deleted; the next run then has no last good counts (no drop check
|
|
79
|
-
once) and reports all ids as added.
|
|
80
|
-
|
|
81
|
-
## Tests
|
|
82
|
-
|
|
83
|
-
`test/criteo-mapping.test.ts` (item mapping) and `test/criteo-feed-run.test.ts`: 20 anonymized DEV cars
|
|
84
|
-
(`test/fixtures/criteo-20-cases.json`, synthetic ids, mapping fields only) plus sold, reserved, offline,
|
|
85
|
-
no image, price change, invalid price, duplicate id, missing id, source outage, partial / stale snapshot,
|
|
86
|
-
count drop + acceptDrop, empty stock, lock, stale last good, atomic publish. Run: `npm test`.
|
|
1
|
+
# Criteo catalog feed — runbook (ADV-319)
|
|
2
|
+
|
|
3
|
+
Status 06.11.2026, `@bytebrand/export-cars` 1.26.40. DEV only; PROD events `4.0` / `4.1` are disabled
|
|
4
|
+
and no Criteo account, contract or importer test exists yet. Nothing here allows a live upload.
|
|
5
|
+
|
|
6
|
+
## What the job does
|
|
7
|
+
|
|
8
|
+
`4.0 export criteo - part 1` (`exportType=criteo`) reads the base export snapshot
|
|
9
|
+
(`resources/cars-export/cars.<n>.json`) and writes the legacy Criteo RSS/XML file
|
|
10
|
+
`resources/criteo-export/api_export_cars_criteo.xml`. The chained `4.1 Export criteo - part 2` uploads that
|
|
11
|
+
file. Format, enums and field order are the legacy ones (decision 20.10.2026: restore as before).
|
|
12
|
+
|
|
13
|
+
Per car, in this order (first match wins, counted in `excluded`):
|
|
14
|
+
|
|
15
|
+
| Reason | Rule |
|
|
16
|
+
|---|---|
|
|
17
|
+
| `no_id` | empty `_id` |
|
|
18
|
+
| `duplicate_id` | `_id` already seen in this run (first one is kept) |
|
|
19
|
+
| `filtered` | shared partner filter: `metaData.imagesCount > 0` and the Cronicle rule params |
|
|
20
|
+
| `reserved_or_sold` | `offer.offerAvailability` is reserved or sold (Criteo 16.10: only marketable stock) |
|
|
21
|
+
| `no_valid_price` | `price.currentSalesPrice` missing, not a number or ≤ 0 |
|
|
22
|
+
| `no_marketing_image` | no generated `criteo-<id>.jpg`; no noimage placeholder is ever sent |
|
|
23
|
+
|
|
24
|
+
A car that went offline is not in the snapshot, so it is missing from the next full file; Criteo's next
|
|
25
|
+
full import removes it. `published-ids.json` keeps the ids of the last published file, and the report
|
|
26
|
+
lists `ids.added` / `ids.removed` / `ids.kept` (+ up to 20 removed ids) for each run.
|
|
27
|
+
|
|
28
|
+
Item fields: `installment` and `custom_label_0` only when the car has a real rate (months and amount > 0);
|
|
29
|
+
`energy_efficiency_class` only for a known WLTP class.
|
|
30
|
+
|
|
31
|
+
## Fail-closed guard
|
|
32
|
+
|
|
33
|
+
Shared `FeedGuard` (same as meinestadt / AutoUncle / OpenAI). The new XML is written to
|
|
34
|
+
`api_export_cars_criteo.xml.tmp` and renamed over the published file only after all checks passed; the
|
|
35
|
+
previous file is kept as `.prev`. A blocked run keeps the published file, moves the temp file to
|
|
36
|
+
`.rejected`, and **fails the job** (Cronicle code 1), so `4.1` does not upload anything.
|
|
37
|
+
|
|
38
|
+
| Code | Meaning | Action |
|
|
39
|
+
|---|---|---|
|
|
40
|
+
| `source_empty` | no snapshot chunk files (base export failed / wiped) | fix the base export, re-run |
|
|
41
|
+
| `source_partial` | chunk files with gaps | re-run the base export |
|
|
42
|
+
| `source_stale` | newest chunk older than `feedGuardMaxSourceAgeHours` (26) | base export did not run |
|
|
43
|
+
| `source_too_small` | snapshot below `feedGuardMinSourceItems` (1000) | check the base export |
|
|
44
|
+
| `published_too_small` | fewer items than `feedGuardMinPublishedItems` (500), e.g. all reserved/sold | check stock / images |
|
|
45
|
+
| `count_drop` | source or items dropped by more than `feedGuardMaxDropPercent` (20 %) vs last good | if the stock really dropped: re-run once with `feedGuardAcceptDrop=true` |
|
|
46
|
+
| `locked` | another run holds `resources/criteo-state/run.lock` (stale after 6 h) | wait / check the other job |
|
|
47
|
+
|
|
48
|
+
Stale fallback: each report carries `lastGoodAgeHours`; after `feedGuardMaxStaleHours` (48) without a
|
|
49
|
+
successful run, `stale: true` and the log line `criteo feed STALE`. The job itself already failed on every
|
|
50
|
+
blocked run, which is the alert. Whether Criteo should then get an empty file (to stop advertising
|
|
51
|
+
old stock) is an open partner question, not built.
|
|
52
|
+
|
|
53
|
+
All thresholds are Cronicle params of the `4.0` event (`feedGuardMinSourceItems`,
|
|
54
|
+
`feedGuardMinPublishedItems`, `feedGuardMaxDropPercent`, `feedGuardMaxSourceAgeHours`,
|
|
55
|
+
`feedGuardMaxStaleHours`, `feedGuardAcceptDrop`).
|
|
56
|
+
|
|
57
|
+
## Files
|
|
58
|
+
|
|
59
|
+
| Path (relative to `/opt/cronicle`) | Content |
|
|
60
|
+
|---|---|
|
|
61
|
+
| `resources/criteo-export/api_export_cars_criteo.xml` | published file (what `4.1` uploads) |
|
|
62
|
+
| `resources/criteo-export/api_export_cars_criteo.xml.prev` | file before the last publish |
|
|
63
|
+
| `resources/criteo-export/api_export_cars_criteo.xml.rejected` | output of the last blocked run |
|
|
64
|
+
| `resources/criteo-state/last-good.json` | counts + time of the last publish |
|
|
65
|
+
| `resources/criteo-state/last-run-report.json` | report of the last run (also printed as `criteo feed report: {...}`) |
|
|
66
|
+
| `resources/criteo-state/published-ids.json` | ids of the published file |
|
|
67
|
+
|
|
68
|
+
Report metrics: source chunk age (`source.ageHours`), run time (`durationMs`), source cars, items,
|
|
69
|
+
exclusions by reason, items with a rate, items without energy class, id changes, last good age.
|
|
70
|
+
Criteo's accepted / rejected records come from the Criteo importer, not from this job.
|
|
71
|
+
|
|
72
|
+
## Rollback
|
|
73
|
+
|
|
74
|
+
1. Code: `sudo npm --prefix /opt/cronicle install @bytebrand/export-cars@<previous> --prefer-online`
|
|
75
|
+
(DEV before this release: 1.26.39). Only the export-cars package changes; back up
|
|
76
|
+
`/opt/cronicle/package.json` and `package-lock.json` first.
|
|
77
|
+
2. File: `cp resources/criteo-export/api_export_cars_criteo.xml.prev resources/criteo-export/api_export_cars_criteo.xml`.
|
|
78
|
+
3. State: `resources/criteo-state/` can be deleted; the next run then has no last good counts (no drop check
|
|
79
|
+
once) and reports all ids as added.
|
|
80
|
+
|
|
81
|
+
## Tests
|
|
82
|
+
|
|
83
|
+
`test/criteo-mapping.test.ts` (item mapping) and `test/criteo-feed-run.test.ts`: 20 anonymized DEV cars
|
|
84
|
+
(`test/fixtures/criteo-20-cases.json`, synthetic ids, mapping fields only) plus sold, reserved, offline,
|
|
85
|
+
no image, price change, invalid price, duplicate id, missing id, source outage, partial / stale snapshot,
|
|
86
|
+
count drop + acceptDrop, empty stock, lock, stale last good, atomic publish. Run: `npm test`.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { toNumber } from 'lodash';
|
|
2
2
|
|
|
3
3
|
import { ExportType, FileType, ImageType } from '../enums';
|
|
4
|
-
import { Params } from '../interfaces/params.interface';
|
|
4
|
+
import { Params, PublicationGateConfig } from '../interfaces/params.interface';
|
|
5
5
|
import { FeedGuardOptions } from '../services/feed-guard';
|
|
6
6
|
import { Config } from './config';
|
|
7
7
|
|
|
@@ -55,8 +55,33 @@ export class AutouncleConfig extends Config {
|
|
|
55
55
|
maxStaleHours: positiveNumber(params.feedGuardMaxStaleHours, AUTOUNCLE_DEFAULTS.maxStaleHours),
|
|
56
56
|
acceptDrop: text(params.feedGuardAcceptDrop).toLowerCase() === 'true',
|
|
57
57
|
};
|
|
58
|
+
this.publicationGate = parsePublicationGate(params);
|
|
59
|
+
// ADV-322 "do not guess emissions": a live file never goes out with unchecked EnVKV estimates.
|
|
60
|
+
if (!this.dryRun && this.publicationGate?.mode !== 'enforce') {
|
|
61
|
+
throw new Error('a live AutoUncle run (autouncleDryRun=false) needs publicationGate=enforce');
|
|
62
|
+
}
|
|
58
63
|
this.outputFiles = [
|
|
59
64
|
`${this.pathesConfig.localConvertedDirectory}/api_export_cars_${ExportType.AUTOUNCLE}.${FileType.JSON}`,
|
|
60
65
|
];
|
|
61
66
|
}
|
|
62
67
|
}
|
|
68
|
+
|
|
69
|
+
// Same params and rules as the Google job (GoogleConfig): off by default; audit / enforce need the
|
|
70
|
+
// enrichment log table (publicationGateBqProject / Dataset / Table).
|
|
71
|
+
function parsePublicationGate(params: Params): PublicationGateConfig | undefined {
|
|
72
|
+
const mode = text(params.publicationGate).toLowerCase();
|
|
73
|
+
if (!mode || mode === 'off') return undefined;
|
|
74
|
+
if (mode !== 'audit' && mode !== 'enforce') {
|
|
75
|
+
throw new Error(`publicationGate must be off, audit or enforce, got "${mode}"`);
|
|
76
|
+
}
|
|
77
|
+
const gate = {
|
|
78
|
+
mode: mode as PublicationGateConfig['mode'],
|
|
79
|
+
project: text(params.publicationGateBqProject),
|
|
80
|
+
dataset: text(params.publicationGateBqDataset),
|
|
81
|
+
table: text(params.publicationGateBqTable),
|
|
82
|
+
};
|
|
83
|
+
if (!gate.project || !gate.dataset || !gate.table) {
|
|
84
|
+
throw new Error(`publicationGate=${mode} needs publicationGateBqProject, publicationGateBqDataset and publicationGateBqTable`);
|
|
85
|
+
}
|
|
86
|
+
return gate;
|
|
87
|
+
}
|
|
@@ -1,54 +1,54 @@
|
|
|
1
|
-
import { toNumber } from 'lodash';
|
|
2
|
-
|
|
3
|
-
import { ExportType, FileType, ImageType } from 'src/enums';
|
|
4
|
-
import { Params } from '../interfaces/params.interface';
|
|
5
|
-
import { FeedGuardOptions } from '../services/feed-guard';
|
|
6
|
-
import { Config } from './config';
|
|
7
|
-
|
|
8
|
-
const text = (value: unknown): string => (typeof value === 'string' ? value.trim() : '');
|
|
9
|
-
|
|
10
|
-
const positiveNumber = (value: unknown, fallback: number): number => {
|
|
11
|
-
const parsed = toNumber(text(value));
|
|
12
|
-
return text(value) && Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
|
|
13
|
-
};
|
|
14
|
-
|
|
15
|
-
// Same guard defaults as the meinestadt / OpenAI feeds; a job can override them via the
|
|
16
|
-
// feedGuard* Cronicle params.
|
|
17
|
-
export const CRITEO_DEFAULTS = {
|
|
18
|
-
minSourceItems: 1000,
|
|
19
|
-
minPublishedItems: 500,
|
|
20
|
-
maxDropPercent: 20,
|
|
21
|
-
maxSourceAgeHours: 26,
|
|
22
|
-
maxStaleHours: 48,
|
|
23
|
-
};
|
|
24
|
-
|
|
25
|
-
/**
|
|
26
|
-
* Criteo catalog (ADV-319): legacy RSS/XML file at the same path as before, which the chained
|
|
27
|
-
* `4.1 Export criteo - part 2` step uploads. Guard state lives in resources/criteo-state.
|
|
28
|
-
*/
|
|
29
|
-
export class CriteoConfig extends Config {
|
|
30
|
-
stateDir: string;
|
|
31
|
-
feedGuard: FeedGuardOptions;
|
|
32
|
-
|
|
33
|
-
constructor(params: Params) {
|
|
34
|
-
super(params);
|
|
35
|
-
this.campaign = ExportType.CRITEO;
|
|
36
|
-
this.pathesConfig.localConvertedDirectory = `resources/${ExportType.CRITEO}-export`;
|
|
37
|
-
this.stateDir = `resources/${ExportType.CRITEO}-state`;
|
|
38
|
-
this.imageType = ImageType.marketing;
|
|
39
|
-
this.marketingImageConfigId = this.environmentConfig[params.environment].criteoMarketingImagesConfigId;
|
|
40
|
-
this.feedGuard = {
|
|
41
|
-
stateDir: this.stateDir,
|
|
42
|
-
minSourceItems: positiveNumber(params.feedGuardMinSourceItems, CRITEO_DEFAULTS.minSourceItems),
|
|
43
|
-
minPublishedItems: positiveNumber(params.feedGuardMinPublishedItems, CRITEO_DEFAULTS.minPublishedItems),
|
|
44
|
-
maxDropPercent: positiveNumber(params.feedGuardMaxDropPercent, CRITEO_DEFAULTS.maxDropPercent),
|
|
45
|
-
maxSourceAgeHours: positiveNumber(params.feedGuardMaxSourceAgeHours, CRITEO_DEFAULTS.maxSourceAgeHours),
|
|
46
|
-
maxStaleHours: positiveNumber(params.feedGuardMaxStaleHours, CRITEO_DEFAULTS.maxStaleHours),
|
|
47
|
-
acceptDrop: text(params.feedGuardAcceptDrop).toLowerCase() === 'true',
|
|
48
|
-
};
|
|
49
|
-
this.outputFiles = [
|
|
50
|
-
`${this.pathesConfig.localConvertedDirectory}/api_export_cars_${ExportType.CRITEO}.${FileType.XML}`,
|
|
51
|
-
];
|
|
52
|
-
this.isCriteo = true;
|
|
53
|
-
}
|
|
54
|
-
}
|
|
1
|
+
import { toNumber } from 'lodash';
|
|
2
|
+
|
|
3
|
+
import { ExportType, FileType, ImageType } from 'src/enums';
|
|
4
|
+
import { Params } from '../interfaces/params.interface';
|
|
5
|
+
import { FeedGuardOptions } from '../services/feed-guard';
|
|
6
|
+
import { Config } from './config';
|
|
7
|
+
|
|
8
|
+
const text = (value: unknown): string => (typeof value === 'string' ? value.trim() : '');
|
|
9
|
+
|
|
10
|
+
const positiveNumber = (value: unknown, fallback: number): number => {
|
|
11
|
+
const parsed = toNumber(text(value));
|
|
12
|
+
return text(value) && Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
|
|
13
|
+
};
|
|
14
|
+
|
|
15
|
+
// Same guard defaults as the meinestadt / OpenAI feeds; a job can override them via the
|
|
16
|
+
// feedGuard* Cronicle params.
|
|
17
|
+
export const CRITEO_DEFAULTS = {
|
|
18
|
+
minSourceItems: 1000,
|
|
19
|
+
minPublishedItems: 500,
|
|
20
|
+
maxDropPercent: 20,
|
|
21
|
+
maxSourceAgeHours: 26,
|
|
22
|
+
maxStaleHours: 48,
|
|
23
|
+
};
|
|
24
|
+
|
|
25
|
+
/**
|
|
26
|
+
* Criteo catalog (ADV-319): legacy RSS/XML file at the same path as before, which the chained
|
|
27
|
+
* `4.1 Export criteo - part 2` step uploads. Guard state lives in resources/criteo-state.
|
|
28
|
+
*/
|
|
29
|
+
export class CriteoConfig extends Config {
|
|
30
|
+
stateDir: string;
|
|
31
|
+
feedGuard: FeedGuardOptions;
|
|
32
|
+
|
|
33
|
+
constructor(params: Params) {
|
|
34
|
+
super(params);
|
|
35
|
+
this.campaign = ExportType.CRITEO;
|
|
36
|
+
this.pathesConfig.localConvertedDirectory = `resources/${ExportType.CRITEO}-export`;
|
|
37
|
+
this.stateDir = `resources/${ExportType.CRITEO}-state`;
|
|
38
|
+
this.imageType = ImageType.marketing;
|
|
39
|
+
this.marketingImageConfigId = this.environmentConfig[params.environment].criteoMarketingImagesConfigId;
|
|
40
|
+
this.feedGuard = {
|
|
41
|
+
stateDir: this.stateDir,
|
|
42
|
+
minSourceItems: positiveNumber(params.feedGuardMinSourceItems, CRITEO_DEFAULTS.minSourceItems),
|
|
43
|
+
minPublishedItems: positiveNumber(params.feedGuardMinPublishedItems, CRITEO_DEFAULTS.minPublishedItems),
|
|
44
|
+
maxDropPercent: positiveNumber(params.feedGuardMaxDropPercent, CRITEO_DEFAULTS.maxDropPercent),
|
|
45
|
+
maxSourceAgeHours: positiveNumber(params.feedGuardMaxSourceAgeHours, CRITEO_DEFAULTS.maxSourceAgeHours),
|
|
46
|
+
maxStaleHours: positiveNumber(params.feedGuardMaxStaleHours, CRITEO_DEFAULTS.maxStaleHours),
|
|
47
|
+
acceptDrop: text(params.feedGuardAcceptDrop).toLowerCase() === 'true',
|
|
48
|
+
};
|
|
49
|
+
this.outputFiles = [
|
|
50
|
+
`${this.pathesConfig.localConvertedDirectory}/api_export_cars_${ExportType.CRITEO}.${FileType.XML}`,
|
|
51
|
+
];
|
|
52
|
+
this.isCriteo = true;
|
|
53
|
+
}
|
|
54
|
+
}
|
|
@@ -9,6 +9,7 @@ import { resolveOpenAiImageFields } from '../transformers/openai-image-fields';
|
|
|
9
9
|
import { CarService, RulesService, LoaderService, ConfigService, ImagesService } from '../services';
|
|
10
10
|
import { ImageUrlValidator } from '../services/image-url-validator.service';
|
|
11
11
|
import { FeedGuard } from '../services/feed-guard';
|
|
12
|
+
import { PublicationGate, loadPublicationGate } from '../services/publication-gate';
|
|
12
13
|
import { AutouncleFeedRun, AutouncleSourceCar } from '../services/autouncle-feed-run';
|
|
13
14
|
import { AutouncleConfig } from '../configs/autouncle.config';
|
|
14
15
|
import { Exporter } from '../interfaces/exporter.interface';
|
|
@@ -36,6 +37,16 @@ export class AutouncleExporter implements Exporter {
|
|
|
36
37
|
const transformer = await this.moduleRef.create(AutouncleTransformer);
|
|
37
38
|
const addresses = new Map<string, Promise<AutouncleAddress | null>>();
|
|
38
39
|
|
|
40
|
+
// EnVKV C8 (ADV-322): estimated values are withheld (enforce) or only counted (audit). enforce fails
|
|
41
|
+
// the job when the enrichment log cannot be read, so the last published file stays.
|
|
42
|
+
let gate: PublicationGate;
|
|
43
|
+
try {
|
|
44
|
+
gate = await loadPublicationGate(config.publicationGate);
|
|
45
|
+
} catch (error) {
|
|
46
|
+
LoggerService.exitError(`autouncle feed not published: ${error?.message ?? error}`);
|
|
47
|
+
process.exit(1);
|
|
48
|
+
}
|
|
49
|
+
|
|
39
50
|
const run = new AutouncleFeedRun(
|
|
40
51
|
{
|
|
41
52
|
forEachChunk: fn => this.loaderService.process((chunk: FsChunkInterface) => fn(chunk.data as AutouncleSourceCar[])),
|
|
@@ -55,6 +66,7 @@ export class AutouncleExporter implements Exporter {
|
|
|
55
66
|
imageStats: () => validator.stats,
|
|
56
67
|
translate: key => transformer.translate(key),
|
|
57
68
|
guard: new FeedGuard(config.feedGuard),
|
|
69
|
+
gate,
|
|
58
70
|
},
|
|
59
71
|
{
|
|
60
72
|
runId: config.runId,
|