pim-import 6.10.0 → 6.11.0

package/README.md

@@ -1,26 +1,548 @@
-# PIM IMPORT
+# pim-import
 
-A set of features needed to import content from pim to Contenteful.
+TypeScript/Node.js library to bulk-import content from a **PIM** (Product Information Management) system into **Contentful CMS**, with integrations for **Amazon S3**, **Algolia** search, and **Imgix** image management.
 
-### Services used:
+## Services
 
-- PIM
-- Contentful
-- Amazon S3
-- Algolia
+| Service    | Purpose                                      |
+| ---------- | -------------------------------------------- |
+| PIM        | Data source (products, catalogs, dictionary) |
+| Contentful | CMS destination (CMA + CDA)                  |
+| Amazon S3  | Asset and JSON snapshot storage              |
+| Algolia    | Search index synchronization                 |
+| Imgix      | Image asset management                       |
 
-## Settings
+---
 
-Depending on what you want to import you will need more or less configurations. [Read More](https://github.com/atoms-studio/pim-import/blob/main/docs/settings.md)
+## Installation
 
-## How to use
+```bash
+yarn add pim-import
+```
 
-The import process takes place through the following steps:
+---
 
-- [Import data from the dictionary](https://github.com/atoms-studio/pim-import/blob/main/docs/import-data-from-the-dictionary.md)
-- [Import taxonomies from the catalog](https://github.com/atoms-studio/pim-import/blob/main/docs/import-taxonomies-from-the-catalog.md)
-- [Import products](https://github.com/atoms-studio/pim-import/blob/main/docs/import-products.md)
-- Saving relationships between color variants
-- Saving relationships between accessories
-- Audit
-- Algolia indexes update
+## Configuration
+
+All services are initialized programmatically or via environment variables (prefixed `FPI_`).
+
+### PIM
+
+```ts
+import { initPim } from "pim-import";
+
+initPim({
+  baseURL: process.env.FPI_PIM_BASE_URL, // required
+  username: process.env.FPI_PIM_USERNAME,
+  password: process.env.FPI_PIM_PASSWORD,
+  timeout: 300, // seconds, default 300
+  sslverify: true, // default true
+});
+```
+
+### Contentful
+
+```ts
+import { initContentful } from "pim-import";
+
+initContentful({
+  accessToken: process.env.FPI_CTF_CMA_ACCESS_TOKEN, // required
+  spaceId: process.env.FPI_CTF_SPACE_ID, // required
+  environmentId: process.env.FPI_CTF_ENVIRONMENT, // required
+});
+```
+
+### Amazon S3
+
+```ts
+import { initS3 } from "pim-import";
+
+initS3({
+  accessKeyId: process.env.FPI_AWS_ACCESS_KEY, // required
+  secretAccessKey: process.env.FPI_AWS_SECRET_ACCESS_KEY, // required
+  region: process.env.FPI_AWS_REGION, // required
+  bucket: process.env.FPI_AWS_S3_BUCKET, // required
+});
+```
+
+### Environment variables reference
+
+```bash
+# PIM
+FPI_PIM_BASE_URL=
+FPI_PIM_USERNAME=
+FPI_PIM_PASSWORD=
+FPI_PIM_TIMEOUT=300
+FPI_PIM_SSL_VERIFY=true
+
+# Contentful
+FPI_CTF_CMA_ACCESS_TOKEN=
+FPI_CTF_SPACE_ID=
+FPI_CTF_ENVIRONMENT=
+
+# Amazon S3
+FPI_AWS_ACCESS_KEY=
+FPI_AWS_SECRET_ACCESS_KEY=
+FPI_AWS_REGION=
+FPI_AWS_S3_BUCKET=
+
+# Algolia
+FPI_ALGOLIA_APP_ID=
+FPI_ALGOLIA_API_KEY=
+FPI_ALGOLIA_INDEX_NAME_SUFFIX=
+FPI_ALGOLIA_INDEX_PRODUCTS_NAME=
+
+# PDF generation (AWS Lambda)
+FPID_AWS_GEN_PDF_LAMBDA_URL=
+FPI_TRIGGER_GEN_PDF_TASK_ID=
+FPI_TRIGGER_SECRET_KEY=
+
+# Notifications
+FPI_TEAMS_NOTIFICATIONS_URL=   # Microsoft Teams incoming webhook
+
+# Monitoring
+SENTRY_DSN=
+SENTRY_TRACE_SAMPLE_RATE=
+LOGDNA_APP_NAME=
+LOGDNA_KEY=
+```
+
+---
+
+## Import flow
+
+A complete import follows these steps in order:
+
+1. [Dictionary](#1-dictionary)
+2. [Taxonomies (categories, families, sub-families)](#2-taxonomies)
+3. [Products](#3-products)
+4. [Product relationships](#4-product-relationships)
+5. [Audit](#5-audit)
+6. [Algolia reindex](#6-algolia-reindex)
+
+---
+
+## 1. Dictionary
+
+Imports product field translations and icons from the `/dictionary` PIM endpoint.
+
+### Import field translations
+
+Saves localized labels for all product fields to Contentful.
+
+**Required:** `FPI_PIM_BASE_URL`, `FPI_CTF_CMA_ACCESS_TOKEN`, `FPI_CTF_SPACE_ID`, `FPI_CTF_ENVIRONMENT`
+
+```ts
+import { importDictionaryFields } from "pim-import";
+
+await importDictionaryFields();
+```
+
+### Import icons to S3
+
+Uploads product field icons retrieved from PIM to Amazon S3.
+
+**Required:** `FPI_PIM_BASE_URL`, `FPI_AWS_*`
+
+```ts
+import { importDictionaryIcons } from "pim-import";
+
+await importDictionaryIcons();
+```
+
+### Import product line taxonomy
+
+```ts
+import {
+  importDictionaryProductLine,
+  importDictionaryProductSubLine,
+} from "pim-import";
+
+await importDictionaryProductLine();
+await importDictionaryProductSubLine();
+```
+
+### Import all dictionary data at once
+
+```ts
+import { importDictionaryData } from "pim-import";
+
+await importDictionaryData();
+```
+
+---
+
+## 2. Taxonomies
+
+Imports the catalog hierarchy from the `/catalogs` PIM endpoint.
+
+Each catalog operation writes the following Contentful content types:
+
+- `topicCategory` — category with name, code, catalog, families
+- `topicSubFamily` — sub-family with name, code, catalog
+- `pageContent` + `page` — CMS pages for each category and sub-family
+
+### Import categories
+
+```ts
+import { importCategories } from "pim-import";
+
+const result = await importCategories(
+  "ARCHITECTURAL", // AvailableCatalogs: 'ARCHITECTURAL' | 'OUTDOOR' | 'DECORATIVE'
+  0, // offset (default 0)
+  -1, // limit, -1 = all (default -1)
+  "", // optional S3 path to a pre-fetched JSON snapshot
+);
+// result: { completed: boolean, offset, limit, total, s3FilePath }
+```
+
+### Import families and sub-families
+
+```ts
+import { importFamilies, importSubFamilies } from "pim-import";
+
+await importFamilies("ARCHITECTURAL", 0, -1);
+await importSubFamilies("ARCHITECTURAL", 0, -1);
+```
+
+### Import models and sub-models
+
+```ts
+import { importModels, importSubModels } from "pim-import";
+
+await importModels("ARCHITECTURAL", 0, -1);
+await importSubModels("ARCHITECTURAL", 0, -1);
+```
+
+---
+
+## 3. Products
+
+### Import by last modification date
+
+Fetches products from the `/latest-products` endpoint and creates/updates Contentful entries.
+
+Each product creates:
+
+- `topicProduct` — product with name, code, status, categories, sub-families, product line, PIM details
+- `pageContent` — page content wrapper
+- `page` — public product page with slug
+
+```ts
+import { importLatestProducts } from "pim-import";
+
+const result = await importLatestProducts(
+  "ARCHITECTURAL", // catalog
+  "20240101T00:00:00", // lastModified — ISO date used as lower bound
+  0, // page (default 0)
+  50, // size per page (default 50)
+  "20240201T00:00:00", // optional lastModifiedTo upper bound
+);
+// result: { completed, page, size, totalPages, nextPage }
+```
+
+### Import a single product by code
+
+```ts
+import { importProductByCode } from "pim-import";
+
+const entry = await importProductByCode("F3030031", "ARCHITECTURAL");
+```
+
+### Reimport products flagged by audit
+
+```ts
+import { reimportAuditProducts } from "pim-import";
+
+await reimportAuditProducts();
+```
+
+---
+
+## 4. Product relationships
+
+### Set relationships (color variants, accessories)
+
+```ts
+import { setProductsRelationships, setProductRelationships } from "pim-import";
+
+// All products in a catalog
+await setProductsRelationships("ARCHITECTURAL");
+
+// Single product
+await setProductRelationships("F3030031");
+```
+
+### Remove relationships
+
+```ts
+import {
+  removeProductFromColorVariantsByProductLine,
+  removeAllProductModelProductRelations,
+} from "pim-import";
+
+await removeProductFromColorVariantsByProductLine();
+await removeAllProductModelProductRelations();
+```
+
+### Populate destination fields
+
+```ts
+import { populateDestinations } from "pim-import";
+
+await populateDestinations();
+```
+
+---
+
+## 5. Audit
+
+Validates product data integrity across Contentful entries.
+
+```ts
+import { audit } from "pim-import";
+
+const results = await audit("ARCHITECTURAL");
+// results: AuditResults — per-product validation status and missing fields
+```
+
+---
+
+## 6. Algolia reindex
+
+All reindex functions accept an optional catalog filter. Individual variants accept a `code` string.
+
+### Products
+
+```ts
+import {
+  reindexProducts,
+  reindexProduct,
+  removeProductObject,
+} from "pim-import";
+
+await reindexProducts("ARCHITECTURAL");
+await reindexProduct("F3030031");
+await removeProductObject("F3030031");
+```
+
+### Families and sub-families
+
+```ts
+import {
+  reindexFamilies,
+  reindexFamily,
+  removeFamilyObject,
+  reindexSubFamilies,
+  reindexSubFamily,
+  removeSubFamilyObject,
+} from "pim-import";
+
+await reindexFamilies("ARCHITECTURAL");
+await reindexSubFamilies("ARCHITECTURAL");
+```
+
+### Models and sub-models
+
+```ts
+import {
+  reindexModels,
+  reindexModel,
+  removeModelObject,
+  reindexSubModels,
+  reindexSubModel,
+  removeSubModelObject,
+} from "pim-import";
+
+await reindexModels();
+await reindexSubModels();
+```
+
+### Editorial content
+
+```ts
+import {
+  reindexDownloads,
+  reindexInspirations,
+  reindexProjects,
+  reindexStories,
+  reindexPressReleases,
+  reindexPressReviews,
+  reindexPosts,
+} from "pim-import";
+
+await reindexDownloads();
+await reindexInspirations();
+await reindexProjects();
+await reindexStories();
+await reindexPressReleases();
+await reindexPressReviews();
+await reindexPosts();
+```
+
+### Clone index settings
+
+```ts
+import { cloneIndexSettings } from "pim-import";
+
+await cloneIndexSettings("products_en", "products_en_staging");
+```
+
+---
+
+## Amazon S3 utilities
+
+```ts
+import { uploadS3, saveJsonToS3, getFileFromS3, savePDFToS3 } from "pim-import";
+
+// Upload a remote file by URL
+const { Location } = await uploadS3(
+  "https://example.com/image.jpg",
+  "image.jpg",
+  "assets/",
+);
+
+// Save a JSON object
+const s3Path = await saveJsonToS3({ key: "value" }, "data.json", "snapshots/");
+
+// Retrieve a file (returns content as string)
+const content = await getFileFromS3("snapshots/data.json");
+
+// Save a PDF buffer
+await savePDFToS3(pdfBuffer, "product.pdf", "pdf/");
+```
+
+---
+
+## PDF generation
+
+Generates a technical specification PDF via AWS Lambda and returns a URL.
+
+```ts
+import { generateTechSpecPdf, generatePDFByUrl } from "pim-import";
+
+// From a product code
+const url = await generateTechSpecPdf("F3030031");
+
+// From an arbitrary URL
+const buffer = await generatePDFByUrl("https://example.com/page", "output.pdf");
+```
+
+---
+
+## Downloads
+
+Imports download resources from a CSV file.
+
+```ts
+import { importDownloads } from "pim-import";
+
+await importDownloads("/path/to/downloads.csv");
+```
+
+---
+
+## Contentful utilities
+
+```ts
+import {
+  initBaseEntries,
+  getEntryByID,
+  getEntries,
+  getTopicPage,
+  getAllProductEntriesByCatalog,
+  publishAllProductDrafts,
+  deleteEntries,
+  deletePages,
+  migrateEntryFields,
+  checkTopicDraftAndPagePublished,
+} from "pim-import";
+
+// Publish all draft products
+await publishAllProductDrafts();
+
+// Get all product entries for a catalog
+const entries = await getAllProductEntriesByCatalog("ARCHITECTURAL");
+
+// Migrate fields between content types
+await migrateEntryFields("topicProduct", { oldField: "newField" });
+
+// Check publish status
+const isReady = await checkTopicDraftAndPagePublished("topic-id-123");
+```
+
+---
+
+## Designers
+
+```ts
+import { importDesigners, importDesigner } from "pim-import";
+
+await importDesigners();
+const entry = await importDesigner("designer-code");
+```
+
+---
+
+## Auto-descriptions
+
+```ts
+import {
+  setProductsAutodescription,
+  getProductAutodescription,
+  setProductAutodescriptionByTopicId,
+} from "pim-import";
+
+await setProductsAutodescription();
+const text = await getProductAutodescription("F3030031");
+await setProductAutodescriptionByTopicId("topic-id-123");
+```
+
+---
+
+## Logging
+
+```ts
+import {
+  log,
+  setLogId,
+  setLogPath,
+  setLogFilename,
+  getLogFolder,
+} from "pim-import";
+
+setLogPath("/var/log/pim-import");
+setLogFilename("import-run");
+setLogId("session-001");
+
+log("Import started", "INFO");
+// Levels: VERBOSE | DEBUG | INFO | WARN | ERROR | SILLY | HTTP
+```
+
+---
+
+## Notifications & CI
+
+```ts
+import { notify, netlifyBuild } from "pim-import";
+
+// Send a message to a Microsoft Teams channel (via FPI_TEAMS_NOTIFICATIONS_URL)
+await notify("Import completed successfully", true);
+
+// Trigger a Netlify build
+await netlifyBuild("site-id", "build-hook-id");
+```
+
+---
+
+## Available catalogs
+
+```ts
+type AvailableCatalogs = "ARCHITECTURAL" | "OUTDOOR" | "DECORATIVE";
+```
+
+---
+
+## License
+
+MIT — [atoms.studio](https://atoms.studio)

package/dist/pim/endpoints.js

@@ -46,12 +46,12 @@ exports.getCatalogTaxonomiesHierarchy = getCatalogTaxonomiesHierarchy;
 const getDictionary = async () => {
     try {
         const timeStart = new Date();
-        const { data } = await (0, request_1.getRequest)("dictionary", {
-            params: { onlyweb: true },
+        const { data } = await (0, request_1.getRequest)("micro-service/dictionary/get/all", {
+            params: { onlyWeb: true },
         });
         const timeEnd = new Date();
         const seconds = (0, utils_1.secondBetweenTwoDate)(timeStart, timeEnd);
-        (0, logs_1.log)(`Request time: ${seconds} seconds - endpoint: dictionary`);
+        (0, logs_1.log)(`Request time: ${seconds} seconds - endpoint: micro-service/dictionary/get/all`);
         return data;
     }
     catch (err) {

package/dist/pim/methods/dictionary.js

@@ -261,11 +261,12 @@ const importDictionaryData = async () => {
     (0, logs_1.log)(`importDictionaryData`, "INFO");
     (0, logs_1.log)("Get dictionary data");
     const pimDictionaryData = await (0, endpoints_1.getDictionary)();
+    console.log("pimDictionaryData", pimDictionaryData);
     if (pimDictionaryData && Object.keys(pimDictionaryData).length) {
         const dictionaryData = {};
         for (const key in pimDictionaryData) {
             if (Array.isArray(pimDictionaryData[key])) {
-                dictionaryData[key] = pimDictionaryData[key].map((item) => ({
+                dictionaryData[key] = pimDictionaryData?.[key]?.map((item) => ({
                     code: item.code,
                     parentName: item.parentName,
                     image: item.image,

package/dist/pim/methods/latestProducts.d.ts

@@ -1,2 +1,2 @@
 import { AvailableCatalogs, PaginationResults } from "../../types";
-export declare const importLatestProducts: (catalog: AvailableCatalogs, lastModified: string, page?: number, limit?: number, lastModifiedTo?: string) => Promise<PaginationResults>;
+export declare const importLatestProducts: (catalog: AvailableCatalogs, lastModified: string, page?: number, size?: number, lastModifiedTo?: string) => Promise<PaginationResults>;

package/dist/pim/methods/latestProducts.js

@@ -5,16 +5,16 @@ const endpoints_1 = require("../endpoints");
 const logs_1 = require("../../libs/logs");
 const products_1 = require("./products");
 const utils_1 = require("../../utils");
-const importLatestProducts = async (catalog, lastModified, page = 0, limit = 100, lastModifiedTo = "") => {
+const importLatestProducts = async (catalog, lastModified, page = 0, size = 100, lastModifiedTo = "") => {
     const timeStart = new Date();
     page = Number(page);
-    limit = Number(limit);
-    (0, logs_1.log)(`importLatestProducts - catalog: ${catalog} lastModified: ${lastModified} page: ${page} limit: ${limit} lastModifiedTo: ${lastModifiedTo}`, "INFO");
-    const data = await (0, endpoints_1.getLatestProducts)(catalog, lastModified, page, limit, lastModifiedTo);
+    size = Number(size);
+    (0, logs_1.log)(`importLatestProducts - catalog: ${catalog} lastModified: ${lastModified} page: ${page} size: ${size} lastModifiedTo: ${lastModifiedTo}`, "INFO");
+    const data = await (0, endpoints_1.getLatestProducts)(catalog, lastModified, page, size, lastModifiedTo);
     const total = data.totalElements;
     const products = data.content;
     (0, logs_1.log)(`${total} products founded`);
-    let count = page * limit + 1;
+    let count = page * size + 1;
     let current = 0;
     for (const product of products) {
         (0, logs_1.log)(`${count} of ${total}`);
@@ -30,11 +30,11 @@ const importLatestProducts = async (catalog, lastModified, page = 0, limit = 100
     const timeEnd = new Date();
     const seconds = (0, utils_1.secondBetweenTwoDate)(timeStart, timeEnd);
     (0, logs_1.log)(`Request time: ${seconds} seconds`);
-    const completed = count >= total;
+    const nextPage = Number(page) + 1;
     return {
-        nextPage: completed ? null : page + 1,
-        limit,
-        completed,
+        page: nextPage,
+        size,
+        completed: nextPage >= data.totalPages,
         total,
         totalPages: data.totalPages,
     };

package/dist/resources/Dictionary.d.ts

@@ -0,0 +1,38 @@
+export type Dictionary = Record<string, DictionaryRecord[]>;
+export interface DictionaryRecord {
+    parentName: string;
+    code: string;
+    image?: string | null;
+    imageAlternative?: string | null;
+    value_en?: string | null;
+    value_it?: string | null;
+    value_es?: string | null;
+    value_de?: string | null;
+    value_fr?: string | null;
+    value_sv?: string | null;
+    value_no?: string | null;
+    value_da?: string | null;
+    value_ru?: string | null;
+    value_en_US?: string | null;
+    value_ja?: string | null;
+    value_zh?: string | null;
+    value_ko?: string | null;
+    note_en?: string | null;
+    note_it?: string | null;
+    note_es?: string | null;
+    note_de?: string | null;
+    note_fr?: string | null;
+    note_sv?: string | null;
+    note_no?: string | null;
+    note_da?: string | null;
+    note_ru?: string | null;
+    note_en_US?: string | null;
+    note_zh?: string | null;
+    note_ja?: string | null;
+    note_ko?: string | null;
+    imgRelUrl?: string | null;
+    imgAltRelUrl?: string | null;
+    priority?: number | null;
+    imageS3?: string | null;
+    imageAlternativeS3?: string | null;
+}

package/dist/resources/Dictionary.js

@@ -0,0 +1,2 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pim-import",
-  "version": "6.10.0",
+  "version": "6.11.0",
   "description": "",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
@@ -15,19 +15,15 @@
   "scripts": {
     "prepublishOnly": "npm run build",
     "publishCmd": "npm publish --access public",
-    "build": "rm -rf dist && rm -rf coverage && tsc",
-    "test": "jest --collect-coverage"
+    "build": "rm -rf dist && rm -rf coverage && tsc"
   },
   "author": "atoms.studio",
   "license": "MIT",
   "devDependencies": {
-    "@types/jest": "^26.0.22",
     "@types/mime-types": "^2.1.0",
     "@types/node-gzip": "^1.1.0",
     "@types/remove-markdown": "^0.3.1",
     "@types/request": "^2.48.5",
-    "jest": "^26.6.3",
-    "ts-jest": "^26.5.5",
     "tslint": "^6.1.3",
     "tslint-language-service": "^0.9.9",
     "typescript": "^5.1.6"