@things-factory/integration-headless 8.0.0 → 9.0.0-beta.3

package/helps/integration/connector/headless-connector.md

@@ -1,57 +1,168 @@
-### Headless Pool Connector
+### 🏆 Headless Connector
 
-**Headless Pool Connector** is designed to manage a pool of headless browser instances. This connector is useful when performing tasks that require multiple headless browser sessions simultaneously, such as PDF generation or web scraping. It efficiently manages the creation, use, and destruction of these headless browser instances.
+The **Headless Connector** is a powerful tool that utilizes headless browsers to perform **automated login**, **web scraping**, **PDF generation**, and **form submission** tasks.  
+It manages multiple browser sessions, maintains sessions, handles cookies, and supports API-based logins.
 
-#### Endpoint
+---
 
-`Headless Pool Connector` does not require a specific endpoint, but a URI can be provided as part of the connection setup. This URI is typically used as a placeholder, with the pool handling the actual connection internally. In other words, any string can be entered as the endpoint.
+## 🚀 1. Endpoint
 
-#### Parameters
+The endpoint for the **Headless Connector** should be the **URL of the target website** that you want to scrape or interact with.  
+For example, you can input `https://example.com` to extract content from that website. If you need to generate a PDF from internal content, you can also use `localhost`.
 
-- **min** (number):
-  - Specifies the minimum number of headless browser instances to maintain in the pool.
-  - Default value: `2`
-- **max** (number):
-  - Specifies the maximum number of headless browser instances the pool can handle.
-  - Default value: `10`
+---
 
-These parameters allow the pool size to be adjusted based on the expected workload. For example, if the application frequently requires multiple browser instances simultaneously, the `max` value can be increased.
+## 🔧 2. Parameters
 
-#### Example Configuration
+### ✅ Login Configuration
+
+| Parameter            | Type     | Description                                       | Default Value |
+| -------------------- | -------- | ------------------------------------------------- | ------------- |
+| `username`           | `string` | Username (login ID)                               | `""`          |
+| `password`           | `string` | Password for login                                | `""`          |
+| `loginPagePath`      | `string` | Path to the login page (e.g., `/login`)           | `"/login"`    |
+| `loginApiUrl`        | `string` | API endpoint for login (if applicable)            | `null`        |
+| `usernameSelector`   | `string` | CSS selector for the username input field         | `"#username"` |
+| `passwordSelector`   | `string` | CSS selector for the password input field         | `"#password"` |
+| `submitSelector`     | `string` | CSS selector for the login button                 | `"#submit"`   |
+| `successSelector`    | `string` | CSS selector to verify login success              | `null`        |
+| `shadowDomSelectors` | `string` | Comma-separated selectors for shadow DOM elements | `""`          |
+| `timeout`            | `number` | Timeout for login and page loading (ms)           | `15000`       |
+| `retries`            | `number` | Number of login retries                           | `3`           |
+
+---
+
+## 🔍 3. Detailed Explanation of `loginApiUrl`
+
+### 🔹 What is `loginApiUrl`?
+
+`loginApiUrl` is used when the website employs **API-based login** instead of traditional HTML form-based login. This parameter helps the connector determine how to handle the login process.
+
+### ✅ If `loginApiUrl` is **NOT set (`null`)**
+
+The **HTML form submission method** is used to log in.  
+The headless browser will manually enter the username and password into the login form and click the submit button.
+
+#### 📌 Example (HTML Form-Based Login)
 
 ```json
 {
-  "name": "headless-browser-pool",
+  "endpoint": "https://example.com",
+  "params": {
+    "username": "user123",
+    "password": "securepassword",
+    "loginPagePath": "/login",
+    "usernameSelector": "#username",
+    "passwordSelector": "#password",
+    "submitSelector": "#submit-button",
+    "successSelector": ".dashboard"
+  }
+}
+```
+
+#### 🛠 How it Works:
+
+1. Navigate to `https://example.com/login`
+2. Enter credentials into `#username` and `#password`
+3. Click `#submit-button` to submit the form
+4. Check if `.dashboard` exists to determine if login was successful
+
+---
+
+### ✅ If `loginApiUrl` **IS set**
+
+If the website uses **API-based authentication**, `loginApiUrl` should be explicitly set to the login API's endpoint.  
+For example, if the login API is `/api/auth/login`, this value must be configured accordingly.
+
+#### 📌 Example (API-Based Login)
+
+```json
+{
+  "endpoint": "https://example.com",
+  "params": {
+    "username": "user123",
+    "password": "securepassword",
+    "loginPagePath": "/login",
+    "loginApiUrl": "/api/auth/login",
+    "usernameSelector": "#username",
+    "passwordSelector": "#password",
+    "submitSelector": "#submit-button",
+    "successSelector": ".dashboard"
+  }
+}
+```
+
+#### 🛠 How it Works:
+
+1. Sends a **POST request** to `https://example.com/api/auth/login`
+2. Receives a response from the server and stores the session cookies
+3. Navigates to the main page and checks for `.dashboard` to verify login success
+
+---
+
+## 🏗 4. Example Configuration
+
+```json
+{
+  "name": "headless-browser-connector",
   "connector": "headless-connector",
-  "endpoint": "1",
+  "endpoint": "https://example.com",
   "params": {
-    "min": 5,
-    "max": 20
+    "username": "user123",
+    "password": "securepassword",
+    "loginPagePath": "/login",
+    "loginApiUrl": "/api/auth/login",
+    "usernameSelector": "#username",
+    "passwordSelector": "#password",
+    "submitSelector": "#submit",
+    "successSelector": ".dashboard",
+    "timeout": 20000,
+    "retries": 5
   }
 }
 ```
 
-In the example above, the pool maintains at least 5 headless browser instances and can scale up to 20 instances as needed.
+---
+
+## 🔄 5. Connection Lifecycle
+
+- **connect**
 
-#### Connection Lifecycle
+  - Performs the login process according to the provided parameters.
+  - Maintains the session and stores cookies for future requests.
 
-- **connect**:
+- **disconnect**
+  - Terminates the browser session and clears cookies.
 
-  - Initializes the connection and creates the pool of headless browser instances based on the configured parameters (`min` and `max`).
-  - The pool automatically manages the lifecycle of these instances, creating new ones as needed and reusing existing ones.
+---
 
-- **disconnect**:
-  - Destroys the headless browser pool, ensuring all instances are properly terminated and resources are released.
+## ⚡ 6. Supported Tasks
 
-#### Usage Scenarios
+| Task Type              | Description                            |
+| ---------------------- | -------------------------------------- |
+| `headless-pdf-capture` | Converts HTML to a PDF                 |
+| `headless-scrap`       | Extracts specific data from a web page |
+| `headless-post`        | Automates form filling and submission  |
 
-`Headless Pool Connector` is primarily used in scenarios where tasks need to be executed in a headless browser environment. For example, it is useful for generating PDFs from HTML content, capturing screenshots of web pages, or scraping data from websites.
+#### 📌 Task Examples
 
-#### Supported Tasks
+### 1️⃣ Web Scraping (`headless-scrap`)
 
-`Headless Pool Connector` supports tasks prefixed with `headless-pdf`. These tasks will utilize the headless browser instances managed by this connector.
+```json
+{
+  "name": "scrape-data",
+  "taskType": "headless-scrap",
+  "params": {
+    "path": "/products",
+    "selectors": [
+      { "text": "Product Name", "value": ".product-title" },
+      { "text": "Price", "value": ".product-price" }
+    ]
+  }
+}
+```
 
-#### Example Usage in Tasks
+### 2️⃣ PDF Generation (`headless-pdf-capture`)
 
 ```json
 {
@@ -66,6 +177,42 @@ In the example above, the pool maintains at least 5 headless browser instances a
 }
 ```
 
-#### Description
+### 3️⃣ Form Submission (`headless-post`)
+
+```json
+{
+  "name": "submit-form",
+  "taskType": "headless-post",
+  "params": {
+    "formPath": "/contact",
+    "fields": {
+      "#name": "John Doe",
+      "#email": "john@example.com",
+      "#message": "Hello, this is a test message!"
+    },
+    "submitSelector": "#submit-button"
+  }
+}
+```
+
+---
+
+## 📢 7. Description
+
+The **Headless Connector** is a powerful tool for **automated web scraping, PDF generation, and form submission** tasks.  
+It supports both **HTML-based logins** and **API-based logins**, making it suitable for a variety of website authentication methods.
+
+---
+
+### 📌 Summary
+
+| Feature                | Description                               |
+| ---------------------- | ----------------------------------------- |
+| **HTML Form Login**    | Simulates user interaction for login      |
+| **API Login**          | Sends login requests via API (faster)     |
+| **Session Management** | Stores cookies for session persistence    |
+| **Web Scraping**       | Extracts specific content from websites   |
+| **PDF Generation**     | Converts HTML pages to PDF                |
+| **Form Submission**    | Fills out and submits forms automatically |
 
-**Headless Pool Connector** efficiently manages the pool of headless browser instances to execute tasks like PDF generation and web scraping, ensuring optimization and scalability of resources in high-demand scenarios.
+📌 **With the Headless Connector, you can easily automate various tasks across different websites!** 🚀

package/helps/integration/connector/headless-connector.ms.md

@@ -1,59 +1,171 @@
-### Penyambung Kolam Tanpa Kepala (Headless Pool Connector)
+### 🏆 **Penyambung Headless (Headless Connector)**
 
-**Penyambung Kolam Tanpa Kepala** direka untuk menguruskan kolam contoh pelayar tanpa kepala. Penyambung ini berguna apabila menjalankan tugas yang memerlukan beberapa sesi pelayar tanpa kepala secara serentak seperti penjanaan PDF atau pengikisan web. Penyambung ini menguruskan penciptaan, penggunaan, dan pemusnahan contoh pelayar tanpa kepala dengan cekap.
+**Penyambung Headless** ialah alat yang kuat yang menggunakan pelayar headless untuk melaksanakan **log masuk automatik**, **pengikisan web (web scraping)**, **penjanaan PDF**, dan **penghantaran borang**.  
+Ia menguruskan berbilang sesi pelayar, mengekalkan sesi, mengendalikan kuki, serta menyokong log masuk berdasarkan API.
 
-#### Titik Akhir
+---
 
-`Penyambung Kolam Tanpa Kepala` tidak memerlukan titik akhir tertentu, tetapi URI boleh disediakan sebagai sebahagian daripada tetapan sambungan. URI ini biasanya digunakan sebagai tempat letak sementara, dan kolam akan mengendalikan sambungan sebenar secara dalaman. Dengan kata lain, sebarang rentetan boleh dimasukkan sebagai titik akhir.
+## 🚀 **1. Titik Akhir (Endpoint)**
 
-#### Parameter
+Titik akhir untuk **Penyambung Headless** ialah **URL laman web sasaran** yang ingin dikikis (scraping) atau berinteraksi dengannya.  
+Sebagai contoh, `https://example.com` boleh digunakan untuk mengekstrak data daripada laman web, atau `localhost` boleh digunakan jika anda ingin menjana fail PDF dari kandungan dalaman.
 
-- **min** (nombor):
-  - Menentukan bilangan minimum contoh pelayar tanpa kepala yang dikekalkan dalam kolam.
-  - Nilai lalai: `2`
-- **max** (nombor):
-  - Menentukan bilangan maksimum contoh pelayar tanpa kepala yang boleh diuruskan oleh kolam.
-  - Nilai lalai: `10`
+---
 
-Parameter ini membolehkan saiz kolam disesuaikan berdasarkan beban kerja yang dijangkakan. Sebagai contoh, jika aplikasi sering memerlukan beberapa contoh pelayar serentak, nilai `max` boleh ditingkatkan.
+## 🔧 **2. Parameter (Parameters)**
 
-#### Contoh Tetapan
+### ✅ **Tetapan Berkaitan Log Masuk**
+
+| Parameter            | Jenis    | Penerangan                                                   | Nilai Lalai   |
+| -------------------- | -------- | ------------------------------------------------------------ | ------------- |
+| `username`           | `string` | ID pengguna untuk log masuk                                  | `""`          |
+| `password`           | `string` | Kata laluan untuk log masuk                                  | `""`          |
+| `loginPagePath`      | `string` | Laluan ke halaman log masuk (contoh: `/login`)               | `"/login"`    |
+| `loginApiUrl`        | `string` | URL API untuk log masuk (jika diperlukan)                    | `null`        |
+| `usernameSelector`   | `string` | Pemilih CSS untuk medan input nama pengguna                  | `"#username"` |
+| `passwordSelector`   | `string` | Pemilih CSS untuk medan input kata laluan                    | `"#password"` |
+| `submitSelector`     | `string` | Pemilih CSS untuk butang log masuk                           | `"#submit"`   |
+| `successSelector`    | `string` | Pemilih CSS untuk elemen yang menunjukkan kejayaan log masuk | `null`        |
+| `shadowDomSelectors` | `string` | Pemilih untuk elemen dalam Shadow DOM (dipisahkan koma)      | `""`          |
+| `timeout`            | `number` | Had masa menunggu untuk log masuk dan pemuatan halaman (ms)  | `15000`       |
+| `retries`            | `number` | Bilangan percubaan untuk log masuk                           | `3`           |
+
+---
+
+## 🔍 **3. Penjelasan Terperinci Mengenai `loginApiUrl`**
+
+### 🔹 **Apa itu `loginApiUrl`?**
+
+`loginApiUrl` ialah parameter yang digunakan untuk **log masuk berdasarkan API**.  
+Sesetengah laman web tidak menggunakan borang HTML biasa untuk log masuk tetapi menggunakan API backend untuk mengesahkan pengguna.
+
+---
+
+### ✅ **Jika `loginApiUrl` Tidak Ditetapkan (`null`)**
+
+Dalam kes ini, **borang HTML biasa** digunakan untuk log masuk.  
+Pelayar headless akan mengisi nama pengguna dan kata laluan secara automatik, kemudian menghantar borang tersebut.
+
+#### 📌 **Contoh (Log Masuk Borang HTML)**
 
 ```json
 {
-  "name": "headless-browser-pool",
+  "endpoint": "https://example.com",
+  "params": {
+    "username": "user123",
+    "password": "securepassword",
+    "loginPagePath": "/login",
+    "usernameSelector": "#username",
+    "passwordSelector": "#password",
+    "submitSelector": "#submit-button",
+    "successSelector": ".dashboard"
+  }
+}
+```
+
+#### 🛠 **Proses Operasi**:
+
+1. Pergi ke `https://example.com/login`.
+2. Isi medan `#username` dan `#password`.
+3. Klik butang `#submit-button` untuk menghantar borang.
+4. Jika `.dashboard` wujud, log masuk dianggap berjaya.
+
+---
+
+### ✅ **Jika `loginApiUrl` Ditentukan**
+
+Dalam kes ini, laman web menggunakan **log masuk berdasarkan API**, jadi URL API perlu dinyatakan secara eksplisit.  
+Sebagai contoh, jika API log masuk ialah `/api/auth/login`, ia mesti dinyatakan dalam parameter.
+
+#### 📌 **Contoh (Log Masuk Berdasarkan API)**
+
+```json
+{
+  "endpoint": "https://example.com",
+  "params": {
+    "username": "user123",
+    "password": "securepassword",
+    "loginPagePath": "/login",
+    "loginApiUrl": "/api/auth/login",
+    "usernameSelector": "#username",
+    "passwordSelector": "#password",
+    "submitSelector": "#submit-button",
+    "successSelector": ".dashboard"
+  }
+}
+```
+
+#### 🛠 **Proses Operasi**:
+
+1. Hantar **permintaan POST** ke `https://example.com/api/auth/login`.
+2. Simpan kuki sesi selepas menerima respons.
+3. Navigasi ke halaman utama dan periksa `.dashboard` untuk memastikan log masuk berjaya.
+
+---
+
+## 🏗 **4. Contoh Konfigurasi**
+
+```json
+{
+  "name": "headless-browser-connector",
   "connector": "headless-connector",
-  "endpoint": "1",
+  "endpoint": "https://example.com",
   "params": {
-    "min": 5,
-    "max": 20
+    "username": "user123",
+    "password": "securepassword",
+    "loginPagePath": "/login",
+    "loginApiUrl": "/api/auth/login",
+    "usernameSelector": "#username",
+    "passwordSelector": "#password",
+    "submitSelector": "#submit",
+    "successSelector": ".dashboard",
+    "timeout": 20000,
+    "retries": 5
   }
 }
 ```
 
-Dalam contoh di atas, kolam akan mengekalkan sekurang-kurangnya 5 contoh pelayar tanpa kepala dan boleh berkembang kepada maksimum 20 contoh
+---
+
+## 🔄 **5. Kitaran Hayat Sambungan (Connection Lifecycle)**
 
-apabila diperlukan.
+- **connect**
 
-#### Kitaran Hidup Sambungan
+  - Mencuba log masuk berdasarkan parameter yang dikonfigurasi.
+  - Mengekalkan sesi dan menyimpan kuki untuk permintaan seterusnya.
 
-- **connect**:
+- **disconnect**
+  - Menamatkan sesi pelayar headless dan memadamkan kuki.
 
-  - Menginisialisasi sambungan dan mencipta kolam contoh pelayar tanpa kepala mengikut parameter yang ditetapkan (`min` dan `max`).
-  - Kolam akan menguruskan kitaran hidup contoh-contoh ini secara automatik, mencipta contoh baru jika diperlukan, dan menggunakan semula contoh sedia ada.
+---
 
-- **disconnect**:
-  - Menghancurkan kolam pelayar tanpa kepala dan memastikan semua contoh ditutup dengan betul dan sumber dilepaskan.
+## ⚡ **6. Tugas yang Disokong (Supported Tasks)**
 
-#### Senario Penggunaan
+| Jenis Tugas            | Penerangan                                     |
+| ---------------------- | ---------------------------------------------- |
+| `headless-pdf-capture` | Menukar HTML kepada PDF                        |
+| `headless-scrap`       | Mengekstrak data tertentu dari laman web       |
+| `headless-post`        | Mengisi dan menghantar borang secara automatik |
 
-`Penyambung Kolam Tanpa Kepala` terutamanya digunakan dalam senario di mana tugas perlu dijalankan dalam persekitaran pelayar tanpa kepala. Contohnya, ia berguna untuk menjana PDF daripada kandungan HTML, menangkap tangkapan skrin halaman web, atau mengikis data daripada laman web.
+#### 📌 **Contoh Penggunaan Tugas**
 
-#### Tugas yang Disokong
+### 1️⃣ **Pengikisan Web (`headless-scrap`)**
 
-`Penyambung Kolam Tanpa Kepala` menyokong tugas yang bermula dengan awalan `headless-pdf`. Tugas-tugas ini akan menggunakan contoh pelayar tanpa kepala yang diuruskan oleh penyambung ini.
+```json
+{
+  "name": "scrape-data",
+  "taskType": "headless-scrap",
+  "params": {
+    "path": "/products",
+    "selectors": [
+      { "text": "Nama Produk", "value": ".product-title" },
+      { "text": "Harga", "value": ".product-price" }
+    ]
+  }
+}
+```
 
-#### Contoh Penggunaan dalam Tugas
+### 2️⃣ **Penjanaan PDF (`headless-pdf-capture`)**
 
 ```json
 {
@@ -68,6 +180,42 @@ apabila diperlukan.
 }
 ```
 
-#### Penjelasan
+### 3️⃣ **Penghantaran Borang (`headless-post`)**
+
+```json
+{
+  "name": "submit-form",
+  "taskType": "headless-post",
+  "params": {
+    "formPath": "/contact",
+    "fields": {
+      "#name": "Ali Bin Ahmad",
+      "#email": "ali@example.com",
+      "#message": "Salam, ini adalah mesej ujian!"
+    },
+    "submitSelector": "#submit-button"
+  }
+}
+```
+
+---
+
+## 📢 **7. Penerangan (Description)**
+
+**Penyambung Headless** membolehkan **pengikisan web, log masuk automatik, penjanaan PDF, dan penghantaran borang** dilakukan secara automatik.  
+Ia menyokong kedua-dua **log masuk berdasarkan borang HTML** dan **log masuk API**, membolehkan sambungan yang stabil di pelbagai laman web.
+
+---
+
+### 📌 **Ringkasan**
+
+| Fungsi                    | Penerangan                                                |
+| ------------------------- | --------------------------------------------------------- |
+| **Log Masuk Borang HTML** | Mensimulasikan interaksi pengguna untuk log masuk         |
+| **Log Masuk API**         | Menggunakan permintaan API untuk log masuk (lebih pantas) |
+| **Penyelenggaraan Sesi**  | Menyimpan kuki untuk memastikan sesi kekal aktif          |
+| **Pengikisan Web**        | Mengekstrak data yang ditentukan secara automatik         |
+| **Penukaran PDF**         | Menukar halaman HTML kepada format PDF                    |
+| **Penghantaran Borang**   | Mengisi dan menghantar borang secara automatik            |
 
-**Penyambung Kolam Tanpa Kepala** menguruskan kolam contoh pelayar tanpa kepala secara cekap untuk melaksanakan tugas seperti penjanaan PDF dan pengikisan web, memastikan pengoptimuman dan kebolehskalaan sumber dalam senario permintaan tinggi.
+📌 **Gunakan Penyambung Headless untuk mengautomasikan pelbagai tugas web dengan lebih mudah dan cekap!** 🚀