data-science-document-ai 1.58.0__py3-none-any.whl → 1.60.0__py3-none-any.whl

src/prompts/library/bookingConfirmation/yangming/prompt.txt

@@ -7,66 +7,54 @@ The Freight Forwarding company receives Booking Confirmation from Yangming Carri
 These Booking Confirmations contain various details related to booking, container pick up and drop off depot details, vessel details, as well as other transport Legs data.
 They may be written in different languages such as English, German, Vietnamese, Chinese, and other European languages, and can appear in a variety of formats and layouts.
 Your role is to accurately extract specific entities from these Booking Confirmations to support efficient processing and accurate record-keeping.
-<CONTEXT>
-
 
-bookingNumber: A unique identifier for the booking.
-cyCutOff: The deadline for cargo to be delivered to the Container Yard.
-gateInReference: A reference code for cargo entering the terminal.
-gateInTerminal: The specific terminal where cargo is gated in.
-mblNumber: The Master Bill of Lading number.
-pickUpReference: A reference code for cargo pickup.
-pickUpTerminal: The specific terminal for cargo pickup.
-siCutOff: The deadline for submitting shipping instructions.
-vgmCutOff: The deadline for submitting the Verified Gross Mass of the cargo.
-transportLegs:
-    eta: The estimated time of arrival for a specific leg.
-    etd: The estimated time of departure for a specific leg.
-    imoNumber: The International Maritime Organization number for a specific leg.
-    portOfDischarge: The port where cargo is unloaded for a specific leg.
-    portOfLoading: The port where cargo is loaded for a specific leg.
-    vesselName: The name of the vessel for a specific leg.
-    voyage: The journey or route taken by the vessel for a specific leg.
+To provide context on the journey of a containers for both Export and Import shipments,
+For Export shipment: An empty container is picked up from a depot (pickupDepotCode) using a pickUpReference and goods loaded into it at a warehouse. Then the loaded container / cargo is transported back to a Container Yard or gateInTerminal before the cyCutOff date for further shipping processes. Then the POL of the First TransportLeg may start from the gateInTerminal or a different POL too.
+For Import Shipment: The loaded container / cargo arrives at a port of discharge then picked up at pickUpTerminal using pickUpReference. After delivery, an empty container is returned to a depot (dropOffDepotCode).
+<CONTEXT>
 
-your task is to extract the text value of the following entities and page numbers starting from 0 where the value was found in the document:
-SCHEMA_PLACEHOLDER
+<INSTRUCTIONS>
+- Populate fields as defined in the response schema.
+- Use the data field description to understand the context of the data.
 
-Keywords for datapoints:
-- bookingNumber: Our Reference
-- cyCutOff: FCL delivery cut-off
-- gateInReference: Our Reference
-- gateInTerminal: Export terminal delivery address
-- mblNumber: BL/SWB No(s).
-- pickUpReference: Export door positioning address(es)
-- siCutOff: shipping instruction closing
-- vgmCutOff: VGM cut-off
-- eta: eta, ETA
-- etd: etd, ETD
-- imoNumber: IMO No, IMO number
-- portOfDischarge: to
-- portOfLoading: from
-- vesselName: vessel
-- voyage: Voy. no
+- gateInTerminalCode: The specific terminal where cargo is gated in. It is mentioned as Delivery Terminal. This sometimes can be the same as portOfLoading of the First transportLeg.
+- cyCutOff: The deadline for cargo to be delivered to the Container Yard. It can be found at Cargo Cut Off or FCL delivery cut-off.
 
-Table Structure to extract TransporLegs:
-- transportlegs table has following colum names: From, To, By, ETD, ETA
-- If the first leg does not have a ETD and ETA, Also By column includes following info: ["Truck", "Combined Waterway"]; skip the row, start from one row below.
+- transportLegs: Multiple Transport Legs entries may exist, capture all instances under "transportLegs". Make sure the order of the legs are important.
+    - eta: The estimated time of arrival for a specific leg.
+    - etd: The estimated time of departure for a specific leg. It can also be referred as ETS (Estimated Time of Sailing).
+    - imoNumber: The International Maritime Organization number for a specific leg.
+    - portOfDischarge: The port where cargo is unloaded for a specific leg.
+    - portOfLoading: The port where cargo is loaded for a specific leg.
+    - vesselName: The name of the vessel for a specific leg.
+    - voyage: The journey or route taken by the vessel for a specific leg.
 
-Further explanation for the transportLegs part as follows:
-- you must differentiate the Voyage from DP Voyage. Never used the value with the key 'DP Voyage'.
-- Vessel name is in 'By' column and under the key Vessel
+IMPORTANT explanation for the transportLegs part as follows:
+- There is at least one leg in each document.
 - 'eta' must be equal or later than 'etd'!
-- portOfLoading and portOfDischarge are name of the Ports. You can rely on the port names from all over the world.
-- portOfLoading and portOfDischarge distinctly denotes the name of the ports. If you find abbreviation of the port use it, if not you can use the full name of the port
-- Abbrevations most likely to be in the paranthesis like follows (DEHAM).
-
-Further explanation for datapoints except transportLegs part as follows:
-- If gateInReference is null, assign it the same value as bookingNumber.
-- If pickUpReference is null, assign it the same value as bookingNumber.
-- If mblNumber is null, assign it the same value as bookingNumber.
-
-You must apply the following rules:
-- The JSON schema must be followed during the extraction.
-- The values must only include text found in the document
-- Do not normalize any entity value.
-- Validate the JSON make sure its a valid JSON ! No extra text, no missing comma!
+- Multiple legs are possible. When there are multiple legs,
+    - Sequential Sorting: You must manually re-order legs based on etd then eta, regardless of their order in the source text.
+    - "Transhipment" indicates the presence of a multi-leg journey.
+    - Transhipment Date is usually empty therefore you may keep ETA of the first leg and ETD of the second leg as null if there is a transhipment.
+    - Transhipment Handling: Treat any mentioned "Transhipment" as the bridge between two legs (Discharge for Leg A and Loading for Leg B).
+    - The Connectivity Rule: For any sequence of legs, the Port of Discharge of the previous leg must match the Port of Loading of the following leg.
+        - First Transhipment is the Port of Discharge for the first transportLegs and Port of Loading for the second transportLegs.
+        - Second Transhipment is the Port of Discharge for the second transportLegs and Port of Loading for the third transportLegs.
+    - Timeline Integrity: Ensure a "No Time Travel" policy: The eta of a previous leg must be earlier than or equal to the etd of the following leg.
+
+
+Structure of Multiple Leg Sequence & Mapping
+ Leg 1 (Initial):
+ - `portOfLoading`: Sailing.
+ - `portOfDischarge`: Transhipment (if exists), otherwise DISCHARGE.
+ - `etd`: Sailing Date ETS.
+ - `eta`: Transhipment Date (if exists), otherwise keep it null in case of multi-legs. Else, DISCHARGE Date for single leg.
+
+
+ Leg 2 (Intermediate):  Trigger: Only if PORT OF TRANSHIPMENT exists.
+ - `portOfLoading`: portOfDischarge of Leg 1 (Transhipment).
+ - `portOfDischarge`: DISCHARGE.
+ - `etd`: Transhipment Date if exists else keep it null
+ - `eta`: DISCHARGE Date. Do not extract Destination Date here.
+
+<INSTRUCTIONS>

src/prompts/library/customsInvoice/other/placeholders.json

@@ -31,7 +31,7 @@
         },
         "documentType": {"type":  "STRING", "nullable": true},
         "dueDate": {"type":  "STRING", "nullable": true,
-            "description": "The date by which the payment should be made by Forto Logistics SE & Co KG. Do Not calculate dueDate based on issueDate or any other date. Extract it directly from the invoice."},
+            "description": "The date by which the payment should be made by Forto Logistics SE & Co KG. If dueDate is not available in the invoice, calculate dueDate based on issueDate and paymentTerm."},
         "eta": {"type":  "STRING", "nullable": true,
             "description": "Estimated Time of Arrival (ETA) is the expected date when the shipment will arrive at its destination."},
         "etd": {"type":  "STRING", "nullable": true,

src/prompts/library/customsInvoice/other/prompt.txt

@@ -46,7 +46,11 @@ Your role is to accurately extract specific entities from these invoices to supp
         - CUSTOMS SUPPORT invoices do not have a vatAmount and vatPercentage.
 
 - issueDate: The date the document was issued.
-- dueDate: The date by which the payment should be made. Do Not calculate dueDate based on issueDate or any other date. Extract it directly from the invoice.
+- dueDate:
+    - The date by which the payment should be made. If dueDate is explicitly mentioned in the invoice, extract that value directly.
+    - If dueDate is not available in the invoice, calculate dueDate based on issueDate and paymentTerm.
+    - Example: If issueDate is 01.03.2023 and paymentTerm is 30 days, then dueDate will be 31.03.2023.
+    - If paymentTerm is not available, do not extract dueDate.
 
 - lineItem: Details of each COGS and Customs line item on the invoice from each page. Make sure to extract each amount and currency separately.
     - uniqueId: A unique id which associated with the lineItem as each line item can belong to a different shipment. Extract only if its available in the line item. Either a shipmentId starting with an S and followed by 6 or 8 numeric values or a mblNumber. If shipmentId or mblNumber does not exist, set it to containerNumber.
@@ -91,7 +95,7 @@ Your role is to accurately extract specific entities from these invoices to supp
         - sentence: A sentence from the invoice indicating the payment status (e.g., "Vorschuss", "Prepayment", "Paid", "Partially Paid", "Unpaid"). This helps summarize the overall payment status of the invoice.
 
 IMPORTANT NOTE:
-- Ensure all extracted values are directly from the document. Do not make assumptions or modifications.
+- Ensure all extracted values are directly from the document. Do not make assumptions, modifications or calculations except dueDate.
 - CustomSized invoices contain line items in a table format in a attached page. Table with headings Shipment ID, Partner Line Item Description, Quantity, Amount, and VAT. Extract all the line items from each tables from each page.
 - Do not normalize or modify any entity values.
 - Pay attention to the line item details and paymentInformation, as they may vary significantly across different invoices.

src/prompts/library/partnerInvoice/other/placeholders.json

@@ -29,7 +29,7 @@
         },
         "documentType": {"type": "STRING", "nullable": true},
         "dueDate": {"type": "STRING", "nullable": true,
-            "description": "The date by which the payment should be made by Forto Logistics SE & Co KG. Do Not calculate dueDate based on issueDate or any other date. Extract it directly from the invoice."},
+            "description": "The date by which the payment should be made by Forto Logistics SE & Co KG. If dueDate is not available in the invoice, calculate dueDate based on issueDate and paymentTerm."},
         "eta": {"type": "STRING", "nullable": true,
             "description": "Estimated Time of Arrival (ETA) is the expected date when the shipment will arrive at its destination."},
         "etd": {"type": "STRING", "nullable": true,

src/prompts/library/partnerInvoice/other/prompt.txt

@@ -42,7 +42,11 @@ Your role is to accurately extract specific entities from these invoices to supp
     - Remove "TVA" characters from VAT ID in MSC invoices. For example, CHE-111.954.803 TVA should be extracted as CHE-111.954.803
 
 - issueDate: The date the document was issued.
-- dueDate: The date by which the payment should be made. Do Not calculate dueDate based on issueDate or any other date. Extract it directly from the invoice.
+- dueDate:
+    - The date by which the payment should be made. If dueDate is explicitly mentioned in the invoice, extract that value directly.
+    - If dueDate is not available in the invoice, calculate dueDate based on issueDate and paymentTerm.
+    - Example: If issueDate is 01.03.2023 and paymentTerm is 30 days, then dueDate will be 31.03.2023.
+    - If paymentTerm is not available, do not extract dueDate.
 
 - eta and etd: Few invoices contains same date for ARRIVED/DEPARTED or ETA/ETD. Extract it for both eta and etd.
 
@@ -81,7 +85,7 @@ Your role is to accurately extract specific entities from these invoices to supp
 - reverseChargeSentence: A sentence which indicate that the reverse charge applies. Mostly fund as Tax Clause.
 
 IMPORTANT NOTE:
-- Ensure all extracted values are directly from the document. Do not make assumptions, modifications or calculations.
+- Ensure all extracted values are directly from the document. Do not make assumptions, modifications or calculations except dueDate.
 - CustomSized invoices contain line items in a table format in the attached page. Table with headings Shipment ID, Partner Line Item Description, Quantity, Amount, and VAT. Extract all the line items from such tables from each page.
 - Do not split the quantity into different line items. e.g., if quantity is 2 or 2 CTR or 2 BIL, do not create 2 separate line items with quantity 1 each.
 - Do not normalize or modify any entity values.

src/utils.py

@@ -361,10 +361,7 @@ def extract_top_pages(pdf_bytes, num_pages=4):
 
 
 async def get_tms_mappings(
-    input_list: List[str],
-    embedding_type: str,
-    llm_ports: Optional[List[str]] = None,
-    input_key: str = None,
+    input_list: List[str], embedding_type: str, llm_ports: Optional[List[str]] = None
 ) -> Dict[str, Any]:
     """Get TMS mappings for the given values.
 
@@ -373,7 +370,6 @@ async def get_tms_mappings(
         embedding_type (str): Type of embedding to use
          (e.g., "container_types", "ports", "depots", "lineitems", "terminals").
         llm_ports (list[str], optional): List of LLM ports to use. Defaults to None.
-        input_key (str, optional): Key to use for input list in payload. Defaults to None.
 
     Returns:
         dict or string: A dictionary or a string with the mapping results.
@@ -393,7 +389,7 @@ async def get_tms_mappings(
         input_list = [input_list]
 
     # Always send a dict with named keys
-    payload = {input_key or embedding_type: input_list}
+    payload = {embedding_type: input_list}
 
     if llm_ports:
         payload["llm_ports"] = llm_ports if isinstance(llm_ports, list) else [llm_ports]