nv-ingest 2025.10.21.dev20251021__tar.gz → 2025.10.23.dev20251023__tar.gz

{nv_ingest-2025.10.21.dev20251021 → nv_ingest-2025.10.23.dev20251023}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nv-ingest
-Version: 2025.10.21.dev20251021
+Version: 2025.10.23.dev20251023
 Summary: Python module for multimodal document ingestion
 Author-email: Jeremy Dyer <jdyer@nvidia.com>
 License:                                  Apache License

{nv_ingest-2025.10.21.dev20251021 → nv_ingest-2025.10.23.dev20251023}/nv_ingest/api/v2/README.md

@@ -40,10 +40,24 @@ This behaviour matches the V1 tracing model and sets the foundation for adding W
 
 The fetch endpoint returns a JSON body shaped like the following:
 
-```
+```json
 {
   "data": [...],
   "status": "success",
+  "trace": {
+    "trace::entry::pdf_extractor": 1000,
+    "trace::exit::pdf_extractor": 2150,
+    "trace::resident_time::pdf_extractor": 250,
+    "trace::entry::table_extractor": 1200,
+    "trace::exit::table_extractor": 2300,
+    "trace::resident_time::table_extractor": 300
+    // ... parent-level aggregated traces only (clean, V1-compatible)
+  },
+  "annotations": {
+    "annotation::uuid-1": {"task_id": "pdf_extractor", "task_result": "SUCCESS"},
+    "annotation::uuid-2": {"task_id": "table_extractor", "task_result": "SUCCESS"}
+    // ... all annotations from all chunks (annotations have unique UUIDs)
+  },
   "metadata": {
     "parent_job_id": "<uuid>",
     "total_pages": 320,
@@ -68,9 +82,9 @@ The fetch endpoint returns a JSON body shaped like the following:
         "chunk_index": 1,
         "start_page": 1,
         "end_page": 32,
-        "trace": {"trace::sink_push": 1.7285796e+18, ...}
+        "trace": {"trace::entry::pdf_extractor": 1.7599e18, ...}
       }
-      // ...
+      // ... per-chunk trace details
     ],
     "annotation_segments": [
       {
@@ -78,17 +92,76 @@ The fetch endpoint returns a JSON body shaped like the following:
         "chunk_index": 1,
         "start_page": 1,
         "end_page": 32,
-        "annotations": {"annotation::stage": "sink", ...}
+        "annotations": {"annotation::uuid": {...}, ...}
+      }
+      // ... per-chunk annotation details
+    ]
+  }
+}
+```
+
+**Top-level trace and annotations** (V1 compatibility):
+- `trace`: Contains **only parent-level aggregated traces** for clean V1 compatibility
+  - `trace::entry::<stage>` - Earliest entry time across all chunks
+  - `trace::exit::<stage>` - Latest exit time across all chunks
+  - `trace::resident_time::<stage>` - Sum of all chunk durations (total compute time)
+- `annotations`: Merged annotations from all chunks (annotations have unique UUIDs so merge safely)
+- These fields match V1 structure, allowing existing client code to work without modification
+
+**Note:** Chunk-level trace details are available in `metadata.trace_segments[]` for granular analysis
+
+**Parent-Level Trace Aggregation:**
+
+For split PDFs, parent-level metrics are automatically computed for each stage (including nested stages):
+
+- `trace::entry::<stage>` - Earliest entry time across all chunks (when first chunk entered stage)
+- `trace::exit::<stage>` - Latest exit time across all chunks (when last chunk exited stage)
+- `trace::resident_time::<stage>` - Sum of all chunk durations (total compute time in stage)
+
+**Supports arbitrary nesting depth:**
+- Simple: `trace::entry::pdf_extractor`
+- Nested: `trace::entry::pdf_extractor::pdf_extraction::pdfium_pages_to_numpy_0`
+
+**Example:**
+```json
+{
+  "trace": {
+    "trace::entry::pdf_extractor": 1000,
+    "trace::exit::pdf_extractor": 2150,
+    "trace::resident_time::pdf_extractor": 250
+    // ... only parent-level aggregations (clean, concise)
+  },
+  "metadata": {
+    "trace_segments": [
+      {
+        "chunk_index": 1,
+        "start_page": 1,
+        "end_page": 32,
+        "trace": {
+          "trace::entry::pdf_extractor": 1000,
+          "trace::exit::pdf_extractor": 1100
+        }
+      },
+      {
+        "chunk_index": 2,
+        "trace": {
+          "trace::entry::pdf_extractor": 2000,
+          "trace::exit::pdf_extractor": 2150
+        }
       }
-      // ...
     ]
   }
 }
 ```
 
-- `trace_segments` and `annotation_segments` appear only when the sink emits telemetry for a given chunk.
-- Clients can correlate chunk data by matching `job_id` or `chunk_index` across `chunks`, `trace_segments`, and `annotation_segments`.
-- Failed chunk entries remain in `failed_subjobs`; if a chunk is missing from the telemetry arrays, the sink did not emit trace/annotation payloads for that chunk.
+**Note:** `resident_time` represents total compute time (sum of chunk durations), while `exit - entry` shows wall-clock span.
+
+**Detailed metadata** (V2-specific):
+- `trace_segments`: **Chunk-level trace data** with page ranges for granular per-chunk analysis
+- `annotation_segments`: Per-chunk annotation data with page ranges
+- Clients can correlate chunk data by matching `job_id` or `chunk_index` across arrays
+- Failed chunk entries remain in `failed_subjobs`; missing chunks indicate the sink did not emit telemetry
+- **To access chunk traces:** Use `metadata.trace_segments[]` - each segment contains the full trace dict for that chunk
 
 ## Testing
 

{nv_ingest-2025.10.21.dev20251021 → nv_ingest-2025.10.23.dev20251023}/nv_ingest/api/v2/ingest.py

@@ -432,6 +432,88 @@ def _extract_ray_telemetry(result: Dict[str, Any]) -> Tuple[Optional[Dict[str, A
     return trace_dict, annotations_dict
 
 
+def _aggregate_parent_traces(chunk_traces: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    Aggregate chunk-level traces into parent-level metrics.
+
+    For each stage found in chunk traces:
+    - trace::entry::<stage> = min(all chunk entries) - earliest start
+    - trace::exit::<stage> = max(all chunk exits) - latest finish
+    - trace::resident_time::<stage> = sum(chunk durations) - total compute
+
+    Parameters
+    ----------
+    chunk_traces : Dict[str, Any]
+        Trace dict with chunk-prefixed keys (chunk_N::trace::entry::stage_name)
+
+    Returns
+    -------
+    Dict[str, Any]
+        Parent-level aggregated traces (trace::entry::stage_name, etc.)
+    """
+    # Group by stage: {stage_name: {chunk_idx: {entry: float, exit: float}}}
+    stage_data: Dict[str, Dict[int, Dict[str, Any]]] = {}
+
+    for key, value in chunk_traces.items():
+        if not key.startswith("chunk_"):
+            continue
+
+        parts = key.split("::")
+        if len(parts) < 4:  # Minimum: chunk_N::trace::entry/exit::stage_name
+            continue
+
+        if parts[1] != "trace":  # Ensure it's a trace key
+            continue
+
+        chunk_idx_str = parts[0].split("_")[1]  # "chunk_1" -> "1"
+        try:
+            chunk_idx = int(chunk_idx_str)
+        except ValueError:
+            continue
+
+        event_type = parts[2]  # "entry" or "exit"
+
+        # Stage name is everything after trace::entry:: or trace::exit::
+        # Handles both simple (pdf_extractor) and nested (pdf_extractor::pdf_extraction::pdfium_0)
+        stage_name = "::".join(parts[3:])  # Join remaining parts
+
+        if event_type not in ("entry", "exit"):
+            continue
+
+        if stage_name not in stage_data:
+            stage_data[stage_name] = {}
+        if chunk_idx not in stage_data[stage_name]:
+            stage_data[stage_name][chunk_idx] = {}
+
+        stage_data[stage_name][chunk_idx][event_type] = value
+
+    # Compute aggregated metrics
+    parent_traces: Dict[str, Any] = {}
+
+    for stage_name, chunks in stage_data.items():
+        entries = []
+        exits = []
+        durations = []
+
+        for chunk_data in chunks.values():
+            entry = chunk_data.get("entry")
+            exit_time = chunk_data.get("exit")
+
+            # Both entry and exit must exist for valid pair
+            if entry is not None and exit_time is not None:
+                entries.append(entry)
+                exits.append(exit_time)
+                durations.append(exit_time - entry)
+
+        # Only add parent traces if we have valid data
+        if entries and exits:
+            parent_traces[f"trace::entry::{stage_name}"] = min(entries)
+            parent_traces[f"trace::exit::{stage_name}"] = max(exits)
+            parent_traces[f"trace::resident_time::{stage_name}"] = sum(durations)
+
+    return parent_traces
+
+
 def _build_aggregated_response(
     parent_job_id: str,
     subjob_results: List[Optional[Dict[str, Any]]],
@@ -469,6 +551,9 @@ def _build_aggregated_response(
         "description": (
             "One or more subjobs failed to complete" if any_failed else "Aggregated result composed from subjob outputs"
         ),
+        # Top-level trace/annotations for V1 compatibility
+        "trace": {},
+        "annotations": {},
         "metadata": {
             "parent_job_id": parent_job_id,
             "total_pages": metadata.get("total_pages", len(subjob_ids)),
@@ -498,6 +583,7 @@ def _build_aggregated_response(
             end_page = descriptor.get("end_page")
 
             if trace_data:
+                # Add to trace_segments (detailed, per-chunk view)
                 aggregated_result["metadata"]["trace_segments"].append(
                     {
                         "job_id": descriptor.get("job_id"),
@@ -507,8 +593,10 @@ def _build_aggregated_response(
                         "trace": trace_data,
                     }
                 )
+                # Chunk traces stay in metadata.trace_segments only (not in top-level)
 
             if annotation_data:
+                # Add to annotation_segments (detailed, per-chunk view)
                 aggregated_result["metadata"]["annotation_segments"].append(
                     {
                         "job_id": descriptor.get("job_id"),
@@ -518,10 +606,28 @@ def _build_aggregated_response(
                         "annotations": annotation_data,
                     }
                 )
+                # Merge into top-level annotations (annotations have unique UUIDs, safe to merge)
+                aggregated_result["annotations"].update(annotation_data)
         else:
             # Note failed page
             logger.warning(f"Page {page_num} failed or missing")
 
+    # Compute parent-level trace aggregations from trace_segments
+    trace_segments = aggregated_result["metadata"]["trace_segments"]
+    if trace_segments:
+        # Build a temporary chunk trace dict for aggregation
+        temp_chunk_traces = {}
+        for segment in trace_segments:
+            chunk_idx = segment.get("chunk_index")
+            chunk_trace = segment.get("trace", {})
+            for trace_key, trace_value in chunk_trace.items():
+                prefixed_key = f"chunk_{chunk_idx}::{trace_key}"
+                temp_chunk_traces[prefixed_key] = trace_value
+
+        # Aggregate and set as top-level trace (only parent traces, no chunk traces)
+        parent_level_traces = _aggregate_parent_traces(temp_chunk_traces)
+        aggregated_result["trace"] = parent_level_traces
+
     return aggregated_result
 
 
@@ -566,11 +672,15 @@ async def submit_job_v2(
         original_source_id = source_ids[0] if source_ids else "unknown_source.pdf"
         original_source_name = source_names[0] if source_names else "unknown_source.pdf"
 
+        # Track page count for all PDFs (used for both splitting logic and metadata)
+        pdf_page_count_cache = None
+
         # Check if this is a PDF that needs splitting
         if document_types and payloads and document_types[0].lower() == "pdf":
             # Decode the payload to check page count
             pdf_content = base64.b64decode(payloads[0])
             page_count = get_pdf_page_count(pdf_content)
+            pdf_page_count_cache = page_count  # Cache for later use
             pages_per_chunk = get_pdf_split_page_count(client_override=client_split_page_count)
 
             # Split if the document has more pages than our chunk size
@@ -656,6 +766,34 @@ async def submit_job_v2(
         await ingest_service.submit_job(updated_job_spec, parent_job_id)
         await ingest_service.set_job_state(parent_job_id, STATE_SUBMITTED)
 
+        # If this was a PDF (even if not split), store page count metadata for tracking
+        if pdf_page_count_cache is not None:
+            try:
+                # Use cached page count from earlier check to avoid re-decoding
+                # Store minimal metadata for non-split PDFs (consistent with split PDFs)
+                single_pdf_metadata: Dict[str, Any] = {
+                    "total_pages": pdf_page_count_cache,
+                    "pages_per_chunk": pdf_page_count_cache,  # Single chunk = entire document
+                    "original_source_id": original_source_id,
+                    "original_source_name": original_source_name,
+                    "document_type": document_types[0],
+                    "subjob_order": [],  # No subjobs for non-split PDFs
+                }
+
+                # Store as parent job metadata with empty subjob list for consistency
+                await ingest_service.set_parent_job_mapping(
+                    parent_job_id,
+                    [],  # Empty subjob list
+                    single_pdf_metadata,
+                    subjob_descriptors=[],
+                )
+                logger.debug(
+                    f"Stored page count metadata for non-split PDF {original_source_name}: {pdf_page_count_cache} pages"
+                )
+            except Exception as metadata_err:
+                # Don't fail the job if metadata storage fails
+                logger.warning(f"Failed to store page count metadata for {parent_job_id}: {metadata_err}")
+
         response.headers["x-trace-id"] = trace.format_trace_id(current_trace_id)
         return parent_job_id
 
@@ -792,6 +930,32 @@ async def fetch_job_v2(job_id: str, ingest_service: INGEST_SERVICE_T):
 
             logger.debug(f"Parent job {job_id} has {len(subjob_ids)} subjobs")
 
+            # Special case: Non-split PDFs have metadata but no subjobs
+            # Fetch the result directly and augment with page count metadata
+            if len(subjob_ids) == 0:
+                logger.debug(f"Job {job_id} is a non-split PDF, fetching result directly")
+                try:
+                    job_response = await ingest_service.fetch_job(job_id)
+
+                    # Augment response with page count metadata
+                    if isinstance(job_response, dict):
+                        if "metadata" not in job_response:
+                            job_response["metadata"] = {}
+                        job_response["metadata"]["total_pages"] = metadata.get("total_pages")
+                        job_response["metadata"]["original_source_id"] = metadata.get("original_source_id")
+                        job_response["metadata"]["original_source_name"] = metadata.get("original_source_name")
+
+                    # Update job state after successful fetch
+                    await _update_job_state_after_fetch(job_id, ingest_service)
+
+                    return _stream_json_response(job_response)
+                except (TimeoutError, RedisError, ConnectionError):
+                    logger.debug(f"Job {job_id} (non-split PDF) not ready yet")
+                    raise HTTPException(status_code=202, detail="Job is processing. Retry later.")
+                except Exception as e:
+                    logger.exception(f"Error fetching non-split PDF job {job_id}: {e}")
+                    raise HTTPException(status_code=500, detail="Internal server error during job fetch.")
+
             # Build ordered descriptors for subjobs
             stored_descriptors = subjob_info.get("subjob_descriptors") or []
             descriptor_lookup = {entry.get("job_id"): entry for entry in stored_descriptors if isinstance(entry, dict)}

{nv_ingest-2025.10.21.dev20251021 → nv_ingest-2025.10.23.dev20251023}/nv_ingest/pipeline/default_pipeline_impl.py

@@ -318,6 +318,7 @@ stages:
     config:
       api_key: $NGC_API_KEY|$NVIDIA_API_KEY
       model_name: $VLM_CAPTION_MODEL_NAME|"nvidia/llama-3.1-nemotron-nano-vl-8b-v1"
+      endpoint_url: $VLM_CAPTION_ENDPOINT|"http://vlm:8000/v1/chat/completions"
       prompt: "Caption the content of this image:"
     replicas:
       min_replicas: 0

{nv_ingest-2025.10.21.dev20251021 → nv_ingest-2025.10.23.dev20251023}/nv_ingest.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nv-ingest
-Version: 2025.10.21.dev20251021
+Version: 2025.10.23.dev20251023
 Summary: Python module for multimodal document ingestion
 Author-email: Jeremy Dyer <jdyer@nvidia.com>
 License:                                  Apache License