vtas-metric 1.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Ritanshu Prasad
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,303 @@
1
+ Metadata-Version: 2.4
2
+ Name: vtas-metric
3
+ Version: 1.3.0
4
+ Summary: VTAS: Visual-Text Alignment Score for Evaluating Vision-Language Models
5
+ Home-page: https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric
6
+ Author: Ritanshu Prasad
7
+ Author-email: ritanshu.prasad@example.com
8
+ Classifier: Programming Language :: Python :: 3
9
+ Classifier: License :: OSI Approved :: MIT License
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.8
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: torch
15
+ Requires-Dist: transformers
16
+ Requires-Dist: spacy
17
+ Requires-Dist: sentence-transformers
18
+ Requires-Dist: Pillow
19
+ Dynamic: author
20
+ Dynamic: author-email
21
+ Dynamic: classifier
22
+ Dynamic: description
23
+ Dynamic: description-content-type
24
+ Dynamic: home-page
25
+ Dynamic: license-file
26
+ Dynamic: requires-dist
27
+ Dynamic: requires-python
28
+ Dynamic: summary
29
+
30
+ # VTAS — Visual-Truth Alignment Score
31
+
32
+ [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
33
+ [![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/)
34
+
35
+ **A white-box, reference-free evaluation metric for image captioning that grounds generated text directly against the visual content of the image.**
36
+
37
+ > *"Does the model see what it says it sees?"* — VTAS answers this question without needing human reference captions.
38
+
39
+ ---
40
+
41
+ ## Why Does This Metric Exist?
42
+
43
+ Image captioning models generate natural language descriptions of images. To measure how "good" a generated caption is, the research community relies on a set of standard metrics. However, every single one of them shares a critical architectural flaw: **they are blind to the image.**
44
+
45
+ They compare the **generated text** against **human-written reference text**, but never verify whether the generated text actually describes what is physically present in the image. This creates three categories of failure:
46
+
47
+ ### The Problem with Existing Metrics
48
+
49
+ #### 1. BLEU (Bilingual Evaluation Understudy)
50
+ BLEU measures n-gram precision — it counts how many word chunks in the generated caption appear in the human reference.
51
+
52
+ **The Flaw:** It is purely lexical. It has no understanding of meaning.
53
+
54
+ | Generated Caption | Reference Caption | BLEU-4 Score | Correct? |
55
+ |---|---|---|---|
56
+ | *"A man relaxes on a sofa"* | *"A person sits on a couch"* | **~0.0** (no 4-gram overlap) | ✅ Yes — semantically identical |
57
+ | *"A person sits on a couch near a dog"* | *"A person sits on a couch"* | **High** (strong overlap) | ❌ Maybe — was there actually a dog? BLEU doesn't check. |
58
+
59
+ > BLEU punishes correct captions that use synonyms, and rewards captions that copy reference words — even if the referenced objects are hallucinated.
60
+
61
+ #### 2. METEOR (Metric for Evaluation of Translation with Explicit ORdering)
62
+ METEOR improves on BLEU by adding stemming ("running" → "run") and synonym matching via external WordNet tables.
63
+
64
+ **The Flaw:** It still compares text-to-text. It relies on pre-built synonym tables that are finite, language-specific, and cannot cover domain-specific vocabulary. It never verifies against the image.
65
+
66
+ #### 3. CIDEr (Consensus-based Image Description Evaluation)
67
+ CIDEr uses TF-IDF weighting to reward captions that mention rare, image-specific words (e.g., "frisbee" scores higher than "the").
68
+
69
+ **The Flaw:** It is entirely dependent on the consensus of human reference captions. If the reference captions are biased, incomplete, or inconsistent, CIDEr scores become unreliable. It cannot detect hallucinations if the hallucinated word happens to appear in a reference.
70
+
71
+ #### 4. CLIPScore
72
+ CLIPScore is the current state-of-the-art reference-free metric. It embeds both the image and the text into a shared vector space using CLIP and computes cosine similarity.
73
+
74
+ **The Flaw:** It is a **black box**. It outputs a single similarity score but provides zero interpretability. If the score is low, there is no way to determine whether the model missed an object, hallucinated one, got a color wrong, or made a spatial error.
75
+
76
+ #### 5. CHAIR (Caption Hallucination Assessment with Image Relevance)
77
+ CHAIR was specifically designed to detect object hallucinations by comparing generated nouns against ground-truth object annotations.
78
+
79
+ **The Flaw:** It requires **expensive, human-annotated object labels** for every image. It cannot run on raw, unannotated images, making it impractical for real-world deployment at scale.
80
+
81
+ ---
82
+
83
+ ### The Gap
84
+
85
+ | Capability | BLEU | METEOR | CIDEr | CLIPScore | CHAIR |
86
+ |---|:---:|:---:|:---:|:---:|:---:|
87
+ | Reference-Free | ❌ | ❌ | ❌ | ✅ | ❌ |
88
+ | Detects Hallucinations | ❌ | ❌ | ❌ | Weak | ✅ |
89
+ | Handles Synonyms | ❌ | Partial | ❌ | ✅ | ❌ |
90
+ | White-Box / Interpretable | ✅ | ✅ | ✅ | ❌ | ✅ |
91
+ | Fully Automated (No Human Labels) | ✅ | ✅ | ✅ | ✅ | ❌ |
92
+
93
+ **No single existing metric is simultaneously: reference-free, hallucination-aware, synonym-robust, interpretable, AND fully automated.**
94
+
95
+ VTAS is engineered to fill that gap.
96
+
97
+ ---
98
+
99
+ ## What is VTAS?
100
+
101
+ VTAS (Visual-Truth Alignment Score) is a **Two-Tier Compound AI System** that evaluates image captions against physical reality:
102
+
103
+ 1. **Tier 1 (Object Grounding):** An Object Detector (DETR) extracts physical objects. An NLP Parser (spaCy) extracts caption nouns. A Semantic Bridge (MiniLM, phrase-based) resolves synonyms.
104
+ 2. **Tier 2 (Scene/Context Grounding):** Unmatched nouns (e.g., "bedroom", "outside", or undetected objects) are verified globally against the image using CLIP zero-shot classification.
105
+
106
+ The final **VTAS Score (v1.2)** uses the mathematically rigorous **F-beta Score** to balance **Object Precision** (absence of hallucinations) and **Visual Recall** (detecting visible objects) into a single metric [0, 1].
107
+
108
+ ### VTAS Properties
109
+ | Property | Value |
110
+ |---|---|
111
+ | Reference-Free | ✅ No human captions needed |
112
+ | Detects Hallucinations | ✅ Explicit precision penalty for invented objects |
113
+ | Handles Synonyms | ✅ Phrase-based semantic embeddings |
114
+ | Scene Aware | ✅ Tier 2 CLIP validates complex environments |
115
+ | Interpretable | ✅ Pinpoints exactly which word failed |
116
+
117
+ ---
118
+
119
+ ## Pictorial Demonstrations (VTAS in Action)
120
+
121
+ The following infographics were generated automatically during a VTAS evaluation run. They visually demonstrate how VTAS solves the flaws of legacy metrics.
122
+
123
+ ### Example 1: Perfect Alignment & Synonym Bridging
124
+ Unlike BLEU, VTAS physically draws bounding boxes to verify reality. Even if the wording differs from human annotators, VTAS bridges the vocabulary gap using phrase-based semantic matching.
125
+ ![Perfect Alignment](assets/best_example.png)
126
+
127
+ ### Example 2: Scene Context & The "Recall Penalty"
128
+ Traditional metrics like CHAIR crash without human bounding boxes. VTAS operates autonomously.
129
+ * **Tier 2 Rescue:** Notice how words like "fireplace" (which DETR missed) are rescued by the CLIP context module!
130
+ * **Visual Recall Penalty:** This caption scored an F1 of 0.00 not because it hallucinated wildly, but because it *described the scene* instead of listing the physical objects detected by DETR.
131
+ ![Hallucination and Recall Penalty](assets/hallucination_example.png)
132
+
133
+ ---
134
+
135
+ ## Quick Start
136
+
137
+ ```python
138
+ from vtas import VTASEvaluator
139
+
140
+ evaluator = VTASEvaluator()
141
+
142
+ result = evaluator.score(
143
+ image_path="path/to/image.jpg",
144
+ caption="A man throws a frisbee to his dog in the park"
145
+ )
146
+
147
+ print(f"VTAS Score (F1): {result['vtas_score']:.4f}")
148
+ print(f"Object Precision: {result['precision']:.4f}")
149
+ print(f"Visual Recall: {result['recall']:.4f}")
150
+ print(f"Matched Objects: {result['matched']}")
151
+ print(f"Hallucinated Objects: {result['hallucinated']}")
152
+ ```
153
+
154
+ ---
155
+
156
+ ## Examples
157
+
158
+ ### Example 1: Perfect Caption
159
+ **Image:** A park scene with a person, a dog, and a frisbee.
160
+
161
+ | Component | Output |
162
+ |---|---|
163
+ | **DETR detects** | `{'person', 'dog', 'frisbee'}` |
164
+ | **Caption says** | *"A man throws a frisbee to his dog"* → `{'man', 'frisbee', 'dog'}` |
165
+ | **Semantic Bridge** | man↔person (0.82 ✅), frisbee↔frisbee (1.0 ✅), dog↔dog (1.0 ✅) |
166
+ | **Visual Recall** | 3/3 = 1.0 |
167
+ | **Hallucination Rate** | 0/3 = 0.0 |
168
+ | **VTAS** | **1.0** ✅ |
169
+
170
+ ### Example 2: Synonym Robustness (BLEU would fail here)
171
+ **Image:** A living room with a person, a couch, and a TV.
172
+
173
+ | Component | Output |
174
+ |---|---|
175
+ | **DETR detects** | `{'person', 'couch', 'tv'}` |
176
+ | **Caption says** | *"A man relaxes on a sofa watching television"* → `{'man', 'sofa', 'television'}` |
177
+ | **Semantic Bridge** | man↔person (0.82 ✅), sofa↔couch (0.89 ✅), television↔tv (0.91 ✅) |
178
+ | **Visual Recall** | 3/3 = 1.0 |
179
+ | **Hallucination Rate** | 0/3 = 0.0 |
180
+ | **VTAS** | **1.0** ✅ |
181
+
182
+ > BLEU-4 would score this near zero because no exact 4-grams match. VTAS correctly identifies it as a perfect caption.
183
+
184
+ ### Example 3: Hallucination Detection
185
+ **Image:** A boy holding an umbrella standing near a cow.
186
+
187
+ | Component | Output |
188
+ |---|---|
189
+ | **DETR detects** | `{'person', 'umbrella', 'cow'}` |
190
+ | **Caption says** | *"A group of people with umbrellas and dogs"* → `{'group', 'people', 'umbrellas', 'dogs'}` |
191
+ | **Semantic Bridge** | people↔person (0.85 ✅), umbrellas↔umbrella (0.97 ✅), group↔? (❌), dogs↔cow (0.45 ❌) |
192
+ | **Visual Recall** | 2/3 = 0.67 (missed 'cow') |
193
+ | **Hallucination Rate** | 2/4 = 0.50 (hallucinated 'group', 'dogs') |
194
+ | **VTAS** | **0.33** ❌ — Correctly penalized |
195
+
196
+ > This is a real hallucination observed during our BLIP zero-shot baseline evaluation. Traditional metrics may still score this reasonably if the reference caption mentions "people." VTAS catches it because DETR confirms there is no "group" or "dogs" in the image.
197
+
198
+ ---
199
+
200
+ ## Installation
201
+
202
+ ```bash
203
+ pip install -r requirements.txt
204
+ ```
205
+
206
+ ### Dependencies
207
+ - `torch` — PyTorch backend
208
+ - `transformers` — Hugging Face (DETR object detection)
209
+ - `sentence-transformers` — MiniLM semantic embeddings
210
+ - `spacy` — NLP noun extraction
211
+ - `Pillow` — Image processing
212
+
213
+ ---
214
+
215
+ ## Project Structure
216
+ ```
217
+ VTAS_Evaluation_Metric/
218
+ ├── src/
219
+ │ ├── vtas.py # Core VTAS evaluator class
220
+ │ ├── visual_grounding.py # DETR-based object detection module
221
+ │ ├── linguistic_extraction.py # spaCy-based noun extraction module
222
+ │ └── semantic_bridge.py # MiniLM-based semantic similarity module
223
+ ├── docs/
224
+ │ └── architecture.md # Full architectural specification & math
225
+ ├── requirements.txt
226
+ ├── .gitignore
227
+ └── README.md
228
+ ```
229
+
230
+ ---
231
+
232
+ ## Known Limitations & Future Work
233
+
234
+ VTAS v1 is a strong first step, but we believe in transparent research. The following are **known architectural limitations** that we have identified through extensive evaluation and are actively working to resolve.
235
+
236
+ ### Limitation 1: Semantic Collapse in the Semantic Bridge
237
+
238
+ **Observed in:** Motorcycle example (Image #78, VTAS = 1.00)
239
+
240
+ ![Semantic Collapse Example](assets/limitation_semantic_collapse.png)
241
+
242
+ **The Problem:** The caption reads *"the man is riding a motorcycle on the road with people watching."* DETR detected only `{'person', 'motorcycle'}`. The Semantic Bridge matched both `"man" → person (0.70)` and `"people" → person (0.76)`. On the surface, this appears correct — VTAS gave a perfect 1.00. But look closely:
243
+
244
+ * `"man"` refers to the **rider** (the main subject).
245
+ * `"people"` refers to the **spectators** in the background (secondary subjects).
246
+
247
+ These are two semantically distinct groups, but both collapsed onto the **single DETR label `"person"`**. VTAS treated them as the same entity. This is a fundamental limitation:
248
+
249
+ > **VTAS v1 operates on unordered sets of labels.** It has no concept of *which* person is being referenced. If DETR detects one `"person"`, any number of human-related nouns ("man", "woman", "people", "boy") in the caption will all match it, even if they refer to different individuals. This inflates both Precision and Recall.
250
+
251
+ **Root Cause:** DETR returns *unique category labels*, not instance-level detections. Even if DETR internally draws 5 bounding boxes for 5 people, the VTAS pipeline reduces them to the single label `"person"`.
252
+
253
+ ### Limitation 2: Scene-Level Descriptions Penalized by Object-Level Recall
254
+
255
+ **Observed in:** Living room example (Image #0, VTAS = 0.00)
256
+
257
+ ![Scene Penalty Example](assets/limitation_scene_penalty.png)
258
+
259
+ **The Problem:** The caption reads *"the living room has a yellow wall and a black fireplace."* DETR detected 9 objects: `{'clock', 'tv', 'dining table', 'refrigerator', 'person', 'vase', 'potted plant', 'bottle', 'chair'}`. The caption mentioned none of these individual items — it described the **collective environment** instead. As a result:
260
+
261
+ * **Object Precision = 0.50** — `"fireplace"` was rescued by CLIP (Tier 2), but `"wall"` was marked as a hallucination.
262
+ * **Visual Recall = 0.00** — The caption mentioned 0 out of 9 detected objects.
263
+ * **VTAS (F1) = 0.00** — Because F1 is a harmonic mean, if either P or R is 0, the entire score collapses.
264
+
265
+ This is arguably the **most important limitation**: a VLM that writes *"the kitchen is clean and ready for us to use"* receives a VTAS of 0.00, even though that caption is a perfectly valid, human-like description of a kitchen. The model correctly identified the scene but chose to describe it holistically rather than itemizing the oven, refrigerator, sink, cups, and bowls that DETR found.
266
+
267
+ > **VTAS v1 is an object-listing metric, not a scene-understanding metric.** It heavily rewards captions that enumerate discrete objects ("there is a clock, a TV, a chair, and a bottle") and penalizes captions that describe the scene as a whole ("this is a clean, well-lit living room"). This is by design — VTAS measures *grounded object alignment* — but it means that VTAS alone cannot evaluate the full quality of a caption.
268
+
269
+ **Root Cause:** The F-beta score requires non-zero Recall. When a caption contains zero object-level nouns that overlap with DETR's detections, Recall = 0, and the harmonic mean forces the entire score to 0 regardless of Precision.
270
+
271
+ ---
272
+
273
+ ### Future Work
274
+
275
+ Based on the limitations above, we propose the following research directions for VTAS v2:
276
+
277
+ | Direction | Addresses | Approach |
278
+ |---|---|---|
279
+ | **Instance-Aware Grounding** | Limitation 1 | Replace set-based matching with instance-level alignment. Use DETR's bounding box coordinates to distinguish "the rider" from "the spectators". Match each caption noun to a specific detection, preventing multiple nouns from collapsing onto one label. |
280
+ | **Scene-Level Recall** | Limitation 2 | Introduce a secondary recall signal that rewards captions for correctly identifying the *type* of scene (kitchen, bedroom, park) using CLIP or a dedicated scene classifier (e.g., Places365). A caption like "the kitchen is clean" would earn partial Recall credit for correctly classifying the environment. |
281
+ | **Attribute-Level Grounding** | New capability | Extend VTAS beyond nouns to verify adjectives ("red car", "large dog"). Use visual question-answering or attribute classifiers to check if described properties (color, size, material) match the image. |
282
+ | **Spatial Awareness** | New capability | Verify spatial claims ("the dog is *next to* the person", "the cup is *on* the table") against the physical coordinates of DETR bounding boxes. |
283
+ | **Dynamic Threshold Calibration** | Both | Replace the fixed similarity thresholds (MiniLM τ=0.65, CLIP τ=0.22) with dataset-adaptive thresholds calibrated on a held-out validation split, similar to ROC-AUC optimal threshold selection. |
284
+
285
+ ---
286
+
287
+ ## License
288
+
289
+ This project is licensed under the [MIT License](LICENSE).
290
+
291
+ ## Citation
292
+
293
+ If you use VTAS in your research, please cite:
294
+
295
+ ```bibtex
296
+ @software{prasad2026vtas,
297
+ author = {Prasad, Ritanshu},
298
+ title = {VTAS: Visual-Truth Alignment Score},
299
+ year = {2026},
300
+ url = {https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric},
301
+ license = {MIT}
302
+ }
303
+ ```
@@ -0,0 +1,274 @@
1
+ # VTAS — Visual-Truth Alignment Score
2
+
3
+ [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
4
+ [![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/)
5
+
6
+ **A white-box, reference-free evaluation metric for image captioning that grounds generated text directly against the visual content of the image.**
7
+
8
+ > *"Does the model see what it says it sees?"* — VTAS answers this question without needing human reference captions.
9
+
10
+ ---
11
+
12
+ ## Why Does This Metric Exist?
13
+
14
+ Image captioning models generate natural language descriptions of images. To measure how "good" a generated caption is, the research community relies on a set of standard metrics. However, every single one of them shares a critical architectural flaw: **they are blind to the image.**
15
+
16
+ They compare the **generated text** against **human-written reference text**, but never verify whether the generated text actually describes what is physically present in the image. This creates three categories of failure:
17
+
18
+ ### The Problem with Existing Metrics
19
+
20
+ #### 1. BLEU (Bilingual Evaluation Understudy)
21
+ BLEU measures n-gram precision — it counts how many word chunks in the generated caption appear in the human reference.
22
+
23
+ **The Flaw:** It is purely lexical. It has no understanding of meaning.
24
+
25
+ | Generated Caption | Reference Caption | BLEU-4 Score | Correct? |
26
+ |---|---|---|---|
27
+ | *"A man relaxes on a sofa"* | *"A person sits on a couch"* | **~0.0** (no 4-gram overlap) | ✅ Yes — semantically identical |
28
+ | *"A person sits on a couch near a dog"* | *"A person sits on a couch"* | **High** (strong overlap) | ❌ Maybe — was there actually a dog? BLEU doesn't check. |
29
+
30
+ > BLEU punishes correct captions that use synonyms, and rewards captions that copy reference words — even if the referenced objects are hallucinated.
31
+
32
+ #### 2. METEOR (Metric for Evaluation of Translation with Explicit ORdering)
33
+ METEOR improves on BLEU by adding stemming ("running" → "run") and synonym matching via external WordNet tables.
34
+
35
+ **The Flaw:** It still compares text-to-text. It relies on pre-built synonym tables that are finite, language-specific, and cannot cover domain-specific vocabulary. It never verifies against the image.
36
+
37
+ #### 3. CIDEr (Consensus-based Image Description Evaluation)
38
+ CIDEr uses TF-IDF weighting to reward captions that mention rare, image-specific words (e.g., "frisbee" scores higher than "the").
39
+
40
+ **The Flaw:** It is entirely dependent on the consensus of human reference captions. If the reference captions are biased, incomplete, or inconsistent, CIDEr scores become unreliable. It cannot detect hallucinations if the hallucinated word happens to appear in a reference.
41
+
42
+ #### 4. CLIPScore
43
+ CLIPScore is the current state-of-the-art reference-free metric. It embeds both the image and the text into a shared vector space using CLIP and computes cosine similarity.
44
+
45
+ **The Flaw:** It is a **black box**. It outputs a single similarity score but provides zero interpretability. If the score is low, there is no way to determine whether the model missed an object, hallucinated one, got a color wrong, or made a spatial error.
46
+
47
+ #### 5. CHAIR (Caption Hallucination Assessment with Image Relevance)
48
+ CHAIR was specifically designed to detect object hallucinations by comparing generated nouns against ground-truth object annotations.
49
+
50
+ **The Flaw:** It requires **expensive, human-annotated object labels** for every image. It cannot run on raw, unannotated images, making it impractical for real-world deployment at scale.
51
+
52
+ ---
53
+
54
+ ### The Gap
55
+
56
+ | Capability | BLEU | METEOR | CIDEr | CLIPScore | CHAIR |
57
+ |---|:---:|:---:|:---:|:---:|:---:|
58
+ | Reference-Free | ❌ | ❌ | ❌ | ✅ | ❌ |
59
+ | Detects Hallucinations | ❌ | ❌ | ❌ | Weak | ✅ |
60
+ | Handles Synonyms | ❌ | Partial | ❌ | ✅ | ❌ |
61
+ | White-Box / Interpretable | ✅ | ✅ | ✅ | ❌ | ✅ |
62
+ | Fully Automated (No Human Labels) | ✅ | ✅ | ✅ | ✅ | ❌ |
63
+
64
+ **No single existing metric is simultaneously: reference-free, hallucination-aware, synonym-robust, interpretable, AND fully automated.**
65
+
66
+ VTAS is engineered to fill that gap.
67
+
68
+ ---
69
+
70
+ ## What is VTAS?
71
+
72
+ VTAS (Visual-Truth Alignment Score) is a **Two-Tier Compound AI System** that evaluates image captions against physical reality:
73
+
74
+ 1. **Tier 1 (Object Grounding):** An Object Detector (DETR) extracts physical objects. An NLP Parser (spaCy) extracts caption nouns. A Semantic Bridge (MiniLM, phrase-based) resolves synonyms.
75
+ 2. **Tier 2 (Scene/Context Grounding):** Unmatched nouns (e.g., "bedroom", "outside", or undetected objects) are verified globally against the image using CLIP zero-shot classification.
76
+
77
+ The final **VTAS Score (v1.2)** uses the mathematically rigorous **F-beta Score** to balance **Object Precision** (absence of hallucinations) and **Visual Recall** (detecting visible objects) into a single metric [0, 1].
78
+
79
+ ### VTAS Properties
80
+ | Property | Value |
81
+ |---|---|
82
+ | Reference-Free | ✅ No human captions needed |
83
+ | Detects Hallucinations | ✅ Explicit precision penalty for invented objects |
84
+ | Handles Synonyms | ✅ Phrase-based semantic embeddings |
85
+ | Scene Aware | ✅ Tier 2 CLIP validates complex environments |
86
+ | Interpretable | ✅ Pinpoints exactly which word failed |
87
+
88
+ ---
89
+
90
+ ## Pictorial Demonstrations (VTAS in Action)
91
+
92
+ The following infographics were generated automatically during a VTAS evaluation run. They visually demonstrate how VTAS solves the flaws of legacy metrics.
93
+
94
+ ### Example 1: Perfect Alignment & Synonym Bridging
95
+ Unlike BLEU, VTAS physically draws bounding boxes to verify reality. Even if the wording differs from human annotators, VTAS bridges the vocabulary gap using phrase-based semantic matching.
96
+ ![Perfect Alignment](assets/best_example.png)
97
+
98
+ ### Example 2: Scene Context & The "Recall Penalty"
99
+ Traditional metrics like CHAIR crash without human bounding boxes. VTAS operates autonomously.
100
+ * **Tier 2 Rescue:** Notice how words like "fireplace" (which DETR missed) are rescued by the CLIP context module!
101
+ * **Visual Recall Penalty:** This caption scored an F1 of 0.00 not because it hallucinated wildly, but because it *described the scene* instead of listing the physical objects detected by DETR.
102
+ ![Hallucination and Recall Penalty](assets/hallucination_example.png)
103
+
104
+ ---
105
+
106
+ ## Quick Start
107
+
108
+ ```python
109
+ from vtas import VTASEvaluator
110
+
111
+ evaluator = VTASEvaluator()
112
+
113
+ result = evaluator.score(
114
+ image_path="path/to/image.jpg",
115
+ caption="A man throws a frisbee to his dog in the park"
116
+ )
117
+
118
+ print(f"VTAS Score (F1): {result['vtas_score']:.4f}")
119
+ print(f"Object Precision: {result['precision']:.4f}")
120
+ print(f"Visual Recall: {result['recall']:.4f}")
121
+ print(f"Matched Objects: {result['matched']}")
122
+ print(f"Hallucinated Objects: {result['hallucinated']}")
123
+ ```
124
+
125
+ ---
126
+
127
+ ## Examples
128
+
129
+ ### Example 1: Perfect Caption
130
+ **Image:** A park scene with a person, a dog, and a frisbee.
131
+
132
+ | Component | Output |
133
+ |---|---|
134
+ | **DETR detects** | `{'person', 'dog', 'frisbee'}` |
135
+ | **Caption says** | *"A man throws a frisbee to his dog"* → `{'man', 'frisbee', 'dog'}` |
136
+ | **Semantic Bridge** | man↔person (0.82 ✅), frisbee↔frisbee (1.0 ✅), dog↔dog (1.0 ✅) |
137
+ | **Visual Recall** | 3/3 = 1.0 |
138
+ | **Hallucination Rate** | 0/3 = 0.0 |
139
+ | **VTAS** | **1.0** ✅ |
140
+
141
+ ### Example 2: Synonym Robustness (BLEU would fail here)
142
+ **Image:** A living room with a person, a couch, and a TV.
143
+
144
+ | Component | Output |
145
+ |---|---|
146
+ | **DETR detects** | `{'person', 'couch', 'tv'}` |
147
+ | **Caption says** | *"A man relaxes on a sofa watching television"* → `{'man', 'sofa', 'television'}` |
148
+ | **Semantic Bridge** | man↔person (0.82 ✅), sofa↔couch (0.89 ✅), television↔tv (0.91 ✅) |
149
+ | **Visual Recall** | 3/3 = 1.0 |
150
+ | **Hallucination Rate** | 0/3 = 0.0 |
151
+ | **VTAS** | **1.0** ✅ |
152
+
153
+ > BLEU-4 would score this near zero because no exact 4-grams match. VTAS correctly identifies it as a perfect caption.
154
+
155
+ ### Example 3: Hallucination Detection
156
+ **Image:** A boy holding an umbrella standing near a cow.
157
+
158
+ | Component | Output |
159
+ |---|---|
160
+ | **DETR detects** | `{'person', 'umbrella', 'cow'}` |
161
+ | **Caption says** | *"A group of people with umbrellas and dogs"* → `{'group', 'people', 'umbrellas', 'dogs'}` |
162
+ | **Semantic Bridge** | people↔person (0.85 ✅), umbrellas↔umbrella (0.97 ✅), group↔? (❌), dogs↔cow (0.45 ❌) |
163
+ | **Visual Recall** | 2/3 = 0.67 (missed 'cow') |
164
+ | **Hallucination Rate** | 2/4 = 0.50 (hallucinated 'group', 'dogs') |
165
+ | **VTAS** | **0.33** ❌ — Correctly penalized |
166
+
167
+ > This is a real hallucination observed during our BLIP zero-shot baseline evaluation. Traditional metrics may still score this reasonably if the reference caption mentions "people." VTAS catches it because DETR confirms there is no "group" or "dogs" in the image.
168
+
169
+ ---
170
+
171
+ ## Installation
172
+
173
+ ```bash
174
+ pip install -r requirements.txt
175
+ ```
176
+
177
+ ### Dependencies
178
+ - `torch` — PyTorch backend
179
+ - `transformers` — Hugging Face (DETR object detection)
180
+ - `sentence-transformers` — MiniLM semantic embeddings
181
+ - `spacy` — NLP noun extraction
182
+ - `Pillow` — Image processing
183
+
184
+ ---
185
+
186
+ ## Project Structure
187
+ ```
188
+ VTAS_Evaluation_Metric/
189
+ ├── src/
190
+ │ ├── vtas.py # Core VTAS evaluator class
191
+ │ ├── visual_grounding.py # DETR-based object detection module
192
+ │ ├── linguistic_extraction.py # spaCy-based noun extraction module
193
+ │ └── semantic_bridge.py # MiniLM-based semantic similarity module
194
+ ├── docs/
195
+ │ └── architecture.md # Full architectural specification & math
196
+ ├── requirements.txt
197
+ ├── .gitignore
198
+ └── README.md
199
+ ```
200
+
201
+ ---
202
+
203
+ ## Known Limitations & Future Work
204
+
205
+ VTAS v1 is a strong first step, but we believe in transparent research. The following are **known architectural limitations** that we have identified through extensive evaluation and are actively working to resolve.
206
+
207
+ ### Limitation 1: Semantic Collapse in the Semantic Bridge
208
+
209
+ **Observed in:** Motorcycle example (Image #78, VTAS = 1.00)
210
+
211
+ ![Semantic Collapse Example](assets/limitation_semantic_collapse.png)
212
+
213
+ **The Problem:** The caption reads *"the man is riding a motorcycle on the road with people watching."* DETR detected only `{'person', 'motorcycle'}`. The Semantic Bridge matched both `"man" → person (0.70)` and `"people" → person (0.76)`. On the surface, this appears correct — VTAS gave a perfect 1.00. But look closely:
214
+
215
+ * `"man"` refers to the **rider** (the main subject).
216
+ * `"people"` refers to the **spectators** in the background (secondary subjects).
217
+
218
+ These are two semantically distinct groups, but both collapsed onto the **single DETR label `"person"`**. VTAS treated them as the same entity. This is a fundamental limitation:
219
+
220
+ > **VTAS v1 operates on unordered sets of labels.** It has no concept of *which* person is being referenced. If DETR detects one `"person"`, any number of human-related nouns ("man", "woman", "people", "boy") in the caption will all match it, even if they refer to different individuals. This inflates both Precision and Recall.
221
+
222
+ **Root Cause:** DETR returns *unique category labels*, not instance-level detections. Even if DETR internally draws 5 bounding boxes for 5 people, the VTAS pipeline reduces them to the single label `"person"`.
223
+
224
+ ### Limitation 2: Scene-Level Descriptions Penalized by Object-Level Recall
225
+
226
+ **Observed in:** Living room example (Image #0, VTAS = 0.00)
227
+
228
+ ![Scene Penalty Example](assets/limitation_scene_penalty.png)
229
+
230
+ **The Problem:** The caption reads *"the living room has a yellow wall and a black fireplace."* DETR detected 9 objects: `{'clock', 'tv', 'dining table', 'refrigerator', 'person', 'vase', 'potted plant', 'bottle', 'chair'}`. The caption mentioned none of these individual items — it described the **collective environment** instead. As a result:
231
+
232
+ * **Object Precision = 0.50** — `"fireplace"` was rescued by CLIP (Tier 2), but `"wall"` was marked as a hallucination.
233
+ * **Visual Recall = 0.00** — The caption mentioned 0 out of 9 detected objects.
234
+ * **VTAS (F1) = 0.00** — Because F1 is a harmonic mean, if either P or R is 0, the entire score collapses.
235
+
236
+ This is arguably the **most important limitation**: a VLM that writes *"the kitchen is clean and ready for us to use"* receives a VTAS of 0.00, even though that caption is a perfectly valid, human-like description of a kitchen. The model correctly identified the scene but chose to describe it holistically rather than itemizing the oven, refrigerator, sink, cups, and bowls that DETR found.
237
+
238
+ > **VTAS v1 is an object-listing metric, not a scene-understanding metric.** It heavily rewards captions that enumerate discrete objects ("there is a clock, a TV, a chair, and a bottle") and penalizes captions that describe the scene as a whole ("this is a clean, well-lit living room"). This is by design — VTAS measures *grounded object alignment* — but it means that VTAS alone cannot evaluate the full quality of a caption.
239
+
240
+ **Root Cause:** The F-beta score requires non-zero Recall. When a caption contains zero object-level nouns that overlap with DETR's detections, Recall = 0, and the harmonic mean forces the entire score to 0 regardless of Precision.
241
+
242
+ ---
243
+
244
+ ### Future Work
245
+
246
+ Based on the limitations above, we propose the following research directions for VTAS v2:
247
+
248
+ | Direction | Addresses | Approach |
249
+ |---|---|---|
250
+ | **Instance-Aware Grounding** | Limitation 1 | Replace set-based matching with instance-level alignment. Use DETR's bounding box coordinates to distinguish "the rider" from "the spectators". Match each caption noun to a specific detection, preventing multiple nouns from collapsing onto one label. |
251
+ | **Scene-Level Recall** | Limitation 2 | Introduce a secondary recall signal that rewards captions for correctly identifying the *type* of scene (kitchen, bedroom, park) using CLIP or a dedicated scene classifier (e.g., Places365). A caption like "the kitchen is clean" would earn partial Recall credit for correctly classifying the environment. |
252
+ | **Attribute-Level Grounding** | New capability | Extend VTAS beyond nouns to verify adjectives ("red car", "large dog"). Use visual question-answering or attribute classifiers to check if described properties (color, size, material) match the image. |
253
+ | **Spatial Awareness** | New capability | Verify spatial claims ("the dog is *next to* the person", "the cup is *on* the table") against the physical coordinates of DETR bounding boxes. |
254
+ | **Dynamic Threshold Calibration** | Both | Replace the fixed similarity thresholds (MiniLM τ=0.65, CLIP τ=0.22) with dataset-adaptive thresholds calibrated on a held-out validation split, similar to ROC-AUC optimal threshold selection. |
255
+
256
+ ---
257
+
258
+ ## License
259
+
260
+ This project is licensed under the [MIT License](LICENSE).
261
+
262
+ ## Citation
263
+
264
+ If you use VTAS in your research, please cite:
265
+
266
+ ```bibtex
267
+ @software{prasad2026vtas,
268
+ author = {Prasad, Ritanshu},
269
+ title = {VTAS: Visual-Truth Alignment Score},
270
+ year = {2026},
271
+ url = {https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric},
272
+ license = {MIT}
273
+ }
274
+ ```
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,29 @@
1
+
2
+ from setuptools import setup, find_packages
3
+
4
+ setup(
5
+ name='vtas-metric',
6
+ version='1.3.0',
7
+ description='VTAS: Visual-Text Alignment Score for Evaluating Vision-Language Models',
8
+ long_description=open('README.md', encoding='utf-8').read(),
9
+ long_description_content_type='text/markdown',
10
+ author='Ritanshu Prasad',
11
+ author_email='ritanshu.prasad@example.com',
12
+ url='https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric',
13
+ package_dir={'': 'src'},
14
+ packages=find_packages(where='src'),
15
+ install_requires=[
16
+ 'torch',
17
+ 'transformers',
18
+ 'spacy',
19
+ 'sentence-transformers',
20
+ 'Pillow'
21
+ ],
22
+ classifiers=[
23
+ 'Programming Language :: Python :: 3',
24
+ 'License :: OSI Approved :: MIT License',
25
+ 'Operating System :: OS Independent',
26
+ ],
27
+ python_requires='>=3.8',
28
+ )
29
+
@@ -0,0 +1 @@
1
+ from .vtas import VTASEvaluator