vtas-metric 1.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- vtas_metric-1.3.0/LICENSE +21 -0
- vtas_metric-1.3.0/PKG-INFO +303 -0
- vtas_metric-1.3.0/README.md +274 -0
- vtas_metric-1.3.0/setup.cfg +4 -0
- vtas_metric-1.3.0/setup.py +29 -0
- vtas_metric-1.3.0/src/vtas/__init__.py +1 -0
- vtas_metric-1.3.0/src/vtas/clip_fallback.py +145 -0
- vtas_metric-1.3.0/src/vtas/generate_illustrations.py +174 -0
- vtas_metric-1.3.0/src/vtas/linguistic_extraction.py +184 -0
- vtas_metric-1.3.0/src/vtas/semantic_bridge.py +244 -0
- vtas_metric-1.3.0/src/vtas/visual_grounding.py +98 -0
- vtas_metric-1.3.0/src/vtas/vtas.py +371 -0
- vtas_metric-1.3.0/src/vtas_metric.egg-info/PKG-INFO +303 -0
- vtas_metric-1.3.0/src/vtas_metric.egg-info/SOURCES.txt +16 -0
- vtas_metric-1.3.0/src/vtas_metric.egg-info/dependency_links.txt +1 -0
- vtas_metric-1.3.0/src/vtas_metric.egg-info/requires.txt +5 -0
- vtas_metric-1.3.0/src/vtas_metric.egg-info/top_level.txt +1 -0
- vtas_metric-1.3.0/tests/test_logic.py +85 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Ritanshu Prasad
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,303 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: vtas-metric
|
|
3
|
+
Version: 1.3.0
|
|
4
|
+
Summary: VTAS: Visual-Text Alignment Score for Evaluating Vision-Language Models
|
|
5
|
+
Home-page: https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric
|
|
6
|
+
Author: Ritanshu Prasad
|
|
7
|
+
Author-email: ritanshu.prasad@example.com
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Requires-Python: >=3.8
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Requires-Dist: torch
|
|
15
|
+
Requires-Dist: transformers
|
|
16
|
+
Requires-Dist: spacy
|
|
17
|
+
Requires-Dist: sentence-transformers
|
|
18
|
+
Requires-Dist: Pillow
|
|
19
|
+
Dynamic: author
|
|
20
|
+
Dynamic: author-email
|
|
21
|
+
Dynamic: classifier
|
|
22
|
+
Dynamic: description
|
|
23
|
+
Dynamic: description-content-type
|
|
24
|
+
Dynamic: home-page
|
|
25
|
+
Dynamic: license-file
|
|
26
|
+
Dynamic: requires-dist
|
|
27
|
+
Dynamic: requires-python
|
|
28
|
+
Dynamic: summary
|
|
29
|
+
|
|
30
|
+
# VTAS — Visual-Truth Alignment Score
|
|
31
|
+
|
|
32
|
+
[](LICENSE)
|
|
33
|
+
[](https://www.python.org/)
|
|
34
|
+
|
|
35
|
+
**A white-box, reference-free evaluation metric for image captioning that grounds generated text directly against the visual content of the image.**
|
|
36
|
+
|
|
37
|
+
> *"Does the model see what it says it sees?"* — VTAS answers this question without needing human reference captions.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Why Does This Metric Exist?
|
|
42
|
+
|
|
43
|
+
Image captioning models generate natural language descriptions of images. To measure how "good" a generated caption is, the research community relies on a set of standard metrics. However, every single one of them shares a critical architectural flaw: **they are blind to the image.**
|
|
44
|
+
|
|
45
|
+
They compare the **generated text** against **human-written reference text**, but never verify whether the generated text actually describes what is physically present in the image. This creates three categories of failure:
|
|
46
|
+
|
|
47
|
+
### The Problem with Existing Metrics
|
|
48
|
+
|
|
49
|
+
#### 1. BLEU (Bilingual Evaluation Understudy)
|
|
50
|
+
BLEU measures n-gram precision — it counts how many word chunks in the generated caption appear in the human reference.
|
|
51
|
+
|
|
52
|
+
**The Flaw:** It is purely lexical. It has no understanding of meaning.
|
|
53
|
+
|
|
54
|
+
| Generated Caption | Reference Caption | BLEU-4 Score | Correct? |
|
|
55
|
+
|---|---|---|---|
|
|
56
|
+
| *"A man relaxes on a sofa"* | *"A person sits on a couch"* | **~0.0** (no 4-gram overlap) | ✅ Yes — semantically identical |
|
|
57
|
+
| *"A person sits on a couch near a dog"* | *"A person sits on a couch"* | **High** (strong overlap) | ❌ Maybe — was there actually a dog? BLEU doesn't check. |
|
|
58
|
+
|
|
59
|
+
> BLEU punishes correct captions that use synonyms, and rewards captions that copy reference words — even if the referenced objects are hallucinated.
|
|
60
|
+
|
|
61
|
+
#### 2. METEOR (Metric for Evaluation of Translation with Explicit ORdering)
|
|
62
|
+
METEOR improves on BLEU by adding stemming ("running" → "run") and synonym matching via external WordNet tables.
|
|
63
|
+
|
|
64
|
+
**The Flaw:** It still compares text-to-text. It relies on pre-built synonym tables that are finite, language-specific, and cannot cover domain-specific vocabulary. It never verifies against the image.
|
|
65
|
+
|
|
66
|
+
#### 3. CIDEr (Consensus-based Image Description Evaluation)
|
|
67
|
+
CIDEr uses TF-IDF weighting to reward captions that mention rare, image-specific words (e.g., "frisbee" scores higher than "the").
|
|
68
|
+
|
|
69
|
+
**The Flaw:** It is entirely dependent on the consensus of human reference captions. If the reference captions are biased, incomplete, or inconsistent, CIDEr scores become unreliable. It cannot detect hallucinations if the hallucinated word happens to appear in a reference.
|
|
70
|
+
|
|
71
|
+
#### 4. CLIPScore
|
|
72
|
+
CLIPScore is the current state-of-the-art reference-free metric. It embeds both the image and the text into a shared vector space using CLIP and computes cosine similarity.
|
|
73
|
+
|
|
74
|
+
**The Flaw:** It is a **black box**. It outputs a single similarity score but provides zero interpretability. If the score is low, there is no way to determine whether the model missed an object, hallucinated one, got a color wrong, or made a spatial error.
|
|
75
|
+
|
|
76
|
+
#### 5. CHAIR (Caption Hallucination Assessment with Image Relevance)
|
|
77
|
+
CHAIR was specifically designed to detect object hallucinations by comparing generated nouns against ground-truth object annotations.
|
|
78
|
+
|
|
79
|
+
**The Flaw:** It requires **expensive, human-annotated object labels** for every image. It cannot run on raw, unannotated images, making it impractical for real-world deployment at scale.
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
### The Gap
|
|
84
|
+
|
|
85
|
+
| Capability | BLEU | METEOR | CIDEr | CLIPScore | CHAIR |
|
|
86
|
+
|---|:---:|:---:|:---:|:---:|:---:|
|
|
87
|
+
| Reference-Free | ❌ | ❌ | ❌ | ✅ | ❌ |
|
|
88
|
+
| Detects Hallucinations | ❌ | ❌ | ❌ | Weak | ✅ |
|
|
89
|
+
| Handles Synonyms | ❌ | Partial | ❌ | ✅ | ❌ |
|
|
90
|
+
| White-Box / Interpretable | ✅ | ✅ | ✅ | ❌ | ✅ |
|
|
91
|
+
| Fully Automated (No Human Labels) | ✅ | ✅ | ✅ | ✅ | ❌ |
|
|
92
|
+
|
|
93
|
+
**No single existing metric is simultaneously: reference-free, hallucination-aware, synonym-robust, interpretable, AND fully automated.**
|
|
94
|
+
|
|
95
|
+
VTAS is engineered to fill that gap.
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## What is VTAS?
|
|
100
|
+
|
|
101
|
+
VTAS (Visual-Truth Alignment Score) is a **Two-Tier Compound AI System** that evaluates image captions against physical reality:
|
|
102
|
+
|
|
103
|
+
1. **Tier 1 (Object Grounding):** An Object Detector (DETR) extracts physical objects. An NLP Parser (spaCy) extracts caption nouns. A Semantic Bridge (MiniLM, phrase-based) resolves synonyms.
|
|
104
|
+
2. **Tier 2 (Scene/Context Grounding):** Unmatched nouns (e.g., "bedroom", "outside", or undetected objects) are verified globally against the image using CLIP zero-shot classification.
|
|
105
|
+
|
|
106
|
+
The final **VTAS Score (v1.2)** uses the mathematically rigorous **F-beta Score** to balance **Object Precision** (absence of hallucinations) and **Visual Recall** (detecting visible objects) into a single metric [0, 1].
|
|
107
|
+
|
|
108
|
+
### VTAS Properties
|
|
109
|
+
| Property | Value |
|
|
110
|
+
|---|---|
|
|
111
|
+
| Reference-Free | ✅ No human captions needed |
|
|
112
|
+
| Detects Hallucinations | ✅ Explicit precision penalty for invented objects |
|
|
113
|
+
| Handles Synonyms | ✅ Phrase-based semantic embeddings |
|
|
114
|
+
| Scene Aware | ✅ Tier 2 CLIP validates complex environments |
|
|
115
|
+
| Interpretable | ✅ Pinpoints exactly which word failed |
|
|
116
|
+
|
|
117
|
+
---
|
|
118
|
+
|
|
119
|
+
## Pictorial Demonstrations (VTAS in Action)
|
|
120
|
+
|
|
121
|
+
The following infographics were generated automatically during a VTAS evaluation run. They visually demonstrate how VTAS solves the flaws of legacy metrics.
|
|
122
|
+
|
|
123
|
+
### Example 1: Perfect Alignment & Synonym Bridging
|
|
124
|
+
Unlike BLEU, VTAS physically draws bounding boxes to verify reality. Even if the wording differs from human annotators, VTAS bridges the vocabulary gap using phrase-based semantic matching.
|
|
125
|
+

|
|
126
|
+
|
|
127
|
+
### Example 2: Scene Context & The "Recall Penalty"
|
|
128
|
+
Traditional metrics like CHAIR crash without human bounding boxes. VTAS operates autonomously.
|
|
129
|
+
* **Tier 2 Rescue:** Notice how words like "fireplace" (which DETR missed) are rescued by the CLIP context module!
|
|
130
|
+
* **Visual Recall Penalty:** This caption scored an F1 of 0.00 not because it hallucinated wildly, but because it *described the scene* instead of listing the physical objects detected by DETR.
|
|
131
|
+

|
|
132
|
+
|
|
133
|
+
---
|
|
134
|
+
|
|
135
|
+
## Quick Start
|
|
136
|
+
|
|
137
|
+
```python
|
|
138
|
+
from vtas import VTASEvaluator
|
|
139
|
+
|
|
140
|
+
evaluator = VTASEvaluator()
|
|
141
|
+
|
|
142
|
+
result = evaluator.score(
|
|
143
|
+
image_path="path/to/image.jpg",
|
|
144
|
+
caption="A man throws a frisbee to his dog in the park"
|
|
145
|
+
)
|
|
146
|
+
|
|
147
|
+
print(f"VTAS Score (F1): {result['vtas_score']:.4f}")
|
|
148
|
+
print(f"Object Precision: {result['precision']:.4f}")
|
|
149
|
+
print(f"Visual Recall: {result['recall']:.4f}")
|
|
150
|
+
print(f"Matched Objects: {result['matched']}")
|
|
151
|
+
print(f"Hallucinated Objects: {result['hallucinated']}")
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
---
|
|
155
|
+
|
|
156
|
+
## Examples
|
|
157
|
+
|
|
158
|
+
### Example 1: Perfect Caption
|
|
159
|
+
**Image:** A park scene with a person, a dog, and a frisbee.
|
|
160
|
+
|
|
161
|
+
| Component | Output |
|
|
162
|
+
|---|---|
|
|
163
|
+
| **DETR detects** | `{'person', 'dog', 'frisbee'}` |
|
|
164
|
+
| **Caption says** | *"A man throws a frisbee to his dog"* → `{'man', 'frisbee', 'dog'}` |
|
|
165
|
+
| **Semantic Bridge** | man↔person (0.82 ✅), frisbee↔frisbee (1.0 ✅), dog↔dog (1.0 ✅) |
|
|
166
|
+
| **Visual Recall** | 3/3 = 1.0 |
|
|
167
|
+
| **Hallucination Rate** | 0/3 = 0.0 |
|
|
168
|
+
| **VTAS** | **1.0** ✅ |
|
|
169
|
+
|
|
170
|
+
### Example 2: Synonym Robustness (BLEU would fail here)
|
|
171
|
+
**Image:** A living room with a person, a couch, and a TV.
|
|
172
|
+
|
|
173
|
+
| Component | Output |
|
|
174
|
+
|---|---|
|
|
175
|
+
| **DETR detects** | `{'person', 'couch', 'tv'}` |
|
|
176
|
+
| **Caption says** | *"A man relaxes on a sofa watching television"* → `{'man', 'sofa', 'television'}` |
|
|
177
|
+
| **Semantic Bridge** | man↔person (0.82 ✅), sofa↔couch (0.89 ✅), television↔tv (0.91 ✅) |
|
|
178
|
+
| **Visual Recall** | 3/3 = 1.0 |
|
|
179
|
+
| **Hallucination Rate** | 0/3 = 0.0 |
|
|
180
|
+
| **VTAS** | **1.0** ✅ |
|
|
181
|
+
|
|
182
|
+
> BLEU-4 would score this near zero because no exact 4-grams match. VTAS correctly identifies it as a perfect caption.
|
|
183
|
+
|
|
184
|
+
### Example 3: Hallucination Detection
|
|
185
|
+
**Image:** A boy holding an umbrella standing near a cow.
|
|
186
|
+
|
|
187
|
+
| Component | Output |
|
|
188
|
+
|---|---|
|
|
189
|
+
| **DETR detects** | `{'person', 'umbrella', 'cow'}` |
|
|
190
|
+
| **Caption says** | *"A group of people with umbrellas and dogs"* → `{'group', 'people', 'umbrellas', 'dogs'}` |
|
|
191
|
+
| **Semantic Bridge** | people↔person (0.85 ✅), umbrellas↔umbrella (0.97 ✅), group↔? (❌), dogs↔cow (0.45 ❌) |
|
|
192
|
+
| **Visual Recall** | 2/3 = 0.67 (missed 'cow') |
|
|
193
|
+
| **Hallucination Rate** | 2/4 = 0.50 (hallucinated 'group', 'dogs') |
|
|
194
|
+
| **VTAS** | **0.33** ❌ — Correctly penalized |
|
|
195
|
+
|
|
196
|
+
> This is a real hallucination observed during our BLIP zero-shot baseline evaluation. Traditional metrics may still score this reasonably if the reference caption mentions "people." VTAS catches it because DETR confirms there is no "group" or "dogs" in the image.
|
|
197
|
+
|
|
198
|
+
---
|
|
199
|
+
|
|
200
|
+
## Installation
|
|
201
|
+
|
|
202
|
+
```bash
|
|
203
|
+
pip install -r requirements.txt
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
### Dependencies
|
|
207
|
+
- `torch` — PyTorch backend
|
|
208
|
+
- `transformers` — Hugging Face (DETR object detection)
|
|
209
|
+
- `sentence-transformers` — MiniLM semantic embeddings
|
|
210
|
+
- `spacy` — NLP noun extraction
|
|
211
|
+
- `Pillow` — Image processing
|
|
212
|
+
|
|
213
|
+
---
|
|
214
|
+
|
|
215
|
+
## Project Structure
|
|
216
|
+
```
|
|
217
|
+
VTAS_Evaluation_Metric/
|
|
218
|
+
├── src/
|
|
219
|
+
│ ├── vtas.py # Core VTAS evaluator class
|
|
220
|
+
│ ├── visual_grounding.py # DETR-based object detection module
|
|
221
|
+
│ ├── linguistic_extraction.py # spaCy-based noun extraction module
|
|
222
|
+
│ └── semantic_bridge.py # MiniLM-based semantic similarity module
|
|
223
|
+
├── docs/
|
|
224
|
+
│ └── architecture.md # Full architectural specification & math
|
|
225
|
+
├── requirements.txt
|
|
226
|
+
├── .gitignore
|
|
227
|
+
└── README.md
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
## Known Limitations & Future Work
|
|
233
|
+
|
|
234
|
+
VTAS v1 is a strong first step, but we believe in transparent research. The following are **known architectural limitations** that we have identified through extensive evaluation and are actively working to resolve.
|
|
235
|
+
|
|
236
|
+
### Limitation 1: Semantic Collapse in the Semantic Bridge
|
|
237
|
+
|
|
238
|
+
**Observed in:** Motorcycle example (Image #78, VTAS = 1.00)
|
|
239
|
+
|
|
240
|
+

|
|
241
|
+
|
|
242
|
+
**The Problem:** The caption reads *"the man is riding a motorcycle on the road with people watching."* DETR detected only `{'person', 'motorcycle'}`. The Semantic Bridge matched both `"man" → person (0.70)` and `"people" → person (0.76)`. On the surface, this appears correct — VTAS gave a perfect 1.00. But look closely:
|
|
243
|
+
|
|
244
|
+
* `"man"` refers to the **rider** (the main subject).
|
|
245
|
+
* `"people"` refers to the **spectators** in the background (secondary subjects).
|
|
246
|
+
|
|
247
|
+
These are two semantically distinct groups, but both collapsed onto the **single DETR label `"person"`**. VTAS treated them as the same entity. This is a fundamental limitation:
|
|
248
|
+
|
|
249
|
+
> **VTAS v1 operates on unordered sets of labels.** It has no concept of *which* person is being referenced. If DETR detects one `"person"`, any number of human-related nouns ("man", "woman", "people", "boy") in the caption will all match it, even if they refer to different individuals. This inflates both Precision and Recall.
|
|
250
|
+
|
|
251
|
+
**Root Cause:** DETR returns *unique category labels*, not instance-level detections. Even if DETR internally draws 5 bounding boxes for 5 people, the VTAS pipeline reduces them to the single label `"person"`.
|
|
252
|
+
|
|
253
|
+
### Limitation 2: Scene-Level Descriptions Penalized by Object-Level Recall
|
|
254
|
+
|
|
255
|
+
**Observed in:** Living room example (Image #0, VTAS = 0.00)
|
|
256
|
+
|
|
257
|
+

|
|
258
|
+
|
|
259
|
+
**The Problem:** The caption reads *"the living room has a yellow wall and a black fireplace."* DETR detected 9 objects: `{'clock', 'tv', 'dining table', 'refrigerator', 'person', 'vase', 'potted plant', 'bottle', 'chair'}`. The caption mentioned none of these individual items — it described the **collective environment** instead. As a result:
|
|
260
|
+
|
|
261
|
+
* **Object Precision = 0.50** — `"fireplace"` was rescued by CLIP (Tier 2), but `"wall"` was marked as a hallucination.
|
|
262
|
+
* **Visual Recall = 0.00** — The caption mentioned 0 out of 9 detected objects.
|
|
263
|
+
* **VTAS (F1) = 0.00** — Because F1 is a harmonic mean, if either P or R is 0, the entire score collapses.
|
|
264
|
+
|
|
265
|
+
This is arguably the **most important limitation**: a VLM that writes *"the kitchen is clean and ready for us to use"* receives a VTAS of 0.00, even though that caption is a perfectly valid, human-like description of a kitchen. The model correctly identified the scene but chose to describe it holistically rather than itemizing the oven, refrigerator, sink, cups, and bowls that DETR found.
|
|
266
|
+
|
|
267
|
+
> **VTAS v1 is an object-listing metric, not a scene-understanding metric.** It heavily rewards captions that enumerate discrete objects ("there is a clock, a TV, a chair, and a bottle") and penalizes captions that describe the scene as a whole ("this is a clean, well-lit living room"). This is by design — VTAS measures *grounded object alignment* — but it means that VTAS alone cannot evaluate the full quality of a caption.
|
|
268
|
+
|
|
269
|
+
**Root Cause:** The F-beta score requires non-zero Recall. When a caption contains zero object-level nouns that overlap with DETR's detections, Recall = 0, and the harmonic mean forces the entire score to 0 regardless of Precision.
|
|
270
|
+
|
|
271
|
+
---
|
|
272
|
+
|
|
273
|
+
### Future Work
|
|
274
|
+
|
|
275
|
+
Based on the limitations above, we propose the following research directions for VTAS v2:
|
|
276
|
+
|
|
277
|
+
| Direction | Addresses | Approach |
|
|
278
|
+
|---|---|---|
|
|
279
|
+
| **Instance-Aware Grounding** | Limitation 1 | Replace set-based matching with instance-level alignment. Use DETR's bounding box coordinates to distinguish "the rider" from "the spectators". Match each caption noun to a specific detection, preventing multiple nouns from collapsing onto one label. |
|
|
280
|
+
| **Scene-Level Recall** | Limitation 2 | Introduce a secondary recall signal that rewards captions for correctly identifying the *type* of scene (kitchen, bedroom, park) using CLIP or a dedicated scene classifier (e.g., Places365). A caption like "the kitchen is clean" would earn partial Recall credit for correctly classifying the environment. |
|
|
281
|
+
| **Attribute-Level Grounding** | New capability | Extend VTAS beyond nouns to verify adjectives ("red car", "large dog"). Use visual question-answering or attribute classifiers to check if described properties (color, size, material) match the image. |
|
|
282
|
+
| **Spatial Awareness** | New capability | Verify spatial claims ("the dog is *next to* the person", "the cup is *on* the table") against the physical coordinates of DETR bounding boxes. |
|
|
283
|
+
| **Dynamic Threshold Calibration** | Both | Replace the fixed similarity thresholds (MiniLM τ=0.65, CLIP τ=0.22) with dataset-adaptive thresholds calibrated on a held-out validation split, similar to ROC-AUC optimal threshold selection. |
|
|
284
|
+
|
|
285
|
+
---
|
|
286
|
+
|
|
287
|
+
## License
|
|
288
|
+
|
|
289
|
+
This project is licensed under the [MIT License](LICENSE).
|
|
290
|
+
|
|
291
|
+
## Citation
|
|
292
|
+
|
|
293
|
+
If you use VTAS in your research, please cite:
|
|
294
|
+
|
|
295
|
+
```bibtex
|
|
296
|
+
@software{prasad2026vtas,
|
|
297
|
+
author = {Prasad, Ritanshu},
|
|
298
|
+
title = {VTAS: Visual-Truth Alignment Score},
|
|
299
|
+
year = {2026},
|
|
300
|
+
url = {https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric},
|
|
301
|
+
license = {MIT}
|
|
302
|
+
}
|
|
303
|
+
```
|
|
@@ -0,0 +1,274 @@
|
|
|
1
|
+
# VTAS — Visual-Truth Alignment Score
|
|
2
|
+
|
|
3
|
+
[](LICENSE)
|
|
4
|
+
[](https://www.python.org/)
|
|
5
|
+
|
|
6
|
+
**A white-box, reference-free evaluation metric for image captioning that grounds generated text directly against the visual content of the image.**
|
|
7
|
+
|
|
8
|
+
> *"Does the model see what it says it sees?"* — VTAS answers this question without needing human reference captions.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Why Does This Metric Exist?
|
|
13
|
+
|
|
14
|
+
Image captioning models generate natural language descriptions of images. To measure how "good" a generated caption is, the research community relies on a set of standard metrics. However, every single one of them shares a critical architectural flaw: **they are blind to the image.**
|
|
15
|
+
|
|
16
|
+
They compare the **generated text** against **human-written reference text**, but never verify whether the generated text actually describes what is physically present in the image. This creates three categories of failure:
|
|
17
|
+
|
|
18
|
+
### The Problem with Existing Metrics
|
|
19
|
+
|
|
20
|
+
#### 1. BLEU (Bilingual Evaluation Understudy)
|
|
21
|
+
BLEU measures n-gram precision — it counts how many word chunks in the generated caption appear in the human reference.
|
|
22
|
+
|
|
23
|
+
**The Flaw:** It is purely lexical. It has no understanding of meaning.
|
|
24
|
+
|
|
25
|
+
| Generated Caption | Reference Caption | BLEU-4 Score | Correct? |
|
|
26
|
+
|---|---|---|---|
|
|
27
|
+
| *"A man relaxes on a sofa"* | *"A person sits on a couch"* | **~0.0** (no 4-gram overlap) | ✅ Yes — semantically identical |
|
|
28
|
+
| *"A person sits on a couch near a dog"* | *"A person sits on a couch"* | **High** (strong overlap) | ❌ Maybe — was there actually a dog? BLEU doesn't check. |
|
|
29
|
+
|
|
30
|
+
> BLEU punishes correct captions that use synonyms, and rewards captions that copy reference words — even if the referenced objects are hallucinated.
|
|
31
|
+
|
|
32
|
+
#### 2. METEOR (Metric for Evaluation of Translation with Explicit ORdering)
|
|
33
|
+
METEOR improves on BLEU by adding stemming ("running" → "run") and synonym matching via external WordNet tables.
|
|
34
|
+
|
|
35
|
+
**The Flaw:** It still compares text-to-text. It relies on pre-built synonym tables that are finite, language-specific, and cannot cover domain-specific vocabulary. It never verifies against the image.
|
|
36
|
+
|
|
37
|
+
#### 3. CIDEr (Consensus-based Image Description Evaluation)
|
|
38
|
+
CIDEr uses TF-IDF weighting to reward captions that mention rare, image-specific words (e.g., "frisbee" scores higher than "the").
|
|
39
|
+
|
|
40
|
+
**The Flaw:** It is entirely dependent on the consensus of human reference captions. If the reference captions are biased, incomplete, or inconsistent, CIDEr scores become unreliable. It cannot detect hallucinations if the hallucinated word happens to appear in a reference.
|
|
41
|
+
|
|
42
|
+
#### 4. CLIPScore
|
|
43
|
+
CLIPScore is the current state-of-the-art reference-free metric. It embeds both the image and the text into a shared vector space using CLIP and computes cosine similarity.
|
|
44
|
+
|
|
45
|
+
**The Flaw:** It is a **black box**. It outputs a single similarity score but provides zero interpretability. If the score is low, there is no way to determine whether the model missed an object, hallucinated one, got a color wrong, or made a spatial error.
|
|
46
|
+
|
|
47
|
+
#### 5. CHAIR (Caption Hallucination Assessment with Image Relevance)
|
|
48
|
+
CHAIR was specifically designed to detect object hallucinations by comparing generated nouns against ground-truth object annotations.
|
|
49
|
+
|
|
50
|
+
**The Flaw:** It requires **expensive, human-annotated object labels** for every image. It cannot run on raw, unannotated images, making it impractical for real-world deployment at scale.
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
### The Gap
|
|
55
|
+
|
|
56
|
+
| Capability | BLEU | METEOR | CIDEr | CLIPScore | CHAIR |
|
|
57
|
+
|---|:---:|:---:|:---:|:---:|:---:|
|
|
58
|
+
| Reference-Free | ❌ | ❌ | ❌ | ✅ | ❌ |
|
|
59
|
+
| Detects Hallucinations | ❌ | ❌ | ❌ | Weak | ✅ |
|
|
60
|
+
| Handles Synonyms | ❌ | Partial | ❌ | ✅ | ❌ |
|
|
61
|
+
| White-Box / Interpretable | ✅ | ✅ | ✅ | ❌ | ✅ |
|
|
62
|
+
| Fully Automated (No Human Labels) | ✅ | ✅ | ✅ | ✅ | ❌ |
|
|
63
|
+
|
|
64
|
+
**No single existing metric is simultaneously: reference-free, hallucination-aware, synonym-robust, interpretable, AND fully automated.**
|
|
65
|
+
|
|
66
|
+
VTAS is engineered to fill that gap.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## What is VTAS?
|
|
71
|
+
|
|
72
|
+
VTAS (Visual-Truth Alignment Score) is a **Two-Tier Compound AI System** that evaluates image captions against physical reality:
|
|
73
|
+
|
|
74
|
+
1. **Tier 1 (Object Grounding):** An Object Detector (DETR) extracts physical objects. An NLP Parser (spaCy) extracts caption nouns. A Semantic Bridge (MiniLM, phrase-based) resolves synonyms.
|
|
75
|
+
2. **Tier 2 (Scene/Context Grounding):** Unmatched nouns (e.g., "bedroom", "outside", or undetected objects) are verified globally against the image using CLIP zero-shot classification.
|
|
76
|
+
|
|
77
|
+
The final **VTAS Score (v1.2)** uses the mathematically rigorous **F-beta Score** to balance **Object Precision** (absence of hallucinations) and **Visual Recall** (detecting visible objects) into a single metric [0, 1].
|
|
78
|
+
|
|
79
|
+
### VTAS Properties
|
|
80
|
+
| Property | Value |
|
|
81
|
+
|---|---|
|
|
82
|
+
| Reference-Free | ✅ No human captions needed |
|
|
83
|
+
| Detects Hallucinations | ✅ Explicit precision penalty for invented objects |
|
|
84
|
+
| Handles Synonyms | ✅ Phrase-based semantic embeddings |
|
|
85
|
+
| Scene Aware | ✅ Tier 2 CLIP validates complex environments |
|
|
86
|
+
| Interpretable | ✅ Pinpoints exactly which word failed |
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## Pictorial Demonstrations (VTAS in Action)
|
|
91
|
+
|
|
92
|
+
The following infographics were generated automatically during a VTAS evaluation run. They visually demonstrate how VTAS solves the flaws of legacy metrics.
|
|
93
|
+
|
|
94
|
+
### Example 1: Perfect Alignment & Synonym Bridging
|
|
95
|
+
Unlike BLEU, VTAS physically draws bounding boxes to verify reality. Even if the wording differs from human annotators, VTAS bridges the vocabulary gap using phrase-based semantic matching.
|
|
96
|
+

|
|
97
|
+
|
|
98
|
+
### Example 2: Scene Context & The "Recall Penalty"
|
|
99
|
+
Traditional metrics like CHAIR crash without human bounding boxes. VTAS operates autonomously.
|
|
100
|
+
* **Tier 2 Rescue:** Notice how words like "fireplace" (which DETR missed) are rescued by the CLIP context module!
|
|
101
|
+
* **Visual Recall Penalty:** This caption scored an F1 of 0.00 not because it hallucinated wildly, but because it *described the scene* instead of listing the physical objects detected by DETR.
|
|
102
|
+

|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## Quick Start
|
|
107
|
+
|
|
108
|
+
```python
|
|
109
|
+
from vtas import VTASEvaluator
|
|
110
|
+
|
|
111
|
+
evaluator = VTASEvaluator()
|
|
112
|
+
|
|
113
|
+
result = evaluator.score(
|
|
114
|
+
image_path="path/to/image.jpg",
|
|
115
|
+
caption="A man throws a frisbee to his dog in the park"
|
|
116
|
+
)
|
|
117
|
+
|
|
118
|
+
print(f"VTAS Score (F1): {result['vtas_score']:.4f}")
|
|
119
|
+
print(f"Object Precision: {result['precision']:.4f}")
|
|
120
|
+
print(f"Visual Recall: {result['recall']:.4f}")
|
|
121
|
+
print(f"Matched Objects: {result['matched']}")
|
|
122
|
+
print(f"Hallucinated Objects: {result['hallucinated']}")
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
## Examples
|
|
128
|
+
|
|
129
|
+
### Example 1: Perfect Caption
|
|
130
|
+
**Image:** A park scene with a person, a dog, and a frisbee.
|
|
131
|
+
|
|
132
|
+
| Component | Output |
|
|
133
|
+
|---|---|
|
|
134
|
+
| **DETR detects** | `{'person', 'dog', 'frisbee'}` |
|
|
135
|
+
| **Caption says** | *"A man throws a frisbee to his dog"* → `{'man', 'frisbee', 'dog'}` |
|
|
136
|
+
| **Semantic Bridge** | man↔person (0.82 ✅), frisbee↔frisbee (1.0 ✅), dog↔dog (1.0 ✅) |
|
|
137
|
+
| **Visual Recall** | 3/3 = 1.0 |
|
|
138
|
+
| **Hallucination Rate** | 0/3 = 0.0 |
|
|
139
|
+
| **VTAS** | **1.0** ✅ |
|
|
140
|
+
|
|
141
|
+
### Example 2: Synonym Robustness (BLEU would fail here)
|
|
142
|
+
**Image:** A living room with a person, a couch, and a TV.
|
|
143
|
+
|
|
144
|
+
| Component | Output |
|
|
145
|
+
|---|---|
|
|
146
|
+
| **DETR detects** | `{'person', 'couch', 'tv'}` |
|
|
147
|
+
| **Caption says** | *"A man relaxes on a sofa watching television"* → `{'man', 'sofa', 'television'}` |
|
|
148
|
+
| **Semantic Bridge** | man↔person (0.82 ✅), sofa↔couch (0.89 ✅), television↔tv (0.91 ✅) |
|
|
149
|
+
| **Visual Recall** | 3/3 = 1.0 |
|
|
150
|
+
| **Hallucination Rate** | 0/3 = 0.0 |
|
|
151
|
+
| **VTAS** | **1.0** ✅ |
|
|
152
|
+
|
|
153
|
+
> BLEU-4 would score this near zero because no exact 4-grams match. VTAS correctly identifies it as a perfect caption.
|
|
154
|
+
|
|
155
|
+
### Example 3: Hallucination Detection
|
|
156
|
+
**Image:** A boy holding an umbrella standing near a cow.
|
|
157
|
+
|
|
158
|
+
| Component | Output |
|
|
159
|
+
|---|---|
|
|
160
|
+
| **DETR detects** | `{'person', 'umbrella', 'cow'}` |
|
|
161
|
+
| **Caption says** | *"A group of people with umbrellas and dogs"* → `{'group', 'people', 'umbrellas', 'dogs'}` |
|
|
162
|
+
| **Semantic Bridge** | people↔person (0.85 ✅), umbrellas↔umbrella (0.97 ✅), group↔? (❌), dogs↔cow (0.45 ❌) |
|
|
163
|
+
| **Visual Recall** | 2/3 = 0.67 (missed 'cow') |
|
|
164
|
+
| **Hallucination Rate** | 2/4 = 0.50 (hallucinated 'group', 'dogs') |
|
|
165
|
+
| **VTAS** | **0.33** ❌ — Correctly penalized |
|
|
166
|
+
|
|
167
|
+
> This is a real hallucination observed during our BLIP zero-shot baseline evaluation. Traditional metrics may still score this reasonably if the reference caption mentions "people." VTAS catches it because DETR confirms there is no "group" or "dogs" in the image.
|
|
168
|
+
|
|
169
|
+
---
|
|
170
|
+
|
|
171
|
+
## Installation
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
pip install -r requirements.txt
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### Dependencies
|
|
178
|
+
- `torch` — PyTorch backend
|
|
179
|
+
- `transformers` — Hugging Face (DETR object detection)
|
|
180
|
+
- `sentence-transformers` — MiniLM semantic embeddings
|
|
181
|
+
- `spacy` — NLP noun extraction
|
|
182
|
+
- `Pillow` — Image processing
|
|
183
|
+
|
|
184
|
+
---
|
|
185
|
+
|
|
186
|
+
## Project Structure
|
|
187
|
+
```
|
|
188
|
+
VTAS_Evaluation_Metric/
|
|
189
|
+
├── src/
|
|
190
|
+
│ ├── vtas.py # Core VTAS evaluator class
|
|
191
|
+
│ ├── visual_grounding.py # DETR-based object detection module
|
|
192
|
+
│ ├── linguistic_extraction.py # spaCy-based noun extraction module
|
|
193
|
+
│ └── semantic_bridge.py # MiniLM-based semantic similarity module
|
|
194
|
+
├── docs/
|
|
195
|
+
│ └── architecture.md # Full architectural specification & math
|
|
196
|
+
├── requirements.txt
|
|
197
|
+
├── .gitignore
|
|
198
|
+
└── README.md
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
---
|
|
202
|
+
|
|
203
|
+
## Known Limitations & Future Work
|
|
204
|
+
|
|
205
|
+
VTAS v1 is a strong first step, but we believe in transparent research. The following are **known architectural limitations** that we have identified through extensive evaluation and are actively working to resolve.
|
|
206
|
+
|
|
207
|
+
### Limitation 1: Semantic Collapse in the Semantic Bridge
|
|
208
|
+
|
|
209
|
+
**Observed in:** Motorcycle example (Image #78, VTAS = 1.00)
|
|
210
|
+
|
|
211
|
+

|
|
212
|
+
|
|
213
|
+
**The Problem:** The caption reads *"the man is riding a motorcycle on the road with people watching."* DETR detected only `{'person', 'motorcycle'}`. The Semantic Bridge matched both `"man" → person (0.70)` and `"people" → person (0.76)`. On the surface, this appears correct — VTAS gave a perfect 1.00. But look closely:
|
|
214
|
+
|
|
215
|
+
* `"man"` refers to the **rider** (the main subject).
|
|
216
|
+
* `"people"` refers to the **spectators** in the background (secondary subjects).
|
|
217
|
+
|
|
218
|
+
These are two semantically distinct groups, but both collapsed onto the **single DETR label `"person"`**. VTAS treated them as the same entity. This is a fundamental limitation:
|
|
219
|
+
|
|
220
|
+
> **VTAS v1 operates on unordered sets of labels.** It has no concept of *which* person is being referenced. If DETR detects one `"person"`, any number of human-related nouns ("man", "woman", "people", "boy") in the caption will all match it, even if they refer to different individuals. This inflates both Precision and Recall.
|
|
221
|
+
|
|
222
|
+
**Root Cause:** DETR returns *unique category labels*, not instance-level detections. Even if DETR internally draws 5 bounding boxes for 5 people, the VTAS pipeline reduces them to the single label `"person"`.
|
|
223
|
+
|
|
224
|
+
### Limitation 2: Scene-Level Descriptions Penalized by Object-Level Recall
|
|
225
|
+
|
|
226
|
+
**Observed in:** Living room example (Image #0, VTAS = 0.00)
|
|
227
|
+
|
|
228
|
+

|
|
229
|
+
|
|
230
|
+
**The Problem:** The caption reads *"the living room has a yellow wall and a black fireplace."* DETR detected 9 objects: `{'clock', 'tv', 'dining table', 'refrigerator', 'person', 'vase', 'potted plant', 'bottle', 'chair'}`. The caption mentioned none of these individual items — it described the **collective environment** instead. As a result:
|
|
231
|
+
|
|
232
|
+
* **Object Precision = 0.50** — `"fireplace"` was rescued by CLIP (Tier 2), but `"wall"` was marked as a hallucination.
|
|
233
|
+
* **Visual Recall = 0.00** — The caption mentioned 0 out of 9 detected objects.
|
|
234
|
+
* **VTAS (F1) = 0.00** — Because F1 is a harmonic mean, if either P or R is 0, the entire score collapses.
|
|
235
|
+
|
|
236
|
+
This is arguably the **most important limitation**: a VLM that writes *"the kitchen is clean and ready for us to use"* receives a VTAS of 0.00, even though that caption is a perfectly valid, human-like description of a kitchen. The model correctly identified the scene but chose to describe it holistically rather than itemizing the oven, refrigerator, sink, cups, and bowls that DETR found.
|
|
237
|
+
|
|
238
|
+
> **VTAS v1 is an object-listing metric, not a scene-understanding metric.** It heavily rewards captions that enumerate discrete objects ("there is a clock, a TV, a chair, and a bottle") and penalizes captions that describe the scene as a whole ("this is a clean, well-lit living room"). This is by design — VTAS measures *grounded object alignment* — but it means that VTAS alone cannot evaluate the full quality of a caption.
|
|
239
|
+
|
|
240
|
+
**Root Cause:** The F-beta score requires non-zero Recall. When a caption contains zero object-level nouns that overlap with DETR's detections, Recall = 0, and the harmonic mean forces the entire score to 0 regardless of Precision.
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
### Future Work
|
|
245
|
+
|
|
246
|
+
Based on the limitations above, we propose the following research directions for VTAS v2:
|
|
247
|
+
|
|
248
|
+
| Direction | Addresses | Approach |
|
|
249
|
+
|---|---|---|
|
|
250
|
+
| **Instance-Aware Grounding** | Limitation 1 | Replace set-based matching with instance-level alignment. Use DETR's bounding box coordinates to distinguish "the rider" from "the spectators". Match each caption noun to a specific detection, preventing multiple nouns from collapsing onto one label. |
|
|
251
|
+
| **Scene-Level Recall** | Limitation 2 | Introduce a secondary recall signal that rewards captions for correctly identifying the *type* of scene (kitchen, bedroom, park) using CLIP or a dedicated scene classifier (e.g., Places365). A caption like "the kitchen is clean" would earn partial Recall credit for correctly classifying the environment. |
|
|
252
|
+
| **Attribute-Level Grounding** | New capability | Extend VTAS beyond nouns to verify adjectives ("red car", "large dog"). Use visual question-answering or attribute classifiers to check if described properties (color, size, material) match the image. |
|
|
253
|
+
| **Spatial Awareness** | New capability | Verify spatial claims ("the dog is *next to* the person", "the cup is *on* the table") against the physical coordinates of DETR bounding boxes. |
|
|
254
|
+
| **Dynamic Threshold Calibration** | Both | Replace the fixed similarity thresholds (MiniLM τ=0.65, CLIP τ=0.22) with dataset-adaptive thresholds calibrated on a held-out validation split, similar to ROC-AUC optimal threshold selection. |
|
|
255
|
+
|
|
256
|
+
---
|
|
257
|
+
|
|
258
|
+
## License
|
|
259
|
+
|
|
260
|
+
This project is licensed under the [MIT License](LICENSE).
|
|
261
|
+
|
|
262
|
+
## Citation
|
|
263
|
+
|
|
264
|
+
If you use VTAS in your research, please cite:
|
|
265
|
+
|
|
266
|
+
```bibtex
|
|
267
|
+
@software{prasad2026vtas,
|
|
268
|
+
author = {Prasad, Ritanshu},
|
|
269
|
+
title = {VTAS: Visual-Truth Alignment Score},
|
|
270
|
+
year = {2026},
|
|
271
|
+
url = {https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric},
|
|
272
|
+
license = {MIT}
|
|
273
|
+
}
|
|
274
|
+
```
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
|
|
2
|
+
from setuptools import setup, find_packages
|
|
3
|
+
|
|
4
|
+
setup(
|
|
5
|
+
name='vtas-metric',
|
|
6
|
+
version='1.3.0',
|
|
7
|
+
description='VTAS: Visual-Text Alignment Score for Evaluating Vision-Language Models',
|
|
8
|
+
long_description=open('README.md', encoding='utf-8').read(),
|
|
9
|
+
long_description_content_type='text/markdown',
|
|
10
|
+
author='Ritanshu Prasad',
|
|
11
|
+
author_email='ritanshu.prasad@example.com',
|
|
12
|
+
url='https://github.com/Ritanshu-Prasad/VTAS_Evaluation_Metric',
|
|
13
|
+
package_dir={'': 'src'},
|
|
14
|
+
packages=find_packages(where='src'),
|
|
15
|
+
install_requires=[
|
|
16
|
+
'torch',
|
|
17
|
+
'transformers',
|
|
18
|
+
'spacy',
|
|
19
|
+
'sentence-transformers',
|
|
20
|
+
'Pillow'
|
|
21
|
+
],
|
|
22
|
+
classifiers=[
|
|
23
|
+
'Programming Language :: Python :: 3',
|
|
24
|
+
'License :: OSI Approved :: MIT License',
|
|
25
|
+
'Operating System :: OS Independent',
|
|
26
|
+
],
|
|
27
|
+
python_requires='>=3.8',
|
|
28
|
+
)
|
|
29
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
from .vtas import VTASEvaluator
|