New metric helps AI systems understand visual relationships in images
Researchers have developed a fairer way to test whether AI vision models can recognize diverse object relationships in images, addressing a major bottleneck in computer vision. They've also created a method to generate high-quality training data automatically, potentially accelerating deployment of more capable visual AI systems across industries from robotics to e-commerce.
Originaltitel: Measuring image-relation alignment: reference-free evaluation of VLMs and synthetic pre-training for open-vocabulary scene graph generation
<p>Scene Graph Generation (SGG) encodes visual relationships between objects in images as graph structures. Thanks to the advances of Vision-Language Models (VLMs), the task of Open-Vocabulary SGG has been recently proposed where models are evaluated on their functionality to learn a wide and diverse range of relations. Current benchmarks in SGG, however, possess a very limited vocabulary, making the evaluation of open-source models inefficient. In this paper, we propose a new reference-free metric to fairly evaluate the open-vocabulary capabilities of VLMs for relation prediction. Another limitation of Open-Vocabulary SGG is the reliance on weakly supervised data of poor quality for pre-training. We also propose a new solution for quickly generating high-quality synthetic data through region-specific prompt tuning of VLMs. Experimental results show that pre-training with this new data split can benefit the generalization capabilities of Open-Voc SGG models11Code and data available at https://github.com/Maelic/OpenVocSGG.</p>