Language Alone Can Spot What People Reference in Video Chats
Researchers found that AI language models can identify what objects people are talking about in conversations just by reading the text—without analyzing the video itself. The finding has major implications for building cheaper, faster AI assistants for customer service, virtual meetings, and accessibility tools.
Originaltitel: Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
<p>In this paper, we explore the use of a text-only, autoregressive language modeling approach for the extraction of referring expressions from visually grounded dialogue. More specifically, the aim is to investigate the extent to which the linguistic context alone can inform the detection of mentions that have a (visually perceivable) referent in the visual context of the conversation. To this end, we adapt a pretrained large language model (LLM) to perform a relatively course-grained annotation of mention spans in unfolding conversations by demarcating mention span boundaries in text via next-token prediction. Our findings indicate that even when using a moderately sized LLM, relatively small datasets, and parameter-efficient fine-tuning, a text-only approach can be effective, highlighting the relative importance of the linguistic context for this task. Nevertheless, we argue that the task represents an inherently multimodal problem and discuss limitations fundamental to unimodal approaches.</p>