Widely Used Language Datasets Hide Quality Problems, Study Finds
Researchers analyzing 100+ language datasets discovered that many appear accurate on standard metrics but contain little useful information for training AI systems. The finding suggests organizations relying on these datasets for natural language processing may be building models on shakier ground than performance scores suggest.
Originaltitel: Investigating UD Treebanks via Dataset Difficulty Measures
<p>Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, 𝒱-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein.</p>