AI Models Show Promise Diagnosing Software Failures in Safety-Critical Train Systems
Researchers benchmarked seven large language models to automatically identify root causes in train control software test failures, a task currently requiring expensive manual labor. The work suggests AI could accelerate quality assurance in safety-critical industries where debugging delays increase costs and risk.
Originaltitel: Benchmarking Large Language Models for Root Cause Analysis in Train Control Software Testing
<p>Software quality assurance is critical in safety-critical domains like railway systems, where failures can have catastrophic consequences. In this context, train control and management systems play a central role, and their software must undergo rigorous validation. Alstom Rail Sweden AB employs a digital twin infrastructure to simulate and validate train control and management systems software. While this setup significantly improves system-level testing, the root cause analysis of test failures remains a manual and time-consuming bottleneck.This study explores the potential of large language models to automate root cause analysis by interpreting execution logs generated during digital twin-based testing. We benchmark seven state-of-the-art large language models, Aion-1.0, DeepSeek R1, DeepSeek V3 0324, Mistral Small 3.1 24B, GPT o3-mini, Gemini 2.5 Pro Experimental, and QwB 32B, using zero-shot chain-of-thought prompting to assess their ability to reason about fault patterns in real-world industrial test execution logs. The logs, sourced from Alstom’s digital twin-based testing environment, capture complex operational behaviour typical of embedded, safety-critical systems.Our results show that Gemini 2.5 Pro Experimental achieved the best performance with 66.7% accuracy and strong reasoning quality in this domain, contributing to the future research agenda to improve the accuracy prediction.</p>