PL/FM/SE Seminar Series: Prof. Darko Marinov - ISSTA'26 Papers

Title 1: The Discreet Charm of the Bugeoisie: A First Look at Bug Reports Created by Researchers

Authors: Ji young Kim, Jana Dragovic, Alessandro Botta, T. M. Rithwanul Islam, Alaa Mohamad, Karim Sharaf, Sejuti Sharmin Siddiqui, Divyanshi Joshi, Harini Anand, Nurjemal Saryyeva, Shubham Chapagain, Saad Nasir, Darko Marinov, and Bogdan Alexandru Stoica

Abstract: Many ISSTA papers present automated tools to find software bugs. Some researchers create bug reports for the bugs they deem worth reporting, e.g., new bugs found by the tools.We study bug reports created by researchers and coin the term Bugeoisie to refer to such bug reports. We collect the Bugeoisie for ISSTA'25 by reading the papers, examining the artifacts, and contacting the authors, trying to "chase" bug report URLs. This process turns out to be surprisingly challenging because only a few authors make such URLs readily available. We then analyze this Bugeoisie to understand potential bad practices. For most papers that claim some bugs reported or confirmed, independently checking those claims is again surprisingly challenging. We also start analyzing Bugeoisie for FSE'25 to check the generalizability of our findings. Our study points out some changes that conference organizers and reviewers may want to require for Bugeoisies. We also provide some suggestions for paper authors to prepare and describe their Bugeoisie in their papers and artifacts.

Title 2: Re-evaluating Detection of Equivalent Mutants Using LLMs: We Should Properly Measure How Far We Are

Authors: Arjun Tandon, Mehmet Fırat Dündar, Milkiyas Gebremichael Gebru, Darko Marinov, Yiling Lou, and Wenxi Wang

Abstract: Mutation testing is a widely used approach for measuring test-suite quality. A critical problem in mutation testing is equivalent mutant detection (EMD), i.e., determining if a mutant semantically behaves the same as the original code despite some syntactic differences. A recent study has shown that LLM-based EMD techniques hold great promise, reporting substantial improvements over traditional compiler- and machine-learning-based approaches. In this work, we revisit those recent results and evaluate the generalization capabilities of the proposed LLM-based EMD techniques across two additional datasets that differ from the prior dataset in mutation operators, programming languages, or source projects. Contrary to prior findings, the proposed LLM-based EMD techniques suffer substantial performance degradation on the two additional datasets. Through an extensive analysis, we identify a key factor underlying the differences as original-method-level data leakage (i.e., the same original method appearing in both training and test sets), indicating that prior results under within-method evaluation do not generalize to cross-method evaluation. We find that the studied LLMs tend to rely on a method-wise majority-voting shortcut rather than reasoning about the semantic effects of mutations. Based on these findings, we call for the adoption of realistic cross-method evaluation and the development of mutation-centric semantic reasoning in future LLM-based EMD research.