Software is an important part of our society, and we depend on softwaredevelopers to implement and maintain the quality of software. In turn, developers rely on regression testing, the practice of running tests after every software change, to check that they do not introduce bugs as they make changes. However, regression testing is plagued by flaky tests that non-deterministically pass and fail independently of software changes. The outcomes of flaky tests often mislead developers about the validity of their changes. Flaky tests are also quite prevalent, e.g., Google once reported 16% of their 4.2 million tests are flaky and that 84% of their changes with test failures involve a flaky test.
In this talk, I will present my work on mitigating the effects of flaky tests by developing techniques to automatically fix and detect flaky tests. I will focus specifically on order-dependent flaky tests, which pass when run in one order of the test suite but fail when run in a different order. First, I will describe iFixFlakies, the first technique for automatically fixing order-dependent flaky tests. Results show that iFixFlakies automatically fixes 101 out of 110 order-dependent flaky tests from a public dataset of flaky tests, and developers accepted patches for 64 of these 101 flaky tests, with the rest pending review. Next, I will describe PolDet, the first technique for proactively detecting order-dependent flaky tests. PolDet detects the root causes of such tests before they fail or even before they exist in the test suite. Results show that PolDet detects 194 root causes that can create order-dependent flaky tests.
August Shi is a PhD candidate in Computer Science at the University of Illinois at Urbana-Champaign. August works with Professor Darko Marinov in the area of Software Engineering, with a focus on Software Testing. His research aims to improve developer productivity by making regression testing (1) more reliable< with respect to flaky tests and (2) faster without loss in quality of testing. August has published 17 full research papers in ICSE, ESEC/FSE, ASE, ISSTA, OOPSLA, ICST, and ISSRE, and his work on improving regression testing received an ACM SIGSOFT Distinguished Paper Award at ICSE 2017. More information is available on his web page: http://mir.cs.illinois.edu/awshi2.