Recording available at: https://mediaspace.illinois.edu/media/t/1_ch98z9nl
Abstract:
As software evolves, developers typically perform regression testing to ensure that their code changes do not break existing functionality. During regression testing, developers often waste time debugging their code changes because of spurious failures from flaky tests, which are tests that nondeterministically pass or fail on the same code. These spurious failures mislead developers because the failures are due to bugs that existed before the code changes. My work on characterizing flaky tests has helped open the research topic of flaky tests, and many companies (e.g., Facebook, Google, Microsoft) have since highlighted flaky tests as a major challenge in their software development.
In this talk, I will describe my recent work on taming flaky tests. Two prominent kinds of flaky tests are order-dependent flaky tests, which pass when run in one order but fail when run in a different order, and async-wait flaky tests, which pass if an asynchronous call finishes on time but fail if it finishes too late. My results include the first automated techniques to (1) fix order-dependent flaky tests, fixing 92% of such flaky tests in a public dataset; (2) reduce the number of spurious failures from order-dependent flaky tests, reducing such failures by 73%; and (3) speed up async-wait flaky tests while also reducing their spurious failures, speeding up such tests by 38%. Overall, my work has helped detect more than 2000 flaky tests and fix more than 500 flaky tests in over 150 open-source projects.
Bio:
Wing Lam is a PhD candidate in the Computer Science department at the University of Illinois at Urbana-Champaign where he is co-advised by Professors Tao Xie and Darko Marinov. He works on several topics in software engineering, with a focus on software testing. Wing's research improves software dependability by characterizing bugs and developing novel techniques to detect and tame bugs. He has published in top-tier conferences such as ESEC/FSE, ICSE, ISSTA, OOPSLA, and TACAS. His techniques have helped detect and fix bugs in open-source projects and have impacted how Microsoft and Tencent developers test their code. Wing has been awarded several fellowships and scholarships, including a Google - CMD-IT Dissertation Fellowship Award. More information is available on his web page: http://mir.cs.illinois.edu/winglam
