Compared to What? A Survey of Tornadic Environments, Baselines, and Forecast Skill
Many statements related to forecast evaluation fall back on unverified assumptions: "this tornado warning had lower skill because the underlying meteorology was a more complicated or atypical scenario" or "that forecast performed worse than we would have expected given the straightforward setup". As the severe storms community turns its attention to tornadoes that occur in non-classic environments (e.g., tornadoes spawned by quasi-linear convective systems), it has become especially important to quantify what is and isn't a reasonable expectation for warning skill.
This work, through the use of a fifteen-year dataset composed of tens of thousands of tornado events and warnings across the contiguous U.S., puts an environmental spin on the concept of a skill score to quantify the effect of the near-storm environment on tornado warning skill. Results show that nocturnal tornadoes have both worse probability of detection (POD) and false alarm ratio (FAR) than even their marginal near-storm environments would suggest, and tornadoes occurring during the summer months also show worse POD and FAR than their environment-based expectation.
Quantifying these shifts in performance in an environmental skill score framework allows us to target the situations in which we can make the greatest improvements in terms of forecaster training and/or conceptual models. This work also highlights the essential question that should always be asked in the context of verification: what, exactly, is the baseline standard to which we are comparing forecast performance?
