It is widely accepted that the assessment and refinement of biomedical imaging technologies should be performed by objective, i.e., task-based, measures of image quality (IQ). However, the objective evaluation of deep learning-based image formation technologies remains largely lacking, despite the breakneck speed at which they are being developed. As such, there is an ever-growing collection of methods whose utility and trustworthiness remains largely unknown. Moreover, such methods have the capability to ‘hallucinate’ false structures, which is of significant concern in medical imaging applications. In this work, we report studies in which the performance of deep a learning-based image restoration method is objectively assessed. The performance of the ideal observer (IO) and common linear numerical observers are quantified, and detection efficiencies are computed to assess the potential impact of deep learning on signal detection performance in this application. The numerical results indicate that, in the cases considered, the application of a deep image formation network can result in a loss of task-relevant information in the image, despite improvement in traditional computer-vision metrics. We also demonstrate that traditional and objective IQ measures can vary in opposite ways as a function of network depth. In a second study, we assess the quality of synthetic images produced by deep generative models and demonstrate how traditional IQ measures used in the field of computer vision can be inadequate. These results highlight the need for the objective evaluation of IQ for deep learning technologies for biomedical imaging and may suggest future avenues for improving the effectiveness of biomedical imaging applications.
