Kavli Affiliate: Ke Wang
| First 5 Authors: Zi Wang, Jihye Choi, Ke Wang, Somesh Jha,
| Summary:
Motivated by the success of traditional software testing, numerous diversity
measures have been proposed for testing deep neural networks (DNNs). In this
study, we propose a shift in perspective, advocating for the consideration of
DNN testing as directed testing problems rather than diversity-based testing
tasks. We note that the objective of testing DNNs is specific and well-defined:
identifying inputs that lead to misclassifications. Consequently, a more
precise testing approach is to prioritize inputs with a higher potential to
induce misclassifications, as opposed to emphasizing inputs that enhance
"diversity."
We derive six directed metrics for DNN testing. Furthermore, we conduct a
careful analysis of the appropriate scope for each metric, as applying metrics
beyond their intended scope could significantly diminish their effectiveness.
Our evaluation demonstrates that (1) diversity metrics are particularly weak
indicators for identifying buggy inputs resulting from small input
perturbations, and (2) our directed metrics consistently outperform diversity
metrics in revealing erroneous behaviors of DNNs across all scenarios.
| Search Query: ArXiv Query: search_query=au:”Ke Wang”&id_list=&start=0&max_results=3