Kavli Affiliate: Jiansheng Chen
| First 5 Authors: Kun Song, Zhiquan Tan, Bochao Zou, Jiansheng Chen, Huimin Ma
| Summary:
In this paper, we utilize information-theoretic metrics like matrix entropy
and mutual information to analyze supervised learning. We explore the
information content of data representations and classification head weights and
their information interplay during supervised training. Experiments show that
matrix entropy cannot solely describe the interaction of the information
content of data representation and classification head weights but it can
effectively reflect the similarity and clustering behavior of the data.
Inspired by this, we propose a cross-modal alignment loss to improve the
alignment between the representations of the same class from different
modalities. Moreover, in order to assess the interaction of the information
content of data representation and classification head weights more accurately,
we utilize new metrics like matrix mutual information ratio (MIR) and matrix
information entropy difference ratio (HDR). Through theory and experiment, we
show that HDR and MIR can not only effectively describe the information
interplay of supervised training but also improve the performance of supervised
and semi-supervised learning.
| Search Query: ArXiv Query: search_query=au:”Jiansheng Chen”&id_list=&start=0&max_results=3