Kavli Affiliate: Zhuo Li
| First 5 Authors: Mouxiang Chen, Lefei Shen, Han Fu, Zhuo Li, Jianling Sun
| Summary:
Recent years have witnessed the success of introducing Transformers to time
series forecasting. From a data generation perspective, we illustrate that
existing Transformers are susceptible to distribution shifts driven by temporal
contexts, whether observed or unobserved. Such context-driven distribution
shift (CDS) introduces biases in predictions within specific contexts and poses
challenges for conventional training paradigm. In this paper, we introduce a
universal calibration methodology for the detection and adaptation of CDS with
a trained Transformer model. To this end, we propose a novel CDS detector,
termed the "residual-based CDS detector" or "Reconditionor", which quantifies
the model’s vulnerability to CDS by evaluating the mutual information between
prediction residuals and their corresponding contexts. A high Reconditionor
score indicates a severe susceptibility, thereby necessitating model
adaptation. In this circumstance, we put forth a straightforward yet potent
adapter framework for model calibration, termed the "sample-level
contextualized adapter" or "SOLID". This framework involves the curation of a
contextually similar dataset to the provided test sample and the subsequent
fine-tuning of the model’s prediction layer with a limited number of steps. Our
theoretical analysis demonstrates that this adaptation strategy is able to
achieve an optimal equilibrium between bias and variance. Notably, our proposed
Reconditionor and SOLID are model-agnostic and readily adaptable to a wide
range of Transformers. Extensive experiments show that SOLID consistently
enhances the performance of current SOTA Transformers on real-world datasets,
especially on cases with substantial CDS detected by the proposed
Reconditionor, thus validate the effectiveness of the calibration approach.
| Search Query: ArXiv Query: search_query=au:”Zhuo Li”&id_list=&start=0&max_results=3