Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data

Page view(s)
111
Checked on Jun 11, 2024
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data
Title:
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data
Journal Title:
Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence
Keywords:
Publication Date:
11 August 2023
Citation:
Xu, Q., Wu, M., Li, X., Mao, K., & Chen, Z. (2023). Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2023/496
Abstract:
For many real-world time series tasks, the computational complexity of prevalent deep leaning models often hinders the deployment on resource limited environments (e.g., smartphones). Moreover, due to the inevitable domain shift between model training (source) and deploying (target) stages, compressing those deep models under cross-domain scenarios becomes more challenging. Although some of existing works have already explored cross-domain knowledge distillation for model compression, they are either biased to source data or heavily tangled between source and target data. To this end, we design a novel end-to-end framework called UNiversal and joInt Knowledge Distillation (UNI-KD) for cross-domain model compression. In particular, we propose to transfer both the universal feature-level knowledge across source and target domains and the joint logit-level knowledge shared by both domains from the teacher to the student model via an adversarial learning scheme. More specifically, a feature-domain discriminator is employed to align teacher’s and student’s representations for universal knowledge transfer. A data-domain discriminator is utilized to prioritize the domain-shared samples for joint knowledge transfer. Extensive experimental results on four time series datasets demonstrate the superiority of our proposed method over state-of-the-art (SOTA) benchmarks. The source code is available at https://github.com/ijcai2023/UNI KD.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - AME Young Individual Research Grant
Grant Reference no. : A2084c1067

This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - AME Programmatic Funds
Grant Reference no. : A20H6b0151
Description:
ISSN:
NA
Files uploaded:

File Size Format Action
distil1.pdf 562.51 KB PDF Open