On Representation Knowledge Distillation for Graph Neural Networks

On Representation Knowledge Distillation for Graph Neural Networks

IEEE Transactions on Neural Networks and Learning Systems

10.1109/TNNLS.2022.3223018

http://dx.doi.org/10.1109/tnnls.2022.3223018

Chaitanya K. Joshi, Fayao Liu, Xun Xu, Jie Lin, Chuan-Sheng Foo

Electrical and Electronic Engineering

06 December 2022

Joshi, C. K., Liu, F., Xun, X., Lin, J., & Foo, C. S. (2022). On Representation Knowledge Distillation for Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 1–12. https://doi.org/10.1109/tnnls.2022.3223018

Knowledge distillation (KD) is a learning paradigm for boosting resource-efficient graph neural networks (GNNs) using more expressive yet cumbersome teacher models. Past work on distillation for GNNs proposed the local structure preserving (LSP) loss, which matches local structural relationships defined over edges across the student and teacher’s node embeddings. This article studies whether preserving the global topology of how the teacher embeds graph data can be a more effective distillation objective for GNNs, as real-world graphs often contain latent interactions and noisy edges. We propose graph contrastive representation distillation (G-CRD), which uses contrastive learning to implicitly preserve global topology by aligning the student node embeddings to those of the teacher in a shared representation space. Additionally, we introduce an expanded set of benchmarks on large-scale real-world datasets where the performance gap between teacher and student GNNs is non-negligible. Experiments across four datasets and 14 heterogeneous GNN architectures show that G-CRD consistently boosts the performance and robustness of lightweight GNNs, outperforming LSP (and a global structure preserving (GSP) variant of LSP) as well as baselines from 2-D computer vision. An analysis of the representational similarity among teacher and student embedding spaces reveals that G-CRD balances preserving local and global relationships, while structure preserving approaches are best at preserving one or the other.

Publisher Copyright

This research / project is supported by the National Research Foundation - AME Programmatic
Grant Reference no. : A20H6b0151

This research / project is supported by the National Research Foundation - AME Programmatic
Grant Reference no. : A19E3b0099

© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

https://oar.a-star.edu.sg/communities-collections/articles/18908

2162-237X
2162-2388

Institute for Infocomm Research