Joshi, C. K., Liu, F., Xun, X., Lin, J., & Foo, C. S. (2022). On Representation Knowledge Distillation for Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 1–12. https://doi.org/10.1109/tnnls.2022.3223018
Abstract:
Knowledge distillation (KD) is a learning paradigm
for boosting resource-efficient graph neural networks (GNNs)
using more expressive yet cumbersome teacher models. Past work
on distillation for GNNs proposed the local structure preserving
(LSP) loss, which matches local structural relationships defined
over edges across the student and teacher’s node embeddings.
This article studies whether preserving the global topology of
how the teacher embeds graph data can be a more effective
distillation objective for GNNs, as real-world graphs often
contain latent interactions and noisy edges. We propose graph
contrastive representation distillation (G-CRD), which uses
contrastive learning to implicitly preserve global topology by
aligning the student node embeddings to those of the teacher
in a shared representation space. Additionally, we introduce an
expanded set of benchmarks on large-scale real-world datasets
where the performance gap between teacher and student
GNNs is non-negligible. Experiments across four datasets
and 14 heterogeneous GNN architectures show that G-CRD
consistently boosts the performance and robustness of lightweight
GNNs, outperforming LSP (and a global structure preserving
(GSP) variant of LSP) as well as baselines from 2-D computer
vision. An analysis of the representational similarity among
teacher and student embedding spaces reveals that G-CRD
balances preserving local and global relationships, while structure
preserving approaches are best at preserving one or the other.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation - AME Programmatic
Grant Reference no. : A20H6b0151
This research / project is supported by the National Research Foundation - AME Programmatic
Grant Reference no. : A19E3b0099