Skellam Mixture Mechanism: a Novel Approach to Federated Learning with Differential Privacy

Skellam Mixture Mechanism: a Novel Approach to Federated Learning with Differential Privacy

Proceedings of the VLDB Endowment

10.14778/3551793.3551798

http://dx.doi.org/10.14778/3551793.3551798

Ergute Bao, Yizheng Zhu, Xiaokui Xiao, Yin Yang, Beng Chin Ooi, Benjamin Hong Meng Tan, Khin Mi Mi Aung

Planning and Development, General Earth and Planetary Sciences, Water Science and Technology, geography

29 September 2022

Bao, E., Zhu, Y., Xiao, X., Yang, Y., Ooi, B. C., Tan, B. H. M., & Aung, K. M. M. (2022). Skellam mixture mechanism. Proceedings of the VLDB Endowment, 15(11), 2348–2360. https://doi.org/10.14778/3551793.3551798

Deep neural networks have strong capabilities of memorizing the underlying training data, which can be a serious privacy concern. An effective solution to this problem is to train models with differential privacy ( DP ), which provides rigorous privacy guarantees by injecting random noise to the gradients. This paper focuses on the scenario where sensitive data are distributed among multiple participants, who jointly train a model through federated learning , using both secure multiparty computation ( MPC ) to ensure the confidentiality of each gradient update, and differential privacy to avoid data leakage in the resulting model. A major challenge in this setting is that common mechanisms for enforcing DP in deep learning, which inject real-valued noise , are fundamentally incompatible with MPC, which exchanges finite-field integers among the participants. Consequently, most existing DP mechanisms require rather high noise levels, leading to poor model utility. Motivated by this, we propose Skellam mixture mechanism (SMM), a novel approach to enforcing DP on models built via federated learning. Compared to existing methods, SMM eliminates the assumption that the input gradients must be integer-valued, and, thus, reduces the amount of noise injected to preserve DP. The theoretical analysis of SMM is highly non-trivial, especially considering (i) the complicated math of DP deep learning in general and (ii) the fact that the mixture of two Skellam distributions is rather complex. Extensive experiments on various practical settings demonstrate that SMM consistently and significantly outperforms existing solutions in terms of the utility of the resulting model.

Publisher Copyright

This research / project is supported by the A*STAR - MTC Programmatic
Grant Reference no. : A19E3b009

This research / project is supported by the Ministry of Education - AcRF-Tier2
Grant Reference no. : MOE2018-T2-2-091

Qatar National Research Fund Qatar Foundation (Number NPRP11C-1229-1700

https://oar.a-star.edu.sg/communities-collections/articles/18640

2150-8097

Institute for Infocomm Research