FedCert: Federated Accuracy Certification

Minh Hieu Nguyen1, Huu Tien Nguyen1, Trung Thanh Nguyen2, Manh Duong Nguyen1, Trong Nghia Hoang3, Truong Thao Nguyen4, Phi Le Nguyen1
1Hanoi University of Science and Technology, Vietnam 2Nagoya University, Japan 3Washington State University, United States 4National Institute of Advanced Industrial Science and Technology (AIST), Japan
FedCert Paper Code

Abstract

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models in a decentralized manner, preserving data privacy by keeping local data on clients. However, evaluating the robustness of these models against data perturbations on clients remains a significant challenge. Previous studies have assessed the effectiveness of models in centralized training based on certified accuracy, which guarantees that a certain percentage of the model's predictions will remain correct even if the input data is perturbed. However, the challenge of extending these evaluations to FL remains unresolved due to the unknown client's local data. To tackle this challenge, this study proposed a method named FedCert to take the first step toward evaluating the robustness of FL systems. The proposed method is designed to approximate the certified accuracy of a global model based on the certified accuracy and class distribution of each client. Additionally, considering the Non-Independent and Identically Distributed (Non-IID) nature of data in real-world scenarios, we introduce the client grouping algorithm to ensure reliable certified accuracy during the aggregation step of the approximation algorithm. Through theoretical analysis, we demonstrate the effectiveness of FedCert in assessing the robustness and reliability of FL systems. Moreover, experimental results on the CIFAR-10 and CIFAR-100 datasets under various scenarios show that FedCert consistently reduces the estimation error compared to baseline methods. This study offers a solution for evaluating the robustness of FL systems and lays the groundwork for future research to enhance the dependability of decentralized learning.

Proposed Method: FedCert

FedCert is a method developed to assess the robustness of a federated learning (FL) system by approximating the certified accuracy of the global model. The certified accuracy represents the portion of predictions that remain robust under small perturbations in the input data. The main contributions of FedCert include:

  • Certified Accuracy Approximation: FedCert uses a combination of local certified accuracies from individual clients to estimate the overall certified accuracy of the global model.
  • Client Grouping Algorithm: To handle the issue of non-IID (non-independent and identically distributed) data across clients, FedCert introduces a client grouping algorithm. This method groups smaller clients together to improve the reliability of the aggregated accuracy.
  • Convex Optimization: FedCert employs convex optimization techniques to minimize the estimation error when combining certified accuracies from multiple clients.

Figure 1 illustrates the main components of the FedCert method, including the certified accuracy approximation and client grouping algorithm.

Figure 1. Overview of FedCert. The method combines local certified accuracies and uses convex optimization to estimate the global model's certified accuracy.

Experimental Results

We evaluated FedCert on two widely used image classification datasets: CIFAR-10 and CIFAR-100. The experiments were conducted across different FL settings, including non-IID data distributions. The results show that FedCert consistently reduces the estimation error of certified accuracy compared to baseline methods such as Volume-based Weighted-sum (VW).

  • Dataset: CIFAR-10, CIFAR-100
  • Baselines: Volume-based Weighted-sum (VW), FedAvg
  • Evaluation Metrics: Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE)

The following figures present the experimental results, highlighting the significant reduction in RMSE and MAPE achieved by FedCert in various scenarios.

Figure 2. Experimental results showing the performance of FedCert compared to baseline methods. FedCert achieves lower RMSE and MAPE across all settings.

BibTeX

@inproceedings{nguyen2024fedcert,
      author={Nguyen, Minh Hieu and Nguyen, Huu Tien and Nguyen, Trung Thanh and Nguyen, Manh Duong and Hoang, Trong Nghia and Nguyen, Truong Thao and Nguyen, Phi Le},
      title={FedCert: Federated Accuracy Certification},
      booktitle={Proceedings of the 22nd International Symposium on Network Computing and Applications (NCA 2024)},
      year={2024},
}