Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Federated Multi-Task Learning Model Based on Adaptive Distributed Data Latent Correlation Analysis

  • Wu, Shengbin (School of Information Engineering, Changsha Medical University) ;
  • Wang, Yibai (School of Information Engineering, Changsha Medical University)
  • Received : 2020.10.21
  • Accepted : 2021.05.04
  • Published : 2021.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Federated learning provides an efficient integrated model for distributed data, allowing the local training of different data. Meanwhile, the goal of multi-task learning is to simultaneously establish models for multiple related tasks, and to obtain the underlying main structure. However, traditional federated multi-task learning models not only have strict requirements for the data distribution, but also demand large amounts of calculation and have slow convergence, which hindered their promotion in many fields. In our work, we apply the rank constraint on weight vectors of the multi-task learning model to adaptively adjust the task's similarity learning, according to the distribution of federal node data. The proposed model has a general framework for solving optimal solutions, which can be used to deal with various data types. Experiments show that our model has achieved the best results in different dataset. Notably, our model can still obtain stable results in datasets with large distribution differences. In addition, compared with traditional federated multi-task learning models, our algorithm is able to converge on a local optimal solution within limited training iterations.

Keywords

Acknowledgement

This work is supported by the Scientific Research Fund of Hunan Education Department (No. 19C0190 and 20C0218).

References

  1. J. Konecny, H. Brendan McMahan, F. X. Yu, A. T. Suresh, D. Bacon, and P. Richtarik, "Federated learning: strategies for improving communication efficiency," 2017 [Online]. Available: https://arxiv.org/abs/1610.05492.
  2. H. Brendan McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Y. Arcas, "Communication-efficient learning of deep networks from decentralized data," in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, 2017, pp. 1273-1282.
  3. Q. Yang, Y. Liu, T. Chen, and Y. Tong, "Federated machine learning: concept and applications," ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, article no. 12, 2019. https://doi.org/10.1145/3298981
  4. H. Brendan McMahan, E. Moore, D. Ramage, and B. A. Y. Arcas, "Federated learning of deep networks using model averaging," 2016 [Online]. Available: https://arxiv.org/abs/1602.05629v1
  5. Y. Xue, X. Liao, L. Carin, and B. Krishnapuram, "Multi-task learning for classification with Dirichlet process priors," Journal of Machine Learning Research, vol. 8, pp. 35-63, 2007.
  6. S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345-1359, 2010. https://doi.org/10.1109/TKDE.2009.191
  7. X. T. Yuan, X. Liu, and S. Yan, "Visual classification with multitask joint sparse representation," IEEE Transactions on Image Processing, vol. 21, no. 10, pp. 4349-4360, 2012. https://doi.org/10.1109/TIP.2012.2205006
  8. L. Argote and E. Miron-Spektor, "Organizational learning: from experience to knowledge," Organization Science, vol. 22, no. 5, pp. 1123-1137, 2011. https://doi.org/10.1287/orsc.1100.0621
  9. C. Vens, J. Struyf, L. Schietgat, S. Dzeroski, and H. Blockeel, "Decision trees for hierarchical multi-label classification," Machine Learning, vol. 73, no. 2, pp. 185-214, 2008. https://doi.org/10.1007/s10994-008-5077-3
  10. T. Evgeniou and M. Pontil, "Regularized multi-task learning," in Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, 2004, pp. 109-117.
  11. S. Rosen, Z. Qian, and Z. M. Mao, "Appprofiler: a flexible method of exposing privacy-related behavior in android applications to end users," in Proceedings of the 3rd ACM Conference on Data and Application Security and Privacy, San Antonio, TX, 2013, pp. 221-232.
  12. J. Wang, M. Kolar, and N. Srerbo, "Distributed multi-task learning," in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Cadiz, Spain, 2016, pp. 751-760.
  13. R. Tibshirani, "Regression shrinkage and selection via the lasso: a retrospective," Journal of the Royal Statistical Societ B, vol. 73, no. 3, pp. 273-282, 2011. https://doi.org/10.1111/j.1467-9868.2011.00771.x
  14. R. G. Brereton and G. R. Lloyd, "Support vector machines for classification and regression," Analyst, vol. 135, no. 2, pp. 230-267, 2010. https://doi.org/10.1039/b918972f
  15. J. Wright, A. Ganesh, S. Rao, and Y. Ma, "Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization," Coordinated Science Laboratory, University of Illinois, Urbana, IL, Report No. UILU-ENG-09-2210(DC-243), 2009.
  16. X. Ding, Y. Chen, Z. Tang, and Y. Huang, "Camera identification based on domain knowledge-driven deep multi-task learning," IEEE Access, vol. 7, pp. 25878-25890, 2019. https://doi.org/10.1109/ACCESS.2019.2897360
  17. D. Mateos-Nunez, J. Cortes, and J. Cortes, "Distributed optimization for multi-task learning via nuclear-norm approximation," IFAC-PapersOnLine, vol. 48, no. 22, pp. 64-69, 2015.
  18. M. Zhao, H. Zhang, W. Cheng, and Z. Zhang, "Joint lp- and l2,p-norm minimization for subspace clustering with outlier pursuit," in Proceedings of 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, Canada, 2016, pp. 3658-3665.
  19. M. Zhang, Y. Yang, H. Zhang, F. Shen, and D. Zhang, "L2,p-norm and sample constraint based feature selection and classification for AD diagnosis," Neurocomputing, vol. 195, pp. 104-111, 2016. https://doi.org/10.1016/j.neucom.2015.08.111
  20. R. Caruana, "Multitask learning," Machine Learning, vol. 28, no. 1, pp. 41-75, 1997. https://doi.org/10.1023/A:1007379606734
  21. D. Zhang and D. Shen, "Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease," NeuroImage, vol. 59, no. 2, pp. 895-907, 2012. https://doi.org/10.1016/j.neuroimage.2011.09.069
  22. Z. Hu, B. Li, and J. Luo, "Time-and cost-efficient task scheduling across geo-distributed data centers," IEEE Transactions on Parallel and Distributed Systems, vol. 29, no. 3, pp. 705-718, 2018. https://doi.org/10.1109/tpds.2017.2773504
  23. Y. Wang, M. Nikkhah, X. Zhu, W. T. Tan, and R. Liston, "Learning geographically distributed data for multiple tasks using generative adversarial networks," in Proceedings of 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 4589-4593.
  24. X. Cai, F. Nie, H. Huang, and C. Ding, "Multi-class l2,1-norm support vector machine," in Proceedings of 2011 IEEE 11th International Conference on Data Mining, Vancouver, Canada, 2011, pp. 91-100.
  25. P. Heins, M. Moeller, and M. Burger, "Locally sparse reconstruction using the l1,∞-norm," Inverse Problems & Imaging, vol. 9, no. pp. 1093-1137, 2015. https://doi.org/10.3934/ipi.2015.9.1093
  26. P. E. Gill, W. Murray, and M. A. Saunders, "SNOPT: an SQP algorithm for large-scale constrained optimization," SIAM Review, vol. 47, no. 1, pp. 99-131, 2005. https://doi.org/10.1137/S0036144504446096
  27. N. Tottenham, J. W. Tanaka, A. C. Leon, T. McCarry, M. Nurse, T. A. Hare, et al., "The NimStim set of facial expressions: judgments from untrained research participants," Psychiatry Research, vol. 168, no. 3, pp. 242-249, 2009. https://doi.org/10.1016/j.psychres.2008.05.006
  28. K. S. Kim and S. Y. Chung, "Greedy subspace pursuit for joint sparse recovery," Journal of Computational and Applied Mathematics, vol. 352, pp. 308-327, 2019. https://doi.org/10.1016/j.cam.2018.11.027
  29. S. Yi, Y. Liang, Z. He, Y. Li, and Y. M. Cheung, "Dual pursuit for subspace learning," IEEE Transactions on Multimedia, vol. 21, no. 6, pp. 1399-1411, 2019. https://doi.org/10.1109/tmm.2018.2877888
  30. V. Smith, C. K. Chiang, M. Sanjabi, and A. Talwalkar, "Federated multi-task learning," 2017 [Online]. Available: https://arxiv.org/abs/1705.10467.