Browse > Article
http://dx.doi.org/10.3745/JIPS.01.0075

A Federated Multi-Task Learning Model Based on Adaptive Distributed Data Latent Correlation Analysis  

Wu, Shengbin (School of Information Engineering, Changsha Medical University)
Wang, Yibai (School of Information Engineering, Changsha Medical University)
Publication Information
Journal of Information Processing Systems / v.17, no.3, 2021 , pp. 441-452 More about this Journal
Abstract
Federated learning provides an efficient integrated model for distributed data, allowing the local training of different data. Meanwhile, the goal of multi-task learning is to simultaneously establish models for multiple related tasks, and to obtain the underlying main structure. However, traditional federated multi-task learning models not only have strict requirements for the data distribution, but also demand large amounts of calculation and have slow convergence, which hindered their promotion in many fields. In our work, we apply the rank constraint on weight vectors of the multi-task learning model to adaptively adjust the task's similarity learning, according to the distribution of federal node data. The proposed model has a general framework for solving optimal solutions, which can be used to deal with various data types. Experiments show that our model has achieved the best results in different dataset. Notably, our model can still obtain stable results in datasets with large distribution differences. In addition, compared with traditional federated multi-task learning models, our algorithm is able to converge on a local optimal solution within limited training iterations.
Keywords
Data Distribution; Federated Multi-Task Learning; Rank Constraint; Underlying Structure;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Wang, M. Kolar, and N. Srerbo, "Distributed multi-task learning," in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Cadiz, Spain, 2016, pp. 751-760.
2 R. Tibshirani, "Regression shrinkage and selection via the lasso: a retrospective," Journal of the Royal Statistical Societ B, vol. 73, no. 3, pp. 273-282, 2011.   DOI
3 J. Wright, A. Ganesh, S. Rao, and Y. Ma, "Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization," Coordinated Science Laboratory, University of Illinois, Urbana, IL, Report No. UILU-ENG-09-2210(DC-243), 2009.
4 J. Konecny, H. Brendan McMahan, F. X. Yu, A. T. Suresh, D. Bacon, and P. Richtarik, "Federated learning: strategies for improving communication efficiency," 2017 [Online]. Available: https://arxiv.org/abs/1610.05492.
5 M. Zhao, H. Zhang, W. Cheng, and Z. Zhang, "Joint lp- and l2,p-norm minimization for subspace clustering with outlier pursuit," in Proceedings of 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, Canada, 2016, pp. 3658-3665.
6 H. Brendan McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Y. Arcas, "Communication-efficient learning of deep networks from decentralized data," in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, 2017, pp. 1273-1282.
7 Q. Yang, Y. Liu, T. Chen, and Y. Tong, "Federated machine learning: concept and applications," ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, article no. 12, 2019. https://doi.org/10.1145/3298981   DOI
8 K. S. Kim and S. Y. Chung, "Greedy subspace pursuit for joint sparse recovery," Journal of Computational and Applied Mathematics, vol. 352, pp. 308-327, 2019.   DOI
9 X. Ding, Y. Chen, Z. Tang, and Y. Huang, "Camera identification based on domain knowledge-driven deep multi-task learning," IEEE Access, vol. 7, pp. 25878-25890, 2019.   DOI
10 D. Mateos-Nunez, J. Cortes, and J. Cortes, "Distributed optimization for multi-task learning via nuclear-norm approximation," IFAC-PapersOnLine, vol. 48, no. 22, pp. 64-69, 2015.
11 M. Zhang, Y. Yang, H. Zhang, F. Shen, and D. Zhang, "L2,p-norm and sample constraint based feature selection and classification for AD diagnosis," Neurocomputing, vol. 195, pp. 104-111, 2016.   DOI
12 R. Caruana, "Multitask learning," Machine Learning, vol. 28, no. 1, pp. 41-75, 1997.   DOI
13 D. Zhang and D. Shen, "Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease," NeuroImage, vol. 59, no. 2, pp. 895-907, 2012.   DOI
14 Z. Hu, B. Li, and J. Luo, "Time-and cost-efficient task scheduling across geo-distributed data centers," IEEE Transactions on Parallel and Distributed Systems, vol. 29, no. 3, pp. 705-718, 2018.   DOI
15 T. Evgeniou and M. Pontil, "Regularized multi-task learning," in Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, 2004, pp. 109-117.
16 X. Cai, F. Nie, H. Huang, and C. Ding, "Multi-class l2,1-norm support vector machine," in Proceedings of 2011 IEEE 11th International Conference on Data Mining, Vancouver, Canada, 2011, pp. 91-100.
17 P. Heins, M. Moeller, and M. Burger, "Locally sparse reconstruction using the l1,∞-norm," Inverse Problems & Imaging, vol. 9, no. pp. 1093-1137, 2015.   DOI
18 P. E. Gill, W. Murray, and M. A. Saunders, "SNOPT: an SQP algorithm for large-scale constrained optimization," SIAM Review, vol. 47, no. 1, pp. 99-131, 2005.   DOI
19 S. Yi, Y. Liang, Z. He, Y. Li, and Y. M. Cheung, "Dual pursuit for subspace learning," IEEE Transactions on Multimedia, vol. 21, no. 6, pp. 1399-1411, 2019.   DOI
20 N. Tottenham, J. W. Tanaka, A. C. Leon, T. McCarry, M. Nurse, T. A. Hare, et al., "The NimStim set of facial expressions: judgments from untrained research participants," Psychiatry Research, vol. 168, no. 3, pp. 242-249, 2009.   DOI
21 V. Smith, C. K. Chiang, M. Sanjabi, and A. Talwalkar, "Federated multi-task learning," 2017 [Online]. Available: https://arxiv.org/abs/1705.10467.
22 C. Vens, J. Struyf, L. Schietgat, S. Dzeroski, and H. Blockeel, "Decision trees for hierarchical multi-label classification," Machine Learning, vol. 73, no. 2, pp. 185-214, 2008.   DOI
23 R. G. Brereton and G. R. Lloyd, "Support vector machines for classification and regression," Analyst, vol. 135, no. 2, pp. 230-267, 2010.   DOI
24 Y. Wang, M. Nikkhah, X. Zhu, W. T. Tan, and R. Liston, "Learning geographically distributed data for multiple tasks using generative adversarial networks," in Proceedings of 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 4589-4593.
25 H. Brendan McMahan, E. Moore, D. Ramage, and B. A. Y. Arcas, "Federated learning of deep networks using model averaging," 2016 [Online]. Available: https://arxiv.org/abs/1602.05629v1
26 Y. Xue, X. Liao, L. Carin, and B. Krishnapuram, "Multi-task learning for classification with Dirichlet process priors," Journal of Machine Learning Research, vol. 8, pp. 35-63, 2007.
27 S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345-1359, 2010.   DOI
28 X. T. Yuan, X. Liu, and S. Yan, "Visual classification with multitask joint sparse representation," IEEE Transactions on Image Processing, vol. 21, no. 10, pp. 4349-4360, 2012.   DOI
29 L. Argote and E. Miron-Spektor, "Organizational learning: from experience to knowledge," Organization Science, vol. 22, no. 5, pp. 1123-1137, 2011.   DOI
30 S. Rosen, Z. Qian, and Z. M. Mao, "Appprofiler: a flexible method of exposing privacy-related behavior in android applications to end users," in Proceedings of the 3rd ACM Conference on Data and Application Security and Privacy, San Antonio, TX, 2013, pp. 221-232.