The Journal of Bigdata (한국빅데이터학회지)

Korea Bigdata Society (한국빅데이터학회)
권호 표지
DOI QR코드

DOI QR Code

A Sequencing Problem with Generalized Due Dates for Distributed Training of Neural Networks

신경망 분산 학습을 위한 일반 납기를 갖는 시퀀싱 문제

  • Received : 2020.08.05
  • Accepted : 2020.08.25
  • Published : 2020.08.30
Download PDF
⟨ Previous Next ⟩

Abstract

We consider the stale problem which makes the training speed slow in the field of deep learning. The problem can be formulated as a single-machine scheduling problem with generalized due dates in which the objective is to minimize the total earliness and tardiness. We show that the problem can be solved in polynomial time if the orders of the small and the large jobs in an optimal schedule are known in advance.

본 논문은 딥러닝을 위한분산학습에서학습속도를 저하시키는 stale 문제를 최소화하기 위한 방법으로 데이터 시퀀싱을 제안하였다. 이데이터 시퀀싱 문제는일반 납기를 갖는 단일 공정 하에서 일찍 혹은 늦음 정도의 총합을 최소화 하는 스케줄링 문제로 모델링할 수 있다. 만약 최적해에서 크기가 작은 작업과 큰 작업의 순서가 미리 알려져 있다면, 이 스케줄링 문제가 효율적으로 풀린다는 것을 보였다.

Keywords

References

  1. R.A. Ahuja, K. Mehlhorn, J.B. Orlin. "Faster algorithm for the shortest path problem," Journal of the Association for Computing Machinery, Vol. 37, pp. 213-223, 1990. https://doi.org/10.1145/77600.77615
  2. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. "Curriculum learning," Proceedings of the 26th International Conference on Machine Learning, pp. 41-48, Montreal, Canada, June 2009.
  3. B. Bahdanau, K. Cho, and Y. Bengio. "Neural machine translation by jointly learning to align and translate." Proceedings of 3rd International Conference on Learning Representations, San Diego, USA, May, 2015.
  4. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. "Project ADAM: Building an efficient and scalable deep learning training system." Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, pp. 571-582, CO, USA, October. 2014.
  5. B.C. Choi, Y.H. Min, and M.J. Park. "Strong NP-hardness of minimizing total deviation with generalized and periodic due dates," Operations Research Letters, Vol. 47, pp. 433-437, Sep. 2019. https://doi.org/10.1016/j.orl.2019.08.002
  6. B.C. Choi, K.M Kim, Y.H. Min, and M.J. Park. "A single machine scheduling with generalized and periodic due dates," Submitted to Discrete Applied Mathematics.
  7. B.C. Choi, and M.J. Park. "Just-in-time scheduling with generalized due dates and identical due date intervals," Asian-Pacific Journal of Operational Research, Vol. 6, pp. 1-13, Nov. 2018.
  8. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A.Y. Ng. "Large scale distributed deep networks," Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems, pp. 1223-1231, Nevada, USA. December, 2012.
  9. C. Farabet, C. Couprie, L. Najman, and Y. LeCun. "Learning hierarchical features for scene labeling," IEEE Transactions in Pattern Aanlysis and Machine Intelligence, Vol. 35, pp. 1915-1929, 2013. https://doi.org/10.1109/TPAMI.2012.231
  10. Y. Gao and J. Yuan. "Unary NP-hardness of minimizing total weighted tardiness with generalized due dates," Operations Research Letters, Vol. 44, pp. 92-95, 2006. https://doi.org/10.1016/j.orl.2015.12.006
  11. N.G. Hall. "Scheduling problems with generalized due dates," IIE Transactions, Vol. 18, pp. 220-222, 1986. https://doi.org/10.1080/07408178608975351
  12. N.G. Hall, W. Kubiak, S.P. Sethi. "Earliness-tardiness scheduling problems II: Deviation of completion times about a restrictive common due date," Operations Research, Vol. 39, pp. 847-856, 1991. https://doi.org/10.1287/opre.39.5.847
  13. G.E. Hinton, S. Osindero, and Y.W. The. "A fast learning algorithm for deep belief nets," Neural Computation, Vol. 18, pp. 1527-1554, 2006. https://doi.org/10.1162/neco.2006.18.7.1527
  14. N.G. Hall and M.E. Posner. "Earliness-tardiness scheduling problems I: Weighted deviation of completion times about a common due date," Operations Research, Vol. 39, pp. 836-846, 1991. https://doi.org/10.1287/opre.39.5.836
  15. N.G. Hall, S.P. Sethi, and S. Srikandarajah. "On the complexity of generalized due date scheduling problems," European Journal of Operational Research, Vol. 51, pp. 100-109, 1991. https://doi.org/10.1016/0377-2217(91)90149-P
  16. K. He, X. Zhang, S. Ren, and J. Sun. "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification," Proceedings of the 2015 International Conference on Computer Vision, pp. 1026-1034, Santiago, Chile, December, 2015.
  17. A. Krizhevsky, I. Sutskever, and G. Hinton. "ImageNet classification with deep convolutional neural networks," Proceedings of 25th International Conference on Advances in Neural Information Processing Systems, pp. 1090-1098, Nevada, USA. December, 2012.
  18. Y. LeCun, Y. Bengio, and G. Hinton. "Deep learning," Nature, Vol. 521, pp. 436-444, 2015. https://doi.org/10.1038/nature14539
  19. V. Nair and G. Hinton. "Rectified linear units improve restricted Boltzmann machines," Proceedings of the 27th International Conference on Machine Learning, pp. 807-814, Haifa, Israel, June. 2010.
  20. S. Srikandarajah. "A note on the generalized due dates scheduling problem," Naval Research Logistics, Vol. 37, pp. 587-597, 1990. https://doi.org/10.1002/1520-6750(199008)37:4<587::AID-NAV3220370411>3.0.CO;2-O
  21. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. "Dropout: A simply way to prevent neural networks from overfitting," Journal of Machine Learning Research, Vol. 15, pp. 1929-1958, 2014.
  22. J. Tsitsiklis, D. Bertsekas, M. Athans. "Distributed asynchronous deterministic and stochastic gradient optimization algorithms," IEEE Transactions on Automatic Control, Vol. 31, pp. 803-812, 1986. https://doi.org/10.1109/TAC.1986.1104412
  23. L. Wa and J. Yuan. "Single-machine scheduling to minimize the total earliness and tardiness is strongly NP-hard," Operations Research Letters, Vol. 41, pp. 363-365, 2013. https://doi.org/10.1016/j.orl.2013.04.007
  24. W. Zhang, S. Gupta, X. Lian, and J. Liu. "Staleness-aware async-SGD for distributed deep learning," Proceedings of the 25th International Joint Conference on Artificial Intelligence, pp. 2350-2356, New York, USA, July. 2016.