Recently, starting with smart construction research, interest in technology that automates construction site management using artificial intelligence technology is increasing. In order to automate construction site management, it is necessary to recognize objects such as construction equipment or workers, and automatically analyze the relationship between them. For example, if the relationship between workers and construction equipment at a construction site can be known, various use cases of site management such as work productivity, equipment operation status monitoring, and safety management can be implemented. This study derives a real-time object detection platform architecture that is required when performing construction site management using deep learning technology, which has recently been increasingly used. To this end, deep learning models that support real-time object detection are investigated and analyzed. Based on this, a deep learning model development process required for real-time construction site object detection is defined. Based on the defined process, a prototype that learns and detects construction site objects is developed, and then platform development considerations and architecture are derived from the results.