Cylindrical worm reducers are generally used in various fields and forms throughout the industry, and demand is increasing due to their role as an integral part of the industry. Market trends require high-load, high-precision components, and small-sized reducers with large loads. When using a cylindrical worm reducer, a reducer designed with a reduced center distance while maintaining the same output torque results in gear wear. To overcome this difficulty, an enveloping worm gear reducer is introduced and studied. In this paper, three types of end mill tools are used to evaluate the tooth profile accuracy for each tool shape during machining of the tooth profile for a non-developed surface worm thread. The effect of the endmill shape on the accuracy of the tooth profile was analyzed by performing 3D modeling of the surrounding worm tooth profile based on the Hindley method. In this study, we analyzed tooth profile accuracy, tooth surface roughness, and tooth surface machining time, etc. Through the study, efficient machining conditions for the enveloping worm gears and the influence of parameters on the process were presented.