Accurate estimation of modal parameters (i.e., natural frequency, damping ratio) of tall buildings is of great importance to their structural design, structural health monitoring, vibration control, and state assessment. Based on the combination of variational mode decomposition, smoothed discrete energy separation algorithm-1, and Half-cycle energy operator (VMD-SH), this paper presents a method for structural modal parameter estimation. The variational mode decomposition is proved to be effective and reliable for decomposing the mixed-signal with low frequencies and damping ratios, and the validity of both smoothed discrete energy separation algorithm-1 and Half-cycle energy operator in the modal identification of a single modal system is verified. By incorporating these techniques, the VMD-SH method is able to accurately identify and extract the various modes present in a signal, providing improved insights into its underlying structure and behavior. Subsequently, a numerical study of a four-story frame structure is conducted using the Newmark-β method, and it is found that the relative errors of natural frequency and damping ratio estimated by the presented method are much smaller than those by traditional methods, validating the effectiveness and accuracy of the combined method for the modal identification of the multi-modal system. Furthermore, the presented method is employed to estimate modal parameters of a full-scale tall building utilizing acceleration responses. The identified results verify the applicability and accuracy of the presented VMD-SH method in field measurements. The study demonstrates the effectiveness and robustness of the proposed VMD-SH method in accurately estimating modal parameters of tall buildings from acceleration response data.