This paper proposes an analytical mode decomposition (AMD) and Hilbert transform method for structural nonlinearity quantification and damage detection under earthquake loads. The measured structural response is first decomposed into several intrinsic mode functions (IMF) using the proposed AMD method. Each IMF is an amplitude modulated-frequency modulated signal with narrow frequency bandwidth. Then, the instantaneous frequencies of the decomposed IMF can be defined with Hilbert transform. However, for a nonlinear structure, the defined instantaneous frequencies from the decomposed IMF are not equal to the instantaneous frequencies of the structure itself. The theoretical derivation in this paper indicates that the instantaneous frequency of the decomposed measured response includes a slowly-varying part which represents the instantaneous frequency of the structure and rapidly-varying part for a nonlinear structure subjected to earthquake excitations. To eliminate the rapidly-varying part effects, the instantaneous frequency is integrated over time duration. Then the degree of nonlinearity index, which represents the damage severity of structure, is defined based on the integrated instantaneous frequency in this paper. A one-story hysteretic nonlinear structure with various earthquake excitations are simulated as numerical examples and the degree of nonlinearity index is obtained. Finally, the degree of nonlinearity index is estimated from the experimental data of a seven-story building under four earthquake excitations. The index values for the building subjected to a low intensity earthquake excitation, two medium intensity earthquake excitations, and a large intensity earthquake excitation are calculated as 12.8%, 23.0%, 23.2%, and 39.5%, respectively.