This paper investigates the problem of noise-induced annoyance level evaluation, and proposes a novel annoyance measurement metric for more efficient and accurate evaluation of annoyance level of different types of noises. Results from a large-scale subjective listening test using 90 different noise clips and 96 subjects show that the proposed method can produce more consistent and reliable annoyance ratings than the widely adopted ISO method. Based on the subjective test results, we further develop an objective noise annoyance level measurement model based on the selected psychoacoustic features extracted from the noise samples. Our evaluation results show that the objective model produces satisfactory prediction accuracy on noise annoyance level.