Most facial emotion recognition algorithms assume that the face is near frontal and the face pose fixed during the recognition process. However, such constrain limits the adoption for real-world applications. To solve this, poseinvariant descriptor for emotion recognition is required. This work proposes a novel pose-invariant dynamic descriptor that encodes the relative movement information of facial landmarks. The proposed feature set is able to handle speed variations and continuous head pose variations while the subject is expressing an emotion. In addition, the proposed method is fast and thus can be realize real-time implementation for real world application. Performance evaluation done using three publicly available databases; Cohn-Kanade (CK+), Amsterdam Dynamic Facial Expression Set (ADFES), and Audio Visual Emotion Challenge (AVEC 2011) showed that our proposed method outperforms the state-of-the-art methods.