An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series

An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series

IEEE Transactions on Neural Networks and Learning Systems

10.1109/TNNLS.2021.3105827

https://doi.org/10.1109/tnnls.2021.3105827

Astha Garg, Wenyu Zhang, Jules Samaran, Savitha Ramasamy, Chuan-Sheng Foo

deep learning, anomaly detection, performance evaluation, Time Series

31 August 2021

Garg, A., Zhang, W., Samaran, J., Savitha, R., & Foo, C.-S. (2021). An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series. IEEE Transactions on Neural Networks and Learning Systems, 1–10. doi:10.1109/tnnls.2021.3105827

Several techniques for multivariate time series anomaly detection have been proposed recently, but a systematic comparison on a common set of datasets and metrics is lacking. This article presents a systematic and comprehensive evaluation of unsupervised and semisupervised deep-learning-based methods for anomaly detection and diagnosis on multivariate time series data from cyberphysical systems. Unlike previous works, we vary the model and post-processing of model errors, i.e., the scoring functions independently of each other, through a grid of ten models and four scoring functions, comparing these variants to state-of-the-art methods. In time-series anomaly detection, detecting anomalous events is more important than detecting individual anomalous time points. Through experiments, we find that the existing evaluation metrics either do not take events into account or cannot distinguish between a good detector and trivial detectors, such as a random or an all-positive detector. We propose a new metric to overcome these drawbacks, namely, the composite F-score (Fc_1), for evaluating time-series anomaly detection. Our study highlights that dynamic scoring functions work much better than static ones for multivariate time series anomaly detection, and the choice of scoring functions often matters more than the choice of the underlying model. We also find that a simple, channel-wise model--the univariate fully connected auto-encoder, with the dynamic Gaussian scoring function emerges as a winning candidate for both anomaly detection and diagnosis, beating state-of-the-art algorithms.

Publisher Copyright

This research / project is supported by the A*STAR - HBMS IAF-PP
Grant Reference no. : H19/01/a0/023

This research / project is supported by the A*STAR - AME Programmatic Funds
Grant Reference no. : A20H6b0151

© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

https://oar.a-star.edu.sg/communities-collections/articles/17968

2162-237X
2162-2388

Institute for Infocomm Research