Unsupervised and semi-supervised anomaly detection using data-driven machine learning

Google Research has published two new papers addressing the challenges of Anomaly Detection (AD), an important task in numerous real-world applications such as detecting defective products from vision sensors in manufacturing, fraudulent behaviours in financial transactions, and network security threats. The Transactions on Machine Learning Research (TMLR) articles present, respectively, a novel unsupervised AD framework and a novel semi-supervised AD framework.

The first paper, titled “Self-supervised, Refine, Repeat: Improving Unsupervised Anomaly Detection,” focuses on unsupervised anomaly detection (AD) and proposes a novel framework based on self-supervised learning without labels and iterative data refinement based on the agreement of one-class classifier (OCC) outputs. This methodology is evaluated on multiple datasets from various domains, including benchmarks for semantic AD, real-world manufacturing visual AD, and real-world tabular AD. Compared to a state-of-the-art one-class deep model on CIFAR-10, this framework improves AD performance by more than 15.0 average precision (AP) with a 10% anomaly ratio, while maintaining outstanding performance on MVTec with less than a 1.0 AUC reduction at a 10% anomaly ratio. It also surpasses a state-of-the-art single-class classifier on Thyroid by 22.9 F1 points and 2.5% anomaly ratio (tabular data).

The second work, titled ‘SPADE: Semi-supervised Anomaly Detection under Distribution Mismatch,’ focuses on semi-supervised AD and presents a robust pseudo-labeling method employing an ensemble of OCCs and a judicious combination of supervised and self-supervised learning. Multiple AD datasets for picture and tabular data, as well as real-world financial fraud detection datasets, are utilised for evaluation. It regularly outperforms alternatives on both datasets, taking use of unlabeled data and demonstrating robustness to changing distributions, according to the obtained results.

The presented frameworks have the potential to have a considerable impact on the field of AD, as their performance in a variety of circumstances is promising. In addition, they provide efficient methods that eliminate the reliance on manual labels during model training, enabling high-performance AD with limited labelling resources.