Gao, Y., Lang, C., Liu, F., Foo, C.-S., Cao, Y., Sun, L., & Wei, Y. (2024). Mining Semantic Correlations Between Mispredictions and Corrections for Interactive Semantic Segmentation. IEEE Transactions on Neural Networks and Learning Systems, 1–14. https://doi.org/10.1109/tnnls.2024.3379585
Abstract:
Interactive semantic segmentation pursues high-
quality segmentation results at the cost of a small number of
user clicks. It is attracting more and more research attention
for its convenience in labeling semantic pixel-level data. Existing
interactive segmentation methods often pursue higher interaction
efficiency by mining the latent information of user clicks or
exploring efficient interaction manners. However, these works
neglect to explicitly exploit the semantic correlations between
user corrections and model mispredictions, thus suffering from
two flaws. Firstly, similar prediction errors frequently occur in
actual use, causing users to repeatedly correct them. Secondly,
the interaction difficulty of different semantic classes varies
across images, but existing models use monotonic parameters
for all images which lack semantic pertinence. Therefore, in this
paper, we explore the semantic correlations existing in corrections
and mispredictions by proposing a simple yet effective online
learning solution to the above problems, named Correction-
Misprediction Correlation Mining (CM2 ). Specifically, we lever-
age the correction-misprediction similarities to design a Con-
fusion Memory Module (CMM) for automatic correction when
similar prediction errors reappear. Furthermore, we measure
the semantic interaction difficulty by counting the correction-
misprediction pairs and design a Challenge Adaptive Convo-
lutional Layer (CACL), which can adaptively switch different
parameters according to interaction difficulties to better segment
the challenging classes. Our method requires no extra training
besides the online learning process and can effectively improve
interaction efficiency. Our proposed CM2 achieves state-of-the-
art results on 3 public semantic segmentation benchmarks.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - MTC Programmatic Funds
Grant Reference no. : Grant No. A20H6b0151