Uncovering the Over-Smoothing Challenge in Image Super-Resolution: Entropy-Based Quantification and Contrastive Optimization

Tianshuo Xu, Lijiang Li, Peng Mi, Xiawu Zheng, Fei Chao, Rongrong Ji, Yonghong Tian, Qiang Shen

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)
4 Downloads (Pure)

Abstract

PSNR-oriented models are a critical class of super-resolution models with applications across various fields. However, these models tend to generate over-smoothed images, a problem that has been analyzed previously from the perspectives of models or loss functions, but without taking into account the impact of data properties. In this paper, we present a novel phenomenon that we term the center-oriented optimization (COO) problem, where a model's output converges towards the center point of similar high-resolution images, rather than towards the ground truth. We demonstrate that the strength of this problem is related to the uncertainty of data, which we quantify using entropy. We prove that as the entropy of high-resolution images increases, their center point will move further away from the clean image distribution, and the model will generate over-smoothed images. Implicitly optimizing the COO problem, perceptual-driven approaches such as perceptual loss, model structure optimization, or GAN-based methods can be viewed. We propose an explicit solution to the COO problem, called Detail Enhanced Contrastive Loss (DECLoss). DECLoss utilizes the clustering property of contrastive learning to directly reduce the variance of the potential high-resolution distribution and thereby decrease the entropy. We evaluate DECLoss on multiple super-resolution benchmarks and demonstrate that it improves the perceptual quality of PSNR-oriented models. Moreover, when applied to GAN-based methods, such as RaGAN, DECLoss helps to achieve state-of-the-art performance, such as 0.093 LPIPS with 24.51 PSNR on 4× downsampled Urban100, validating the effectiveness and generalization of our approach.
Original languageEnglish
Pages (from-to)6199-6215
Number of pages17
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume46
Issue number9
Early online date19 Mar 2024
DOIs
Publication statusPublished - 01 Sept 2024

Keywords

  • Analytical models
  • Contrastive learning
  • Distortion
  • Entropy
  • Image restoration
  • image super-resolution
  • information entropy
  • Optimization
  • Self-supervised learning
  • Superresolution
  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Software
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'Uncovering the Over-Smoothing Challenge in Image Super-Resolution: Entropy-Based Quantification and Contrastive Optimization'. Together they form a unique fingerprint.

Cite this