Learning Steerable Function for Efficient Image Resampling

1University of Science and Technology of China,

2Huawei Noah’s Ark Lab

*Equal contribution #Correspondence author

In CVPR 2023

Abstract

Image resampling is a basic technique that is widely employed in daily applications. Existing deep neural networks (DNNs) have made impressive progress in resampling performance. Yet these methods are still not the perfect substitute for interpolation, due to the issues of efficiency and continuous resampling. In this work, we propose a novel method of Learning Resampling Function (termed LeRF), which takes advantage of both the structural priors learned by DNNs and the locally continuous assumption of interpolation methods. Specifically, LeRF assigns spatially-varying steerable resampling functions to input image pixels and learns to predict the hyper-parameters that determine the orientations of these resampling functions with a neural network. To achieve highly efficient inference, we adopt look-up tables (LUTs) to accelerate the inference of the learned neural network. Furthermore, we design a directional ensemble strategy and edge-sensitive indexing patterns to better capture local structures. Extensive experiments show that our method runs as fast as interpolation, generalizes well to arbitrary transformations, and outperforms interpolation significantly, e.g., up to 3dB PSNR gain over bicubic for x2 upsampling on Manga109.

this slowpoke moves

Continuous Resampling Resulsts

Homographic Transformation

Learn More

Related Works

LUT acceleration is a technique to cache learned neural networks into look-up tables for highly efficient inference. Please learn more about LUT acceleration at MuLUT.

BibTeX

@InProceedings{Li_2022_ECCV,
      author    = {Li, Jiacheng and Chen, Chang and Cheng, Zhen and Xiong, Zhiwei},
      title     = {MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution},
      booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
      year      = {2022},
  }

@arxiv{Li_2023_DNN_LUT,
  author    = {Li, Jiacheng and Chen, Chang and Cheng, Zhen and Xiong, Zhiwei},
  title     = {Toward {DNN} of {LUTs}: Learning Efficient Image Restoration with Multiple Look-Up Tables},
  booktitle = {arxiv},
  year      = {2023},
}

  @InProceedings{Li_2023_CVPR,
    author    = {Li, Jiacheng and Chen, Chang and Huang, Wei and Lang, Zhiqiang and Song, Fenglong and Yan, Youliang and Xiong, Zhiwei},
    title     = {Learning Steerable Function for Efficient Image Resampling},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {5866-5875}
}

Acknowledgement

We would like to thank colleagues at Noah's Ark Lab for helpful discussions.