Convolutional LSTM-Based Hierarchical Feature Fusion for Multispectral Pan-Sharpening

Dong Wang, Yunpeng Bai, Chanyue Wu, Ying Li*, Changjing Shang, Qiang Shen

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

12 Citations (Scopus)
88 Downloads (Pure)

Abstract

Multispectral (MS) pan-sharpening aims at producing high-resolution (HR) MS images in both spatial and spectral domains, by merging single-band panchromatic (PAN) images and corresponding MS images with low spatial resolution. The intuitive way to accomplish such MS pan-sharpening tasks, or to reconstruct ideal HR-MS images, is to extract feature pairs from the given PAN and MS images and to fuse the results. Therefore, feature extraction and feature fusion are two key components for MS pan-sharpening. This article presents a novel MS pan-sharpening network (MPNet), including a heterogeneous pair of feature extraction pathways (FEPs) and a convolutional long short-term memory (ConvLSTM)-based hierarchical feature fusion module (HFFM). Specifically, we design a PAN FEP to extract 2-D feature maps via 2-D convolutions and dual attention, while an MS FEP is introduced in an effort to obtain 3-D representations of MS image by 3-D convolutions and triple attention. To merge the resulting hierarchical features, the ConvLSTM-based HFFM is developed, leveraging intralevel fusion, interlevel fusion, and information exchange within one single framework. Here, the interlevel fusion is implemented with the ConvLSTM to capture the dependencies among hierarchical features, reduce redundant information, and effectively integrate them via its recurrent architecture. The information exchange between different FEPs helps enhance the representations for subsequent processing. Systematic comparative experiments have been conducted on three publicly available datasets at both reduced resolution and full resolution, demonstrating that the proposed MPNet outperforms state-of-the-art methods in the literature.

Original languageEnglish
Article number5404016
JournalIEEE Transactions on Geoscience and Remote Sensing
Volume60
Early online date20 Aug 2021
DOIs
Publication statusPublished - 26 Jan 2022

Keywords

  • Computer architecture
  • Feature extraction
  • Fuses
  • Information exchange
  • Remote sensing
  • Spatial resolution
  • Task analysis

Fingerprint

Dive into the research topics of 'Convolutional LSTM-Based Hierarchical Feature Fusion for Multispectral Pan-Sharpening'. Together they form a unique fingerprint.

Cite this