Convolutional LSTM-Based Hierarchical Feature Fusion for Multispectral Pan-Sharpening

Multispectral (MS) pan-sharpening aims at producing high-resolution (HR) MS images in both spatial and spectral domains, by merging single-band panchromatic (PAN) images and corresponding MS images with low spatial resolution. The intuitive way to accomplish such MS pan-sharpening tasks, or to reconstruct ideal HR-MS images, is to extract feature pairs from the given PAN and MS images and to fuse the results. Therefore, feature extraction and feature fusion are two key components for MS pan-sharpening. This article presents a novel MS pan-sharpening network (MPNet), including a heterogeneous pair of feature extraction pathways (FEPs) and a convolutional long short-term memory (ConvLSTM)-based hierarchical feature fusion module (HFFM). Specifically, we design a PAN FEP to extract 2-D feature maps via 2-D convolutions and dual attention, while an MS FEP is introduced in an effort to obtain 3-D representations of MS image by 3-D convolutions and triple attention. To merge the resulting hierarchical features, the ConvLSTM-based HFFM is developed, leveraging intralevel fusion, interlevel fusion, and information exchange within one single framework. Here, the interlevel fusion is implemented with the ConvLSTM to capture the dependencies among hierarchical features, reduce redundant information, and effectively integrate them via its recurrent architecture. The information exchange between different FEPs helps enhance the representations for subsequent processing. Systematic comparative experiments have been conducted on three publicly available datasets at both reduced resolution and full resolution, demonstrating that the proposed MPNet outperforms state-of-the-art methods in the literature.