Pansharpening is a process used to extract high spatial resolution from panchromatic images and high spectral resolution from multispectral images at the same time. The intensity–hue–saturation (IHS) is the simplest and most efficient fusion method in current fusion methods, which could preserve spatial information well, but there is some spectral distortion. The main reason is that, in the IHS space, the estimated intensity image used to replace the original intensity image is not accurate enough. A method is proposed to overcome this defect, which aims to combine the fast generalized IHS (FGIHS) fusion technique based on spectral adjustment with a variational optimization model to estimate a more accurate intensity image. In the variational model, it is assumed that the down-sampled intensity image should be close to the original intensity image, retaining the spectral information effectively. The experimental results show that this method not only preserves the spectral information effectively but also improves the spatial quality of the fusion results in the view of the subjective and objective evaluations.