Opportunity
In real-world applications such as video surveillance and autonomous driving, images captured through windows or camera lenses often suffer from raindrop occlusion, which severely degrades visibility and hampers downstream computer vision tasks. While existing raindrop removal methods, including convolutional neural network (CNN)-based and transformer-based approaches, have made progress, they are primarily designed for uncompressed or lightly compressed images. In practice, images are routinely compressed using standards like JPEG to reduce storage and transmission overhead. This compression introduces additional distortions, such as loss of high-frequency details and blocking artifacts. When raindrop occlusion and compression artifacts are intertwined, the resulting hybrid degradation becomes significantly more complex. Raindrops obscure background textures, while compression erodes high-frequency information in non-raindrop regions. Moreover, block-based coding paradigms can partition raindrops into irregular shapes, corrupting rain-related context and deviating raindrop distributions. Existing methods lack the capability to effectively address this combined challenge, creating a critical gap in restoring compressed images contaminated by raindrops for reliable real-world deployment.
Technology
The patent introduces a novel transformer-based architecture named HFGlobalFormer, specifically designed to restore compressed images with raindrops. The core innovation lies in a multi-level approach that integrates global contextual modeling with high-frequency detail recovery. At the framework level, the Low-High-Frequency Transformer (LHFT) module employs dual complementary branches: a low-frequency branch using a self-attention mechanism (e.g., Relative Position Multi-Head Self-Attention) to capture global contextual information under raindrops, and a high-frequency branch utilizing a specially designed High-Frequency Depth-wise Convolution (HFDC) to extract local high-frequency details lost due to compression. At the component level, the HFDC incorporates zero-mean kernels, a constraint inspired by traditional high-pass filters, which suppresses direct current bias and enhances the extraction of high-frequency-dominated features. At the module level, a Low-High-Attention Module (LHAM) adaptively fuses the features from both branches by allocating channel-wise and window-wise importance weights, allowing the model to adjust to diverse degradation patterns across different image regions and channels. The overall system is structured as a hierarchical U-shaped encoder-decoder network with residual learning, facilitating multi-scale feature extraction and reconstruction. This integrated design enables simultaneous removal of raindrops and restoration of compression-induced artifacts.
Advantages
- Outperforms existing raindrop removal methods on compressed images across various compression rates (quality factors), as demonstrated by higher PSNR and SSIM metrics.
- Effectively addresses the hybrid degradation of combined raindrop occlusion and compression artifacts, a scenario not adequately handled by prior art.
- Introduces a high-frequency-friendly design (HFDC with zero-mean kernels) that specifically enhances the recovery of lost textural details.
- Achieves adaptive feature fusion through LHAM, optimizing the combination of low-frequency and high-frequency information based on channel-specific and content-specific characteristics.
- Maintains computational efficiency, achieving superior performance without increasing multiply-accumulate operations (MACs) compared to existing methods.
- Establishes and validates performance on a dedicated JPEG compressed raindrop image dataset, addressing a previously unexplored practical problem.
Applications
- Autonomous driving systems, to clean raindrop-obscured and compressed video feeds for improved object detection and scene understanding.
- Video surveillance and security cameras, enhancing image clarity from outdoor cameras subject to rain and bandwidth-limited compression.
- Mobile photography and smartphone applications, improving photos taken through rainy windows or with lenses affected by water droplets.
- Remote sensing and aerial imaging, where transmitted images may be compressed and affected by atmospheric conditions.
- Teleconferencing and live streaming, mitigating visual degradation from rain on windows and network-induced compression.
- Medical imaging and diagnostic systems, potentially applied to enhance compressed images affected by similar obstructive artifacts.
