Robust Motion Compensated Frame Interpolation Using Weight-Overlapped Block Motion Compensation with Variable Block Sizes to Reduce LCD Motion Blurs

Abstract

Liquid crystal displays (LCDs) have slow responses, so motion blurs are often perceived in fast moving scenes. To reduce this motion blur, we propose a novel method of robust motion compensated frame interpolation (MCFI) based on bidirectional motion estimation (BME) and weight-overlapped block motion compensation (WOBMC) with variable block sizes. In most MCFI methods, a static block size is used, so some block artefacts and motion blurs are observed. However, the proposed method adjusts motion block sizes and search ranges by comparing matching scores, so the precise motion vectors can be estimated in accordance with motions. In the MCFI, overlapping ranges for WOBMC are also determined by adjusted block sizes, so the accurate MCFI can be performed. In the experimental results, the proposed method strongly reduced motion blurs arisen from large motions, and yielded interpolated images with high visual performance and peak signal-to-noise ratio (PSNR).

I. INTRODUCTION

Liquid crystal displays (LCDs) have slow responses as compared with cathode ray tube (CRT) displays because of LCD hold time [1-3], so ghost phenomenon or motion blurs are often perceived around fast moving objects in LCDs. To reduce the motion blurs, backlight flashing [4], signal overdrive [5], or frame rate up-conversion (FRUC) [6-12] can be used. However, luminance fluctuation may be seen when the flashing rate is not high enough in backlight flashing, and signal overdrive does not consider image contents [1]

To increase video frame rates for high temporal resolution, frame repetition, back/gray frame insertion and linear interpolation may be used; however, these methods cannot remove motion blurs and judders because they do not use motion information [13]. To produce unaffected frames without block artefacts and motion blurs, motion compensated frame interpolation (MCFI) must be used, where motion vectors (MVs) must be correctly estimated by means of various motion estimation (ME) methods.

The conventional MCFI utilizes unidirectional ME (UME); forward ME, backward ME and bilateral ME [14-17]. The critical drawback of the UME is shown in Fig. 1. Forward and backward MEs cannot estimate all pixels’ MVs, so holes and overlaps result from multiple MVs and no MVs. Meanwhile, bilateral ME may not estimate correct MVs when a matching score of a block including moving objects is lower than matching scores of other blocks. In the result of MCFI using bilateral ME of Fig. 1, the block including a moving ball is not compensated [18].

FIG. 1.Various UME methods for MCFI.

For the outstanding interpolation results, bidirectional ME (BME) and overlapped block motion compensation (OBMC) are used [19-22]. The results of BME + OBMC are very sensitive to the sizes of blocks and search regions for block matching as shown in Fig. 2. When the sizes are small, the large motions are not estimated, while the detailed motions are not compensated and huge processing time is needed when large sizes are utilized.

FIG. 2.BME + OBMC using various block sizes and search regions for FRUC.

Various methods have been proposed to improve the performance or computation of FRUC [23-25]. Regions overlapping with large or complex motions are not, however, correctly compensated. To solve this problem, variable block sizes and overlapping ranges are needed. A method using variable block sizes was presented [26]; however, this used block splitting by quad scales (coarse-grained block size is 16×16, and fine-grained block size is 4×4), and refined motion estimation is always performed for all blocks whether or not a block is located at an object boundary.

This paper is aimed at developing a novel FRUC method that robustly compensates regions overlapped with large or complex motions. Accordingly, the proposed method uses variable block sizes and overlapping ranges. The block sizes are enlarged by checking a matching score, and the weighted overlapping ranges (WOBMC, weight-OBMC) are also enlarged by the ratio of block enlargement. Therefore, the block sizes and the weighted overlapping ranges can be suitably determined according to the movement and the size of objects, so the proposed method can interpolate frames with high visual and objective performance.

 

II. MCFI USING BME AND WOBMC WITH VARIABLE BLOCK SIZES

To estimate forward MVs, the proposed method uses a block matching algorithm (BMA) using mean of absolute differences (MAD), which can directly compare matching scores for blocks with variable sizes in contrast to sum of absolute differences (SAD) [27], in the direction from the current frame (It) to the previous frame (It−1) as follows:

where the block size is b×b, b is an odd positive integer, and i and j are integers, where b/2 is the largest integer that is less than or equal to the real value of b/2. The block sizes to estimate MVs are normally used as even windows; however, we use odd windows to estimate pixel’s MVs instead of block’s MVs. Therefore, the MV of It(x, y) can be determined by

where u∈[x−r/2, x+r/2] and v∈[y−r/2, y+r/2] so the search range is r×r and r is an odd integer.

To estimate an MV at pixel It(x, y) with variable block sizes, an initial block size is set to b0×b0, and its matching block is found within the initial search range r0×r0. If the matching score S0, which is the reciprocal of the MAD, is less than τm, the block size and the search range are set to α1b0 and α1r0×α1r0, respectively (α>1); otherwise the MV computed in the initial size and the initial expansion rate (α0=1) are stored without expansion. If the matching score S1 using the enlarged block and search range is less than τm (matching score threshold for expansion of block sizes), the block size and the search region are enlarged by α2 to compute large MVs; otherwise the MV and the expansion rate are stored and the iterative expansion is terminated. When the expansion rate αk is greater than τα (expansion limit), the iterative adjustment is terminated and the MV and the αk with the greatest matching score are selected. The ME using variable block sizes is shown in Fig. 3. Backward MVs are estimated in the same manner as forward MVs, and are found by using the direction from the previous frame to the current frame.

FIG. 3.Concept of motion estimation using variable block sizes.

Using the estimated forward and backward MVs, vf and vb, WOBMC is performed to interpolate the intermediate frame It−0.5 as shown in Fig. 4. In the conventional OBMC, motion blocks are enlarged by predetermined sizes, and then overlapped region are estimated by the weights. Whereas the weight for the motion compensation (MC) is the 2-dimensional Gaussian kernel, and pixels within the overlapping range (all pixels within the enlarged block) are compensated in proposed WOBMC. The overlapped range is assigned as αkO0, where O0 is the initial overlapping range for b0.

FIG. 4.WOBMC with variable block sizes.

Finally, the brightness of the interpolated frame may be reduced owing to the OBMC; the method compensates for this by using the difference with the interpolated image using the averaging Mt−0.5 = (It+It−1) / 2. The compensated brightness is

where denotes the low-pass filtering of f and the windows size of the median filter is large for large motion regions. In our method, the median filter is used for , because the median filter can ignore irregular variations.

 

III. EXPERIMENTAL RESULTS

To evaluate the proposed method, the proposed method tested 12 image sequences; CIF (common intermediate format, 352×288) with 30 Hz, full HD (high definition, 1920×1080) with 50 Hz, and used 51 odd frames and produced 50 intermediate frames. The 50 images were compared with the 50 real even frames to evaluate their peak signal-to-noise ratio (PSNR) as shown in Fig. 5. The initial sizes of the block, search range and overlapping range are set to 5×5, 9×9 and 7×7, respectively; other parameters were selected as follows: is the typical value for upscaling of images, and τα =7, and τm= 0.05 were experimentally selected to reduce computation complexities and to find optimal block sizes. Some test results with variable parameter values are shown in Table 1. Although the PSNR results with , τα =7 and τm= 0.05 are not the highest, we used these parameters to consider the computation complexities. Therefore, the parameter values can be changed for the variable environments of FRUC such as characteristics of image sequences, hardware performance and network environments; however, we used , τα =7 and τm= 0.05 in these experiments to objectively compare the proposed method. In addition, one MV is estimated for 2×2 pixels to reduce complexities.

FIG. 5.Quality comparison.

TABLE 1.Comparison of Variable Parameter values (PSNR [DB] and processing time [ms])

Visual comparison of the results of OBMC, WOBMC and the proposed method is shown in Fig. 6. The result images interpolated by WOBMC have fewer block artefacts than the images interpolated OBMC in almost all regions because WOBMC uses weights of all pixels. The proposed method, moreover, removed most block artefacts and motion blurs, and was able to produce unaffected intermediate frames owing to the variable block sizes

FIG. 6.Visual comparison of result images (‘foreman’ sequence); (a) Results of OBMC, (b) Results of WOBMC, and (c) Results of the proposed method.

Some result images are shown in Fig. 7; many regions of motion blurs in averaging images (Fig. 7(a)) are observed, small motion regions are compensated in the results by static block size (Fig. 7(b), green boxes), and most motion regions are more clearly interpolated by the proposed method (Fig. 7(c), red boxes). In the results of our method, the face region is preserved in the ‘foreman’ sequence, and the shape of the boat and the river bank regions is clearly interpolated in the ‘coastguard’ sequence. The spectators are clearly separated, the text of the billboard is distinctly shown, and the motion blur of the play is reduced in the ‘stefan’ sequence. Especially, the table tennis ball is shown with one object in the ‘tennis’ sequence.

FIG. 7.Sample result images (from top to bottom: ‘foreman’, ‘coastguard’ ‘stefan’ and ‘tennis’ sequences); (a) Images interpolated by averaging, (b) Images interpolated by static block size, and (c) Results of the proposed method.

For objective comparison, the proposed method was compared with other recent methods [23-26] as shown in Table 2. In most test sequences, the PSNRs are higher than those of other methods. Especially, in sequences having large motions, such as ‘football’, ‘tennis’ and ‘stefan’, the performance of the proposed method is improved. In Fig. 8, the resultant images of the proposed method and other conventional methods [23-26] are shown (significant motion blurs and block artifacts are marked as red regions). Even though variable block size method [26] can reduce some motion blurs, large motion blurs are not corrected; however, the proposed method reduced most motion blurs and block artifacts.

TABLE 2.PSNR Comparison with Other Methods (PSNR [DB])

FIG. 8.Sample result images (from top to bottom: replica [23], replica [24], replica [25], replica [26] and the proposed method); (a) ‘flower’, (b) ‘football’, and (c) ‘stefan’ sequences.

In the test of full HD image sequences, the results of our method are very similar to the original frames as shown in Figs. 9 and 10, where difference values between the results of the proposed method and the original images are very low values.

FIG. 9.Sample result images (‘into tree’ Full HD, from top to bottom: interpolated images and difference images with an original image); (a) Averaging (PSNR: 30.95dB) and (b) Proposed method (PSNR: 35.83dB).

FIG. 10.Sample result images (‘crowd run’ Full HD, from top to bottom: interpolated images and difference images with an original image); (a) Averaging (PSNR: 25.21dB) and (b) Proposed method (PSNR: 29.82dB).

To evaluate the complexities of the proposed method, we compared the proposed method with a symmetric motion estimation method with variable search ranges [28], which was implemented by software. The Lim’s method [28] uses variable search ranges, while our method uses variable MV sizes and variable search ranges. The average PSNR of the proposed method is a little higher than the Lim’s method, while our method can speed up about 144 times on average. In addition, we compared our method and the replica of Lee’s method [26] as shown in Table 3, where we tested the proposed method with two parameter sets; one set is the same parameters of the above experiments, and the other set (say speed-up parameters) is 7×7 and 5×5 for initial search range and overlapping range, respectively, and one MV is estimated for 5×5 pixels. The proposed method has the most complexity; however, the result with speed-up parameters has the least complexity, and has higher PSNR than the result of the replica of Lee’s method [26].

TABLE 3.Comparison with replica [26] (PSNR [DB] and processing time [ms])

The average processing time of the proposed method for 10 CIF sequences is 4,746 ms; however, this is the processing time under software implementation. The processing time of variable block size ME realized by dedicated hardware for HD images is fast enough for real-time processing [29, 30], so the proposed method can be used for real-time FRUC processing by hardware implementation.

 

IV. CONCLUSIONS

A FRUC method using WOBMC with variable block sizes has been proposed in this paper; in the proposed method, bidirectional MVs are estimated using variable block sizes and search ranges comparing matching scores. The overlapping range for WOBMC is also enlarged by using the ME expansion rate. In addition, brightness compensation is utilized by comparing with the median filtered image of averaging interpolation. Therefore, the proposed method can provide high-performance FRUC regardless of the magnitude and complexity of motions to reduce LCD motion blurs.