Distributed Video Compressive Sensing Reconstruction by Adaptive PCA Sparse Basis and Nonlocal Similarity

Abstract

To improve the rate-distortion performance of distributed video compressive sensing (DVCS), the adaptive sparse basis and nonlocal similarity of video are proposed to jointly reconstruct the video signal in this paper. Due to the lack of motion information between frames and the appearance of some noises in the reference frames, the sparse dictionary, which is constructed using the examples directly extracted from the reference frames, has already not better obtained the sparse representation of the interpolated block. This paper proposes a method to construct the sparse dictionary. Firstly, the example-based data matrix is constructed by using the motion information between frames, and then the principle components analysis (PCA) is used to compute some significant principle components of data matrix. Finally, the sparse dictionary is constructed by these significant principle components. The merit of the proposed sparse dictionary is that it can not only adaptively change in terms of the spatial-temporal characteristics, but also has ability to suppress noises. Besides, considering that the sparse priors cannot preserve the edges and textures of video frames well, the nonlocal similarity regularization term has also been introduced into reconstruction model. Experimental results show that the proposed algorithm can improve the objective and subjective quality of video frame, and achieve the better rate-distortion performance of DVCS system at the cost of a certain computational complexity.

1. Introduction

The basic idea of Compressive Sensing (CS) is to sample the signal by the way of direct dimensionality reduction while compressing the signal and then recover the original signal by exploiting the sparse prior of signal. Due to its ability to sample signal at the sub-Nyquist rate, the theory of CS has been widely applied into the various fields of image and video processing [1],[2]. The measurement approach of CS is realized by the linear inner-product and thus it has a low computational complexity, however, it requires the high computational costs to non-linearly reconstruct signal. This feature of light coding and heavy decoding makes CS theory easily be combined into the Distributed Video Coding (DVC) [3], which produces a new video compression technology -- Distributed Video Compressive Sensing (DVCS) [4]-[6].

In the DVCS system, the primary problem is the requirement of the huge memory burden in CS measurement. Currently there are two schemes to effectively resolve this problem. The first method is to use the Structurally Radom Matrices (SRMs) [7],[8] to achieve the measurement data. The SRMs use the fast orthogonal transformation to realize CS measurement, and thus avoid to construct the measurement matrix requiring lots of memory. The another method is to perform CS measurement by the Block Compressed Sensing (BCS) [9]. This approach can not only realize a low-memory CS measurement but also measure and transmit the video block one by one, and therefore it is very appropriate for the real-time applications and widely used in various DVCS systems [10]-[11]. The DVCS firstly divides the video stream into the key frames and non-key frames. The key frame can realize codec by either the traditional video coding technology (e.g., H.264) or measuring video frame at a higher measurement rate and using still-image CS reconstruction algorithm [12]-[14] to recover the original video frame. Due to the low measurement rate of non-key frames, its reconstruction requires to combine intra and inter frame correlation. Ref. [5] uses the previous and following frames to interpolate the Side Information (SI) of non-key frame by motion compensation and then regards the SI as the initial solution of GPSR algorithm [15] to construct the final interpolated frame. Ref. [6] uses the temporal-neighboring blocks to construct the sparse dictionary of each interpolated block in the non-key frame and then performs the appropriate minimum l1-norm algorithm to predict the SI, and finally reconstructs the residual frame between the SI and original frame by using the still-image CS reconstruction algorithm. Ref. [16] firstly uses CS reconstruction algorithm to independently perform intra-frame recovery and then utilizes the previous and following frames to predict the SI by motion estimation and motion compensation, and finally recovers the residual. Ref. [17] uses the Multiple Hypotheses (MH) concept in the traditional video coding to construct the candidate set of each interpolated block, and then replaces the sparse regularization item in the way of l1-norm with the Tikhonov regularization item in the way of l2-norm to predict the SI of non-key frame, and this method can effectively improve the predictive precision and reconstruction speed.

Although the above methods can obtain the better reconstructed quality of non-key frame, there are still the two defects: (a) the sparse dictionary cannot adaptively change in terms of the reconstructed quality of reference frame and remove the noise; (b) they only use the sparse prior and overlook the other prior knowledge of video frame. Aim to the first defect, an adaptive construction of sparse dictionary is proposed in this paper. Firstly, it uses the motion information between frames to find the best-matching block in reference frames of each interpolated block and extracts its temporal-neighboring blocks to produce the data matrix. Due to the noises existing in the reference frames, the Principle Components Analysis (PCA) is then used to compute the significant principle components, and finally these significant principle components are used to construct the sparse dictionary. The PCA-based sparse dictionary has a big correlation with the interpolated block, and therefore it can exploit the sparse property of non-key frame to improve the accuracy of reconstruction. For the second defect, this paper uses the Non-local Similarity (NL) of video frame to model the regularization item and combines the sparse prior knowledge to generate the joint CS reconstruction model, and finally an appropriate reconstruction algorithm is designed to solve the joint model. Since the NL is help for preserving edge details and suppressing noises, the proposed joint model can improve the performance of CS reconstruction algorithm. Experimental results show that the proposed joint reconstruction algorithm can effectively improve the rate-distortion performance of DVCS system and achieve the better objective and subjective quality of reconstructed non-key frame.

 

2. Framework of Proposed DVCS System

The framework of proposed DVCS system is shown in Fig. 1. The original video stream is firstly divided into key frames and non-key frames, and they are measured by the BCS proposed by Ref. [9]. An Ic×Ir video frame xt with N = Ic×Ir pixels in total is divided into L small blocks with size of B×B. Let xt,n represents the vectorized signal of the n-th block though raster scanning, and each block xt,n is measured by using the same Gaussian random measurement matrix ΦB, and the corresponding output CS vector yt,n with the length MB can be obtained. The above process can be described as

Fig. 1.Framework of proposed DVCS system

The measurement rate is defined as S = MB/B2. When the non-key frame is reconstructed jointly, the reconstruction quality of previous and following frame can affect seriously the performance of joint reconstruction model. Therefore, the measurement rate SK of key frame should be higher than the measurement rate SNK of non-key frame. The high measurement rate of key frame guarantees also the better reconstruction quality by only using the still-image CS reconstruction algorithm to independently reconstruct key frame, and therefore the key frame is also called as I frame.

Since the non-key frame is measured at a low measurement rate, the sufficient employment of inter-frames and spatial correlation can just guarantee the high quality of reconstructed non-key frame. If the previous key frame is only used, then the current non-key frame is called as P frame. If the previous and following frame are both used, then the current non-key frame is called as B frame. The adaptive PCA sparse dictionary and non-local similarity can be generated by using the neighboring reference frames and the current non-key frame, and then they are used to construct joint reconstruction model, and then the corresponding algorithm is performed to solve the SI xSI of current non-key frame. To further improve the reconstruction quality of non-key frame, the residual between SI and original frame is reconstructed , and the steps are described as follows,

Step 1) Initialization: xt(0) = xSI, the initial iteration k is set to 0, the maximum number iterations maxiter is set to 5.

Step 2) The CS measurement of residual between SI and original frame can be calculated as

Step 3) The residual frame rt,n(k) is computed by using BCS-SPL-DCT algorithm proposed by Ref. [13], and the k+1 iteration solution xt(k+1) can be get as follows,

Step 4) k = k+1, if k ≤ maxiter and ║rt,n(k)║2 ≥ 10-4·N, then go back to Step 2) and continue to the process of iteration, otherwise stopping the iteration.

 

3. Proposed Joint CS Reconstruction

3.1 Construction of Adaptive PCA Sparse Dictionary

Since the statistic characteristic of video frame is non-stationary, there is not the best fixed sparse dictionary (e.g., DCT dictionary, wavelet dictionary, etc.). To exploit the sparse property of video frame, the adaptive sparse dictionary correlated with the content of video frame should be constructed. Ref. [6] and Ref. [7] use directly the temporal-neighboring blocks to construct sparse dictionary, however, although this dictionary can adaptively be adjusted with the variational statistic characteristic of video frame, it cannot always keep the high correlation with the interpolated block. The main reasons of this problem have the following two points: (a) the motion information between frames; (b) the reconstructed key frames contain some noises. To overcome the above defects, we firstly use the CS measurement of the interpolated block to do motion estimation and find its best matching block in the reference frame, and then the spatial neighboring blocks of the best matching block in the reference frame are extracted to generate the data matrix. However, the data matrix contains a certain noises, and therefore the PCA is used to compute the principle components of data matrix, and then we select the significant principle components to construct the final sparse dictionary to suppress the noises. Take the situation of P frame as a example, the concrete construction steps of proposed sparse dictionary are described as follows:

Step 1) Suppose the CS measurement of the interpolated block xt,n is yt,n. Due to the Restricted Isometry Property (RIP) [18] of Gaussian measurement matrix, the matching error between xt,n and the candidate matching block xc,j retains approximately unchanged, i.e.,

Therefore, the block-matching based motion estimation can be performed in the measurement domain as follows:

where S1 is the search window with size of 2S1×2S1. As shown in Fig. 2, we extract the blocks xp,k with size of B×B pixel-by-pixel in the search window with the centre xb,n, and then each extracted block is converted into the vector by raster scanning, and all extracted blocks are combinend into the data matrix Xp = [xp,1, xp,2, …, xp,K] in which K = 2S2×2S2.

Fig. 2.Illustration of data matrix Xp construction

Step 2) Each block xp,k in data matrix Xp contains noises, and thus it is not the best scheme that Xp is directly regarded as the sparse dictionary. The PCA can compute the orthogonal transformation matrix P which can remove the redundant information between pixels in xp,k. If P is used to transform image blocks, and the useful information and noises of Xp can be effectively divided. Firstly, the covariance matrix Ωp with size of d×d (d = B2) corresponding to Xp can be calculated as follows,

and then we can compute d eigenvalues η1 ≥ η2 ≥ … ≥ ηd of the covariance matrix Ωp and their corresponding normalized eigenvectors (principle components) p1, p2, …, pd, and finally we can construct the orthogonal transformation matrix P = [p1, p2, …, pd].

Step 3) To effectively divide noises and useful information in the data matrix Xp, we should be to find the sparse dictionary Dn which can sparsely represent all blocks in Xp as far as possible, i.e., the Dn should satisfy the following formula,

where Ʌn is coefficient matrix of Xp, ║·║F is Frobenius norm. The r significant principle components in P are used to generate the dictionary Dn,r = [p1, p2, …, pr], and the coefficient matrix Ʌn can be simply calculated by Ʌn,r = Dn,rT·Xp. The reconstruction error ║Xp - Dn,rɅn,r║F2 in Eq. (8) will decrease as r increases, and the item ║Ʌn,r║1 is otherwise increasing. Therefore, the best value r* of r can be selected by the following formula,

Finally, the sparse dictionary Dn = [p1, p2, …, pr*] of the interpolated block xt,n can be achieved.

Step 4) The CS reconstruction model can be constructed by using the Dn from PCA training as follows,

The sparse representation αt,n of xt,n is obtained by using GPSR algorithm to solve Eq. (10), and finally the interpolated block is reconstructed by

3.2 Non-local Similarity Regularization Item

Although the adaptive PCA sparse dictionary can exploit the sparse property of video frame, it cannot preserve edge and texture features well since the edge and texture features have a low sparse degree. Fig. 3 shows that the reconstructed Foreman 13-th frame (P situation) when the measurement rate SNK is 0.1 and block size B is 16. It can be observed that edge and texture regions appear the obvious blurring and blocking artifacts. Therefore, to retain the clear edge and texture details, in addition to using the sparse priori knowledge, the other priori knowledge requires also to be added.

Fig. 3Comparison between original frame and the reconstructed frame from adaptive PCA sparse dictionary for Foreman 13-th frame.

For image and video, the pixel is not isolated but jointly describe the image features with its neighboring pixels. The window with center pixel (it is also called as patch) can usually present details of a pixel. The center of patch is corresponding to a pixel of image, then an image can be represented by the over-complete set composed by all patches. In the edge and texture regions usually exist lots of periodical repetitive patterns and they have a high self-similarity, and therefore the patches locating at the different positions have a strong similarity. This property of image and video is called as non-local similarity [19]-[21]. The non-local similarity of video presents that patches have not only spatial correlation but also temporal correlation. As shown in Fig. 4, the patch labeled by red color and the patch labeled by blue color can find the similar patches in spatial and temporal neighboring regions. The non-local similarity is very helpful to improve the quality of reconstructed frame, especially for preserving edge and texture structure features, and therefore this property can become a priori knowledge to mix into Eq. (10) and effectively remove the blurring and blocking artifacts in edge and texture regions.

Fig. 4.Non-local similarity of video

Take the P situation as an example, the following content describes the construction of non-local similarity regularization item in details. Any pixel in xt,n is denoted as xt,n(i), i = 1,2, …,d, and xt,n(i) denotes the patch whose center and radius are xt,n(i) and b respectively. For each patch xt,n(i), we find its similar patches in the current block xt,n and the best-matching block xb,n in the previous frame, and each patch xt,nm(i) should satisfy eim = ║ xt,n(i) -xt,nm(i)║2 ≤ t, therefore xt,n(i) can be predicted by

where ni is the additional noise item. Suppose βi is the vector containing all elements βim, xt,nm(i) corresponding to βim can be generated as gi, and thus Eq. (12) can be equal to

Considering the non-local similarity of video, the predictive error ║xt,n(i)- βiT·gi║2 should be smaller, and thus it can be regarded as the regularization item to mix Eq.(10) as follows,

where λ2 is the regularization factor used to balance the non-local similarity item. Eq. (15) can be equal to

where I is the identify matrix, Hn,1 and Hn,2 satisfy

To solve Eq. (16), it can be further simplified as the following l1-l2 norm minimum model,

where

Since the construction of Hn,1 and Hn,2 requires the interpolated block xt,n, however xt,n is unavailable in the process of reconstruction. Therefore, Hn,1 and Hn,2 will be updated using the iteration solution in the process of solving Eq. (19). The steps of solving Eq. (19) are described as follows,

Step 1) Initialization:

a) the initial solution xt(0) is firstly acquired by using Eq. (10) and Eq. (11);

b) H(0)n,1 and H(0)n,2 are constructed by using the initial solution xt(0) in term of Eq. (17) and Eq. (18), and then we use them to generate ỹ(0)t,n and Φ(0)n;

c) the number of iteration k is set to 0, and the maximum number of iteration maxiter is set to 10.

Step 2) Combining ỹ(k)t,n and Φ(k)n into Eq. (19), and GPSR algorithm is used to compute the sparse representation coefficients α(k)t,n, and then we use Eq. (11) to obtain the (k+1)-th iteration solution x(k+1)t,n of each block. Finally, all the interpolated blocks are combined into the estimation xt(k+1) of current frame.

Step 3) k = k+1, if k ≤ maxiter and ║ xt(k+1)- xt(k)║2 ≥ 10-4·N, then H(k)n,1, H(k)n,2, ỹ(k)t,n and Φ(k)n can be updated as H(k+1)n,1, H(k+1)n,2、ỹ(k+1)t,n and Φ(k+1)n by using xt(k+1) and the iteration goes back to Step 2), otherwise the algorithm will be stopped.

The predict frame xSI can be obtained by using CS joint reconstruction after the above steps perform several iterations, and finally the reconstruction of residual frame is performed to achieve final non-key reconstructed frame .

 

4. Simulation results and analysis

The proposed algorithm is evaluated by using the first 61 frames of four test sequences with CIF formant including Foreman, Mobile, Bus and News. The key frame is the odd frame (I frame), and the non-key frame is the even frame (P or B frame). In terms of the style of non-key frame, the proposed algorithm is performed under the two different predictive model, i.e., I-P-I model and I-B-I model. The key frame is independently reconstructed by the MH-BCS-SPL algorithm proposed by Ref. [14], and the non-key frame is reconstructed by the proposed algorithm and the four compared algorithms proposed by Ref. [5], Ref. [6], Ref. [16] and Ref. [17] respectively. The proposed algorithm is divided into two parts to do the comparison experiments: the algorithm uses only adaptive PCA sparse dictionary (i.e., reconstruction model (10)), and it is named as APCA; the algorithm uses adaptive PCA sparse dictionary and non-local similarity regularization item (i.e., reconstruction model (16)), and it is named as APCA-NL. The block size B in all algorithms is set to 16, the measurement rate SK of key frame is set to 0.7, the range of the measurement rate SNK of key frame is [0.1, 0.5]. The parameter setting of proposed algorithm is as follows: the radiuses S1 and S2 of search window are both set to B; the radius b of patch is set to 3; the threshold t selecting patch is set to 20; the regularization factors λ1 and λ2 are set to 0.2 and 0.5/k respectively; the other parameter c is 10.

The objective quality of reconstructed frame is evaluated by using the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity (SSIM) [22], and the reconstruction time reveals the computational complexity. The hardware platform of experiments is a PC with 3.20 GHz CPU and 8 GB RAM, and the software platform is the MATLAB 7.6 under the system Windows 7 64 bits.

Table 1 presents the average PSNR and SSIM of all reconstructed non-key frames at the different measurement rate when the predictive model is I-P-I. It can be observed that the proposed algorithms APCA and APCA-NL have the higher PSNR and SSIM than the other compared algorithm at any measurement rate. Comparing APCA algorithm with APCA-NL algorithm , it can be seen that the performance of APCA-NL algorithm outperforms the APCA algorithm at the high measurement rate (SNK is 0.4 or 0.5), e.g., when SNK is 0.5, the APCA-NL algorithm obtains the PSNR gain 2.09 dB and SSIM gain 0.0058 than APCA algorithm for all test sequences, and but APCA-NL algorithm acquires a little performance improvement at the low measurement rate since the inaccurate motion estimation in measurement domain and lots of noises in the initial solution result in the fact that the added regularization item cannot better describe the non-local similarity of video. Besides, since the edge and texture regions of Mobile and Bus sequences have the complex structural features and do not contain lots of periodical predictive patterns, and their non-local similarity is low, which causes that APCA-NL algorithm cannot effectively improve performance for Mobile and Bus sequences in the basis of APCA algorithm and even degrades the quality of the reconstructed frame. Fig. 5 shows the subjective visual quality of the reconstructed Foreman 8-th frame for various algorithm when SNK is 0.3. It can be seen that the proposed algorithm can remove the blurring and blocking artifacts around lap, and the better subjective visual quality is obtained.

Table 1.Average PSNR (dB) and SSIM of test sequences for the proposed and existing algorithms under I-P-I Model

Fig. 5.When SNK is 0.3, the comparison of subjective visual quality on Foreman 8-th frame for various algorithms under I-P-I model.

Table 2 presents the average PSNR and SSIM of all reconstructed non-key frames at the different measurement rate when the predictive model is I-B-I. Firstly, when compared with I-P-I model, the reconstructed quality of all test sequences is effectively improved, and this is because that the situation of B frame uses not only the information on the previous reconstructed frame but also performs the information on the following reconstructed frame. The performance variances of different algorithms are similar to the those of I-P-I model, the performance of proposed algorithms APCA and APCA-NL outperforms the other compared algorithm, and the APCA-NL algorithm can effectively improve the quality of reconstructed video frame at the high measurement rate. Fig. 6 shows the subjective quality of the Mobile 4-th frame for various algorithms, and it can be seen that the proposed algorithms obtain the better subjective visual quality.

Table 2.Average PSNR (dB) and SSIM of test sequences for the proposed and existing algorithms under I-B-I Model

Fig. 6.When SNK is 0.3, the comparison of subjective visual quality on Mobile 4-th frame for various algorithms under I-B-I model.

Table 3 presents the average reconstruction time (s/frame) of various algorithms. It can be observed that the reconstruction time under I-P-I model is lower than that of I-B-I model, which presents that the reconstructed quality is improved at the cost of the increasing computational complexity under I-B-I model. Besides, the proposed two algorithms increase the computational complexity and obtain the improvement of reconstructed quality, which presents that the better performance of proposed algorithms are achieved at the cost of the high computational complexity.

Table 3.Average reconstruction time (s/frame) comparison of various algorithms

 

5. Conclusions

This paper combines the adaptive PCA sparse dictionary constructed by the correlation between frames and the regularization item constructed by the non-local similarity to propose a joint reconstruction algorithm for improving the rate-distortion performance of DVCS system. With the various temporal-spatial statistic characteristic, the fixed sparse dictionary cannot effectively exploit the sparse property of video frame, and although the sparse dictionary extracted from neighboring frames can change as the content of video frame is change, it is not the best one, this is because that the example-based sparse dictionary lacks the motion estimation between frames and the reference frame contains noises. The proposed construction of sparse dictionary firstly uses the CS measurements of current non-key frame to perform motion estimation in measurement domain, and then uses the motion information between frames to extract the example to produce the data matrix, and finally uses PCA to compute the significant principle components of data matrix for constructing the sparse dictionary. The sparse priori knowledge cannot still recover the edge and texture details of video frame well. To improve the quality of edge and texture regions, the non-local similarity of video frame is used to construct the regularization item, and the regularization item is mixed into the joint CS reconstruction model to remove the blurring and blocking artifacts in edge and texture regions. Experimental results show that the proposed algorithm can effectively improve the rate-distortion performance of DVCS system at the cost of a certain computational complexity, and achieve the better subjective and objective reconstructed quality.