A model based on an autoregressive process

In [17], Campbell et al. introduce another way of generating video sequences of faces based on an existing video clip without direct reuse of the original frames. They encode frames from the original sequence in a way similar to our method. An appearance model is used to model the face through the video sequence and a trajectory is obtained in the appearance parameter space. This trajectory is then used to train a second order autoregressive process.

An autoregressive process predicts the position of a point ${\bf y_k}$ in the appearance parameter space, given the two previous points ${\bf y_{k-1}}$ and ${\bf y_{k-2}}$ where ${\bf k}$ represents the frame number. This prediction is produced by the equation:

$${\bf y_k}-{\bf\overline{y}}={\bf A_2}\left({\bf y_{k-2}}-{\b... ...eft({\bf y_{k-1}}-{\bf\overline{y}}\right)+{\bf B_0} {\bf w_k}$$ (37)

where ${\bf\overline{y}}$ is the limit of the mean value of ${\bf y_k}$ as ${\bf k}$ tends to infinity, ${\bf w_k}$ contains white noise ( ${\bf w_k} \sim {\cal{N}} \left(0,1\right)$), ${\bf A_2}$, ${\bf A_1}$ and ${\bf B_0}$ are parameter matrices. ${\bf\overline{y}}$, ${\bf A_2}$, ${\bf A_1}$ and ${\bf B_0}$ can be learnt from the original data set. The learning method used in this work is due to Reynard et al. and is described in [79] and [8].

Given two initial points in the parameter space, a new trajectory can be generated by repeatedly applying equation 8.1. This new trajectory in the parameter space fed into an appearance model to generate a video sequence. Figure 8.1 shows an example of a video clip generated by an autoregressive process. The original video sequence V3 is shown on figure 8.2. For comparison figure 8.3 shows the generation of similar video sequences with our model.

Figure 8.1: Frames taken every 4 seconds from the video sequence generated with an autoregressive process.

Figure 8.2: Frames taken every 4 seconds from the original long video sequence.

Figure 8.3: Frames taken every 4 seconds from the video sequence generated with our model and the linear residual model.

The corresponding video sequences can be seen on the accompanying CD-ROM. The file examples/V3/V3_orig.m1v shows the original video sequence. The file examples/V3/V3_arp.m1v shows the video sequence generated with the autoregressive process. The file examples/V3/V3_wr.m1v shows the video sequence generated with our model if we use the linear model for the residuals. The file examples/V3/V3_wor.m1v shows the video sequence generated with our model when we do not use the residuals.

Another example can be seen on figures 8.4(a), 8.4(b), 8.4(c) and 8.4(d). Those figures describe the trajectory T3 (figure 8.4(a)) as well as the corresponding generated trajectory using our model with or without the linear model of residuals (figures 8.4(c), 8.4(b) respectively). Finally figure 8.4(d) shows the trajectory generated by the autoregressive process in the appearance parameter space. The corresponding video sequences can be seen on the accompanying CD-ROM. The file examples/V1/V1_orig.m1v shows the original video sequence. The file examples/V1/V1_arp.m1v shows the video sequence generated with the autoregressive process. The file examples/V1/V1_wr.m1v shows the video sequence generated with our model if we use the linear model for the residuals. The file examples/V1/V1_wor.m1v shows the video sequence generated with our model when we do not use the residuals.

Figure 8.4: Comparison of generated trajectories. Figure 8.4(a) represents points on the trajectory that corresponds to the original video sequence. Figure 8.4(b) represents points on the trajectory generated by our model if the residuals used are set to zero. Figure 8.4(c) represents the equivalent with a linear model for residuals. Figure 8.4(d) represents points on the trajectory generated by an autoregressive process.

[Training sequence extracted from video V1.] [Generated sequence with our model without using the residuals.] [Generated sequence with our model and a linear model for the residuals.] [Generated sequence with an autoregressive process.]