Generation with a linear residual model

If we use the spatiotemporal model without modelling the residuals, the ends of generated pathlets might not always match each other. Indeed, even if the mean pathlets of two consecutive pathlet models join properly, it does not mean that sampled pathlets from those two groups join properly. This is a problem for the corresponding generated video sequences since one can see sharp changes of the movement of the face when generating parts of the video corresponding to changes of pathlet. This creates a impression of a face jumping which is easily spotted by people.

In order to reduce this effect, we tried to add constraints on pathlet models. Unfortunately, it leads to an over-constrained problem. Figure 7.8 illustrates the problem. It shows a previously generated pathlet on the left. We want to generate the next pathlet. Dotted lines show possible pathlets we can generate from the next pathlet model. Due to the small number of modes describing the possible generated pathlets, the beginnings of those pathlets describe a subspace of the appearance parameter space. This happens when the number of modes of the pathlet model is lower than the dimension of the appearance parameter space. Matching the end of the previous generated pathlet is impossible if this point does not belong to the subspace described by the beginnings of possible generated pathlets.

Figure 7.8: Constraining the generation of pathlets. The beginning of any generated pathlet cannot match the end of the previous generated pathlet.

Since adding constraints did not work, we use the linear residual model described in section 5.3.3. For each pathlet model, a pathlet is generated using equation 5.6 as in the previous section. The beginning of each pathlet is then forced to match the end of the previous one. The other points of the pathlet are moved according to equation 5.9 in section 5.3.3.

Figures 7.9 and 7.10 show generated trajectories using the normalised cut algorithm and the greedy algorithm respectively. The linear model for the residuals has been used to generate both figures.

Figure 7.9: Trajectories generated with the normalised cut algorithm, the dynamic time warping algorithm and a linear residual model. Figure 7.9(a) shows a trajectory generated from the trajectory T1. Figure 7.9(b) shows a trajectory generated from the trajectory T2. Finally, figure 7.9(c) shows a trajectory generated from the trajectory T3.

[Generated from trajectory T1] [Generated from trajectory T2] [Generated from trajectory T3]

Figure 7.10: Trajectories generated with the greedy algorithm and a linear residual model. Figure 7.7(a) shows a trajectory generated from the trajectory T1. Figure 7.7(b) shows a trajectory generated from the trajectory T2. Finally, figure 7.7(c) shows a trajectory generated from the trajectory T3.

[Generated from trajectory T1] [Generated from trajectory T2] [Generated from trajectory T3]