Statistical model of appearance

The aim of the statistical model of appearance is to add information about the texture of an object to the information about the shape of the object. The approach is the same as the statistical shape models. The model is learned from a set of annotated images of the object.

First, a statistical shape model is built from the training set and can be generated by the equation:

$${\bf x}=\overline{\bf x}+\Phi_s {\bf b}_s$$

where ${\bf b}_s$ is a vector of parameters.

Then, given the mean shape we can warp the training images into a shape-free patch of texture. Figure 3.3 shows how a face is decompose into a shape and a patch-free texture.

Figure 3.3: Decomposition of a face into a shape and a patch-free texture.

The set of patch-free textures is then normalized in order to reduce the effect of lighting. The pixel values are changed by a linear function so that the patch-free texture of a image is as close as possible to the patch-free texture of the normalized mean.

A principle component analysis is then applied to the patch-free texture in order to model this patch by a linear equation:

$${\bf g}=\overline{\bf g}+\Phi_g {\bf b}_g$$

where ${\bf b}_g$ represents the parameters, as it was already the case with the statistical shape model.

${\bf b}_s$ and ${\bf b}_g$ can then describe an object and its texture. Nevertheless, the two vectors can still be correlated. In order to reduce the dimensionality again, we compute the vector:

$${\bf b}=\left(\begin{array}{l} {\bf W}_s{\bf b}_s \\ {\bf b}_g \\ \end{array}\right)$$

for each training image, where $W_s$ is a diagonal matrix of weights that allows to compare $b_s$ and $b_g$. A third principle component analysis is applied to these data to obtain the model :

$${\bf b}={\bf Q}{\bf c}$$

where $c$ describes the new set of parameters that controls the model. For every $c$, a object can be synthetize. This is an important characteristic of the model. It will allow us to synthetize behaviour once the proper parameter stream is found.

The figure 3.4 shows the first mode of variation of the model.

Figure 3.4: Example of the first mode of variation of the face using a statistical model of appearance. The parameter of ${\bf c}$ corresponding to the larger eigenvalue varies from $-3\sqrt{\lambda_1}$ to $3\sqrt{\lambda_1}$.