This chapter explains how we can create an active appearance model that can track a face in a long video sequence. This approach is semi-automatic since we iteratively refine the appearance model by adding hand-labelled images in its training set. We then describe the video sequences acquired to assess our model of behaviour.