This chapter described how we can transform a sequence of video frames into a sequence of prototypes that are learned in a unsupervised manner from training sequences. These prototypes can then be used to learn sequences of prototypes using predictions algorithm over a finite set of states. Such learning algorithms include different variants hidden Markov model and in particular, the variable length Markov model that is described in the next chapter. The vector quantization algorithm as described by Johnson and Hogg will be taken as a reference in order to compare our future work with this way of clustering vectors.