The main aim in machine vision is to interpret images. By extension, it also aims at interpreting image sequences. In this chapter, we review how people have tried to model behaviour and interactions between different actors in a video sequence.