Description of the experiment

After a short description of the experiment (see figure 8.7), people are asked to answer 52 questions. Each question consists of a comparison between two video sequences. The user is asked to choose the most realistic video between the two videos displayed (see figure 8.8).

Figure 8.7: Screen capture of the description of the experiment.

Figure 8.8: Screen capture of a question during the experiment.

Tables 8.2 and 8.3 summarise the content of the questions. The questions are the same for each person completing the experiment. They have been selected in such a way that:

• each type of video is shown the same number of times.
• each type of video appears the same number of times on the right as on the left.
• generated videos are compared to the original video.
• videos generated with one model are compared to video generated with all the other models.
• for each pair of generated videos another pair is shown with the same models in the same order.
• for each pair of generated videos another pair is shown with the same models in a different order.

Table 8.2: Summary of the content of the questions in the experiment (questions 1 to 34). Arrows pointing to the left means that the video is shown on the left while arrows pointing on the right means that the video is shown on the right. An arrow pointing in both directions means that the clips used on the right as well as the one on the left are extracted from the same video.

Question	Video V2: expressions				Video V3: dialog
number	Original	WR	WOR	ARP	Original	WR	WOR	ARP
1					$\leftarrow$	$\rightarrow$
2						$\rightarrow$		$\leftarrow$
3					$\leftrightarrow$
4					$\leftarrow$			$\rightarrow$
5	$\leftarrow$	$\rightarrow$
6		$\leftarrow$	$\rightarrow$
7	$\rightarrow$			$\leftarrow$
8		$\leftarrow$		$\rightarrow$
9			$\leftarrow$	$\rightarrow$
10	$\leftarrow$	$\rightarrow$
11							$\leftarrow$	$\rightarrow$
12						$\rightarrow$	$\leftarrow$
13						$\leftarrow$		$\rightarrow$
14					$\leftarrow$		$\rightarrow$
15	$\leftrightarrow$
16	$\rightarrow$			$\leftarrow$
17					$\leftarrow$		$\rightarrow$
18	$\leftarrow$		$\rightarrow$
19						$\leftarrow$	$\rightarrow$
20					$\leftarrow$			$\rightarrow$
21					$\rightarrow$	$\leftarrow$
22		$\leftarrow$	$\rightarrow$
23		$\leftarrow$		$\rightarrow$
24			$\rightarrow$	$\leftarrow$
25			$\rightarrow$	$\leftarrow$
26	$\rightarrow$		$\leftarrow$
27					$\leftarrow$	$\rightarrow$
28						$\rightarrow$		$\leftarrow$
29					$\rightarrow$	$\leftarrow$
30	$\rightarrow$	$\leftarrow$
31					$\rightarrow$		$\leftarrow$
32	$\rightarrow$		$\leftarrow$
33					$\rightarrow$		$\leftarrow$
34		$\rightarrow$	$\leftarrow$

Table 8.3: Summary of the content of the questions in the experiment (questions 35 to 52). Arrows pointing to the left means that the video is shown on the left while arrows pointing on the right means that the video is shown on the right. An arrow pointing in both directions means that the clips used on the right as well as the one on the left are extracted from the same video.

Question	Video V2: expressions				Video V3: dialog
number	Original	WR	WOR	ARP	Original	WR	WOR	ARP
35							$\rightarrow$	$\leftarrow$
36	$\leftarrow$			$\rightarrow$
37						$\leftarrow$		$\rightarrow$
38					$\rightarrow$			$\leftarrow$
39		$\rightarrow$	$\leftarrow$
40					$\leftrightarrow$
41						$\rightarrow$	$\leftarrow$
42					$\rightarrow$			$\leftarrow$
43		$\rightarrow$		$\leftarrow$
44	$\leftrightarrow$
45	$\leftarrow$			$\rightarrow$
46							$\leftarrow$	$\rightarrow$
47		$\rightarrow$		$\leftarrow$
48	$\leftarrow$		$\rightarrow$
49							$\rightarrow$	$\leftarrow$
50	$\rightarrow$	$\leftarrow$
51			$\leftarrow$	$\rightarrow$
52						$\leftarrow$	$\rightarrow$

In addition to those questions, people are asked to compare four times the original video to itself. Those questions are used to verify whether people prefer the right or the left given no real difference. The questions are presented in random order to avoid potential psychological effects on sequences of similar questions.

The experiment has been done online to allow a wide range of persons to participate. Two video formats have been provided (mpeg and animated gif). The mpeg format provided the best quality and people were advised to use it in the presentation of the experiment.

The accompanying CD-ROM contains the video files used for the experiments. The files exp/m1v/q01.m1v to exp/m1v/q52.m1v, corresponding to the videos used for questions 1 to 52 respectively, are encoded using the mpeg format, while the files exp/gif/q01.gif to exp/gif/q52.gif are the corresponding videos encoded in the animated gif format. The mpeg files are each 405 frames long. This correspond to a runtime of about 16 seconds. The animated gif version contains half of the frames with half the frame rate. So each person completing the experiment to the end has seen about 14 minutes of video sequences (either generated or original videos). This time appeared to be globally too long for people to concentrate, which explains why only video V2 and video V3 have been included in the experiment. Video V2 represents a structured video while the video V3 is unstructured.

Finally, the time taken to answer each question has been recorded although no indication about timings were given to the volunteers.