Statistics

Since only two choices are available for each question, the statistics of the set of answers describe a binomial distribution. If we are comparing videos generated from a model A with videos generated from a model B, the probability $P(r;p,n)$ of choosing the videos generated by the model A $r$ times out of $n$ trials is given by the equation:

$$P(r;p,n)=\frac{n!}{r!(n-r)!}p^r(1-p)^{n-r}$$ (40)

where $p$ is the hypothetical probability of choosing A for each trial.

We want to answer the question: "can people distinguish between two different models by looking at generated videos from the two models"?

We consider the null hypothesis that the videos are indistinguishable. In this case, $p=0.5$ and the distribution of $r$ is given by:

$$P(r;p,n)=\frac{n!}{2^n r!(n-r)!}$$ (41)

We reject the null hypothesis if:

$$\sum_{i=min(r,n-r)+1}^{max(r,n-r)-1}\frac{n!}{2^n i!(n-i)!}>0.95$$ (42)

i.e. there is a $95\%$ chance that we would have obtained a smaller difference if the videos had been indistinguishable.

Similar equations can be used to check whether people prefer the left or the right when choosing videos from two choices extracted from the original sequence.