Suppose we want to select between two prediction models M1 and M2. We have performed 10-fold cross validation on each. The error rates obtained for M1 are 30.5, 32.2, 20.7, 20.6,31.0, 41.0, 27.7, 26.0, 21.5, 26.0. The error rates for M1 are 22.4, 14.5, 22.4, 19.6,20.7, 20.4, 22.1, 19.4, 16.2, 35.0. Is one model significantly better than the other considering a significance level of 1%.?

Since the calculated t value does not fall in the critical region we accept H0 and  conclude that  one model is not significantly better than the other at 1% level of significance.

Explanation:

When the observations from two samples are paired either naturally or by design , we find the difference between two observations of each pair. Treating the differences as a random sample from a normal population with mean μd =μ1-μ2 and unknown standard deviation σd we perform one sample t- test on them. This is called paired difference t- test.

M1               M2           Difference            d²

                                  d= (M1-M2)

30.5,         22.4,         8.1                       65.61          

32.2,       14.5,           17.7                      313.29

20.7,       22.4,          -1.7                       8.3521

20.6,       19.6,             1.0                        1

31.0,       20.7,             10.3                     106.09

41.0,      20.4,             20.6                    424.36

27.7,      22.1,              5.6                      31.36

26.0,      19.4,            6.6                         43.56

21.5,       16.2,            5.3                         28.09                    

26.0.       35.0.            -9.0                      81.0          

∑ 329.2     213.1           64.5                    1102.7121

Now

d` = ∑di/n = 64.5/10= 6.45

sd² = ∑(di-d`)²/n-1= 1/n-1 [ ∑di²- (∑di)²/n]

     =1/9[ 1102.7121 - (64.5)²/10 ]

       =[1102.7121 - 416.025/9]

       = 79.298

sd= 8.734

We state our null and alternate hypotheses as

H0 : μd= 0   and Ha: μd≠0

The significance level is set at ∝ = 0.01

The test statistic under H0  is

t= d`/ sd/√n

which has a t distribution with n-1 degrees of freedom.

The critical region is  t ≥ t (0.005,9)= 3.250

Calculating t

t= 6.45 / 8.734/ √10

t = 6.45 / 8.734/3.162

t= 6.45 / 2.7621

t= 2.335

Since the calculated t value does not fall in the critical region we accept H0 and  conclude that  one model is not significantly better than the other at 1% level of significance.

answer///quick access toolbar;

the major difference between the two is the objective and the reward of the game. candy crush has a puzzle theme with the objective of meeting a specific quota of points or objects, you are able to gain access to the next level and compete with peers on the process. on duolingo the objective is rather different, you are given sets of questions for you to answer for you to move to the next level and as for the reward, you are able to learn available language as this is an app developed to teach the user about other languages.