Question

In the course of the thesis work, a student develops a new approach for the solution of a problem (here referred to as method B). The current state-of-the-art approach, method A, is well published in the literature and has been applied to a large standard problem set where its average performance was discovered to be (and published in the main paper by the developers as) 7 with a standard deviation of 3 across the different problems in the problem set. In addition to the publication, the developers of method A also provide their code for anyone to be able to experiment with and the student decides to pick a random set of 15 problems from the standard problem set and apply both methods to these problems, resulting in the following performance numbers for method A: {8, 3, 10, 8, 11, 4, 6, 4, 12, 4, 5, 10, 6, 2, 10}, and the following performance numbers for the student’s method B: {9, 5, 9, 10, 15, 4, 7, 4, 12, 7, 8, 10, 6, 4, 12}. Looking at this data, the student discovers that it seems that method B outperforms method A and sets out to prove this using significance testing with a two-tailed 5% significance threshold. Given that both published performance results as well as the student’s experimental results are available, a number of tests can be performed.

Evaluate the results in terms of the hypothesis that method B has a higher performance than method A. List all the steps (and formulas) involved in the test and what the result implies for the significance of the hypothesis.

Answer 1

The hypothesis would be as followed:

H₀: Difference=0

H₁: Difference ≠hypothesized difference

we will perform paired t-test for this problem, it would be calculated as followed:

Method A Method BD 5 10 4 12 10

The following table "D" is the difference of method a and b

use the following formula, to get test-statistic value;

t =

where,

D-bar= mean of the D = -1.267

S_d= standard deviation of D= 1.438

n= 15

So, putting these values in the above formula,

t= -3.41

Now, using t-table, p-value with the df=14 would be as followed

p-value= 0.004

Now, as you can see that p-value< significance value(0.05), the result is statistically significant.

Since, we may reject the null hypothesis. We can say that there is some difference in both the methods.