Question

Can I have the answer with explaination in own word.

1.
(a) What is the motivation behind “subset selection”?
(b) Explain what the term “shrinkage” means.
(c) Discuss one shrinkage method that we have discussed in class.
(d) What is the difference between linear and quadratic discriminant analysis?

Answer 1

(a) Subset selection reduces overall computational cost by ignoring irrelevant input features. Thus this provides optimal subset for effective performance by eliminating noisy and redundant data in a faster manner.

(b) The term 'shrinkage' denotes a class of regularisation approach that reduces the size of coefficient estimates towards zero relative to the least square estimates. This improves model's stability and feature selection. Generally, this is a tool to improve estimation of covariance matrices in situations where the number of training samples is small compared to the number of features.

(c) LASSO [Least Absolute Shrinkage and Selection Operator]

This method penalizes the absolute value of coefficients (L1 penalty) which enforces some of them to be exactly zero.

Lasso subject to:    

While ridge penalty encompasses the norm of the estimates vector, the lasso makes use of the norm:

where λ≥ 0 is again a complexity parameter.

Lasso also allows automatic variable selection. Extensions include: group-lasso, fused lasso.

(d) LDA [Linear Discriminant Analysis]

- For p predictors and one common covarince matrix for all classes, we need to estimate k*p parameters

- Have linear decision boundary

- Fast classification

- Easy to implement

   - Uses Gaussian assumptions

- Better with less training observations

- Less flexibility and varriance

QDA [Quadratic Discriminant Analysis]

   - For p predictors, 1 covariance matrix, we need to estimate p(p+1)/2 parameters. For K covariance matrices we need to estimate K *p(p+1)/2

- Have quadratic decision boundary

  - More accurate classification

- Outperforms LDA

- Uses Gaussian assumptions

- Recommended with lots of training observations

- Requires more training time and complex matrix ops