Question

1)Give a visual example that PCA is not a good way to reduce the dimension?

2)Based on Bayes theorem, express P(c|x) in terms of likelihood and prior.

Answer 1

Since there was a brief hint of SVD, I'd suggest you consider that independently of PCA. SVD can be used directly on a table of records versus attributes, rather than doing the covariance thing that PCA does. This is the basic approach of latent semantic analysis tthe advantage is that using SVD on the raw record vs attribute data means that the singular values are actually very easy to interpret, as they are directly related to both the records and the attributes. Typical LSA will normalize the attributes just slightly, maybe using log counts rather than integer counts, and it's still fairly easy to interpret.

Why is this useful? Imagine that the dimensionality of the feature set is larger than just two or three Using a PCA we can now identify what are the most important dimensions and just keep a few of them to explain most of the variance we see in out data. Hence we can drastically reduce the dimensionality of the data and make EDA feasible again. Moreover, it will also enable us to identify what the most important variables in the original feature space are, that contribute most to the most important PCs. Intuitively, one can imagine, that a dimension that has not much variability cannot explain much of the happenings and thus is not as important as more variable dimensions How do we get there? Let's talk some math (feel free to skip if you don't want to). Technically performing a PCA is just finding the eigenvalues and -vectors of the data's correlation matrix, where and is an matrix of data points: each row is one of samples with each column representing one of the features. is the empirical mean value of the data, and we will come back later to why we need to substract it