Question

What does it means to get the same feature selected for all classifier in machine learning? Give a reaseach base answer of atleast 500 words. Also look at the table 5 below for more understanding of my question.

In the filter methods, the results showed that all tested classifiers left with same top ten features in the same order. (Table 5). This could indicate what? Table 5. List of the ranked features by filter method ML Classifiers Global minimum of the top ten features SMO 5,3,6,4,30,29,33,34,35,38 REPTree 5,3,6,4,30,29,33,34,35,38 NB Tree 5,3,6,4,30,29,33,34,35,38 RBF 5,3,6,4,30,29,33,34,35,38 Logit Boost 5,3,6,4,30,29,33,34,35,38 BayesNet 5,3,6,4,30,29,33,34,35,38 + +

Answer 1

In the question, it is noted that the same set of features that are chosen by various classifiers are the best top performing.

Here. I would like to point out certain few things in advance for a better understanding of the feature selection algorithms. They are: 1. What technique of filter feature selection is used for choosing the best set of features which is not specified in the question. The question is relevant because for classifiers like Support Vector Machine, Naive Bayes, and many other classifiers that are based on the dimension of the classifier, the distance between the sample points of the labels will be relevant. For example, Fisher Score is an example of a metric that is used to measure the class seperability of each labels. If the class seperation between the labels are high, then the features are said to be highly distinguishable. That is why the high ranked features using Fisher Score tend to have produced good results. Similarily, information gain is a probability based feature selection approach in which the measurement of how much information is contained within each feature is measured. This metric is very much suited for the classifiers such as Decision Tree and Random Forest that are based on the information theory measures. In short, the kind of feature selection used could dramatically influence the performance of the classifiers.

There is also one more important thing to note it is regarding the multi-dimension combination factor for the features. It is an important thing to address regarding the features because the combination of certain top ranked features could give a better classifier performance. In this scenario, the most important thing to address is regarding the ability of the combinations to have a better classifier performance. Let us say we ranked the top 10 features for the classifier according to some criteria. Then, the combination of the first 10 features need not give the best result. In such case, one combination among the top 10 features could give the best classifier performance. Finding out that best combination of classifiers is again a mystery and a lot of research efforts are being put forward for implementing such a metric that incorporate the multi dimensionality aspect of the feature.

Any way, the observations made by the questionnaire on the uniformity of top ranked features over various classifiers is a clear indication of two things:

1. The chosen top-ranked features are promising and has high seperability because every classifier is good with this set of features. By good seperability, it would mean that the between class distance and within class distance for each feature with respect to the labels are high. This is an indication of how distinguishable are the features.

2, One good combination can be obtained from this top ranked features that could give the best result. A supervised wrapper model could be used to accomplish this task.

Hope, I have given a research point of view answer.