Question

Suppose you built a system to predict stock prices. How would you evaluate it?

Answer 1

First of all it is very difficult to predict stockprices with current available information because it is very random .So there isnt any models with which I would compare my model with to evaluate it . Some prefer to use accuracy of the model to evaluate the models performance. The problem with using accuracy as a performance measure is that it is difficult to assert whether the built models are valid, due to the fact that the data in the test set is generally not uniformly distributed.Area Under the Receiver Operating Characteristic Curve(AUC) is a metric that allows easy evaluation of the discriminative power of a model by measuring the performance of the classification model over the complete range of possible cut-off values.The AUC is a generally accepted performance metric to assess the predictive performance of classification models with respect to ranking in data mining. One of the key advantages of using AUC is its ability to cope with skewed distributions of target label data. Furthermore, it allows for easy comparison with random predictions.AUC is a useful metric for measuring model discrimination power. Even so, just like accuracy, it proves to be less useful in evaluating the real world operational value of a classifier since it makes assumptions about the data that might not be portable to a real setting.In order to properly evaluate the prediction performance of a model, no training information can be used in the evaluation. Usually a 'hold out' set is kept aside for evaluation purposes. In time series prediction, an additional concern is that we cannot use any future information in the training phase of a model, this is usually referred to as 'out of time'.

It would be optimal when the test set is withheld from the training process and is only used in the evaluation of the trained model. There is  a '5 fold in time' cross validation scheme which might be usefull.By using this scheme we can ensure a more robust evaluation of the model.Once the optimal parameters have been determined, a final model can be built on the full training set. Note that no 'out of time' information may be used when building the final system to avoid using information of a future event in a prediction since this information would not be available in a trading system either.

Furthermore constant backtesting is advised in order to ensure to continuing correctness of the models.

Its a way long answer ,but still I guess you have got a brief idea about limitations of stock price models and some possible ways for effective evaluation of these models.

THANK YOU