Question

Robust data loading poses a challenge in database systems because the input data are often dirty. In many cases and input record may miss multiple values; some records could be contaminated, with some data values out of range or of a different data type than expected. Work out an automated data cleaning and loading algorithm so that the erroneous data will be marked and contaminated data will not be mistakenly inserted into the database during data loading.

Answer 1

Begin
                  for each record r
begin
            check r for missing values
            If possible, Fll in missing values according to domain knowledge
                             (e.g. mean, mode, most likely value, etc.).
            check r for out of range values
                             If possible, correct out of range values according to domain knowledge
                             (e.g. min or max value for the attribute).
            check r for erroneous data types
                                      If possible, correct data type using domain knowledge
                              If r could not be corrected, mark it as bad and output it to a log, otherwise load r into                               the database.
        end
end

The domain knowledge can be a combination of manual and automatic work. We can, for example, use the data in the database to construct a decision tree to induce missing values for a given attribute, and at the same time have human-entered rules on how to correct wrong data types. - 4 -