Confusion Matrix and Example

Confusion matrix, also known as error matrix is a table layout that visualizes the performance of a supervised machine learning algorithm, mainly classification algorithm. The name stems from the fact that the table makes it easy to see if the system is confusing two/multiple classes in the target variable.

For example, if we are researching a new test that can quickly test a new type of flu, lets say, flu-C. We randomly selected 1000 subjects for the test, and the test results shows that 530 are positive, 470are negative. The actual results are that, 480 are positive, 520 are negative. Therefore, the new test has errors.

1. 470 are tested positive. 400 are tested positive and actually positive. Therefore, 400 are true positive. 70 are tested positive but actually negative. Therefore, 70 are false positive, which is type I error.
2. 530 are tested negative. 450 are tested negative and actually negative. Therefore, true negative is 450. 80 tested negative are actually positive. Therefore, 80 are false negative, which is type II error.

There are four basic metrics in the confusion matrix to evaluate a classification model.

1. Accuracy: overall, how often is the classifier correct?

Accuracy = (𝑇𝑃+𝑇𝑁) /(𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁)

2. Recall: When it is actually yes, how often the classifier predict yes?

Recall = 𝑇𝑃 / (TP+FN)

3. Precision: When it predicts yes, how often it is correct?

Precision = 𝑇𝑃 / (TP+FP)

4. F-1 score: is a weighted average of the recall and precision

F-1 Score = 2𝑃𝑟𝑅𝑐/(Pr+Rc)= 2𝑇𝑃 / (2TP+FP+FN)

• high recall + high precision : the class is perfectly handled by the model
• low recall + high precision : the model can’t detect the class well but is highly trustable when it does
• high recall + low precision : the class is well detected but the model also include points of other classes in it
• low recall + low precision : the class is poorly handled by the model

(https://towardsdatascience.com/handling-imbalanced-datasets-in-machine-learning-7a0e84220f28)

It is not hard to calculate the metrics one by one, nevertheless, sklearn has a really nice package that could provide these four basic metrics quickly. It is very convenient especially when there are multiple classes. In the following I will provide an example of generating confusion matrix in a classification model and how to evaluate the model according to the metrics in the confusion matrix.

Import the data from cars.csv datafile, and selected a few interested features as a subset.

Interpret the results: from the confusion matrix and the classification report, we can tell that the model predicts the target variable very well with the test data set. The false positive and false negative are both 0. The accuracy, recall, precision, and F1 score are all 1. This is mainly because we split the selling price into only two categories around mean. and it is easier for the model to classify.

Hope you enjoy reading it and have fun with confusion matrix.