How To Evaluate Classifiers with Imbalanced Dataset Part C ROC AUC,Worthless Test Set up threshold

For all tutorials: muratkarakaya.net Google Colab: https://colab.research.google.com/dri... Github: https://github.com/kmkarakaya/ML_tuto... Github pages: https://kmkarakaya.github.io/Deep-Lea... Github Repo: https://github.com/kmkarakaya/Deep-Le... Medium:   / how-to-evaluate-a-classifier-trained-with-...   ------------------------------------------------------------------ Related Tutorial Playlists: Classification with Keras Tensorflow:    • Classification with Keras / Tensorflow   Applied Machine Learning with Python:    • Applied Machine Learning with Python   How to evaluate a TensorFlow Keras model by using correct performance metrics?    • How to evaluate a TensorFlow Keras model b...   All Tutorials in English: https://www.youtube.com/c/MuratKaraka... -------------------------------------------------------------------- How to Evaluate a Classifier Trained with an Imbalanced Dataset? Why Accuracy is not Enough? Imbalanced Dataset Classification Metrics Description: In this tutorial, we will learn how to select THE BEST classification Machine Learning algorithm among 8 alternative models using and understanding the various classification metrics, especially, when you have an imbalanced dataset. We will implement the solutions by Python and SciKit Learn library. Parts I will deliver the content in 3 parts: Part A: Fundamentals, Metrics, Synthetic Dataset Part B: Dummy Classifiers, Accuracy, Precision, Recall, F1 Part C: ROC, AUC, Worthless Test, Setting up threshold Part A: Fundamentals, Metrics, Synthetic Dataset At the end of the tutorial, we will learn the answers to these questions: what is an imbalanced dataset what is a dummy classifier how to measure the performance of a binary classifier how to compare the performances of classification algorithms which metrics are meaningful which is useless how to interpret the metric scores or graphics why “accuracy” is not a good metric why a fixed threshold value is not useful why 99% accuracy is not enough why accuracy is not a good performance metric in some classification tasks what about recall, precision & f1 how to interpret ROC curve & AUC which classifier is better among the alternatives how to create a random binary classification dataset by SciKit Learn Synthetic data is information that’s artificially manufactured rather than generated by real-world events. Imbalanced datasets are a special case for classification problems where the class distribution is not uniform among the classes. Classes that make up a large proportion of the data set are called majority classes. Those that make up a smaller proportion are minority classes. Part C: ROC, AUC, Worthless Test, Setting up threshold Receiver Operator Curve (ROC) & Area Under Curve (AUC) So far, for any classifier, the threshold value is fixed at 0.5 for deciding a class label That is, the class label is decided as "0" if the class probability is less than 0.5 The class label is decided as "1" if the class probability is greater than or equal to 0.5 In ROC metric we try all possible threshold values for class probabilities to decide class labels In ROC, for each possible threshold value, we calculate the True Positive Rate (TPR) and False Positive Rate (FPR) values and draw their values on a plot: the ROC plot In the ROC plot, these TPR & FPR pairs generate a curve and we can calculate the Area Under Curve (AUC). The AUC score gives an average performance of the classifier when all possible threshold values are considered NOTE: THE WORTHLESS TEST When we have a complete overlap between the results from the positive and the results from the negative population, we have a worthless test. A worthless test has a discriminating ability equal to flipping a coin. The ROC curve of the worthless test falls on the diagonal line. It includes the point with 50 % sensitivity and 50 % specificity. The area under the ROC curve (ROC AUC) of the worthless test is 0.5. Above, you can see that by changing threshold values from 0.1 to 1.0, you can end up with different prediction results for positive class! Actually, you can use the ROC curve information to locate a specific threshold value which provides you better recall value for the positive class. Below, we will observe the TPR and FPR values for the calculated thresholds: The recall is the number of correct positive class predictions divided by the total number of positive samples in the dataset. Precision is the number of positive class predictions divided by the number of samples that actually belong to the positive class. F-Measure (score) is a single score that balances both the precision and recall in one number.