How To Evaluate Classifiers with Imbalanced Dataset Part B Accuracy Precision Recall F1

For all tutorials: muratkarakaya.net Google Colab: https://colab.research.google.com/dri... Github: https://github.com/kmkarakaya/ML_tuto... Github pages: https://kmkarakaya.github.io/Deep-Lea... Github Repo: https://github.com/kmkarakaya/Deep-Le... Medium:   / how-to-evaluate-a-classifier-trained-with-...   ------------------------------------------------------------------ Related Tutorial Playlists: Classification with Keras Tensorflow:    • Classification with Keras / Tensorflow   Applied Machine Learning with Python:    • Applied Machine Learning with Python   How to evaluate a TensorFlow Keras model by using correct performance metrics?    • How to evaluate a TensorFlow Keras model b...   All Tutorials in English: https://www.youtube.com/c/MuratKaraka... -------------------------------------------------------------------- How to Evaluate a Classifier Trained with an Imbalanced Dataset? Why Accuracy is not Enough? Imbalanced Dataset Classification Metrics Description: In this tutorial, we will learn how to select THE BEST classification Machine Learning algorithm among 8 alternative models using and understanding the various classification metrics, especially, when you have an imbalanced dataset. We will implement the solutions by Python and SciKit Learn library. Parts I will deliver the content in 3 parts: Part A: Fundamentals, Metrics, Synthetic Dataset Part B: Dummy Classifiers, Accuracy, Precision, Recall, F1 Part C: ROC, AUC, Worthless Test, Setting up threshold Part A: Fundamentals, Metrics, Synthetic Dataset At the end of the tutorial, we will learn the answers to these questions: what is an imbalanced dataset what is a dummy classifier how to measure the performance of a binary classifier how to compare the performances of classification algorithms which metrics are meaningful which is useless how to interpret the metric scores or graphics why “accuracy” is not a good metric why a fixed threshold value is not useful why 99% accuracy is not enough why accuracy is not a good performance metric in some classification tasks what about recall, precision & f1 how to interpret ROC curve & AUC which classifier is better among the alternatives how to create a random binary classification dataset by SciKit Learn Synthetic data is information that’s artificially manufactured rather than generated by real-world events. Imbalanced datasets are a special case for classification problems where the class distribution is not uniform among the classes. Classes that make up a large proportion of the data set are called majority classes. Those that make up a smaller proportion are minority classes. Part B: Dummy Classifiers, Accuracy, Precision, Recall, F1 Function to create Dummy Classifiers We will use the SciKit Learn DummyClassifier method which is a classifier that makes predictions using simple rules. This classifier is useful as a simple baseline to compare with other (real) classifiers. class sklearn.dummy.DummyClassifier( strategy='warn', random_state=None, constant=None) Parameters: strategy default="stratified". Strategy to use to generate predictions. “stratified”: generates predictions by respecting the training set’s class distribution. “most_frequent”: always predicts the most frequent label in the training set. “prior”: always predicts the class that maximizes the class prior (like “most_frequent”) and predict_proba returns the class prior. “uniform”: generates predictions uniformly at random. “constant”: always predicts a constant label that is provided by the user. This is useful for metrics that evaluate a non-majority class Do not use it for real problems. More metrics, More Insight: Precision, Recall, F1 We can observe more metrics by using the SciKit Learn classification_report() method. Before investigating the results remember that there are only 13 positive case (1) in the test dataset and our aim is to detect possible patients! So notice the recall values in the reports to understand how many of the positive cases are predicted correctly by looking the second line of each report. accuracy is the number of correct predictions divided by the total number of predictions. The recall is the number of correct positive class predictions divided by the total number of positive samples in the dataset. Precision is the number of positive class predictions divided by the number of samples that actually belong to the positive class. F-Measure (score) is a single score that balances both the precision and recall in one number.