Machine Learning (Chapter 38): Evaluation Measures 1

By


 


Machine Learning (Chapter 38): Evaluation Measures 1

In machine learning, evaluation measures are crucial to assess the performance of a model. This chapter will focus on key metrics for evaluating classification models, including accuracy, precision, recall, F1 score, and the confusion matrix. We will also look at mathematical formulas and provide Python/Java examples for better understanding.

1. Confusion Matrix

The confusion matrix is a tabular representation of actual vs. predicted classes. It helps to visualize the performance of a classification model by categorizing predictions into:

  • True Positive (TP): Correctly predicted positive classes.
  • True Negative (TN): Correctly predicted negative classes.
  • False Positive (FP): Incorrectly predicted as positive (Type I error).
  • False Negative (FN): Incorrectly predicted as negative (Type II error).
Predicted PositivePredicted Negative
Actual PositiveTrue Positive (TP)False Negative (FN)
Actual NegativeFalse Positive (FP)True Negative (TN)

2. Accuracy

Accuracy is the ratio of correctly predicted observations to the total observations.

Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}

Example in Python:

python
from sklearn.metrics import confusion_matrix, accuracy_score

# Example data
actual = [1, 0, 1, 1, 0, 1, 0, 0]
predicted = [1, 0, 1, 0, 0, 1, 1, 0]

# Confusion Matrix
conf_matrix = confusion_matrix(actual, predicted)
print(f"Confusion Matrix:\n{conf_matrix}")

# Accuracy
accuracy = accuracy_score(actual, predicted)
print(f"Accuracy: {accuracy}")

Example in Java:

java
import java.util.Arrays;

public class EvaluationMetrics {
    public static void main(String[] args) {
        int[] actual = {1, 0, 1, 1, 0, 1, 0, 0};
        int[] predicted = {1, 0, 1, 0, 0, 1, 1, 0};
        
        int TP = 0, TN = 0, FP = 0, FN = 0;
        
        for (int i = 0; i < actual.length; i++) {
            if (actual[i] == 1 && predicted[i] == 1) TP++;
            if (actual[i] == 0 && predicted[i] == 0) TN++;
            if (actual[i] == 0 && predicted[i] == 1) FP++;
            if (actual[i] == 1 && predicted[i] == 0) FN++;
        }

        double accuracy = (double)(TP + TN) / (TP + TN + FP + FN);
        System.out.println("Confusion Matrix: ");
        System.out.println("TP: " + TP + " FP: " + FP);
        System.out.println("FN: " + FN + " TN: " + TN);
        System.out.println("Accuracy: " + accuracy);
    }
}

3. Precision

Precision measures the accuracy of the positive predictions. It is the ratio of true positives to all positive predictions.

Precision=TPTP+FP\text{Precision} = \frac{TP}{TP + FP}

Example in Python:

python
from sklearn.metrics import precision_score

# Precision
precision = precision_score(actual, predicted)
print(f"Precision: {precision}")

Example in Java:

java
double precision = (double) TP / (TP + FP);
System.out.println("Precision: " + precision);

4. Recall (Sensitivity)

Recall (or Sensitivity) measures how well the model identifies positive cases. It is the ratio of true positives to all actual positive cases.

Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}

Example in Python:

python
from sklearn.metrics import recall_score

# Recall
recall = recall_score(actual, predicted)
print(f"Recall: {recall}")

Example in Java:

java
double recall = (double) TP / (TP + FN);
System.out.println("Recall: " + recall);

5. F1 Score

The F1 Score is the harmonic mean of precision and recall. It balances both metrics, especially when you want to give equal importance to precision and recall.

F1 Score=2×Precision×RecallPrecision+Recall\text{F1 Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

Example in Python:

python
from sklearn.metrics import f1_score

# F1 Score
f1 = f1_score(actual, predicted)
print(f"F1 Score: {f1}")

Example in Java:

java
double f1 = 2 * (precision * recall) / (precision + recall);
System.out.println("F1 Score: " + f1);

6. Conclusion

These evaluation measures provide essential insights into how well your model is performing. While accuracy is the most basic metric, it may not always be the best measure, especially when the classes are imbalanced. Precision, recall, and the F1 score offer deeper insights into the model's behavior, particularly in critical applications like medical diagnoses or fraud detection.

By calculating these metrics and analyzing the confusion matrix, we can better understand our model's strengths and areas of improvement.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 41): The ROC Curve

By
Image
  Machine Learning (Chapter 39): The ROC Curve Introduction to ROC Curve The ROC (Receiver Operating Characteristic) curve is a graphical representation used in binary classification problems to evaluate the performance of a classifier. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The curve illustrates the trade-offs between sensitivity (recall) and specificity, helping to visualize a classifier’s capability. A perfect classifier would have a curve that reaches the top-left corner of the plot, whereas a random classifier would result in a diagonal line from (0, 0) to (1, 1). Key Definitions Before diving into the mathematics, let's define key metrics: True Positive (TP) : Correctly predicted positives. False Positive (FP) : Incorrectly predicted positives. True Negative (TN) : Correctly predicted negatives. False Negative (FN) : Incorrectly predicted negatives. From these, we calculate: True Positive Rate (TPR) , also kn...
Read more

Machine Learning (Chapter 39): Bootstrapping & Cross-Validation

By
Image
  Machine Learning (Chapter 39): Bootstrapping & Cross-Validation Introduction In the context of machine learning, evaluating the performance of models is a critical step. Two essential techniques widely used for model evaluation and resampling are Bootstrapping and Cross-Validation . Both help estimate the performance of a model by making efficient use of available data, reducing overfitting, and improving generalization. 1. Bootstrapping Bootstrapping is a resampling method where subsets of data are sampled with replacement. The goal is to create multiple training datasets, then build models for each dataset, and assess model variability and performance. Key Concept : When we sample with replacement, some observations may appear multiple times in the bootstrap sample, while others may not appear at all. Mathematical Foundation Given a dataset D = { x 1 , x 2 , … , x n } D = \{x_1, x_2, \dots, x_n\} D = { x 1 ​ , x 2 ​ , … , x n ​ } , a bootstrap sample D b D_b D b ​ is crea...
Read more

Machine Learning (Chapter 40): Class Evaluation Measures

By
Image
  Machine Learning (Chapter 39): Class Evaluation Measures In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples. 1. Accuracy Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets. Accuracy = T P + T N T P + T N + F P + F N \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} Accuracy = TP + TN + FP + FN TP + TN ​ where: T P TP TP = True Positives T N TN TN = True Negatives F P FP FP = False Positives F N FN FN = False Negatives Example: Let’s calculate accuracy in Python. python Copy code from sklearn.metrics import ac...
Read more