Machine Learning (Chapter 40): Class Evaluation Measures

By


 


Machine Learning (Chapter 39): Class Evaluation Measures

In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples.

1. Accuracy

Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets.

Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}

where:

  • TPTP = True Positives
  • TNTN = True Negatives
  • FPFP = False Positives
  • FNFN = False Negatives

Example:

Let’s calculate accuracy in Python.

python
from sklearn.metrics import accuracy_score

# True labels
y_true = [1, 0, 1, 1, 0, 1, 0]

# Predicted labels
y_pred = [1, 0, 1, 0, 0, 1, 1]

# Calculate accuracy
accuracy = accuracy_score(y_true, y_pred)
print("Accuracy:", accuracy)

Output:

makefile
Accuracy: 0.7142857142857143

2. Precision

Precision is the ratio of correctly predicted positive observations to the total predicted positives. High precision relates to the low false positive rate.

Precision=TPTP+FP\text{Precision} = \frac{TP}{TP + FP}

Example:

Let’s calculate precision in Python.

python
from sklearn.metrics import precision_score

# Calculate precision
precision = precision_score(y_true, y_pred)
print("Precision:", precision)

Output:

makefile
Precision: 0.75

3. Recall (Sensitivity or True Positive Rate)

Recall is the ratio of correctly predicted positive observations to all observations in the actual class.

Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}

Example:

Let’s calculate recall in Python.

python
from sklearn.metrics import recall_score

# Calculate recall
recall = recall_score(y_true, y_pred)
print("Recall:", recall)

Output:

makefile
Recall: 0.75

4. F1-Score

The F1-score is the harmonic mean of precision and recall, providing a balance between the two.

F1-Score=2×Precision×RecallPrecision+Recall\text{F1-Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

Example:

Let’s calculate the F1-Score in Python.

python
from sklearn.metrics import f1_score

# Calculate F1 score
f1 = f1_score(y_true, y_pred)
print("F1-Score:", f1)

Output:

makefile
F1-Score: 0.75

5. Confusion Matrix

The confusion matrix shows the actual versus predicted classifications and provides insights into how well the model differentiates between classes.

The confusion matrix for a binary classification problem is as follows:

Actual\PredictedPositive (1)Negative (0)
Positive (1)TPFN
Negative (0)FPTN

Example:

Let’s generate a confusion matrix in Python.

python
from sklearn.metrics import confusion_matrix

# Generate confusion matrix
cm = confusion_matrix(y_true, y_pred)
print("Confusion Matrix:\n", cm)

Output:

lua
Confusion Matrix:
[[2 1]
 [1 3]]

6. Specificity (True Negative Rate)

Specificity measures the proportion of actual negatives that are correctly identified.

Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}

Example:

We can calculate specificity manually as it's not directly available in scikit-learn:

python
# Manually calculate specificity
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
specificity = tn / (tn + fp)
print("Specificity:", specificity)

Output:

makefile
Specificity: 0.6666666666666666

7. ROC-AUC (Receiver Operating Characteristic - Area Under Curve)

The ROC-AUC score summarizes the model's ability to distinguish between classes across different thresholds. AUC (Area Under the Curve) ranges from 0 to 1, where a value close to 1 indicates a good classifier.

The ROC curve plots the true positive rate (Recall) against the false positive rate (1 - Specificity).

Example:

Let’s compute ROC-AUC in Python.

python
from sklearn.metrics import roc_auc_score

# True labels (y_true) and predicted probabilities (y_pred_prob)
y_pred_prob = [0.9, 0.1, 0.8, 0.4, 0.2, 0.7, 0.6]

# Calculate ROC-AUC
roc_auc = roc_auc_score(y_true, y_pred_prob)
print("ROC-AUC:", roc_auc)

Output:

makefile
ROC-AUC: 0.8333333333333334

Java Implementation

Below is a simple Java code snippet for calculating accuracy, precision, and recall using Apache Commons Math library:

java
import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;

public class ClassEvaluationMeasures {

    public static void main(String[] args) {
        // True and predicted labels
        int[] yTrue = {1, 0, 1, 1, 0, 1, 0};
        int[] yPred = {1, 0, 1, 0, 0, 1, 1};

        // Calculate Accuracy
        double accuracy = calculateAccuracy(yTrue, yPred);
        System.out.println("Accuracy: " + accuracy);

        // Calculate Precision and Recall
        double precision = calculatePrecision(yTrue, yPred);
        double recall = calculateRecall(yTrue, yPred);
        System.out.println("Precision: " + precision);
        System.out.println("Recall: " + recall);
    }

    // Accuracy
    public static double calculateAccuracy(int[] yTrue, int[] yPred) {
        int correct = 0;
        for (int i = 0; i < yTrue.length; i++) {
            if (yTrue[i] == yPred[i]) {
                correct++;
            }
        }
        return (double) correct / yTrue.length;
    }

    // Precision
    public static double calculatePrecision(int[] yTrue, int[] yPred) {
        int tp = 0, fp = 0;
        for (int i = 0; i < yTrue.length; i++) {
            if (yPred[i] == 1) {
                if (yTrue[i] == 1) {
                    tp++;
                } else {
                    fp++;
                }
            }
        }
        return tp / (double) (tp + fp);
    }

    // Recall
    public static double calculateRecall(int[] yTrue, int[] yPred) {
        int tp = 0, fn = 0;
        for (int i = 0; i < yTrue.length; i++) {
            if (yTrue[i] == 1) {
                if (yPred[i] == 1) {
                    tp++;
                } else {
                    fn++;
                }
            }
        }
        return tp / (double) (tp + fn);
    }
}

Conclusion

In this chapter, we covered various class evaluation measures such as accuracy, precision, recall, F1-score, specificity, ROC-AUC, and how they can be calculated both mathematically and programmatically using Python and Java. These metrics are crucial in determining the effectiveness of a classification model and in comparing different models.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 35): Decision Trees - Multiway Splits

By
Image
  Chapter 35: Decision Trees - Multiway Splits Introduction Decision trees are a powerful and intuitive method for both classification and regression tasks in machine learning. They work by recursively splitting the data into subsets based on the value of input features, eventually leading to a final prediction. A key decision when building a decision tree is how to split the data at each node. In many cases, binary splits (two-way splits) are used, but multiway splits, where a node is split into more than two branches, can also be beneficial in certain scenarios. Multiway Splits A multiway split occurs when a decision node divides the dataset into more than two subsets based on the possible values of a feature. This approach is especially useful for categorical features with multiple levels, where splitting into all possible categories at once might lead to a simpler and more interpretable tree. Mathematics Behind Multiway Splits The goal of a multiway split is to maximize the sep...
Read more

Machine Learning (Chapter 6): Statistical Decision Theory - Classification

By
Image
  Machine Learning: Chapter 6 - Statistical Decision Theory: Classification Introduction Statistical decision theory forms the backbone of many machine learning techniques, especially in the domain of classification. Classification is a fundamental problem in machine learning, where the goal is to assign a label to a given input based on its features. In this article, we will delve into the mathematical foundation of classification using statistical decision theory, explore key concepts, and provide Python code to illustrate these ideas. 1. The Basics of Classification Classification is the task of predicting a discrete label y y y (such as a class) given an input vector x \mathbf{x} x . The input vector x \mathbf{x} x represents the features of the data, and y y y is the class label. In the context of statistical decision theory, we aim to find a decision rule that minimizes the probability of misclassification. 2. Probability Distributions in Classification In statistical deci...
Read more

Machine Learning (Chapter 32): Stopping Criteria & Pruning

By
Image
  Machine Learning (Chapter 32): Stopping Criteria & Pruning Stopping Criteria In machine learning, particularly in iterative algorithms like decision trees, neural networks, and gradient-based optimization methods, stopping criteria are crucial for ensuring that the algorithm converges efficiently without overfitting or underfitting. Stopping criteria determine when to halt the training process based on certain conditions. 1. Stopping Criteria for Iterative Algorithms For iterative algorithms, stopping criteria can include: Maximum Iterations : The algorithm stops after a predefined number of iterations. Stop if  k ≥ K \text{Stop if } k \geq K Stop if  k ≥ K where k k k is the current iteration and K K K is the maximum number of iterations. Convergence of Loss Function : The algorithm stops if the change in the loss function between iterations is below a threshold ϵ \epsilon ϵ . Stop if  ∣ L k + 1 − L k ∣ < ϵ \text{Stop if } |L_{k+1} - L_k| ...
Read more