Machine Learning (Chapter 17): Perceptron Learning

By

 



Machine Learning (Chapter 17): Perceptron Learning

Introduction

Perceptron learning is a fundamental concept in machine learning and neural networks, providing the basis for understanding more complex algorithms. This chapter delves into the Perceptron algorithm, which is a type of linear classifier used for binary classification tasks.

1. The Perceptron Model

The Perceptron is a simple linear classifier that maps input features to a binary output. It consists of input features, weights, a bias term, and an activation function. The goal of the Perceptron algorithm is to find a set of weights and bias that correctly classify the input data.

The Perceptron model can be described mathematically as follows:

  1. Mathematical Formulation

    Given an input vector x=[x1,x2,…,xn]\mathbf{x} = [x_1, x_2, \ldots, x_n] and a weight vector w=[w1,w2,…,wn]\mathbf{w} = [w_1, w_2, \ldots, w_n], the output yy of the Perceptron is computed using the following formula:

    y=sign(w⋅x+b)y = \text{sign}( \mathbf{w} \cdot \mathbf{x} + b )

    where:

    • w⋅x\mathbf{w} \cdot \mathbf{x} denotes the dot product between the weight vector w\mathbf{w} and the input vector x\mathbf{x}.
    • bb is the bias term.
    • sign\text{sign} is the activation function that returns 11 if the argument is positive, and −1-1 otherwise.

  2. Perceptron Learning Algorithm

    The Perceptron Learning Algorithm adjusts the weights and bias based on the prediction error. Given a dataset with input features xi\mathbf{x}_i and corresponding labels yiy_i, the weight update rule is:

    w←w+η(yi−y^i)xi\mathbf{w} \leftarrow \mathbf{w} + \eta (y_i - \hat{y}_i) \mathbf{x}_i b←b+η(yi−y^i)b \leftarrow b + \eta (y_i - \hat{y}_i)

    where:

    • η\eta is the learning rate.
    • yiy_i is the true label.
    • y^i\hat{y}_i is the predicted label.

    The algorithm iteratively updates the weights and bias until convergence or until a predefined number of iterations is reached.

2. Python Code Example

Let's illustrate the Perceptron learning algorithm with a Python example using the numpy library. We'll classify points in a 2D space into two classes.

python:

import numpy as np

class Perceptron:
    def __init__(self, learning_rate=0.01, n_iter=1000):
        self.learning_rate = learning_rate
        self.n_iter = n_iter

    def fit(self, X, y):
        self.weights = np.zeros(X.shape[1])
        self.bias = 0
        self.errors_ = []

        for _ in range(self.n_iter):
            errors = 0
            for xi, target in zip(X, y):
                update = self.learning_rate * (target - self.predict(xi))
                self.weights += update * xi
                self.bias += update
                errors += int(update != 0.0)
            self.errors_.append(errors)

    def predict(self, X):
        return np.where(self.net_input(X) >= 0.0, 1, -1)

    def net_input(self, X):
        return np.dot(X, self.weights) + self.bias

# Example usage
if __name__ == "__main__":
    # Training data: XOR problem
    X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
    y = np.array([-1, 1, 1, -1])  # Labels for XOR

    perceptron = Perceptron(learning_rate=0.1, n_iter=10)
    perceptron.fit(X, y)

    print("Weights:", perceptron.weights)
    print("Bias:", perceptron.bias)

    # Predictions
    predictions = perceptron.predict(X)
    print("Predictions:", predictions)

Explanation of the Code:

  • Initialization: We initialize the Perceptron with a learning rate and number of iterations.
  • Training (fit method): We initialize weights and bias, then update them based on the prediction errors.
  • Prediction (predict method): We use the trained weights and bias to predict labels for new data.
  • Example Usage: The XOR problem is used as a simple test case, though the Perceptron may not solve it perfectly due to its linear nature.

Conclusion

The Perceptron algorithm is a foundational technique in machine learning, providing insight into linear classification and the learning process. Despite its simplicity, it lays the groundwork for more advanced models and neural networks. Understanding its mathematical formulation and implementation helps build a strong foundation for studying more complex algorithms.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 41): The ROC Curve

By
Image
  Machine Learning (Chapter 39): The ROC Curve Introduction to ROC Curve The ROC (Receiver Operating Characteristic) curve is a graphical representation used in binary classification problems to evaluate the performance of a classifier. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The curve illustrates the trade-offs between sensitivity (recall) and specificity, helping to visualize a classifier’s capability. A perfect classifier would have a curve that reaches the top-left corner of the plot, whereas a random classifier would result in a diagonal line from (0, 0) to (1, 1). Key Definitions Before diving into the mathematics, let's define key metrics: True Positive (TP) : Correctly predicted positives. False Positive (FP) : Incorrectly predicted positives. True Negative (TN) : Correctly predicted negatives. False Negative (FN) : Incorrectly predicted negatives. From these, we calculate: True Positive Rate (TPR) , also kn...
Read more

Machine Learning (Chapter 39): Bootstrapping & Cross-Validation

By
Image
  Machine Learning (Chapter 39): Bootstrapping & Cross-Validation Introduction In the context of machine learning, evaluating the performance of models is a critical step. Two essential techniques widely used for model evaluation and resampling are Bootstrapping and Cross-Validation . Both help estimate the performance of a model by making efficient use of available data, reducing overfitting, and improving generalization. 1. Bootstrapping Bootstrapping is a resampling method where subsets of data are sampled with replacement. The goal is to create multiple training datasets, then build models for each dataset, and assess model variability and performance. Key Concept : When we sample with replacement, some observations may appear multiple times in the bootstrap sample, while others may not appear at all. Mathematical Foundation Given a dataset D = { x 1 , x 2 , … , x n } D = \{x_1, x_2, \dots, x_n\} D = { x 1 ​ , x 2 ​ , … , x n ​ } , a bootstrap sample D b D_b D b ​ is crea...
Read more

Machine Learning (Chapter 40): Class Evaluation Measures

By
Image
  Machine Learning (Chapter 39): Class Evaluation Measures In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples. 1. Accuracy Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets. Accuracy = T P + T N T P + T N + F P + F N \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} Accuracy = TP + TN + FP + FN TP + TN ​ where: T P TP TP = True Positives T N TN TN = True Negatives F P FP FP = False Positives F N FN FN = False Negatives Example: Let’s calculate accuracy in Python. python Copy code from sklearn.metrics import ac...
Read more