Polynomial Regression
ML From Scratch Series
Prerequisite
Linear Regression
Before diving into Polynomial Regression, let’s first take a look at Linear Regression. It shares some similarities with Polynomial Regression, but the key difference is that Linear Regression defines a linear relationship between the output and a single input. The model created by Linear Regression is a straight line, and this line tends to approximate a plane that best fits the surrounding data.
Compared to Linear Regression, Polynomial Regression makes it easier to capture relationships in complex data, but one should be cautious as Polynomial Regression can lead to the problem of overfitting the data.
Gradient Descent
To make the model fit the data better, Gradient Descent is one of the methods used.
Gradient Descent is a technique used to find the parameters that minimize the loss function. By continuously updating the model’s parameters through Gradient Descent, we can make the model fit the given dataset better, leading to a reduction in the value of the loss function.
💡 What is a Loss Function / Loss Value?
When using a regression model like Polynomial Regression, our goal is to make the model closely fit the given data, allowing it to accurately predict the corresponding output for a given input.
To assess how well our model matches the actual output, we use a loss function. The loss value it produces represents the difference between the actual and predicted values.
Among the commonly used loss functions for Regression models, one of them is:
- Mean Square Error (MSE)
- Mean Absolute Error (MAE)
- Root Mean Square Error (RMSE)
In order to prevent the Mean Squared Error (MSE) loss value from becoming too large, the Root Mean Square Error (RMSE) takes its square root.
Choose MSE or MAE ?
- MAE (Mean Absolute Error):
The advantage of choosing MAE is that its loss function is less sensitive to outliers, which are data points that significantly differ from the majority. However, MAE cannot predict bias because its loss function only considers absolute differences.
- MSE (Mean Squared Error):
MSE is a convex function and is easy to optimize using gradient descent algorithms. It calculates the differences between data using squared errors, which makes its loss function sensitive to bias.
However, when choosing between MAE or MSE, the nature of the data and the specific requirements of the analysis model should be the focus in deciding which loss function to use in the Regression model.
Goal
Polynomial Regression is a regression model that aims to model data using a polynomial predictive function, with unknown model parameters estimated through data.
Background
Here, the input can be referred to as the independent variable, and the output can be referred to as the dependent variable.
Polynomial Regression aims to model the relationship between the input x and the output y, where the determining function is an n-degree polynomial. In comparison to Linear Regression, where n=1, y = β_0 + β_1*x, Polynomial Regression is more suitable for modeling relationships among nonlinear variables.
The goal in both cases is to determine the values of coefficients that minimize the difference between the estimated output and the actual values.
The basic idea of Polynomial Regression is to build polynomial terms from the original inputs, allowing the predictive function to be used in a linear regression model, represented as a polynomial equation.
In this context, x represents the input features of the original data, while y represents the output. β_0 is the bias term for y, and β_1, β_2, …, β_n are the coefficients of the polynomial.
The purpose of creating a Polynomial Regression model is to determine the coefficient values in such a way that the difference between the actual output values and the predicted values is minimized.
The optimization process can be achieved using optimization algorithms such as gradient descent or the method of least squares.
Once the coefficients are determined, the polynomial equation can be used to make predictions based on new values of the input feature.
Implementation
In this section, we can start by using the scikit-learn library for implementation.
If it’s not already installed, you can use pip install scikit-learn to install it.
Prepare data
Create the polynomial features
Create Model
Evaluation
From Scratch
In this section, we will implement Polynomial Regression using only the numpy library.
Create the feature matrix
Create Model
Evaluation
More example
Here, I try to use Polynomial Regression from scratch to participate the Kaggle competition.
Titanic — Machine Learning from Disaster
Thanks for reading my article. See you next time.
