In the previous post, we discussed that how we can assess the performance of a Machine learning model using a k-fold cross-validation method. In this post, we will discuss that how we can leverage the GridSearchCV and RandomizedSearchCV methods to find the optimal hyperparameter values. The hyperparameter value is the value that is required before the model is exposed to the training dataset. That means the hyperparameter is the parameter that is not directly learned from the data by the estimator. Therefore, we need to explicitly pass this parameter value to the model before we use it on the training dataset. So, let’s discuss GridSearchCV and RandomizedSearchCV in detail. Also, let’s discuss the pros and cons of these methods at the end of this post.
What is GridSearchCV
The GridSearchCV is a method that is available out of the box with the sci-kit learn library. This method helps in the hyperparameter tuning of an estimator. We fed some estimated values for the hyperparameter into a parameter grid and then we choose one of the scoring metrics (accuracy, precision, recall, F1 score, etc) to evaluate the performance of the model. The GridSearchCV method loops through all the predefined values in the parameter grid and finds the best combination of the hyperparameters using the given scoring metrics. That is how it helps to fine-tune an ML model.
For example, Before we fit a regularized linear regression (Ridge or Lasso regression), we need to choose the value of an alpha parameter. The model cannot learn the value of alpha from the training dataset directly. Similarly, before fitting the k-Nearest neighbors model, we need to choose the value of parameter n_neighbors. Such parameter values cannot be learned by the estimator from the training dataset directly. Most importantly, the overall performance of the model is directly dependent on these parameters. These parameters have a significant impact on the overall model performance. So, we need to evaluate the model performance with a bunch of different hyperparameter values and then we choose the best performing one. That is why we use the GridSearchCV method to perform these tasks automatically.
The below image summarizes the way GridSearchCV works:
In the above image, the yellow highlighted score is the best score. That means the best value for Hyperparameter 1 is 0.2 and for Hyperparameter 2 is 0.2 respectively.
What is RandomizedSearchCV
The RandomizedSearchCV is also another method that can help us in hyperparameter tuning. The only difference between GridSearchCV and RandomizeddGridSearchCV is the way they scan the predefined parameter grid values. The Randomized Search CV method randomly scans the grid however the Grid Search CV method scan the parameter grid in a sequential way. This is the main difference between both of these methods.
When to use GridSearchCV and RandomizedSearchCV methods
We can use these methods if we wanted to test our model performance for a given set of hyperparameter values. However, it will use only the predefined set of values to scan and test the model performance. In addition to this, we can also use k-fold cross-validation to further enhance the model capability for unseen data. That way, we can make our model more generic and unbiased. The bias and variance trade-off is a very important topic in the Machine learning field. However, we are not going to discuss that here. Most importantly, we need to be very cautious while choosing the hyperparameter values for an ML model. So that we can have a generic model which can work well if it is exposed to the unseen dataset.
Pros and Cons of GridSearchCV and RandomizedSearchCV
Pros:
- Helps in hyperparameter tuning of the estimator with given scoring method
- Automatically scan through all the fed up values into the param grid
Cons:
- Time consuming process which can significantly delay the training time of the model especially if the input dataset is large
- Limited to the manually estimated set of values
Thanks for the reading. Please share your inputs in the comment section.
