SCD

 

STOCHASTIC GRADIENT DESCENT ALGORITHM

Introduction:

Gradient Descent is an iterative optimization technique that seeks the optimal value (Minimum/Maximum) of an objective function. It is one of the most used techniques for altering a model's parameters in machine learning projects in order to lower a cost function. 

Typically, there are three types of Gradient Descent:  

1.     Batch Gradient Descent

2.     Stochastic Gradient Descent

3.     Mini-batch Gradient Descent

What is Stochastic Gradient Descent?

Stochastic gradient descent is an optimization algorithm often used in machine learning applications to find the model parameters that correspond to the best fit between predicted and actual outputs. It’s an inexact but powerful technique.

SGD has the benefit of being computationally efficient, particularly when working with huge datasets. Compared to conventional Gradient Descent techniques that demand processing the complete dataset, employing a single sample or a small batch considerably lowers the computing cost per iteration.

Here's how the stochastic gradient descent algorithm works:

1.     Initialization: Randomly initialize the parameters of the model.

2.     Set Parameters: Determine the number of iterations and the learning rate (alpha) for updating the parameters.

3.     Stochastic Gradient Descent Loop: Repeat the following steps until the model converges or reaches the maximum number of iterations: 

                  a. Shuffle the training dataset to introduce randomness. 

                  b. Iterate over each training example (or a small batch) in the shuffled order. 

                  c. Compute the gradient of the cost function with respect to the model parameters using the current training                         example (or batch).                   

                 d. Update the model parameters by taking a step in the direction of the negative gradient, scaled by the learning rate. 

                  e. Evaluate the convergence criteria, such as the difference in the cost function between iterations of the gradient.

4.     Return Optimized Parameters: Once the convergence criteria are met or the maximum number of iterations is reached, return the optimized model parameters.

 

In SGD, since only one sample from the dataset is chosen at random for each iteration, the path taken by the algorithm to reach the minima is usually noisier than your typical Gradient Descent algorithm. But that doesn’t matter all that much because the path taken by the algorithm does not matter, as long as we reach the minimum and with a significantly shorter training time.

 

 

 

 

 

 

                                    Fig: stochastic gradient optimization path 

 

Conclusion:

Large-scale machine learning models frequently use SGD for training, especially when the full dataset does not fit in memory. SGD offers effective and scalable training by randomly picking tiny chunks of the data. As a result of the numerous parameters and data samples, batch gradient descent is frequently utilized in deep learning algorithms like those for training neural networks.

References

·       https://realpython.com/gradient-descent-algorithm-python/

·       https://www.geeksforgeeks.org/ml-stochastic-gradient-descent-sgd/

 

Narsima Ahmed

@INTERNATIONAL SCHOOL OF MANAGEMENT EXCELLENCE

Intern @Hunnarvi Technologies under guidance of Nanobi data and analytics pvt ltd.

Views are personal.

#gradientdecent #SCD #algorithm #nanobi #hunnarvi #ISME

 

Comments

Post a Comment

Popular posts from this blog

Koala: A Dialogue Model for Academic Research

Image
  Koala: A Dialogue Model for Academic Research Introduction The accidental leak of the weights associated with Meta AI’s LLM LLaMA has sparked tremendous innovation in the open-source LLM space. Since the furious leak, we have seen models like Alpaca, Vicuna, ChatLlama, and several others expand on the foundations of LLaMA to build innovative conversational agents that match the capabilities of ChatGPT. One of the latest additions to the list is Koala (yes, I know, another animal-named model), a chatbot created by Berkeley AI Research (BAIR) that fine-tunes LLaMA on conversations gathered from the internet. The core goal of Koala is to highlight the balance between mega-large closed-source LLMs and smaller, open-source LLMs. BAIR’s thesis is that smaller models can achieve performance that matches mega models like ChatGPT with a fraction of the cost while also improving in areas such as fine-tuning, transparency, and many others. Koala Koala is a version of LLaMA fine-tu...
Read more
  What is FLAN-T5? Artificial Intelligence (AI) has been making strides in recent times, with advancements in Large Language Models (LLMs) and their applications taking much of the credit. One of the most prominent models in this domain is GPT-3, developed by OpenAI. However, it is not the only model making waves. FLAN-T5, developed by Google Research, has been getting a lot of eyes on it as a potential alternative to GPT-3. ·        FLAN stands for “Fine-tuned Language Net”         ·        T-5 stands for “Text-To-Text Transfer Transformer” Back in 2019, Google's first published a paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", introducing a novel where they introduced the original T5 architecture. The pretrained encoder-decoder model worked well on multiple tasks, and particlarly well suited for translation and summarization tasks. In 2022 G...
Read more