Great on article on the difference between L1 and L2 regularization.

By
I find this to be a pretty complex topic, but I think that this article explains the differences very intuitively.

http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/



 What is the simple intuition?

 I am by no means an expert, but here is some basic intuition for why:

 1. L1 regularization (Least absolute errors) produces sparse solutions, and therefore has built in feature selection
2. L2 regularization (Least squares error) does not

 Suppose you had 6 weights, and your L1 regularization term  had a choice between a few sparse weights, or many smaller weights:

 L1 = |0|+|0|+|0|+|-5|+|0|+|1.4| = 4.8
OR 
L1 = |1.2|+|1.3|+|.8|+|2.4|+|1.8|+|1.4| = 8.9

 In this case, your optimization algorithm would converge towards fewer sparse weights because of the absolute value term.

 Now lets take a look at the situation with the L2-norm:

 L2 = 0^2 +0^2+0^2+-5^2+0^2+1.4^2 = 26.96
OR 
L2 = 1.2^2+1.3^2+.8^2+2.4^2+1.8^2+1.4^2 = 14.73

 Now, your optimization algorithm would prefer the smaller but non-sparse weights.

 Thus, you can see how L1 prefers sparse weights, but L2 prefers smaller but non-sparse weights.
Additional Resources:

Great Quora answers to this question: https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization

Why does the l1 norm produce sparse solutions?: https://medium.com/mlreview/l1-norm-regularization-and-sparsity-explained-for-dummies-5b0e4be3938a

Again, the article from above: http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/



Comments

Post a Comment

Popular posts from this blog

grandmaster level chess AI using python - Part 2 (the code)

By
Image
In continuation of part 1, I will now building my chess AI in python.  As mentioned before, my goal is to create a chess AI at the Grandmaster level (elo > 2500) Here is the code for the chess AI. https://github.com/luweizhang/chess_ai Key components of the chess game / AI: Python Classes: ChessGame instantiates the chessboard and pieces keeps track of whose turn it is RulesEnforcer Logic for how all the pieces move, castling Makes sure you cant make illegal moves Used in the chess AI to generation possible positions ChessAi Tree generator generates the tree of position moves Position Evaluator evaluates the quality of chess positions Minimax Determines the optimum move from the chess tree ChessDB Opening database Explanation of the tree generator: The tree generator component of the chess AI generates the possible future board positions based on the possible moves that the pieces can make.  At any given position, there are o...
Read more

building a chess ai - part 4: learning an evaluation function using deep learning (keras)

By
Image
Note: this post is still in progress. Below I will discuss approaches for training a deep learning chess game and the results of my implementation. Approach 1: Train against the outcome of the game *y* is the outcome of the game (1 is win for white, 0 is loss for white, .5 is draw) We want to learn a function * f(p)* that can approximate this. *p*  is the chess position (8x8 chess board), actually is an 8x8x12 = 768 dimensional vector (since there are 12 pieces)      8x8x6  = 384 dimensional vector with positive values for squares with a white piece and negative values for squares with a black piece.  (Using 64 x 12 introduces too many degrees of freedom) The goal is to try to learn a function * f(p)* that can predict the winner of the game * y* given a chess position * p* . The model: Dataset: http://www.ficsgames.org/download.html millions of high quality games played by grand masters or international masters Objective/Loss Function...
Read more

Brief intro to recurrent neural networks

By
Image
Note: This post is meant to be a brief and intuitive summary of recurrent neural networks.   I have adapted this material from the Coursera deep learning course.  The value I hope to add here is that I have attempted to summarize the information in a way that is easy (hopefully) to understand, and can be used as reference or refresher material for the future. Part 1: Recurrent Neural Networks: Recurrent neural networks are a class of neural networks where the nodes/neurons form a directed graph along a sequence.  They are very effective at tasks such as Natural Language Processing because they have a "memory," in other words they can receive context from previous inputs.  They can take in input one at a time, and can pass information from one node to the next via hidden activation layers.  This information serves as the "memory" or "context" layer of the network, which can be used in conjunction as more new input is being processed. RNN Cell: ...
Read more