2 minute read

Review of SGD / Mini-Batch, Online-Learning, intro Map-reduce, Linking posts in Jekyll, R loopy functions (lapply, sapply, tapply, vapply, split)

Batch gradient descent can be slow for large training sets \(m>100,000,000\) since it only updates once per epoch. We can instead update the parameters with every training example, This is Stochastic Gradient Descent, this results in a lot of variance and oscillations in our parameters and doesn’t really converge to a single value unless we decrease the learning rate \(\alpha\) which reduces the oscillations. Mini-Batch gradient descent is a halfway compromise between Batch and SGD where in each iteration we go over a subset of our training set. This also allows us to take advantage of vectorization and parallelization libraries and exploit GPUs.

Map-Reduce is a way to parallelize our computations over multiple machines / cores. For example if we have a summation involving our entire training set we can send subsets of that summation to different machines and they can return their answers to a final aggregating machine which combines their results together.

Online-Learning is just stochastic gradient descent where we are using live data as our training example instead of an instance from our training set. This refers to cases where we have a stream of data like from users clicking on websites or some sort of sensory input etc. This also has the advantage that it allows the algorithm to adapt to changing user preferences since our data is dynamic.

Can include links to posts in jekyll by using this syntax:


[Name of Link]({% post_url 2020-02-28-name-of-post %})

Funny thing is, trying to type the above example in a code block causes jekyll to break and github pages to fail to build page since it cant find any post with that name, but doesnt give this error of post not found … This was fixed by using raw as follows:

R has some loopy functions that map functions to lists. The most simple ones are lapply which applies a function to a list and returns a list, sapply is a wrapper on lapply which returns a matrix or vector or a list depending on the shape of the result of applying our function. tapply is another function which applies a function to every element in the list based on some other factor list that we provide. vapply is useful when we want to enforce vector output. split helps to break apart lists based on factors. tapply is a sort of wraper for split+lapply.