AI Activated

Originally written on April 15th, 2019. Updated on:- December,2019 In this tutorial, we will take a look into Parallel Processing on MMUs and the need for matrix multiplication. Parallel Processing on MMUs Typical RISC processors provide instructions for simple calculations such as multiplying or adding numbers. These are  scalar processors, as they process a single operation (= scalar operation) with each instruction. Even though CPUs run at clock speeds in the gigahertz range, it can still take a long time to execute large matrix operations via a sequence of scalar operations. One effective and well-known way to improve the performance of such large matrix operations is through vector processing, where the same operation is performed concurrently across a large number of data elements at the same time. CPUs incorporate instruction set extensions that express such vector operations. The streaming multiprocessors (SMs) of GPUs are effectively vector processors, with...

Originally written on April 15th, 2019. Updated on:- December,2019 In the 2nd part, we take a look into TPUs and their Architecture. Continuing from the 1st tutorial, a Neural network models consist of matrix multiplies of various sizes — that’s what forms a fully connected layer, or in a CNN, it tends to be smaller matrix multiplies. This architecture is about doing those things — when you’ve accumulated all the partial sums and are outputting from the accumulators, everything goes through this activation pipeline. The non-linearity is what makes it a neural network even if it’s mostly linear algebra. Neural networks are just a series of matrix operations applied to input data And if there’s a lot of data to input, that’s a lot of matrix operations to compute. Like a lot. Matrices full of numbers all being multiplied in parallel -Most of the math is just 'multiply a bunch of numbers, and add the results' .We can connect these two together in a single oper...

Originally written on April 15th, 2019. Updated on:- December,2019 In this tutorial series, we will be taking a look into Tensor Processing Units or TPUs. I have divided the whole series into a number of posts. What is a Tensor Processing Unit? A tensor processing unit (TPU) is an AI accelerator application-specific integrated circuit (ASIC) developed by Google specifically for neural network machine learning. Google has been using TPUs in their data centers since 2015. In the above graph you can see how the Deep Learning demand at Google has increased since 2013. The TPU production started in 2015. Google has designed them specifically for machine learning applications. They use them for Google Translate, Photos, Search Assistant, Gmail, Cloud, etc. Why did they make their own chip? 1) Neural Networks in particular almost always outperform other machine learning models if given enough data & compute. 2) Neural networks require...