Many modern neural network implementations are based on GPUs, which were originally developed as dedicated hardware components for graphics applications. Therefore, neural networks benefit from the development of the gaming industry.
Video game rendering requires many operations to be executed quickly and in parallel. The environment and character models are defined by a series of 3D coordinates of vertices. In order to project a large number of 3D coordinates onto the 2D space of a display screen, the GPU must perform matrix multiplication and division on many vertices in parallel.
After this, the GPU must also perform many operations in parallel on each pixel to determine its color. In both cases, the calculations are very simple and do not involve the complex branching operations typically encountered by CPUs.
For example, every vertex in the same object will be multiplied by the same matrix, meaning there is no need for “if” statements to determine which matrix each vertex should be multiplied by. The computations are also independent of each other, enabling parallel execution. The computations also involve handling large memory buffers and bitmap information describing the textures of each object to be rendered.
Overall, this design makes GPUs highly parallel and with very high memory bandwidth. While clock speeds and branching computation capabilities are much weaker than CPUs, these factors are irrelevant during neural network training.
The performance characteristics required for neural network training are actually very similar to graphics algorithms. Neural networks typically involve large buffers for parameters, activation values, and gradients, each of which is updated in every training iteration. These buffers are often too large to fit into the cache of a traditional desktop computer, so memory bandwidth often becomes the main bottleneck. A significant advantage of GPUs over CPUs is their extremely high memory bandwidth.
Neural network training typically does not involve large amounts of branching or complex control instructions, making it more suitable for training on GPUs. Since neural networks can be divided into multiple independent “neurons,” and each neuron is independent of the others within the same layer, neural networks can greatly benefit from the parallel characteristics of GPUs.
However, writing efficient code on a GPU is not always an easy task. The techniques required to achieve good performance on a GPU are very different from those used on a CPU. For example, well-optimized CPU code is often designed to read more information from the cache as much as possible. In a GPU, however, most writable memory locations are not cached, so calculating a value twice is often faster than calculating it once and then reading it from memory.
In summary, a CPU is like a university student who can do a surface integral in 10 minutes, but if you ask him to do 1000 addition problems with three-digit numbers in 10 minutes, he won’t finish. A GPU, on the other hand, is like 100 elementary school students. While each of them may not be as capable as you, together they can complete 1000 three-digit addition problems in 10 minutes.
Disclaimer:
- This channel does not make any representations or warranties regarding the availability, accuracy, timeliness, effectiveness, or completeness of any information posted. It hereby disclaims any liability or consequences arising from the use of the information.
- This channel is non-commercial and non-profit. The re-posted content does not signify endorsement of its views or responsibility for its authenticity. It does not intend to constitute any other guidance. This channel is not liable for any inaccuracies or errors in the re-posted or published information, directly or indirectly.
- Some data, materials, text, images, etc., used in this channel are sourced from the internet, and all reposts are duly credited to their sources. If you discover any work that infringes on your intellectual property rights or personal legal interests, please contact us, and we will promptly modify or remove it.