Are Neural Networks Really Similar To The Brain?

Deep learning has made tremendous progress in various fields such as image recognition, natural language processing, and machine translation. Many of the techniques used in modern deep learning were inspired by biological systems, such as the structure of the brain and the way it processes information. However, it is also true that some of the techniques that were once thought to be biologically inspired have been largely abandoned in favor of newer techniques that have proven to be more effective.

Comparisons

One example of this is the use of sigmoid and tanh activation functions, which were once popular in deep learning. These functions were inspired by the way neurons in the brain process information, but they have largely been replaced by the rectified linear unit (ReLU) activation function. The ReLU function has been found to be more effective because it is able to learn faster and more accurately than the sigmoid and tanh functions.

Another example is the use of spiking neural networks (SNNs), which were inspired by the way neurons in the brain communicate with each other through action potentials or “spikes.” While SNNs have the potential to be more energy efficient than traditional artificial neural networks (ANNs), they have not yet been able to outperform ANNs on a wide range of tasks.

Hebbian learning, which is based on the idea that neurons that fire together wire together, has also been largely abandoned in favor of…