This post is part of an ongoing online book. To access the other parts, please refer to the contents page of the book.

Beyond Ghor, there was a city. All its inhabitants were blind. A king with his entourage arrived nearby; he brought his army and camped in the desert. He had a mighty elephant, which he used to increase the people’s awe. The populace became anxious to see the elephant, and some sightless from among this blind community ran like fools to find it. As they did not even know the form or shape of the elephant, they groped sightlessly, gathering information by touching some part of it. Each thought that he knew something, because he could feel a part…. The man whose hand had reached an ear… said: “It is a large, rough thing, wide and broad, like a rug.” And the one who had felt the trunk said: “I have the real facts about it. It is like a straight and hollow pipe, awful and destructive.”. The one who had felt its feet and legs said: “It is mighty and firm, like a pillar.” Each had felt one part out of many. Each had perceived it wrongly….

Idries Shah - Tales of the Dervishes, 1967

This ancient story was told to teach a simple lesson that is often ignored: The behaviour of a system cannot be known just by knowing the elements of which the system is made.

Introduction

We have come a long way since Santiago Ramón y Cajal published his first iconic drawings of neurons back in the late 19th century. The sheer amount of discoveries made by scientists since then proves that we are gradually converging on an understanding of how the brain works. The ingenious techniques that neuroscientists are developing to record and analyse our brains, are now helping us illuminate one by one parts of a once unknown world. Whereas the neuronal morphology and the molecular mechanisms of different neural structures are relatively well understood, the bigger picture of how neuronal interactions form emergent phenomena remains enigmatic. More than a century after the neuron’s discovery, we still do not know how the neural circuity in our heads gives rise to the mind. Like the blind men beyond Ghor, we are still trying to make sense of the elephant ourselves.

Humans in fact tried to understand the nature of the mind for centuries. In ancient India for example, philosophers believed in a theory called “Samskara”. Samskara meant different things to different people, and it was always somehow mixed with religion and mysticism. It represented the mental impressions, or psychological imprints of a person. Samskaras were explained as characteristics, or behavioural traits that one either possessed from the moment of birth, or that got shaped over time. Indian philosophers of the Nyaya school of Hinduism understood that a newborn child has imprinted memories (even though they did not refer to it like that). They argued that a baby’s instinctive reach for the mother’s breast was a sign that the baby had some prior Samskara. Since no one provided the knowledge of the necessity of the mother’s breast to the baby, and since the baby did not form any samskaras so far, philosophers believed that the newborn’s knowledge came from a ‘‘prior experience’’.

As always, it was the Ancient Greeks however that hit the nail on its head. Alcmaeon of Croton, one of the greatest minds of Ancient Greece, was the first one to propose that the brain is the organ of the mind. Even though this revelation might not sound like a big deal today, back then it was a revolution in human knowledge. Up until then, it was not that obvious to people that the thinking happens in the brain. The brain was just another organ. Even the father of modern science, Democritus of Abdera, who formulated the atomic theory of the universe, was inspired by Alcmaeon’s discovery. Democritus concurred with Alcmaeon’s discovery and argued that perception is a purely mechanistic (or one might say algorithmic) process. He argued that thinking and feeling were simply features of matter that emerge when organised in a sufficiently fine and complex way and not due to some spirit infused into matter by the gods. During the time of Democritus, where everything was fused with spirituality and mysticism, these were not only bold statements, they were world-shattering. We know that Democritus wrote several books about the mind and senses. Some of the known book titles were “On the Mind”, “On the Senses”, “On Flavours”, “On Colours”, and “On Logic” (book titles back then were still simple). Unfortunately, none of Alcmaeon’s and Democritus’ books survived the passage of time, all we know are the book titles and the references from other philosophers. Who knows where we would have been today, if only we would have managed to preserve the memory of these ancient giants.

The Ancient Greeks answered the question of ‘‘where’’ the mind takes place. Today’s scientists on the other hand are trying to answer ‘‘how’’ the mind does what it does. How do we learn? Where and how are memories stored? How does consciousness form? These are quite abstract questions, which because of the way they are asked, are difficult to answer. I strongly believe that to answer these questions, we will have to question every neurobiological structure and biochemical process we see. Why is there a very long dendrite that emerges from the cell body of pyramidal cells? Why are so many excitatory neurons covered with dendritic spines? Why are most interneurons spineless? What’s the purpose of the back-propagating signal that occurs inside a neuron? Why do excitatory and inhibitory neurons look so different? What is the purpose of the enigmatic spine apparatus and why do axons also have such a similar organelle?

I believe that these are the kind of questions that will allow us to reverse engineer the circuits that give rise to the algorithms of our minds. As Albert Szent-Györgyi once said: “If structure does not tell us anything about function, it only means we have not looked at it correctly.” Patterns that repeat over and over again in nature tend to have an important role. In biology after all, form follows function. Instead of focusing on big philosophical questions, I argue that we can deduce and understand the algorithmic parts of the brain by questioning the patterns in the morphology and biochemical processes of neurons. Using a reductionist approach alone however, will not be enough to decipher the mind. a bottom-up approach might generate a lot of facts, but not necessarily new ideas that will lead to discoveries. For that, a holistic mindset is necessary. I am not advocating one mode of thinking over another, on the contrary. We will need to be willing to constantly switch between modes of thinking, to truly tackle complex problems like the mind. As you can quickly notice, one needs to know some basic neurobiology, to understand the questions this book tries to answer. That’s why in the first part of the book, we are going to learn some basic neuroscience.

The main focus of this book will be about how biological memories form and are stored. The whole first part of the book in fact will deal with that topic. I will describe a concrete algorithm for how memories are formed and stored in neural circuits, and how a phenomena called ‘‘neuronal specialisation’’ can emerge by following very simple rules. My proposed model aims to extend Hebb’s rule, with the necessary temporal attributes for an asynchronous system to function. We will see how neural competition is key for Hebb’s neural assemblies to form. Once we know how this model works, I will also explain what the purpose of some neural structures might be from the context of the mind.

My interest is not to write a philosophical, vaguely defined proposal for how memories are formed and stored. Instead I will use a more pragmatic approach and explain a concrete, simple to understand process for it. Even though we do not know all the biochemical puzzle pieces for this problem yet, we will see how by looking at many experiments, we can already come up with a simple unsupervised learning algorithm for neural circuits. I feel it is important to emphasise that while designing the model, I tried to be as strict as possible on its biological plausibility, putting neurobiology first and machine learning second. There will be no no unnecessary abstractions that have little to do with actual biology. This book is after all meant to help us get closer to deciphering the brain.

Instead of immediately trying to answer the big questions, we are going to ask well defined biological questions, like the ones mentioned earlier. We are then going to use the proposed answers to those questions as puzzle pieces for bigger questions, like ‘‘How are memories stored?’’. Often times knowing how to ask the right questions is even more important than the question itself. We will not only examine the proposed model, but the biological structures from which it emerges as well. We will see how many of the questions in neurobiology, including the ones mentioned earlier in this introduction, can be indirectly explained and understood through this model. Moreover, the model that I will propose will also reveal the existence of a mysterious code inside neurons. We will discuss the implications of such a code, as well as the evidence supporting the existence of such a code, in great detail.

As you have probably seen in the contents page, the first part of the book is called “Biology”, whereas the second part “Technology”. While the first part describes the neurobiological structures and the emergent phenomena of neural circuits, the second part of the book focuses on the potential real-world applications of the proposed model. In the second part of the book, I will discuss about artificial neural networks, as well as the emergent field of neuromorphic computing. We will see what the basic idea of machine learning is and how the concepts of machine learning can be quite a useful mental model to reverse engineer the algorithms of biological systems. I will also explain how todays neural networks work, and why I believe that it is so important to figure out alternatives to the famous backpropagation algorithm. At the end of the second part, we will explore some of the existing neural network architectures and I will propose a new kind of architecture based on the proposed model from the first part of the book.

Writing this book turned out to be tricky. On the one hand, the book has to be technical enough so that scientists and researchers do not get annoyed, yet it has to be simple and clear enough, so that people from different backgrounds can nonetheless understand everything. Because of this, I decided that any term or concept used after the introduction, be it neuroscience related or machine learning related, has to be explained in advance. My interest is after all to reach as many people as possible. If we want to inspire someone to adopt a new idea after all, the things we write have to be readable. I also hope that this book will blur a bit the lines between fields. The brain is a complex system, meaning that it consists of many components that interact with one another. Complex systems are inherently hard to model. Knowing the components of the system is only one part of the solution, we also have to figure out how those parts interact. To achieve this, I argue that we have to use an interdisciplinary approach. When studying things, we should not only strive for depth of knowledge, but range as well. Reading papers and books from other fields, can foster lateral thinking and help us come up with completely new ideas. Even if the idea that I will present will turn out not to reflect reality, I hope that the facts presented in this book, and the questions asked will help others decipher the algorithms of the mind.