What happens in one’s brain when two objects, never before seen together (say, an apple on top of a dolphin), are imagined together for the first time? The scientific consensus is that a familiar object, such as an apple or a dolphin, is encoded in the brain by a network of neurons known as a neuronal ensemble (Hebb 1949). When one recalls such an object, the neurons of that object’s neuronal ensemble tend to activate into synchronous resonant activity (Quiroga et al. 2008). The neuronal ensemble binding mechanism, based on the Hebbian principle “neurons that fire together, wire together,” came to be known as the binding-by-synchrony hypothesis (Singer 1995, Singer 2007). However, while the Hebbian principle explains how we perceive a familiar object, it does not explain the infinite number of novel objects that humans can purposefully imagine. The neuronal ensembles encoding those objects cannot jump into spontaneous synchronized activity on their own since the parts forming those novel images have never been seen together and therefore lack enhanced synaptic connections. Vyshedskiy proposed that to account for the limitless human imagination, the binding-by-synchrony hypothesis would need to be extended to include the phenomenon of mental synthesis whereby the prefrontal cortex actively and intentionally synchronizes independent neuronal ensembles into one morphed image  (Vyshedskiy 2013). Thus, to imagine an apple on top of a dolphin, the firing of the apple neuronal ensemble would have to be synchronized with the firing of the dolphin neuronal ensemble, enabling the perception of the two disparate objects together, in one mental frame.

The synchronization mechanism of mental synthesis is likely responsible for many imaginative and creative traits that philosophers and scientists have recognized as being uniquely human, despite not having a precise neurological understanding of the process. For example, Lev Vygotsky claims, “Imagination is a new formation that is not present in the consciousness of the very young child, is totally absent in animals, and represents a specifically human form of conscious activity” (Vygotsky 1967). Ian Tattersall writes, “... if there is one single thing that distinguishes humans from other life-forms, living or extinct, it is the capacity for symbolic thought: the ability to generate complex mental symbols and to manipulate them into new combinations. This is the very foundation of imagination and creativity: of the unique ability of humans to create a world in the mind...” (Tattersall 1999). Envisioning fictional characters such as a three-headed dog, counterfactual thinking, playing strategy games, engineering, design, scientific modeling: all of these endeavors rely heavily on one’s ability to synthesize novel images in the mind’s eye.

Mental synthesis is also an integral part of human language. When we speak we use mental synthesis to describe a novel image (“My house is the second one on the left, just across the road from the red gate”) and we rely on the listener to use mental synthesis in order to visualize the novel image.  Mental synthesis is essential for understanding common elements of language, such as: flexible syntax (“My friend ate a fish” vs. “A fish ate my friend” – notice that you have to mentally synthesize two disparate novel images involving your friend and a fish  in order to correctly understand the meaning of the two statements), verb tenses (“My friend ate a fish” vs. “My friend was eaten by a fish”), and spatial prepositions (“put a bowl behind a cup” vs. “put a bowl in front of the cup”). Flexible syntax, verb tenses, prepositions, adjectives, and other common elements of grammar facilitate the human ability to communicate an infinite number of novel images with the use of a finite number of words – the quality of language that linguists refer to as “recursion.” Mental synthesis is acquired by children uniformly across all cultures around the age of three along with a culturally determined vocabulary. Understanding the basis of mental synthesis can shed light on the evolution of the brain in general and on the evolution of language in particular.

What are the neurological bases of mental synthesis? Neuronal ensembles encoding sensory memories are largely located in the posterior cortex (temporal, parietal and occipital lobes), while the temporal organization of behavior is primarily a function of the frontal cortex (Fuster 2008). Single neuron recordings in monkeys (Miyashita 2004) and functional brain imaging studies in humans (Fuster 2008) demonstrate that memory recall is associated with activation of neuronal ensembles in the posterior sensory cortex and is under the executive control of the prefrontal cortex (PFC). Vyshedskiy hypothesized that mental synthesis is primarily orchestrated by the PFC, which activates and synchronizes independent neuronal ensembles in-phase with each other (Vyshedskiy 2013). When two or more independent neuronal ensembles are activated to fire synchronously in one mental frame, they are consciously experienced as one unified object or scene. In this process humans can manufacture an unlimited number of novel mental images and can plan their future actions through mental simulation of the physical world. The PFC can be viewed as a puppeteer controlling its puppets (memories encoded in neuronal ensembles stored in the posterior cortex). By pulling the strings, the PFC puppeteer activates and changes the firing phase of the neuronal ensemble puppets. Phase-synchronized neuronal ensembles are consciously experienced as a novel whole object or scene. For example, to imagine something you have never seen before, such as your favorite cup on top of your computer’s keyboard, your PFC (1) activates the neuronal ensemble of the cup, (2) activates the neuronal ensemble of the keyboard, and then (3) synchronizes the firing of the two ensembles in time.

The Mental Synthesis Theory can be tested in a conceptually simple experiment using the established technique of single-cell recording in patients undergoing treatment for epilepsy.

Over the past two decades, considerable excitement was generated within the neuroscience community by experiments which recorded the neuronal activity of fully conscious and responsive human patients (Quiroga et al. 2008, Quiroga et al. 2007, Quiroga et al. 2005, Gelbard-Sagiv et al. 2008). Working with patients awaiting treatment for epilepsy, scientists were able to successfully record the neuronal activity within the patient’s brain. A single implanted electrode is able to record the activity of up to eight nearby neurons. Patients were then shown thousands of images presented to them by the researchers with the goal of triggering activity in one of the recorded neurons. In addition, neuronal activity was measured when patients were not actively looking at an image, but were either visualizing the object or freely recalling images in the mind’s eye, all while reporting their perception. 

Recoding from hundreds of patients demonstrated that most neurons in the temporal lobe are object-selective. That is, a given neuron fires at a high rate only when the patient perceives a specific object, such as an animal, a baseball, the Sydney Opera House, or Bill Clinton, and not when any other object is perceived  (Kreiman et al. 2000, Gelbard-Sagiv et al. 2008). For example, a neuron selective to Bill Clinton would fire at a high rate of over 30 action potentials per second only when the patient perceived Bill Clinton. Remarkably, the neuron was activated regardless of whether the patient was shown a photograph of Bill Clinton (in a variety of positions and postures), heard the words “Bill Clinton” or simply recalled Bill Clinton from memory. The neuron would not fire when the patient perceived any other object.  

It is important to note that the Bill Clinton neuron is not the only neuron activated when a patient perceives Bill Clinton. Likely it is one of thousands of neurons encoding the perception of Bill Clinton; it just happens to be the neuron that the scientists are recording from. There is significant evidence that the perception of every object is associated with the synchronous firing of thousands of neurons located throughout the brain (for review see Quiroga et al. 2008, Singer 2007; Buzsaki 2004). The neurons encode the various characteristics of an object, such as its shape, color, and texture; they fire synchronously whenever someone perceives the object. As noted above, this group of synchronized neurons encoding a particular object is referred to as a neuronal ensemble (Quiroga and Kreiman 2010, Waydo et al. 2006​). 

Now that we have reviewed the neurology behind the perception of a familiar object, let’s return to the question raised in this manuscript: what happens when two objects that have never been seen together are imagined together for the first time? Such integration or mental synthesis occurs routinely in our mind, however the mechanism of this process is unclear. To answer this question, we propose the following experiment, which is an extension of the scientific paradigm described above. Two or more very selective neurons encoding disparate objects would be identified in patients with implanted electrodes: for example, a neuron that is part of the Bill Clinton neuronal ensemble, and another neuron that is part of the neuronal ensemble of a lion. Let us call them the Clinton neuron and the lion neuron for the sake of brevity. Whenever the patient recalls Bill Clinton, the Clinton neuron we are recording from would fire at a high rate. Similarly when the patient recalls a lion, the lion neuron would fire at a high rate. The proposed experiment, however, would record what happens when the patient imagines these two objects together, for example: Bill Clinton holding the lion in his lap, Fig. 1. If the Mental Synthesis Theory is correct, both the Clinton neuron and the lion neuron will increase their firing rate and, more importantly, their firing activity would be synchronized, implying the synchronization of the Clinton and the lion neuronal ensembles, Fig. 2.

Figure 1.

Mental synthesis of Bill Clinton holding a lion. Once selective neurons for Bill Clinton and the lion are identified, a subject can be asked to imagine Bill Clinton holding the lion on his lap. The Mental Synthesis theory predicts that both the Clinton neuron and the lion neuron will increase their firing rate and that their activity will be synchronized. 

Images modified from: 1. William J. Clinton at the Parliament in London, United Kingdom, November 29, 1995.  https://commons.wikimedia.org/wiki/File:Bill_Clinton_1995_im_Parlament_in_London.jpg 2. Lioness in the Olomouc Zoo at Svatý kopeček, Czech Republic. This image is licensed under the CC BY-SA license.  https://commons.wikimedia.org/wiki/File:Lioness,_Olomouc.jpg

Figure 2.

On a neurological level, mentally forming the image of Bill Clinton and the lion consists of the following steps:

Step 1 - Recall of Bill Clinton: The prefrontal cortex (PFC) activates the ensemble of neurons representing Bill Clinton to fire synchronous actions potentials. Bill Clinton is perceived by the patient. The electrode implanted into the temporal lobe (TL) records an increased rate of action potentials.

Step 2 - Recall of the lion: The PFC activates the ensemble of neurons representing the lion to fire synchronous actions potentials. The lion is perceived. The second electrode implanted into the TL records an increased rate of action potentials.

Step 3 - The patient mentally integrates the images of Bill Clinton and the lion into one scene. The Mental Synthesis Theory hypothesizes that integration is accomplished by the PFC synchronizing the two neuronal ensembles in time.

Step 4 - When synchronization of the Clinton and the lion neuronal ensembles is achieved, a new, never-before-seen mental image of Bill Clinton holding the lion on his lap is perceived by the patient. At that moment the two implanted electrodes are predicted to record synchronous action potentials, implying the synchronization of the Clinton and lion neuronal ensembles.

In addition to the visualization task which would give unprecedented insight into the nature of imagination, patients could also be shown an actual image of Bill Clinton holding the lion on his lap. Again, we would expect to see both the Clinton neuron as well as the lion neuron increase their firing rate and for their action potentials to synchronize.

Since researchers can often identify several object-selective neurons within a single patient, multiple novel pairings of objects can be studied. Furthermore, morphing of more than two objects into one mental frame can also be investigated. For example, if researchers happen to identify selective neurons for Bill Clinton, the Sydney Opera house, and a lion, the subject can be asked to imagine Bill Clinton sitting next to the Sydney Opera house and holding the lion. In this case, all three neurons would be expected to fire synchronously. This experimental paradigm also paves the way for many other interesting experiments studying the neuroscience of imagination. For example, what would happen on the neuronal level if: Bill Clinton was imagined as a statue rather than as a human being?; the lion was seen fighting Bill Clinton, rather than sitting on his lap?; the subject was to imagine the lion swallowing Bill Clinton?; etc.

Conclusions