At the very beginning of my project, I conducted a literature review on the neural mechanism of how the visual and auditory stimuli could elicit a facial expression. Most of the contents below were obtained and re-arranged from Wikipedia. I acquired a rough understanding of the topic after writing this short article. In this article, we will first sketch out the structure and function for some of the parts that are associated with this study, followed by a stepwise description of how the visual and auditory stimulus can result in facial expression. Finally, we will talk about the human reaction time. Throughout this article, we do not consider the factor of emotion.
It is worth noting that some of the theories included in this article are controversial and debatable, e.g., the two-stream hypothesis proposed in 1991. Also, the exact and detailed functions or mechanisms for lots of brain components are still unknown. Fortunately, my project does not require me to be a physiologist. What I care about is the fact that there exists a delay between the stimulus received and the facial expression elicited, and how long it roughly takes.
Human Nervous System
The human nervous system consists of two components which are the central nervous system (CNS) and the peripheral nervous system (PNS). The CNS includes the brain and the spinal cord. The PNS includes the cranial nerves, spinal nerves with their roots and branches, peripheral nerves, and neuromuscular junctions. They are nerves and ganglia outside of the brain and spinal cord.
Figure 1. The illustration of the brain.
The brain consists of the cerebrum, brainstem, and cerebellum, as shown in Fig. 1. The cerebrum is the largest part of the brain which performs high-level functions, such as interpreting touch, vision, and hearing, or speech, reasoning emotions, etc. The cerebellum locates under the cerebrum. It mainly coordinates muscle movements, maintains posture and balance. The brainstem connects the cerebrum and cerebellum to the spinal cord. It controls automatic functions such as breathing, heart rate, digestion, vomiting, swallowing and sneezing, etc.
The cerebrum consists of four lobes, which are the frontal lobe, parietal lobe, occipital lobe, and temporal lobe, as shown in Fig. 2.
Figure 2. The illustration of the cerebrum. As the color depicted, the sensory strip belongs to the parietal lobe, the motor strip and Broca area belong to the frontal lobe, and the Wernicke area is part of both the parietal and temporal lobe.
- The frontal lobe is involved in emotions, reasoning, motor control, and language. It contains the motor strip, which plans and coordinates movement; the Broca's area, which is basically for language production.
- The parietal lobe integrates sensory information among various modalities, such as touch, odor, pressure, pain, and temperature, etc.
- The occipital lobe interprets vision, such as color and light. The visual cortex is located in the occipital lobe which receives the electrical nerve signal from a visionary stimulus. The primary visual cortex (PVC) is one component of the whole visual cortex.
- The temporal lobe processes sound, interpret languages and speeches. The auditory cortex is located in the temporal lobe. The primary auditory cortex (PAC) is one component of the whole auditory cortex.
The limbic system, also referred to as the paleomammalian cortex, is a set of brain structures surrounding the thalamus, immediately beneath the temporal lobe. It supports a variety of functions including emotion, behavior, motivation, memory, and olfaction. Emotion is largely related to the limbic system. The limbic system is shown in Fig. 3.
Figure 3. The illustration of the limbic system.
- The hypothalamus nuclei controls body temperature, hunger, parenting and attachment behaviors, thirst, fatigue, sleep, and circadian rhythm.
- The cingulate gyrus is included by the cingulate cortex, the latter is highly influential in linking motivational outcomes to behavior.
- The amygdala is involved in many cognitive processes, e.g., memory, attention, and emotion (including fear, anxiety, and aggression). It helps an organism to define a stimulus and therefore responds accordingly, e.g., fight or flight. The left and right amygdala have different functions. In one study, electrical stimulations of the right amygdala induced negative emotions, especially fear and sadness. While for the left amygdala both the pleasant (happiness) or unpleasant (fear, anxiety, sadness) could be induced.
- The corpus callosum is a wide thick bundle of nerve fibers. It connects the left and right cerebral hemispheres, enables the communication between them. It is the largest white matter structure in the human brain.
- The hippocampus is involved with various processes such as spatial memory and learning. It is critical for effectively combining the what, when and where status of each experience to compose the retrieved memory. It also contributes to learning, as researchers discovered an upsurge of new neurons and neural circuits in it after some training tasks.
- The thalamus is a large mass of gray matter with several functions such as relaying of sensory signals. including motor signals to the cerebral cortex and the regulation of consciousness, sleep, and alertness. It is like a relay station or hub. Visual stimuli from the retina are sent to it first and later projected to the visual cortex in the occipital lobe.
The Cranial Nerves
Twelve pairs of cranial nerves are included in the PNS (they always present symmetrically on both sides of the body). Together with the cerebrum, they control much of the motor and sensory functions of the head, neck, and shoulder. An overview of the cranial nerves is shown in Fig. 4.
Figure 4. The illustration of the twelve pairs of cranial nerves.
Of all the twelve pairs of cranial nerves, the optic nerve carries visual information from the retina of the eye, the vestibulocochlear nerve carries the information from the inner ear about hearing and balance, and the facial nerve controls the muscle of facial expression. They mainly transfer the signals from eyes and ears to the primary visual and auditory cortex or transfer the command from the motor cortex to the muscles in an opposite direction.
The Pathway from Stimulus to Facial Expression
Vision is generated when the stimulus hits the retina of the eye. The photoreceptors in the retina converse the stimulus into an electrical nerve signal and send it to the visual cortex in the occipital lobe. As shown in Fig. 5, starting from the eyes, the signal is transferred along the optic nerves all the way down to the primary visual cortex, located in the occipital lobe.
Figure 5. The illustration of the optic nerve.
When the acoustic energy (in the form of sound waves) impinges the tympanic membrane, the cochheal nerve (part of the vestibulocochlear nerve) will conduct the information to cochheal nuclei, the superior olivary nucleus, the medial geniculate nucleus, and finally the auditory radiation to the primary auditory cortex.
The information at the primary visual and auditory cortex are then processed by the two-streams system. The two-streams hypothesis is a model of the neural processing of vision as well as hearing, which is illustrated in Fig. 6. It argues that when the visual information exits the primary visual cortex, and when the auditory information exits the primary auditory cortex, they follow two pathways, the ventral stream (also referred to as "what" pathway) and the dorsal stream (also referred to as "where/how" pathway). The former leads to the temporal lobe, and is involved with object identification and recognition, and the latter leads to the parietal lobe, and is involved with coordinating the object relative to the viewer, and with the speech repetition.
Figure 6. The illustration of the two-streams hypothesis. The arrows along the purple and green areas stand for the ventral and dorsal streams, respectively.
The multi-sensory information outputted by the two-streams system is integrated by the posterior parietal cortex (PPC). The PPC acts as a bridge between the parietal lobe and the frontal lobe (as shown in Fig. 7) and is responsible for transforming multi-sensory information into motor commands or motor planning. Subsequently, the primary motor cortex contributes to generating neural impulses that pass down to control the execution of movement. The impulses for facial expression are transferred to the pons of the brainstem and then the facial nerve emerges.
Figure 7. The illustration for the posterior parietal cortex.
The facial nerve involves the motor part, which arises from the facial nerve nucleus in the pons; and the sensory part, which arises from the intermediate nerve. These two components are responsible for the controlling of facial expression and the sensory from the face, as shown in Fig. 8. The former is the motor control of all the muscles related to facial expression. which finally results in the facial muscle movement according to the impulses from the facial nerve.
Figure 8. The illustration of the facial nerve emerged from the pons of the brainstem.
Examples of the Pathway
The overall transition of the signal from the retina and tympanic membrane to the facial muscles are shown in Fig. 9.
Figure 9. The journey of how a stimulus finally elicits a facial expression. (i): visual area in yellow, where the primary visual cortex is located. (ii): auditory area in green, where the primary auditory area is located. (iii): sensory area, where the information is integrated. (iv): motor area in red, where the motor impulse is generated. (v): pons, slightly below the cerebrum, where the facial nerve emerges.
The stimulus is first received by the retina and tympanic membrane. It is then passed to the primary visual（Fig. 9-i) and auditory（Fig. 9-ii) cortex through the optic and vestibulocochlear nerves, respectively. The information is then processed by the two-stream system, and is passed to the parietal lobe (Fig. 9-iii） for sensory integration. After which it is sent to the frontal lobe (Fig. 9-iv) for generating the impulse of muscle movement. The impulse is sent to pons (Fig. 9-v), facial nerve, and finally to the facial muscle to elicit the facial expression.
Human Reaction Time
Supposing that a ball is approaching you, what will happen to you physiologically? Let's see the figure below.
Figure 10. The reaction to an approaching ball. Recall the aforementioned example we know that between Step 3 and Step 4 the information is processed by the two-stream system. Unlike the facial muscle movement which is controlled by the cranial nerve located in our head, this time the signal will travel through our spinal cord to reach our arms and hands. Illustration: https://backyardbrains.com/experiments/reactiontime.
This kind of study is called Mental Chronometry. It studies the reaction time (RT) in perceptual-motor tasks to infer the content, duration, and temporal sequencing of mental operations. The RT is constrained by the speed of signal transmission in the white matter, and the processing efficiency of neocortical gray matter. The mean RT for college-age individuals is about 160 milliseconds (ms) and 190 milliseconds to detect auditory and visual stimuli, respectively.
There are basically four paradigms to measure the RT.
- The first one is Simple RT. It simply requires the subject to respond to a stimulus. For example, to press a button when the light or sound appears.
- The second one is Recognition RT. It requires a subject to press a button for one stimulus and withhold the response for another type of stimulus. For example, the subject has to press the button when a green light appears and not press when a blue light appears.
- The third one is Choice RT. It requires distinct responses for each possible class of stimulus. For example, the subject is asked to press one button if a red light appears and a different button if a yellow light appears.
- The fourth one it Discrimination RT. It first requires the subject to compare the simultaneously presented stimulus, and then press one of two buttons accordingly.
As the complexity increases, the actual RT increases as well. Fig. 11 illustrate the timeline for the brain to recognize, compare, move the hands, and finally to correct the error.
Figure 11. The timeline for a brain to respond to the number correction task. In this experiment, the subject's event-related potentials from electroencephalography (EEG) signal is recorded. The subject is required to press a key with their left or right hand, as fast as they can, to indicate whether the numbers they see are larger or smaller than 5. Illustration: Dehaene, S. (2011). The number sense: How the mind creates mathematics. OUP USA.
At least four processing stages are identified. At 150ms the brain recognized that These are Arabic digits (or a spelled number). At 190ms the brain made the comparison. At 330ms the brain planed and execute the manual response. At 470ms the brain might correct occasional false pressing.
In this article, we described some key structures and functions of the brain components. We then explained the pathway of how a visual or auditory stimulus can finally elicit a facial expression. Finally, we discussed the response time for the stimulus-to-motor pathway to complete.