Facial Coding 2023-05-05T14:34:57+00:00

UNDERSTANDING EMOTIONS THROUGH FACIAL CODING

Emotions

Emotion can be defined as any relatively brief conscious experience characterized by intense mental activity and a high degree of pleasure or displeasure. Scientific discourse has drifted to other meanings and there is no consensus on a single definition. Emotion is often intertwined with mood, temperament, personality, disposition, and motivation.

Emotions are
multicomponent
response tendencies
that unfold over
relatively short timespans.

Emotions begin with an individual’s assessment of personal meaning of some antecedent event
– environment
relationship, or
encounter.

This appraisal process (conscious or unconscious) triggers a cascade of response tendencies manifesting across loosely coupled component systems.

Subjective experience

Facial expressions

Physiological change

Facial coding is something that we do every day. Our face is capable of making 10,000 unique expressions, but only 7 of them are universal.
Face provides different types of signals to convey different kinds of messages.

MULTISIGNAL

  • STATIC (such as skin color)
  • SLOW (such as permanent wrinkles)
  • RAPID (such as raising the eye-brows)

MULTIMESSAGE

  • EMOTIONS
  • MOOD
  • AGE
  • SEX
  • INTELLIGENCE

Emotional messages are transmitted by rapid facial signals. These are quite interesting because when you feel an emotion, it blinks on your face.

Universal emotions are basically the same in all cultures, although some negative emotions may not be shown (e.g. in Japanese culture, it is not customary to express negative emotions in public).

Regarding the basic expressions, there are:

  • Happiness
  • Surprise
  • Anger
  • Fear
  • Disgust (the only one that is unliteral)
  • Sadness

People experience a number of distinct emotions.

Adult human can distinguish at least 50 to 100 emotional states.

Some emotion theorists have identified 6-10 so-called basic emotions,
such as

anger, fear, sadness, excitement,
happiness, surprise, shame, joy,
disgust and anxiety.

Facial expressions are spontaneous and difficult to suppress as the muscles responsible for their change are directly linked to the brain.

Explicit insights

Implicit insights

Tracking and measuring emotional
reactions provide information on the
subconscious and, therefore, more
honest responses to ads end
products than stated reactions.

Logical thinking

Conscious behavior

Language and meaning

Emotion

Attention

Non-verbal

Emotions are key in communication as the biggest part of our decision making process does not include rational reasoning.

Emotional engagement is the key to content marketing success.
One of the definitions proposes that engagement is “the amount of subconscious ‘feeling’ going on when advertisement is being processed.” (R. Heath)
Emotional engagement shows us the “hidden face” of stated behavior.

From the moment we are born, we start to “measure” emotional reactions of others. First coders in our life are our mothers – does she smile, is she afraid, sad or even angry? Facial (de)coding is not some superpower ability – we all know how to read faces – only some are better than others, while some are trained.
Paul Ekman and Wallace Friesen gathered human experience of face reading in one place and named it FACS – Facial Action Coding System (first time published in 1978).

History

In 1872, Charles Darwin published “The Expression of the Emotions in Man and Animals”. He compared numerous images of humans and animals in different emotional states and suggested that some basic emotions, like anger, fear and sadness, are universal and present across ethnicities, cultures and even species. According to Darwin, facial expressions are inborn (not learned) and common to humans and animals (some human characteristics, like clenching teeth in anger or tears in eyes when sad, have animal origin).

Despite Darwin’s theory, the prevailing belief in 1950 was that facial expressions were determined by cultural background and learning process.

In 1960s, American psychologist Paul Ekman set out to visit people from different nations (including an isolated indigenous tribe in Papua New Guinea) to study non-verbal behavior across cultures. His research showed that Darwin was right and that facial expressions and emotions are universal as people from diverse cultural background interpreted expressions in photos in similar way. Ekman’s work indicated existence of 7 basic emotions that are universally present: happiness, surprise, fear, anger, disgust, sadness and neutral.
In 1978, Ekman and Friesen updated Facial Action Coding System (FACS), originally developed by a Swedish anatomist Carl-Herman Hjortsjö. FACS is a tool for classification of all facial expressions that humans can make. Each component of facial movement is called an action unit (AU) and all facial expressions can be broken down to action units. Ekman and Friesen identified 46 different units to describe each facial movement.

Ekman, Friesen, and Joseph C. Hager published a significant update to FACS in 2002. Major updates, beside the general changes, consist of elimination of some AUs, elimination and reduction of scoring rules, and modification of some scoring rules.

Types of facial coding

What is facial coding?

Facial coding is the process of measuring human emotions through facial expressions. Emotions can be detected by FACS trained coders or by computer algorithms for automatic emotion recognition that record facial expressions via webcam. This can be applied to better understanding of people’s reactions to visual stimuli.

Manual facial coding

The most comprehensive catalogue of unique facial action units (AUs) is The Facial Action Coding System (FACS). It describes each independent motion of the face and their groups, showing patterns of facial expressions which correspond to experienced emotion.

Facial Action Coding System (FACS) is a system used for classifying human facial movements by their appearance on the face. Movements of individual facial muscles are encoded by FACS as slightly different instant changes in facial appearance. It is a common standard to systematically categorize the physical expression of emotions.
FACS allows measurement and scoring of facial expressions in an objective, reliable and quantitative way.

Main strength of FACS is the high level of detail contained within coding scheme, while the biggest setback is the time consuming process which includes at least two FACS-trained coders in order to get accurate results.

Automated facial coding

The computer algorithm for facial coding extracts the main features of face (mouth, eyebrows etc.) and analyzes movement, shape and texture composition of these regions to identify facial action units. Therefore, it is possible to track tiny movements of facial muscles in the individual’s face and translate them into universal facial expressions that convey happiness, surprise, sadness, anger and others.

Automatic facial expression recognition procedures are built on groundbreaking research and made available to the general public.

These technologies use cameras embedded in laptops, tablets, and mobile phones or standalone webcams mounted to computer screens to capture videos of respondents as they are exposed to content of various categories.
Companies like EyeSee can use inexpensive webcams that eliminates the requirement for specialized high-class devices, making automatic expression coding ideally suited to capture face videos in a wide variety of naturalistic environmental settings such as respondents’ homes, workplace, car, public transportation, and many more.

Emotions behind expressions

Neutral – Content elicits no emotion.

Fear – Great motivator and causes people to take action. Must be combined with possible solutions for the fear-causing problems.

Happiness/Smile/Joy – Content is positive, inspirational, has bigger chances to be shared or talked about.

Sadness – Usually a response to unfortunate events, rarely used in adverts, unless is followed by the happy ending.

Surprise  – Common element of advertisement content. Can not stand by itself, must be combined with other emotions. Implies something unexpected.

Puzzlement – The content is not clear enough. Person can put some mental effort (focus) in getting the content right, or not (confusion).

Disgust – Content is repulsive. Can be shared among small groups. Implies that the content is not accepted positively.

Discontent (Contempt and Not-Face) – Content is not likable.

Explanation of Heightened Interest – Puzzled expression

Puzzled expression shows that the content is not clear enough. Two facial expressions are typical for this mental state:

Puzzled Focus

The head is getting closer to the zone/content of interest

Eyebrows down
and pulled together

Focus in eyes

Eyebrows raised and
pulled together

Losing focus in eyes

Upper lip pulled down
Lower lip thrown out

Puzzled Confused

Focus /concentration shows heightened interest in some content.
When person is focused it means that she/he is putting some mental effort to get the content right.

When person shows confused face, it means that she/he has no interest in figuring out what is the meaning of content.
No mental effort is put in the getting the content right.

Applications

Medicine

  • Pain detection
  • Monitoring of depression
  • Improving communication for individuals on the autism spectrum

Human resources

  • Online employee recruitment

Gaming

  • Games that adapt to emotions of the players

Advertising and media testing

  • Prediction of advertising campaign success
  • Prediction of advertised content virality.

Automotive industry

  • Prevention of falling asleep on the wheel.

Automatic emotion recognition

  • Construction of different devices (music players that detect emotional state and play appropriate tunes)
  • Robotics
  • Video communication

Progress in facial coding technology and its accessibility enabled application in the field of market research. It can be used to test marketing communication such as advertising, shopper and digital campaigns. Respondents are exposed to visual stimuli (TV Commercial, Animatic, Pre-roll, Website, DM etc.), while algorithm registers and records their facial expressions via their webcam. The analysis of the obtained data provides results that indicate valence over time, engagement level, emotional peaks and possibilities for improvement. Some companies conduct this type of research in-house, while others engage private companies specialized in facial coding services such as EyeSee.

Facial coding is an objective method for measuring emotions. There are two reasons for that:

  • facial expressions are spontaneous
  • muscles responsible for facial coding are directly linked to the brain

The main advantages of facial coding are:

  • It is not based on stated behavior (read: no biases)
  • Cost effective, fast and scalable (N>50; fMRI or EEG)
  • Facial expressions are universal

Facial coding measures emotions through facial expressions and helps answering:

  • TV Commercial: Will it arouse emotions? Which emotions?
  • New concepts: Which messages evoke emotions?
  • Websites: Are visitors frustrated, confused or surprised?

The results of facial coding provide insight into viewers’ spontaneous, unfiltered reactions to visual content, by recording and automatically analyzing their facial expressions. It provides moment-by-moment emotional and cognitive metrics. Facial expressions are tracked in real time using key points on the viewer’s face to recognize a rich array of both emotional and cognitive states such as enjoyment, attention and confusion. Many of the users’ responses are so quick and fleeting that viewers may not even remember them, let alone be able to objectively report about them.

Automatic facial coding of naturalistic and spontaneous facial expressions has many applications in various fields – from medical applications, through human resources, automotive and entertainment industry, to commercial uses such as advertising research and media testing.

Technology

How does it work?

1. Face detection in video or in a single image – face recognition algorithm detects the face. The most frequently used algorithm is Viola Jones Cascaded Classifier, often used in the camera of your smartphone or laptop. The result is a box frame around the face.

2. Accurate modeling of the face – prominent facial features (eyes, brows, mouth, nose, etc.) are detected and algorithm’s landmarks are positioned on them. This process makes internal face model which matches the respondent’s actual face. Face model is a simplified version of the actual face – it has fewer details (face features), but it contains all face features involved in making universal facial expressions. Whenever the respondent’s face is moving or changing the expression, the face model follows up and adapts itself to the current state.

3. Emotion detection – differently positioned and orientated algorithm’s landmarks on the face model are fed as an input into classification part of the algorithm which compares it to other face models in the database (dataset) and translate those face features into labeled emotional expressions, Action Units codes and other “emotional” metrics. Comparing the actual face model with other face models in dataset and translating face features into desirable metrics is accomplished statistically – the dataset contains statistics and normative distribution of all features across respondents from multiple world regions, demographic profiles and recording conditions (dataset must contain data recorded “in the wild”, as well as data recorded in the lab condition – perfect illumination, lenses, etc). After comparison, classifier returns a probabilistic result – expectancy that the position and orientation of facial landmarks match one of the 7 universal expressions.

Contact

Contact us for a talk

UNDERSTANDING EMOTIONS THROUGH FACIAL CODING

Emotions

Emotion can be defined as any relatively brief conscious experience characterized by intense mental activity and a high degree of pleasure or displeasure. Scientific discourse has drifted to other meanings and there is no consensus on a single definition. Emotion is often intertwined with mood, temperament, personality, disposition, and motivation.

Emotions are multicomponent response tendencies that unfold over relatively short timespans.

Emotions begin with an individual’s assessment of personal meaning of some antecedent event – environment relationship, or encounter.

This appraisal process (conscious or unconscious) triggers a cascade of response tendencies manifesting across loosely coupled component systems.

Subjective experience

Facial expressions

Physiological change

Facial coding is something that we do every day. Our face is capable of making 10,000 unique expressions, but only 7 of them are universal. Face provides different types of signals to convey different kinds of messages.

MULTISIGNAL

  • STATIC (such as skin color)
  • SLOW (such as permanent wrinkles)
  • RAPID (such as raising the eye-brows)

MULTIMESSAGE

  • EMOTIONS
  • MOOD
  • AGE
  • SEX
  • INTELLIGENCE
  • ...

Emotional messages are transmitted by rapid facial signals. These are quite interesting because when you feel an emotion, it blinks on your face.

Universal emotions are basically the same in all cultures, although some negative emotions may not be shown (e.g. in Japanese culture, it is not customary to express negative emotions in public).

Regarding the basic expressions, there are:

  • Happiness
  • Surprise
  • Anger
  • Fear
  • Disgust (the only one that is unliteral)
  • Sadness

People experience a number of distinct emotions.

Some emotion theorists have identified 6-10 so-called basic emotions, such as

anger, fear, sadness, excitement, happiness, surprise, shame, joy, disgust and anxiety.

Adult human can distinguish at least 50 to 100 emotional states.

Facial expressions are spontaneous and difficult to suppress as the muscles responsible for their change are directly linked to the brain.

Explicit insights

Implicit insights

Tracking and measuring emotional reactions provide information on the subconscious and, therefore, more honest responses to ads end products than stated reactions.

Logical thinking

Conscious behavior

Language and meaning

Emotion

Attention

Non-verbal

Emotions are key in communication as the biggest part of our decision making process does not include rational reasoning.

Emotional engagement is the key to content marketing success. One of the definitions proposes that engagement is “the amount of subconscious ‘feeling’ going on when advertisement is being processed.” (R. Heath) Emotional engagement shows us the “hidden face” of stated behavior.

From the moment we are born, we start to “measure” emotional reactions of others. First coders in our life are our mothers – does she smile, is she afraid, sad or even angry? Facial (de)coding is not some superpower ability – we all know how to read faces – only some are better than others, while some are trained. Paul Ekman and Wallace Friesen gathered human experience of face reading in one place and named it FACS - Facial Action Coding System (first time published in 1978).

History

In 1872, Charles Darwin published “The Expression of the Emotions in Man and Animals”. He compared numerous images of humans and animals in different emotional states and suggested that some basic emotions, like anger, fear and sadness, are universal and present across ethnicities, cultures and even species. According to Darwin, facial expressions are inborn (not learned) and common to humans and animals (some human characteristics, like clenching teeth in anger or tears in eyes when sad, have animal origin).

Despite Darwin’s theory, the prevailing belief in 1950 was that facial expressions were determined by cultural background and learning process.

In 1960s, American psychologist Paul Ekman set out to visit people from different nations (including an isolated indigenous tribe in Papua New Guinea) to study non-verbal behavior across cultures. His research showed that Darwin was right and that facial expressions and emotions are universal as people from diverse cultural background interpreted expressions in photos in similar way. Ekman’s work indicated existence of 7 basic emotions that are universally present: happiness, surprise, fear, anger, disgust, sadness and neutral. In 1978, Ekman and Friesen updated Facial Action Coding System (FACS), originally developed by a Swedish anatomist Carl-Herman Hjortsjö. FACS is a tool for classification of all facial expressions that humans can make. Each component of facial movement is called an action unit (AU) and all facial expressions can be broken down to action units. Ekman and Friesen identified 46 different units to describe each facial movement.

Ekman, Friesen, and Joseph C. Hager published a significant update to FACS in 2002. Major updates, beside the general changes, consist of elimination of some AUs, elimination and reduction of scoring rules, and modification of some scoring rules.

Types of facial coding

What is facial coding?

Facial coding is the process of measuring human emotions through facial expressions. Emotions can be detected by FACS trained coders or by computer algorithms for automatic emotion recognition that record facial expressions via webcam. This can be applied to better understanding of people’s reactions to visual stimuli.

Manual facial coding

The most comprehensive catalogue of unique facial action units (AUs) is The Facial Action Coding System (FACS). It describes each independent motion of the face and their groups, showing patterns of facial expressions which correspond to experienced emotion.

Facial Action Coding System (FACS) is a system used for classifying human facial movements by their appearance on the face. Movements of individual facial muscles are encoded by FACS as slightly different instant changes in facial appearance. It is a common standard to systematically categorize the physical expression of emotions. FACS allows measurement and scoring of facial expressions in an objective, reliable and quantitative way.

Main strength of FACS is the high level of detail contained within coding scheme, while the biggest setback is the time consuming process which includes at least two FACS-trained coders in order to get accurate results.

Automated facial coding

The computer algorithm for facial coding extracts the main features of face (mouth, eyebrows etc.) and analyzes movement, shape and texture composition of these regions to identify facial action units. Therefore, it is possible to track tiny movements of facial muscles in the individual’s face and translate them into universal facial expressions that convey happiness, surprise, sadness, anger and others.

Automatic facial expression recognition procedures are built on groundbreaking research and made available to the general public.

These technologies use cameras embedded in laptops, tablets, and mobile phones or standalone webcams mounted to computer screens to capture videos of respondents as they are exposed to content of various categories. Companies like EyeSee can use inexpensive webcams that eliminates the requirement for specialized high-class devices, making automatic expression coding ideally suited to capture face videos in a wide variety of naturalistic environmental settings such as respondents’ homes, workplace, car, public transportation, and many more.

Emotions behind expressions

Neutral – Content elicits no emotion.

Fear – Great motivator and causes people to take action. Must be combined with possible solutions for the fear-causing problems.

Happiness/Smile/Joy – Content is positive, inspirational, has bigger chances to be shared or talked about.

Sadness – Usually a response to unfortunate events, rarely used in adverts, unless is followed by the happy ending.

Surprise  – Common element of advertisement content. Can not stand by itself, must be combined with other emotions. Implies something unexpected.

Disgust – Content is repulsive. Can be shared among small groups. Implies that the content is not accepted positively.

Puzzlement – The content is not clear enough. Person can put some mental effort (focus) in getting the content right, or not (confusion).

Discontent (Contempt and Not-Face) – Content is not likable.

Explanation of Heightened Interest – Puzzled expression

Puzzled expression shows that the content is not clear enough. Two facial expressions are typical for this mental state:

Puzzled Focus

The head is getting closer to the zone/content of interest

Eyebrows down and pulled together

Focus in eyes

Focus /concentration shows heightened interest in some content. When person is focused it means that she/he is putting some mental effort to get the content right.

Eyebrows raised and pulled together

Losing focus in eyes

Upper lip pulled down Lower lip thrown out

Puzzled Confused

When person shows confused face, it means that she/he has no interest in figuring out what is the meaning of content. No mental effort is put in the getting the content right.

Applications

Medicine

  • Pain detection
  • Monitoring of depression
  • Improving communication for individuals on the autism spectrum

Human resources

  • Online employee recruitment

Gaming

  • Games that adapt to emotions of the players

Advertising and media testing

  • Prediction of advertising campaign success
  • Prediction of advertised content virality.

Automotive industry

  • Prevention of falling asleep on the wheel.

Automatic emotion recognition

  • Construction of different devices (music players that detect emotional state and play appropriate tunes)
  • Robotics
  • Video communication

Progress in facial coding technology and its accessibility enabled application in the field of market research. It can be used to test marketing communication such as advertising, shopper and digital campaigns. Respondents are exposed to visual stimuli (TV Commercial, Animatic, Pre-roll, Website, DM etc.), while algorithm registers and records their facial expressions via their webcam. The analysis of the obtained data provides results that indicate valence over time, engagement level, emotional peaks and possibilities for improvement. Some companies conduct this type of research in-house, while others engage private companies specialized in facial coding services such as EyeSee.

Facial coding is an objective method for measuring emotions. There are two reasons for that:

  • facial expressions are spontaneous
  • muscles responsible for facial coding are directly linked to the brain

The main advantages of facial coding are:

  • It is not based on stated behavior (read: no biases)
  • Cost effective, fast and scalable (N>50; fMRI or EEG)
  • Facial expressions are universal

Facial coding measures emotions through facial expressions and helps answering:

  • TV Commercial: Will it arouse emotions? Which emotions?
  • New concepts: Which messages evoke emotions?
  • Websites: Are visitors frustrated, confused or surprised?

The results of facial coding provide insight into viewers’ spontaneous, unfiltered reactions to visual content, by recording and automatically analyzing their facial expressions. It provides moment-by-moment emotional and cognitive metrics. Facial expressions are tracked in real time using key points on the viewer’s face to recognize a rich array of both emotional and cognitive states such as enjoyment, attention and confusion. Many of the users’ responses are so quick and fleeting that viewers may not even remember them, let alone be able to objectively report about them.

Automatic facial coding of naturalistic and spontaneous facial expressions has many applications in various fields – from medical applications, through human resources, automotive and entertainment industry, to commercial uses such as advertising research and media testing.

Technology

How does it work?

1. Face detection in video or in a single image – face recognition algorithm detects the face. The most frequently used algorithm is Viola Jones Cascaded Classifier, often used in the camera of your smartphone or laptop. The result is a box frame around the face.

2. Accurate modeling of the face – prominent facial features (eyes, brows, mouth, nose, etc.) are detected and algorithm’s landmarks are positioned on them. This process makes internal face model which matches the respondent’s actual face. Face model is a simplified version of the actual face – it has fewer details (face features), but it contains all face features involved in making universal facial expressions. Whenever the respondent’s face is moving or changing the expression, the face model follows up and adapts itself to the current state.

3. Emotion detection – differently positioned and orientated algorithm’s landmarks on the face model are fed as an input into classification part of the algorithm which compares it to other face models in the database (dataset) and translate those face features into labeled emotional expressions, Action Units codes and other “emotional” metrics. Comparing the actual face model with other face models in dataset and translating face features into desirable metrics is accomplished statistically – the dataset contains statistics and normative distribution of all features across respondents from multiple world regions, demographic profiles and recording conditions (dataset must contain data recorded “in the wild”, as well as data recorded in the lab condition – perfect illumination, lenses, etc). After comparison, classifier returns a probabilistic result – expectancy that the position and orientation of facial landmarks match one of the 7 universal expressions.

Contact

Contact us for a talk