How Image Annotation Can Support AI-Based Emotion Detection

Developed by Paul Ekman, Ekman taxonomy classifies human emotions based on facial expressions. These are built on 7-emotion taxonomy. Paul Ekman’s emotions include anger, fear, disgust, happiness, sadness, surprise, and contempt.

This has become industry standard for labelling facial emotions and major datasets such as FER2013 (≈35,000 images) and AffectNet (1M+ images) use these emotion classes to train emotion recognition models.

But these human emotions may not always appear pure and isolated; they often get blended. Sometimes you see emotions that is a combination of multiple emotions such as fear, surprise or even happy. This adds to the confusion, making labelling difficult.

A single emotion is assigned to each image, which speeds up dataset creation, but it struggles to capture the complex human expressions. The training datasets starts looking at emotions as a single category, which can often mislead.

This creates issues because in real world situations simple emotions don’t work. Human emotions are often mixed and complex. This is a challenge in emotion detection systems. Here Facial Action Coding System (FACS) developed by Ekman and Friesen helps. Expressions are broken and identified into 46 distinct facial action units, each linked to a specific muscle movement. This supports correct emotion identification.

This also has its own challenges. Your annotators must be FACS-certified, and such talent is not easily available. FACS-based annotation is slower, more complex, and more expensive than standard emotion labeling.

Different cultures interpret emotions in different ways as people react differently to each situation. To address this the annotation team must be from different cultural backgrounds so that any kind of bias can be avoided. Cultural and geographical diversity among annotators results in balanced emotion labels.

This technique captures the geometry of the face by marking specific points on facial locations. Usually, 68 landmark points, which is a dlib standard, is used to map the facial structure. For more detailed tracking, higher density schemas with 98 points are also used.

Here AI models can measure how facial features move during expressions and correspond closely to Facial Action Units (FACS). This makes landmark annotation effective in capturing emotions without FACS.

This technique struggles when facial features are partially hidden and when landmark points are blocked, the model may lose the geometric structure needed for reliable analysis. In such cases polygon or segmentation annotation can be used to capture the visible regions of the face.

Bounding boxes annotation draw rectangular boxes around faces, giving clarity to the area that model must focus on. This helps AI systems locate faces within an image. Many face detection models such as MTCNN, RetinaFace etc. use these labelled boxes to identify faces.

But again, accuracy is very important here. The box should be perfect; a little loose or too tight may misguide the model. Loose box may include elements not needed, while a tight box can crop things. A perfect box must focus only on the face. Any inaccuracy here may continue in the later stages such as detection, expression classification, and action unit analysis.

Polygon and instance segmentation support AI-based emotion detection by isolating facial regions, which helps them capture expression-relevant areas more accurately than simple bounding boxes.

Polygons, unlike bounding boxes, outline the face following boundaries around lips, eyebrows and eyes. These areas express emotions and they need to be captured and analysed. In crowded areas where faces are often partially covered or angled. Here polygons help by outlining the exact contours of the face.

And then instance segmentation separates each detected face as an independent object. This gives each face its own segmentation mask. This helps in the analysis of emotional expressions for multiple individuals simultaneously. Check image segmentation services for a better understanding of the process.

Traditional emotion recognition systems focus primarily on facial expressions which can provide ambiguous emotional signals. There are many other elements like surrounding people, posture or the context that defines the true emotion. So, just the facial expression cannot be a correct emotion.

Semantic segmentation assigns a label to every pixel in an image. It is not just about the face; the model identifies multiple scene elements simultaneously, like the human body, background objects, surrounding people in the scene, and other environmental features. It is a step beyond just face analysis and moves towards scene understanding.

The context is important as the cues help disambiguate similar facial expressions. The integration of scene semantics with emotional annotation helps infer context and combined emotions. AI can interpret emotional states by capturing signals beyond just facial expressions.

Here multiple labels are assigned to a single face, which helps annotators to interpret the single strongest emotion from the multiple ones.

It could be happiness plus surprise, or fear plus surprise or maybe sadness with anger. So, it is no longer about exclusive categories but more of overlapping emotional states.

This is necessary for naturalistic data because in such a scenario pure Ekman expressions occur rarely. Most frames capture transitional or mixed emotional states. Real world emotional complexity gets challenging when emotion detection models are trained on single-label datasets.

Disagreements between annotators, inconsistent tagging and rare emotions require advanced quality control mechanisms.

Cohen’s Kappa is a measure that decides the level of annotator agreement. Higher level is required in emotion labelling for better performance. Any score below 0.60 is unreliable, while 0.70 is considered good. For high stakes or clinical use 0.80 is desired. High agreement between annotators is needed for high-performing models.

The threshold is applied to each label and not to each image. For instance, if there is an image that is labelled as ‘fearful surprise’ we need to check agreement on both fear and surprise separately. So, we have two separate Kappa measurements.

There is a difference between consensus and a majority vote. Majority vote works faster but can often get biased while the consensus vote is slow but accurate. In case of a majority vote if 2 out of 3 annotators agree on a certain emotion, it becomes the final label. But this ignores minority opinions.

In case of weighted consensus, each annotator’s confidence is considered, which produces more accurate labels, especially for tricky cases.

Annotator drift is a common problem that needs to be fixed. It happens when over a time the label accuracy starts getting inconsistent. Repetitive work often leads to this kind of drift. This kind of drift needs to be fixed. You can insert gold-standard images that is the known correct labels. Once this is into the workflow, the annotators’ labels can be compared with the correct answers using kappa. It is important to run regular calibration checks and realign annotators.

• What Happens: Blur correction, histogram normalization, resolution repair. Morphological processing (boundary extraction, erosion) applied. Faces with extreme occlusion flagged for multi-annotator queue.
• QA Checkpoint: Automated resolution and histogram checks; rejection rate logged per batch.

• What Happens: MTCNN or RetinaFace detects face ROIs. Bounding boxes generated and validated against ground truth.
• QA Checkpoint: IoU > 0.85 threshold; outliers queued for manual review.

• What Happens: 68-point or 98-point keypoint sets placed per FACS-aligned facial geometry. AU-relevant landmarks annotated with sub-pixel precision.
• QA Checkpoint: Landmark position error < 2px against reference model output.

• What Happens: Primary Ekman class label applied. Secondary label added for compound states. Valence/arousal/dominance scores added for AffectNet-compatible pipelines.
• QA Checkpoint: Minimum 3 annotators per image; Cohen’s Kappa > 0.70 required.

• What Happens: Polygon or semantic masks applied for occluded or multi-subject scenes. DMS datasets receive pixel-level body pose labels.
• QA Checkpoint: mIoU validated against held-out test set.

• What Happens: Annotations exported in client-specified format: COCO JSON, Pascal VOC, or CSV keypoints. Schema validation run. Dataset split into train/val/test at agreed ratios.
• QA Checkpoint: Zero schema errors; class distribution checked against project brief.

Cameras are used by retail stores to capture the sentiments of shoppers. It could be frustration during self-checkout, confusion or engagement with displays. Once these get captured the retail staff can be made available to them immediately to solve the issue. It helps identify customer behaviour that is used later to improve customer experiences.

This presents a challenge to the retail industry as people appear at distances or different angles, and often the faces are blocked by carts or shelves. Here the basic model fails as only the frontal face model may not work. Here one needs to use polygon segmentation to isolate the face. The expressions need to be captured accurately and for that 68-point landmarks are combined.

In one Hitech BPO retail project 1.2M+ fashion and décor images were annotated in just 12 days. The California based retail AI company saw a 96% boost in productivity while maintaining strong annotation precision at scale. Read the complete case study here.

EU regulations (ISO 15008, UN ECE R158) require driver monitoring systems in vehicles. These systems must detect drowsiness, signs of inattention, and any kind of distraction. Even the Euro NCAP sets a high bar where eye closure detection must be 95%+ accurate. And to meet these standards, one requires very high-quality training data.

DMS cameras use infrared and not normal RGB. It is important for the data to include head angles also and not just frontal views, as drivers look in many directions. This is managed by using 98-point landmarks, which supports precise eye tracking. Segmentation is used to separate driver from background while gaze direction labels as an extra layer.

It is important to spot unusual behaviour at airports and other similar facilities through emotion detection and facial recognition. It is to identify people showing reactions that don’t match their situation. Poor image quality is again a challenge here. Blurry images, faces blocked in crowds, matching faces from old footage are often a struggle.

Multiple viewers help get a clear label, avoiding vague tagging and ensuring consistency even with blurry or ambiguous faces. Guesswork is completely avoided and constant updates enable long-term, reliable identification.

Understanding patient feelings and emotions without relying on self-reports is important in mental health. Teletherapy and clinical tools use emotion AI to capture the right emotion, but they need to follow strict rules as data privacy is important.

They must capture micro expressions under 200ms and detect subtle or suppressed emotions and follow HIPAA rules with full de-identification before annotation begins. Clinical emotion AI demands ultra-precise labeling and strict data privacy compliance.

To capture very subtle emotions, annotation uses FACS coding at the AU (muscle movement) level by trained experts. It also adds valence, arousal, and dominance scores for deeper emotion analysis. Video is annotated frame-by-frame (at least 30 fps) to track how expressions change over time.