ResidualMaskingNetwork

Facial Expression Recognition using Residual Masking Network

(*) State-of-the-art Facial Expression Recognition / Emotion Detection.

Live Demo

Try it in your browser — huggingface.co/spaces/phamquiluan/ResidualMaskingNetwork. Inference runs entirely client-side through ONNX Runtime Web, so images never leave your machine. Source in space/.

Installation

1. Install from pip

pip install rmn

2. Or build from source

git clone [email protected]:phamquiluan/ResidualMaskingNetwork.git
cd ResidualMaskingNetwork
pip install -e .

Quick Start

from rmn import RMN
import cv2

# Initialize the model
m = RMN()

# Detect emotions from an image
image = cv2.imread("your-image.png")
results = m.detect_emotion_for_single_frame(image)
print(results)

# Draw results on the image
image = m.draw(image, results)
cv2.imwrite("output.png", image)

Webcam Demo

from rmn import RMN
m = RMN()
m.video_demo()

Table of Contents

• Benchmarking on FER2013
• Benchmarking on ImageNet
• Download datasets
• Training on FER2013
• Training on ImageNet
• Evaluation results
• Download dissertation and slide

Benchmarking on FER2013

We benchmark our code thoroughly on two datasets: FER2013 and VEMO. Below are the results and trained weights:

Model	Accuracy
VGG19	70.80
EfficientNet_b2b	70.80
Googlenet	71.97
Resnet34	72.42
Inception_v3	72.72
Bam_Resnet50	73.14
Densenet121	73.16
Resnet152	73.22
Cbam_Resnet50	73.39
ResMaskingNet	74.14
ResMaskingNet + 6	76.82

Results in VEMO dataset could be found in my thesis or slide (attached below)

Benchmarking on ImageNet

We also benchmark our model on ImageNet dataset.

Model	Top-1 Accuracy	Top-5 Accuracy
Resnet34	72.59	90.92
CBAM Resnet34	73.77	91.72
ResidualMaskingNetwork	74.16	91.91

Datasets

• FER2013 Dataset (locate it in saved/data/fer2013 like saved/data/fer2013/train.csv)
• ImageNet 1K Dataset (ensure it can be loaded by torchvision.datasets.Imagenet)

Training on FER2013

To train the networks, you need to specify the model name and other hyperparameters in the config file (located at configs/*) then ensure it is loaded in main file, then run training procedure by simply run main file, for example:

python main_fer.py  # Example for fer2013_config.json file

The best checkpoints will chosen at term of best validation accuracy, located at saved/checkpoints. By default, it will train alexnet model, you can switch to another model by edit configs/fer2013_config.json file (to resnet18 or cbam_resnet50 or my network resmasking_dropout1.

Training on the Imagenet dataset

To perform training resnet34 on 4 V100 GPUs on a single machine:

python ./main_imagenet.py -a resnet34 --dist-url 'tcp://127.0.0.1:12345' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0

Ensemble method

I used the no-weighted sum average ensemble method to fuse 7 different models together, to reproduce results, you need to do some steps:

1. Download all needed trained weights and locate them on the ./saved/checkpoints/ directory. The link to download can be found in the Benchmarking section.
2. Edit file gen_results and run it to generate result offline for each model.
3. Run the gen_ensemble.py file to generate accuracy for example methods.

Dissertation and Slide

• Dissertation PDF (in Vietnamese)
• Dissertation Overleaf Source
• Presentation slide PDF (in English) with full appendix
• Presentation slide Overleaf Source
• ICPR Paper
• Hugging Face Paper Page
• Hugging Face Model (trained weights)
• ICPR Poster Overleaf Source

Citation

Pham Luan, The Huynh Vu, and Tuan Anh Tran. “Facial Expression Recognition using Residual Masking Network”. In: Proc. ICPR. 2020.

@inproceedings{pham2021facial,
  title={Facial expression recognition using residual masking network},
  author={Pham, Luan and Vu, The Huynh and Tran, Tuan Anh},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={4513--4519},
  year={2021},
  organization={IEEE}
}

Star History

License

This project is licensed under the MIT License - see the LICENSE file for details.