Face analysis is essential for a large number of applications such as human–computer interaction or multimedia (e.g. content indexing and retrieval). Although many approaches have been proposed, performance under uncontrolled conditions is still not satisfactory. The variations that may impact facial appearance (e.g. pose, expression, illumination, occlusion, motion blur) make it a difficult problem to solve.

This book is composed of two parts based on the recent PhD work of Belmonte in Part 1 (P1) and Allaert in Part 2 (P2). The focus is on an updated review of the literature. Some experiments and benchmarks are also included.

Part 1 focuses on one of the core building blocks of facial analysis: facial landmark detection. After an introduction that provides the background on the problem, Chapter 2 provides an overview of the literature (methods, data, metrics). Supervised 2D detection in images and image sequences are mainly studied. Chapter 3 presents an analysis of the performance of current approaches and highlights the most difficult poses and expressions to handle. It also illustrates the importance of a suitable temporal modeling to benefit from the dynamic nature of the face. Finally, Chapter 4 addresses the modeling of temporal information. A benchmark of various architecture is proposed to determine the best design for video-based facial landmark detection. It also helps us to understand the complementarity between spatial and temporal information as well as local and global motion. The conclusion discusses possible research perspectives.

Part 2 focuses on a more higher-level task within facial analysis: expression recognition. After an introduction explaining the problem and its challenges, Chapter 2 describes how a face can be characterized according to the literature. Chapter 3 explains how to model the expressions and consider both micro- and macro-expressions. Chapter 4 presents way to characterize facial motion that takes into account mechanical facial skin deformation properties, and a detailed evaluation is provided in Chapter 5. Finally, the acquisition of a dataset, called SNaP-2DFe, to analyze facial expressions in the presence of head pose variations is included in Chapter 6. Research perspectives are discussed in the conclusion.

Although landmark detection is used for expression recognition, the two parts of this book are independent and can be read in either order. Each chapter is also relatively self-contained. To go further, we invite the interested readers to look at other facial analysis tasks, to move towards more practical content, and to try to implement the algorithms mentioned in this book and why not transpose them into useful applications.

We would like to thank Hichem Sahbi for inviting us to contribute to the SCIENCES project. We would like to thank our PhD supervisor Chaabane Djeraba who trusted us and who always remained very available. We also thank our co-supervisors Ioan Marius Bilasco (P1 and P2), Pierre Tirilly (P1) and Nacim Ihaddadene (P1). We are sincerely grateful for the time they spent with us, their patience and the extensive advice they gave us. Also, we would like to thank our PhD reviewers (P1: Jean-Luc Dugelay and Hichem Sahbi; P2: Monique Noirhomme-Fraiture and Jenny Benois-Pineau) and examinators (P1: Karine Zeitouni and Nicu Sebe; P2: Moncef Gabbouj), who agreed to be part of this work and have greatly contributed to its improvement before being published as a book. Many thanks to all our colleagues at the University of Lille and ISEN-Lille, who have always been very kind and supportive. Finally, we would like to thank our family and friends for their constant support.

June 2022