Display Features

The quality of the display varies widely from model to model. This huge variety causes problems when attempting to retarget content for the best viewing experience [10]. Here challenges relate to small size, the available resolution, color gamut, and peak luminance [11].

The first research that investigated the impact of display size on TMO performance was by Urbano et al. [12]. This work only considered static HDR images and compared tone mapping across three different displays: two 17″ displays (one TFT LCD and one CRT) and one 2.8″ display of a smartphone. The experiments considered seven TMOs that were evaluated by a total of 114 participants divided into six groups, randomly assigned by the displays and the scenes. A pairwise comparisons were carried out. The main conclusion of this study was that the participants’ preference regarding the tone-mapping performance changes across displays, and that for smaller displays, participants preferred stronger detail reproduction, more saturated colors, and overall brighter image appearance.

Inspired by this work, Melo et al. [13] conducted a series of experiments designed to study the impact of the HDR video tone mapping. This study consisted of a series of psychophysical experiments that were undertaken by a total of 60 participants. Each participant had to rank seven different HDR videos tone mapped by six TMOs. The experiments considered a conventional-sized display of 37″ and a 9.7″ mobile device. The comparisons were made by having the original HDR video displayed on an HDR display as reference. The experiments were divided into two scenarios, one that evaluated the 37″ LDR display, and the other that evaluated the 9.7″ display (Fig. 2).

The author had expected to corroborate the findings of Urbano et al. [12]; that is, that the tone-mapping performance is different across conventional-sized displays and mobile devices. However, interestingly, this study did not corroborate such findings as the HDR video evaluation showed that participants’ preference regarding HDR video tone-mapping performance did not change across displays. The authors explain this result because, with video the differences between displays are not as obvious, as participants did not have much time to focus on the spatial details of the moving images of the video as they could when observing still images.

Local Storage Availability

Mobile devices typically have much less local storage available than conventional PCs. The consumer magazine, Which? measured the storage available of eight state-of-the-art smartphones and reached the conclusion that brands announce a certain memory size, but the real memory size available is always considerably lower [14]. For example, in one device for which the manufacturer had announced a memory size of 16 GB, on average, there was only about 11 GB available. This is simply insufficient when one considers that each frame of an uncompressed HDR video sequence at full HD resolution requires 24 MB, and a minute at 30 fps is 42 GB [15]. Although efficient HDR video codecs can help, it is simply not possible to store a collection of HDR videos on a mobile device, as one does with LDR videos.

Power Supply

Portability is a key feature of mobile devices, but the “downside” of this is battery life. When processing high-quality multimedia content on mobile devices, there is a higher battery consumption [16]. Mobile devices’ computational power has reached near Moore’s law with an almost exponential evolution of technology, but unfortunately this cannot be said for batteries, in which technology has been developing at a much slower pace.

The wide spectrum of mobile devices gives a large range of battery life as shown in Expert Reviews [17] that evaluated the battery life of 60 state-of-the-art mobile devices. The evaluation consisted of playing the Spider-Man 2 movie continuously with the sound output by a set of headphones. The average battery life for the iOS-based smartphones tested was of 13 hours and 25 minutes, followed by the Android-based smartphones with an average battery life of 11 hours and 40 minutes, and Windows-based with an average battery life of 10 hours.

As the average length of a feature film is 1 hour and 30 minutes, the evaluation results show that it is possible to watch a movie without a concern about battery life. There are, however, two facts that one must account for: (1) users do not want to have their battery totally drained by multimedia consumption and compromise the use of the mobile device for communications or something similar and (2) that HDR video has more impact on battery drain than LDR video. Regarding (1) besides the average battery drain caused by the playing the video, users are unaware of power-saving settings on smartphones [18]. Furthermore, they do not respect the charging cycles leading to a significant reduction of battery life [19]. This can be problematic because the multimedia consumption can drain most of the battery, leaving the mobile device unusable for other purposes such as communications. With (2), HDR can aggravate the problem by up to 30% [20], as it requires significant resources due to constant heavy processing. More recent mobile devices are overcoming this possible limitation with more powerful graphics processing units that aim to relieve the CPU load, leading to a more efficient processing and power consumption [16].

Ambient Illuminance

Studies such as Rempel et al. [21] and Melo et al. [22] were conducted to evaluate the impact of the ambient lighting levels on the HDR video visualization. The first study focused on HDR video visualization under different lighting levels, and proved that the ambient lighting levels does have a significant impact on the viewing experience [21] studied the video viewing preferences for HDR displays under varying ambient illumination. Their results show that there are significant differences in preferred display settings under different ambient lighting conditions. This study consisted of two psychophysical experiments that analyzed the potential for visual fatigue and the viewing preferences on next-generation HDR displays. The first was performed by 10 subjects who had to watch 5 movies of approximately 90 minutes each. Five different ambient lighting levels (0.01, 0.75, 8.5, 28, and 74 lux) were evaluated that ranged from a dark ambient lighting that only has the HDR display as the light source, to a poorly lit environment. To avoid subject fatigue due to the length of the video content, the visualization was undertaken in five different sessions. A total of 17 participants did the study. Each subject watched three television episodes of 22 minutes in a single session. Three different ambient lighting (0.01, 70, and 700 lux) were used that ranged from a dark room to a normal indoors environment.

For both studies, participants were distributed randomly across the lighting levels and the video content and placed at a distance of approximately 1.5 m from the display. Participants were encouraged to adjust the brightness and contrast of the display at the start of each session. They were also informed that the brightness and contrast could change spontaneously during the session, in which case they should adjust brightness and contrast to their desired levels. These data were retrieved from a time-stamped log of all brightness and contrast changes made during a session, using a visual fatigue questionnaire, and a more general questionnaire. For this first psychophysical experiment, the relatively small variations in ambient lighting levels did not have an impact on the preferences for brightness or contrast. With regard to visual fatigue, the results showed that participants experienced remarkably little visual fatigue throughout the 90 minutes sessions, as half of participants reported a value of 0 out of 10 and the average level of visual fatigue reported was 1.18 out of 10. The results were uniform across the tested lighting levels. This second study showed that there is a preference toward maximizing the available display contrast. Furthermore, there is a direct relationship between the preferred display brightness and the level of ambient illumination. Participants tended to prefer more brightness for brighter environments.

The Rempel et al. study [21] focused on HDR displays and does not consider tone mapping needed for visualizing HDR content on conventional displays. As HDR displays are not yet widespread, it is also important to understand the impact of the ambient lighting level of HDR video tone mapping. Such a study was conducted by Melo et al. [22]. This considered both conventional-sized displays and small screen devices typical of mobile devices. This work extends the previous study on HDR TMO performance across different sized displays [22]. Three distinct ambient lighting environments were considered: dark, medium, and bright. The dark ambient lighting had a measured lighting levels of about 15 cd/m², which corresponds essentially to a dark room. The only light was the luminance emitted by the displays used in the experiments. The medium ambient lighting level was about 55 cd/m². This roughly corresponds to the average lighting levels of a living room. The bright ambient light scenario had a luminance level of about 1450 cd/m², which is similar to indoors on an average cloudy day. Six state-of-the-art TMOs were applied to seven different video sequences. A total of 180 participants were randomly distributed across the different ambient luminance scenarios. They ranked the HDR tone-mapped content on a 37″ LDR display, or on a 9.7″ mobile device display using an HDR display as reference. Fig. 2 shows the experimental setup.

This study corroborated the findings of Melo et al. [22]. One interesting contribution of this work is that results showed that while the TMOs were equally ranked on dark and medium ambient lighting environments, the TMO preference is significantly different when compared within the bright lighting environments. This proves that bright ambient lighting levels have a clear impact on the viewing experience and thus, need to be taken into account. This is also in line with the findings by Rempel et al. [21] that verified differences within the participants’ preferred configuration parameters of the display under different ambient lighting levels. Melo et al. also noted that under dark and medium lighting levels, the tone-mapping process should focus more on color and details, while under environments with bright lighting levels, the tone-mapping process should prioritize contrast and naturalness.

One limitation of this study is that the 9.7″ mobile device display is not representative of the wide spectrum of display sizes and features of mobile devices. Smaller display sizes also need to be investigated to see if there is any additional impact on the HDR video tone mapping, as well as to identify any differences between the upper and lower scale of small screen devices as, for example, a tablet and a smartphone.

Reflections

Due to their portability, mobile devices can be easily subject to screen reflections. It is thus important to understand the possible impact of such reflections on the viewing experience. Melo [20] conducted the first study that investigated tone-mapping performance in situations where a mobile device display is under direct reflections. In total, three experimental scenarios were tested: having the display fully under reflections, only half of the display under reflections, and as a benchmark, without reflections on the display (Fig. 3). As reflections are typical of outdoors scenarios, the authors ensured all the experimental scenarios had a measured ambient lighting level of about 1450 cd/m²; approximately the same lighting level as a cloudy day.

The experiments considered two state-of-the-art TMOs, as well as a hybrid approach of both in order to understand if such a hybrid could deal with reflections better. A total of 90 participants ranked 6 videos that were all tone mapped with the chosen TMOs. The comparison was made using a reference shown on an HDR display. The results showed that the greater the area exposed to reflections, the larger the negative impact on a TMO’s perceptual accuracy. Results also showed that a hybrid approach for tone mapping does not outperform the standard TMOs.