10.4.1 Smart homes and cities

In the smart home, town or cities, robotic vision is gaining pace in order to accommodate the growing demands of fully automated systems. These systems are supposed to detect and localize any object or person and can make decisions based on those detections. For example, a person enters a room, the system should be able to identify if it’s one of the residents or an intruder and can raise the alarm. In smart towns or cities, detecting a pedestrian can be a first step toward a robust surveillance system that can identify any wanted person or detect any suspicious activity. Garcia et al. [20] presented the basic idea of using an IP camera to automate the process of person detection in smart homes, smart towns, and smart cities. The authors have used MIDGAR architecture which is a popular IOT platform and integrated it with a computer vision module to analyze images from the IP cameras. The real-time monitoring system developed can be used to define actions owing to the presence of a person in an area. Ibrahim et al. [21] designs a framework using Deep Convolutional Neural Network for detecting pedestrians in an urban scene, and also mapping slums and transport modes among other things. The Urban-I architecture comprises three parts. One of which is used for identifying pedestrians using the CNN and Single Shot Detector technique. The framework is useful in smart cities as it serves multiple purposes. Othman et al. [22] combined the IOT system and computer vision module like Garcia et al. [20] to detect people. Using the PIR sensor on a Raspberry 3 card, motion is detected in an area and an image is captured. The computer vision module is then used to detect the position of the pedestrian in that image. The resultant image is sent to a smartphone along with a notification. The proposed system is useful in maintaining security in a smart home. For detecting pedestrians at distance and to improve the resolution of input images, Dinakaran et al. [23] made use of Generative Adversarial Networks (GANs) in combination with a cascaded Single Shot Detector. GAN is helpful in image reconstruction to discriminate features of objects present in the image. This discriminative feature of the proposed system is helpful in differentiating vehicles and pedestrians from a street surveillance camera to make it suitable for smart cities applications. Lwowski et al. [24] proposes a fast and reliable regional detection system integrated with deep convolutional networks for real-time PD. This study has its application in robotics, surveillance, and driving. The proposed study used rg-chromaticity to extract relevant red, green, blue (RGB) channels features and a further sliding window algorithm was utilized for the detection of feature points for classification purposes.

10.4.2 Autonomous driving

Designing an autonomous vehicle or mobile robots crucially requires a robust and accurate PD system. Khalifa et al. [25] propose a dynamic approach to detect moving pedestrians using a moving camera placed on top of a car’s windshield. This framework models the motion of the background using several consecutive frames then differentiates between the background and foreground, then checks for the presence of pedestrians in the scene. The main idea by Wang [26] is to develop a system of PD based on panoramic vision of the vehicle. Center and Scale Prediction, which is a two-stage process of feature extraction and detection, has been designed. Resnet-101 is used as base model which is used for feature extraction using multiple channels. In the detection stage, small-sized convolutions are used to predict the center, scale, and offsets. Pop et al. [27] proposes PD along with activity recognition and time estimation to cross the road for driver assistance systems in order to increase road safety. PD is done using RetinaNet, which is based on ResNet50, using the JAAD dataset for training that is annotated to derive the pedestrian actions before crossing the streets. The actions are classified into four categories: pedestrian is preparing to cross the street (PPC), pedestrian is crossing the street (PC), pedestrian is about to cross the street (PAC), and pedestrian intention is ambiguous (PA).

10.4.3 Tracking

For the purpose of crowd surveillance, it is important to detect pedestrians in a crowded scene, it is also important to track the pedestrians in a scene. The trajectory of pedestrian’s movement is crucial in understanding the flow of the crowd, analysis of crowd behavior, and detection of anomalous behavior in crowd. Mateus et al. [17] proposes a PD and tracking mechanism using multicamera sensors. Detection is done by combining ACF detectors along with Deep Convolutional Neural Network. HAN for a robot in a social context is handled using Gaussian as the cost function for different constraints. In Chen et al. [28] a pedestrian detector is developed using a faster region based convolutional neural networks (R-CNN) and then a tracker is designed using a target matching mechanism by utilizing a color histogram in combination with scale invariant feature transformed into a Fully Convolutional Network to extract pedestrian information. It is also shown to reduce background noise in a much more efficient manner and works well in occluded conditions. Zhong et al. [29] presents a novel technique for estimating the trajectory of pedestrians’ motion in 3D space rather than 2D space. A stereo camera is used to detect and estimate a human pose using the Hourglass Network. For predicting trajectory, SocialGAN is extended from 2D to 3D. The detection of the CNN can be improved by enhancing the representational ability. Yu et al. [30] proposes fusing extracted features such as color, texture, and semantics using different channels of the CNN by emphasizing the correlations among them via learning mechanisms. Yang et al. [31] proposes a pedestrian trajectory extraction technique using a single camera that doesn’t acquire any blind space. A Deep CNN in combination with Kalman filter and Hungarian algorithm to detect pedestrian heads are used to obtain the coordinates of the trajectory points using a novel height estimation technique.

10.4.4 Reidentification

Reidentifying pedestrians is a challenging task. It involves tracking a pedestrian across multiple cameras/scenes as well distinguishing different people at the same time. A number of factors come into play like similarity in appearances, varying camera angles, pedestrian pose, and so on. Fu et al. [32] proposes a two-stream network based on ResNet50 that uses spatial segmentation and identifies global as well as local features. One of the two branches in the network learns global spatial features using adaptive average pooling, whereas the other branch learns the local features using horizontal average pooling to depict the pose of the pedestrians. Qu et al. [33] proposes a Deep Convolutional Neural Network that identifies differences in the local features between the two input images. The DCNN extracts feature vectors from the input images then utilize Euclidean distance to calculate the similarity. For training, the focal loss is used to handle the class imbalance problem in the dataset, thus improving the recognition accuracy of the overall system.

10.4.5 Anomaly detection

Anomaly in terms of surveillance refers to any suspicious activity that is different from the activities occurring in the scene. A dual channel CNN [34] enables identification of suspicious crowd behavior by integrating motion and scene-related information of the crowd extracted from input video frames. This highlights the significance of raw frames along with feature descriptors in identifying an anomaly in crowded scenes. Fully Convolutional Neural Networks (FCN) have also been used recently for detecting abnormal crowd behaviors. These networks can be used in combination with temporal data to detect and localize abnormalities in crowded scenes [35]. Transfer Learning (pretrained CNN) can be adapted to an FCN (by removing the Fully Connected Layers). This FCN-based structure extracts distinctive features of input regions and verifies them with the help of a special-design classifier. Ganokratanaa et al. [36] attempted to detect anomalies and localize them using a deep spatiotemporal translation network based on GAN. Using normal activities framed as a training set, the authors train a system to generate dense optical flow as temporal features of normal events. For application, the input videos are fed into the system and any event that was not in the training set is categorized as anomaly.

10.5.1 Illumination conditions

Illumination conditions play the most crucial role in accurate PD. Several factors can contribute to bad lightening conditions, such as bad weather, nighttime, or insufficient light in an indoor scene.

Researchers [37] designed a multispectral PD system which consists of two subnetworks designed using region-based FCN, by combining color and thermal image information. The model fuses the color and thermal images, and nonmaximum suppression is used to fuse the detection results of the two subnetworks. Researchers [38] proposed a multispectral Deep Convolutional Neural Network that combines the visible and infrared images using a two-branch network which is merged together using three different fusion techniques at different convolutional stages. Color and thermal images and their correlation can also be used as a measure to detect pedestrians in nighttime videos. This illumination measure [39] can be used as a facilitator in the detection process or can be used to design a CNN for the same. CNN fusion architectures have also proved useful in detecting pedestrian in images with illumination problem. Vanilla architectures have been used to obtain and combine the detection results of the fusion architectures. In outdoor scenarios, weather plays an important role as visibility is totally dependent on it. Tumas et al. [40] focused on detecting pedestrians in adverse weather conditions. A new dataset named ZUT captured during severe weather conditions (cloudy, rainy, foggy, etc.) was used to train a system trained using You Only Look Once (YOLO) v3 detector. The detector performs well in varying lighting conditions and under motion blur as well.

10.5.2 Instance size

Position of the camera that captures a scene can have positive as well as negative effects on how a system detects pedestrians in that scene. For example, pedestrians that are closer to the camera can be easily identified but the small instances can even be missed by a human eye. When Deep Neural Networks process these scenes, there is a possibility that they may miss the small instances.

A Scale Aware Fast R-CNN synthesizes two subnetworks into a single framework [41], and detects pedestrians in outdoor scenes where different pedestrian instances possess different spatial scales. The framework takes the location of people (proposal) and the feature map of the scene as inputs, and outputs a prediction scores and a boundary for each input proposals. The framework in [38] uses a Feature Pyramid Network that is shown to improve the detection results for different scales of instances (especially, small-scale instances). A two-stream CNN [42], replicating the human brain, captures spatial and optical flow of the input video sequences for detecting pedestrian using deep motion features and deep still image features. Zhang et al. [43] focused on the detection of small-scale pedestrians using an asymmetrical CNN by considering the shape of the human body as rectangular. This facilitates to generate rectangular proposals that capture the complex features of a human body. A three-stage framework for the CNN deploys coarse-to-fine features to gently dismiss the false detections.

10.5.3 Occlusion

Occlusion has proved to be one of the major hindrances in PD. It becomes quite difficult to detect a pedestrian who is being blocked by an object or another pedestrian.

Mask R-CNN can be used to detect and segment faces in obscured images [44]. This is one way of handling occlusion and increasing robustness of face detection mechanism. GANs can be another intuitive way to handle occlusion. A generative and adversarial architecture integrated with attribute classification can be trained using transfer learning and a discriminator [45]. The generator generates the new image and the discriminator tries to authenticate it, depending on how well it is trained. This is a cool way of reconstructing occlusion-free images of a person which is credible in possible scenarios. Extensive Deep Part detectors are another way of handling occlusion. These detectors cover all the scales of different parts of human body and choose significant parts for occlusion handling [46]. Each part detector can be individually trained for different body parts using a quality training data or transfer learning. The output of each part detector can then be combined using weighted scheme or a direct score. A semantic head detection system [47] in parallel with the body detection system is used to handle occlusion. The head is one of the most crucial parts while detecting pedestrians. Predictions are inferred from the body detector and are used to manually label the semantic head regions for the training process. A Deep Neural Network has taken under consideration the semantic correlation among different attributes of the body parts [48]. The system uses element wise multiplication layer that avails the feature maps to extract different body feature representations from the input. Fei et al. [49] uses context information from the extracted features from the network to handle occlusion. A pixel-level context embedding technique to access multiple regions within the same image for context information using multibranched layers in CNN with varying input channels and an instance-level context prediction framework identifies distinguishing features among different pedestrians in a group.

10.5.4 Scene specific data

One of the factors in the limitation of the performance of the detection models is unavailability of the scene-specific dataset. A model can be trained on different types of generic dataset, but when it comes to real-world applications, it suffers due to scarcity of scene-specific datasets.

In an online learning technique to detect pedestrian in a scene-specific environment, Zhang et al. [50] designed a model on an augmented reality dataset which addresses the issue of unavailability of scene-specific data. Synthetic data is used to simulate scenes of need (which is also available publicly in [51]). The same model is gradually improved over the time as a specific scene changes. Cyget et al. [52] explored the importance of data augmentation toward improving the robustness of PD systems, evaluated the robustness of different available augmentation techniques, and then proposed a novel technique that uses Style-transfer, combining the input image and stylized version of the image in patches to maintain balance. Inspired from Patched Gaussian Augmentation, this work adds patches from the styled image to the original image at the same location to improve detection. Tumas et al. [40] introduced a new dataset, ZUT (Zachodniopomorski Uniwersytet Technologiczny), captured during severe weather conditions (cloudy, rainy, foggy, etc.) from four EU countries (Denmark, Germany, Poland, and Lithuania). It was collected using a 16 bit thermal camera at 30fps and is fine-grained annotated in nine different classes: pedestrian, occluded, body parts, cyclist, motorcyclist, scooterist, unknowns, baby carriage, pets and animals.