7.2.1. Human haptic perception

Human haptic receptors reside and are distributed in the skin to form the sense of touch. These receptors can detect pressure, heat, cold and pain (Schmidt 1986). Pressure sensors or mechanoreceptors are the ones responsible for haptic and tactile sensations. These receptors have an absolute and relative threshold value of detection. Concerning the absolute perception, this results from the deformation of the skin due to applied pressure. As such, indentations of the skin of the order of magnitude of 0.01 mm (10 μm) suffice to produce tactile sensations when they occur on the inner surface

[7.1]

where ΔI is the difference threshold (JND), I is the initial stimulus and k is the constant that signifies that the ratio stays constant. The above Weber equation states that the psychophysical perception of a signal change is therefore proportional to the stimulus intensity itself. In general, the JND for human haptic perception ranges from k ≤ 15%. If a change in haptic force magnitude is less than this range, the user would not perceive that force (Nitsch et al. 2010). The margin of no perception can be used for data compression, as we will see in section 7.4.

7.2.2. Telerobotic tactile and haptic perception

The main control operation of a teleoperated robot in the Tactile Internet comes from the operator at the master domain. The role of sensing is to detect information before encoding it and conveying it by a communication channel for reproduction on the teleoperator side. The information consists of the status of the environment surrounding the slave robot. The teleoperation is then composed of a fast continuous, uninterrupted sensing, encoding and reproduction of tactile sensations. However, intermittent autonomous behavior may also be required on the slave side to encounter any extended delays in the sensory signals’ continuous feedback loop to the master operator. This is similar to a human who acquires sensory-motor coordination through learning based on a considerable amount of sensed data accumulated in past practices, as shown by Arimoto et al. (2005); Inoue and Hirai (2014), who show the analogy of this human capacity on the two-finger robotic hand. In these works, a model and a proportional–integrative–derivative (PID) control system are used to grasp objects without the need for tactile sensing on the robotic fingertips. However, most grasping robotic hand systems depend on tactile sensing for the controller feedback. In such cases, perception of the object’s material and shape is essential for a local assistance of the master-slave controlled robots.

A perception process is required to interpret and represent touch sensing information to observe and determine the object properties and a model for eventual local decision-making. It is essential in the absence or interruption of master control signals to let the telerobotic recognize unintentional collisions or make intentional physical contact with objects or humans, or in other words to improve environmental awareness when dealing with objects and humans.

The detection and perception of the materials and objects by a telerobot is not only useful for autonomously executed tasks and local decisions, but it can also be used in the compression of feedback signals used for the master domain reconstruction since realistic sensing is required for the distant human operator with minimum bandwidth utilization. As a result of the perception, some packed parameters extracted from a model can be fed back to the master to reconstruct the haptic effects on the master side. In addition to models, machine learning can also be used by the robot tactile sensing to recognize objects according to their properties and materials, which can be done to a very good extent (Xie et al. 2019).

For authentic feedback, the relevant tactile information for TI or telepresence applications extracted from sensing data are the shape, material properties and the object pose (Luo et al. 2017). In the following sections, the different techniques for these types of sensing are covered.

7.2.3. Tactile sensing for material recognition

The functioning of the Tactile Internet is comparable to a prosthetic hand device with tactile sensing feedback used to improve the grasp task’s success rate. Therefore, an object’s surface material properties are essential for an interactive action with the surrounding environment. The most critical parameters to be detected and conveyed by a slave robot to the human master are the surface texture and object stiffness. Tactile sensors detect the object’s surface by a sliding exploratory movement on their surfaces, while the object stiffness can be detected using force sensors. Concerning the texture, many types of sensors with their own data processing methods are available, such as acoustic sensors with a frequency domain analysis of the sensed data to detect the different textures. The stiffness, which is the inverse of compliance, shows the rigidity of the object. Compliance can be described as the relationship between an object’s deformation and the force applied to it. Therefore, one method to estimate the compliance is by measuring a given skin-like sensor’s deformation when using a constant force on it. Piezoresistive sensors with a linear variation of conductance as a function of the applied force, as shown in Figure 7.1, can be used in this case.

7.2.4. Tactile sensing for object shape recognition

The telepresence system assistance for the master side can be through identifying the shape of tackled objects. This can be for the aims of reconstruction, local control of objects or feature extraction for data compression. An object’s perception can be done using image processing and recognition techniques invariant with image scale, translation and rotation (Lowe 1999). However, this may not be very practical in the control tasks of telerobotic and telepresence applications. In the latter cases, tactile object perception is necessary. Two tactile senses will interact to perceive the objects: the cutaneous sense to recognize the local shape and the combination of cutaneous and kinesthetic senses to detect the object’s global contours. This can be summarized in the perception of a Rubik’s cube and a simple smooth cube shown in Figure 7.2.

The sensors for this type of perception are mostly a two-dimensional array of sensors to detect the different amounts of applied force on the object. Figure 7.3 shows a tactile sensor organized in a matrix array of pressure sensors. Many data processing methods exist to recognize the local shape, such as raw data manipulation or statistical feature extraction. Otherwise, the data collected from the array of sensors can be seen as a two-dimensional image, with each sensor representing a taxel.

The local object recognition is then done by features adapted from computer vision applied to the array taxels such as moment estimation, feature extraction from the tactile image histogram, principal component analysis (PCA) and the data extract features. Finally, neural networks can be applied with unsupervised learning and incremental learning to classify objects. The latter method is also useful for global shape perception.

To estimate the global shape, the point cloud method is also used.

7.2.5. Tactile sensing for pose estimation

The object pose is not essential for Tactile Internet applications since vision assistance is also available for the master side. This helps with the localization and manipulation of objects with telerobotic arms and hands.

In summary, Table 7.1 shows the computational techniques used in object recognition using different tactile sensors and application domains. The table is extracted from a detailed review of the tactile sensing techniques of robot hands given in Kappassov et al. (2015).

7.3.1. Haptic interface for telepresence

Haptic perception refers to both the cutaneous (tactile) and kinesthetic sense, which conveys important information about distal objects and events (Loomis and Lederman 1986). Haptic perception plays a significant role in telerobotics to enable an effective human–machine interface in addition to video and audio interfaces. These interfaces are also used to manipulate virtual objects, in CAD/CAM applications, before manufacturing them. Artificial sensors and actuators are used to capture and convey this information to remote sites. For example, in the bilateral telepresence system, both the master and slave domains are equipped with these sensors and actuators. On the master domain side, as explained before, the ensemble of the sensors and actuators is called the haptic interface. It is used to translate the operator’s manual dexterity while giving a realistic force reflection, position and touch feeling from the manipulator (slave domain).

Haptic interfaces enable human–machine communication through kinesthetic and tactile senses. Many anatomical parts of the human body, such as hands, skin and limbs, can provide the haptic communication channel between human and machine. However, the primary currently available commercial haptic interfaces are based on manual touch. A haptic exoskeleton is another way to communicate with a device. Commercial exoskeleton products for health and work, also called wearable robots, are available (https://exoskeletonreport.com/). These products are mainly known for personal and local uses other than for remote-controlled parts. However, in space research, more specifically the European Space Agency (ESA), a demonstration example of telepresence with exoskeleton can be found in Schiele (2014) and ESA (2014).

The haptic interface hardware design is based on a human contact scenario using hands, limbs or skin. It will also depend on the target application because of the complexity required to interface the human haptic sensory and mechanical movement. For example, suppose we need a plier gripper effector to execute a task at the slave domain in a telerobotic application. In that case, it is not necessary to use an exoskeleton-type haptic interface on the master side. A haptic interface’s hardware can vary from a gaming joystick, a multiple DOF stylus to a wearable exoskeleton device (Malley and Gupta 2008), for the kinesthetic to an array of factors for tactile sensing. Following the studies of Millman and Colgate (1991), Adelstein and Rosen (1992) and Minsky (1995) in force-reflecting and tactile interfaces, respectively, commercial haptic devices are now increasingly available. Figures 7.4 and 7.5 show some of the commercial haptic interface products available. Both haptic and tactile interfaces can be integrated into a single interface to feedback force and tactile from the remote device.

In 1980, Bejczy (1980) listed the difficulties in the mechanical design of haptic interfaces due to several problems, “These problems concern linkage of joint geometry, kinematic redundancy and dexterity, structural stiffness and dynamics, actuators for power and precision, motion transmission within the manipulator, mathematics for geometry and dynamics of mechanical arms, properties of end effectors and mechanical evaluation criteria.”. He then listed the challenges in the human–machine interface research domain. Here are some that are still valid:

• – the form of the sensors should be adapted to the real-world environment (task);
• – the control and command language should be tailored to the mechanical, sensing and electronic properties of the manipulator system;
• – the interface with the operator should be such that to intensify his or her capabilities by using increased degree of freedom (DOF), 3- to 6-DOF devices are available commercially.

To conclude this section, we cannot neglect the importance of video and audio information in telepresence applications. In fact, in industrial robotics, where automation plays the leading role, visual technologies are used by robots to outline objects. However, this is not sufficient; a more detailed and precise definition of objects and their handling are obtained using force-reflective sensing. In his visionary article, Sutherland (1965) argued that there is no reason only hands and arms should be used to control the computer due to their high dexterity. He adds that eye dexterity is also high and that machines to interpret eye motion will also be built. The range of interface devices, including haptic ones, can still increase based on new sensor and actuator technologies. In the following section, a list of haptic sensors and actuators is given as an example.

7.3.2. Haptic and tactile sensors and actuators

Tactile sensors are defined according to Lee and Nicholls (1999) as: “a device or system that can measure a given property of an object or contact event through physical contact between the sensor and the object”. The property of the object includes its shape and texture. Many authors provide reviews on the types of sensors and their working principles (Rossi 1991; Amato et al. 2013; Dahiya and Valle 2013). This section presents the different kinds of tactile and kinesthetic sensing devices and a brief about the basics of their functioning.

• – Resistive and piezoresistive sensors: the resistance of the sensor changes according to two variables; first, the contact position, and second, the contact force. The first variable is of the potentiometer type, and the second is of the piezoresistance type. In the potentiometer type, two resistive sheets are stacked in two layers. Each sheet is made of many lines of resistive voltage dividers, one horizontally and the other vertically. When stacked on each other, the two layers form a grid of resister circuits. When a point is pressed on the sheet, contact is made between the horizontal and vertical voltage dividers. The voltage dividers’ output voltage in x and y directions is a function of the position of the contact. In the same way as the resistive sensor, to add a contact force sensing capability, two-layer lines of conductors are stacked with a piezoresistance material between two layers (Robertson and Walkden 1985). A pressure exerted on the surface will create a variable resistance that can be scanned with a particular electronic circuit to locate the position and evaluate the force as shown in Figure 7.6. Another technique that allows measurement of multiple hybrid resistive tactile sensing is given by Hong Zhang and So (2002). Another type of sensor based on piezoresistive touch sensors is also given in Goger et al. (2009).

• – Capacitive sensors: capacitive measurement methods are widely used nowadays in human-computer interfaces such as touch screens and trackpads. The functioning principles are based on the capacitance variation, which is a function of the distance between the plates and the area of its plates as shown in Figure 7.7. When a force is applied to the capacitor where the normal force component changes the plates’ distance, the tangential force component changes the plates’ effective area. This change in capacitance is then converted into a voltage change.

• – Optical sensors: one example of this type of sensor is based on the refraction in an optical waveguide plate (Ohka 2007) as shown in Figure 7.8. When a light beam is directed into the plate, it remains within it unless the surrounding refractive index is higher than the optical waveguide. This happens when the rubber sheet, featuring an array of conical feelers, is pressed. The canonical feelers are deformed on the plate surface, which reflects the light totally. The distribution of contact pressure is calculated from the bright areas viewed from the plate’s reverse surface. Another technique based on fiber Bragg grating (FBG) is given in Heo et al. (2006). Its working principle is based on the shifts in the reflected Bragg signal’s wavelength as a function of external parameters such as strain or force.

• – Ultrasonic sensors: this involves a medium for ultrasonic signals composed of thin rubber. An ultrasonic transmitter and receiver are placed underneath this rubber covering. When an object, such as a finger, presses into the rubber, it is compressed and the path is reduced. By estimating this path and comparing it with the path of other regions, the difference can be found from the pressed area. The amount of path length depends on the magnitude of the applied force (Chuang et al. 2018).
• – Magnetism-based sensors: these sensors are based on either the Hall effect or the inductance change of a coil. In Torres-Jara et al. (2006), the Hall effect is used to measure the deformation of a silicon dome when a force is applied to it. At the dome base, magnetic sensors are used to measure the deformation, which is a function of the applied force, as shown in Figure 7.9.

  

• – Microelectromechanical system (MEMS) sensors are realized by silicon micromachining, which integrates the micromechanical components and electronics on the same substrate. These devices are quite sensitive and result in higher spatial resolution. In Kane et al. (2000), the authors suggest a structure consisting of 4,096 elements arranged in an array of sensors on a silicon substrate. Each element is composed of a central shuttle plate suspended by four bridges over an etched pit. Each of the four bridges has a piezoresistor embedded in it. The structure’s suspension over the substrate allows a deformation to occur in the bridges when normal and shear loads are applied to the central plate. The piezoresistors in each leg acts as the variable resistive half-bridge circuit. The strain can be measured by monitoring the intermediate node voltage of the half-bridge for the four bridges. Figure 7.10 shows the structure of one cell of the array.

Different types of tactile actuators are available based on some of the principles shown in the sensors above, such as piezoelectric actuators. In this case, the piezoelectric actuator converts the electrical energy into mechanical energy. Magnetic, electroactive polymer actuators and shape memory alloy actuators are other tactile actuators, as shown in Amato et al. (2013) and Lucia et al. (2019).

7.5.1. Application layer protocols

The protocols in the application layer have the objective of customizing haptic communication protocols. This makes the protocol more agile. For example, a multi-modal or other additional information encapsulation is possible at this layer.

• – Application layer protocol for haptic networking (ALPHAN) (Osman et al. 2008): this protocol operates on top of the UDP since it does not impose any reliability or flow control schemes. The protocol supports the notion of key updates widely endorsed by most haptic communication transport layer protocols. This is done in the application layer by implementing a reliability mechanism applied to such updates, while regular updates remain unaffected. Figure 7.14 shows the header format of the protocol, with the frame field description in Table 7.3

Multiple buffering schemes are adapted in ALPHAN, where each haptic variable representing the environment is allocated a sending buffer. Two queues are arranged for each variable in case of retransmission to improve the application layer’s reliability since the UDP is not providing it. Another advantage of buffering is in prioritizing variables to be sent according to their importance.

The I packets are sent reliably and are considered to be key packets. P packets are encoded differentially based on previous I or P packets, while B packets are encoded based on previous and future I and/or P packets.

• – The synchronous collaboration transport protocol (SCTP) (Shirmohammadi and Georganas 2001): this is a host-to-host layer protocol, comparable to the transport layer. It is encapsulated into UDP packets and assumes that the underlying physical network supports IP multicasting. It was initially designed for interaction with a shared collaborative virtual environment. Figure 7.15 shows the packet format.

The protocol adopts the concept of reliable packet delivery of key updates and unreliable delivery of regular updates. The object ID indicates which object is being interacted in the header; the key update specifies if the packet is a key update or not. The stream ID and sequence number can be used together to discard late arriving messages. The protocol is inherently time-sensitive since updates are immediately taken into consideration.

Another version of this protocol called the smoothed SCTP has been devised by Dodeller (2004) to overcome the receiver’s jitter problem. A timestamp is added in the packet format. The time is divided into buckets of time of δt milliseconds at the receiver side. Each received update is put into a bucket according to its timestamp. A constant delay is added to the update; thus, all the update messages are processed with a fixed delay higher than the network delay, but constant.

• – Perception-based adaptive haptic communication protocol (PAHCP) (Nasir and Khalil 2012). In this protocol, the author devised a modified version of the smoothed SCTP and took into consideration extra features from other protocols such as reliable transmission protocol (SRTP), reliable multicast transport protocol (RMTP), synchronous collaboration transport protocol (SCTP) and selective reliable multicast (SRM). With a fixed threshold variable, Weber’s law is used to reduce the number of packets transmitted.

  

• – ADMUX, Adaptive Multiplexer (Eid et al. 2011) is an adaptive application layer multiplexing framework including a communication protocol. It is designed for multimedia applications incorporating haptic, visual, auditory and scent data for non-dedicated networks. It uses statistical multiplexing to multiplex the different media. The statistical multiplexing principle is when a group of media channels shares a limited quantity of bandwidth. The allocation of bandwidth is on a frame-by-frame basis, which is controlled by the multiplexer. The multiplexer is based on a mathematical model that dynamically adapts to both the application needs and the network conditions (Eid et al. 2009). The haptic applications meta language (HAML) is used to initiate and functionally modify all the components of the communication framework as shown in Figure 7.16 (Eid et al. 2009). The components include:
  • - the transport protocol, which defines the transport layer protocol, its reliability mechanism and its QoS parameters;
  • - the synchronization scheme, which defines both the intermodal and intramodal synchronization models used in the communication;
  • - the compression scheme, which defines the data preprocessing method (if used) and the codec configurations;
  • - the control method scheme to describe the algorithms that are used at both ends of the network to compensate for the network deficiencies.

An advantage of the ADMUX is synchronizing haptic data with audio/video data and that it adapts to application level events and interactions.

• – RTP/I real-time application level protocol for distributed interactive media (Mauve et al. 2001): initially, this was developed to implement distributive interactive media such as remote collaborative whiteboards and 3D design applications. The protocol reuses some of the RTP properties, such as using two protocols, one for data and the other for control, called RTP/I and RTCP/I, respectively, both carried over separate transport addresses. The protocol relies on the exchange of state information modified by either events or the passage of time. The technical details of this protocol are given in the document (Mauve et al. 2000) and the web resource¹.
• – Haptic over Internet protocol (HoIP) (Gokhale et al. 2013, 2015). In this protocol, adaptive sampling (from the highest haptic rate 1,000 packets/s to a minimum of 10 packets/s) of the haptic variables is implemented based on Weber’s law, where successive samples are compared to decide about the transmission or not of the new sample. HoIP uses the existing transport layer UDP for a better real-time performance and IP implementations. The frame of the HoIP protocol is shown in Figure 7.17. The fields of the frame are described in Table 7.4.

7.5.2. Transport layer protocols

• – Interactive real-time protocol (IRTP): this is a transport layer protocol based on the IP; it is different from TCP and UDP but inherits their advantages (Ping et al. 2005). It is designed for interactive Internet-based service with a reduced end-to-end delay. It is a connection-oriented protocol, i.e. a connection must be established before transmitting data. This decreases its efficiency compared with connection-less protocols since in each connection some inquiry-reply happens before establishing a connection. But it has the advantage of managing the users and implementing reliable transmission since it is necessary. This protocol’s short header is shown in Figure 7.18 with the fields’ description given in Table 7.5.

The protocol interacts with the upper-layer protocol, in order to transmit both reliable and unreliable data. It distinguishes reliable transmission and unreliable transmission by the command segment in the header.

• – Real-time protocol (RTP) (Schulzrinne et al. 1996) This provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. It was primarily designed to satisfy the needs for multiple-participant multimedia conferences. RTP is carried on top of IP and UDP, where an RTP header precedes a chunk of data; both are in turn contained in a UDP packet. For example, in audio signal transmission, the RTP header indicates the type of audio encoding (PCM, ADPCM or LPC). The RTP header contains timing information to reconstruct the timing produced by the source.

The RTP is used in conjunction with the real-time control protocol (RTCP) and is also needed to establish a multiuser conference, where each instance of the audio, video or any other application data periodically multicasts a reception report plus the name of its user on the RTCP (control) port. The reception report indicates how well the current data is being received and may be used to control adaptive encodings. Other identifying information may also be included in this protocol subject to control bandwidth limits, such as the inter-arrival jitter estimate, information about the highest sequence number received, a fraction of packets lost and a cumulative number of packets lost.

• – The stream control transmission protocol (SCTP) is a general-purpose transport layer protocol. The services it provides are similar to TCP and a set of advanced features (Stewart 2007) to use the enhanced capabilities of modern IP networks and support increased application requirements.
• – The efficient transport protocol (ETP) (Wirz et al. 2008) is a transport layer protocol that optimizes the available bandwidth by sending the highest packets with a reduced round trip time (RTT) and the time between two packets’ inter-packet gap (IPG). It is designed for haptic applications to integrate the good characteristics of many other transport protocols. The protocol’s performance is based on reducing the IPG to reduce the RTT, which represents the frequency of closing the control loop of the remote haptic system RTT = IGP.N, where N is the sum of the number of sent and received packets through the network, ETP uses the UDP protocol for transporting its data.
• – Real-time network protocol (RTNP) (Uchimura and Yakoh 2004): this is specially designed for bilateral teleoperation using master/slave manipulators and force feedback. It is specially designed for the UNIX environment. Its priority determines each packet order in a queue at the sender and the intermediate buffers over the network.
• – Bidirectional transport protocol (BTP) provides a novel congestion control technique that enhances application and transport layer performance (Wirz et al. 2009). The protocol improves bilateral flow tasks for real-time telerobotics by minimizing the round trip time (RTT) while maximizing the transmission frequency. To do this, the network congestion is controlled by modifying the elapsed time between two consecutive packets sent: Inter-Packet Gap (IPG) . IPG is related to the interarrival time (IAT), which is the time elapsed between two consecutive arriving packets. Both are related and equal in the case of no congestion. The BTP packet information is shown in Figure 7.19, where it is shown that it uses the UDP protocol with an additional header of the BTP.