15.3.1 Smart Homes

NTS is one of the testing service providers that provides validation of home area network (HAN) devices such as smart meters, smart door locks, light controls, thermostats, and smoke sensors. It tests the interoperability of appliances and reflects the energy consumption by various devices and thus helps in an effective energy management 3, 4. The testing tool also supports clients with self‐testing by simulating the functionality of the appliances. NTS has been designated by ZigBee Alliance to test wireless products for smart energy, and ZigBee Smart Energy is nominated by the US Department of Energy and the National Institute of Standards and Technology (NIST) as an initial interoperable standard for HAN. NTS also works for iControl Platforms to test its security and home automation commodities such as smart door locks, light controls, thermostats, and smoke sensors.

Corporate major players in the field of mobile phones manufacturing (such as Apple and MI) also provide smart home applications that help in the attainment of security, effective energy management, and automated detection of smoke or gas through mobile phone applications. Security of smart homes is ensured by setting up security sensor systems for doors and windows. Smoke or gas detectors can be turned on through mobile applications. In a similar way, smart light schedule and brightness can also be remotely controlled. Allion Smart Home provides testing and validation services, which support clients in the development, testing, and debugging of products for the three most important smart home environments – named as Cloud Service/Data Exchange,” “UI/APP,” and “End User Device” [5]. The lab established at Allion simulates a real home environment with three bedrooms, two living rooms, and two bathrooms including home items such as a sofa, TV cabinets, beds, desks, wardrobes, and so forth. Common appliances and electronics, such as a television, wireless speakers, computers (desktop and laptop), wireless LED lights, and so on have been installed in compartments with powerline wireless extenders, one‐in‐three wireless phones, and a microwave in the kitchen to introduce interference from other electrical products in the 2.4 GHz band. This is done to simulate behavior patterns and user habits in the real world [5].

eInfochips carries out performance testing for iOS and Android apps and redesigned the UI for Android and iOS platforms to improve the performance of home devices and to avoid inconsistency between iOS and Android platforms. Application response time is measured by using 24 × 7 performance evaluation tools and carries out bottleneck analysis to identify performance inefficiencies with the help of data flows and log files. Performance optimization techniques are implemented using cost–benefit analysis. Crash issues are resolved by utilizing detailed analysis and creating a crash log review. Code analysis is done with the help of SonarQube and XClarify tools [6].

UL has established the UL living lab in a 2500‐square‐foot fully furnished home situated near Silicon Valley campus and it thus enables testing of smart home devices in real‐world user scenarios and provides various benefits such as ecosystem integration, large‐scale interoperability, RF performance, and audio quality [7].

TUV is the third‐party testing provider that tests smart home products to ensure privacy of the data as per the guidelines of data protection regulations. Various types of tests such as device default settings, local communication testing for encrypted data, interoperability testing etc. are carried out to test the effectiveness of privacy of user data. Smart home devices are tested to certify their functionality and mechanical and electrical safety by testing the products such as motion sensors and smoke alarms. In addition, usability tests are also carried out for the smart home devices [8].

Smart Home Test platform established at VDE Institute conducts tests to evaluate and certify smart home network devices for compliance, faultless functionality, user data protection and interoperability [9].

The National Renewable Energy Laboratory (NREL) has devised a smart home test bed to simulate power distribution grid for industry, manufacturers, universities, and other government organizations. The NREL test bed includes the combination of powered hardware and software simulations. The smart home hardware comprises electric vehicle supply equipment (EVSE), home loads, a water heater, a thermostat, and an air conditioner, all powered (via red lines) by a photovoltaic inverter and an alternating current (AC) power amplifier, which emulates grid power. A high‐performance computer (HPC), Peregrine, has been utilized to execute advanced home energy management system (HEMS) optimization algorithms that simulates power distribution feeder, also uses weather and price data to determine control signals sent to simulated homes and to the smart home's hardware via the HEMS. The key component of a smart home test bed is a co‐simulation tool, integrated energy system model (IESM) that is responsible for managing the power system and home simulations, the HEMS algorithms, communications with the HEMS hardware, and a simulation of the smart home (using EnergyPlus) that runs on the hardware‐in‐the‐loop (HIL) control computer in the laboratory. The IESM also provides price signals as inputs to the HEMS, allowing users to evaluate how smart home technologies respond to different retail price structures [10]. Zipperer et al. [11] have also worked in this direction and developed a mechanism for electric energy management in the smart home. Cordopatri et al. [12] established test lab in the campus of the University of Calabria to experiment with various management systems for smart homes such as energy flow and comfort management systems. The main objective of the energy and comfort management system (ECMS) developed at the University of Calabria is to attain reduction in the cost and usage of energy along with improved comfort and safety of the smart home systems. Several authors have proposed similar kind of frameworks based on fuzzy logic, neural networks, and genetic algorithms [13–16]. Hu et al. [17] developed an open and smart home test bed named as SHEMS that can be used for educational purposes. The summary of these products is depicted in Table 15.1.

15.3.2 Smart Health

The main objective of the healthcare industry is to provide patients with quality healing services round the clock in a cost‐effective manner. The software industry enables the smooth functioning of the healthcare industry by providing software applications that assist in the functioning of various hospital operations and at the same also maintains the privacy of patients. Hence, crashing of an application would severely impact healthcare process and may also adversely affect the health of the patient. Therefore, testing of healthcare software is very essential as it ensures the quality and productivity of a healthcare service. The healthcare industry needs to follow strict regulatory and compliance norms, and it is bound to identify novel revenue generation strategies and to effectively utilize R&D budgets. This raises the need of software professionals to have thorough understanding of domain and industry regulations and standards. Significant work done in this direction is explained below and is portrayed in Table 15.2.

Virtusa has established dedicated center of excellence that provides healthcare domain testing, user acceptance testing (UAT) optimization, ICD‐10 testing, and enterprise end‐to‐end testing [18]. Mindfiresolutions provides a manual as well as automated healthcare application testing services by using various tools such as QTP, Selenium, Appium, and Robotium over several platforms. The testing services offered are: conformance testing, interoperability testing, functional testing, security testing, platform testing, load and performance testing, system integration and interface testing, and enterprise workflow testing [19]. The healthcare testing services provided by QAInfotech include functional testing, database testing, performance testing, content QA testing and development and implementation of QA and test strategies. In addition, testing professionals also take care of HIPAA guidelines and carries out performance and security tests [20]. Cloud lab established by ALTEN Calsoft Labs' provides healthcare domain testing in the area of clinical systems, nonclinical systems, and specialized testing services. Clinical systems include EHR/EMR, hospital ERP, radiology information systems, imaging systems, and compliance‐related standards and guidelines such as HIPAA. Nonclinical system contains the modules of pharmacy, billing, and revenue cycle management. Specialized testing services comprise compatibility and localization, security testing, performance testing, legacy modernization and testing, mobile healthcare, BI/analytics, and cloud migration and testing [21]. Precise Testing Solution delivers healthcare application testing in the domain of electronic medical records, patient survey solutions, quality and compliance solutions, enterprise content management, medical equipment software solution and compliance testing services [22].

ZenQ helps healthcare organizations in attaining quality, efficiency, and cost‐effectiveness by providing specialized healthcare testing solutions in the area of electronic health records (EHRs) electronic medical records (EMRs), hospital management systems, healthcare data interoperability and messaging standards conformation, and mobile health. Testing services include functional/regression testing, usability testing, interoperability testing, mobile apps testing, conformance/certification testing, performance testing and security testing [23]. Testree offers a complete package of quality assurance and healthcare application testing that includes certification for automatic compliance of various standards, appropriate administration, and control of policy claims and benefits, patient and disease management, billing and reporting, etc. [24]. The healthcare testing services offered by KiwiQA encompass compliance conformance testing, product consistency testing, platform testing, and security testing [25].

XBOSoft makes the provision of testing services in the domain of healthcare and ensures the compliant working of electronic health records (EHR), automated drug dispensing machines, pharmacy management, EMAR, and EPCS with mobile apps. This is done by careful design of test cases that ensures test coverage, cross‐platform, multidevice, and multibrowser compatibility [26]. The lab setup at Infoicon Technologies Pvt. Ltd. dedicatedly provides the cost‐effective healthcare testing services covering the domain of pharmaceutical industry, clinical systems, healthcare startups, body fitness, dental care, physiotherapy, doctor consultation, and homeopathy. It provides multiple platforms for manual as well as automated testing services that include interoperability testing, functional testing, security testing, load and performance testing, system integration testing, and acceptance testing [27]. W3Softech offers agile‐based healthcare and pharmaceutical testing services [28].

In the similar way, Prova also provides cost‐effective software testing and QA services for the healthcare industry [29]. Calpion's offers convenient and fast‐testing framework that works for both web and mobile healthcare application by utilizing HP quality center (QC), quick test professional (QTP) and HP ALM [30]. Abstracta provides healthcare testing system for patient portals, medical imaging, and electronic health records (EHR) while adhering to the standards and regulations. It provides automated functional testing, security testing, and performance testing services [31]. The 360logica labs offers cost‐effective, reliable, and standard compliant healthcare software testing services. The testing services are in the area of hospitals, pharmaceutical and clinical labs, which include healthcare billing software testing, R&D software testing, and embedded application testing [32].

Löffler et al. [35] devised a model‐based test‐case generation strategy from use case scenarios described with their newly introduced formal specification language by extending UML2.0 sequence diagrams. Test models have been derived from specifications, which are then used to generate test cases corresponding to each and every flow in the test model. J.M.C. Bastien et al. [36] carried out user‐based evaluation for healthcare applications to assess the usability of the application by employing single user and paired‐user testing. In this approach, users are asked to carry out certain tasks, and performance of the users is noted such as task completion rate, types of error accorded, etc. to recognize certain design flaws that causes user errors. Based on these observations, design changes can be suggested to front‐end designers. Snelick [33] investigated conformance testing and the tools that are used to perform HL7 (Health Level Seven) v2‐based conformance testing for certification of EHR technologies. Scott et al. [34] demonstrated the development of conformance methods based on the professional standards. Table 15.2 summarizes work done in smart health.

15.3.3 Smart Transport

The researchers at UMTRI carry out development and testing of intelligent transportation systems off‐the‐road to prevent collisions in passenger vehicles. Exhaustive study is carried out in the direction of automotive collision avoidance, in‐vehicle driver‐assistance and safety systems, and integrated technologies between the vehicle and the infrastructure [38]. Connected Vehicle Test Bed has been established in Michigan, Virginia, Florida, California, New York, and Arizona to facilitate a real environment where intersections, roadways, and vehicles are able to communicate through wireless connectivity by the US Department of Transportation (USDOT), and it comprises of a network of 50 roadside equipment (RSE) units installed along various segments of live interstate roadways, arterials, and signalized and unsignalized intersections, in Novi, Michigan. These RSEs communicate messages over 5.9 Ghz dedicated short‐range communication (DSRC). This test bed provisions testing of new hardware and software for the evolution in connected vehicle technology. Various types of tests (such as signal phase and timing (SPaT) communications; security system operations; and other connected vehicle applications, concepts, and equipment) can be successfully carried out for free. In addition, there is a provision of experts to carry out complex scenario tests. Also, there is no need to make any testing arrangements because of prior contracts between the local agencies and roadway operators. Test beds frequently undergo upgrades and enrichments to provision the changing requirements of users. Clients of Connected Vehicle Test Bed include Denso, Delphi, Hirschmann, Eaton, Argenia, Wayne State University, MET Labs, Ricardo, and University of North Texas [39].

The test lab instituted at IBS provides end‐to‐end software testing services to travel, transportation and logistics enterprises. It provides four types of testing services which includes Enterprise QA Automation Services, Product Acceptance Test Services, Managed Testing Services and NFR Testing services. Enterprise QA Automation Services provides automation to support DevOps environment, process automation to validate build to release quality, reusable frameworks for TTL customers and transformational models to support guaranteed outcome, Product Acceptance Test Services involves system Integration, final acceptance and UAT support, domain experts to validate business requirements, reusable assets for TDM (Test Data Management), automation, and performance and multivendor management for airlines' IT solutions testing. Managed Testing Services comprises consulting services for outsourcing, transition from incumbent vendors/captive organization, end‐to‐end testing from functional to acceptance test, and assured output/outcome model for delivery. NFR Testing services consist of performance benchmarking and capacity planning, SMAC, usability, security, performance covered, projects supported with dedicated lab facility and compliance, industry standards and frameworks in mobility and multitenancy/cloud [40].

ETSI worked in collaboration with Telecom Italia, ERTICO, the regional government, local highway authorities and port authority to launch the ITS test bed in Livorno. The test bed contains traffic lights, IoT sensors, cameras, variable message signs, and connectivity with a highway control center. RSUs and on‐board units within vehicles can be tested effectively by deploying it in the road sideways. Other ITS testing activities such as traffic sign violation, road hazards, intersections and collision warnings, and loading zones can also be carried out successfully [41].

Woo et al. [42] have designed a test bed to handle testing on various ITS and advanced driver assistance system (ADAS) technologies, such as adaptive cruise control (ACC), lane departure warning system (LDWS), cooperative intersection warning system, as well as rollover stability control (RSC) and electronic stability control (ESC). The test bed has been devised to meet the requirements of ISO/TC204 standards. The test bed for ITS encompass three tracks named as ITS high‐speed track, Cooperative vehicle‐infra test intersections, and Special test track. The main purpose of ITS high‐speed track is to test performance of ACC, LDWS, LKAS, etc. It has three lanes of high‐speed track of length equal to 1,360 m with maximum allowable speed of 204 km/h. The total length of Cooperative vehicle‐infra test intersections is 1,200 m and there are three intersections. The main objective is to test pedestrian protection and intersection safety. Special test track comprises of four lanes of test road with the total area of 490 × 35 m. It includes Belgian road, washboard road, cobblestone road, water splash shower tunnel, for example. Durability and reliability test are carried out it these tracks.

The government of Estonia plans to restructure its public transportation system by adopting autonomous vehicles and thus legalized testing of autonomous vehicles on national and local roads of the country. Rigorous efforts are put into developing a cyber‐risk management framework for autonomous vehicles in regular road and traffic conditions. The government has planned to create a fleet management system and integration of vehicles into the public transport system and the implementation of call‐to‐order bus stops [44].

Transit Windsor provides the development as well as testing services for intelligent transportation systems. The company has produced 10 buses furnished with an efficient, safer, and more user‐friendly system. It provides onboard voice and visual announcements on the display boards for the upcoming bus stop messages. It also stipulates real‐time Transit Windsor bus arrival information as well as route for the bus progress via the Internet [45].

Siphen has achieved an intelligent transportation system (ITS) product compliance with UBS II and ARAI testing. It is known for its rigorous testing procedure. It is working with the government of India to equip the country with ITS by providing 24 × 7 bus operation service with the features such as automatic vehicle location, vehicle health monitoring, and diagnostics. In addition, it is carrying out end‐to‐end testing as well as a certification process according to the timelines given by government authorities [46].

Anritsu provides ITS solutions for V2X, testing, and manufacturing in a very efficient manner with reduced test time and test cycles. Testing solutions are provided with the help of four components: MD8475A Signalling Tester, MS2830A Spectrum Analyzer, MS269xA series, and V2X 802.11p Message Evaluation Software. MD8475A Signalling Tester is similar in that it supports cellular as well as M2M standards. The services supported are eCall, IMS, VoLTE, WLAN off‐load tests, and call‐processing tests for vehicles. The testing tasks are easy, fast, and reliable due to GUI‐based SmartStudio software and supplied test sequences for automatic remote control of the GUI. Multimode terminals and all cellular standards, such as LTE (2×2 MIMO) and LTE‐Advanced (Carrier Aggregation) are well supported. SmartStudio GUI provisions easy setup of test environments and functional tests. It also carries out automated mobile terminal verification testing with the available test sequences. MS2830A Spectrum Analyzer is used for testing of 2G, 3G, LTE, and LTE‐Advanced signals on a vehicle‐to‐vehicle or vehicle‐to‐x test environment. To improve the product quality, capture and replay functions are compared with the real‐world effects with simulated designs and performance. The supported frequency range is 9 kHz to 26.5 GHz/43. MS269xA series units contain swept spectrum analysis, FFT signal analysis, and a precision digitizer function and are the latest high‐performance signal analyzers for next‐generation communication applications. It has One‐Box Tester with the addition of the signal generator option. Due to the support of batch capture measurements, analysis time gets faster [47].

Penta Security Systems has launched secured smart transportation with the secure data solution AutoCrypt, implemented on the connected vehicles in the three cities of South Korea. It has also established the second‐largest test bed named as K‐City to test and certify autonomous cars. Public key infrastructure and V2X security system has been implemented to ensure secure and encrypted communication between vehicle‐to‐vehicle and vehicle‐to‐infrastructure, as well as the security and encryption of roadside units [43].

Simulation‐based test bed has been developed at Georgia Institute of Technology by the School of Civil and Environmental Engineering, which can be used for fast assessment and incorporation of sensor and actuator systems in ITS. The test can also be used to study and examine various data networks architectural possibilities to support ITS applications. The test bed supports integrated parallel simulation ability and also involves interoperable simulations of transportation infrastructures, wired and wireless communication networks, and distributed computing applications. In addition, it possesses emulation ability that allows conducting live experiments with prototype hardware and software embedded into virtual transportation systems. The test bed incorporates data generated from sensors embedded in the vehicles (such as location, velocity, and acceleration etc.) functioning in the Atlanta metropolitan area. This data are also used for modeling and scenario development as well as validation of simulations [37]. The above stated work is summarized in Table 15.3.

15.4.1 Smart Homes

We have proposed the following set of criteria to evaluate the existing work in smart home test beds. The pertinence of these criteria is described in this section:

• Energy efficiency testing. The testing is used to verify reduction in energy consumption in the smart homes.
• Reliability testing. It ensures the stability of the system under various specific tests, which include stress testing, network testing, along with functional testing.
• Functionality testing. It is required for verification of each function of the software application in conformance with the requirement specification. It encompasses all the scenarios related to failure paths and boundary cases.
• Interoperability testing. Interoperability determines how devices communicate with each other and, upon receiving information, how processing is done and corresponding actions are generated. If a device can't receive information, process it, and act upon that information, it won't function as consumers hope. Without full functionality, the product may not provide value. Real‐world test labs are the best way to solve interoperability issues, as they depict actual scenarios of the problem.
• Performance testing. It is required to ensure software applications will perform well under their expected workload. It determines responsiveness and stability of a system under various workloads and measures the quality attributes of the system, such as scalability, reliability, and resource usage.
• Usability testing. Usability testing measures the convenient level for learning of the system by end users, which include parameters such as level of skill required to understand the system, time requirement to attain familiarity, and user's productivity.
• Security testing. Security testing is a testing technique to determine whether the application or the product is secured. It aims at verifying basic principles such as confidentiality, integrity, authentication, authorization, availability and nonrepudiation.

Based on the evaluation criteria already described, limitations and research directions as well as suggestions for the future work have been elucidated here and portrayed in Table 15.4. The first hindrance in acceptance of the smart home technology is that smart homes are vulnerable to hacking. Hence, a test bed should be established that takes into account cyber‐security measures to protect the smart home. The second hindrance is the high cost; measures should be taken to develop a technology that can be made available to users at a lower cost. Combinatorial testing strategy can be used to ensure the low price suggested by the pricing model that supports the pooling of distributed, dispersed resources in fog computing and the IoT. The third hindrance is the learning curve for non‐tech‐savvy people with smart home. Hence, usability testing should also be given topmost priority. Another most important factor that hinders the acceptance is lack of industry standardization, as use of proprietary technology can cause problems smart home users. Hence, conformance testing should also be given priority. Dependency on Internet connection should also be tackled, and reliability testing methodology specifically designed to suit the environment of the smart system need to be addressed.

15.4.2 Smart Health

To assess the existing smart health test beds, the following set of criteria have been suggested:

• Conformance testing. Conformance testing is performed to ensure adherence to the standards such as Sarbanes‐Oxley, HIPAA, FDA etc.
• Platform testing. It ensures applications executes well across different platforms, which includes operating systems, different browsers, and multiple devices.
• Interoperability testing. Interoperability testing assesses whether connected devices and EHR systems communicate with one another effectively and correctly. It also ensures seamless operations between HL7 and DICOM transactions.
• Functionality testing. This is required for verification of each function of the software application in conformance with the requirement specification. It encompasses all the scenarios related to failure paths and boundary cases.
• Enterprise workflow testing. It checks whether the expected activities are executed and workflow data properties have correct values.
• Performance testing. It is required to ensure software applications will perform well under their expected workload. It determines responsiveness and stability of a system under various workloads and measures the quality attributes of the system, such as scalability, reliability, and resource usage.
• Usability testing. Usability testing measures the convenient level for learning of the system by end users, which include parameters such as level of skill required to understand the system, time requirement to attain familiarity, and user's productivity.
• Security testing. Security testing is a testing technique to determine whether the application or the product is secured. It aims at verifying basic principles such as confidentiality, integrity, authentication, authorization, availability and nonrepudiation.
• Mobile app testing. Mobile application testing is a procedure by which application software developed for handheld mobile devices is tested for its functionality, usability, and consistency.

Evaluation criteria described above helped in deducing the limitations and research directions in the existing smart health testing solutions. The same have been illustrated in this section and depicted in Table 15.5, along with research suggestions for the future work.

It has been implied that there is a lack of effective methodology that provides a systematic way to manage the data collected from various wearable devices. To combat this challenge, big‐data, machine learning, and AI can be used. To ensure the attainment of the mentioned functionality, a test bed is required that executes blockchain‐based repeatable tests with massive data received from wearable devices such as smart watches, eyeglass displays, and electroluminescent clothing, for example.

Further, it has been found that despite the various benefits of smart healthcare, it is not well adopted and market growth is restrained. It may be due to the high cost of IoT infrastructure and data privacy and security apprehensions. This can be resolved by building confidence among various stakeholders, which can be brought into practice by executing security tests specifically designed to examine the cybersecurity measures taken up to address the above‐mentioned issue.

Another challenge is the management of connected devices and a lack of interoperability with EHR systems. This can be ensured by executing context‐aware testing techniques. Thus, context‐aware test case generation methodologies need to be worked out for smart health systems. To address the limitations associated with smart glasses (i.e. short battery life and inability to understand the medical terms of doctors by voice‐control system), context‐aware test data generation must be applied to ensure that the system will work. Blockchain technology should be reconnoitered to solve the problems of large‐scale data sharing, ensuring data privacy and security and transparency between patient and doctors and between various healthcare providers. In this case, blockchain‐based repeatable regression tests can be employed for assurance of the privacy and security of data shared between doctors and patients.

Genomics is a field that deals with genes editing and genomic sequencing in which robotics plays a major role. Such test beds that ensure the proper functioning of genomics would help patients to recover from diseases like central nervous system and infectious diseases. Thus, for this purpose an efficient testing strategy must be identified. Prospects of the utilization of virtual reality for rehabilitation in orthopedics need more exploration. Context‐aware test case design would strengthen confidence in the system.

Augmented reality should also be surveyed broadly so that it can be used effectively for gathering 3D data sets of a patient in real time using sensors like magnetic resonance imaging (MRI), ultrasound imaging, or CT scans. It should also be investigated for its use as a visualization tool during surgeries. Appropriate testing mechanisms need to be identified that work well in this direction. In addition, exploration of 5G applications for its use in the smart devices (such as wearable sensors) to monitor the health condition of patients is the need of the hour. To ensure the attainment of desired functionality, a comprehensive and customized testing strategy need to be devised. Also, a transparent pricing model is required to be implemented that ensures cost reduction of the associated IoT infrastructure by promoting the pooling of distributed, dispersed resources in fog computing and the Internet of Things. This also demands the establishment of test beds that make use of customized testing strategies to ensure the attainment of desired functionality of the ubiquitous system (such as smart home, smart health, and smart transport). Such test beds should be freely available to the research community to carry out extensive studies in this area.

15.4.3 Smart Transport

The existing work toward the implementation of test beds for smart transport system has been evaluated as per the following set of verification criteria and the associated research directions along with limitation are provided in Table 15.6:

• Privacy testing. There is the need to ensure the privacy and security of transport devices, encryption of the data communicated between vehicles, and the roadside infrastructures for privacy and security. This can be accomplished by the establishment of dedicated transportation cybersecurity test labs for intrusion detection/prevention systems, sensor spoofing/manipulation, secure controller area network, secure software updates, resiliency and recovery, sensor spoofing/manipulation, etc.
• Energy efficiency testing. ITS systems need to be tested for fuel consumption. The lesser the fuel consumed by the transport vehicle to travel per unit distance, the more will be the efficiency and lower will be cost. This can be achieved by avoiding vehicles standing idle in traffic jams or circling around looking for parking spaces. The better alternative would be to design vehicles based on sustainable resources such as electric and solar vehicles. Such engines must be tested comprehensively before making them fully functional in a real‐world environment.
• Collision avoidance testing. Collision avoidance algorithms employed to prevent road accidents need to be tested to validate the ITS effectiveness.
• Autonomous vehicle testing. The autonomous vehicle operates without hands on the wheel, and the safety of such vehicle is of utmost importance, as failure causes life hazards and hence must be verified comprehensively.
• Traffic congestion management. Traffic congestion is one of the biggest challenges faced by commuters, as it leads to the wastage of time, fuel, and money. Unnecessary burning of fuel also increases the level of carbon emissions, which causes air pollution. This issue can be addressed by installing fog nodes in vehicles and roadside to send and receive information related to traffic jams and accidents. The generated information can thus be used to trigger certain actions such as activation of automatic brakes or issue of warning messages to slow down speed or to avoid specific lanes and intersection points. The test bed should be equipped to monitor and evaluate this mechanism.
• Connected vehicle technology. Connected vehicle technology utilizes wireless communication to transfer information regarding road accidents, jams etc. by one vehicle to another vehicle and to the road‐side infrastructures. This helps in preventing road accidents and avoids unnecessarily getting stuck in traffic jams. The test bed should incorporate such facilities where new hardware and software could be tested before putting vehicles into real operating conditions.
• Compliance with standards. Test beds should comply with the standards of transport so that they provide the real picture of the working system before actual launch on the road and streets among public.
• Reliability testing. Transport system should be robust and resilient in case of any device failure or lost Internet connectivity. There should be strong mechanism to verify the reliability of the transportation system.
• Performance and usability testing. The testing must ensure that the devices perform well and the mobile apps are user‐friendly.
• Pollution control testing. Carbon emissions should be reduced and proper checks need to be maintained to control air pollution. Roadside monitoring units installed to monitor and measure emission gases (such as carbon dioxide (CO₂) or nitrogen oxide (NO)) are required to be testing for its effectiveness. It can be controlled by using sustainable vehicles such electric cars.
• Interoperability testing. Poor interoperability is one of the largest barriers to smart transport implementation. Interoperability determines how devices communicate with each other and, upon receiving information, how processing is done and corresponding actions are generated. If a device can't receive information, process it, and act upon that information, it won't function as consumers hope. Without full functionality, the product may not provide value. Real‐world test labs are the best way to solve interoperability issues, as they depict actual scenarios of the problem.

Although many corporate provides the testing solution for smart transport, inadequate work has been done in academia, and this requires special attention of researchers. In addition, several works discusses the importance of cyber‐physical systems in transportation, but no work has been found describing the novel testing methodology that verifies the security of smart transport vehicle. Similarly, numerous works have been found that discuss the importance of connected vehicle technology in reducing air pollution and improve efficiency but no test bed has been found that quantitatively measures the percentage level of air pollution reduction and up to what percentage travel experience gets enriched. Further, no case study has been discussed that empirically proves the benefits of the technology. Also, very few test beds have been developed that actively carry out testing of autonomous vehicles, and no work has been found that tests user friendliness of transportation systems. Test‐bed executing of the repeatable regression tests based on blockchain technology must be studied to address quality assurance issues of the smart transportation system.

Pollution‐monitoring devices are also required to be verified for effectiveness. No test bed has been suggested that works in this direction. Reliability is one of the most crucial feature that should be possessed by transport devices and related infrastructure; hence, there should be appropriate methodology that verifies the cybersecurity measures to ensure the resilience and robustness of the system. Only one research work has been found that works in this direction. Novel testing methodologies should be proposed for the comprehensive testing of collision avoidance algorithms, and the test bed should be designed to be portable so that it can be freely used by the research community.