2.1 Sources and Target Data Warehouse

Sources in a data warehousing system store data belonging to one or more organizations for daily transactions or business purposes. The target data warehouse, on the other hand, stores large volumes of data for long-term analysis and mining purposes. Sources and target data warehouses can be designed and implemented using a variety of technologies including data models and data management systems.

A data model describes business terms and their relationships, often in a pictorial manner [12]. The following data models are typically used to design the source and target schemas:

• • Relational data model: Such a model organizes data as collections of two-dimensional tables [5] with all the data represented in terms of tuples. The tables are relations of rows and columns, with a unique key for each row. Entity relationship (ER) diagrams [13] are generally used to design the relational data models.
• • Nonrelational data model: Such a model organizes data without a structured mechanism to link data of different buckets (segments) [6]. These models use means other than the tables used in relational models. Instead, different data structures are used, such as graphs or documents. These models are typically used to organize extremely large datasets used for data mining because unlike the relational models, the nonrelational models do not have complex dependencies between their buckets.
• • Dimensional data model: Such a model uses structures optimized for end-user queries and data warehousing tools. These structures include fact tables that keep measurements of a business process, and dimension tables that contain descriptive attributes [14]. Unlike relational models that minimize data redundancies and improve transaction processing, the dimensional model is intended to support and optimize queries. The dimensional models are more scalable than relational models because they eliminate the complex dependencies that exist between relational tables [15].
  The dimensional model can be represented by star or snowflake schemas [16] and is often used in designing data warehouses. These types of schemas are as follows:
  • – Star: This type of schema has a fact table at the center. The table contains the keys to dimension tables. Each dimension includes a set of attributes and is represented via a one dimension table [17].
  • – Snowflake: Unlike the star schema, the snowflake schema has normalized dimensions that are split into more than one dimension tables. The star schema is a special case of the snowflake schema with a single level hierarchy.

The sources and data warehouses use various data management systems to collect and organize their data. The following is a list of data management systems generally used to implement the source and target data stores.

• • Relational database management system (RDBMS): An RDBMS is based on the relational data model that allows linking of information from different tables. A table must contain what is called a key or index, and other tables may refer to that key to create a link between their data [6]. RDBMSs typically use Structured Query Language (SQL) [18] and are appropriate to manage structured data. RDBMSs are able to handle queries and transactions that ensure efficient, correct, and robust data processing even in the presence of failures.
• • Nonrelational database management system: A nonrelational DBMS is based on a nonrelational data model. The most popular nonrelational database is Not Only SQL (NoSQL) [6], which has many forms, such as document-based, graph-based, and object-based. A nonrelational DBMS is typically used to store and manage large volumes of unstructured data.
• • Big data management system: Management systems for big data need to store and process large volumes of both structured and unstructured data. They incorporate technologies that are suited to managing nontransactional forms of data. A big data management system seamlessly incorporates relational and nonrelational database management systems.
• • Data warehouse appliance (DWA): DWA was first proposed by Hinshaw [19] as an architecture suitable for data warehousing. DWAs are designed for high-speed analysis of large volumes of data. A DWA integrates database, server, storage, and analytics into an easy-to-manage system.
• • Cloud data warehouse appliance: Cloud DWA is a data warehouse appliance that runs on a cloud computing platform. This appliance benefits from all the features provided by cloud computing, such as collecting and organizing all the data online, obtaining infinite computing resources on demand, and multiplexing workloads from different organizations [20].

Table 1 presents some of the available products used in managing the data in the sources and target data warehouses. The design and implementation of the databases in the sources are typically based on the organizational requirements, while those of the data warehouses are based on the requirements of data analyzers and researchers.

For example, the sources for a health data warehouse are databases in hospitals and clinic centers that keep patient, medication, and treatment information in several formats. Fig. 2 shows an example of possible sources in the health data warehouse. Hospital A uses a flat spreadsheet to keep records of patient data. Hospital B uses an RDBMS for its data. Hospital C also uses an RDBMS but has a different schema than Hospital B. The data from different hospitals must be converted to a common model in the data warehouse.

The target data warehouse for health data may need to conform to a standard data model designed for electronic health records such as Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) [32]. The OMOP CDM is a dimensional model that includes all the observational health data elements that are required for analysis use cases. The model supports the generation of reliable scientific evidence about disease, medications, and health outcomes.

2.2 Extract, Transform, and Load

The ETL process extracts data from sources, transforms it to a common model, and loads it to the target data warehouse. Fig. 3 shows the components involved in the ETL process, namely, extract, transform, and load.

1. 1. Extract: This component retrieves data from heterogeneous sources that have different formats and converts the source data into a single format suitable for the transformation phase. Different procedural languages such as Transact-SQL or COBOL are required to query the source data. Most extraction approaches use Java Database Connectivity (JDBC) or Open Database Connectivity (ODBC) drivers to connect to sources that are in DBMS or flat file formats [33].
   Data extraction is performed in two phases. Full extraction is performed when the entire data is extracted for the first time. Incremental extraction happens when new or modified data are retrieved from the sources. Incremental extraction employs strategies such as log-based, trigger-based, or timestamp-based techniques to detect the newly added or modified data. In the log-based technique, the DBMS log files are used to find the newly added or modified data in the source databases. Trigger-based techniques create triggers on each source table to capture changed data. A trigger automatically executes when data is created or modified through a Data Manipulation Language (DML) event. Some database management systems use timestamp columns to specify the time and date that a given row was last modified. Using these columns, the timestamp-based technique can easily identify the latest data.
2. 2. Transform: This component propagates data to an intermediate data staging area (DSA) where it is cleansed, reformatted, and integrated to suit the format of the model of a target data warehouse [3]. This component has two objectives.
   First, the transformation process cleans the data by identifying and fixing (or removing) the existing problems in the data and prepares the data for integration. The goal is to prevent the transformation of so-called dirty data [34, 35]. The data extracted from the sources is validated both syntactically and semantically to ensure that it is correct based on the source constraints. Data quality validation and data auditing approaches can be utilized in this step to detect the problems in the data. Data quality validation approaches apply quality rules to detect syntactic and semantic violations in the data. Data auditing approaches use statistical and database methods to detect anomalies and contradictions in the data [36]. In Table 2 we present some examples of data quality validation applied to data cleansing of patients in our health data warehouse.

   Table 2

   Examples of Validation Applied to Data Cleansing

   Validation	Example of a Violation
   Incorrect value check	Birth_date=70045 is not a legal date format
   Uniqueness violation check	Same SSN=‘123456789’ presented for two people
   Missing value check	Gender is null for some records
   Wrong reference check	Referenced hospital=1002 does not exist
   Value dependency violation check	Country=‘Germany’ does not match zip code=‘77’

   
   Second, it makes the data conform to the target format through the application of a set of transformation rules described in the source-to-target mapping documents provided by the data warehouse designers [33]. Table 3 presents examples of source-to-target mappings for generating a target table called Patient in our health data warehouse. The mappings include the names of the corresponding source and target tables, the source and target columns with their types, and selection conditions.

   Table 3

   Transforming Source Data to Generate Target Table Patient

   Source			Target
   Table Name	Column Name	Data Type	Table Name	Column Name	Data Type	Selection Condition
   PersonDim	PersonKey	Integer	Patient	Patient_id	Integer	Transform all the current patients
   PersonDim	Name	String	Patient	Patient_name	String	Transform all the patients
   PersonDim, AddressDim	AddressKey	Integer	Patient	Location_id	Integer	Transform all the patients with new addresses (after year 2000)
   PersonDim, Concept	Sex	String	Patient	Gender	Integer	Transform all the patients with their sex using concept values female, male, other

   

3. 3. Load: This component writes the extracted and transformed data from the staging area to the target data warehouse [1]. The loading process varies widely based on the organizational requirements. Some data warehouses may overwrite existing data with new data on a daily, weekly, or monthly basis, while other data warehouses may keep the history of data by adding new data at regular intervals. The load component is often implemented using loading jobs that fully or incrementally transform data from DSA to the data warehouse. The full load transforms the entire data from the DSA, while the incremental load updates newly added or modified data to the data warehouse based on logs, triggers, or timestamps defined in the DSA.

The ETL components, namely, extract, transform, and load, are not independent tasks, and they need to be executed in the correct sequence for any given data. However, parallelization can be achieved if different components execute on distinct blocks of data. For example, in the incremental mode the different components can be executed simultaneously; the newly added data can be extracted from the sources while the previously extracted block of data is being transformed and loaded into the target data warehouse.

2.3 Front-End Applications

Front-end applications present the data to end users who perform analysis for the purpose of reporting, discovering patterns, predicting, or making complex decisions. These applications can be any of the following tools:

• • OLAP report generators: These applications enable users and analysts to extract and access a wide variety of views of data for multidimensional analysis [37]. Unlike traditional relational reports that represent data in two-dimensional row and column format, OLAP report generators represent their aggregated data in a multidimensional structure called cube to facilitate the analysis of data from multiple perspectives [38]. OLAP supports complicated queries involving facts to be measured across different dimensions. For example, as Fig. 4 shows, an OLAP report can present a comparison of the number (fact) of cases reported for a disease (dimension) over years (dimension), in the same region (dimension).

• • Analysis and data mining: These applications discover patterns in large datasets helping users and data analysts understand data to make better decisions [4]. These tools use various algorithms and techniques, such as classification and clustering, regression, neural networks, decision trees, nearest neighbor, and evolutionary algorithms for knowledge discovery from data warehouses. For example, clinical data mining techniques [39] are aimed at discovering knowledge from health data to extract valuable information, such as the probable causes of diseases, nature of progression, and drug effects.
• • Decision support: These applications support the analysis involved in complex decision-making and problem solving processes [40] that involve sorting, ranking, or choosing from options. These tools typically use Artificial Intelligence techniques, such as knowledge base or machine learning to analyze the data. For example, a Clinical Decision Support [41] application provides alerts and reminders, clinical guidelines, patient data reports, and diagnostic support based on the clinical data.

4.1 Testing Underlying Data

In this testing activity, the data stored in the sources and the target data warehouse is validated against organizational requirements, which are provided by domain experts in the form of a set of rules and definitions for valid data. If the underlying data fails to meet the requirements, any knowledge derived from the data warehouse will be incorrect [43].

We describe existing functional and security testing approaches based on testing the underlying data in data warehouses as well as propose approaches based on our experience to achieve high quality data in a health data warehouse.

4.2 Testing the Data Model

As the data model is the foundation for any database, it is critical to get the model right because a flawed model directly affects the quality of information. Data model tests ensure that the design of the model follows its standards both conceptually and logically and meets the organizational specifications. Documentation for the source and target model help equip testers with the required information for the systematic testing of data models.

4.3 Testing Data Management Product

Using the right product for data management is critical to the success of data warehouse systems. There are many categories of products used in data warehousing, such as DBMSs, big data management systems, data warehouse appliances, and cloud data warehouses that should be tested to ensure that it is the right technology for the organization. In the following sections, we describe the existing approaches for performance, stress, and recovery testing of the data management products.

4.4 Summary

Table 7 summarizes the testing approaches that have been applied to the sources and the target data warehouse that we discussed in this section. There are no methods proposed for the security testing of underlying data in data warehouse systems (as indicated by the gray shaded row in the table).

Table 7

Testing the Sources and the Target Data Warehouse

Test Category	Component	GuardianIQ [48]	Informatica [49]	Hoberman [12]	Golfarelli and Rizzi [8]	MySQL plug-in [54]	Calero et. al, [56]	Slutz [58]	Gunawi et al. [60]	Smith and Klingman [62]

Gray shaded rows indicate that we did not find approaches or tools to support that kind of testing activity even though they are necessary for a real-world data-warehouse.

We have identified the following open problems in testing the sources and the target data warehouse.

• • In the area of functional testing of underlying data, there is no systematic way to assure the completeness of the test cases written/generated by different data quality assurance tools. We suggest that new research approaches be developed using a test adequacy criterion, such as number of fields, tables, or constraints as properties that must be exercised to constitute a thorough test.
• • Data quality rules are not formally specified in the business requirements for the functional testing of the underlying data. A tester needs to bridge the gap between informal specifications and formal quality rules.
• • It is difficult to design a generalized data quality test applicable to all data warehouse systems because data warehouse projects are typically designed for specific business domains. There are a number of generalized tools that let users define their desired data quality rules.
• • The fault finding ability of the data quality tests are not evaluated in the literature. One can use mutation analysis techniques to perform this evaluation.
• • No approach has been proposed for the security testing of underlying data. One can compare the access privileges defined for the target data with the ones defined for the corresponding source data to ensure that all the required protections are correctly observed in the target data warehouse.
• • There is a lack of automatic functional evaluation techniques for data models. The existing functional evaluation activities are manually performed through human inspections.
• • There is a lack of structural evaluation techniques for nonrelational and dimensional schema. The existing approaches focus on the relational data schema.
• • No formal technique has been proposed for the usability testing of data models. The proposed approaches are typically human inspections.
• • In the area of maintainability testing of data models, a number of design complexity metrics have been proposed to get an insight into the capability of the data model to sustain changes. However, there is no information on how to design maintainability tests based on the metrics.
• • The heterogeneous data involved in the data warehousing systems make the performance and stress testing of data management products difficult. Testers must use large datasets in order to perform performance and stress tests. Generating this voluminous data that reflect the real characteristics of the data is an open problem in these testing activities.
• • There is a lack of work in performance and stress testing of data warehouse appliance and cloud data warehouses. The proposed approaches in the literature typically focus on testing DBMSs.

5.1 Functional Testing of ETL Process

Functional testing of ETL process ensures that any changes in the source systems are captured correctly and propagated completely into the target data warehouse [3]. Two types of testing have been used for evaluating the functionality of ETL process, namely, data quality and balancing tests.

5.2 Performance, Stress, and Scalability Testing of ETL Process

Performance tests assess whether or not the entire ETL process is performed within the agreed time frames by the organizations [70]. The goal of performance testing of ETL is to assess ETL processing time under typical workloads that are comparable with the average expected data volume [42].

Stress tests also assess the ETL processing time but under a workload which is significantly larger than the expected data volume. The goal of stress testing of ETL is to assess ETL tolerance by verifying whether or not it crashes or fails when dealing with an extraordinarily large volume of data.

Scalability testing of ETL is performed to assess the process in terms of its capability to sustain further growth in data warehouse workload and organizational requirements [1, 70]. The goal of scalability testing is to ensure that the ETL process meets future needs of the organization. Mathen [1] stated that this growth mostly includes an increase in the volume of data to be processed through the ETL. As the data warehouse workload grows, the organizations expect ETL to sustain extract, transform, and load times. Mathen [1] proposed an approach to test the scalability of ETL by executing ETL loads with different volumes of data and comparing the times used to complete those loads.

In all of these testing activities, the processing time of ETL is evaluated when a specific amount of data is extracted, transformed, and loaded into the data warehouse [8]. The goal is to determine any potential weaknesses in ETL design and implementation, such as reading some files multiple times or using unnecessary intermediate files or storage [1]. The initial extract and load process, along with the incremental update process must be evaluated through these testing activities.

Wyatt et al. [71] introduced two primary ways to measure the performance of ETL, i.e., time-based and workload-based. In the time-based method, they check if the ETL process was completed in a specific time frame. In the workload-based approach they test the ETL process using a known size of data as test data, and measure the time to execute the workload. Higher performing ETL processes will transfer the same volume of data faster. The two approaches can be blended to check if the ETL process was completed in a specific time frame using a specific size of data.

The above testing approaches test the entire ETL process under different workload conditions. However, the tests should also focus on the extraction, transformation, and load components separately and validate each component under specific workloads. For example, the performance testing of the extraction component verifies whether or not a typical sized data can be extracted from the sources in an expected time frame. Applying the tests separately on the constituent ETL components and procedures helps localize existing issues in the ETL design and implementation, and determine the areas of weaknesses. The weaknesses can be addressed by using an alternative technology, language, algorithm, or intermediate files. For example, consider the performance issue in the Load component of ETL, which incorrectly uses the full mode and loads the entire data every time instead of loading only the new added or modified data. In this case, the execution time of the Load component is considerably longer than the Extract and Transform components. This problem can be localized if we apply the tests on the individual components instead of on the entire ETL process.

5.3 Reliability Testing of ETL Process

This type of testing ensures the correctness of the ETL process under both normal and failure conditions [71]. Normal conditions represent situations in which there are no external disturbances or unexpected terminations in the ETL process. To validate the reliability of the ETL process under normal conditions, we want to make sure that given the same set of inputs and initial states, two runs of ETL will produce the same results. We can compare the two result sets using properties such as completeness, consistency, and validity.

Failure conditions represent abnormal termination of ETL as a result of loss of connection to a database or network, power failure, or a user terminating the ETL process. In such cases the process should be able to either complete the task later or restore the process to its starting point. To test the reliability of the ETL process under abnormal conditions, we can simulate the failure conditions and compare the results from a failure run with the results of a successful run to check if the results are correct and complete. For example, we should check that no records were loaded twice to the target data warehouse as a result of the failure condition followed by rerunning the ETL process.

Most ETL implementations indirectly demonstrate reliability features by relying on the ACID properties of DBMSs [71]. If DBMSs are used in a data warehousing system to implement the sources or the target data warehouse components, these components recover from problems in the Extract, Transform, or Load processes. However, there are data management products other than DBMSs used in the data warehousing systems that do not support ACID properties (e.g., Google Cloud Bigtable [72] that is a NoSQL data management product). In such cases, reliability needs to be addressed separately and appropriate test cases must be designed.

Note that balancing tests may be performed to compare the two result datasets in both normal and abnormal conditions to verify the proposed properties in addition to other reliability tests.

5.4 Regression Testing of ETL Process

Regression tests check if the system still functions correctly after a modification. This testing phase is important for ETL because of its evolving nature [8]. With every new data warehouse release, the ETL process needs to evolve to enable the extraction of data from new sources for the new applications. The goal of regression testing of ETL is to ensure that the enhancements and modifications to the ETL modules do not introduce new faults [73]. If a new program is added to the ETL, interactions between the new and old programs should be tested.

Manjunath et al. [74] automated regression testing of ETL to save effort and resources with a reduction of 84% in regression test time. They used Informatica [49] to automatically generate test cases in SQL format, execute test cases, and compare the results of the source and target data. However, their approach uses the retest all [75] strategy, which reruns the entire set of test cases for regression testing of the ETL process. Instead, they could use regression test selection [75] techniques to run a subset of test cases to test only the parts of ETL that are affected by project changes. These techniques classify the set of test cases into retestable and reusable tests for regression testing purposes in order to save testing cost and time. A retestable test case tests the modified parts of the ETL process and needs to be rerun for the safety of regression testing. A reusable test case tests the unmodified parts of the ETL process and does not need to be rerun, while it is still valid [76].

Mathen [1] proposed to perform regression testing by storing test inputs and their results as expected outputs from successful runs of ETL. One can use the same test inputs to compare the regression test results with the previous results instead of generating a new set of test inputs for every regression test [1].

5.5 Usability Testing of ETL Process

The ETL process consists of various components, modules, databases, and intermediate files with different technologies, DBMSs, and languages that require many prerequisites and settings to be executed on different platforms. ETL is not a one-time process; it needs to be executed frequently or any time data is added or modified in the sources. As a result, it is important to execute the entire process with configurations that are easy to set up and modify.

Usability testing of ETL process assesses whether or not the ETL process is easy to use by the data warehouse implementer. This testing activity determines how easy it is to configure and execute ETL in a data warehouse project.

We suggest to assess the usability of the ETL process by measuring the manual effort involved in configuring ETL in terms of time. The configuration effort includes (1) providing connection information to the sources, data staging area, or target data warehouse, (2) installing prerequisite packages, (3) preprocessing of data before starting the ETL process, and (4) human interference to execute jobs that run each of the extract, transform, and load components. We can also do a survey with different users that gives us more information about the difficulty level.

5.6 Summary

Table 10 summarizes the existing approaches to test different aspects of the ETL process. As can be seen from the table, scalability, reliability, and usability testing were not reported in the literature even though they are critical for a comprehensive testing of the process. We identified the following open problems in ETL testing. We summarize areas and ideas for future investigation.

• • In the functional testing of ETL, there is not a systematic way to assure the completeness of the test cases written/generated as a set of queries. As with the functional testing of underlying data, we can use appropriate test adequacy criteria to evaluate and create a thorough test.
• • There is a lack of systematic techniques to generate mock test inputs for the functional testing of the ETL process. We can use input space partitioning techniques to generate test data for all the equivalent classes of data. Current tools generate random test data with not much similarity with the characteristics of real data.
• • The fault finding ability of the balancing tests is not evaluated in the surveyed approaches. We can use mutation analysis for this evaluation.
• • In the performance, stress, and scalability testing of ETL, the existing approaches test the entire ETL process under different workloads. We can apply tests to the individual components of ETL to determine the areas of weaknesses.
• • The heterogeneous data involved in the data warehousing systems make the performance, stress, and scalability testing of the ETL process difficult. Testers must use large heterogeneous datasets in order to perform tests.
• • The existing ETL implementations rely on ACID properties of transaction systems, and ignore the reliability testing of the ETL process. We can perform the balancing tests proposed in Section 5.1.2 to compare the results of the ETL process under normal conditions with the ones under abnormal conditions to verify the properties, namely, completeness, consistency, and syntactic validity.
• • No approach has been proposed to test the usability of the ETL process. We define this testing activity as the process of determining whether or not the ETL process is easy to use by the data warehouse implementer. One can test the usability of the ETL process by assessing the manual effort involved in configuring ETL that is measured in terms of time.

Table 10

Testing Extract, Transform, and Load (ETL)

Testing Category	GuardianIQ [48]	Informatica [49]	QuerySurge [65]	Wyatt et al. [71]	Mathen [1]	Manjunath et al. [74]

Gray shaded rows indicate that we did not find approaches or tools to support that kind of testing activity even though they are necessary for a real-world data-warehouse.

6.1 Functional Testing of Front-End Applications

This testing activity ensures that the data is correctly selected and displayed by the applications to the end users. The goal of testing the functionality of the front-end applications is to recognize whether the analysis or end result in a report is incorrect, and whether the cause of the problem is the front-end application rather than the other components or processes in the data warehouse.

Golfarelli and Rizzi [8] compared the results of analyses queries displayed by the application with the results obtained by executing the same queries directly (i.e., without using the application as an interface) on the target data warehouse. They suggested two different ways to create test cases as queries for functionality testing. In a black-box approach, test cases are a set of queries based on user requirements. In a white-box approach, the test cases are determined by defining appropriate coverage criteria for the dimensional data. For example, test cases are created to test all the facts, dimensions, and attributes of the dimensional data.

The approaches proposed by Golfarelli and Rizzi are promising. In our project, we have used Achilles, which is a front-end application that performs quality assurance and analysis checks on health data warehouses in the OMOP [77] data model. The queries in Achilles are executable on OMOP.

The functional testing of the front-end applications must consider all possible test inputs for adequate testing. As it is impossible to test the functionality of the front-end applications with all possible test inputs, we can use systematic input space partitioning [67] techniques to generate test data.

6.2 Usability Testing of Front-End Applications

Two different aspects of usability of front-end applications need to be evaluated during usability testing, namely, ease of configuring and understandability.

First, we must ensure that the front-end application can be easily configured to be connected to the data warehouse. The technologies used in the front-end application should be compatible with the ones used in the data warehouse; otherwise, it will require several intermediate tools and configurators to use the data warehouse as the application's back-end. For example, if an application uses JDBC drivers to connect to a data warehouse, and the technology used to implement the data warehouse does not support JDBC drivers, it will be difficult to connect the front-end apps to the back-end data warehouse. We may need to reimplement parts of the application that set up connections to the data warehouse or change the query languages that are used. We suggest evaluating this usability characteristic by measuring the time and effort required to configure the front-end application and connect it to the target data warehouse.

Second, we must ensure that the front-end applications are understandable by the end users [70], and the reports are represented and described in a way that avoids ambiguities about the meaning of the data. Existing approaches to evaluate the usability of generic software systems [78] can be used to test this aspect of front-end applications. These evaluations involve a number of end users to verify the application. Several instruments are utilized to gather feedback from users on the application being tested, such as paper prototypes [79], and pretest and posttest questionnaires.

6.3 Performance and Stress Testing of Front-End Applications

Performance testing evaluates the response time of front-end applications under typical workloads, while stress testing evaluates whether the application performs without failures under significantly heavy workloads. The workloads are identified in terms of number of concurrent users, data volumes, and number of queries. The tests provide various types of workloads to the front-end applications to evaluate the application response time.

Filho et al. [80] introduced the OLAP Benchmark for Analysis Services (OBAS) that assesses the performance of OLAP analysis services responsible for the analytical process of queries. The benchmark uses a workload-based evaluation that processes a variable number of concurrent executions using variable-sized dimensional datasets. It uses the Multidimensional Expressions (MDX) [81] query language to perform queries over multidimensional data cubes.

Bai [82] presented a performance testing approach that assesses the performance of reporting systems built using the SQL Server Analysis Services (SSAS) technology. The tool uses the MDX query language to simulate user requests under various cube loads. Metrics such as average query response time and number of queries answered are defined to measure the performance of these types of reporting services.

The above two tools compare the performance of analysis services such as SSAS or Pentaho Mondrian. However, the tools do not compare analysis services that support communication interfaces other than XML for Analysis (XMLA), such as OLAP4J.

6.4 Summary

Table 11 summarizes existing approaches that test front-end applications in data warehousing systems. Although it is important to assess usability, none of the approaches addressed usability testing. Stress testing of the front-end applications is not reported in the surveyed approaches. Below is a summary of the open problems in testing front-end applications, and ideas for future investigation.

• • Existing approaches proposed for functionality testing of the front-end applications compare the results of queries in the application with the ones obtained by directly executing the same queries on the target data warehouse.
• • Functional testing of the front-end applications must consider all types of test inputs. We can use input space partitioning techniques to generate test data for this testing activity.
• • To support usability testing of front-end applications, we define a new aspect of testing to assess how easy it is to configure the application. We can test this aspect of usability by measuring the manual effort involved in configuring the front-end application in terms of time.
• • The voluminous data involved in the data warehousing systems makes performance and stress testing of the front-end applications difficult. Testers must use large datasets in order to perform realistic tests.

Table 11

Testing Front-End Applications

Test Category	Golfarelli and Rizzi [8]	Filho et al. [80]	Bai [82]

Gray shaded rows indicate that we did not find approaches or tools to support that kind of testing activity even though they are necessary for a real-world data-warehouse.