7.2.1. Content validity

Often confused with face validity, content validity, sometimes called consensus validity, seeks to determine whether the scale reproduces the content of the phenomenon. In other words, it is a question of assessing whether the items generated reproduce the content of the phenomenon studied in its different facets. This underlines the importance of the phase of defining the phenomenon and generating indicators that can explain it. Chandler and Lyon (2001, p. 106) propose to assess face validity by “comparing the defined construct to the applied measures for that construct”, which is the first step in verifying content validity. According to the same authors “content validity is an iterative process, with evidence for content validity compiling with each successive study in a research stream” (pp. 106–107). Hardesty and Bearden (2004) argue that face validity is the ability to ensure that each item measures what it is supposed to measure, while content validity assumes that the sample of selected items represents the entire domain of the construct. Rossiter (2002), although he distinguishes between face validity and content validity, assumes that the latter, which he equates with an appeal to reason, is the most decisive aspect, or even the only element to be considered for a construct.

Representing the content of a phenomenon through a set of items first involves a literature review to better conceptually define the phenomenon, its dimensions, its manifestations, etc. and then the opinions of a number of experts (researchers, managers, consumers, etc.). These experts have to do two principal activities: first propose items to explicate the phenomenon, second evaluate the clarity, quality and importance of the indicators in order to make a first selection and identify those that seem most representative of the mental processes that refer to the construct studied. The use of experts very early in the scale development process in this quest for content validation also helps, in addition to the conceptual framework, to inform decisions about the nature of the specification of construct-indicator relationships (formative or reflective).

As noted in Chapter 3, such a review avoids errors in the development and evaluation procedures of the scale. The choice of experts is therefore crucial: they must be well informed about the construct, its dimensions or any other aspect that may help in its understanding. An expert is any person who is concerned, involved or has knowledge of the phenomenon to be measured. From one construct to another, the potentially solicited experts can be very different.

The subjective judgments provided by the researcher about the construct are thus largely based on conceptual definitions and exploratory qualitative investigations with experts. Content validity is the first to be assessed before embarking on data collection from the population concerned. This is the minimum level of validity required to consider field tests. Without questioning the other types of validity of a scale, it is recurrent issue to insist on the relevance of content validity and to recall that some researchers, notably Rossiter (2002; 2008), attest to its privileged status in the conception of a measurement (see chapters 3 and 4). Hair et al. (2011, p. 146) have even proposed that for the evaluation of formative indicators, the literature review and expert judgment are the most important aspects.

Ensuring the content validity therefore means that the selected items are significantly representative of the domain of the construct to be measured, and that it is then possible to undertake empirical tests. MacKenzie et al. (2011, p. 304) suggest two decisions when assessing content validity:

• – verify that each item taken individually is representative of an aspect of the content domain of the construct;
• – check if all the items taken collectively are representative of the entire content domain of the construct.

Although some researchers have attempted to structure (Schriesheim et al. 1993; MacKenzie et al. 2011; Malhotra et al. 2012), use (Uhrich and Benkenstein 2010) or support (Tojib and Sugianto 2006) a quantitative procedure that provides statistical indicators to demonstrate the degree of agreement between different judges in assessing content, the assessment of this validity remains more qualitative. On this subject, Hinkin (1995) specifies that content validity is made through the construction of items that capture the domain or construct being studied, adding that according to Stone (1978, cited by Hinkin 1995, p. 969) there are no quantitative indices to establish it but judgments must be made by evaluating the correspondence of the measure to the theoretical construct.

In order to clarify these apparently divergent points of view, we will note that quantitative indices are welcome. They reassure the scale developer about the a priori adequacy of its content to a construct. These indices function as indicators of the reliability of qualitative data derived from judges’ assessments. As an indication, Perreault and Leigh (1989) presented some indices to attest to inter-judge reliability, including Cohen’s Kappa coefficient, which is often used for this purpose. As in any qualitative protocol, it is obviously fruitful to ensure the coherence of the approach and the results that emerge. Nevertheless, estimates of the degree of agreement between the experts cannot be used as sufficient evidence of the quality of the measuring instrument, but can be used as an indication to continue the scale development process.

It should be recalled that, although content validity is necessary to conduct the rest of the scale construction process, it is not sufficient to attest to the correspondence of scale to the measurement of the phenomenon (Malhotra et al. 2012). Indeed, other levels of validity, which we will discuss below, must be established to authorize the use of the measuring instrument. It must also be stressed that content validity is not only an assessment of quality when developing a scale, but is also almost decisive when adapting or even adopting a scale. Each time a scale is borrowed, modified or constructed, it is imperative to focus on the items one by one to verify their correspondence with certain aspects of the phenomenon to be evaluated (face validity) and to verify if all the selected items are capable of reproducing the content of the phenomenon to be operationalized (content validity), in the context of the proposed study.

Let us add that the inspection of the format of the items (type of scale, number of response points, response labels, etc.) is also important in order to reach content validation. Indeed, it is not only the content of an item that contributes to the meaning created in the mind of a respondent regarding a phenomenon. All the design attributes of a scale are to be considered because they form a cluster of signals on which respondents base their understanding of content and therefore of the construct to be examined. From one format to another, the content of the items may differ significantly according to other perceptions (see Chapter 5). Even if this seems to make the process of operationalizing a construct more cumbersome, such an undertaking makes it possible to have more confidence in the choice of indicators and a faster and more precise adjustment of the scale for the indicated phenomenon.

Hinkin (1998) has attested in this regard that investing time and effort to assess the adequacy of content at the very beginning of the research process reduces subsequent problems concerning the inadequacy of the psychometric properties of the measure. Moreover, as MacKenzie et al. (2011) have pointed out, all the care taken in defining the focal construct must also be considered for its different dimensions when it is multidimensional (definition, distinction from other constructs, nature of the relationships with the focal construct, etc.). There is no doubt about the relevance of this phase of content validation for each latent construct and its indicators.

Figure 7.1 proposes, for a construct with two dimensions A and B, an illustration to support the understanding of the concepts of face validity and content validity of a sample of items.

7.2.2. Construct validity

This aspect of validity, also called trait validity, “refers to how well the items in the questionnaire represent the underlying conceptual structure” (Rattray and Jones 2007, p. 238). According to Peter (1981, p. 134), “the term construct validity generally is used to refer to the vertical correspondence between a construct which is at an unobservable, conceptual level and a purported measure of it which is at an operational level”. Bagozzi and Yi (2012, p. 18) define the construct validity as: “the extent to which the indicators of a construct measure what they are purported to measure”. The construct validity thus seeks to establish whether the scale provides an adequate representation of the phenomenon studied. To do this, two components of the construct validity are to be estimated, namely, convergent validity and discriminant validity, defined as follows:

• – convergent validity consists of seeing whether the items establishing a priori the same phenomenon are strongly correlated;
• – discriminant validity consists of seeing whether the items establishing a priori different phenomena (or distinct dimensions of the same phenomenon) are not correlated or at most weakly correlated.

This construct validity has mobilized the most investigations by developers and users of measurement scales, particularly reflective ones. Moreover, it seems to be the most difficult to demonstrate. The presentation of the main protocols of its assessment will be discussed in section 7.3.

7.2.3. Nomological validity

This validity, sometimes called predictive validity, focuses on studying the potential of a measurement scale of a phenomenon to establish links with other measurement scales of other constructs that are theoretically supposed to be linked. Bergkvist (2015, p. 247) notes that some definitions of predictive validity are not universally accepted. In fact, according to several researchers (Venkatraman and Grant 1986; Gurviez and Korchia 2002; Brahma 2009), it is generally accepted that:

• – predictive validity seeks to link the construct, through its measurement, to a single antecedent or consequence;
• – nomological validity attempts to ensure that the construct can be associated with several constructs in a network of more complex relationships.

According to Bagozzi (1981b, p. 327), the difference between predictive validity and nomological validity is a matter of degree, not nature. Figure 7.2 illustrates the distinction between these two often confused types of validity.

That being said, can a measurement scale that proves to have good content validity and reliability, while demonstrating good construct validity, be considered scientifically interesting to use if it does not enable identification of the planned conceptual links? We do not believe so; the nomological validity must be demonstrated to prove the adequacy of the measure to identify the phenomenon under study and its interdependence with other phenomena. This validity may not have received the attention it deserves in the development processes of many scales, as researchers often overlook this step. However, it must not be forgotten that the objective of the measure is not only to enable an adequate reading of a construct, but also and above all, to allow a confrontation of this construct with others that can lead to a better interpretation of the underlying phenomena.

To establish this validity, it is a question of choosing, according to the available theoretical foundations, a hypothetical framework supporting relationships between the phenomenon under study and other phenomena. Let us add that when the scale is first-order reflective multidimensional, which is often the case with many marketing constructs, it is necessary to identify the specific hypotheses for each of the dimensions. It should be noted, however, that the objective of nomological validity is not to seek to test hypotheses relating to problem-defining hypothetico-deductive research, but rather to focus on a few relationships identified and approved in literature in order to test the nomological validity of a new scale under construction.

On the technical side, the data obtained from the administration of different scales relating to the different phenomena studied with a sample of respondents would make it possible, through the appropriate statistical analyses (e.g. regression, structural equation methods), to test whether such relationships can exist. When relationships are established, the nomological validity of the scale under construction is a priori shown. However, it should be noted that the absence of links does not make it possible to definitively decide whether nomological validity is sufficient or not. Indeed, the absence of relationships can reflect one of the following two alternatives:

• – absence of nomological validity;
• – absence of the presumed conceptual links.

What then can be done to reassure oneself as to this nomological validity when the tested links are not verified? It is possible, for example, to re-compare the newly designed scale with other scales testing other links between the construct under study and other phenomena. Moreover, as we pointed out in Chapter 2, replicating the measurement of a phenomenon by confronting it with other construct measures makes it possible to support the nomological validity of a scale, even when it is already established. However, Peter (1981) points out that even when links are confirmed between two concepts (C1 and C2) by two measures (M1 and M2), the results do not definitively confirm nomological validity. He also argues that other aspects may explain such a relationship, including the possibility that the two instruments (M1 and M2) actually measure a single construct (C1, C2 or even C3).

This suggests that nomological validity is only obtained when several theoretically proven links between the studied construct and other constructs are empirically confirmed through the confrontation of the scale to be validated with others already validated. If the confirmation of nomological validity fails, the test must be repeated with other variables to verify other conceptual links or, if necessary, challenge the scale, while reverting to previous steps in developing the scale. Figure 7.3 shows this search for nomological validity.

Note that nomological validity is essential for all types of specifications (reflective and formative) and cannot be assessed if the previous validation steps (example: content validity) are not passed. Indeed, nomological validation of a measurement is a step that is at the end of the scale development process. Of course, this does not mean that once the scale is validated, its future adaptations (or adoptions) will be exempt from investigation (for reflective scales: content validity, dimensionality, reliability and construct validity). In fact, once developed and validated, an instrument has the potential to be reused, but all the verifications we have previously presented must be implemented again to verify the equivalence and adequacy of the measure of the studied construct and the context of its use.

7.3.1. The multitrait-multimethod matrix (MTMM)

This matrix is developed by Campbell and Fiske (1959) to assess the construct validity (convergent validity and discriminant validity). This approach has been the subject of some discussion on its potential to estimate validity (Bagozzi and Yi 1991; 1993; Cote 1995). Although not very popular with very modest applications (compared to the CFA), mainly because of the difficulty of using multiple measurement methods with the same items (traits), the MTMM approach seems relevant for studying the validity of several constructs and measurement methods (Heeler and Ray 1972; Lammers 1985; Tein et al. 1994; NeiBecker 1997) and its use has been highly recommended. For example, in the field of research on family purchasing decision making, such an approach has often been proved to attest to the validity of the constructs studied and to make comparisons between different traits (items) and measurement methods; in short, to construct and compare measurement scales (Szybillo et al. 1979; Seymour and Lessne 1984; Hopper et al. 1989; Frikha 2009).

MTMM is a matrix of correlations between different items (traits) measured by different methods, which of course presumes that the data has metric properties. In the case where three traits are measured by three methods, the matrix takes the following form (Figure 7.4).

Malhotra et al. (2012) have suggested that the MTMM approach can be used to establish the convergent and discriminant validity of single-item measurements. However, the use of such an approach seems to be encouraged when several methods of measuring different traits (items) are considered. While the difficulty of finding different measurement methods for the same traits has sometimes been pointed out (Peter 1981; Brée 1991), this approach offers great richness where applied in that it provides a wealth of information (see section 7.4). The three main benefits are as follows:

• – first, it allows us to verify if a construct (trait) can be measured by different methods. This possibility is of paramount importance because it can inform us, for example, about the possibility of having several formats of a scale (measurement method) that can be associated with the selected items;
• – second, it allows us to study whether the traits (items) can be differentiated from each other;
• – third, it offers the possibility of distinguishing variations between different traits (items) caused by their individual characteristics from those due to measurement methods.

Moreover, Heeler and Ray (1972) have long pointed out that the possibilities for validating studies seem to expand with the MTMM approach.

It should be noted that it is also possible to introduce in this matrix the monotrait-monomethod reliability diagonal which represents the correlations of the measurements of the same trait using the same method, for example by making measurements spaced by a time interval, which recalls the principle of test-retest reliability. However, this reliability diagonal is not often indicated. Particularly adapted to multi-item reflective scales, for each of the two facets of the construct validity, the MTMM matrix operates as follows.

7.3.2. Confirmatory factor analysis

Structural equation modeling (SEM) methods examine the relationships that may exist between variables (items) and constructs to evaluate both measurement models (scales) and underlying structural models (Gerbing and Anderson 1988). These methods can thus be used at two stages of the measurement scale validation process:

• – when establishing the construct validity (convergent validity and discriminant validity) while testing the links between the observable variables (items) and the construct(s) reflecting them (measurement model);
• – when establishing nomological validity by confronting the scale with others according to a theoretical framework assuming a set of links between the different crossed constructs (structural model).

Apart from its potential to evaluate all the properties of a scale (reliability, convergent validity, discriminant validity and nomological validity), the success of structural equation modelling methods, also known as structural equations with latent variables and measurement errors, has been greatly facilitated and encouraged by the development of ad hoc software, the consideration of measurement errors and, in addition, the possibility of using a single measurement method to collect data used to establish the validity of the construct, unlike the MTMM approach. Listing the scales that have been subjected to this construct’s validation protocol has now become impossible given its use. Indeed, the application of SEM for scale development covers many areas and is widespread (Guiry et al. 2006; Frikha and Khrouf 2013; Hollebeek et al. 2014). Figure 7.6 illustrates the measurement models of three constructs (X, Y and Z) related in a structural model.

In this section, we will focus only on the evaluation of a reflective measurement model to attest to the quality of the fit of the measurement model and to evaluate, in addition to reliability, the construct validity (convergent validity and discriminant validity). The evaluation of a measurement model by structural equation methods thus seeks to confirm (or not) the structure of the scale (to empirical data) where a set of items is identified as reflecting a latent construct (reflective model). Although nomological validity is also important to verify and structural equation modeling methods are very efficient tools for this undertaking, it should be noted that depending on the constructs to be crossed, several statistical methods can be used to verify the established relationships. Without dwelling on the technical considerations that are often quite elaborate but very explicit in several writings (Jöreskog and Sörbom 1982; Bagozzi and Yi 1988; Akrout 2010), we will present in the following the main foundations and the general approach to be followed during a confirmatory factor analysis.

The structural equation modeling methods, while using a correlation or covariance matrix, compare the collected data with the model of the configured relationships. Several methods are possible, some based on covariance estimation (CB-SEM) and others based on variance estimation (PLS-SEM). Before implementing a confirmatory factor analysis, it is necessary to ensure that a set of conditions for applying the proposed estimation method are met, in particular in terms of sample size, the nature of the variables, etc. For example, for the Maximum Likelihood (ML) estimation method, often used for its robustness in estimating a measurement model, it is imperative, among other things, to verify the normality of the data and to have a sufficient number of observations. For this last point, the debates are very rich: a number 10 times greater than the number of observable variables (items) seems to be a minimum requirement. Admittedly, other estimation protocols are possible when restrictions exist in terms of sample size, data normality or, in the context of a purely formative measurement model, such as the partial least squares method (PLS) (Roy 2008; Henseler et al. 2009; Wetzels et al. 2009; Hair et al. 2011; Wong 2013; Hair et al. 2014).

More generally, several estimation methods are possible, each with its own constraints and advantages. Once all the conditions required for the application of the chosen estimation technique have been met, the researcher can progress in the analysis of the measurement model. He or she specifies all the links between the items associated with the latent variables they are supposed to capture and analyzes the quality of the scale. In this perspective of evaluating the validity of a reflective measurement model, a series of steps are generally to be followed. These are mainly the verification of the quality of the model fit, the assessment of reliability, convergent validity and discriminant validity.

7.4.1. Focus on confirmatory factor analysis (CFA)

Confirmatory factor analysis, which is widely used for validation purposes, has some limitations and precautions are required, in particular two: reading the resulting indices and selecting the proposed modifications.

7.4.2. Focus on the multitrait-multimethod matrix (MTMM)

Although often neglected in favor of confirmatory factor analysis (CFA), the multitrait-multimethod approach (MTMM) to assess the construct validity accommodates to several situations (multiple products, multiple contexts, multiple respondents, multiple scale formats, etc.) with a large potential for exploitation, even if it is subject to some limitations. We will discuss below some of its potential applications and some of its limitations.

7.4.3. Synergy in perspective

The growing interest in protocol development to ensure the construct validity should not lead scale developers to turn a blind eye to the need to establish other forms of validity associated with a scale. Indeed, there is no reason to believe that the construct validity is sufficient to establish the validity of a measurement scale.

Validity should be seen as a sequential process in which all steps are involved at some moment in the scale development process. The most important thing is to take full care of each of the components of validity to ensure a faster, less costly process and, above all, to ensure that the results are as close as possible to the true value of the phenomenon being studied. We will relay below some of the synergy effects that can result from the different variants of the validity of a scale and/or their validation protocols:

• – it is clear that reading the indices for assessing convergent and discriminant validity, according to the two validation protocols of a construct (CFA and MTMM), are largely based on the researcher’s judgments, the thresholds defined by experts and research in a specific field. Indeed, as we have already pointed out, the levels of the various indices considered acceptable to attest to the construct validity are more often based on use than on a real fixed scientific assessment. It is also very useful to note that each approach has its advantages and disadvantages. This shows the value of triangulation, not only of the indices, but also of the approaches in particular of the validation that the researcher could consider. No one construct validity method (MTMM and CFA) can be considered superior to the other. Each can be used for a relevant evaluation of certain aspects of a measure. These approaches seem rather complementary. For example, Kim and Lee (1997) used the logic supported by the two approaches to construct validation (MTMM and CFA) to allow for a more detailed examination of the measurements. Bagozzi (2011, pp. 288–289) explicitly supported the benefits that can be derived from exploiting different approaches to establishing the construct validity and even the potential gains that can be made by combining different protocols in the same study;
• – whatever the method used to test the construct validity (MTMM or CFA), the examination of discriminant validity is of no interest when convergent validity is not established. Moreover, this logic can be applied to the whole process. Thus, for a reflective model, unless content validity is established, it is not appropriate to test the dimensionality of a scale (EFA). Similarly, it is only when reliability is confirmed for each of the established dimensions that it is possible to focus on testing the construct validity and finally, of the nomological validity. Of course, this in no way excludes the interest of undertaking, at times, a back and forth between steps in order to adjust the scale. For example, when the dimensions identified through an exploratory factor analysis do not match the researcher’s expectations, it is necessary to turn back, return to the conceptual framework and potentially solicit some experts to better explain the factors obtained and attest to the validity of their content. Once the explanations have been found, come back to the field to collect new data, test dimensionality, reliability, convergent validity, etc.;
• – it is very useful to document the entire procedure used for testing and validating the measurement model. Indeed, in addition to all the details relating to the definition of the construct, the production of items, etc. highlighting all of the considerations and indicators used to evaluate the scale effectively ensures its relevance in identifying the expected realities. Admittedly, the objective of this undertaking is not to overload the discussion with overly technical comments that are useless for understanding the general approach adopted, but to provide a framework for analyzing the scale in order to facilitate the study of its equivalence with other contexts and thus facilitate its future adoption;
• – evaluating the content validity of a set of items is of great interest not only for establishing a first diagnosis concerning the correspondence of indicators to the definition of the construct and its different facets, but also to undertake an arbitration for a first purification of the items. Indeed, given the considerations in terms of the sample size required for adequate field tests and its dependence on the number of items, this qualitative purification has the advantage of making the number of essential observations less constraining. Admittedly, this judgment must not reduce the number of indicators to be kept, but must be able to justify the interest of the items to be used for the next steps and to verify for the formative models that all the items are identified without omission. Content validity is by no means an unnecessary step, it ensures that the development of the scale progresses more seamlessly and facilitates the other steps of the process.