4.2.2.1. Content validation

In the progression of steps proposed by Churchill (1979), content validity was addressed. However, it is not uncommon to observe that during the construction of many scales, this validity seems illusory. Bergkvist and Rossiter (2008), considering their work as an extension of Churchill’s scale development procedure, note Churchill’s insistence on giving importance to theoretical considerations as a crucial step in his procedure. Bergkvist and Rossiter (2008) note that, unfortunately, researchers have not paid enough attention to this aspect. This is not a recent observation; Spector (1992, p. 7) has noted that many scale developers do not do enough work at this stage to define the construct. Mowen and Voss (2008, p. 488) noted that “while Churchill’s (1979) approach includes construct definition and domain specification, his discussion of these ideas is limited”; they therefore propose that the definition of a construct should also take into account its antecedents and consequences (determining its nomological network). MacKenzie et al. (2011, p. 295) have pointed out that a poor definition of a construct leads to serious problems including:

• – confusion about what the construct refers to (or does not refer to);
• – deficient indicators because the definition of the focal construct is insufficiently developed or contaminated by the definition of other constructs;
• – invalid conclusions about relationships with others constructs.

In short, without a clear and precise definition of the content of a construct, it is almost impossible to progress in the construction of a scale, whatever the envisaged specification (reflective or formative). One of the undeniable contributions of the C-OAR-SE paradigm proposed by Rossiter (2002) is the particular attention that must be given to the definition of the construct and the validity of its content. In order to obtain a scale that can list all facets of a behavior, it is crucial that the observable indicators selected can be perfectly adjusted to the latent phenomenon in which the researcher is interested. All the steps of purification, validation and empirical testing undertaken for reflective scales are insufficient and unsuccessful if the researcher is not able, from the beginning of the process of genesis of the items, to find the essential and unavoidable qualifiers to cover the phenomenon in question. Straub (1989, p. 150) pointed out that an instrument with content validity “is difficult to create and perhaps even more difficult to verify because the universe of possible contents is virtually infinite”. In a formative model, this step is also fundamental to ensure that all the indicators forming the construct are identified and that no omission of any part of the content to which the phenomenon refers is made.

The first steps of a scale building process are too often rushed. However, it is at this stage that the researcher will find (or not) the key indicators referring to latent content. Sometimes he is confronted by items that seem redundant (same idea formulated in different ways), too long (several words make up the items) or ambiguous (containing difficult or confusing words). A series of questions come to the fore: which items should be retained? Which ones should be deleted? What words and adjectives should be used? At this level, there is also the question of reversed items, the importance of their use and their number. Of course, at this stage, it is not a question of reducing the scale in terms of number of items, but it is necessary to make some judgments about the potential of each item to capture as much reliable and valid information as possible while reducing some possible response biases. In a way, it is a question of undertaking an initial qualitative purification of the list of indicators selected.

With regard to reflective models, the researcher may decide to keep certain items that are considered redundant if they consider them to be potentially interesting. On the other hand, in the case of formative models, the researcher must ensure that all possible indicators are effectively identified and that there is no duplication. To assist in their judgment, the researcher may use experts. Moreover, Rossiter (2002) strongly supports the use of experts not only to generate items, but also to analyze those that have already been identified. Experts can effectively assist the scale developer in assessing each of the items, opposing adjectives, etc., while keeping in mind the definition of the phenomenon to be measured, in order to select those that:

• – best represent the construct;
• – are equivalent for the entire population;
• – limit a priori response bias.

These qualitative explorations with experts should only terminate once an agreement has been reached between the judges. While analyses at later stages of the development process, particularly for reflective models (exploratory factorial analyses, reliability tests, confirmatory analyses, etc.), can provide statistical values to inform these decisions, the use of judges is irreplaceable in order to decipher the meaning of each item and its consistency with the construct under study. This makes it possible to assert the content validity that is often overlooked compared to other types of validity, in particular convergent validity and discriminant validity.

When examining items, one must focus on their ability not only to reproduce the content of a phenomenon, but also to reduce the biases they may generate during administration. Academic experts in scale design can be helpful when discussing and informing choices on topics concerning response formulation, reversed items, response formats, number of echelons, etc., which we will discuss in more detail later (Chapter 5).

Although content validity has often been examined with reference to indicators, it is worth noting that other components of a scale may be important in its assessment. Moreover, some have pointed out the importance of certain scale components, including format (Rossiter 2002) and item formulation (Churchill 1979; MacKenzie et al. 2011) when understanding a construct, but the scale construction processes they have proposed do not highlight this important step when evaluating items and their content.

It is also easy to observe that decisions relating to the design of a scale are often rushed, hence why they ought to be proposed as a separate step in scale development processes in order to highlight their relevance to the production of indicators and the measurement of a construct. While some components of the design of a scale appear to be rather closely associated with data collection systems, the attributes of a scale, or at least some of them, do not serve exclusively to facilitate the collection of quality information from respondents in the population concerned, but guide them in order to gain an understanding of the meaning of the indicators selected in the scale and to choose the best attitudinal response.

4.3.2.1. Defining the construct and the attributes validating its content

One of the major concerns of the first five steps is to produce a set of attributes that can capture the true meaning of the content of the phenomenon. This implies: 1) a broad understanding of the latent construct, its extent and its relationships with other phenomena; 2) an active search for indicators that can represent it; 3) reading the nature of the “construct-indicator” links (reflective or formative) to choose the indicators that will represent it; 4) a choice of the different evaluation attributes of the phenomenon (items, response labels, etc.); 5) verification in order to finally evaluate whether all the attributes identified (items, response format, etc.) offer the most appropriate way to apprehend the entire content of the construct and do not induce bias.

This reading of the construct and the different components of the scale, which makes it possible to discover a valid content of the phenomenon and then undertake field tests (axis 2), is based on two important types of investigations: the literature review and qualitative exploratory studies with experts. This first axis thus mobilizes a triangulation between a deductive approach (literature review) and an inductive approach (qualitative exploratory research).

The literature review can concern not only the construct in question (extended to various fields) but also its antecedents, consequences and related concepts in order to have the richest possible vision that delimits it, identifies its dimensions, examines its potential stability over time, allows elements of reflection on the attributes of the scale, etc. These conceptual investigations seek to establish an observation (measurement) close to the latent content and to justify the interest of developing a scale for the phenomenon if none of the existing ones can properly apprehend it.

With regard to empirical explorations, the experts mobilized are essentially individuals who know the subject (construct) or who are interested in the phenomenon to be operationalized. This potentially includes consumers, managers, researchers, etc. Several exploratory techniques (group interviews, individual interviews, etc.) can be used depending on the nature of the phenomenon, the characteristics of the experts, etc. These explorations must involve all categories of the population concerned (cultural groups, age groups, etc.) when the phenomenon is likely to be perceived differently from one group to another. Let us add that the number of these explorations cannot be defined in absolute terms, but the more abstract the construct is, the more fruitful such explorations are in terms of content validity.

It should be recalled that, for formative specifications, the content validity can only be asserted when all formative indicators are identified because no omissions are allowed, unlike reflective specifications where a sample of items may be sufficient for the validity of its content. The latter is the first type of validity to be established, very early in the process of constructing a scale. Indeed, it is not possible to advance in the development of a scale (whatever its specification) if the content validity is not established. Admittedly, it is always possible to decide to turn back and re-examine the correspondence of a set of items to conceptual content in order to ensure the content validity when a doubt arises around the correspondence to the conceptual framework of the structure of a reflective scale (following for example an exploratory factorial analysis).

4.3.2.2. Data collection and scale purifications

This axis mainly concerns four stages (6, 7, 8, 9) whose main concern is to test the draft scale obtained following the previous stages (1, 2, 3, 4, 5), in order to establish a set of purifications on the basis of data from the field (and far from the judgments of experts qualified by some as subjective). This test covers a multitude of aspects:

• – some, applicable to all specifications (reflective and formative), concern the overall assessment of the scale by the population (understanding of the items, ease of administration, understanding of the response format, etc.);
• – others, more technical, are specifically applicable to reflective models (exploratory factorial analyses and classical reliability tests). For formative models, test-retest reliability is possible;
• – and finally, examinations of conceptual attributes (correspondence of the obtained structure to the theoretical definition of the construct and its different facets) are also important to conduct.

Thus, once a set of items has been selected and formatted, a succession of data collections-purifications is required. For these different collections, it is imperative to select individuals from the population concerned because obtaining data from other sources is likely to provoke measurement errors. In addition, as mentioned above, taking into consideration the number of individuals to be retained for these tests is a guarantee of quality. As such, if the objective is to have exclusively a global appreciation of the items formatted (example: in a Likert format), a reduced sample (20 to 30 individuals) may be sufficient. But if the researcher tries to analyze the factor structure and reliability of the scale (in a reflective model), a larger sample is required to allow statistical processing. For the latter, it is commonly accepted to start with exploratory factorial analyses (EFA) and then to carry out reliability tests on each of the dimensions obtained. Reliability can be assessed through several indices, such as Cronbach’s alpha (for reflective models). At this level of reflection, it should be mentioned that decisions are often to be made surrounding the choice of items to be retained (or removed) definitively from the scale (see Chapter 6). In order to make all these judgments operational, a back and forth between reading the statistical results obtained and the conceptual definition of the construct is necessary. Although we have already mentioned the number of possible collections for such an undertaking, it is interesting to recall that this succession of purification collections can only end once a stable factorial structure is obtained that is statistically and conceptually acceptable. Let us not forget that a purification, especially the first one based on the items generated, leads to the elimination of certain indicators, hence the importance of collecting additional data to review the structure of the scale. Although Figure 4.1 highlights two data collection processes for such purifications (steps 6 and 8), it should be noted that this is a minimum amount. Indeed, undertaking further collections, especially for some constructs that appear less stable, is often necessary before progressing to the next steps. This is the case when the first two collections do not result in a close structure, in accordance with the planned conceptualization. This becomes more complicated when the results (EFA, reliability) do not correspond to a potential theoretical structure, hence the urgency to turn back and check the whole process undertaken in the previous steps (1, 2, 3, 4, 5).

4.3.2.3. Validation of the scale and its ability to identify the phenomena studied

If the sequence of steps recommended in the two previous sections is properly carried out, it then gives rise to a series of items potentially distributed over one or more dimensions with good content validity and acceptable reliability for each of them. However, the fact remains that this series is not yet usable. Indeed, the content validity, established on a conceptual basis and based on expert judgments during the initial phases of the process, only makes it possible to reassure oneself that the scale captures the entire domain of the construct studied, but does not make it possible to know if it is able to reproduce the real phenomenon with a population. It is still necessary to test its validity with samples of individuals and to examine its explanatory power on a certain number of phenomena (steps 10, 11, 12, 13).

The validity of the scale used at this stage must be assessed in two steps:

• – first, it is a question of assessing the validity of the construct by testing the convergent validity and the discriminant validity. To do this, particularly for reflective models, alongside the protocol suggested by Campbell and Fiske (1959), it is fruitful to use structural equation methods. For formative models, a few indices can be used to attest to the validity of the construct, in particular the importance of the error term associated with the latent variable;
• – second, it is necessary to test whether the scale obtained is capable of capturing the phenomenon studied in relation to others associated with it in the literature; it is then a question of examining the nomological validity. To this end, the conceptually defined relationships between the phenomenon to be operationalized and the others identified are tested by a variety of methods including structural equations (SEM).

We will discuss the different variants of validity later (Chapter 7). However, it should be noted here that each of these validities (construct validity and nomological validity) should, whenever possible, be based on a new data collection. Replication of collections is indeed likely to promote a more reliable and valid instrument. It is therefore important, in order to refine a scale that can be trusted, to undertake data collection assessing the construct validity (convergent validity and discriminant validity) and another collection to relate this scale to others, in order to ensure its potential to explain the theoretical links that may exist between several constructs (hence the test of nomological validity). It should be noted that, for each new collection, it is imperative, before carrying out the recommended validity test (construct validity or nomological validity), to check again for dimensionality, reliability, etc.

Let us add that the last point to be considered when constructing a scale is the development of a set of standards through the use of a few indices (example: average) to facilitate the interpretation of the scores obtained by the scale. Although this last aspect is very often neglected, we point out its importance for better understanding the distribution of scores among individuals in the studied population and, consequently, in assessing the ability of the scale to provide an interpretation of individuals’ positions on different characteristics, the main objective of the measure.