Appendix 2.B Item Reliability

Reliability refers to the consistency or repeatability of measurement. It is defined as the extent to which the variance of an observed variable is explained by the true score that the variable is designed to measure. In other words, the reliability is defined as the ratio of the true score variance to the total variance of the observed measure. Let us define where is the true score that y is to measure and is the measurement error. The reliability of variable y is defined as the ratio of variance explained by the true score to the variance to the observed variance of y:

(2.10)

where and are variance of the true score and variance of measurement error; respectively, and is the factor loading. Note that the reliability defined in Equation (2.10) is in fact the squared correlation between the observed variable y and its true score:

(2.11)

where is the unstandardized factor loading of y on . This also supports that the squared standardized factor loading () of an item estimated in a CFA model is the estimated reliability of the item.

In test theory there are three major types of observed variables: parallel, tau- equivalent, and congeneric measures. As an example, let us define variable y measured at times 1 and 2 as y₁ and y₂, respectively:

(2.12)

where the measurement errors are assumed not to be correlated.

If and , then y₁ and y₂ are parallel measures.

If and , then y₁ and y₂ are tau- equivalent measures.

If and , then y₁ and y₂ are congeneric measures, that is, the most general measure.

Assuming parallel measures, the correlation between y₁ and y₂ can be described as:

(2.13)

where is the item reliability. Equation (2.13) implies that the item reliability can be estimated using test– retest measures of the same variable x, assuming: (1) the test and retest measures of y are parallel measures; (2) participants' responses at time 2 were not dependent on their previous responses; and (3) there was no change in the true score between test and retest time. Assuming parallel measures, the reliability of y is simply the correlation between the test– retest measures of y. Though the assumption of parallel measures may not be realized in real research, the test– retest reliability is widely used as an approximate estimate of item reliability in social science studies. Note that in application of this method, appropriately choosing the time interval for test– retest interviews is important. The time interval should be short enough to prevent changes over time in the measures under test, and long enough to prevent memory effect of responses in the first test.