Psychological Testing: Chapter 5: Reliability

Consistency in measurement; the total variance in an observed distribution of test scores equals the sum of the true variance plus the error variance

Index of reliability; proportiion that indicates the ratio between the true score variance on a test and the total variance

X = T+ E
X = Observed score
T = True score
E = Error

Also true that the magnitude of the presence of a certain psychological trait as measured b a test of that trait will be due to the true amount of that trait and other factors

Statistic useful in describing sources of a test score variability; useful because it can be broken down into components

Variance from true differences

Variance from irrelevant, random sources

The greater the proportion of the total variance attributed to true variance, the more reliable the test

Test Construction
Administration
Scoring
Interpretation

Terms that refer to variation among items within a test as well as to variation among items between tests

Maximize the proportion of the total variance that is true variance and to minimize the proportion of the total variance that is error variance

Room temperature
Level of Lighting
Amount of ventilation and noise
Instrument used to enter responses and even the writing surface on which responses are written

Pressing emotional problems
Physical discomfort
Lack of sleep
Effects of drugs or medication

Examiner’s physical appearance and demeanor; presence or absence of an examiner

Technical glitches may contaminate data

Using the same instrument to measure the same thing at two points in time

Result of a reliability evaluation; estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test

Appropriate when evaluating the reliability of a test that purports to measure something that is relatively stable over time;

Estimate of test-retest reliability when the interval between testing is greater than six months

Alternate-Forms or Parraled forms coefficient of reliability

Exist when for each form of the test, the means and the variances of observed test scores are equal; means of scores obtained on parallel forms correlate equally with the tue score; scores obtained on parallel test correlate equally with other measures

Different versions of a test that have been constructed so as to be parallel; designed to be equivalent with respect to variables such as content and level of difficulty

Two test administrations with the same group are required
Test scores may be affected by factors such as motivation, fatigue, or intervening events such as practice, learning or therapy

Inherent in the computation of an alternate- or parallel-forms reliability coefficient; testtakers may do better or worse on a specific form of the test not as a function of their true ability but simply because of the particular items that were selected for inclusion in the test

Obtaining an estimate of the reliability of a test without developing an alternate form of the test and without having to administer the test twice to the same people

Obtained by correlating two pairs of scores obtained from equivalent halves of a single test administered once; useful measure of reliability when it is impractical or undersirable to assess reliability with two tests or to administer a test twice

Divide the test into equivalent halves.
Calculate a Pearson r between scores on the two halves of the test
Adjust the half-test reliability using the Spearman-Brown formula

Randomly assign items to one or the other half of the test; assign odd-numbered items to one half of the test and even-numbered items to the other half

assign odd-numbered items to one half of the test and even-numbered items to the other half

Each half equal to the other in format, stylistic, statistical, and related aspect

Allows a test developer or user to estimate internal consistency and reliability from a correlation of two halves of a test; Specific application to estimate the reliability of a test that is legnthened or shhortened by any number of items; used to determine the number of items needed to attain a desired level of reliability

The rule is that new items must be equivalent in content and difficulty so that the longer test still measures what the original test measured

When measuring the reliability of a heterogeneous test and speed test

Refers to the degree of correlation among all the items on a scale; calculated from a single administration of a single form on a test; useful in assessing the homogeneity of a test

Degree to which a test measures a single factor; extent to which items in a scale are unifactorial

Degree to which a test measures different factors; composed of items that measure more than one trait

The more homogeneous a test is, the more inter-item consistency it can be expected to have;
Desirable because it allows relatively straighforward test-score interpretation

Have similar abilities in the area tested

May have different abilities

Insufficient tool for measuring multifaceted psychological variables such as intelligence or personality

Developed their own measures for estimating reliability; Kuder-Richardson Formula 20 (KR-20)

Most popular formula

KR-20 and split-half reliability estimates will be similar

KR-20 will yield lower reliability estimates than the split-half method

Items that can be scored right or wrong, such as multiple choice items

A selected assortment of tests and assessment procedures in the process of evaluation; typically composed of tests designed to measure different variables

The Kuder-Richardson Formula 20 Reliability Coefficient

Used if there is reason to assume that all the test items have approximately the same degree of difficulty; Outdated in an era of calculators and computers

Variant of the KR-20 that has received the most acceptance and is in widest used today; mean of all possible split-half correlations, corrected by the Spearman-Brown formula; approriate for use on tests containing nondichotomous items; preferred statistic for obtaining an estimate of internal consistency reliability; formula yields an estimate of the mean of all possible test-retest, split-half coefficients; widely used as a measure of reliability, in part because it requires only one administration of the test; gives information about the test scores and not the test itself

Ranges in value from 0 to 1; impossible to yield a negative value of alpha, if negative, report as zero

0 Absolutely no similarity
1 Perfectly identical

• Alpha is usually reported as Zero

Degree of agreement or consistency between two or more scorers (or judges or raters) with regard to a particular measure

A way to determine the degree of consistency among scorers

Test-Retest
Alternate or Parallel Forms
Internal or Inter-Item Consistency

On a continuum relative to the purpose and importance of the decisions to be made on the basis of scores on the test

Test items are homogeneous or heterogeneous in nature
The characteristic, ability, or trait being measured is presumed to be dynamic or static
The range of test scores is or is not restricted
Test is a speed or a power test
Test is or is not criterion-referenced

True Variance 67%
Error due to Test Construction 18%
Administration Error 5%
Unidentified Error 5%
Scorer Error 5%

HOmogeneous in items if it is functionally uniform throughout

An estimate of internal consistency might be low relative to a more appropriate estimate of test-retest reliability

A trait, state, or ability presumed to be ever-changing as a function of situational and cognitive experiences; Obtained measurement would not be expected to vary significantly as a function of time, and either the test-retest or the alternate forms method would be appropriate;

Trait, state, or ability resumed to be relatively unchanging

If Variance of either variable in a correlational analysis is restricted by the sampling procedure used, then the resulting correlation coefficient tends to be lower; if the variance of either variable in a correlational analysis is inflated by the sampling procedure, then the resulting correlation coefficient tends to be higher

when a time limit is long enough to allow testtakers to attempt all items and if some items are so difficult that no testtaker is able to obtain a perfect score

Generally contains items of uniform level of difficulty so that when gien generous time limits, all testtakers should be able to complete all test items correctly; based on performance speed; time limit is established so that few, if any, of the testtakers will be able to complete the entire test

Based on performance from two independent testng periods using one of the following:
Test-Retest Reliability
Alternate-Forms Reliability
Split-Half Reliability from two separately timed half tests

The obtained reliability coeffiient is for a half test and should be adjusted using the Spearman-Brown formula

Result will be a spuriously high reliability coefficient; two people, one who completes 82 items of a speed test and another who completes 61 items of the same speed test; correlation of the two will be close to 1 but will not say anything about response consistency

Designed to provide an indication of whether a testtaker stands with respect to some variable or cirterion, such as an educational or a vocational objective; tend to contain material that has been mastered in heirarchical fashion; tend to be interpreted in pass-fail terms, and any scrutiny of performance on individual items tends to be for diagnostic and remedial purpose

Based on the correlation between the total scores on two admnistrations of the same test

A reliability estimate is based on the correlation between scores on two halves of the test and is then adjusted using the Spearman-Brown formula to obtain a reliability estimate of the whole test

Seek to estimate the extent to which specific sources of variation under defined conditions are contributing to the test score; A test's reliability is conceived of as an objective measure ofhow precisely the test score assesses the domain from which the test draws a sample

Universe of items that could conceivably measure that behavior; hypothetical construct: one that shares certain characteristics with (and is measured by) the sample of items that make up the test

May be viewed as an extension of true score theory wherein the concept of a universe score replaces that of a true score; developed by Lee J. Cronbach; Given the same conditions of all the facets in the universe, the exact same test score should be obtained

Encouraged test deelopers and researchers to describe the details of the particular test situation (universe) leading to a speciic test score

Described in terms of its facets

Include things like the number of items in the test, the amount of training the test scorers have had, and the purose of the test administration

The test score; analogous to a true score in the true score model

Examines how generalizable scores from a particular test are if the test is administered in different situations; examines how much of an impact different facets of the universe have on the test score

Influence of particular facets on the test score; similar to reliability coefficients in the true score model

Developers examine the usefulness of test scores in helping the test user make decisions; designed to tell the test user how test scores should be used and how dependable those scores are as a basis for decisions, depending on the context of their use

Provide a way to model the probability that a person with X ability will be able to perform at a level of Y; Stated in terms of personality assessment, it models the probability that a person with X amount of a particular personality trait will exhibit Y amount of that trait on a personality test designed to measure it; not a term used to reer to a single theory or method

Physically unobservable

Synonym for IRT; Propose models that describe how the latent trait influences performance on each test item; theoretically can take on values from -infinity to +infinity;

Difficulty Leel of an Item

| Item's Level of Discrimination

Refers to the attribute of not being easily accomplished, solved, or comprehended; May also refer to physical difficulty

How hard or easy it is for a person to engage in a particular activity

Signifies the degree to which an item differentiates among people with higher or lower levels of the trait, ability, or whatever it is that is being measured

Test items or questions that can be answered with only one of two alternate responses, such as true-false, yes-no, or correct-incorrect questions

TEst items or questions with three or more alternative responses, where only one is scored correct or scored as being consistent with a targeted trait or other construct

Developed a group of IRT models; each item on the test is assumed to have an equivalent relationship with the construct being measured by the test;

Helps the test developer build an adequate measuring instrument
Helps the test user select a suitable test
Its usefulness does not end with test construction and selection

SEM; provides a measure of the precision of an obsered test score; provides an estimate of the amount of error inherent in an obsered score or measurement; inverse relationship between SEM and reliability of a test; the higher the reliability of a test (or individual subtest within a test) the lower the SEM; tool used to estimate or infer the extent to which an observed score deviates from a ture score; standard deviation of a theoretically normal distribution of test scores obtained by one person on equivalent tests

Another term for Standard Error of Measurement; Index of the extent to which one's individual's scores vary over tests presumed to be parallel

Range or band of test scores that is likely to contain the true score

A statistical measure that can aid a test user in determining how large a difference should be before it is considered statistically significant

How did this individual's performance on test 1 compare with his or her performance on test 2?
How did this individual's performance on test 1 compare with someone else's performance on test 1?
How did thisindividual's performance on test 1 compare with someone else's performance on test 2?