Summary of Findings

This assessment evaluates the efficacy of continuous performance tests (CPTs) in the diagnosis of attention-deficit hyperactivity disorder (ADHD) and its potential use in titrating pharmacotherapy for this disorder.

ADHD is the most commonly diagnosed behavioral disorder of childhood, and is characterized by a persistent pattern of inattention and/or hyperactivity-impulsivity in multiple settings including home, school, and peer relationships. Without identification and proper treatment, ADHD can have serious consequences, including school failure, depression, conduct disorder, failed relationships, and substance abuse. The diagnosis of ADHD typically involves obtaining information from multiple sources including parent and teacher ratings of the child’s behavior, clinical interviews of the parents and child, the clinician’s observations, and neuropsychologcial testing.

The CPT is used frequently in research on attention deficits in children and adults. This computer-based test involves the rapid presentation of stimuli (typically numbers, letters or number/letter sequences) for up to 30 minutes. Children taking the CPT are instructed to respond to the “target” stimulus by pressing a button and to refrain from responding to “non-target” stimuli. Children’s responses, in particular, the types of errors made, are thought to indicate the level of inattention or distractibility. CPT performance measures include correct responses, omission errors, commission errors and reaction time.

This assessment addresses four questions about the efficacy of CPTs in diagnosing ADHD and in titrating pharmacotherapy. Answers to two of these questions employ scientific evidence from the published, peer-reviewed literature. Answers to two other questions require the use of reports, position statements and clinical guidelines by published national professional medical organizations to determine whether there is consensus on the use of CPTs in diagnosis and titration of therapy. The questions are:

1) Does the peer-reviewed published literature establish the reliability and validity of continuous performance tests (CPTs) for the purpose of diagnosing attention deficit disorder (ADD)/attention deficit hyperactivity disorder (ADHD)?

To evaluate the utility of CPTs in diagnosing ADHD, we calculated measures of sensitivity, specificity, positive predictive value and negative predictive value from eight studies that addressed this question. Our analysis revealed low to moderate measures of sensitivity (range = 9% to 88%). Measures of specificity ranged from 23% to 100%; however, the studies with higher specificities (100% and 94%) had low sensitivities (13% and 62%, respectively). Thus, the utility of the CPT as a stand-alone diagnostic tool is not high. Even so, we would not expect the diagnostic utility of any test or behavioral rating scale to be high when it is used as the sole instrument for diagnosis. Comparing the performance of the CPT to the performance of any other diagnostic test for ADHD would require a full assessment of the other test.

It is possible that CPTs may be useful in measuring some isolated symptoms of ADHD; however, we did not consider this use in this assessment because it is not clear how these symptoms would relate to the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) or to diagnostic judgements of clinicians. Furthermore, there is little research addressing how or whether the CPT, or any other test, could be used in conjunction with other assessments as part of a clinical algorithm for diagnosing ADHD.

2) Does the peer-reviewed published literature establish the reliability and validity of continuous performance tests (CPTs) for the purpose of titrating pharmacotherapy levels in treating patients with attention deficit disorder (ADD)/attention deficit hyperactivity disorder (ADHD)?

We were unable to demonstrate a strong correspondence between CPTs and ADHD as measured by the DSM-IV or clinical judgment. Thus, the only type of study that can answer Question 2 would be one that correlated measures of drug dosage, CPT scores, and DSM-IV evaluation (or a reference standard validated by the DSM). We found no suitable studies of this type. Thus, we were unable to find literature that either supports or does not support the use of CPTs for titration of medications.

Although there is clinical evidence indicating that the dose of methylphenidate given to a child affects CPT performance, these trials did not validate the CPT by correlating its results to diagnosis of ADHD or to individual treatment related outcomes such as school performance. Associations between drug dose and such treatment related outcomes were not covered by the key questions in this assessment. If other evidence establishing an association between CPT scores and ADHD diagnosis or school performance is available, it may be possible to establish the validity of the CPT in titrating methylphenidate dose.

3) Do the published reports of national professional medical associations, national medical policy organization positions, or reports of national expert opinion organizations demonstrate consensus in the medical community that the safety and efficacy of continuous performance tests (CPTs) are accepted for the purpose of diagnosing attention deficit disorder (ADD)/attention deficit hyperactivity disorder (ADHD)?

All of the guidelines we retrieved advised against use of CPT scores alone for diagnosis of ADHD. Although some guidelines recognized some usefulness of the CPT during diagnosis and management, none specified any particular purpose for CPTs other than research.

4) Do the published reports of national professional medical associations, national medical policy organization positions, or reports of national expert opinion organizations demonstrate consensus in the medical community that the safety and efficacy of continuous performance tests (CPTs) are accepted for the purpose of titrating pharmacotherapy levels in treating patients with attention deficit disorder (ADD)/attention deficit hyperactivity disorder (ADHD)?

We identified only one guideline that addressed the issue of CPTs and medication for ADHD. The guideline questioned whether the behaviors measured by CPT tests were representative of behaviors likely to be expressed by children in more natural settings (e.g., home or school). The guideline concluded that the applicability of CPTs to monitor treatment for ADHD is “unproven or even absent.”

Preface

This assessment is organized into three major sections: 1) Overview, 2) Analysis of the Currently Available Evidence, and 3) Economic and Regulatory Issues. In the Overview section, we provide background information related to the particular health condition or illness under evaluation, including details about the epidemiology, diagnosis and treatment of the condition or illness. We provide background information on the specific instrument(s) often used for diagnosing the condition or illness, and some details on competing or complementary diagnostic technologies. The final part of the Overview section addresses issues relating to patient populations under study, patient safety and care setting.

The Analysis of the Currently Available Evidence section details the methods we used to evaluate currently available data. We detail the strategies employed for our searches of the literature, which includes an exhaustive list of the electronic databases searched and the protocol for hand-searches of the non-journal literature. We describe the inclusion and exclusion criteria used to identify, retrieve and analyze studies. When appropriate, we provide details of the characteristics of those studies included, and we list and define the key outcomes for analysis. We also describe any statistical methods that we employed. Finally, for each of the key questions addressed in the assessment, the synthesis of results is presented and discussed.

In the Economic and Regulatory Issues section, we provide information on the manufacturers of devices or technologies used in the studies analyzed for this assessment. Where available, we also provide cost information for the device, instrument or technology. We include information on whether the technology is regulated by the U.S. Food and Drug Administration (FDA) and if so, the status of the technology in the FDA approval process. Lastly, we provide information on health insurance coverage for the technology under evaluation. This includes discussion of the coverage policies of Medicare, Medicaid and other third party payers.

Overview

Attention Deficit Disorder (ADD)/Attention Deficit Hyperactivity Disorder (ADHD)

Attention deficit hyperactivity disorder (ADHD) is the most commonly diagnosed behavioral disorder of childhood and is characterized by a persistent pattern of inattention, impulsivity and hyperactivity. (1,2) Children with ADHD usually have functional impairment across multiple settings including home, school, and peer relationships. ADHD also has been shown to have long-term adverse effects on academic performance, vocational success and social-emotional development. Without identification and proper treatment, ADHD can have serious consequences, including school failure, depression, conduct disorder, failed relationships, and substance abuse. (3)

Diagnosis

There is no laboratory test or set of physiological features that has been identified as an unequivocal marker for ADHD. That is, there is no “gold standard” for diagnosing ADHD. The disorder is behaviorally based; thus, behavioral observations are required to identify and correctly diagnose the disorder. It has been argued that ADHD is not a distinct diagnostic entity, but that it is a “symptom complex” characterized by multiple possible etiologies and a constellation of pathologic behaviors. (4,5) The observed behaviors are interpreted subjectively by parents and teachers who describe these observations to clinicians. Clinicians often observe the child during clinical interviews and psychometric testing. Typically, parent, teacher and clinical observations are incorporated into a diagnostic decision. However, this subjective interpretation can lead to inter-observer differences, and can make ADHD diagnosis difficult. For example, the prevalence of behaviors related to hyperactivity as rated by a teacher can be higher than that rated by a clinician. (6) In contrast, the prevalence of behaviors related to hyperactivity can be lower if these behaviors must be judged to be present by more than one source (e.g., parent and teacher). (7)

Because there is no gold standard for diagnosing ADHD, it is important to make a distinction between how ADHD is defined and how it is diagnosed. The disorder is currently defined by criteria contained in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV). (3) The DSM-IV defines ADHD according to two behavioral domains: inattention and hyperactivity-impulsivity. Each domain contains nine possible symptoms; a child must have at least six of the nine symptoms to qualify for a diagnosis of ADHD. If the child has at least six symptoms on the inattention domain, s/he qualifies for the “ADHD-Predominantly Inattentive Type” diagnosis. If the child has at least six symptoms on the hyperactivity-impulsivity domain, s/he qualifies for the “ADHD-Predominantly Hyperactive-Impulsive Type” diagnosis. If the child has at least six symptoms on both inattention and hyperactivity-impulsivity domains, s/he qualifies for the “ADHD-Combined Type” diagnosis. (3)

Additional DSM-IV criteria specify that some inattentive or hyperactive-impulsive symptoms must have been present before the age of seven years, although the diagnosis can be made at older ages. In addition, the symptoms must be present in at least two settings (e.g., home and school) and must cause impairment. That is, there must be evidence of interference with developmentally appropriate social, academic, or occupational functioning. The symptoms should not occur exclusively during the course of a pervasive developmental disorder (e.g., autism), schizophrenia, or other psychotic disorder. Furthermore, the symptoms should not be better accounted for by another mental disorder (e.g., mood, anxiety, dissociative or personality disorders). (3) The DSM‑IV criteria for ADHD are listed in Table 1 .

Table 1. Diagnostic criteria for Attention-Deficit/Hyperactivity Disorder

Either (1) or (2):

1. Six (or more) of the following symptoms of inattention have persisted for at least 6 months to a degree that is maladaptive and inconsistent with developmental level:

Inattention

a. often fails to give close attention to details or makes careless mistakes in schoolwork, work, or other activities

b. often has difficulty sustaining attention in tasks or play activities

c. often does not seem to listen when spoken to directly

d. often does not follow through on instructions and fails to finish schoolwork, chores, or duties in the workplace (not due to oppositional behavior or failure to understand instructions)

e. often has difficulty organizing tasks and activities

f. often avoids, dislikes, or is reluctant to engage in tasks that require sustained mental effort (such as schoolwork or homework)

g. often loses things necessary for tasks or activities (e.g., toys, school assignments, pencils, books, or tools)

h. is often easily distracted by extraneous stimuli

i. is often forgetful in daily activities

2. six (or more) of the following symptoms of hyperactivity-impulsivity have persisted for at least 6 months to a degree that is maladaptive and inconsistent with developmental level:

Hyperactivity

a. often fidgets with hands or feet or squirms in seat

b. often leaves seat in classroom or in other situations in which remaining seated is expected

c. often runs about or climbs excessively in situations in which it is inappropriate (in adolescents or adults, may be limited to subjective feelings of restlessness)

d. often has difficulty playing or engaging in leisure activities quietly

e. is often “on the go” or often acts as if “driven by a motor”

f. often talks excessively

Impulsivity

g. often blurts out answers before questions have been completed

h. often has difficulty awaiting turn

i. often interrupts or intrudes on others (e.g., butts into conversations or games)

Some hyperactive-impulsive or inattentive symptoms that cause impairment were present before age 7 years.

Some impairment from the symptoms is present in two or more settings (e.g., school [or work] and at home).

There must be clear evidence of clinically significant impairment in social, academic, or occupational functioning.

The symptoms do not occur exclusively during the course of a Pervasive Developmental Disorder, Schizophrenia, or other Psychotic Disorder and are not better accounted for by another mental disorder (e.g., Mood Disorder, Anxiety Disorder, Dissociative Disorder, or a Personality Disorder.

(Continued on next page)

Table 1. Diagnostic criteria for Attention-Deficit/Hyperactivity Disorder (cont’d)

Code based on type:

314.01 Attention-Deficit/Hyperactivity Disorder, Combined Type: if both Criteria A1 and A2 are met for the past 6 months

314.00 Attention-Deficit/Hyperactivity Disorder, Predominantly Inattentive Type: if Criterion A1 is met but Criterion A2 is not met for the past 6 months

314.01 Attention-Deficit/Hyperactivity Disorder, Predominantly Hyperactive-Impulsive Type: if Criterion A2 is met but Criterion A1 is not met for the past 6 months

Coding note: For individuals (especially adolescents and adults) who currently have symptoms that no longer meet full criterion, “In Partial Remission” should be specified.

While ADHD is defined by the DSM-IV criteria, the symptom complex is diagnosed by a clinician. In the absence of a gold standard, the “reference standard” is the clinician’s judgment. Ideally, this decision would be based on information gathered from a number of sources (e.g., parent, teacher, observations of the child), and would be reached by consensus. That is, a number of qualified clinicians would confer in making the appropriate diagnosis. Nevertheless, the clinician’s decision is ultimately a subjective one, and this introduces a level of variability that is difficult to control for in evaluating any tool used for diagnosing ADHD. Moreover, the DSM has undergone several iterations over the past two decades (see below), suggesting that ADHD is indeed a “symptom complex” characterized by behaviors that are difficult to agree upon.

The terminology for attention deficit disorder has changed over the past 70 years. Previous terms included “minimal brain dysfunction”, (8,9) “hyperkinetic syndrome”, “attention deficit disorder” (ADD), and “attention deficit disorder with hyperactivity” (ADDH). The DSM-II, published in 1968, (10) referred to the symptom complex as the “hyperkinetic reaction of childhood”, and described it as “characterized by overactivity, restlessness, distractibility, and short attention span, especially in young children; the behavior usually diminishes in adolescence.” (10) In the DSM-III (1980), ADHD was classified among the Disruptive Behavior Disorders and the terminology was changed to Attention Deficit Disorder (ADD). (11) DSM-III differentiated two types of ADD based on the presence or absence of hyperactivity: attention deficit disorder with hyperactivity (ADDH) and attention deficit disorder without hyperactivity (ADDnoH). (11) In DSM-III-R (1987), the distinction between attention deficit disorder with and without hyperactivity was shifted by focusing on a new term: attention deficit hyperactivity disorder (ADHD). (12) Attention deficit disorder without hyperactivity was included in a category referred to as undifferentiated attention deficit disorder (UAD). (12) DSM-IV (1994) reapplied the distinction between attention deficit disorder with and without hyperactivity by including categories and criteria for all three behavioral domains: inattention, hyperactivity and impulsivity (see Table 1). (3)

Because the DSM-IV refers to the disorder as Attention Deficit Hyperactivity Disorder (ADHD), we use the term for this assessment. However, in systematically reviewing the literature, we have identified studies published as early as 1938. Thus, as we discuss the results of individual studies that relied on prior versions of the DSM and its accepted terminology, we apply the terms used by the original investigators.

Epidemiology

The prevalence of ADHD has been investigated in the school, community, and clinic settings. One systematic review of the literature found that school-based studies reported prevalence rates for ADHD ranging from 4% to 26%; community-based studies reported prevalence rates between 4.5% and 12%. (1) Multiple logistic regression revealed that gender, setting and diagnostic definition (i.e., different versions of the DSM) were significant contributors to the variability in the prevalence of ADHD. (1) In the Ontario Child Health Study, ADHD prevalence rates also varied widely, ranging from 1% to 14%. (13) These findings are consistent with the notion that the diagnosis of ADHD is made in the absence of an appropriate reference standard. The prevalence of ADHD diagnoses is substantially higher in boys than it is in girls. Prevalence ratios in boys and girls range from 2:1 to 5:1. (2,13) The prevalence of ADHD appears to decline during adolescence for both boys and girls. (13,14)

ADHD can occur in individuals with considerable and varied comorbidity. (15,16) Systematic reviews of the literature examining the comorbidity of ADHD with other disorders, have reported prevalence rates for ADHD with conduct disorder (CD), oppositional defiant disorder (ODD), mood and anxiety disorders, and learning disabilities (LD). (1,15) For externalizing disorders (disruptive behavior disorders), ADHD and CD reportedly occur together in 26% to 50% of all cases studied in community and clinical samples, respectively. The comorbidity of ADHD and ODD in children is at least 35%, either for ODD alone or combined with CD in cases studied. (1,15) For internalizing disorders (mood and anxiety disorders), ADHD and mood disorders reportedly occur together in 15% to 75% of cases from community and clinical samples. (15) For anxiety disorders, ADHD co-occurs in approximately 25% of cases surveyed. (1,15) The proportion of ADHD children with multiple comorbidities was 28%. (1) The co-occurrence of ADHD and LD has been consistently reported in the literature, (15) but the reported rates of co-occurrence are variable, ranging from as low as 10% to as high as 92%. (16) The prevalence of LD in children with ADHD was about 9%-10% in hyperactive boys (17) and 8% in a sample of Connecticut school children. (18) Lambert and Sandoval (1980) found that 43% of hyperactive children met objective criteria for LD. (19)

To evaluate the efficacy of the CPT in diagnosing ADHD, it is important to consider ADHD children with comorbidities. That is, in the clinical setting children are likely to have ADHD as well as CD, ODD or LD. (3,20,21) It is not enough to differentiate ADHD from normal children (non-specific diagnosis), but one must distinguish among disorders that tend to occur together (differential diagnosis). Because ADHD frequently co-occurs with CD, ODD and LD, the DSM-IV criteria for these disorders have been appended (Appendices A, B and C).

Treatment

The most common treatment for ADHD is pharmacological intervention. It has been estimated that 2-3% of all elementary school-age children in North America receive some form of pharmacotherapy for hyperactivity. (22) While a variety of medications for ADHD have been used, including anti-depressants and neuroleptics, by far the most widely prescribed and extensively studied are the psychostimulants, especially Ritalin (methylphenidate). Other common stimulant medications include Adderall, Dexedrine, and Cylert (pemoline). These are considered the “first line” of pharmacotherapy for ADHD. (23)

More recently, psychosocial treatments have been examined as potential treatment strategies, either alone or in combination with stimulant medication. Such psychosocial treatments include classroom-based behavior modification, social skills and cognitive training, parent training and home-based interventions, and intensive summer treatment programs.

Overall, there appears to be a trend towards a multimodal approach to ADHD treatment. With this approach, clinicians are using multiple treatment strategies in combination. That is, stimulant medication is combined with cognitive and/or behavioral modification programs. Such programs may be individualized or designed to involve the child, his or her family and relevant school personnel. (24,25)

Continuous Performance Tests

Continuous performance tests (CPTs) are computer-based tests designed to measure inattention and impulsivity. The CPT is one of the most widely used measures of attentional deficits in research on children and adults. (26) . The original version of the CPT, (27) and more recent versions of the test, involve the rapid presentation of a series of visual or auditory stimuli over a period of time, typically from nine (28-30) to 22.5 (31,32) minutes. Specifically, the task requires a subject to respond to “target” stimuli and inhibit responding to “non-target” stimuli that appear on the monitor. In one of the most common CPT formats, the letter “X” is the predetermined target stimulus. For example, during a typical trial, 500 letters will be presented in rapid succession. When the target “X” appears, the child responds as quickly as he or she can by pushing a button on a control panel or computer keyboard. An early modification of the CPT “X” design made this task slightly more difficult by requiring subjects to respond to the letter “X” only when it directly followed the letter “A.” This format is known as the “AX” CPT. Other ways of making the task more difficult include degrading the stimuli by blurring them on the screen or adding adjacent, “distracting” stimuli. Another CPT format is the “identical pairs” CPT. In this test, there is no fixed target; instead, the target is the occurrence of the same stimulus (e.g., playing card) two times in a row. (33,34) This type of CPT is more attentionally demanding and can be associated with more complex stimuli, e.g., 4-digit numbers. (33) The child’s age is generally taken into account when selecting the appropriate CPT format, given that some are more difficult than others.

Stimuli other than letters are also used, including numbers (e.g., “5”), a series of numbers (e.g., a “1” followed by a “9”), geometric forms (e.g., T.O.V.A., see below), and pictures of animals for younger children. While most versions of the CPT are visual, others use an auditory mode (e.g., Auditory Continuous Performance Test (35) ), where numbers, letters or words are spoken and the subject responds to the target stimuli.

Several different performance outcomes are measured during testing. Responding to the designated target is referred to as a correct response, or a hit. Missing a target is referred to as an omission error. Omission errors are thought to measure inattention. (36,37) Responding to any stimulus other than the target is referred to as a commission error, or false alarm. Commission errors are considered to measure impulsivity. (36,37) Other measures of CPT responses include the number of correct responses, reaction time, and variability in reaction time.~~, and activity level~~

T.O.V.A.

The Test of Variables of Attention (T.O.V.A.) is a standardized, visual CPT designed specifically as a clinical tool to screen, assist in the diagnosis and to predict and monitor medication effects in children with attention deficit disorders. (38,39) The T.O.V.A. is a 22.5-minute computerized test that requires no use of language, no right-left discrimination, nor recognition of letters or numerals. Rather, the two visual stimuli are a large box containing a smaller box that is adjacent to the top or bottom edge of the larger box (see Figure 1 ). The stimulus of the inner box adjacent to the top edge is the designated target. The T.O.V.A. presents two 11-minute test conditions that involve frequent and infrequent presentation of targets. These two conditions were designed to measure attention and impulsivity, respectively.

Figure 1. Example of target and non-target stimuli for the T.O.V.A.

Reliability and Validity of CPTs

The reliability of a diagnostic test refers to the extent to which the test results are consistent over time or across conditions. Test-retest reliability is generally expressed as a correlation coefficient, or r. If there is a perfect positive correlation, r equals 1.0; for a perfect negative correlation r equals –1.0. If there is no correlation (i.e., no consistency), r equals 0. A desirable level of reliability for diagnostic tests is .80 or above. (40) For the CPT, the reliability of the test can be assessed by comparing performance scores over time and in different settings.

For the traditional “A-X” CPT, Halperin et al. (1991) demonstrated test-retest reliabilities ranging from 0.65 to 0.74 for hits, misses, hit reaction time and the derived inattention and impulsivity scores. (41) For the T.O.V.A., Greenberg and Waldman (1993) (31) showed test-retest correlations of 0.80 or greater for response times, 0.50 or greater for commission errors, and 0.14 for omission errors. On the SCAT (Seidel Continuous Attention Task) CPT, Seidel and Joschko (1991) found test-retest reliability measures ranging from 0.36 for commission error to 0.82 for reaction time. (42) For the GDS, test-retest reliability for vigilance task total commission errors ranged from 0.52 to 0.94. (43,44)

The validity of a diagnostic test refers to what the test measures and how well it does so. The validity of a test is the most important consideration in test evaluation and must be addressed for three categories: content-related validity, construct-related validity and criterion-related validity. (45) By design, CPT tests measure constructs. That is, the target and non-target letters, numbers or words are aids used to test the constructs of inattention and impulsivity. Two types of evidence are used to demonstrate construct validity: convergent and discriminant. The CPT would have high convergent validity if the results agreed with the results of other tests designed to measure the same constructs. The CPT would have high discriminant validity if the results objectively measured inattention and impulsivity independent of verbal, perceptual and other cognitive processes. (41,42,46)

Klee and Garfinkel (1983) (47) showed that CPT performance was small and positively correlated with behavioral ratings of inattention (r = 0.31 to 0.33) and hyperactivity (r = 0.34 to 0.36). Halperin et al. (1988) (46) showed that omission and commission errors correlate with teacher ratings of inattention (r = 0.25) and impulsivity (r = 0.37), respectively. For the SCAT, Seidel and Joschko (1990) (42) showed correlations with Teacher Questionnaire factors including Hyperactivity (canonical R = 0.38) and Inattention (canonical R = 0.35). Lovejoy and Rasmussen (1990) (48) reported very low correlations between teacher and parent ratings of child behavior and CPT scores. For commission errors, the correlations were 0.01 and -0.01 for teacher and parent ratings, respectively. The correlation for omission errors was the same for both ratings (0.01). DuPaul et al. (1992) (49) showed that teacher ratings of inattention and overactivity were not more than weakly correlated with either the total number of CPT correct responses (rs = 0.08 and 0.06) or commission errors (rs = 0.18 and 0.10). Comparing the correlations of the CPT with behavior ratings and correlations of other tests with behavior ratings would require a full assessment of these other tests.

Criterion-related validity refers to the degree to which the test correlates with a direct and independent measure of what the test is designed to predict. If the criterion measure is obtained at the same time as the test of interest, it is referred to as concurrent validity. Fisher et al. (1995) showed that diagnostic classifications based on CPT performance agreed with those based on diagnostic criteria 70% to 80% of the time. By contrast, DuPaul et al. (1992) showed a 44% concordance rate for CPT outcome measures and clinical criterion measures. (49) Criterion-related evidence of the CPT has been demonstrated in a number of studies. (42,50-52)

Role of the CPT in diagnosis of ADHD

In the absence of any unequivocal markers for ADHD, it is generally agreed that no single procedure, observation or behavioral characteristic is sufficient to support a diagnosis of ADHD. (3,26,32,53) Cohen et al. (1989) suggest a “trilateral” approach to diagnosing ADHD. (54) These authors suggest that clinicians include instruments that assess: 1) parent observations, 2) teacher observations and 3) the child’s test performance (e.g., CPT) when making diagnostic and treatment decisions. In addition, guides to professional practice and diagnosis emphasize the importance of obtaining a wide range of information from a variety of sources (see Question #3, below). Therefore, it is unlikely that CPTs will supercede other methods of assessment for the diagnosis of ADHD. Indeed, it is unlikely that any test, rating scale or behavioral assessment will be used as a sole-source diagnostic tool. In general, CPTs have been utilized as an objective method for assessing attention deficits. Such tests are not subject to the biases that can occur with more subjective assessment techniques including clinical interviews and rating scales.

Furthermore, because the underlying causes of ADHD are unknown, and because the disorder is, in practice, variably diagnosed in terms of subjective observation of symptoms, there is no reliable standard to which the CPT, or any diagnostic test, can be compared. Only trials that measure behavioral outcomes will be able to ascertain the clinical value of the CPT or any other diagnostic test for ADHD. Such a trial could be a randomized trial where the investigator applies a diagnostic algorithm with and without a CPT. The diagnostic classifications based on results of each group’s algorithm would then be compared with a gold standard diagnosis made by a consensus panel of clinical experts who observe the subject in the home and school settings and decide whether the DSM-IV criteria are met. Admittedly, this is a difficult study to perform, but it is the only way to definitively measure the effectiveness of the CPT. Indeed, this design is required to evaluate the effectiveness of any diagnostic test. No diagnostic test for ADHD has been subjected to such rigorous experimental scrutiny.

While a number of computerized CPTs are available, (55) the studies analyzed for this assessment used visual CPTs, including an “L” target and “A-X” sequence target CPT, the Gordon Diagnostic System (GDS), and the T.O.V.A. One auditory version of the CPT, the Auditory Continuous Performance Test (ACPT) was also analyzed in this assessment.

Competing/Complementary Technologies

Standard measures used in assessing ADHD include a diagnostic interview with the parent and/or child, completion of behavior rating scales by the parent and teacher, direct observations of behavior, and administration of clinic-based tests. A multidisciplinary approach to assessment is used to determine the presence and severity of ADHD symptoms across settings, tasks, and caretakers, and to rule out other conditions that may account for a child’s attentional problems (e.g., overanxious behavior, learning disability). (49) Many of these assessment tools have been shown to have high test-retest reliability and validity; however, discussion of such is beyond the scope of this assessment.

Despite the wide range of instruments, rating scales, questionnaires and surveys that have been developed to assess ADHD symptoms, neither the extant literature nor practice guidelines provide a clinical algorithm for how these instruments should be used and in what combinations. A cost-effectiveness analysis of the use of the standard diagnostic tools as well as the CPT to determine the most effective test battery is warranted.

Patient Populations

While ADHD affects people of all ages, this assessment focuses on studies of ADHD in children aged 6 to 13 years. Although our literature searches identified studies on the use of CPTs with preschoolers (Harper and Ottinger, 1992; (56) Byrne et al., 1998 (57) ) and adults,(Roy-Byrne et al., 1997; (58) Epstein et al., 1998 (59) ) we did not evaluate these reports in this assessment.

Study subjects have included normal populations of children from schools and neighborhoods. Clinical populations have come from referrals to pediatricians, child psychologists, child neurologists, outpatient clinics and other academic-, community-, or hospital-based programs.

Safety Issues

There are no known hazards associated with administering or performing CPTs for clinicians and researchers or study subjects, respectively.

Care Setting

CPTs are frequently administered in a university- or hospital-based research laboratory that typically is associated with an outpatient or inpatient clinic. For screening studies involving large samples of children from the general population, CPTs can be performed in the school setting. In clinical practice, patients can be tested as part of the initial multi-method assessment and/or monitoring of treatment. Again, these tests can be performed by clinicians in the practice setting or by investigators affiliated with the clinic (e.g., university-, hospital- or community-based). Universal Attention Disorders, Inc. (Los Alamitos, CA)~~CPT~~ offers a version of the CPT for administration in the home. This group developed the HomeTOVA~~can be administered at home is a version~~ ~~of the T.O.V.A.~~ that can be loaded onto a home computer and administered by parents. However, the manufacturer provides the disclaimer that “the HomeTOVA should not take the place of an evaluation done by a trained health care professional.” (60)

Analysis of the Currently Available Evidence

Key Questions

This assessment addresses four questions:

1. Does the peer reviewed published literature establish the reliability and validity of CPTs for the purpose of diagnosing ADHD?

2. Does the peer reviewed published literature establish the reliability and validity of CPTs for the purpose of titrating pharmacotherapy levels in treating patients with ADHD?

3. Do the published reports of national professional medical associations, national medical policy organization positions, or reports of national expert opinion organizations demonstrate consensus in the medical community that the safety and efficacy of CPTs are accepted for the purpose of diagnosing ADHD?

4. Do the published reports of national professional medical associations, national medical policy organization positions, or reports of national expert opinion organizations demonstrate consensus in the medical community that the safety and efficacy of CPTs are accepted for the purpose of titrating pharmacotherapy levels in treating patients with ADHD?

Evidence Base

Identification of Clinical Studies

To ensure that we comprehensively evaluated the literature for studies using the CPT in ADHD, the following databases were searched for relevant information:

The Cochrane Database of Systemic Reviews (through 2000, Issue 3)

The Cochrane Registry of Clinical Trials (through 2000, Issue 3)

The Cochrane Review Methodology Database (through 2000 Issue 3)

Current Contents (through September 2000)

The Database of Reviews of Effectiveness (Cochrane Library) (through 2000, Issue 3)

ECRI Health Technology Assessment Information Service Master Database
(through September 2000)

ECRI Library Catalog (through September 2000)

Embase (Excerpta Medica) (1975 through July 6, 2000)

ERIC (through July 2000)

Heath Care Financing Administration (HCFA) Website (through September, 2000)

Healthcare Standards (1975 through September 2000)

Health Devices Alerts (1977 through September 2000)

Health Devices Sourcebase (through September 2000)

Health and Psychosocial Instruments (HAPI) (through July 7, 2000)

HealthSTAR (Health Services, Technology, Administration, and Research)

(1975 through July 6, 2000)

International Health Technology Assessment (IHTA) (through September 2000)

MEDLINE (1964 through July 5, 2000)

National Guideline Clearinghouse (NGC) (through September 2000)

NHS Economic Evaluation Database (through July, 2000)

NIH Website (September 8, 2000)

PsycINFO (1990 through July 6, 2000)

TARGET™ (through September 2000)

U.S. Food and Drug Administration (FDA) Website (through September 8, 2000)

The search strategies employed a number of freetext keywords as well as controlled vocabulary terms including (but not limited to) the following concepts:

Controlled Trials: Randomized controlled trials; controlled clinical trials (MeSH heading, publication type, and textword); meta-analysis; random allocation; single‑blind method; double-blind method, evidence based medicine (includes randomized controlled trials, outcomes research, and meta‑analysis)

Technology: attention; attention test; continuous; continuous performance task; continuous performance test; test of variables of attention; TOVA;

Disease: attention deficit disorder with hyperactivity; attention deficit disorder; attention deficit; minimal brain dysfunction; ADD; ADHD; AD/HD

Reimbursement Information

The following web sites were searched for reimbursement policies:

Aetna US Healthcare (http://www.aetnaushc.com/cpb/cpb_alpha.html)

Allina Health System (http://www2.allina.com/hnf/mpol.nsf/index?openview&count=200)

Blue Cross/Blue Shield of Massachusetts (http://www.bcbsma.com/hresource/fs411.htm)

HCFA Coverage Issues Manuals (http://www.hcfa.gov/pubforms/06%5Fcim/ci00.htm

Humana (http://providers.humana.com/ciinter/cihome.asp)

Magellan Health Services (http://www.magellanhealth.com)

Hand Searches of Journal and Nonjournal Literature

In addition to searching Current Contents - Clinical Medicine on a weekly basis, over 1,600 journals and supplements maintained in ECRI's collections were routinely reviewed. Nonjournal publications and conference proceedings from professional organizations, private agencies, and government agencies were also screened. Other mechanisms used to retrieve additional relevant information included review of bibliographies/reference lists from peer‑reviewed and gray literature. (Gray literature consists of reports, studies, articles, and monographs produced by federal and local government agencies, private organizations, educational facilities, consulting firms, and corporations. These documents do not appear in the peer-reviewed journal literature.)

We identified 533 documents by the above-described methods. Of this total, we procured 302 articles for review. Not all of these articles were clinical trials; some documents were obtained for relevant background information. By applying the a priori inclusion criteria (see below) we excluded a large percentage of these articles from further analysis. This assessment is based upon the results reported and/or derived from 8 published studies.

Study Selection

To ensure this assessment was free from bias, we adopted a priori criteria for study inclusion for each of the two questions regarding diagnosis of ADHD and titration of pharmacotherapy. This approach differs from that typically employed in a narrative review where strict inclusion and exclusion criteria are not pre-determined. Adopting such criteria ensures that articles are not consciously or unconsciously chosen because their conclusions agreed with a preconceived idea about the effectiveness of the technology under consideration. Our a priori inclusion criteria for Question 1 were:

· Studies must have conducted a diagnostic classification analysis

Diagnostic classification analysis determines the ability of a test (e.g., CPT) or test component (e.g., omission errors, commission errors) to assign children to diagnostic categories that are then compared with the true state of the patient as measured by another independent criterion (i.e., a reference standard). (61,62) This study design enables the construction of 2 x 2 tables and the calculation of the efficacy indices: sensitivity, specificity, PPV and NPV. Thus, classification analysis is the most appropriate approach to determining the diagnostic utility of a test. Figure 2 is a schematic representation of a classification analysis.

Figure 2. Schematic of classification analysis

· Studies must have provided a detailed description of the independent criterion (reference standard) used to classify study subjects

Studies must include details regarding the independent, reference criterion used to define the true state of the patient and to which the test classifications were compared. This typically includes a series of assessments (e.g., clinical interviews, rating scales, behavior observations) designed to assess whether a child’s behavior meets the DSM criteria.

· Studies written in the English language

Our a priori criteria for the second question regarding titration of pharmacotherapy for ADHD were:

· Studies must have examined more than one dose of medication

To adequately assess issues relating to the titration of medication, studies must have examined more than one dose and compared the effects across doses.

· Studies must have used a CPT to evaluate the treatment effects on ADHD symptoms

· Studies must have used at least one independent criterion (reference standard) to evaluate medication effects on ADHD symptoms

To evaluate the utility of the CPT in diagnosing ADHD (as in Question 1) and titrating pharmacotherapy for ADHD, we must be able to compare the CPT results with a previously validated measure of ADHD symptoms.

· Studies must be controlled trials

· Studies written in the English language

Our a priori criterion for the third and fourth questions (those questions regarding possible consensus in the medical community on using CPTs for diagnosis and titrating pharmacotherapy) was all-inclusive. Specifically, we reviewed all position statements and clinical guidelines that addressed diagnosis and/or treatment of ADHD that were identified by our searches.

Study Characteristics

Eight studies met our a priori inclusion criteria for question 1. All eight studies used a version of the CPT to classify samples of referred children into ADHD and non-ADHD groups. The authors of these studies determined thresholds for performance measures (e.g., omission errors, commission errors, reaction time, etc.) prior to administering the CPT. Based on the results of the tests and using the selected thresholds, they classified children into ADHD and non-ADHD groups. These groupings were compared with the groupings arrived at using independent criteria (see Figure 2).

Two studies that met our initial inclusion criteria were subsequently eliminated from further analysis. Klee and Garfinkel (1983) combined data from 7 ADD patients and 23 CD patients and analyzed these data as if they were from a single group. (47) Because we could not separately ascertain the performance of the patients with ADHD, we excluded this study from our analysis. Levy and Hobbes (1997) developed a regression model for distinguishing ADHD patients from those without ADHD. (63) To this end, they used a retrospective sample of children with ADHD and matched controls. However, the authors measured the sensitivity and specificity of their model on the same subjects for which it was developed. This is problematic, for it is not uncommon for regression models to accurately describe the data from which they were developed, but to have little descriptive ability when the models are applied to data from different patient samples. The optimal approach is to develop a model on one group of patients and then validate its diagnostic utility on a separate group of patients. Levy and Hobbes did not employ this latter validating step; therefore, we excluded this study from our analysis.

The remaining six studies were included in our analysis for question 1 and are summarized in Table 2 . The studies are distinguished from each other in the text and tables by the presence or absence of spectrum bias. Spectrum bias occurs when one uses idealized populations with and without a given condition to determine the effectiveness of a diagnostic test. Use of such “ideal” populations causes overestimates of the sensitivity and specificity of the test because patients who are difficult to diagnose are not considered. The Matier-Sharma et al. study (28) contains spectrum bias because children who were judged by clinicians to have between five and seven ADHD symptoms (eight are required for diagnosis), were eliminated “to ensure a clear clinical distinction” between the ADHD and non-ADHD groups. That is, “borderline” ADHD patients were not considered as part of the diagnostic classification analysis. Furthermore, the authors were not explicit in their handling of 25 patients who did not meet the full diagnostic criteria for any of the non-ADHD diagnoses assessed. It is likely that these children were clinically “normal.” From the group numbers provided, it is also likely that these 25 patients were excluded from analysis.

The Barkley and Grodinsky (1994) study contains spectrum bias because the normal controls were selected only if: 1) they had no history of mental health services for behavior or emotional problems; 2) there were no parental or teacher complaints of behavior problems; and 3) they obtained scores below the 84^th percentile on the Child Behavior Checklist (CBCL) and the Child Attention Profile (CAP). The ADD+H children were eligible only if they scored above the 93^rd percentile on the CAP; ADD-H children were eligible if they scored above the 93^rd percentile on the Inattention scale and below the 84^th percentile on the Overactivity scale of the CAP. Finally, the LD children were eligible for study inclusion only if they had no teacher complaints of inattention, overactivity or impulsivity. Like the Matier-Sharma et al. study, (28) these criteria eliminate any difficult or borderline children and ensure a large difference between the groups. This defines spectrum bias.

Similarly, Greenberg and Crosby (38) excluded all subjects with co-existing psychiatric problems. Moreover, the authors used a normal control group that was not derived from the same population as the ADHD patients. Results from the Greenberg and Crosby (1992) study have been published in the T.O.V.A. manual; (64) the manuscript per se, has not been peer-reviewed to date.

We identified eight studies that met are inclusion criteria for Question 2. Rapport et al. (1987) examined the effects of five doses of MPH on the attention of children with ADHD in school and on the GDS CPT. Classroom measures included teacher ratings of behavior, behavioral observations by one of the investigators, and performance on academic assignments. (65) Barkley and colleagues examined the effects of stimulant medication in children with ADHD in a series of four studies. In all four studies, the drug responses were measured by using a battery of assessments that included parent and teacher ratings of behavior, direct observations by the investigators, and laboratory tests including the GDS CPT. In 1987 Barkley et al. assessed the effects of two doses of MPH in children with ADHD. (66) In 1988 the group examined two doses of MPH and the possible differential effects in children with aggressive or non-aggressive ADHD. (67) In 1991 they investigated three doses of MPH and effects in children with ADD with hyperactivity or ADD without hyperactivity. (68) Fisher and Newby (1991) refined the previously described assessment battery and examined the modified battery in clinic-referred children with ADHD. For this study, a larger sample of children was examined, two lower doses of MPH were used to reduce the occurrence of side effects, the Delay Task of the GDS was eliminated, and parents were not allowed in the playroom during behavioral observations. (69) Raymond et al. (1993) used the T.O.V.A. and the Connors Parent-Teacher Questionnaire, Abbreviated Form (70) to determine an optimal dose of MPH for children with ADHD. (71) Finally, Nigg et al. (1996) examined the effects of two MPH doses on “A-X” CPT performance in boys with ADHD. (72)

All eight studies examined different doses of stimulant medication and CPT performance. Each used an assessment battery that consisted of laboratory tests, behavioral observations, and academic performance. Some of these batteries might have been useful if they had components that corresponded with the attention and impulsivity factors that CPTs attempt to measure, and if the CPT scores were correlated with observational or academic assessments that purported to measure attention or impulsivity at appropriate drug dosages. However, with one exception, (72) these studies did not correlate CPT scores and observational assessments. Even in the Nigg et al. (1996) study, the observational assessments were for aggression and noncompliance; they were not for attention or impulsivity as measured by the CPT.

Because it was not possible to analyze the data from these eight studies or to derive conclusions about the correlations between drug dosage, CPT scores and a validated reference standard, we were unable to consider these studies further. Thus, we were unable to identify any appropriately designed studies that either supported or did not support the use of CPTs for titration of medication.

Key Outcomes

The efficacy of a diagnostic test is defined as its ability to indicate the presence or absence of a disease or condition. (75) Theoretically, if a test was perfect, it would correctly identify the “true disease state” 100 percent of the time. That is, a positive test result would identify all individuals with the disorder, or “True Positives” (TP; cell a in Figure 3 ). A negative test result would identify all those without the disorder, or “True Negatives” (TN; cell d).

The efficacy of a diagnostic test can be evaluated by comparing the number of correct classifications (i.e., true positives and true negatives) made on the basis of the test results with the actual number of patients in each group as determined by an independent diagnostic criterion. If this criterion was a true gold standard, the test could be found as good, but never better than the gold standard.

Efficacy is expressed as four indices: sensitivity, specificity, positive predictive value, and negative predictive value. Sensitivity answers the question: “If the patient has the disorder, how likely is s/he to have a positive test?” Sensitivity is the proportion of patients with the disorder who receive a positive test result [a / (a + c)]. Specificity answers the question: “If the patient does not have the disorder, how likely is s/he to have a negative test?” Specificity is the proportion of patients without the disorder who receive a negative test result [d / (b + d)]. (75) Sensitivity and specificity are the preferred measures for evaluating tests because they are independent of the prevalence of a condition in the study population.

In clinical practice however, clinicians are faced with the opposite task. (75) For example, a clinician is likely to ask: “If the patient has a positive test, how likely is s/he to have the disease?” or “If the patient has a negative test, how likely is s/he not to have the disease.? The indices that answer these questions are positive predictive value [PPV = a / (a + b)] and negative predictive value [NPV = d / (c + d)], respectively, and these expressions must not be mistaken for sensitivity and specificity. (76) PPV and NPV depend on not only sensitivity and specificity, but also on the prevalence of the disease [(a + c) / N] in the population examined. (75) Thus, the key outcomes in this assessment include measures of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). These outcomes are listed and defined in Table 3 .

Figure 3. Results of diagnostic tests

	Disease State
Test Result	Present	Absent	TOTAL
Positive	TP a	FP b	a + b
Negative	FN c	TN d	c + d
TOTAL	a + c	b + d	N
Sensitivity = a / (a + c)		Specificity = d / (b + d)
PPV = a / (a + b)		NPV = d / (c + d)

Table 3. Outcome measures

Reported Outcome	Definition
Prevalence	The proportion of subjects in a sample who have ADHD.
True Positive (TP)	A subject who has ADHD and was correctly identified as having the disorder by the CPT.
False Positive (FP)	A subject who does not have ADHD and was incorrectly identified as having the disorder by the CPT.
True Negative (TN)	A subject who does not have ADHD and was correctly identified as not having the disorder by the CPT.
False Negative (FN)	A subject who has ADHD and was incorrectly identified as not having the disorder by the CPT.
Sensitivity	The proportion of individuals with ADHD who are detected by the CPT as having the disorder [TP/(TP + FN)].
Specificity	The proportion of individuals without ADHD who are detected by the CPT as not having the disorder [TN/(TN + FP)].
Positive Predictive Value (PPV)	The proportion of those individuals identified by the CPT as having ADHD and who actually had the disorder [(TP/TP + FP)].
Negative Predictive Value (NPV)	The proportion of those individuals who were not identified by the CPT as having ADHD and who actually did not have the disorder [TN/(TN + FN)].

An efficacious diagnostic test will have high sensitivity, thereby maximizing the number of true positive classifications. The test also will have high specificity, maximizing the number of true negative classifications. However, there is a tradeoff between the sensitivity and specificity in that increasing the value of one will decrease the value of the other. For screening purposes, one should maximize specificity; for diagnosis, one should maximize sensitivity. One way to alter the levels of sensitivity and specificity is to raise or lower the threshold of the test. For example, if the threshold is set very high (e.g., children need to obtain very high scores on the CPT to be classified as ADHD), there will be relatively few test positives and more test negatives. This would have the desirable effect of decreasing false positives and increasing the number of true negatives, but would have the undesirable effect of increasing false negatives. Conversely, if the threshold is set very low (e.g., children can obtain lower scores on the CPT and be classified as ADHD), there will be many test positives and relatively few test negatives. This would have the desirable effect of fewer false negatives, but would have the undesirable effect of increasing false positives.

It is also important to consider the consequences of any errors made in classifying patients on the basis of test scores. We can assess this by calculating the false negative and false positive rates from the same 2 x 2 tables. To apply the tradeoff to the diagnosis of ADHD, we must ask what are the potential consequences of incorrectly diagnosing a normal child as ADHD (false positive). Possible harms include unnecessary treatments such as stimulant medication, behavioral therapy or special education programs. It would also be possible that parents, teachers and peers will falsely stigmatize the child. The potential consequences of failing to identify a child with ADHD (false negative) include not prescribing effective treatments and instead having continued disruptions at home or in school, poor academic performance and strained social relationships.

Because the consequences of false positives and false negatives differ from each other, it is not often the case that the ideal threshold of a diagnostic test is the threshold where sensitivity and specificity are equal. With decision modeling techniques, and the knowledge of the tradeoff between sensitivity and specificity, one can determine the threshold yielding the optimum utility. Development of such a model is outside the scope of this assessment; but selection of the optimum threshold is an important task in the development of a diagnostic test for clinical use.

Synthesis of Results

Question 1: Does the peer reviewed published literature establish the reliability and validity of CPTs for the purpose of diagnosing ADHD?

Table 4 presents the outcome measures from the five studies included in our analysis for Question 1. Prevalence, sensitivity, specificity, PPV and NPV were derived from data reported in the published articles.

Among the studies that did not have spectrum bias, Rielly et al. (1999) used the GDS CPT to evaluate and classify children with a preschool history of language disorders for the presence of ADHD. (73) The sample was drawn from children consecutively assessed in a diagnostic clinic for children with a preschool history of language disorders. The dependent measures obtained during the assessment were intelligence quotients, parent and teacher ratings of behavior, and the GDS. The reference standard used for ADHD diagnosis was the parent and teacher ratings. Because this reference standard is not perfect, the results of this study will underestimate sensitivity and specificity of the CPT. For classifying children based on CPT results, the authors used the 5^th and 25^th percentiles as thresholds for 11 possible outcomes on three GDS tasks. Only the results from the vigilance portion of the GDS are presented in Table 4 . For total commission errors from the vigilance portion of the CPT, sensitivity was 60% and specificity was 46% at the 5^th percentile; at the 25^th percentile these values were 88% and 23%, respectively. At the 5^th percentile PPV was 27% and NPV was 77%; for the 25^th percentile, PPV was 28% and NPV was 85%. The authors concluded that the GDS may have some clinical utility in ruling out a diagnosis of ADHD in children with a history of language disorders, but may not be sufficient to confirm an ADHD diagnosis in this population. (73)

In the Forbes (1998) study, participants were children referred to a private practice of clinical child psychology to determine if they had ADHD. The reference standard for ADHD diagnosis was based on five assessment procedures: 1) behavioral ratings by the mother; 2) behavioral ratings by the teacher; 3) parental reports of school problems; 4) parental reports of developmental and behavioral history; and 5) behavioral observations of the child during the initial interview. A diagnosis of ADHD was made only when three or more of the five assessments were in agreement. For classification of children based on CPT performance, Forbes studied four different thresholds based on recommendations in the T.O.V.A. manual. (64) The manual recommends that a diagnosis of ADHD should be considered when any one outcome (omission errors, response time or variability; commission errors are excluded) exceeds 1.5 standard deviations (SD) on the age-adjusted mean (denoted in Table 6 as “1 > 1.5 SD”), or when any two outcomes exceed 1.0 SD (“2 > 1.0 SD”). (32) . The additional thresholds of two outcomes exceeding 1.5 SD from the mean and all three outcomes exceeding 1.5 SD from the mean demonstrate the tradeoff between sensitivity and specificity. With the increased threshold, sensitivity decreased from 80% for one outcome greater than 1.5 SD to 13% for three outcomes greater than 1.5 SD, while specificity increased from 72% to 100% respectively. Using a threshold of three outcomes greater than 1.5 SD was too strict. The trade‑off for high specificity (100%) was low sensitivity (13%). Likewise, the two outcomes greater than 1.5 SD threshold also was too strict, as sensitivity was only 52%. If a clinician were to use these thresholds in practice, he or she would fail to diagnose many children with ADHD. Overall, the T.O.V.A. classification criterion of any one variable (not including commission errors) greater than 1.5 SD correctly identified 80% of the ADHD and 72% of the “Other” groups.

Table 4. Study results

Study	N	Threshold^a	TP (N)	FP (N)	TN (N)	FN (N)	Prev (%)	Sens (%)	Spec (%)	PPV^b (%)	NPV^b (%)
Included Studies
Rielly et al., 1999 (73)	148	5^th percentile 25^th percentile	15 22	40 57	34 17	10 3	25	60 88	46 23	27 28	77 85
Forbes, 1998 (32)	146	1 > 1.5 SD^a 2 > 1.5 SD 3 > 1.5 SD 2 > 1.0 SD	94 61 15 78	8 3 0 4	21 26 29 25	23 56 102 39	80	80 52 13 67	72 90 100 86	92 95 100 95	47 32 22 39
Oyler et al., 1998 (74)	23	Error scores Vigilance decrement	1 1	2 1	10 11	10 10	48	9 9	83 92	33 9	50 92
Included Studies with Spectrum Bias
Matier-Sharma et al., 1995 (28)	97	Inattention Impulsivity	28 9	28 7	29 50	12 31	41	70 23	51 88	50 56	71 62
Barkley and Grodinsky, 1994 (61)	47	Correct responses Commission errors Omission errors	11 8 11	1 0 1	22 23 22	13 16 13	51	46 33 46	96 100 96	92 100 92	63 59 63
Greenberg and Crosby, 1992 (38)	848	Discriminant analysis Equal weighting	25 23	60 22	331 369	12 14	9	66 62	85 94	29 51	97 96

a. Thresholds should be read as: any one outcome greater than 1.5 standard deviations (SD) above the norm and age-adjusted mean, etc.
b. PPV, positive predictive value; NPV, negative predictive value

Oyler et al. (1998) used the ACPT to evaluate a small sample of children for the presence or absence of ADHD and central auditory processing disorder (CAPD). (74) A chart review was conducted to identify children who had been evaluated for both ADHD and CAPD. A group of presumed normal controls was recruited from among the investigators’ children, their friends and the children of acquaintances. A developmental pediatrician diagnosed all children for suspected ADHD. The diagnosis was made using parental interviews, behavioral observations and DSM criteria. Children were classified as ADHD (i.e., ADHD and ADHD/CAPD children) or no ADHD (i.e., normal and CAPD only children). For classification based on CPT performance, the authors examined the results for two different ACPT performance measures. The total number of errors included both omission and commission errors; vigilance decrement was measured as the difference between the number of errors made during the first and last of six trials. The results showed moderate to high specificity levels: 83% for total errors and 92% for vigilance decrement. But sensitivity measures for both outcomes were less than 10% (see Table 4 ). Altering the threshold so more results would be called positive would increase sensitivity but decrease specificity, but there is not enough data in this article to permit us to measure this effect. The authors recommended that the ACPT not be used as a screening tool for ADHD in this group of children.

For the two studies that had spectrum bias, Matier-Sharma et al. (1995) used discriminant function analysis to assess the ability of objective measures of ADHD symptom dimensions to classify ADHD patients, non-ADHD patients and normal children. (28) The patient group consisted of consecutive, unmedicated referrals to a child psychiatric outpatient clinic in an urban medical center. The normal controls were recruited from a neighborhood school. As mentioned above, the authors excluded “borderline ADHD” children who displayed between five and seven ADHD symptoms (eight are required for diagnosis). This is exactly the definition of spectrum bias. The reference standard for diagnosis is the DSM-III-R criteria. For classification based on “A-X” CPT performance, the authors used a score for inattention and a score for impulsivity. Despite the presence of spectrum bias in this study, measures of sensitivity and specificity were not higher than those reported in the studies without spectrum bias. For the inattention score, sensitivity was 70% and specificity was 51%. For the impulsivity score, sensitivity and specificity were 23% and 88%, respectively (see Table 4 ). Thus, even under these “ideal” conditions where clinically distinct patient samples are diagnosed and difficult or borderline cases have been removed, the clinical utility of the CPT is marginal.

Greenberg and Crosby (1992) used four performance measures from the T.O.V.A. to classify children diagnosed with ADHD and normal controls. The diagnosticians were careful to rule out any children with co-existing psychiatric problems, such as depression, CD and ODD. Removing these children creates spectrum bias. Outcomes evaluated included omission errors, commission errors, mean correct response time, and response time variability. The diagnosis was made on the basis of history, psychiatric interview by a psychiatrist or psychologist, psychological testing (excluding the T.O.V.A.) and teacher behavior ratings. The authors used two alternative approaches to classify children as ADHD or normal: discriminant analysis and equal weighting. For both approaches, one half of the ADHD and normative samples are randomly selected and test scores are standardized (mean = 0; SD = 1). However, in discriminant analysis items are weighted based on the regression coefficients; in equal weighting, the standardized scores are summed across performance indices. Two distinct thresholds were determined for each method and then validated on the remaining half of the ADHD and normative samples. Only the results from the latter validation steps for each approach are presented in Table 4 . This avoids the bias that caused us to exclude the study by Levy and Hobbes. (63)

Again, despite the presence of spectrum bias, measures of sensitivity and specificity were not high in this study. In the discriminant function analysis sensitivity was 66% and specificity was 85%. For the equal weighting approach, sensitivity was 62% and specificity was 94%. Unlike the Forbes study, which reported high PPVs and low NPVs, Greenberg and Crosby (1992) showed the reverse, low PPVs and high NPVs. This is likely due to differences in the prevalence of ADHD between the two studies (i.e., 80% in Forbes, 1998 versus 9% in Greenberg and Crosby, 1992).

The prevalence of a disorder can affect the utility of a test, meaning that higher prevalence rates result in higher PPVs and lower prevalence rates result in higher NPVs. The highest prevalence rate for ADHD that we calculated was 80% for the Forbes (1998) study. (32) Likewise, the PPVs were also high (92%–100%). The lowest prevalence rate was calculated for the Rielly et al. (1999) study at 25%; the NPVs were also high (77% and 85%). The prevalence in these studies may approximate the prevalence in the referred clinical population. Aylward et al. (1990) reported a prevalence rate of 65% in a group of children referred for attention and behavior difficulties. This sample was considered typical of children who would be given computerized tests for ADHD assessment. (77) Thus, the PPVs and NPVs calculated for the studies analyzed in this report are likely to be similar to those obtained in the clinical setting.

Figure 4 is a ROC (receiver operating characteristic) plot of the sensitivities and specificities from the studies listed in Table 4 . This graph illustrates the relationship between sensitivity and specificity and facilitates the comparison of these values from the studies discussed above. Sensitivity is on the ordinate; specificity is on the abscissa. By convention, the specificity axis is inverted in a ROC plot. The “perfect” diagnostic test would appear in the upper left-hand corner of the graph, with values of 100% for sensitivity and specificity. On the graph, the dotted line represents chance levels of sensitivity and specificity. That is, the levels one would achieve if one were guessing a person’s true state.

The studies of Rielly et al., (73) Oyler et al. (74) and Matier-Sharma et al. have measures of sensitivity and specificity that fall on, or close to, the chance line. These results illustrate that, under the conditions in which they were tested, the CPTs are of minimal (if any) diagnostic utility. By contrast, the Forbes (32) and Greenberg and Crosby (38) studies have measures of sensitivity and specificity that are farther from the chance line. That is, under the conditions in which they were tested, their CPTs had higher diagnostic utility than those of Rielly et al., Oyler et al. and Matier-Sharma et al.

We can speculate as to the reasons why the results of these studies are so different. First, Oyler et al.(1998) (74) used an auditory rather than a visual CPT to classify children with CAPD according to the presence or absence of ADHD. It is possible that the auditory mode is not as effective in measuring inattention and impulsivity in children. In addition, children with CAPD may have other deficits that confound measures of attentional deficits associated with ADHD. In this case, performance on the CPT would not provide useful information for classifying these children by their ADHD symptoms. Likewise, Rielly et al. (1999) (73) examined ADHD in children with a history of language disorders. While children who could not understand instructions were excluded from the study, cognitive deficits associated with language disorders may confound measurement of attentional deficits associated with ADHD. Again, the CPT was not designed to measure deficits associated with CAPD. For the Matier-Sharma et al. (1995) study, (28) it is possible that the CPT performance scores used to classify children as ADHD or non-ADHD may not be as predictive as other standard CPT measures. That is, the investigators used an “inattention” score comprised of omission errors, long latency commission errors, and very long latency correct responses. The “impulsivity” score is comprised of fast “A-not-X” commission errors and slow “A-only” errors. This is the only study that used such performance scores.

Figure 4. Study results: ROC plot of sensitivities and specificities

The Forbes (1998) data demonstrate that, as with any other diagnostic test, selection of the threshold has an important effect on the performance of the CPT. Forbes (1998) was the only investigator who reported results at three or more diagnostic thresholds. As expected, his data points illustrate the trade-off between sensitivity and specificity (Figure 5 ). A line can be fitted to these points using the logit regression method of Littenberg and Moses. (78) Theoretically, any sensitivity/specificity combination on that line should be attainable by shifting the threshold separating positive from negative test results.

Figure 5. ROC plot of data from Forbes, 1998 (32)

One purpose of CPTs is their potential use in making a differential diagnosis. Thus, in ~~actual~~ practice, these tests might be used to attempt to distinguish between children who have ADHD and children with other disorders. Currently available literature does not allow one to address this issue. Thus, there is no study that contains a control group that mimics the non-ADHD population in which these tests are likely to be administered. Further, there ~~are~~ is an insufficient number of studies with different control groups to allow us to infer how such a “non-ADHD” population would perform on CPTs. This is illustrated in Table 5 , where important study characteristics are cross tabulated to identify studies with these characteristics in common.

There is very little overlap among the six studies (represented by 4) for either control group or type of CPT. We have discussed the problem of spectrum bias in relation to using normal children as controls. Results from studies involving restricted control groups such as children with language disorders or CAPD are not likely generalizable to a clinic-referred population of children. Moreover, very different CPT formats are used in these studies: The T.O.V.A. is a geometric visual CPT, the GDS uses numeric visual stimuli (e.g., “9-1”), the ACPT is auditory rather than visual, and the “A-X” CPT uses alphabetic visual stimuli. Another difference among these studies that is not depicted in Table 5 (but is provided in Table 2 ) is the reference standard used for diagnosis of ADHD with which the CPT classifications are calculated. Only two studies (28,74) use an edition of the DSM. Rielly uses parent and teacher ratings on the Disruptive Behavior Disorders Rating Scale; (73) Forbes uses parental interviews and a number of rating scales. (32) Greenberg and Crosby use patient history, psychiatric interview and psychological testing. (38) Given these differences we cannot derive firm conclusions about the diagnostic utility of the CPT in clinical practice.

Table 5. Pattern of evidence by control group and CPT type

Control Group	T.O.V.A.	GDS	ACPT	“A-X” CPT
Normal	4 (38)
Language Disorder		4 (73)
CAPD			4 (74)
Learning Disability		4 (61)		4 (28)
Any non-ADHD	4 (32)

Abbreviations: T.O.V.A., Test of Variables of Attention; GDS, Gordon Diagnostic System; ACPT, Auditory Continuous Performance Test; CAPD; Central Auditory Processing Disorder.

Question 2: Does the peer reviewed published literature establish the reliability and validity of CPTs for the purpose of titrating pharmacotherapy levels in treating patients with ADHD?

The ultimate question concerning drug dosage titration is whether there is a strong correspondence between drug dosage and ADHD symptoms as measured by clinical evaluation. Assuming that there is such a correspondence, the next question would be whether CPTs can act as a surrogate for this evaluation. If they could, then it would be more convenient and more objective to titrate drug dosage, either for groups or individuals, by using CPTs rather than by conducting lengthy interviews and/or completing rating scales and questionnaires. The usefulness of CPTs for this purpose depends on the diagnostic utility of the CPTs as a surrogate for these other assessment tools. Our analysis for Question 1 above indicates that the correspondence between CPTs and these reference standards in terms of sensitivity, specificity and predictive values is not high. Because of this, even if there was a strong correspondence between drug dosage and CPT score, there might be only a moderate correspondence, or a different degree of correspondence, between drug dosage and the appropriate behavior changes as defined by a reference standard.

We illustrate this point with the following example. Suppose we administer a CPT to a group of ADHD children and record their performance before and after drug treatment. In one half of the group, CPT performance improves but ADHD symptoms do not improve after treatment. In the other half of the group, CPT performance does not improve but ADHD symptoms do improve after treatment. When the group means are calculated for CPT scores and ADHD symptoms, the results show that the group has improved by a similar percentage on both measures, but actually, a different subgroup of children improved on each measure.

In the absence of studies that demonstrate a strong correspondence between CPTs and ADHD as measured by some independent criterion, the only type of study that can answer Question 2 is one with triple measures of drug dosage, CPT scores, and a reference standard for ADHD. Because our primary interest was the use of CPTs for titrating pharmacotherapy, we identified five studies (see Study Characteristics, page 14 ) that examined different doses of stimulant medication and CPT performance. The studies reported statistically significant effects of stimulant medication on group CPT performance. They also reported statistically significant effects of medication on group behavioral observations. However, as described in the above example, these correlations can only reveal that the direction of scores on a CPT or other test change in the same direction for the group. They do not provide information about whether an individual child who scores near normal on one test also scores near normal on a CPT. As such, the formal possibility remains that a child’s performance on CPTs and other tests may be independent. Addressing this latter issue requires that a study provides individual data or directly computes the correlation between a CPT score and a reference standard for ADHD.

Only one study calculated individual correlations between CPT scores and observational assessments. (72) But the observational assessments in this study were for aggression and noncompliance, not for attention or ~~compulsivity~~impulsivity as measured by the CPT used in the study. In this situation it is not clear that there should be any correlations; and indeed, the correlations observed between the CPT and observational assessments at various drug doses were low. Moreover, the degree to which the behavioral observations in this study correlate with DSM criteria has not been determined. Thus, these finding are difficult to interpret.

Question 3: Do the published reports of national professional medical associations, national medical policy organization positions, or reports of national expert opinion organizations demonstrate consensus in the medical community that the safety and efficacy of CPTs are accepted for the purpose of diagnosing ADHD?

Our searches identified five guidelines for diagnosis and management of ADHD. All of the guidelines advised against use of CPT scores alone for diagnosis of ADHD. Although some guidelines recognized some usefulness of the CPT during diagnosis and management, none specified any particular purpose for CPTs other than research.

In 1997, the American Academy of Child and Adolescent Psychiatry (AACAP) published practice parameters for assessment and treatment of ADHD based on a literature review and expert opinion consensus. (79) They stated that,

Computerized tests of attention and vigilance (CPTs) generally are not useful in diagnosis because they suffer from low specificity and sensitivity. They are useful, however, as research tools. Behavioral observations while performing the CPT discriminate ADHD children from other groups as well as or better than the CPT scores. The correspondence between impulsive errors on the CPT and behavioral impulsivity has not been established…CPTs are not consistently sensitive to stimulation effects. Also, task and contextual factors, such as the presence or absence of an adult, the instructions given to the patient, and the nature of feedback and contingencies, can substantially affect scores. Concerns have been expressed regarding commercial CPT products.

However, the AACAP guideline stated that, “An index combining characteristics of movement measured by infrared motion analysis and accuracy and variability of response on a CPT accurately distinguished boys with ADHD from normal controls.” However, the study (80) from which this conclusion was based had spectrum bias. Fifteen of the 18 ADHD children had comorbid diagnoses and the normal children had no behavioral or attentional deficits as revealed by clinical assessment. Moreover, the sample size was small (N=18) and the authors did not attempt to replicate the results of their discriminant analysis in an independent sample of children.

In 1998 the National Institutes of Health (NIH) issued a consensus statement entitled, Diagnosis and Treatment of Attention Deficit Hyperactivity Disorder (ADHD). (81) This was a consensus of non-advocate, non-Federal experts, based on presentations by investigators in the field and questions and statements from participants in a two-day public conference. The relevant statement of the consensus document was “Although an independent diagnostic test for ADHD does not exist, there is evidence supporting the validity of the disorder. Further research is needed on the dimensional aspects of ADHD, as well as the comorbid (coexisting) conditions present in both childhood and adult forms.”

In 1999 the Educational Testing Service (Princeton, NJ) posted on their Internet web site a Policy Statement for Documentation of Attention-Deficit/Hyperactivity Disorder in Adolescents and Adults. (82) This document was prepared by the Consortium on ADHD Documentation, for use by post-secondary education personnel, licensing and testing services, and consumers, to aid in determining appropriate accommodations for individuals with ADHD. The Consortium consisted of representatives of the Education Testing Service, Law School Admission Council, University of Tennessee-Memphis, National Board of Medical Examiners, University of Connecticut-Storrs, Dartmouth College, and Harvard University. This document stated that, “Test scores or sub-test scores alone should not be used as a sole measure for the diagnostic decision regarding ADHD. Selected sub-test scores from measures of intellectual ability, memory functions tests, attention or tracking tests, or continuous performance tests do not in and of themselves establish the presence or absence of ADHD.”

In January 2000 the Institute of Clinical Systems Improvement issued a Health Care Guideline on ADHD. (83) In regard to CPTs this guideline stated that,

Although these instruments appear to discriminate between children with ADHD and their normal counterparts at a group level, the usefulness of these measures in assessing individual children is limited. Due to significant false negative rates (estimated at 15-30%), these instruments are not considered pathognomonic of ADHD and are of limited utility in screening and evaluation. They are most useful in research settings and the complex individual patient where more extensive data may be useful.

In May of 2000 the American Academy of Pediatrics (AAP), Committee on Quality Improvement, Subcommittee on Attention-Deficit/Hyperactivity Disorder issued Clinical Practice Guideline: Diagnosis and Evaluation of the Child With Attention-Deficit/Hyperactivity Disorder. (22) The guideline was arrived at by consensus of experts after reviewing an evidence report prepared under the auspices of the Agency for Healthcare Research and Quality [AHRQ; formerly the Agency for Health Care Policy and Research (AHCPR)] by Technical Resources International (Washington, DC). In regard to CPTs this guideline stated that,

Several such tests [measuring vigilance or distractibility] have been developed and tested, but all of these have low odds ratios (all <1.2, equivalent to a sensitivity and specificity <70%) in studies differentiating children with ADHD from normal comparison controls. Therefore, current data do not support the use of any available continuous performance tests in the diagnosis of ADHD.

We also examined the AHRQ evidenced-based report, Diagnosis of Attention-Deficit/Hyperactivity Disorder, (1) that formed the basis for the AAP guideline. The AHRQ report only assessed studies making the non-specific diagnosis of ADHD compared to normal children. The authors claimed there was insufficient data for assessment of studies of CPTs used for specific or differential diagnosis comparing ADHD children to children with other disorders. While the authors pointed out that the normal-ADHD comparison is an “ideal” situation, they did not assess these studies for spectrum bias or exclude any studies on that basis. Also, they did not calculate sensitivities, specificities and predictive values for the studies. Instead, they compared the mean CPT scores of the normal and ADHD children and presented the differences between the means as effect sizes (differences between the means standardized to standard deviation units). It is possible to estimate sensitivities and specificities from these effect sizes; however, this requires certain assumptions, and is not as accurate or reliable as calculating the sensitivities and specificities directly from the data as we have done in our analysis. This report concluded that, even for these “ideal” comparisons of CPT scores of normal and ADHD children, the effect sizes were mostly less than 1.2, corresponding to sensitivities and specificities below 70%, which is not considered useful. Apparently the AAP committee misinterpreted the effect sizes in the AHRQ report as odds ratios; however, the conclusion of poor usefulness is the same. The authors believed that specific comparisons of ADHD children to those with other disorders would result in even lower effect sizes. This does not appear to us to be self-evident. Comparing children with ADHD to children with other disorders may increase or decrease the effect size, depending on the behavioral manifestations of the other disorder. If the other disorders involve more extreme behaviors than ADHD, it is also possible that CPTs could more easily discriminate between ADHD and the other disorders, than between ADHD and normal children. In either case, there are no data that address the issue of differential diagnosis. Thus, no evidence-based conclusion on this matter can currently be reached.

Question 4: Do the published reports of national professional medical associations, national medical policy organization positions, or reports of national expert opinion organizations demonstrate consensus in the medical community that the safety and efficacy of CPTs are accepted for the purpose of titrating pharmacotherapy levels in treating patients with ADHD?

Only one of the five guidelines made any reference to the use of CPTs in monitoring the effects of treatment or titrating optimal doses. In regard to this issue, the AACAP (79) stated that, “When used for assessment of medication efficacy, the applicability of (84) results to the patient’s natural environment is unproven or even absent.” This quote refers to the ecological validity of the CPT, which refers to the degree to which test performance in a laboratory is representative of behaviors that occur in more naturalistic (e.g., home or school) settings. The CPT has been shown to have low to moderate ecological validity. (85)

Economic and Regulatory Issues

Manufacturers and Costs

A number of CPTs are commercially available. Table 7 lists a representative sample of some CPT versions, their manufacturers and associated costs. Most of the test packages include the computer software, a technical manual and unlimited use of the test. The Gordon Diagnostic System (Gordon Systems, Inc.) is the most expensive ($1,595), most likely because a device (rather than just software) must be purchased. The Intermediate Visual and Auditory (IVA) CPT (BrainTrain), which was not used in the studies reviewed for this assessment, is also relatively expensive ($1,495). The T.O.V.A. is the least expensive ($295) commercially available CPT. The costs presented in Table 8 are for the hardware and/or software needed to run the CPTs. Not included are charges associated with personnel time required to provide instructions and supervise test administration.

FDA Status

To the extent that computer products used in medicine are intended to affect the diagnosis and treatment of patients, such systems are considered medical devices; thus, the FDA must provide reasonable assurance that these products are safe and effective. Computer products are subject to four levels of regulatory control; two of these levels describe exemptions from registration, listing, and pre-market notification.

Table 6. Commercially available CPTs

Test	Publisher / Manufacturer	Materials	Cost
Conners’ CPT II	Multi-Health Systems, Inc.	CPT program, manual (84)	$ 495^a
Gordon Diagnostic System (GDS)	Gordon Systems, Inc.	GDS III microprocessor-based portable unit, manual, interpretive guide, 50 record forms, 4 issues of ADHD/Hyperactivity newsletter, 1-year warranty (43)	$1,595
Intermediate Visual and Auditory CPT (IVA)	BrainTrain	IVA program, test and interpretation manual, 10 tests (86)	$1,495^a
Test of Variables of Attention (T.O.V.A.)	Universal Attention Disorders, Inc.	T.O.V.A. program, T.O.V.A. microswitch, EVE3 hardware, manual, video (87)	$ 295^a

a. Price reflects unlimited use of programs; for the T.O.V.A., there is an additional charge after the first five interpretations.

The Gordon Diagnostic System (GDS; Model 1) is the only CPT that has been cleared for marketing under the premarket notification [501(k)] process by the FDA. The GDS was approved in June, 1986. (88) Because personal computers were not readily available in the mid-eighties, the GDS was developed as a portable, single-component computer-based device, and was thus subject to the FDA’s medical device approval process.

Most other commercially available CPTs are software packages that can be loaded onto a personal computer. The FDA has only recently developed guidelines for the use of computer software in medical devices. (89) Given the FDA’s current policy and the intended use of CPTs, they are exempt from regulation as General Purpose Articles (21 CFR 807.65(c)). (90) These articles “either pose little or no risk, or are appropriately the sole responsibility of the health care professionals who have used them in medical applications.” CPTs also are exempt from regulation under the second level of regulatory control. Under Future Exemptions, manufacturers of computer products that are “intended to involve competent human intervention before any impact on human health occurs…will be exempt from registration…” (90) Thus, CPTs are exempt from FDA regulation because they pose no health risk, and because test results must be interpreted by a clinician before making diagnostic or treatment decisions.

Medicaid Coverage Policy

Medicaid has no national coverage policy for using CPTs in diagnosing ADHD. Coverage is at the discretion of the individual state’s programs and local carriers.

Third Party Payer Coverage

Blue Cross/Blue Shield of Massachusetts covers outpatient services for the diagnosis, assessment, and treatment of mental health disorders for, among others, the “identification of the presence or absence of a disorder, as outlined in the current DSM, including assessment of the acuity and severity of a condition.” Blue Cross/Blue Shield does not provide information on specific procedures or tests covered. (91)

Aetna US Healthcare covers Neuropsychological Testing (NPT) when provided to and in the diagnosis of “medical” and “mental” health conditions. NPT requested for mental health evaluation is covered through the mental health benefit. This coverage policy applies to fully insured Aetna US Healthcare HMO, POS and PPO plans and to all other plans unless a specific limitation or exception exists. (92)

Magellan Behavioral Health reviews mental health and substance abuse treatment for medical necessity before making coverage decisions. Magellan defines medical necessity as: “Services by a provider to identify or treat an illness that has been diagnosed or suspected.” Magellan’s National Provider Handbook does not provide details on the service options. (93) However, according to the Handbook, Magellan has adopted clinical practice guidelines developed by the American Psychiatric Association and encourages the use of these guidelines among its providers.

Conclusions

This assessment addresses four key questions:

1. Does the peer reviewed published literature establish the reliability and validity of CPTs for the purpose of diagnosing ADHD?

2. Does the peer reviewed published literature establish the reliability and validity of CPTs for the purpose of titrating pharmacotherapy levels in treating patients with ADHD?

To answer Question 1, we retrieved six studies that were of the design required to adequately address this question. Three of the studies had spectrum bias, which leads to overestimates of sensitivity and specificity. We included them in the analysis to see if they had any effect on the conclusions. Diagnostic classifications were made on the basis of CPT scores and then compared with an appropriate reference standard for ADHD. Reported sensitivities and specificities were highly variable and depended on the threshold levels used (as is the case with any diagnostic test) and on the individual study. Even with the studies with spectrum bias included, the CPT had low to moderate measures of sensitivity (range = 9% to 88%). The results with highest sensitivity (88% and 80%) did not have good specificity (23% and 72%, respectively).

Based on the results of this analysis, it does not appear that CPTs can be used as a stand-alone instrument for diagnosing ADHD. However, it is unlikely that the diagnosis of ADHD will ever be made on the basis of the results from one test or assessment procedure. In practice, diagnostic decisions are made by incorporating information and observations obtained from multiple sources and various settings (i.e., multiple diagnostic tests). No study to date has evaluated any multi-test diagnostic algorithm. In the absence of studies that report appropriate data, we cannot conclude that the CPT contributes additional and useful information in diagnosing ADHD. Nor can we conclude that the CPT does not.

To answer Question 2, we identified and reviewed eight studies that met our inclusion criteria. We were unable to demonstrate a strong correspondence between CPTs and ADHD as measured by the DSM-IV or clinical judgment. The only type of study that can answer Question 2 would be one that correlated measures of drug dosage, CPT scores, and DSM-IV evaluation (or a reference standard validated by the DSM). We found no suitable studies of this type. If other evidence establishing an association between CPT scores and ADHD diagnosis or other relevant outcome such as school performance is available, it may be possible to establish the validity of the CPT in titrating methylphenidate dose.

To answer Questions 3 and 4, we identified five guidelines that addressed the diagnosis of ADHD. All of the guidelines advised against use of CPT scores alone for diagnosis of ADHD. This concurs with our answer to Question 1. Although some guidelines recognized some usefulness of the CPT during diagnosis and management, none specified any particular purpose for CPTs other than research. Only one guideline that we identified addressed the issue of CPTs and medication for ADHD. The guideline questioned whether the behaviors measured by CPT tests were representative of behaviors likely to be expressed by children in more natural settings (e.g., home or school). The guideline concluded that the applicability of CPTs to monitor treatment for ADHD is “unproven or even absent.”

The CPT was not intended as a “stand alone” measure of attention deficits. Rather, the intent was to provide objective behavior-based information that can provide meaningful additions to a comprehensive, multi-modal battery of rating scales and interviews. (42,44) Currently, the published literature does not provide the data needed to determine whether the CPT provides useful additional information as part of a multi-test battery for diagnosis of ADHD.

Bibliography

1. Agency for Healthcare Research and Quality. Technical review: number 3. Diagnosis of attention-deficit/hyperactivity disorder (Full Report). Rockville (MD): AHRQ; 1999 Aug. 103 p.

2. Anderson JC, Williams S, McGee R, Silva PA. DSM-III disorders in preadolescent children. Prevalence in a large sample from the general population. Arch Gen Psychiatry 1987 Jan;44(1):69-76.

3. American Psychiatric Association (APA), Task Force on DSM-IV. First MB, editor(s). Diagnostic and statistical manual of mental disorders: DSM-IV. 4th ed. Washington (DC): American Psychiatric Press, Inc; 1994 Jan. 886 p.

4. Weinberg WA, Brumback RA. The myth of attention deficit-hyperactivity disorder: symptoms resulting from multiple causes. J Child Neurol 1992 Oct;7(4):431-45; discussion 446-61.

5. Weinberg WA, Harper CR. Vigilance and its disorders. Neurol Clin 1993 Feb;11(1):59-78.

6. Lambert NM, Sandoval J, Sassone D. Prevalence of hyperactivity in elementary school children as a function of social system definers. Am J Orthopsychiatry 1973;48:446-63.

7. Sandberg ST, Wieselberg M, Shaffer D. Hyperkinetic and conduct problem children in a primary school population: some epidemiological considerations. J Child Psychol Psychiatry 1980 Oct;21(4):293-311.

8. Wender P. Minimal brain dysfunction in children. New York: Wiley-InterScience; 1971. 242 p.

9. Wender PH, editor(s). Minimal brain dysfunction in children. New York: Wiley-InterScience; 1971. Prevalence and diagnosis of the MBD syndrome. p. 59-73.

10. American Psychiatric Association (APA). Diagnostic and statistical manual of mental disorders (DSM-II). 2nd ed. Washington (DC): American Psychiatric Association (APA); 1968.

11. American Psychiatric Association (APA). Diagnostic and statistical manual of mental disorders (DSM-III). 3rd ed. Washington (DC): American Psychiatric Association (APA); 1987. 494 p.

12. American Psychiatric Association (APA). Diagnostic and statistical manual of mental disorders (DSM-III-R). 3rd(rev.) ed. Washington (DC): American Psychiatric Association (APA); 1987.

13. Szatmari P, Offord DR, Boyle MH. Ontario Child Health Study: prevalence of attention deficit disorder with hyperactivity. J Child Psychol Psychiatry 1989 Mar;30(2):219-30.

14. Cohen P, Cohen J, Kasen S, Velez CN, Hartmark C, Johnson J, Rojas M, Brook J, Streuning EL. An epidemiological study of disorders in late childhood and adolescence I. Age- and gender-specific prevalence. J Child Psychol Psychiatry 1993 Sep;34(6):851-67.

15. Biederman J, Newcorn J, Sprich S. Comorbidity of attention deficit hyperactivity disorder with conduct, depressive, anxiety, and other disorders. Am J Psychiatry 1991 May;148(5):564-77.

16. Newcorn JH, Halperin JM. Comorbidity among disruptive behavior disorders: impact on severity, impairment, and response to treatment. Child Adolesc Psychiatr Clin North Am 1994;3:227-52.

17. Halperin JM, Gittelman R, Klein DF, Rudel RG. Reading-disabled hyperactive children: a distinct subgroup of attention deficit disorder with hyperactivity? J Abnorm Child Psychol 1984 Mar;12(1):1-14.

18. Early recognition of vulnarability (EREV). Hartford (CT): Connecticut State Department of Education; 1986. (Technical Report)

19. Lambert NM, Sandoval J. The prevalence of learning disabilities in a sample of children considered hyperactive.
J Abnorm Child Psychol 1980 Mar;8(1):33-50.

20. Loney J, Milich R. Hyperactivity, inattention, and aggression in clinical practice. In: Wolraich M, Routh D, editor(s). Advances in developemental behavior pediatrics. Vol. 3. Greenwich(CT): JAI Press; 1982. p. 113-47.

21. Epstein MA, Shaywitz SE, Shaywitz BA, Woolston JL. The boundaries of attention deficit disorder. J Learn Disabil 1991 Feb;24(2):78-86.

22. American Academy of Pediatrics. Clinical practice guideline: diagnosis and evaluation of the child with attention-deficit/hyperactivity disorder. Pediatrics 2000 May;105(5):1158-70.

23. Richters JE, Arnold LE, Jensen PS, Abikoff H, Conners CK, Greenhill LL, Hechtman L, Hinshaw SP, Pelham WE, Swanson JM. NIMH collaborative multisite multimodal treatment study of children with ADHD: I. Background and rationale. J Am Acad Child Adolesc Psychiatry 1995 Aug;34(8):987-1000.

24. Meents CK. Attention deficit disorder: A review of the literature. Psychol Sch 1989;26:168-78.

25. Achenbach TM. Diagnosis, assessment, and comorbidity in psychosocial treatment research. J Abnorm Child Psychol 1995 Feb;23(1):45-65.

26. Barkley RA, DuPaul GJ, McMurray MB. Comprehensive evaluation of attention deficit disorder with and without hyperactivity as defined by research criteria. J Consult Clin Psychol 1990 Dec;58(6):775-89.

27. Rosvold HE, Mirsky AF, Sarason I, Bransome ED, Beck LH. A continuous performance test of brain damage. J Consult Psychol 1956;22:343-50.

28. Matier-Sharma K, Perachio N, Newcorn JH, Sharma V, Halperin JM. Differential diagnosis of adhd: are objective measures of attention, impulsivity, and activity level helpful? Child Neuropsychol 1995;1(2):118-27.

29. O'Brien JD, Halperin JM, Newcorn JH, Sharma V, Wolf L, Morganstein A. Psychometric differentiation of conduct disorder and attention deficit disorder with hyperactivity. J Dev Behav Pediatr 1992;13(4):274-277.

30. Koriath U. Construct validity of clinical diagnosis in pediatric psychiatry: relationship among measures. J Am Acad Child Psychiatry 1985;24(4):429-36.

31. Greenberg LM, Waldman ID. Developmental normative data on the test of variables of attention (T.O.V.A.). J Child Psychol Psychiatry 1993 Sep;34(6):1019-30.

32. Forbes GB. Clinical utility of the Test of Variables of Attention (T.O.V.A.) in the diagnosis of attention-deficit/hyperactivity disorder. J Clin Psychol 1998 Jun;54(4):461-76.

33. Hinton VJ, Halperin JM, Dobkin CS, Ding XH, Brown WT, Miezejeski CM. Cognitive and molecular aspects of fragile X. J Clin Exp Neuropsychol 1995 Aug;17(4):518-28.

34. Cornblatt BA, Erlenmeyer-Kimling L. Global attentional deviance as a marker of risk for schizophrenia: specificity and predictive validity. J Abnorm Psychol 1985 Nov;94(4):470-86.

35. Keith. Auditory continuous performance test: Exam manual. San Antonio (TX): Harcourt Brace; 1994.

36. Sostek AJ, Buchsbaum MS, Rapoport JL. Effects of amphetamine on vigilance performance in normal and hyperactive children. J Abnorm Child Psychol 1980 Dec;8(4):491-500.

37. Krupski A. Attention problems in youngsters with learning handicaps. In: Torgesen JK, Wong BY, editor(s). Psychological and educational perspectives on learning disabilities. Orlando (FL): Academic Press, Inc; 1986. p. 161-92.

38. Greenberg LM, Crosby RD. Specificity and sensitivity of the test of variables of attention (T.O.V.A) [unpublished]. 20 p.

39. Impara JC, Plake BS, editor(s). The thirteenth mental measurements yearbook. Lincoln (NE): University of Nebraska Press; 1998. Test of variables of attention. p. 1058-62.

40. Anastasi A. Psychological testing. 6th ed. New York (NY): Macmillan Publishing Company; 1988. 817 p.

41. Halperin JM, Sharma V, Greenblatt E, Schwartz S. Assessment of the continuous performance test: Reliability and validity in a nonreferred sample. J Consult Clin Psychol 1991;3(4):603-8.

42. Seidel WT, Joschko M. Assessment of attention in children. Clin Neuropsychol 1991;5(1):53-66.

43. Impara JC, Plake BS, editor(s). The thirteenth mental measurements yearbook. Lincoln (NE): University of Nebraska Press; 1998. The Gordon Diagnostic System. p. 457-66.

44. Gordon M, Mettelman BB. The assessment of attention: I. Standardization and reliability of a behavior-based measure. J Clin Psychol 1988 Sep;44(5):682-90.

45. Prostate cancer treatment study gets under way. Hosp Technol Scanner 1994 Nov;13(14):13.

46. Halperin JM, Wolf LE, Pascualvaca DM, Newcorn JH. Differential assessment of attention and impulsivity in children.
J Am Acad Child Adolesc Psychiatry 1988;27(3):326-9.

47. Klee SH, Garfinkel BD. The computerized continuous performance task: a new measure of inattention. J Abnorm Child Psychol 1983 Dec;11(4):487-95.

48. Lovejoy MC, Rasmussen NH. The validity of vigilance tasks in differential diagnosis of children referred for attention and learning problems. J Abnorm Child Psychol 1990 Dec;18(6):671-81.

49. DuPaul GJ, Anastopoulos AD, Shelton TL, Guevremont D. Multimethod assessment of attention-deficit hyperactivity disorder: the diagnostic utility of clinic-based tests. J Clin Child Psychol 1992;21(4):394-402.

50. Sykes DH, Douglas VI, Morgenstern. Sustained attention in hyperactive children. J Child Psychol Psychiatry 1973;14:213-20.

51. O'Dougherty, Nuechterlein KH, Drew B. Hyperactive and hypoxic children: signal detection, sustained attention, and behavior. J Abnorm Psychol 1984;92:4-28.

52. Tarnowski KJ, Prinz RJ, Nay SM. Comparative analysis of attentional deficits in hyperactive and learning-disabled children. J Abnorm Psychol 1986 Nov;95(4):341-5.

53. Offord DR, Boyle MH, Racine Y, Szatmari P, Fleming JE, Sanford M, Lipman EL. Integrating assessment data from multiple informants. J Am Acad Child Adolesc Psychiatry 1996 Aug;35(8):1078-85.

54. Cohen ML, Kelly PC, Atkinson AW. Parent, teacher, child. A trilateral approach to attention deficit disorder. Am J Dis Child 1989 Oct;143(10):1229-33.

55. Kane RL, Kay GG. Computerized assessment in neuropsychology: a review of tests and test batteries. Neuropsychol Rev 1992 Mar;3(1):1-117.

56. Harper GW, Ottinger DR. The performance of hyperactive and control preschoolers on a new computerized measure of visual vigilance: the Preschool Vigilance Task. J Child Psychol Psychiatry 1992 Nov;33(8):1365-72.

57. Byrne JM, Bawden HN, DeWolfe NA, Beattie TL. Clinical assessment of psychopharmacological treatment of preschoolers with ADHD. J Clin Exp Neuropsychol 1998;20(5):613-27.

58. Roy-Byrne P, Scheele L, Brinkley J, Ward N, Wiatrak C, Russo J, Townes B, Varley C. Adult attention-deficit hyperactivity disorder: assessment guidelines based on clinical presentation to a specialty clinic. Compr Psychiatry 1997 May-Jun;
38(3):133-40.

59. Epstein JN, Conners CK, Sitarenios G, Erhardt D. Continuous performance test results of adults with attention deficit hyperactivity disorder. Clin Neuropsychol 1998;12(2):155-168.

60. HomeTOVA. Test of variables of attention [promotional material online]. HomeTOVA; [cited 2000 Apr 25]. [4p]. Available: http://www.hometova.com/HomeMain.cfm.

61. Barkley RA, Grodzinsky GM. Are test of frontal lobe function useful in the diagnosis of ADD. Clin Neuropsychol 1994;8:121-39.

62. Frick PJ, Lahey BB, Applegate B, Kerdyck L, Ollendick T, Hynd GW, Garfinkel B, Greenhill L, Biederman J, Barkley RA, McBurnett K, Newcorn J, Waldman I. DSM-IV field trials for the disruptive behavior disorders: symptom utility estimates. J Am Acad Child Adolesc Psychiatry 1994 May;33(4):529-39.

63. Levy F, Hobbes G. Discrimination of attention deficit hyperactivity disorder by the continuous performance test.
J Paediatr Child Health 1997 Oct;33(5):384-7.

64. Leark RA, Dupuy TR, Greenberg LM, Corman CL, Kindschi CL. T.O.V.A. test of variables of attention. Professional guide. Los Alamitos (CA): Universal Attention Disorders, Inc.; 1996 Aug 1. 105 p.

65. Rapport MD, Jones JT, DuPaul GJ, Kelly KL. Attention deficit disorder and methylphenidate: group and single-subject analyses of dose effects on attention in clinic and classroom settings. J Clin Child Psychol 1987;16(4):329-38.

66. Barkley RA, Fischer M, Newby RF, Breen MJ. Development of a multimethod clinical protocol for assessing stimulant drug response in children with attention deficit disorder. J Clin Child Psychol 1988;17:14-24.

67. Barkley RA, McMurray MB, Edelbrock CS, Robbins K. The response of aggressive and nonaggressive ADHD children to two doses of methylphenidate [published erratum appears in J Am Acad Child Adolesc Psychiatry 1990 Jul;29(4):670]. J Am Acad Child Adolesc Psychiatry 1989 Nov;28(6):873-81.

68. Barkley RA, DuPaul GJ, McMurray MB. Attention deficit disorder with and without hyperactivity: clinical response to three dose levels of methylphenidate. Pediatrics 1991 Apr 1;87(4):519-31.

69. Fisher M, Newby RF. Assessment of stimulant response in ADHD children using a refined multimethod clinical protocol. J Clin Child Psychol 1991;20:232-44.

70. Plake BS, Impara JC, editor(s). The supplement to the thirteenth mental measurements yearbook. Lincoln (NE): University of Nebraska Press; 1999. 411 p p.

71. Raymond N, Crosby RD, Corman CL, Greenberg LM. Determining optimal dose of methylphenidate [unpublished].

72. Nigg JT, Hinshaw SP, Halperin JM. Continuous performance test in boys with attention deficit hyperactivity disorder: methylphenidate dose response and relations with observed behaviors. J Clin Child Psychol 1996;25(3):330-40.

73. Rielly NE, Cunningham CE, Richards JE, Elbard H, Mahoney WJ. Detecting attention deficit hyperactivity disorder in a communications clinic: diagnostic utility of the gordon diagnostic system. J Clin Exp Neuropsychol 1999 Oct;21(5):685-700.

74. Oyler RF, Rosenhagen KM, Michal ML. Sensitivity and specificity of keith's auditory continuous performance test. Lang Speech Hear Serv Sch 1998;29(3):180-5.

75. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 1978 Oct 26;299(17):926-30.

76. Elwood RW. Clinical discriminations and neuropsychological tests: an appeal to Bayes Theorem. Clin Neuropsychol 1993;7:224-33.

77. Aylward GP, Verhulst SJ, Bell S. Individual and combined effects of attention deficit and learning disabilities on computerized ADHD assessment. J Psychoeduc Assess 1990;8:497-508.

Study	Patient Source	Age (years)	Sex (%male)	ADHD Group^a		Control Group^a	Reference Standard for Diagnosis^b	CPT type^c
Included studies
Rielly et al., 1999 (73)	Consecutive assessments in a clinic for children with a preschool history of language disorders	7-9	100%	ADHD		Language disorder	Parent and teacher ratings on the DBD scale; ADHD if 8 or more of 14 ADHD symptoms, ODD if 5 or more of 9 ODD symptoms, CD if 3 or more of 13 CD symptoms by either parent or teacher	GDS
Forbes, 1998 (32)	Referred to private clinical child psychology practice	6-13	75%	ADHD, ADHD+ODD/CD		OTHER (ODD/CD, LD, ED)	Agreement on 3 or more: parental interview, DCBRS / CBCL, RCTRS / ACTeRS, Parental report of school problems, PPVT-R and Bender-Gestalt	T.O.V.A.
Oyler et al., 1998 (74)	Chart review of patients evaluated for CAPD and ADHD normal children recruited from investigators, friends and acquaintances	7-11	65%	ADHD, ADHD+CAPD		Normal, normal+CAPD	DSM-III, DSM-III-R or DSM-IV criteria	ACPT
Included studies with spectrum bias
Matier-Sharma et al., 1995 (28)	Consecutive unmedicated referrals to child psychiatry outpatient clinic at medical center	6.5-13	74%	ADHD	Non-ADHD (CD, ODD, ANX, AFF)		CBCL, CTQ, computer algorithm based on DSM-III-R criteria	“A-X” CPT
Barkley and Grodinsky, 1994 (61)	Consecutive referrals to outpatient clinics at medical center	6-12	100%	ADD+H, ADD-H	LD referrals to pediatric or psychiatric clinic and newspaper ads		Some DSM-III criteria, CAP	GDS
Greenberg and Crosby, 1992 (38)	Not reported	6-13	85%	ADHD	Normal		History, psychiatric interview, psychological testing, CPTQ-A and ACTeRs	T.O.V.A.