This month’s ASAT feature comes to us from Daniel W. Mruzek, PhD, BCBA-D, University of Rochester. To learn more about ASAT, please visit their website at www.asatonline.org. You can also sign up for ASAT’s free newsletter, Science in Autism Treatment, and like them on Facebook!
Marketers of purported interventions for autism spectrum disorder (ASD), whether they are pills, devices, or exercises, claim that their products are effective. As proof, they point to any number of measures: some valid, some questionable, and some potentially misleading. Given that many of these “treatments” may be costly, ineffective and even dangerous, it is good to consider what constitutes legitimate measures of therapeutic benefit. How will we know if the intervention actually works?
A first step when presented with a potential treatment option is to investigate its scientific record. One can certainly ask the marketer (or therapist, interventionist, clinician, etc.) for examples of already published and peer-reviewed studies examining the effectiveness of their recommended intervention. An honest marketer will be glad to give you what they have in this regard or freely disclose that none exist. A good second step is to consult with a trusted professional (e.g., physician, psychologist, or behavior analyst who knows your family member) to get an objective appraisal of the intervention. If, after this first level of investigation is completed, a decision is made to pursue a particular intervention for a family member, there are additional questions that one can ask the marketer prior to implementation. Such questions may prove very helpful in determining effectiveness after the intervention has been employed. These include the following:
Question 1: “What behaviors should change as a result of the intervention?”
Virtually any ASD intervention that is truly effective will result in observable change in behavior. For example, a speech intervention may very well result in increased spoken language (e.g., novel words, greater rate of utterances). An academic intervention should result in specific new academic skills (e.g., independent proficiency with particular math operations). An exercise purported to decrease the occurrence of challenging behavior will, if effective, result in a lower rate of specific challenging behaviors (e.g., tantrums, self-injury). As “consumers” of ASD interventions, you and your family member have every right to expect that the marketer will identify specific, objective, and measurable changes in behaviors that indicate treatment efficacy. Scientists refer to such definitions as “operational definitions” – these are definitions that are written using observable and measurable terms. If the marketer insists on using ill-defined, “fuzzy” descriptions of treatment benefit (e.g., “increased sense of well-being”, “greater focus and intentionality”, an increased “inner balance” or “regulation”), then “Buyer Beware!” These kinds of outcome goals will leave you guessing about treatment effect. Insist that operational definitions of target behaviors be agreed upon prior to starting the intervention.
Question 2: “How will these behavior changes be measured?”
Behavior change is often gradual and may occur in “fits and starts” (i.e., the change is variable). In some cases, the behavior may initially deteriorate. Also, our perception of behavior change can be impacted by any number of events (e.g., the co-occurrence of other therapies, our expectations for change). Therefore, it is the marketer’s responsibility to offer up a plan for collecting data regarding any change in the identified “target” behaviors. Usually, it is best to record numerical data (e.g., number of new words spoken by the individual, number of bladder accidents, duration [in minutes] of tantrums). The use of numerical data to measure the change of operationally defined target behaviors is one of the best ways for a treatment team to elevate their discussion above opinion, conjecture, and misrepresentation. If a pill, therapy, or gadget is helpful, there is almost assuredly a change in behavior. And, that change is almost always quantifiable. Setting up a system to collect these numerical data prior to the initiation of the new intervention is a key to objective evaluation of intervention. Don’t do intervention without it.
Question 3: “When will we look at these intervention data and how will they be presented?”
Of course, it is not enough to collect data; these data need to be regularly reviewed by the team! One of the best ways to portray data is “graphically”, such as plotting points on a graph, so that they can be inspected visually. This gives the team a chance to monitor overall rates or levels of target behaviors, as well as identify possible trends (i.e., the “direction” of the data over time, such as decreasing or increasing rates) and look for change that may occur after the start of the new intervention. Note that the review of treatment data is generally a team process, meaning that relevant members of the team, including the clinicians (or educators), parents, the individual with ASD (as appropriate) often should look at these data together. Science is a communal process, and this is one of the things that makes it a powerful agent of change.
An interventionist with a background in behavior analysis can set up strategies for evaluating a possible treatment effect. For example, in order to gauge the effectiveness of a new intervention, a team may elect to use a “reversal design,” in which the target behaviors are monitored with and without the intervention in place. If, for example, a team wishes to assess the helpfulness of a weighted blanket in promoting a child’s healthful sleep through the night, data regarding duration of sleep and number of times out of bed might be looked at during a week with the blanket available at bedtime and a week without the blanket available. Another strategy is to use the intervention on “odd” days and not use it on “even” days. Data from both “odd” and “even” days can be graphed for visual inspection, and, if the intervention is helpful, a “gap” will appear between the data sets representing the two conditions. These strategies are not complex, but they give the team an opportunity to objectively appraise whether or not a specific intervention is helpful, which is much better than informal observation. Few things are as clarifying in a team discussion as plotted data placed on the table of a team meeting.
If the marketer does not answer these questions directly and satisfactorily, consider turning to a trusted professional (e.g., psychologist, physician, or behavior analyst) for help. Families have a right to know whether their hard-earned money, as well as their time and energy, are being spent wisely. Asking these questions “up front” when confronted with a new intervention idea will help. Marketers have a responsibility to present their evidence – both the “state-of-the-science” as reflected in peer-reviewed research, as well as their plans to measure the potential effectiveness of their intervention for the individual whom they are serving.
Citation for this article:
Mruzek, D. W. (2014). ASD intervention: How do we measure effectiveness? Science in Autism Treatment, 11(3), 20-21
About the Author
Daniel W. Mruzek, PhD, BCBA-D has been a member of the faculty at the University of Rochester since 2002 and currently maintains an appointment in the Departments of Brain and Cognitive Sciences and Psychology. Also, he is CEO of Elevation Behavioral Services LLC, based at the Golisano Autism Center in Rochester, NY. As a psychologist and behavior analyst, Dr. Mruzek specializes in direct consultation and technical assistance to school districts and agencies through the region, nationally, and internationally. His areas of expertise include clinical and psychoeducational assessment of individuals with developmental and learning disabilities, assessment and treatment of challenging behavior, and promotion of inclusive practices (e.g., staff training and direct consultation to instructional teams). As a researcher, he has been the primary investigator and investigator on projects including the MCH Autism Intervention Research Program, the Adolescent Brain Cognitive Development (ABCD) Study, Seychelles Child Development Study, and the UR Intellectual and Developmental Disabilities Research Center (IDDRC). A member of numerous committees and professional organizations, Dr. Mruzek is on the board of reviewers for several peer-reviewed scientific journals. His research has been funded by the National Institutes of Health, and he has published chapters in various books and monographs and in peer-reviewed journals including The Journal of the American Medical Association, Autism, and Environmental International.