By: Ruth Stoeckel, Ph.D., CCC-SLP
An important aspect of providing competent intervention is determining if a child is making adequate progress. We want to know whether a child’s speech-language skills are changing, to have an idea of the rate of change over time, and to develop evidence that whatever change is occurring is likely to be a result of our intervention efforts. If we see evidence of slowing rate of progress, we may want to consider changes such as increasing frequency or length of sessions or modifying the approach. If a child is close to achieving all of their goals, we may consider decreasing frequency or length of sessions as generalization is being facilitated to contexts outside of the therapy room. There will be times when it may be the goals rather than the approach that need to be modified, either to provide greater challenge for a child, or to adjust a goal that may have been too ambitious. These are all decisions that are best made based on data collected in conjunction with clinical observations.
The ASHA Code of Ethics tells us that we should evaluate the effectiveness of our services, and that we should not provide services without exercising our own independent, professional judgment. As helping professionals, we want children to improve in the skills we are teaching. We can certainly exercise professional judgment based on clinical experience and knowledge, but it is all the more powerful when we can back up those qualitative impressions with objective data. We cannot know for sure if what we are doing is effective without data. Having specific, measurable outcomes helps to reduce (not necessarily eliminate) the natural bias that we as clinicians have for seeing expected progress. Additionally, progress may not occur at a consistent rate, and we can use data to be sure that a child doesn’t get stuck at a plateau point.
In this climate of expectations for evidence-based practice, we have an obligation to provide services based on more than “I heard in school/at a workshop that this technique works” or “In my opinion, this intervention works.” Whether we are using approaches that have empirical support or basing our work on clinical experience, it is best practice to have clear documentation of what we are doing in intervention and how the child’s skills seem to be changing as a result.
When writing IEPs and clinical notes, we are encouraged to write objective, measurable, behavioral goals for treatment. That means identifying behaviors that more than one person would likely interpret in the same way, such as waving a hand in the air for “bye-bye” when leaving a room, or producing a target syllable or word to a specified level of accuracy. Measurement frequently involves teaching to a criterion level of performance, whether we are documenting improvement relative to baseline or looking at how often the behavior is produced accurately out of the total number of opportunities provided. Examples of increase relative to baseline for a language goal might be “increase MLU from 1.2 to 3.1” or “increase number of initiations with peers from 1 time per 20-minute free play session to 4 times per 20-minute free play session.” An example of comparison to baseline for a speech goal might be “increase intelligibility from 25% to 75% based on ratings of a spontaneous speech sample by naïve listeners.” Measuring performance in terms of accurate productions relative to total number of opportunities is often expressed in percentages. For example, a language goal saying that a child will produce present progressive tense sentences correctly 80% of the time in a structured picture description task. Or as an example of a speech goal, specifying that the child will produce a target sound with 80% accuracy in sentence-level practice.
Standardized tests are not the best way to measure gains in speech or language skills, and should never be used as the only metric of progress. Even the best tests provide a limited number of opportunities to evaluate a given behavior (e.g., a specific sentence form, linguistic concept, or speech sound). We may see a child making incremental progress (e.g. beginning to use a particular sentence form spontaneously, but not yet consistently; “marking” a consonant that they previously omitted) that can’t be captured by the format of a test. In addition, if we teach to a test, we may lose sight of the overall goal of helping the child to communicate more effectively. Changes in test scores do not automatically translate to changes in functional ability to communicate.
A powerful way to document progress and to demonstrate the probability that treatment is resulting in change is to develop sets of stimuli that are used as 1) treatment probes; 2) generalization probes (similar to target stimuli, but not the same); and 3) control probes (unrelated to treatment or generalization probes). We would expect to see the greatest gains in treatment stimuli, with a slight delay before seeing gains in generalization stimuli, and little or no change in the control probes. Let’s use an example of a child with a phonologic disorder who exhibits the process of stopping continuant sounds. You determine that the child is stimulable for /s/ and decide to make that the target. You choose specific words containing the /s/ sound for treatment. You choose words containing another continuant sound, such as /f/, for your generalization probes. You choose an unrelated sound, such as /r/, as your control. Every third or fourth session, you ask the child to produce the stimuli you have in each of those probe sets and document the accuracy of their productions. Let’s discuss four different outcome scenarios that might result from this type of data collection. In each of these scenarios, the child has been seen for around 30 total therapy sessions. The probe lists have been presented to the child every 3rd or 4th session, for a total of 10 probe sessions. There is some variability from one probe session to the next, but the overall trend is as described in each scenario. (Thanks to Benjamin Munson, Ph.D. at University of Minnesota for the basis for these scenarios)
In the first scenario, the child demonstrated consistent improvement in the target behavior (/s/), no improvement in the generalization behavior (/f/), and no improvement in the control behavior ( /r/). What do we infer from this? 1) The child is making progress in the target behavior, suggesting that treatment is effective. 2) The child did not make progress in the generalization behavior, and seems to be learning only what is taught. While the treatment is effective, we may want to consider how to increase effectiveness of therapy and to promote generalization, perhaps by calling attention to the feature of continuance. 3) No progress in the control behavior, as expected.
In the second scenario, the child improves in the target behavior shortly after treatment begins and shows improvement in generalization behaviors starting at the 3rd probe session, but shows no improvement in the unrelated behavior. What can we infer from this? 1) The child is making progress in the target behavior, suggesting that treatment is effective. 2) The child is making progress with the generalization behavior, which suggests that treatment effects are spreading, a desirable outcome. 3) No progress in the control behavior, as expected.
In the third scenario, the child shows improvement in the target, generalization, and control behaviors from the initial probe session through all 10 probe sessions. What can we infer from this? We cannot attribute progress to treatment effects, since all of the behaviors are improving. We have to consider the possibility of maturation or some other factor as the reason for change. We should probe additional sounds to determine whether the attention to these particular sounds was sufficient to promote change in a child with a relatively mild problem. We cannot attribute improvement to intervention without developing additional information.
In the fourth scenario, there is no progress in target, generalization, or control targets across probe sessions. In this case, intervention does not appear to be effective. We need to consider making changes. Options for modification could include utilizing a different treatment approach, or increasing frequency of sessions and intensity of intervention. Another option would be to determine whether goals need to be modified, choosing a different set of sounds that may lead to greater success for this child. In addition, we need to take a look at factors such as motivation and attention that may also be contributing to the lack of progress.
The above scenarios describe a situation with 10 probe sessions for convenience; it is hoped that in cases, such as the third and fourth scenario, that an alert clinician would begin making modifications sooner rather than later. Also, in each of these scenarios, clinical observations are very important. The child’s level of engagement is a qualitative observation that will help to guide what stimuli are used and how they are presented for maximizing learning for this particular child. Clinical observations combined with objective data forms a basis from which we can make appropriate decisions regarding intervention.
It is a challenge for those of us in clinical practice to collect data, given heavy caseloads and busy schedules. A system such as the one described above is not time-intensive endeavor, and can yield helpful information to inform treatment. Regardless of our practice setting, finding ways to collect objective data to document change as a result of therapy is a way to live up to the obligation we have as speech-language pathologists to provide competent, ethical services.
Featured Author: Ruth Stoekel, Ph.D. CCC-SLP
Dr. Ruth Stoeckel has worked in a variety of settings, including schools, private practice and clinic. She is currently employed at Mayo Clinic in Rochester, Minnesota. Her clinical interests include childhood apraxia/severe phonological disorders, autism, cochlear implants and early language development. She has presented at local training workshops, state associations and ASHA. Ms. Stoeckel is a member of the Childhood Apraxia of Speech Association’s Professional Advisory Board.