by: Alejandro Brice, Ph.D, CCC-SLP

Introduction
Pragmatics is the ability to use language in specific contexts and for specific purposes (Brice, 1992). School based speech-language pathologists are often faced with the task of differentiating between English learners (ELs) and bilingual students with language learning disabilities.  One method of achieving this outcome is to measure their pragmatic abilities, or social language use (Brice & Montgomery, 1996).  However, assessment of language, particularly language use in social interaction, is a difficult task.   In a review of currently available tests and procedures for use with students, Cummins (1984) stated that assessment must place “more emphasis on sensitive and informed interpretation of students’ behaviour” in everyday communication situations (p. 206).

SLPs who need to assess language use in social interaction are provided several options in determining a child’s language use skills.   Manners in which to assess language use in social interaction include (but are not limited to) the following:

1. Observing the child in naturalistic communication exchanges;
2. Interviewing others familiar with the child’s language use in social communication abilities (e.g., family members, teachers);
3. Using an existing inventory or checklist (i.e., a taxonomy of behaviors) that samples a valid representation of language use in social communication abilities. The behaviors may be observed in real-time or completed after the observation.  Real-time observations will be more accurate (i.e., valid), however, some behaviors may be overlooked or not exhibited while completing the inventory;
4. Administering a language use in social interactions test or subtest; and/or
5. Developing a new test or taxonomy of behaviors.

Observing the Child in Naturalistic Communication Exchanges
Qualitative research methods (Bogdan & Biklen, 1992; Glesne & Peshkin, 1992) indicate that observers may assume one of three roles:

1.  Non-participant observer.  This is more structured and more removed than that of participant observation.  Use of predefined categories of behavior (i.e., language use in social interaction taxonomies) for observations may be used.

2.  Participant observer.  The degree of participation is generally moderate with the observer maintaining some degree of detachment.  In child directed language samples, the degree of participation increases.

3.  Participant.  The clinician becomes totally involved with the clinical population to the extent that data collection may become difficult (e.g., becoming an advocate for a client).

Obtaining a language sample is also akin to observing a child in naturalistic settings.  Spontaneous language samples, historically, have been a staple of speech-language pathology clinical work (Bloom & Lahey, 1978; Fujiki & Brinton, 1987;  Fujiki & Brinton, 1983; Miller, 1981).   Highly structured samples are efficient and allow the clinician to collect language use in social interaction data in a relatively easy manner.  With more structured samples and interactions, the reliability of the data collected increases.  However, the variance or range of the data collected will be limited resulting in decreased variance or representativeness of the sample (i.e., validity).  As a consequence, most language samples have tended to be spontaneous and child directed.  The major disadvantages to obtaining language use in social interaction sampling data have been that it is time consuming to obtain and to analyze.  In addition, language samples are affected by the types of materials used, the elicitation procedures used by the clinician, the environmental context, the child’s demeanor, and other factors. Hence, many factors may affect the validity of the results.

Interviewing Others

Interviewing is key to clinical diagnostics for speech-language pathologists (Haynes & Pindzola, 1998).  Interviewing for assessment purposes is to obtain information about a child’s level and type of language use in social interaction functioning in classrooms, with peers, with teachers, and with others.  A good interviewer will ask open-ended questions, ask specific questions, and allow for topics to shift in order to obtain the fullest amount of information (Haynes & Pindzola, 1998).  Some errors to avoid in interviewing others are presented (adapted from Haynes & Pindzola, 1998):

1.  Avoid asking questions that will yield a “yes” or “no” response only;

2.  Avoid questions that lead the interviewee or inhibit freedom of responses;

3.  Avoid indirect questions. Please note that direct questioning is a cultural trait seen in individualistic cultures (e.g., North American) and typically not reinforced in collectivistic Latino, African-American, Asian, or Native-American cultures (Brice & Campbell, 1999);

4.  Avoid abrupt transitions when brining the interviewee back to the topic; and,

5.  Avoid obtaining superficial answers.

In conclusion, interviewing is a process skill that is usually learned with practice.  An interview is a purposeful interaction with someone else in order to obtain information.  Clinicians should prepare for interviews and allow appropriate time devoted to the task.  With culturally and linguistically diverse individuals more time may be needed as clinicians may need to adjust to different ways of obtaining information from the interviewee (Brice, 2002).

Using an Existing Inventor
The SLP may wish to adopt an existing inventory and observe the child in interaction (e.g., Dore, 1979; Halliday, 1978).  The SLP must elicit the sample in a manner that is qualitatively valid (i.e., an authentic or representative sample).  The reader is referred to language sampling procedures discussed by Miller (1981).  Use of existing inventories will be valid only if the clinician is completely familiar with the inventory and the behaviors being observed.  It is suggested that beginning clinicians establish their rating skills by obtaining high inter-rater reliability with a trained observer.  Issues of inter-rater reliability are discussed by Miller (1981).

Administering a Test
Many speech-language pathologists may choose to use a standardized, norm referenced or criterion referenced instrument.  In using a published test or protocol, the SLP must be knowledgeable of psychometric assessment criteria.  These include, but are not limited to knowledge of: (a) normative samples; (b) sample sizes; (c) item analysis; (d) instrument means and standard deviations; (e) concurrent validity; (f) predictive validity; (g) test-retest reliability; (h) inter-rater reliability; (i) test procedures; and (j) examiner qualifications (American Psychological Association 1979; Salvia & Ysseldyke 1981).  The main disadvantage to the use of tests is that they do not gather information in a naturalistic communicative environment, e.g., the classroom.  Consequently, formalized tests may not necessarily predict a child’s naturalistic language use in all social interactions (Evans & Craig, 1992).   In addition, tests may sample a limited repertoire of language use in social interaction behaviors (Evans & Craig, 1992) increasing sample reliability, yet, decreasing overall validity of the sample.  Please refer to Table 1 for a list of tests that assess language use in social interaction.  This table is not meant to be exhaustive, but rather to provide some examples of tests currently available that address language use in social interaction.

Table 1.

Available assessment instruments for measuring language use in social interaction

Developing a New Test or Taxonomy
The speech-language pathologist may develop a new test, protocol, or a new language use in social interaction taxonomy.  This approach is very time consuming; yet, it may provide an authentic sample of the behaviors to be elicited.   Any measure of language use in social interaction (e.g., a test, protocol or taxonomy of behaviors) should be developed according to a systematic process ensuring adequate reliability and validity.  One example is to construct the language measure using the guidelines set forth by Crocker and Algina (1986).  The following eight-step process for the development of a language use in social interaction measure follows:

1.   Identification of the primary purpose(s) for the language use in social interaction results;

2.   Identification of behaviors that define the domain (content validity);

3.   Preparing of instrument specifications, delineating the proportion of items that focus on each behavior as identified in step two;

4.   Construction of the initial pool of items;

5.   Reviewing and refining items as necessary and conducting preliminary item tryouts;

6.   Field-testing of the items on a large representative sample;

7.   Analyzing the statistical properties of the item scores with an elimination of items that did not meet the established criteria;

8.   Development of the guidelines for administration, scoring, and interpretation of the instrument behaviors.

Identification of the Primary Purposes for the Language Use in Social Interaction Results

      Most child language use in social interaction instruments will be used to screen, diagnose language use in social interaction problems, or gauge progress in language use in social interaction.  Therefore, the primary purposes will be for evaluation purposes.  In order to accomplish this evaluation, the child’s language will need to be assessed in social interaction.  Language use in social interaction may be assessed through the use of a test (e.g., The Test of Pragmatic Language, Phelps-Teraski & Phelps-Gunn, 1992), an observational scale (e.g., the Adolescent Pragmatics Screening Scale, Brice & Montgomery, 1996), or obtained through a language sample and use of a taxonomy (e.g., Dore, 1979; Halliday, 1978).

Identification Of Behaviors That Define The Domain (Content Validity)
Content validity can be established via three methods.  Initially a comprehensive review of the language use in social interaction use literature may be conducted in order to establish a list of topics and behaviors.  A draft of the test/protocol/taxonomy may then be constructed and sent to various reviewers, whereupon each is to make comments regarding suitability of the instrument and its items.  Finally, the instrument may used in preliminary try-out testing using professionals working with children.  After administering the test/protocol/taxonomy, these professionals may then make comments upon which revisions were made.

Preparing of Instrument Specifications
Each item should be defined as to weight of importance.  That is, should each item possess the same weight of importance or are some items deemed. to be more important and carry more weight in the assessment?   Most tests, scales, and taxonomies have been developed on the premise that no one language use in social interaction behavior in itself yields a normal or disordered classification in and of itself.

Construction of the Initial Pool of Items
One guideline in writing the items is that the individual language use in social interaction behaviors be objective and observable in settings where children use language.  A thorough and systematic review of the literature should provide an initial pool of items for the test/protocol/taxonomy. The language use in social interaction behaviors should be arranged according to a best fit classification and verified by an item-to-topic matching procedure, item analysis, and/or regression statistical procedure.  These procedures ensure that the items fit the language use in social interaction measure and are a good fit.

Reviewing and Refining the Items
All items should be reviewed by a panel of expert professionals knowledgeable about language use in social interaction.  These professionals may include researchers and scholars, speech-language pathologists working with children, and/or other professionals knowledgeable of language use in social interaction.  The experts should provide detailed feedback about the pool of items.  The items should then be modified according to the feedback provided.

Preliminary Item Tryouts
The test/protocol/taxonomy should be trialed with preliminary item tryouts.  A representative sample, for whom the instrument will be used, should serve as the initial group of participants.   The results from the trialing will indicate what items may need to be deleted, how the instrument should be administered, the time length for administration, etc.  In essence, this trialing should serve as a standardization for using the language use in social interaction instrument.

Filed Testing the Items on a Large Representative Sample
Once the test or taxonomy of items is refined, then the next step is to conduct large scale field testing.  Again, the participants in the field testing should match for whom the instrument is intended.  The sample size should large enough so that each item has sufficient discriminate power (i.e., can the items differentiate between children with language disorders versus children without language disorders?).  In addition, the items should possess sufficient power to minimize the internal validity effects of mortality (i.e., participants eliminated or dropped from the field testing).  Please refer to Hair (1998) as a reference in making this determination.

Analyzing The Statistical Properties of the Item Scores
Once the instrument has been tested through filed testing, then statistical analyses should be conducted to determine appropriateness of the items (often referred to goodness-of-fit) (Crocker & Algina, 1986).  This may be accomplished through the use of statistical procedures such as item analysis or regression.

Development of The Guidelines For Administration, Scoring, and Interpretation of the Instrument Behaviors
Guidelines for administering, scoring, and interpreting the language use in social interaction behaviors should be developed and discussed.

Conclusion
Assessment of pragmatics or social use of language in children is a task that takes concerted effort.  If conducted appropriately, it can yield valid and reliable measures of social language abilities.  Assessment of English learners suspected of a language learning disability may be evaluated using social language measures.  As with any language measurement, particularly of EL students, extra care must be taken to ensure accurate (i.e., valid and reliable results) measurement.  Speech-language pathologists must ensure administration of this care.

References

American Psychological Association. (1974). Standards for educational and psychological tests. Washington, DC: American Psychological Association.

Bloom, L. & Lahey, M. (1978). Language development and language disorders. New York: John Wiley and Sons.

Bogdan, R., & Biklen, S. K.  (1992).  Qualitative research for education.  Boston, MA:  Allyn and Bacon.

Brice, A. (1992). The Adolescent Pragmatics Screening Scale: A comparison of language-impaired students, bilingual/Hispanic students, and regular education students. Howard Journal of Communications, 4, 143-156.

Brice, A.  (2002). The Hispanic child:  Speech, language, culture and education.             Boston, MA: Allyn and Bacon.

Brice, A., & Campbell, L.  (1999).  Cross-cultural communication.  In R. Leavitt (Ed.), Cross-cultural health care: An international perspective for rehabilitation professionals (pp. 83-94). London,  England: W. B. Saunders.

Brice, A., & Montgomery, J. (1996). Adolescent pragmatic skills: A comparison of Latino students in ESL and speech and language programs. Language, Speech, and Hearing Services in Schools, 27, 68-81.

Crocker, L.,  & Algina, J.  (1986).  Process of test construction.  Introduction to classical and modern test theory.  (pp. 66-86) .  New York: CBS College Publishing.

Cummins, J.  (1984).  Bilingualism and special education : Issues in assessment and pedagogy.  San Diego: College-Hill Press.

Dore, J.   (1979).  Conversation and preschool language development.  In P. Fletcher & M.  Garman (Eds.), Language acquisition (pp. 337-361).  New York: Cambridge University Press.

Evans, J. L. & Craig, H. K.  (1992).  Language sample collection and analysis: Interview compared to freeplay assessment contexts.  Journal of Speech and Hearing Research, 35(2),  343-353.

Fujiki, M. & Brinton, B. (1983). Sampling reliability in elicited imitation. Journal of Speech and Hearing Disorders, 48, 85-89.

Fujiki, M. & Brinton, B. (1987). Elicited imitation revisited: A comparison with spontaneous language production. Language, Speech, and Hearing Services in Schools, 18, 301-311.

Glesne, C., & Peshkin, A.  (1992).  Becoming qualitative researchers.  White Plains, NY: Longman.

Hair, J.F. (1998).  Multivariate data analysis (5th ed.).  Upper Saddle River, NJ: Prentice Hall.

Halliday, M.  (1978).  Language as social semiotic.  Baltimore, MD: Edward Arnold.

Haynes, W.  O. & Pindzola, R. H.  (1998).  Diagnosis and evaluation in speech pathology (7^th ed.).  Boston, MA: Allyn and Bacon.

Miller, J., F.  (1981).  Assessing language production in children: Experimental            procedures.  Baltimore, MD: University Park Press.

Phelps-Teraski, D. & Phelps-Gunn, T.  (1992).  Test of Pragmatic Language (TOPL).         San Antonio, TX: The Psychological Corporation.

Salvia, J., & Ysseldyke, J. E. (1988). Assessment in special and remedial education (3rd ed.). Boston:  Houghton Mifflin.

This Month’s Featured Authors:  Alejandro Brice, Ph.D., CCC-SLP University of South Florida St. Petersburg

Dr. Alejandro E. Brice is an Associate Professor at the University of South Florida St. Petersburg in Secondary/ESOL Education. His research has focused on issues of transference or interference between two languages in the areas of phonetics, phonology, semantics, and pragmatics related to speech-language pathology. In addition, his clinical expertise relates to the appropriate assessment and treatment of Spanish-English speaking students and clients. Please visit his website at http://scholar.google.com/citations?user=LkQG42oAAAAJ&hl=en or reach him by email at [email protected]

dup 1419