Evaluating Phonological Skills in Adult ESOL Learners

Part Two: Testing phonological skills in adult learners

Testing phonological skills in adult ELLs presents some challenges because it is different from testing monolingual children and adolescents. For these latter groups testing phonological skills is a straightforward process, with many tests commercially available for that purpose (e.g. The CTOPP --Wagner, Torgeson & Rashotte, 1999). There are also tests now available in other languages (e.g. Portuguese) (Santos & Bueno, 2003). Because of an assumption that adult learners already have most phonological processing skills in place, commercial tests are not generally designed to test weaknesses in these skills in adults. Moreover, the tests for children are predicated on learners’ having difficulty with literacy or language skills in some way, not on adult learners’ having low or no literacy skills.

Furthermore, testing phonological skills across languages is more complex than testing in the native language of the speaker. Though no lexical knowledge is needed to do the tasks on these tests well, and test items are seen to carry very little cultural bias in the items (Laing & Kamhi, 2003), native speakers of a language normally develop a knowledge of some other aspects of a language that can influence testing positively. These are factors such as the length of words used, complexity and placement of consonant combinations, familiarity with the construct of rhyme, the advantage of knowing the words that need to be blended or segmented, an awareness of the concept of phonemes. Not having this knowledge can negatively impact the effectiveness of phonological skills tests that were conceived in English when used with non-native English speakers (Wagner, R. personal communication, April, 2004).

However, in spite of those problems, evidence of the bi-directionality of phonological skills and more compellingly, evidence of the critical need to have phonological skills adequately developed to be able to learn a new language all point to the possibility and preference for testing new learners of a language in the language they need to learn.

The tool used for this project: The Assessment of Phonological Skills (APS)

The Assessment of Phonological Skills (APS) was developed for the purpose of evaluating phonological skills in adult ELLs. It was intended to meet several criteria.

First, the test needed to measure the core skills of phonological processing that have been found to be key to learning language and acquiring literacy. To assure this is what it would do, it was modeled on currently available commercial tests, but items are original in word choice and only core tasks measuring the two types of skills that are critical to adult foreign/other language learners were chosen.

Phonological awareness is measured on the APS by tasks of rhyme perception, counting of large chunks of sound—words and syllables—and a deletion task. Deletion (the act of removing a segment from a word and saying what is left: “bat” without /b/ = “at”) is considered a robust measure of phoneme awareness, a skill in English and other alphabetic languages that develops with literacy and critical to efficient reading and writing (Marshall, Snowling and Bailey, 2001). It was reasoned that ability on this task would normally correspond to a learner’s literacy level and phoneme awareness (if the learner was literate in an alphabetic language), while weakness in deletion, despite high levels of literacy in another alphabetic language, would point to difficulties in encoding and decoding English at the phoneme level. Weakness in this skill could also indicate unfamiliarity with the concept of phonemes as a result of being literate in a non-alphabetic language.

Phonological memory on the APS is measured by production of rhyme (having to hold the pattern in phonological memory and then produce a similar sound) and simple repetition tasks—sentences, words and nonsense words. Rhyme is seen as a critical skill for sensitivity to word segments (much of English is patterned on the onset-rime construction, e.g. c-at; gr-een) (Gathercole & Willis, 1991, Leong, 1999). Although for monolinguals, nonsense word repetition is considered to be an effective measure of phonological memory (Gathercole & Willis, 1991), two factors make it less compelling as a task for adult non-native speakers of a language: 1) Repeating words is rarely difficult for adults and 2) since nearly all words of a new language are in essence nonsense words for foreign language learners, repeating low frequency real words in the new language should be just as much a test of phonological memory as nonsense words. In fact, in terms of learning the language, weakness in reproducing real words would be of some concern. To test this theory, tasks of both real and nonsense word repetition are included on the APS.

A sentence repetition task was also added to test adults’ phonological memory on a more practical and challenging level. Adult language learners are expected to be able to understand and either repeat or respond to whole sentences in their classes and in their everyday lives. If they have difficulty repeating sentences of only four or five words accurately, their ability to absorb information both at the word and sentence level might be severely limited (McCollin & O’Shea, 2004). Native speakers of a language are able to repeat sentences of 15-20 words easily, and many can repeat even longer utterances well. An additional factor in the repetition tasks is that poor auditory discrimination of sounds in English could negatively impact the learner’s accuracy in perceiving words in sentences much more than it would word repetition. Consequently, the sentence repetition task has two purposes—first to measure what the learner can remember and second to indicate the need for direct instruction in the sounds and patterns of words in sentences to increase accurate input for learners.

A second criterion for the test was that it had to be brief enough to be administered easily within the context of adult ESOL programs, which is the intended audience of the APS. The APS currently consists of eight tasks, four each of phonological awareness and phonological memory and is intended to be administered in 15-20 minutes. It was anticipated that the results of this project would indicate which tasks could be eliminated, which needed to be lengthened and which were satisfactory as they were to be able to administer the test in that time. The rhyme tasks have 10 items in the current version, since experience has shown that it often takes that many items for a real trend towards understanding or not understanding to be discerned. Other tasks consist of five items, since it is easier to discern adults’ ability to do well on these in fewer items.

Another criterion was that the test had to reduce the advantage a native speaker would have on some of the tasks on commercial tests for English speakers. Rhyme is a new concept for many ELLs, so the rhyme items had to be very simple. The word counting and sentence repetition items needed to be constructed from vocabulary that is likely to be used in situations adult ELLs would encounter. The deletion task was constructed so that the the new word created after removing a segment or phoneme was still a real, and somewhat familiar, word. Some of these goals had to be balanced against demands of phonotactic differences—that is, the interference that is caused naturally by the difficulty of pronouncing sounds and sound combinations that are not in the learner’s language. This latter requirement continues to be adjusted, and finding out about the items on the current version of the test in terms of pronounceability was an intended outcome of the testing.

Fourth, the test had to be simple enough for low literate or non-literate learners to do easily. It is intended to measure core pre-literacy phonological skills, not school-based skills. Tests that are currently available for evaluating phonological processing skills are complex in nature and often consist of items with which children would likely have had experience in school. The intention of the APS was to make tasks as simple as possible so as to reduce bias from lack of experience with such tasks, and to make directions very easy.

Similarly, ease of administration of the test to those with very limited English skills was a goal. Items were generally designed to be as simple as possible so that directions would be limited and demonstration and modeling could aid in giving directions.

Finally, the test was intended to be a way to evaluate phonological skills, not vocabulary. It was important to know if items on the test were more or less difficult because the participants were attempting to understand the words or sentences and not just paying attention to sounds as it was intended they should.

Since skills cannot be tested if they are unknown, all skills on the APS are taught to learners before they are tested. Directions include explanations of tasks for the adminstrator to use and encouragement to do as much practice as seems necessary. If, when testing begins, the learner still appears to misunderstand, it is possible to start over and review the practice until understanding increases. If the learner does not understand at all, the item is skipped and targeted in the learning situation until such time as the learner has understood the task.

Scoring

There are no ceilings on the tasks, and only complete lack of understanding is a floor. Neither are there time limits on any task. Learners are encouraged to take their time, and some may require multiple examples or explanations before they can proceed with confidence.

Items are all scored as correct or incorrect; correct answers are tallied and skills are rated “good” (80-100% correct), “fair” (50-70% correct), to “poor” (< 50% correct). For the purposes of coming up with a global score in the two types of phonological skills, scores of tasks were weighted. (Poor = 1, fair = 2 and good =3.) Total scores of the four tasks in each type of skill ( phonological memory and phonological awareness) are added to create the global score. These totals are: 5 or under = poor; 6-9 = fair; 10 -12 =good. Thus, for example, a score of one poor (1), two fairs, (4) and a good (3) yields 8 points, or “fair” overall in one area (phonological memory or awareness). If a participant does not understand a task at all, it is marked as DU and scored in the poor range.

Using the scores         

Competent adult ELLs are expected to be able to do all tasks perfectly. Consequently, the intention of the scores is to show in which areas a learner has weaknesses. If a learner has skills that fall into the “fair” or “poor” range he or she will receive instruction targeting those skills. Learners having skills entirely in the “fair” or “poor” range would have most instruction pointed at strengthening phonological skills and auditory discrimination. For those not understanding a task, activities targeting that task would be explicitly included in instruction until the instructor was sure the learner understood the task and then it would be re-tested.

Next | Table of Contents | Previous