Evaluation of the STAR Training Initiative
and the Applicability of Its Content for Texas Adult Basic Education
by Kenneth L. Appelt, Donna Mancuso, and San Juanita Garcia (July 2009)
STAR -- STudent Achievement in Reading
Evidence-based Reading Instruction for Adults:
A Professional Development Pilot Study
Since the STudent Achievement in Reading (STAR) professional development training on evidence-based reading instruction for adults was launched in 2006, state directors of adult education have been encouraged by the Office of Vocational and Adult Education to participate. The STAR training is part of a larger Office of Vocational and Adult Education initiative called the National Diffusion of Reading Research. Because 56% of the students in Texas are placed in ESL classes, professional development leaders wondered whether the STAR training would be cost effective for Texas. However, there were nearly 23,000 ABE students (27%) in Texas during the 2006-2007 academic year with reading comprehension scores in the range that the STAR training addresses.
During the 2008-2009 academic year, the Texas adult education leadership chose to participate in the STAR series of trainings and conduct a pilot study to determine both the quality of the professional development design and how well the training meets the needs of teachers who teach students reading at a 4 to 8.9 grade level equivalent.
Research Questions
The primary research question was: How well does the STAR training assist teachers in implementing evidence-based reading instructional practices in the classroom? Subsidiary research questions were: 1) How useful and relevant are the materials used in the training? 2) Are there any additional costs for the program associated with STAR implementation? 3) What kinds of technical assistance and support will be required to help teachers fully implement evidence-based reading in their classroom? 4) How much of the training might be useful with reading instruction for ESL students?
Rationale and Significance of the Study
Intermediate level ABE students have difficulty transitioning to Adult Secondary Education. Texas data tables show that a significant number of students in Texas have difficulty improving reading skills beyond the intermediate reading level and cannot show progress. For example, in the 2006-2007 academic year, Texas had 12,855 students at the ABE intermediate-low level; of those, 39% completed the level. Only 32% of the 11,377 Texas students at the ABE intermediate-high level completed; this is the lowest completion rate for any level.
Improving the reading ability of these students is critical for the students and for adult education in Texas. If the students are going to reach their long-term goals of better employment, they will need to move beyond the intermediate reading level. Transitioning to Adult Secondary Education requires a higher level of reading for the more demanding assignments and for students to work independently. Transitioning to post-secondary training in certified workforce areas or enrolling in community colleges also require strong reading skills if the student is to be successful. Texas adult education programs must also improve student reading ability to improve completion rates in the intermediate-low and intermediate-high levels. The success of programs in Texas depends on it.
This pilot study was designed to monitor and observe the STAR training and technical assistance visits during the year and collect information from the participants as to the quality and usefulness of the training. Although the training is supported by a strong research base, state leaders wanted to be sure they understood the training requirements and demands of program implementation clearly before fully investing in this professional development program.
Review of the Literature
The professional development design of the STAR training draws on the results of the How Teachers Change study (Smith, 2003). This study found that administrators play an important role in how effectively teachers can make changes in their teaching approach and implement new teaching methods in their classrooms. Since the goals of the STAR training are the implementation of and the long-term sustainability of evidence-based reading instruction, administrators are included in the cohort. Smith (2003) also found it important for teachers to support each other through informal conversation, observation, and problem solving, so the STAR training encourages sending several teachers from the same program to facilitate collegial support. This, in combination with administrative participation, should increase the ability of programs to implement what is learned in the training sessions.
Three research publications support the content of the STAR training institutes. First, the Adult Reading Component Study (ARCS), led by John Strucker (2003), informs the diagnostic testing element of the STAR training. Strucker found that adult readers with the same score on conventional reading comprehension tests, such as the TABE, can actually have very different skill levels in word recognition skills, spelling, word meaning, and oral reading rate. These students, therefore, need instruction that is targeted based on the area of greatest need.
The second and third studies inform the instructional practices selected for inclusion in the STAR training: Research-Based Principles for Adult Basic Education Reading Instruction by John R. Kruidenier (2002) and Applying Research in Reading Instruction for Adults: First Steps for Teachers by Susan McShane (2005). These reports reflect a consensus among experts on the implications for instruction of the existing research on reading instruction for adults. Intermediate level adult students should have specific instruction that meets their differing needs, so the developers of the STAR training selected research-based instructional methods that have proven effective in developing the different skill areas.
Methodology
This research study collected pre-training and post-training data from the participants. The four open-ended questions of the Pre-training Survey (Appendix A) gave participants the opportunity to state their current classroom practices regarding reading instruction as well as any previous training in reading instruction. The Post-training Survey (Appendix B) gave all participants nine open-ended questions to answer on changes to their teaching practices, training specific content, and the applicability of the instructional methods with ESL students; there were two additional questions for administrators only regarding required implementation changes to the program and additional program costs. With these two measures, researchers hoped to get a sense of how much the participants knew about reading instruction before the STAR training and how much they learned during their participation and by implementing STAR in their programs. The surveys also contained questions that asked the cohort members about the usefulness of the testing procedures and teaching strategies with their students.
The Program Manager also observed the STAR training sessions and observed 50% of the technical assistance visits to programs. He was also in conversation with the STAR trainers on a regular basis to make arrangements for the training sessions and the technical assistance site visits. The trainers had good insight on implementing STAR and the difficulties that can arise from their personal experiences implementing STAR and training teachers within their own states.
Study Participants.
The Texas STAR pilot includes 13 programs and 3 professional development centers. STAR was introduced to program directors during the July 2008 Texas Adult Education Administrators Institute. We recruited programs willing to meet the STAR recommendations of closed enrollment, administrative participation, and commitment to implementation. The trainings were planned for Houston, so we recruited most heavily in that area; however, 6 of the 8 professional development service regions participants. Both large urban programs and smaller rural programs are represented. Some smaller programs that we tried to recruit did not participate because they could not see how they could have teachers attend three separate two-day trainings during the year without disrupting classes significantly. Some programs worried about how the travel expense would impact their professional development budget, others thought it may conflict with work on a Program Improvement Plan. Texas professional developers (GREAT Centers) were also recruited to become an integral part of ongoing support of implementation through technical support and participation in future trainings.
Programs Participating in the STAR Institutes
Dallas Independent School District - Dallas, Texas
Northeast Texas Community College - Mt. Pleasant, Texas
Tyler Junior College - Tyler, Texas
Educational Service Center Region 6 - Huntsville, Texas
Bryan Adult Learning Center - Bryan, Texas
Bowie & Cass County Adult Education Coop - Texarkana, Texas
Kilgore Junior College - Longview, Texas
College of the Mainland - Texas City, Texas
Harris County Dept. of Education - Houston, Texas
Coastal GREAT Center - Cypress-Fairbanks, Texas
Educational Service Center Region 20 - San Antonio, Texas
San Antonio Independent School District - San Antonio, Texas
South GREAT Center - Kingsville, Texas
Houston Community College - Houston, Texas
Southwest Texas Junior College - Uvalde, Texas
Overview of the STAR Institutes
The STAR training consists of three 2-day training institutes. The three STAR institutes for Texas took place in October 2008, January 2009, and April 2009. In the first institute, participants learn how to conduct diagnostic reading assessments which measure student skills in four of the major components of reading: alphabetics, vocabulary, fluency, and comprehension. They learn to create reading profiles that characterize the strengths and needs of each student and will learn how to target instruction to meet student needs. Participants begin work on teaching strategies to help develop fluency in oral reading. The first session also includes time to examine the existing program and classroom structures to determine any changes necessary to support implementation of these methods. Finally, both teachers and administrators work to develop a program action plan for implementation.
Immediately following the first institute, all participants were given access to the participant areas of the online STAR Toolkit. The toolkit has many resources to support and amplify the face-to-face training sessions. Many of the research documents mentioned during the training are there for downloading, consumable handouts and student tracking forms are available, and participant assignment materials are available. Videos of student-teacher interaction are also part of the toolkit; for example, videos of students reading at the four fluency levels are online to help participants calibrate their assessments of fluency skills. These can be reviewed periodically prior to testing sessions. Once trained, participants retain their use of the STAR Toolkit.
The second 2-day institute explores the evidence-based strategies for teaching alphabetic skills, vocabulary, and fluency. Participants learn how to plan lessons that include instruction and practice in these skills. Attention is paid to grouping students by their skill levels to work on specific areas of need. The program and state support available to effectively implement these teaching practices are evaluated. Then, implementation action plans for classroom instructors, local programs, and state level support are developed.
The final 2-day institute covers comprehension strategies and planning lessons that include comprehension instruction. Participants will discuss key elements of successful implementation and evaluate the status of implementation in their local program. Any questions they have about the entire training are answered. Then everyone sets goals and develops action plans to support implementation of evidence-based reading instruction for the classroom, the program, and at the state level. This session closes with a discussion of what must happen to sustain professional development in evidence-based reading instruction in Texas.
Between institute sessions, the national trainers returned to Texas for technical assistance visits in the programs. Between institutes 1 and 2 participants received email or telephone assistance. Following the second and third institutes, trainers tried to visit the sites of all programs requesting a site visit to observed how the program and teachers followed their implementation action plans. They met with teachers and administrators, answering questions, and offering suggestions to enhance implementation. Administrators and GREAT Center personnel were invited to “shadow” the trainers on these visits so that Texas’ professional development personnel will be able to provide technical assistance for teachers in the future.
Data Collection
The Pre-training Survey was given just prior to the first STAR training institute. Follow-up emails to those who had not yet turned in the Pre-training Survey eventually produced a 91% response rate. The Post-training Survey was distributed following the third STAR training institute. With follow-up emails, researchers obtained an 86% response rate.
The researchers compiled the responses from the participants to each of the survey questions. Then the responses to each of the questions could be examined and characterized question by question.
Pre-training Survey
Before attending the STAR training institutes, almost all of the participants claimed to use the comprehension measure of the TABE test for reading assessment. 43% of the participants mentioned using additional methods of measuring reading level – observing oral reading ability in class, using computer software of various brands, and making observations of the students as they work in class.
In terms of previous training in reading instruction, 20% of the participants had no reading specific training. Of the participants claiming college reading courses, 46% had K-12 certification in Texas, and 60% of the cohort mentioned professional development workshops.
When asked how they would help students who were having trouble decoding words, only 35% of the participants mentioned instructional strategies that could be classified as phonics or phonetic instruction. Many mentioned using dictionaries, determining the word from context, or using picture information to help determine a word, all of which are not direct instruction in alphabetics. Many stated that the struggling readers had phonics help through workbooks, computer programs, or one-on-one tutoring.
Only 35% mentioned specific comprehension strategies they taught to students. Many said that they would use follow-up questions to help students think about the text, show students how to use context clues, and assign vocabulary study.
Post-training Survey
Both administrators and teachers felt that the information gained through diagnostic testing is critical to planning appropriate reading instruction to meet the needs of each student. Four of the respondents had not implemented the diagnostic reading assessments in their program yet. 86% of the cohort responded that STAR improved reading assessment in their program, allowing teachers to plan instruction to needs of the student. Here is an example of the responses:
Before STAR, reading assessment in my classroom began with the TABE reading test. I further evaluated reading skill by listening to student reading and evaluating written responses to reading. Now I use the STAR assessment to break down reading skill into its four components. Knowing specific strengths and weaknesses allows me to guide students in exercising specific areas of alphabetics, fluency, vocabulary, and comprehension. (Respondent 3)
Changes in vocabulary instruction included focus on Tier II words, shorter word lists to allow depth of learning, more oral and written exercises using vocabulary words in different ways by adding suffixes to transform them into different parts of speech. Example:
First, STAR has improved my teaching of vocabulary by helping me understand the importance of vocabulary to intermediate readers. Second, the program has helped me to see the importance of Tier 2 words for GED instruction. Third, STAR has given me guidelines for delivering evidence-based instruction in vocabulary. (Respondent 3)
After the STAR training, teachers named specific strategies for teaching alphabetics and phonemic awareness to students who have difficulty identifying and sounding out words. They also mentioned sequences of instruction. Response examples:
This is where the training has opened my eyes to areas of instruction that I was missing in my classes. I would use the assessment to see where and how the student needed instruction, and begin to group students accordingly with appropriate assignments for their level, more one-on-one teaching as well as group instruction. (Respondent 15)
Through decoding, students are taught not to halt when they come to a new word in their reading text, but to practice decoding skills they are taught in class. Students are taught alphabetics/phonemic awareness. These skills help the student to "sound out" words if they do not know them on sight. Students realize and are taught that letters have sounds, how to connect letters to sounds, how to manipulate these sounds, and decoding rules, for example, silent letters, syllabification, prefixes and suffixes. (Respondent 16)
Responses to the question about teaching comprehension strategies were greatly different on the Post-training Survey. Most named several activities or teaching strategies they learned during the STAR Reading Institutes which they now use to help students improve reading comprehension skills. Some of the activities and strategies that respondents mentioned are: summarizing and questioning chart, fix-up strategies, graphic organizers, questioning strategies, and popcorn reading. Response example:
Comprehension is the most important component of reading. It is the product of reading; the understanding. Our program now incorporates graphic organizers into comprehension instruction. For example, Summarizing and Questioning Charts, K-W-L Chart (demonstrates the thought process through activating prior content knowledge), hands on activities that include graphic organizers in the Five Text Structures: Cause and Effect, Story structure, Sequencing, Compare and Contrast, and Description. Graphic organizers assist the visual learner. Also, instructors teach Fix-Up Strategies such as re-reading, vocabulary check, text aids, retelling, and asking someone for information. (Respondent 16)
When asked if any of the strategies learned in the STAR training would apply to ESL learners, most thought the alphabetic and vocabulary strategies would work well with ESL students and those who recently move into ABE classes. One of the respondents had concerns about the oral nature of the diagnostic testing and whether that would give useful results with ESL students who had limited speaking vocabularies. Examples:
The Alphabetics and Vocabulary are applicable to all ESL learners to some degree. Many ESL learners will automatically get the fluency and comprehension as soon as they get the foundation in Alphabetics and Vocabulary. I would consider giving the Sylvia Greene assessment to ESL learners as rule in order to check their understanding of the English phonetic system. I think all ESL students, regardless of the level, would benefit from focused instruction using the Alphabetic and Vocabulary strategies presented in the STAR materials. (Respondent 29)
Respondent 19 answered, “I tried to give diagnostic testing to ESL transition students and it was difficult. The oral vocabulary skill just wasn't there. Since most of the diagnostic testing is oral response, it was difficult, even though the students had scored Intermediate ABE in Reading. However, the instructional strategies will definitely be incorporated into ESL professional development. Learning the different reading components while in ESL will be useful as they transition to ABE.” This quote provides insight into how the instructional strategies taught in the STAR Institutes can better assist ESL learners as they transition into ABE.
Eight respondents to the Post-training Survey were administrators. Responses varied when asked about program changes necessary to implement STAR depending on the size of the program and its current organization.
I have not fully decided this. Presently we are working it with a pull out group on a volunteer basis. My plan is to train more teachers for next program year and make instructional strategies, especially vocabulary, fluency, and comprehension a part of the instructional program in all reading related classes at the intermediate level. I have yet to figure out how the more needy students who need intensive phonics or who are working at much lower levels than the other students will be scheduled at the sites where one teacher teaches all levels. (Respondent 1)
None…just basic modifications and budget allocations. (Respondent 5)
Implement Standard "Reading with Understanding" and introduce strategies involving vocabulary, fluency, and comprehension. Use graphic organizers and other techniques. (Respondent 7)
We do open enrollment every Monday. We will have to set up a time to go through the TABE tests and find the students that will be eligible for the program. We then have to call these students and make an appointment for them to come in and do the diagnostic testing. (Respondent 8)
We're going to have to train test administrators and implement a broader and more comprehensive assessment process. (Respondent 13)
When asked about any additional costs to implementing STAR in their program, 6 out of 8 administrators thought there would be some additional cost.
Certainly, I will need more staff to do the assessment as well as the instruction for those students who are working outside the norm whether it is higher or lower. We will also need more materials and more planning time for the teachers who are learning these methods and finding ways to implement the strategies into their instruction. (Respondent 1)
Yes, but we believe that there are many benefits to the program. We are already implementing many of the strategies program-wide. We just need to continue training ALL our teachers on reading strategies. (Respondent 5)
I hope to go through the GREAT center to reduce the costs of training. Extra-duty pay for instructor, if needed, and hiring a sub so the instructor can complete preparation. (Respondent 7)
Oh, yeah. The testing instruments, staff training time, and both administrative and instructional planning time are significant costs to the program. (Respondent 13)
We have several of the books already in our library that are being used for the STAR program in Illinois and California. To implement STAR reading in our program has not cost us anything except time and training. (Respondent 8)
We pay our instructors for professional development, which is included in our 2009-2010 budget. (Respondent 19)
