Using Evaluation Data to Document the Value of Even
Start Family Literacy Programs

Chia-Yin Chen, Don F. Seaman, Yvette D. Perry

Introduction

The importance of involving the family in educational endeavors has become the focus of much attention in recent years. The impact of educating parents and children together has been discussed in the literature for well over a decade, with most of the authors indicating that "intergenerational" or "family" literacy enhances the learning of both parents and children in ways greater than educating them separately (Field, Widmayer, Stringer, and Ignatoff, 1982; Nickse, 1989; Weiss, 1988). Despite this, several years of isolated efforts took place before Even Start Family Literacy legislation was enacted as part of the Elementary and Secondary Education Act in 1989. Questions regarding the value and accomplishments of the Even Start program arose soon after. Were the participants’ goals and expectations being met? Were participants’ lives improving as a result of participation in Even Start? Was the intent of the legislation being met? If so, what evidence existed to support those conclusions? Answers to those questions have become especially important since recommendations materialized in Washington, D.C. in 2002 to reduce the amount of federal funding for Even Start programs.

Good program evaluation empowers local literacy providers by providing them with a sense of direction, continuous assessment of program progress, a focus for program development, and a basis for making decisions about future program changes. Yeh (2000) points out that program evaluation is a way to strengthen many aspects of a program. When a program becomes strengthened through effective evaluation activities, it can produce the results that help policy-makers, the public, and the participants themselves perceive the program to be worthy of support and continued funding.

Although there is limited research data to substantiate positive outcomes of any great magnitude, a brief overview of early Even Start evaluations can provide a foundation upon which to build current efforts. Early evaluations suggested that Even Start programs produce positive outcomes, but only documented short-term outcomes. One study combined quantitative and qualitative research procedures to identify the effects that the Kenan Model of Family Literacy was having upon participants (Seaman, 1992). The short-term outcomes indicated that the lives of the parents and children in these states were positively affected by their program participation, particularly in regard to parent-child interactions in the home. However, as this study primarily produced qualitative data, generalization to other programs was not possible.

A later research study showed that, with certain limitations, literacy can be transferred from parents to children within almost any cultural setting (Snow and Tabors, 1996). Data showed that regardless of the social class differences of the families, transfer of knowledge can be attained at similar rates. This supports one of the goals of Even Start programs: to serve those families “most in need” of the services available.

Researchers also have emphasized the importance of data collection with the implication that practice can be improved through quality evaluation and related research (St. Pierre, et al., 1995; Powell, 1996). St. Pierre and Layzer (1996) indicate that only high quality services can produce large effects upon family literacy participants. They cite data from the Perry Preschool Program and the Infant Health and Development Program to document those effects, and infer that without good evaluation and research procedures, such documentation cannot be acquired.

The evaluation requirements for Even Start family literacy programs demonstrate the concern of policy-makers for information and data to show that their funds are being used properly (i.e., that Even Start programs are meeting the intent of the federal No Child Left Behind legislation and the accompanying appropriations). In response to this call for accountability, programs must demonstrate compliance. They also must show that they are meeting the needs of the participants, while meeting the requirement of “continuous improvement” through the use of data to improve program operations constantly and consistently.

Staff members at the Texas Center for Adult Literacy and Learning (TCALL) have been evaluating local Even Start programs since 1992. We have observed numerous ways in which program coordinators have been able to utilize the information that has been derived from evaluation activities (data from both standardized instruments and alternative assessments) to improve their programs. Although there is much disagreement about the value of using standardized tests or alternative assessment tools for assessing literacy skills in family literacy programs, our purpose is to share how both formal and informal assessments can be used to gauge overall program quality. Effective evaluations utilize both kinds of assessment (Holt & Van Duzer, 2000). Data from alternative assessment procedures may support findings from standardized instruments and provide explanations for the results while providing the “in-depth” information that enables staff to determine more clearly how to improve a program.

Using standardized assessment instruments

In Texas, selected standardized assessment instruments such as the Pre-school Language Scale, Third Revision (PLS-3), Basic English Skills Test (BEST), and Test of Adult Basic Education (TABE) are required by the state to determine literacy skills improvement in both children and parents. This enables State Even Start staff to (a) attain a statewide assessment, (b) compare programs, and (c) report common data to policy-makers and legislators at the state and federal levels. All of those actions also enable state-level staff to identify areas for improvement in local programs.

For children, the most commonly utilized test is the PLS-3 which assesses receptive and expressive language skills in infants and young children. Data from the PLS-3 can provide an example of how test data may be used to suggest modifications in the program’s emphasis, when appropriate. When data from pre- and post-tests on the PLS-3 are analyzed, data on two sub-scales are provided: Auditory Comprehension and Expressive Communications (plus a combination score from both). If a child’s Auditory Comprehension score reflects a statistically significant difference between the two pre- and post-test scores, he or she has demonstrated an understanding of the language and directions from the program staff. However, a slight difference in pre- and post-test scores on Expressive Communications indicates that the child is not expressing him or herself at a satisfactory level, despite understanding what is being said. Therefore, evaluators will indicate to program staff that more time and emphasis should be given to the children for self-expression as shown on the formal assessment. This can lead to improvement in the children’s learning and will produce the kind of data that will show program improvement as measured by the PLS-3. This in turn will enable Even Start programs to demonstrate the effectiveness of the literacy program for children.

The Basic English Skills Test (BEST) is used to assess parents learning English as another language. The purpose of this instrument is to assess speaking, listening comprehension, reading, and writing skills. The BEST test provides data on five sub-scale scores: listening, communications, fluency, reading, and writing. By analyzing pre- and post-test scores in these areas, evaluators can suggest program changes to staff. For example, in one program, the pre- and post-test analysis showed significant differences in reading, but small differences in writing. Therefore, the evaluation team advised the program staff to focus more on developing the parents’ writing skills in English. The next year, the differences in the writing scores were significant. Similar to children, as parents acquire needed literacy skills and begin to utilize those skills in their daily lives, they become “living proof” of the value of the Even Start program and this can be communicated to the various publics as needed.

Program evaluators in Texas also frequently examine the relationship between attendance and test performance to determine whether the amount of time a parent participates in classes is related to how the individual performs on a specific test (e.g., the TABE). The TABE is the most widely used assessment test of reading, mathematics, language, and spelling for adults and older youth. It determines grade equivalency in each area and directs accurate prescriptive instruction to meet student needs. However, as with any assessment instrument, care should be taken in reviewing and discussing those results.

For example, if teen parents who have recently left high school are part of the class, our TCALL evaluation team has found that they often influence the assessment results. Since they have recently been attending school, teens often can test higher and may pass the GED test after spending a relatively short time in the program. Older adults who have been away from school for some time (perhaps more than a decade) usually will require a much longer time to improve their test scores. Therefore, the low correlation that may be produced in this case has little to do with the quality of the instructional program, but is instead due to the unique enrollment in the program. Another factor to consider is the literacy level of the parents. Data from the programs evaluated by TCALL staff usually show that parents who have been assessed at lower literacy levels seem to progress more slowly than those who are at higher literacy levels.

Measuring increases in parenting skills generates some debate among family literacy providers because these skills are particularly difficult to assess on any one instrument. There are several different standardized assessment instruments being used to assess parenting skills, e.g., The Home Observation for Measurement of the Environment (HOME) Inventory and the Adult and Adolescent Parenting Inventory (AAPI-2.) HOME is used to assess aspects of the environment such as: frequency and stability of adult contact, amount of developmental and vocal stimulation, need gratification, emotional climate, avoidance of restriction on motor and exploratory behavior, types of play materials available, and home characteristics indicative of parental concern with achievement. AAPI-2 is used to assess the parenting and child rearing attitudes of adult and adolescent parent populations. None of these instruments, however, has acquired much universal support among family literacy practitioners.

As with other standardized assessments, evaluators examining the parenting component can acquire pre- and post-test data, analyze the differences in the scores, determine what kinds of gains have been achieved, and document the findings. The results can be interpreted as providing some indication of the quality of the parent education component of the Even Start program. However, as we discuss later in this article, there may be other factors to be considered in assessing the quality of any program component.

Program providers also can use the results of standardized assessments to gauge, at least to some extent, how the “program as a whole” is functioning. Are the parents and children improving their literacy skills (as measured by the assessments being used)? How might the academic instructional program be revised to produce better results? Should the curriculum be changed to enable the participants to make more progress toward their literacy goals? Does an assessment instrument that measures the parenting curriculum show positive gains in knowledge by the parents?

Using alternative assessments

Alternative assessment is needed in family literacy programs because standardized assessment instruments: (a) only measure what is on the instrument; (b) cannot provide any “in depth” information about facts, concepts, or behaviors; and (c) do not allow any input from those being assessed other than what is requested on the instrument. However, credible input in verifying progress toward meeting the intent of the legislation that funds Even Start often can be acquired from non-standardized kinds of assessment. Alternative assessment tools such as oral interviews, surveys, and observations by staff or evaluators provide another medium for examining student progress for determining any needed changes in the curriculum.

The oral interview can be used to determine whether parents feel their needs are being met through participation in the program, to document how parents are using their new knowledge in dealing with real-life situations, and to validate home educators’ perceptions of positive changes that are occurring in the home. None of these outcomes are measured easily by formal assessments. If parents are uncomfortable in individual interviews, they can be interviewed in small groups. Not only will this improve the “comfort zone” for the parents being interviewed, but it also will provide for the possibility that comments by one person will stimulate additional ideas and comments by others in the group.
This type of assessment also can supplement data acquired from formal assessments. For example, the assessment data in one program indicated low achievement among some participants as measured by the HOME. An examination of their records showed that they had not been to many of the parenting classes offered. Through interviews with these families, staff members were able to ascertain that the schedule for the parenting classes, a required element in the curriculum, made attendance difficult. When adjustments were made to the schedule, the attendance rate as well as the performance on the HOME Inventory improved.

Surveys can also provide progress data about several components of the program. Our evaluation staff members at TCALL use a survey to determine the extent to which changes have occurred in: (a) parent-child relationships related to the children’s learning; (b) parent-school relationships; (c) parent-community relationships, and (d) how parents’ lives have changed in regard to better health and more control of their lives, for example. A follow-up survey, utilized after a local program has existed at least 2-3 years, is sent to parents who have left the program for whatever reason. It assesses such questions as: (a) Did the parents achieve their goals while in the program? (b) Have they been able to use the skills they acquired in the program to secure employment or pursue further education? and (c) Have they continued working with their children’s learning, participating in school activities, and utilizing their new learning to improve their families’ lives in other ways? Data compiled from these surveys supplement data acquired through formal assessments in the evaluation report and often offer new insights into needed program changes.

Observations by evaluators can be used to offer recommendations to program staff concerning, for example:

1. Re-arranging equipment and/or facilities for more effective learning;
2. Improving teaching methods and techniques, participating in selected staff development programs, or using peer-learning activities;
3. Changing PACT time to emphasize more interaction between parents and children; or
4. Acquiring materials or equipment that will help improve the quality of the instructional program.

In one program, a member of the evaluation team observed three classes of different levels (levels 1, 2, and 3) for adults learning English. However, there wasn’t much difference in the amount of English being spoken by the parents in any of the classes. The teacher indicated that parents were reluctant to use English although she knew they had the knowledge and skills to speak it. The evaluator made several recommendations for motivating parents to use more English and indicated to the program coordinator that a teacher who could only speak English be used in the level 2 and 3 classes. The change was made and at the next site visit, only English was being used in those two classes. The parents were not hesitant to try to speak it. Some even indicated to the evaluator that the change motivated them to speak English outside of the Even Start classes and that they were pleased with their improvement.

In addition to using evaluation data for program improvement, local program evaluation data can be used to justify requests for additional funding, either from a collaborating agency or from outside sources.
One of the best ways to justify continued funding for Even Start is to focus upon the human capital return from the investment of public monies. Reliable assessment data can influence policy-makers and educational leaders to support certain programs rather than others. With an ever-increasing student population and dwindling resources, school administrators prefer to allocate space and personnel to those programs that can document improved student learning. Evaluation data documenting the early learning achievement and smooth transition into school by Even Start children can justify requests for additional resources.

When the TCALL evaluation team is asked to assist the local Even Start coordinator in presenting the value of the program to the school administration staff or to the Board of Education, it is reliable data from local program achievements that convinces those in attendance of the program’s value. This type of presentation also provides support for requests for additional resources. In one school district, the Even Start coordinator gathered evidence from kindergarten, first-grade and second-grade teachers regarding the differences between children in their classrooms who had and had not been in the Even Start program. A “matched pairs” research design was used. When the differences in rates of attendance, classroom behaviors, and degrees of parental involvement were presented to school administrators with supporting anecdotal evidence, the administration committed additional resources to the program the following year.
Collaborating agencies that provide resources to the program also can be instrumental in finding additional support in the community and in communicating the value of the program to their own state and national offices. According to Darling (1999), although securing support may be one of the reasons for developing collaborative relationships, the overall purpose must always be to help families. Good collaborations should do both, and good evaluation data can be the basis for accomplishing the two. One of the best ways to help families is to provide a quality program while documenting the achievements of the participants.

When we, as family literacy practitioners, evaluators, or other supporters, petition lawmakers to continue funding this important program, we need to have solid data, both qualitative and quantitative. In 2002, a sub-committee in the U.S. Senate expressed the opinion that Even Start had not proven its worth and recommended a 20% reduction in the funding for the program. President Bush’s budget recommendations included the same reduction in Even Start funding. Many local program coordinators feared that the program would eventually be eliminated. With leadership from the National Even Start Association and the National Center for Family Literacy, literacy practitioners around the country organized to blunt that effort. However, a large body of reliable research and evaluation data would have been more effective when presented to those policy-makers who were unaware of the value of this important program.
Data that reflect Even Start’s accomplishments are being collected by local and/or state evaluators around the country. Those data should be compiled and shared regularly with those in positions of leadership at local, state, and national levels. We suggest that the U.S. Department of Education staff, one of our national literacy centers, or one of our professional literacy associations assume the leadership in that endeavor.

In essence, reliable evaluation data from local Even Start programs that document the value and worth of those programs are available. It is a requirement that those data be submitted from state agencies to the U.S. Department of Education every year. What is needed is some kind of effort to assimilate those data into composite findings and disseminate them to the policy-makers, legislators, and other audiences who can then use those data in making their decisions about continuing this valuable program. After all, what could be more important for our profession than demonstrating that our programs are worthy of continued funding because of their value to children and parents?

Darling, S. (August, 1999). Collaboration: The three-legged race. Momentum (2).

Field, T.M., Widmeyer, S.M., Stringer, S., & Ignatoff, E. (1982). Teenage, lower-class, black mothers and their preterm infants: An intervention and developmental follow-up. Child Development, 51, 426-36.

Holt, D. D. & Van Duzer, C. (2000). Assessing success in family literacy and adult ESL. Washington, D. C.: Center for Applied Linguistics.

Nickse, R. (1989). Noises of literacy: An overview of intergenerational and family literacy programs. Washington, D.C.: U.S. Department of Education.

Powell, D. (1996). Teaching parenting and basic skills to parents: What we know. In L.A. Benjamin and J. Ford (Eds.), Family literacy: Directions in research and implications for practice (pp. 65-69). Washington, D.C.: U.S. Department of Education.

Seaman, D. (1992). Follow-up evaluation of the Kenan model for family literacy. Adult Basic Education, 2, 71-83.

Snow, C., & Tabors, P. (1996). Intergenerational transfer of literacy. In L.A. Benjamin & J. Ford (Eds.), Family literacy: Directions in research and implications for practice (pp.73-79). Washington, D.C.: U.S. Department of Education.

St. Pierre, R., & Layzer, J. (1996). Informing approaches to serving families in family literacy programs: Lessons from other family intervention programs. Family literacy: Directions in research and implications for practice (pp.81-88). Washington, D.C.: U.S. Department of Education.

St. Pierre, R., Swartz, J., Gamse, B., Murray, S., Deck, D., & Nickel, P. (1995). National evaluation of the Even Start family literacy program. Washington, D.C.: U.S. Department of Education.

Weiss, H.B. (1988). Family support and education programs: Working through ecological theories of human development. In H.B. Weiss & F.H. Jacobs, (Eds.), Evaluating Family Programs (pp. 3-36). New York: De Gruyter.

Yeh, S.S. (2000). Building the knowledge base for improving educational and social programs through planned variation evaluations. American Journal of Evaluation, 21(1), 27-40.