Skip to main content;
![]() |
||||||
| 6005 Westview Dr. | Houston,
Texas | 77055-5419 | 713-696-0700 | 1-866-696-4233 |
||||||
Charting a Course: Responding to the
Industry-Related
|
Assessment and Evaluation |
How should a needs analysis be
conducted?
The needs analysis is perhaps the most crucial of the steps, because
the remaining steps are based on it. Much has been written about how
and why to do a needs analysis. Philippi (1991) describes a detailed
process of observing workers on the job, interviewing all stakeholders,
and collecting all written material to determine the basic skills needed
on the job to do a specific job. Thomas, Grover, Cichon, Bird, and Harns
(1991) provide a step-by-step guide on how to perform a task analysis
for language minority employees. Burt and Saccomano (1995) discuss the
value of a needs analysis that goes beyond the work floor to include
union meetings and other places where workers interact on the job. Auerbach
and Wallerstein (1987) talk about a needs assessment process that is
more participatory as workers themselves identify the issues they wish
to explore in the class. And Taggart (1996) points out that the emergent
curriculum development process that takes place as the class progresses
provides timely information to service providers and is less costly for
employers.
Participatory learner-generated needs assessment is not antithetical
to the traditional needs assessment process. Grognet (1994) stresses
that for adults learning English as a second language, any instruction
to help them succeed in the workplace is in their best interest and is
by definition learner-centered. Lomperis (in press) asserts that having
a curriculum framework generated from a pre-program needs assessment
can facilitate the process of soliciting input from learners in the classroom.
Finally, Mansoor (1995) speaks of the necessity for the needs analysis
to be performed not solely for the jobs the participants have, but for
the positions they aspire to, as well.
If the learners are already on the job, the analysis is conducted in
that specific workplace. If learners are preparing for a job, several
different environments in that occupation can be used for the needs analysis.
In interviewing or surveying supervisors, managers, and nonnative and
English-speaking employees, the same kinds of questions should be asked
so that information from all these sources can be compared (Alamprese,
1994; Lynch, 1990).
For example, managers and supervisors might be asked if they perceive
their employees experiencing difficulty in such common workplace tasks
as following spoken instructions; explaining or giving instructions;
reporting problems; asking questions if they don't understand something;
communicating with co-workers; communicating on the telephone; communicating
in group or team meetings; making suggestions; reading job-related manuals;
filling out forms; writing memos, letters, or reports; reading notices,
newsletters, or short reports; doing job-related math computations; interpreting
graphs, charts, or diagrams; or following safety standards and measures.
Employees or learners should also be asked if they have difficulties
with these tasks. Next, or simultaneously, educators go to the workplace
to see the jobs performed and the language used on the job. At the same
time, all of the written materials used in the workplace or in that occupation-for
example, manuals, notices, safety instructions, and office forms-should
be collected and analyzed for linguistic difficulty. Meetings and other
team activities should also be observed for language use.
Perhaps the most important part of the needs analysis is the reconciliation,
where one takes the information from managers and supervisors, employees
and learners, puts it together with personal observation, and lists and
prioritizes the language needed on the job. This in turn leads to forming
the objectives for the program. Program objectives developed in this
way are based not only on what one party has reported, and not solely
on observation, but on a combination of factors.
What major areas should be considered in curriculum development?
While needs vary within each worksite or occupation, there are general
areas that should be considered in curriculum development. Some of these
areas, with examples of specific linguistic and cultural competencies,
are outlined here. Not all tasks and functions are taught at every worksite
to every participant. However, along with the information from the needs
analysis and from learner input, these topics form the backbone of the
curriculum.
1. Workplace Communication Expectations
2. Following Directions and Instructions
3. Job-Specific Terminology
4. Cross-cultural Factors
5. Company Organization and Culture
6. Upgrading and Training
Other factors also matter. Understanding situations in which pronunciation
makes a difference, such as in describing work processes and procedures
or in giving oral instructions, is important as are literacy initiatives
(e.g., reading posted notices, production reports, and forms; writing
an accident report; and keeping a written log). However, for the language
minority worker, the curriculum should start with workplace communication
and end with company organization and culture, and skills upgrading.
What should be considered when
planning lessons?
Lesson planning includes gathering text material and realia (e.g., those
manuals, signs, and job aids that were analyzed during the needs analysis
process) and any tools and equipment possible. From these, classroom
activities that involve listening, speaking, reading, and writing can
then be designed. However, language practice should not be limited to
the classroom. Learners should leave the classroom after each session
able to perform at least one new linguistic skill. For example, they
might be able to pronounce the names of three pieces of equipment, know
how to interrupt politely, or use the index of their personnel manual
to find information on sick leave policy. To this end, instruction must
include activities that use language needed by learners either on the
job or in the wider community.
The educator may have input into revising written materials used at the
worksite as a way of resolving worker performance problems on the job
(Westerfield & Burt, 1996). Guidelines for adapting written material
found on the job follow:
What are characteristics
of learner-centered instruction?
All workplace ESL (and all adult ESL in general) should be learner-centered.
If language learning is to be successful, the learners' needs, rather
than the grammar or functions of language, must form the core of the
curriculum and the instruction.
Many educators, among them Auerbach (1992), Auerbach and Wallerstein (1987), and Nash, Cason, Rhum, McGrail, and Gomez-Sanford (1992), have written about the learner-centered ESL class. In a learner-centered class, the teacher creates a supportive environment in which learners can take initiative in choosing what and how they want to learn. The teacher does not give up control of the classroom, but rather structures and orders the learning process, guiding and giving feedback to learners so that their needs, as well as the needs of the workplace, are being addressed. In a traditional teacher-centered classroom, where the teacher makes all the decisions, learners are sometimes stifled. At the same time, too much freedom given to learners, especially those from cultures where the teacher is the sole and absolute classroom authority, may cause learners to feel that the teacher has abandoned them (Shank & Terrill, 1995). The teacher must determine the right mix of license and guidance.
The following are characteristics of learner-centered classrooms:What are learner-centered instructional strategies?
Some strategies that are especially useful for workplace ESL programs are:
Various types of exercises and activities can be used in a learner-centered
environment. These include question and answer, matching, identification, interview,
fill-in, labeling, and alphabetizing; using charts and graphs; doing a Total
Physical Response (TPR) activity; playing games such as Concentration and Twenty
Questions; creating role-plays and simulations; developing a Language Experience
Approach (LEA) story; or writing in a dialogue journal. (See Holt, 1995, and
Peyton and Crandall, 1995, for a discussion of these and other adult ESL class
activities.)
What about assessing learner
progress?
Testing is part of teaching. Funders may mandate that programs use commercially
available tests such as the Basic English Skills Test (BEST) and the Comprehensive
Adult Student Achievement System (CASAS). These tests, when used in combination
with program-developed, performance-based measures, can provide a clear picture
of what has been learned in the class. (See Burt and Keenan, 1995, for a discussion
of learner assessment in adult ESL instruction.) Performance-based tests measure
the learner's ability to apply what has been learned to specific, real-life
tasks. Actual job artifacts such as pay stubs, job schedules, and company manuals
can be used to assess linguistic skills. Further, program-developed materials
lend themselves well to workplace ESL instruction in that they allow both learners
and teachers to see progress in the outlined objectives over time. Some program-developed
assessment instruments are discussed below.
Program-Developed Assessment Instruments
Checklists. Objectives for the course, or even for each lesson, can form the basis of a checklist. For instance, an aural/oral checklist for high-beginning learners might include such items as 1) uses level-appropriate words and phrases to respond verbally to spoken language; 2) uses extended speech to respond verbally to spoken language; 3) initiates conversation; 4) participates in small group or paired activities; 5) follows oral directions for a process; and 6) asks for clarification.
A checklist for reading might include such items as 1) recognizes appropriate sight words(e.g.,words on safety signs); 2) recognizes words in context; 3) shows evidence of skimming; 4) shows evidence of scanning; 5) reads simplified job aids or manuals; and 6) reads paycheck information.
A checklist for writing might include entries such as 1) fills out simple forms; 2) makes entries into work log; and 3) writes requests for time-off.
Learner-generated learning logs. In a notebook, such page headings as "Things I Learned This Month" "Things I Find Easy in English" "Things I Find Hard in English" "Things I Would Like to Be Able to Do in My Work in English" create categories that help learners see growth in their English language skills over time. If learners make an entry on one or more pages every week, then review the logs with their teachers every three months, they usually see progress, even if it is slight. This also helps teachers to individualize instruction.
Portfolios. These individual learner folders include samples of written work, all pre- and post-testing, self analysis, and program-developed assessment instruments. Portfolio contents also tend to show growth in vocabulary, fluency, and the mechanics of writing over time.
What kind of program evaluation is necessary?
Formative evaluation, performed while a program is in operation,
should be a joint process between a third-party evaluator and program personnel.
Together, they should review the curriculum to make sure it reflects the
program objectives as formulated through the needs analysis process. They
should also review all instructional materials (e.g., commercial texts and
program-developed materials) to see that they meet workplace and learner
needs. Finally, the third-party evaluator should periodically observe the
classroom to evaluate instruction and learner/teacher interaction.
Summative evaluation, done at the completion of a program,
should evaluate both the learner and the program. Learner evaluation
data can be taken from formal pre- and post-tests as well as from
learner self-analysis, learner writings, interviews, and program-developed
assessments (Burt & Saccomano, 1995).
A summative program evaluation should be completed by a third party. The third
party evaluator analyzes the above summative data that includes information
from all the stakeholders (i.e., teachers, employers, union representatives,
and learners) about what worked and did not work in the program, and why. The
evaluator also looks at relationships among all the stakeholders. This analysis
will yield more qualitative than quantitative data. However, there are processes
to quantify qualitative information through matrices, scales, and charts, as
discussed in Alamprese, 1994; Lynch, 1990; and Sperazi & Jurmo, 1994.
Conclusion
By following the steps discussed in this digest, a workplace or pre-employment
ESL program should meet the needs of employers, outside funders, and learners.
The best advertisement for a workplace program is employers choosing to continue
instructional programs because they see marked improvement in their employees'
work performance. The best advertisement for a pre-employment program is learners
using English skills on jobs they have acquired because of their training.
References
Alamprese, J. (1994). Current practice in workplace literacy evaluation. MOSAIC: Research Notes on Literacy, 4(1), 2.
Auerbach, E. (1992). Making meaning, making change: Participatory curriculum development for adult ESL literacy. Washington, DC and McHenry, IL: Center for Applied Linguistics and Delta Systems.
Auerbach, E., & Wallerstein, N. (1987). ESL for action: Problem posing at work. Reading, MA: Addison Wesley.
Burt, M. & Keenan, F. (1995). Adult ESL learner assessment: Purposes and tools. ERIC Digest. Washington, DC: National Center for ESL Literacy Education. (ERIC No. ED 386 962)
Burt, M., & Saccomano, M. (1995). Evaluating workplace ESL instructional programs. ERIC Digest. Washington, DC: National Center for ESL Literacy Education. (ERIC No. ED 386 961)
Grognet, A.G. (1994, November). ESL and the employment connection. Presentation at the Office of Refugee Resettlement English Language Training Consultations, Washington, DC. (EDRS No. ED 378 843)
Holt, G.M. (1995). Teaching low-level adult ESL learners. ERIC Digest. Washington, DC: National Center for ESL Literacy Education. (ERIC No. ED 379 965)
[Kirby, M.] (1989). Perspectives on organizing a workplace literacy program. Arlington, VA: Arlington Education and Employment Program. (ERIC No. ED 313 927)
Lynch, B.K. (1990). A context-adaptive model for program evaluation. TESOL Quarterly, 24(1), 23-42.
Mansoor, I. (1995, Fall/Winter). Program profile: The evolution of a workplace ESL program. The Connector, 3, 1, 4.
Nash, A., Cason, A., Rhum, M., McGrail, L., & Gomez-Sanford, R. (1992). Talking shop: A curriculum soucebook for participatory adult ESL. Washington, DC and McHenry, IL: Center for Applied Linguistics and Delta Systems.
Peyton, J.K., & Crandall, J.A. (1995). Philosophies and approaches in adult ESL literacy instruction. ERIC Digest. Washington, DC: National Center for ESL Literacy Education.
Philippi, J. (1991). Literacy at work: The handbook for program developers. New York: Simon and Schuster.
Shank, C.C., & Terrill, L.R. (1995). Teaching multilevel adult ESL classes. ERIC Digest. Washington, DC: National Center for ESL Literacy Education. (ERIC No. ED 383 242)
Sperazi, L., & Jurmo, P. (1994). Team evaluation: A guide for workplace education programs. East Brunswick, NJ: Literacy Partnerships. (EDRS No. ED 372 284)
Taggart, K. (1996, Spring). Preparing ESL workers to work in teams. The Connector, 4, 3-4.
Thomas, R.J., Grover, J., Cichon, D.J., Bird, L.A., & Harns, C.M. (1991). Job-related language training for limited English proficient employees: A handbook for program developers and a guide for decision makers in business and industry. Washington, DC: Development Associates. (ERIC No. ED 342 277)
Westerfield, K., & Burt, M. (1996). Assessing workplace performance problems: A checklist. Washington, DC: Center for Applied Linguistics, Project in Adult Immigrant Education.
This document was produced by the Project in Adult Immigrant Education, funded by the Andrew W. Mellon Foundation through a grant to the Center for Applied Linguistics (4646 40th Street, NW, Washington, DC 20016 202-362-0700). Additional funding was from the U.S. Department of Education (ED), Office of Educational Research and Improvement, under contract no. RR 93002010, The opinions expressed in this report do not necessarily reflect the positions or policies of ED or the Andrew W. Mellon Foundation. This document is in the public domain and may be reproduced without permission.