State Assessment and
Goal Setting/Attainment Policy for Adult Education

Effective Date – July 1, 2011
Revised April 1, 2011

IV. Quality Control Procedures

A. Purchasing Assessment Materials

• Programs may purchase TABE testing materials through the Texas CTB McGraw-Hill Sales Representative. The contact information is located in the Appendix.
• Programs may purchase BEST Literacy testing materials in a bulk discount through TCALL. For more information call 1-800-441-READ.
• Programs may purchase BEST Plus materials through a CAL certified BEST Plus Test Administrator only or call TCALL toll free at 1-800-441-READ for information.
  For more information about how to find a BEST Plus Test Administrator Trainer to train a local staff, call your Regional teacher training center - GREAT Center. The contact information is located on the TCALL website (here) or call TCALL for contact information toll free at 1-800-441-READ.
• Programs may purchase GAIN testing materials through:
  Wonderlic, Inc.
  400 Lakeview Parkway, Suite 200
  Vernon Hills, IL 60061
  1-877-605-9496

B. Electronic Applications for Program Data Quality

• Texas Educating Adults Management System (TEAMS) is the Management Information System for adult education in Texas. Programs are required to hire or contract with a trained data entry staff person. GREAT Centers are required to have trainers available to train TEAMS data entry staff. TEAMS added a new feature in March 2011 to allow programs the ability to download class rosters into an EXCEL worksheet. The worksheet may be used to record hours for students from the original student sign-in sheets which contain student signatures and actual time and time out. The worksheets may be emailed electronically or saved on a flash drive or CD so that supervisors or other persons with access to TEAMS can upload the EXCEL worksheets into TEAMS electronically. This function has added integrity to the data by allowing more than one person to verify the original student sign in data entered. Direct data entry continues to be available. Programs are encouraged to do regular random selection of student sign in sheets to verify that data is accurate.
• Data Entered at least Quarterly – Programs are required to keep data entry current. Programs must enter data at least quarterly and complete a data sign off quarterly.
• Data Sign Off (DSO) – All program directors are required to complete a Data Sign-Off Report in TEAMS quarterly and in final compliance documentation after all data has been inputted and validated by the local program director.
• Adult Education Guidance Information System (AEGIS) is an electronic desk review tool that provides quarterly reports to the state office and local programs. Report Issues are sent out as informational in the 2nd and 3rd Quarters. The AEGIS report is sent out for local program required responses in the final compliance reports. Local programs may request that an AEGIS report be provided at anytime during the fiscal year by contacting the Grant Services Manager at Texas LEARNS. Call 1-866-696-4233. AEGIS automatically sends out letters of commendation after the final compliance reports are configured to qualifying programs.
• Program Improvement Plans (PIP) – Programs failing to meet the state target performance measures may be placed on program improvement plans if continuous improvement is not maintained. AEGIS automatically notifies programs and provides access to the Program Improvement Program template if a PIP is required.

C. Student Orientation

Student Orientation is a required activity for all students entering the program.  Contact hours must be documented, entered, and counted in the management information system Texas Educating Adults Management System (TEAMS).  Activities may include but are not limited to the following:

1. Goal-setting and/or identifying student goals related to core measures
2. Informing students of procedures including use of Social Security numbers (if applicable) for data match to determine follow-up outcomes;
3. Data collection of demographics;
4. Administration and scoring of assessment to determine Domain of Significance (DOS) (e.g. Reading, Language, Math, Oral, or Literacy);
5. Enrollment/registration;
6. Familiarizing the student with adult education and student-related program operating procedures;
7. Learning Styles Inventory; and
8. Individual Learning Plan.

D. Placement by Lowest Functioning Level

Multiple skill areas or domains are assessed and the student will likely have different abilities or functioning levels in each different domain.  Per NRS, students are placed according to the lowest functioning level.  The lowest functioning level determines the Domain of Significance (DOS) for the student for the present program year.  For example, during the pre-assessment, if a student assessed at the Beginning Level in Reading and the Low Intermediate Level in Mathematics the student would be placed in the Beginning Level and Reading would be the DOS; if the student takes the BEST Plus and BEST Literacy, and the lowest functioning level is in BEST Plus, the Domain of Significance is the BEST Plus.  Each year the lowest educational functioning level determines the DOS. Refer to Tables II, III, and IV for score ranges.  The lowest educational functioning level and the DOS will also be used to determine educational gain in subsequent or post-test assessments for federal reporting.  TEAMS, the management information system automatically calculates the DOS once the scale scores are entered properly.  Please note that for “state gain” performance measures all skill areas and domains are considered and calculated in the electronic desk review performed in the Adult Education Guidance Information System (AEGIS).  Programs have state gain performance measures in all skill areas or domains regardless of the DOS.

E. Placement by Specific Skill Area as an Exception

An adult learner may be assessed in the skill area or domain that reflects the student's needs as an exception to the rule of administering all three sections of the TABE or both the BEST Plus and the BEST Literacy tests. The NRS level descriptors may be instrumental in assisting students in determining their needs. During student orientation, the skill area or domain most relevant to student needs is determined, and the learner is assessed accordingly. For example, if a student's goal is to improve mathematics skills only, the mathematic/numeracy descriptors would serve as the primary guide for placement, and the TABE Math would be administered.

NOTE: Programs may not decide program-wide to test in one subject area.

F. Student Goal Setting

Within the NRS framework, all students are presumed to have at least one goal: development of literacy skills. That is, all students are assumed to be in the program to improve their literacy skills, and thus have the default goal of either improving literacy skills or improving English language skills. This presumed goal is the reason that all students are counted in the educational gain measure. Students often have other goals, but only four are directly relevant to NRS accountability requirements: obtaining employment, retaining employment, obtaining a GED or high school credential, and entering postsecondary education. Note that the default goal of educational gain remains, regardless of whether the student designates any of the additional goals.

1. Goal Setting Policy
   The Goal Setting Policy in Attachment P gives more detailed information about Policy points 1.1 through 1.8. Below the main points of the Goal Setting Policy are listed. Program directors who need to arrange training for staff on Goal Setting, should contact the GREAT Center for information about the face-to-face training (preferred training) and the two-hour online course.
   • Policy 1.1
     All students with 12 hours or more of attendance are held accountable for educational gain goals.
   • Policy 1.2
     Programs must adhere to goal-setting policies to ensure that learners select reasonable and attainable follow-up goals that satisfy NRS reporting requirements. All NRS follow-up goals must be attainable within the fiscal year. Follow-up goals should not be set until after pre-testing and a deliberate goal setting process are conducted.
   • Policy 1.3
     Programs must provide a goal setting process that includes completion of a student learning plan (SLP) within the first few weeks of instruction for all learners reported in the Texas Educating Adults Management System (TEAMS).
   • Policy 1.4
     Programs should review the SLP ideally after every 30 - 40 hours of instruction or at least after 60 hours to coincide with progress testing, making revisions as appropriate. Any revisions that affect NRS or state reporting must be identified in TEAMS.
   • Policy 1.5
     Programs must report initial NRS follow-up goals in TEAMS within the first few weeks of the student goal being set.
   • Policy 1.6
     If a program and student determines that a follow-up goal needs to be changed, the reason for the goal change should be documented in the student's folder. The goal change should then be edited/recorded in TEAMS.
   • Policy 1.7
     All program staff members who participate in student goal setting processes must successfully complete training on the Texas goal setting policy and attune annually.
   • Policy 1.8
     Learners with professionally certified documentation who self-identify a given physical, mental, or emotional disability must be provided appropriate accommodations for completing goals.
2. Goal-Setting Process
   Programs should have a goal-setting process whereby students meet with teachers or an intake counselor to help identify and set goals for instruction. The best time for this process to occur is when the learner first enters the program. The goal-setting process should help learners set both a realistic timeline for attaining each goal and a means for determining whether the goal is achieved. Since learners often change their goals after they begin instruction, it is often advisable to extend goal setting over additional orientation sessions during the first few weeks of class.
3. Identify Attainable Short and Long-Term Goals
   Setting the timeline and evidence of achievement will help the learner realize whether the goal is short or long term and whether it is achievable. For example, when learners enter a program, many of them state very broad goals, such as attaining a GED or getting a job. Breaking the goal down into discrete steps—with short and long-term milestones along the way—establishes a series of goals that help learners and teachers design instruction and identify the appropriate goals for NRS purposes. All goals entered into TEAMS are short term goals attainable within the program year (July 1 – June 30). Long term goals are captured at the local level and entered into the student's portfolio.
   
   When a student has one of the follow-up goals, the program is held accountable for helping the student attain the goal. The State must obtain information on whether the student achieved the goal after he or she leaves the program. For this reason, not only is it important that the student attain the goal during the year but also that the program's instruction, services, and referral to another agency for assistance be oriented toward helping the student achieve the goal.
   
   For example, a student with a goal of GED attainment should be at a literacy level that makes passing the GED tests likely within the calendar year. Data matching to determine whether the student has passed the GED and obtained their short term goal takes place from the beginning of the fiscal year, i.e., July 1, through the end of the next calendar year, i.e. December 31, for a total of 18 months. (See Appendix H for the timelines for collecting, matching and reporting data.)
   
   The student also should receive instruction that helps him or her acquire the additional skills needed for passing the tests. Similarly, if the student's goal is to obtain a job, the program should provide instruction, referral, and/or services to help the student acquire the skills needed to obtain employment. Participation in the many Texas Workforce Commission services or programs that the student may be eligible will assist the student meet his/her employment related goals.
   
   While setting a realistic goal is important for accountability, students' long-term goals should not be ignored simply because they are not obtainable during the NRS reporting period. Local programs must assist learners with setting goals appropriately and not avoid setting goals because of difficulty in following up with learners.
4. Secondary Measures
   The NRS secondary measures are optional measures of student outcomes and status that local programs are not required to use and should not be used as a basis for assessing performance under WIA. No performance standards are tied to these measures. The NRS includes these measures because many stakeholders during the consensus-building process believed these measures were important to the identity of the program and the goals and purposes of adult education.
   
   The secondary measures are in the areas of employment, community, and family. The employment measure indicates whether the student's public assistance grant was reduced or eliminated due to employment. This measure applies only to students receiving public assistance upon entry.
   
   In the area of community, there are three measures covering citizenship, voting, and community involvement. For students enrolled in EL Civics and citizenship programs, there is a measure of whether students have achieved citizenship skills. Voting for the first time or registering to vote and more involvement in community groups or activities are the remaining measures. The family measures include increased involvement in children's literacy activities and in children's education.
   
   Secondary student status measures of low-income status, displaced homemaker, and single-parent status are included, because these groups are specific target populations under WIA.
   
   NRS Implementation Guidelines, 2006

G. Data Collection Policies and Procedures

1. Programs must provide staff with clear written description and understanding of its roles and responsibilities for data collection.
   
   Collecting data involves every staff member in an adult education program. Intake staff may collect student demographic data and goals; teachers report attendance, may administer tests and report other outcomes; administrators must review and make decisions based on data tables; and administrative staff may be involved in checking forms and data entry. Program should have a clear written description of local data collection processes and the role of every individual in that process—and every program staff member should know his or her role. Job descriptions for all staff should incorporate the data collection responsibilities of the job and performance reviews should consider how well staff fulfilled these functions.
2. Clear definitions for each measure have been established.
   
   Staff members must know their role in data collection. It is critical that they have an understanding of what they need to collect. Local procedures should include a written, precise definition for each data item that is compatible with the state definitions. To resolve ambiguities please refer to: Improving NRS Data Quality, Chapter III., The Data Collection Process 23
3. Programs use standard forms tied to the program database for collecting data.
   
   Staff must record information on intake and other data forms and administrative staff then keys the information from these forms into TEAMS. Consequently, programs should use standard forms for data collection that include all the data elements and categories TEAMS uses.

H. Data Quality

1. Programs must have an error-checking and quality control system for identifying missing and inaccurate data.
   
   In an activity as complex and stressful as data collection, mistakes and missing data are inevitable. Staff may fail to complete forms fully due to high workload or simple oversight. In addition, the required information may not be available when it is needed. Programs must have procedures for checking data for completeness and accuracy at several times during the process. Data checking should follow a regular, prescribed schedule with clear deadlines. Programs should assign one or more staff persons to perform these data checking functions explicitly and make the job of this person known to all other staff in the program. Data checkers should review all data forms as soon as possible for completeness and accuracy and should receive error reports from the database to check immediately after data entry. To do their job, data checkers must have access to all staff— teachers, intake staff, counselors and administrative staff—and the authority to obtain cooperation.
2. Programs must have ongoing training on data collection.
   
   Local program directors are required annually to be up-to-date on National Reporting System policy, accountability policies, data collection processes, definitions of measures, and how to administer assessments. Texas LEARNS and the GREAT Centers provide NRS and data collection training twice annually to local directors and twice annually to data managers/coordinators.
   
   Data collection procedures will result in valid and reliable data if staff understand and follow them. Therefore, part of the program's data collection process must include training of all data collection staff on their roles and responsibilities, as well as the importance of data collection. Programs must offer training to all staff, and it should be offered several times during the year, if possible, to accommodate new staff and to allow existing staff to take follow-up training. It is also a good idea to have regular meetings or in-service trainings on data issues to give staff opportunities to discuss problems and issues that arise. (Improving NRS Data Quality 24, Chapter III., The Data Collection Process)
3. Program's use of the student-level, relational database system.
   For program improvement, look at student outcomes and demographics by student according to such variables as: number of instructional hours received, length of enrollment, the teachers and classes enrolled and by student educational functioning level. This type of analysis is available in TEAMS which stores information by individual student and allows programs to link different pieces of data for each student for reports or other output.
4. Data entry procedures must be clear and timely.
   Program procedures for data entry should specify at least one person whose job is to enter the information from forms into TEAMS. All staff members should know this person's role and he or she should have the authority to request clarification or resolution of errors. In addition, data entry should be scheduled at frequent, regular intervals—such as weekly or monthly. Without frequent data entry, not only will there be a large backlog of forms to enter, but programs may not become aware of errors and missing data until too late to correct them. Part of the data entry procedure should also include a prompt, organized way to identify and resolve errors. Data Sign-Off (DSO) Reports are due quarterly and 45 days after the closing of the program. Data Sign-Off (DSO) Reports are the program directors' certification that the data is accurate, valid, and reliable.
5. Staff must have timely or direct access to information from the database.
   A frequent complaint of staff involved in data collection is that they fill out the forms, enter the data and then never see it again. If staff members cannot access data, they cannot learn how data is used or how they themselves can use it for program improvement and management. TEAMS has the capability for local program staff to access their data in useful ways. It is best if this access is direct if possible so that staff at the local level can query the database to print a report locally. Program directors may assign "read-only" access to staff and teachers by completing a TEA Secure Environment (TEASE) access application. The TEASE application may be accessed online (here) 8 Read-only access to TEAMS can be at the class level and directors are encouraged to approve access for classroom teachers and supervisors.
6. Staff regularly reviews data.
   The program's data collection procedures should include a regular review of data soon after entry into the database. Regular data reviews allow directors to spot errors, missing data and other data that "don't make sense." Programs can use data reviews as a staff development opportunity to examine problems and issues to support program improvement. Data can help programs understand issues such as the impact of instructional arrangements, learner retention, and learner progress. Having staff learn to use data will foster program improvement and improve data quality; staff will see the importance of data collection to produce valuable information.

I. Data Matching

Collecting, matching, and reporting GED Obtainment, Entry into Postsecondary or Training, Employment Obtainment, and Employment Retention is performed according to the guidelines set up in the National Reporting System. The chart in Appendix K shows by program year and quarter when the data is collected, matched, and reported. This information is important to the goal setting process so programs know when a goal will be matched to a central data base.

Table of Contents