How do you know if your after school program is meeting the needs of the youth and community it serves? Like the students it serves, you measure its strengths and challenges using a variety of tests designed to show you where your program is succeeding and where you need to improve.
Before you measure, however, you must clearly define your goals and set a benchmark for each, so you have something against which to measure.
While many programs focus on the changes in school test scores and crime rates, you need to consider a broader range of desired outcomes, especially in the areas of social and emotional development.
This variety of outcomes steers the goals you set, the way you structure your program, and how you measure the success of your childcare service. Every evaluation method has multiple measures, and you must find the ones pertinent to your program and community.
Key performance indicators and measurement
Key performance indicators, or KPIs, are metrics used to evaluate the success of a given process or program. One example of a KPI measures homework completion rates; another is school attendance. Both of these are reported as a volume or percentage.
Other KPIs are reported along a spectrum or range. Many surveys designed to provide feedback ask participants to place their answers on a scale such as from "strongly agree" to "strongly disagree." Other evaluations ask observers to provide a number from one to ten indicating how often a measurement is met.
After school programs are assessed with a variety of methods providing reports with exact numbers as well as more subjective evaluations. While gathering data on staff attendance is easily measured with a time clock application, determining how effectively the staff supports youth is a bit more subjective.
Your organization must commit to collecting data as a routine part of operations to generate accurate and helpful metrics.
|
Example: Activity Structure and Management by Staff 1. Does staff arrive on time? OR What is the on-time arrival rate of the staff? Yes No Percentage 2. How well does the staff handle disputes among the students on a scale of 1-5? 3. Do daily tasks address the stated goals? Yes No
Examination of Relationships Between Youth and Staff 1. How responsive are the youth to the staff? Very Responsive Moderately Responsive Little to No Response 2. How effective is the staff in providing support to youth while they are accomplishing tasks? Very Effective Moderately Effective Not Effective 3. What is the emotional tenor of the relationship between youth and staff on a scale of 1-5?
Examine an Activity’s Level of Challenge 1. Do the students report they are challenged? Yes No Sometimes 2. Does the staff monitor a youth’s level of frustration and provide clear direction? Yes No Sometimes |
Instruments and tools for measuring after school program quality and environment
Numerous tools and instruments are available from a wide variety of associations, groups, and retailers that specialize in childhood development and childcare programming. A few of these tests are described below.
After School Activity Observation Instrument (AOI) adapted for WCER/PSA
This tool is used in site visits to collect observational data. It measures the quality of interactions, opportunities, and resources at a particular program. An observer records indicator data during a sequence of five-minute observations, including:
- Youth interaction
- Staff to youth interactions
- Youth engagement
- Activity organization
- Setting and resources
- Opportunities for skill building and mastery
After School Environment Scale adapted for WCER/UW
The After School Environment Scale contains 36 items to measure three main components of an after school program’s environment as it is reported by the youth participants. It is a survey type of tool in which the students report on a frequency scale that ranges from “Never” to “Always” in answer to questions such as:
- "I get to know other kids well here."
- "I help plan what we do here."
Through the reported reactions your program can judge the enthusiasm for and perceptions of the provided emotional support, the students’ belief in their autonomy and privacy, and the opportunity for peer affiliations.
After School Program Practice Tool - Research Version (APT-R)
From the Intercultural Center for Research in Education and the National Institute on Out-of-School Time (2005)
The APT-R measures the quality of an after school program by observing the overall program, each activity, and homework assistance and completion. It reacts to statements such as:
- “There is an established arrival routine that seems familiar to staff and youth.”
- “The staff uses positive reinforcement to encourage appropriate behavior."
Most items are rated on a scale of one to four that ranges from “not true” to “very true.”
Caregiver Interaction Scale (CIS) from Caregivers in Day-Care Centers: Does Training Matter? By Arnett, J.
Observations on 26 different elements are used to rate a single staff member on his or her interaction quality with children. Sample elements include:
- “Speaks warmly to the children.”
- “Does not supervise the children very closely.”
The elements are scored from 1 (never true) to 4 (often observed).
National After School Association's (NSACA) Standards for Quality School-Age Care
The NAA, a membership association for professionals who work with children in school and community settings, developed standards in the following care areas:
- Human relationships
- Indoor and outdoor environment
- Activities
- Safety
- Health
- Nutrition
- Administration
Assessment is according to the degree to which the program meets various quality standards, such as:
- The staff makes children feel welcome and comfortable.
- There are regular opportunities for creative arts and dramatic play
- The staff has access to adequate and convenient storage.
Observational responses are provided on a scale from zero (no evidence or not met) to 3 (fully met).
Using reports to determine changes
Once the observations have been recorded, the results are placed into a report. If an outside agency performed the assessment, that agency should provide an expert to help you translate the various answers into a format that shows clearly where your program is operating to standard and where it falls short.
Take the areas where a weakness is shown and clearly define each problem. Perform a root-cause analysis to determine exactly what is causing the issue and come up with a resolution to put into place to correct the deficiency. All stakeholders (program staff, a board of directors, school staff, and members of the community) should help formulate potential solutions.
After school programs provide wide-ranging benefits to the youth it serves and the community itself. In some areas, it is the only place for children to safely stay where they can be assisted with schoolwork and provided with real-world applications of what they learned.
You want to know that your program is delivering the level of quality the community, youth, and parents expect. If one area is deficient, the youth are not receiving the full benefits of the program. Use the results of the assessments and reports to optimize your after school program.
