Dimensions of Success
DoS Overview | DoS History | DoS Tool | DoS Research | Certification | Training Schedule | DoS Program Planning Tool | DoS Brochure
The Dimensions of Success observation tool, or DoS, defines twelve indicators of STEM program quality in out-of-school time (e.g., afterschool, summer camps, etc.). It was developed and studied with funding from the National Science Foundation (NSF) by the Program in Education, Afterschool and Resiliency (PEAR), along with partners at Educational Testing Services (ETS) and Project Liftoff. The DoS tool allows researchers, practitioners, funders, and other stakeholders to track the quality of STEM learning opportunities and to pinpoint strengths and weaknesses.
DoS measures twelve dimensions that fall in 4 broad domains: Features of the Learning Environment, Activity Engagement, STEM Knowledge and Practices, and Youth Development in STEM.
Development of an Observation Tool for the Informal Science Field: Refinement, Field-testing and Establishment of Psychometric Properties research project has been supported by the National Science Foundation (NSF) since 2010. The study was conducted by the Program in Education, Afterschool and Resiliency (PEAR) and partners at Educational Testing Services (ETS) and Project Liftoff. The goal was to develop and psychometrically evaluate a program observation tool for the informal out-of-school time environment. DoS was developed based on the National Science Foundation's evaluation framework (Friedman, 2008) which assesses informal STEM program quality in out-of-school time settings such as afterschool programs, summer programs, museums, field trips, community centers. It was also inspired by the National Research Council's report, Learning Science in Informal Environments: People, Places and Pursuits (NRC, 2009).
The DoS observation tool consists of detailed explanations of each dimension and a 4-level rubric defining a range of quality associated with the dimension. Once certified, raters visit sites that are enacting STEM activities and take detailed field-notes. They then use the rubrics to assign ratings for each dimension that are backed up with evidence from the observation. Scores can then be aggregated across settings in particular geographic regions to look at trends or used for feedback and coaching at a particular site. All DoS observers must complete training and certification steps to maintain the reliable and valid use of the tool in the field.
The first three dimensions look at the logistics and preparation of the activity, whether or not the materials are appealing and appropriate for the learning goals, and how the learning environment creates a suitable space where students can explore science informally.
The second three dimensions look at how the activity engages students: for example, the dimensions examine whether or not all students are getting access to the activity, whether activities are moving towards STEM concepts and practices purposefully or superficially, and whether or not the activities are hands-on, and designed to support students to think for themselves versus being given the answer.
The next domain looks at how the informal STEM activities are helping students understand STEM concepts and make connections. It also looks at whether or not students have opportunities to participate in the inquiry practices that STEM professionals use (e.g., collecting data, using scientific models, building explanations, etc.), and whether students have time to reflect and make meaning of their experiences.
Finally, the last domain assesses the student-facilitator and student-student interactions and how they encourage or discourage participation in STEM activities, whether or not the activities make STEM relevant and meaningful to students' everyday lives, and how the interactions allow youth to make decisions and have a voice in the learning environment and community.
For a detailed introduction to DoS, including sample rubrics, data reports, and summaries of each dimension, please download our DoS Guide here: DoS Guide for Organizations (pdf).
Technical Report: The technical report that describes the development of the DoS observation tool as well as the results of the National Science Foundation-funded study that led to the validity argument for the tool can be found at this link: Dimensions of Success Technical Report (pdf).
Key Findings include:
- For each dimension, the full score range was used across 284 observations in Missouri, Kansas, Ohio, Massachusetts, Rhode Island, and New Hampshire.
- Dimensions that supported youth development but did not directly require STEM focus (e.g., Organization, Materials, Relationships) received higher ratings by observers, while the STEM-focused dimensions (e.g., Engagement with STEM, Inquiry, Content Learning) tended to receive lower scores. This indicates the tool can be critical for identifying areas of weakness and providing goals for STEM programs to strive towards.
- Pairs of observers reported similar scores across the dimensions when observing the same activity. We note, however, that informal STEM programs have many moving parts, and a one-time visit is unlikely to provide an accurate understanding of its overall strengths and weaknesses. Therefore, to understand the quality of a program, results from multiple observations would need to be combined.
Reading about DoS:
Quest for Quality in Afterschool Science: The Development and Application of a New Tool
DoS used as an observation measure used for "Evaluation Study of Summer of Innovation Stand-Alone Program Model FY2013: Outcomes Report"
How to become a certified DoS observer:
The training process involves a two-day live webinar training, a calibration conference call, and successful completion of two practice observation in the field. PEAR staff offer live, webinar DoS trainings monthly and in-person trainings for state networks or organizations can be arranged as needed for an additional cost. PEAR staff provide ongoing technical assistance, quarterly data reports, and online data management support for all DoS observers for two years post-certification.
Click here to read more about the DoS Certification process.
DoS Training Schedule Update:
- July 20-21, 10am- 4pm EST; SPACES AVAILABLE
- August 10-11, 10am- 4pm EST; SPACES AVAILABLE
- September 21- 22, 10am- 4pm EST; SPACES AVAILABLE
Trainings are offered monthly, and new dates are posted as scheduled. For private training arrangements, please see contact information below.
Registration: Please follow this link to begin the registration process: https://www.surveymonkey.com/r/DoS_Training_Registration
Please contact our STEM Client Relations Coordinator, Rebecca Browne, for more information about the training/certification process or to schedule a private training: firstname.lastname@example.org.
For questions about the DoS tool itself, data collection, and development of the tool, please contact Dr. Ashima Shah at email@example.com.
RECERTIFICATION as a DoS OBSERVER: Please note that observers that have been certified for two or more years need to complete re-certification to make sure they are continuing to use the tool reliably. Recertification trainings are only offered a few times a year. The next recertification training will be held August 30th, 2016 from 12pm-4pm EST. To register, please complete the form using this link: https://www.surveymonkey.com/r/DoS_Recertification_Registration
DoS Executive Training: The DoS Executive Training provides an introduction to the DoS Framework for quality STEM programming. This 3-hour session is ideal for network leads, program administrators, or funders seeking to better understand how to design and improve high-quality STEM learning experiences for youth. Suggestions for setting up a system for program quality including issues of data collection, outcomes and reporting will also be shared. While practitioners may benefit from this training, we suggest our DoS Framework Workshop for those that work directly to implement STEM programming. The next executive training will be held on August 19th, 2016 from 1:00pm- 4:00pm EST. To register, please complete the form using this link: https://www.surveymonkey.com/r/DoS_Executive_Training_Registration
Dimensions of Success (DoS) Program Planning Tool:
The DoS Program Planning Tool or DoS PPT is a new tool being developed by PEAR for practitioners in the field to use as they prepare staff to lead STEM activities. The DoS PPT aligns with the DoS observation tool, and focuses on how facilitators can prepare activities and prepare a teaching strategy that will allow them to reach high quality on all twelve dimensions. The DoS PPT is available in its first beta version before another round of revisions. The PEAR team hopes to collect feedback on its early use in the field, so please make sure you complete the form with your contact information before you download the tool so you can be contacted for feedback in the future.
Click here to access the DoS Program Planning Tool, The DoS Feedback Report, and The DoS Coaching Guide.
For more information on DoS, please download our DoS brochure.