We are working to understand the current literature on poverty, equity of education, and effective management of non-profits, to identify any credible research associated with each of these hypotheses. We refer to this as our “external evidence”. (Please see our attached hypothesis testing rubric.) Our primary work in testing the efficacy of our Theory of Change Logic Model consists of collecting and analyzing data from our own LET’S GO program. We refer to this as our “internal evidence”.
Click on the image to open.
Our evidence of mission success is presented in an infographic (above) to demonstrate the current success for change steps 5, 6, 7, 9, 10, and 11. This infographic and the more detailed data analyses behind it are used to determine our “internal evidence” scores. We intend to continue with both our external and internal testing of each hypothesis associated with each change step in our logic model. We believe in the adage: “What gets measured, gets managed”. We fully expect that our Program Support/Delivery operating model will require continuous improvement in order to achieve the internal test scores of all 15 hypotheses to a sufficiently high level that would lead a thoughtful evaluator and investor to consider the overall LET’S GO Theory of Change Logic Model valid. When this level of confidence in the efficacy of LET’S GO’s outcomes is achieved, we will be in a position to state that LET’S GO is fully capable (with other partners) of “Breaking the cycle of poverty through STEM education and workforce development”. When this occurs, LET’S GO will be in a position to scale our program exponentially.
Simultaneously with our big picture thinking, LET’S GO holds itself accountable to partners, funders and our team as a whole, to deliver on our mission and be able to report how we are accomplishing this mission. In addition to the efforts made to collect monitoring data (outputs already reported), LET’S GO chose to partner with a third-party evaluator to make sure we are truly doing what we say we are doing. Policy Studies Associates (PSA) is an evaluation firm used by some of our partners. As a STEM provider for these partners, some of our results are reported through the PSA Evaluation Report.
In addition, LET’S GO has partnered with The PEAR Institute: Partnerships in Education and Resilience, at Harvard Medical School and McLean Hospital (PEAR), to measure the impact of out-of-school time (OST) STEM programming on students’ science-related attitudes. Students’ STEM interest, identity and workplace skills (relationships with others, critical thinking and perseverance) were measured using the Common Instrument Suite (CIS) survey which includes a Holistic Student Assessment (HSA). Program quality was assessed using the Dimensions of Success (DoS) observation tool. All data collected using the DoS tool, were collected by PEAR-certified observers. Permission to use the CIS survey and DoS observation tool was obtained from The PEAR Institute at Harvard Medical School and McLean Hospital.
We know positive informal STEM experiences influence peer and adult relationships (teamwork/collaboration) as well as creative thinking, logic skills, and grit (PEAR). Based on assessments using validated tools from PEAR, our key findings from 2018 (see attached) show that LET’S GO students consistently out-perform students compared to the national norm sample on several STEM-related areas of measurement (p<.05): Interest, Enjoyment, Identity, Career Interest and Career Knowledge. In addition, on average, over 80% of LET’S GO students report a positive change in 21st century skills (critical thinking, perseverance, relationships with adults and peers) with significantly higher levels of perseverance and critical thinking compared to similar students in PEAR’s national norm sample (p<.05). Results from our 2018-19 school year indicated again that LET’S GO youth, both male and female, reported statistically significant positive change (i.e. mean or median score greater than 3.0) on 7 of the 9 scales on the CIS (p’s <.05). These scales were: STEM engagement, STEM career interest, STEM identity, critical thinking, perseverance, relationships with adults and relationships with peers. (PEAR, 2019). No significant change was observed for STEM Career Knowledge or pursuing STEM activities on their own outside of their school environment. No significant negative changes were found.
Finally, 35.7% of students who participated in LET’S GO’s STEM programming met or exceeded growth expectations on the NWEA Reading Assessment, compared to only 24.4% of students who did not participate, indicating the positive effect of our informal STEM programs on literacy (PSA, 2018). Teachers noted that our “STEM offerings provided the opportunity for students to practice a range of new literacy skills including reading and communication” (PSA, 2018).
High quality STEM programming is key to impacting student outcomes. Using Dimensions of Success (DoS) ratings (PEAR), the quality of our STEM programs was assessed to meet or exceed national benchmarks in the following dimensions: usage of materials, organization, space utilization, STEM content learning, STEM engagement and inquiry. Instructors have observed highly positive changes in their students in STEM confidence in science, technology, engineering, math and social skills as a result of our programs and anecdotal reports suggest that student participation is higher at STEM-related out-of-school events than those of other subject areas. In addition, pre/post assessments from select schools show students in LET’S GO STEM programs improve from 42% to 73% in their STEM content knowledge. Encouraged by the positive results they experience in our STEM programs, many of our students are pursuing other advanced learning opportunities.
All of these measured and reported outcomes are directly linked to our logic model. This Theory of Change Logic Model holds LET’S GO accountable to deliver on its mission. We are confident in the strides made that reinforce the direction of our organizational strategy to break the cycle of poverty through STEM education and workforce development.