Case Studies
We know that the right results are different for every client. For some, the result they are looking for is an improved understanding of the problem they are addressing or the clients they serve. For others, obtaining the right results means designing and monitoring new strategies to address these issues.
1. Rebuilding Together Embarks on New Effort to Measure Impact
The Challenge:
Rebuilding Together, a national housing nonprofit, made a long-term commitment to collect key outcomes about the impact of their services on the lives and communities they serve. Through their network of affiliates, they wished to strengthen their measurement and evaluation practices through long-term investment in impact measurement. Rebuilding Together chose a small subset of affiliates with which to pilot an impact measurement project, and wanted an external partner with evaluation expertise to conduct the project.
The Solution:
Actionable Insights helped Rebuilding Together leadership and affiliates to align what they do and their desired outcomes for low-income homeowners. We developed an evaluation plan, revised a survey to measure impact, and trained affiliates to collect survey data from homeowners. Ultimately, our report showed the measurable impact affiliates are having in their communities. We are continuing to work with Rebuilding Together on a second year of impact measurement with 12 additional affiliates from across the country.
Benefits and Results:
This work built consensus around the purpose and intended impact of Rebuilding Together’s core business and demonstrated credibility and value to existing and potential donors. The ongoing efforts promote an organizational culture of learning and continuous improvement. Rebuilding Together and its affiliates also improved their story-telling through data and evidence, enhancing their ability to compete for funding and the national organization’s reputation as a leader in the field.
“Actionable Insights has brought a collaborative, professional, and customized approach to the table, which has given our leadership great comfort throughout a new and complex process. I highly recommend this team!”
David Cattivera, Executive Director, Rebuilding Together Peninsula
2. Nonprofit Research Institution Uses Actionable Insights to Evaluate and Improve STEM/Health Internship Program
The Challenge:
A biomedical college internship program,“PUMAS,” received a five-year NIH grant that required annual program evaluation. Program staff did not have the capacity or expertise to evaluate the program themselves.
The Solution:
Actionable Insights developed a logic model and a streamlined evaluation plan for this new program in collaboration with the PUMAS team. We adopted instruments to measure many of the elements of the logic model, and developed tools to assess success of participants at various time points after they left the program.
Benefits and Results:
There is a robust logic model and evaluation plan that the PUMAS team can share with internal and external stakeholders and funders. Our annual evaluation reports helped the team think about the program differently, and identified actionable ways to make the program even better.
“Your evaluations really made me think about the PUMAS program through a different lens, and now I have a bunch of ideas on ways we can make it even better. So happy we are getting your evaluation services… very helpful!”
Shannon Noonan, Program Manager of Diversity, Outreach & Mentoring, Gladstone Institutes
3. Data Entry and Analysis Assists UCSF in Improving Program Recruitment and Content Delivery
The Challenge:
UCSF’s annual Plan on College event attendance dropped over time from nearly 1,000 eighth-graders and their families to barely 100. UCSF’s EAOP unit did not have continued capacity to evaluate the event, including marketing/recruitment efforts and information delivery. Program organizers were reduced to briefly reviewing participant surveys and then boxing them up, hoping there would be time for later study.
The Solution:
UCSF hired Dr. van Stelle as full-time program evaluator. Data-entry of registration forms and participant surveys coupled with analysis showed that the average event attendee was not from the target populations, and that few students brought a parent or guardian, even though specific event information was aimed at parents. The evaluation also showed that the program did a good job of delivering certain content, while other information delivery could be improved.
Benefits and Results:
Clear reporting of the findings with suggestions for improvement based both on data and on discussions with the program staff galvanized the organizers to quadruple event attendance the following year, with greater representation from the target populations. Program organizers also improved information delivery overall, and adjusted delivery of certain content to specific sub-populations.
“Jen immediately impressed me […] with her ability to multi-task and yet still have a keen eye for detail. She is adaptable, proficient in methodology and stats, and able describe complex issues with clear and concise report writing”
Holly Child, Ph.D., Project Manager, Applied Survey Research
4. Professional Evaluation Measures Impact of Student-Run Program to Gain Sustained Funding & Support
The Challenge:
A student-run summer science camp needed external support in order to be sustainable. The students understood that funders would want to see clear evidence that the children served by the camp were benefiting as intended. The students had good data but did not have the time or expertise to fully evaluate the camp. In addition, since the organizers were students, the camp was faced with the need to maintain continuity from year to year as students graduated and left the institution.
The Solution:
Dr. van Stelle evaluated the camp annually for four years, collaborating on data collection with each year’s new graduate student camp director. All aspects of the program were evaluated annually for continuous program improvement. Changes to camper recruitment, counselor training, camp procedures/activities, and new elements such as orientation night and alumni day were made based on Dr. van Stelle’s evaluations and subsequent recommendations.
Benefits and Results:
Strong evaluation results showing program impact allowed the camp team to win several multi-year grants for the camp. Camp directors were able to engage in continuous program improvement, better meeting the camp’s mission and goals.
“Jen and her team frequently met tight deadlines, which was crucial to [us] receiving grant funding. Jennifer always maintained open communication and provided valuable input in helping … evaluate [our] programs.”
Carlos Austin, Senior Management Analyst, County of Santa Clara
5. Nonprofits Engage in Continuous Program Improvement Supported by Actionable Insights
The Mix of Challenges:
(1) The federal government requires nonprofit hospitals to adopt and report on implementation strategies (IS) to meet the health needs identified through their IRS-mandated community health needs assessments (CHNAs).
(2) Funders did not have enough bandwidth to go back and forth with potential grantees to improve their applications, and wanted to build grantee/applicant capacity.
(3) Funders needed to monitor grantee processes and performance to understand how resources were being allocated, how programs were being managed, and to support their boards’ confidence in the funded work.
The Various Solutions:
(1) We facilitated hospitals’ selection of needs to address, conducted research on strategies, held community engagement events to present results and obtain community input, and wrote hospitals’ federally-mandated reports.
(2) We reviewed preliminary grant applications of community-based organizations and held one-on-one coaching calls with them to offer recommendations for improving the materials.
(3) We created a monitoring visit protocol, conducted site visits with grantees, wrote summaries and a high-level report of all findings and recommendations for grantee and hospital process improvements, and presented key insights to the funder’s board.
Benefits and Results:
Clients received thorough research with customized reporting to meet federal requirements, professionally-conducted presentations and small-group facilitation to support internal stakeholders and external community engagement, coaching to help build grantee capacity, and clear and specific recommendations to improve grantee and hospital processes.
“Jen and Melanie did a fantastic job helping our hospital. We appreciate their expert help in engaging the community including planning and facilitating our community forum. They were responsive, collaborative, and committed to getting our work done on time and exactly to our specifications.”
CHNA Client (Community Benefit Hospital Manager)
6. Nonprofit Hospitals Rely on Actionable Insights’ Principals for Community Assessment Research and Reporting
The Challenge:
It is a federal requirement that tax-exempt hospitals conduct a community health needs assessment (CHNA) every three years to identify and report on the greatest health needs affecting the communities they serve. There are specific IRS requirements related to conducting a proper CHNA, including consultation with public health and community representatives.
The Solution:
In compliance with IRS requirements, Actionable Insights collected statistical data from a variety of sources, conducted key informant interviews, and synthesized the qualitative and quantitative data to identify priority community health needs. We presented the findings to the hospital’s internal stakeholders, then wrote the CHNA report for the hospital to post publicly.
Benefits and Results:
Our streamlined community assessment process met federal requirements and fit the client’s compressed timeline, including professional presentations to obtain buy-in from internal stakeholders, customized reporting for public consumption, and responsive consulting support throughout the process.
“Actionable Insights has been exceptional. We appreciated your responsiveness, professionalism, and willingness to do whatever it took to deliver exactly what we needed, while keeping to our very tight deadlines.”
Adrienne Laurent, Chief Strategic Communications Officer, Salinas Valley Memorial Healthcare System
7. Schools Use Health and Wellness Assessment to Drive Action on Improving Student Health
The Challenge:
The Young Women’s Leadership Network (YWLN), the operator of a network of public schools that serve girls in New York City, had a strong interest in improving students’ health and wellness outcomes, but students’ needs were not fully understood by YWLN. In order to provide appropriate health and wellness supports to students, YWLN needed a window into their needs.
The Solution:
Actionable Insights teamed up with Stanford University to build a set of appropriately sensitive, validated survey questions for staff and students. IRB approval for the study was obtained from both Stanford and the New York City Department of Education. The surveys were administered online to consented students and staff. Actionable Insights analyzed the survey data and provided a timely report to YWLN, including findings about students’ fitness and nutrition, mental health, bullying, and trauma.
Benefits and Results:
Based on recommendations from Actionable Insights and Stanford University, YWLN has used data from the report to secure on-going funding for social work interns, obtained funding to bring training in trauma-informed instruction to school staff, and convened TYWLS leadership to identify next steps in the areas of youth mental health, bullying and relationship aggression, nutrition and physical health, and teacher training/professional development.
“Jen and Melanie supported us through our first-ever health and wellness assessment at our organization. We could not have done it without their hard work and expertise. The results have enabled us to continue our social work program in our five schools and we are in the process of developing curriculum to address the findings.”
Amanda Rosenblum, M.A., Associate Director, Girls’ Education, Young Women’s Leadership Network
8. Predictive Analysis Drives Decisions on Long-Term Emergency Response
The Challenge:
An aging population, combined with frequent calls from a small number of individuals, place serious strain on the medical system nationwide, from front-line ambulance and fire department personnel to emergency department and hospital staff. The Palo Alto Fire Department (PAFD) wanted to understand how its 911 call volume, particularly for emergency medical services (EMS), would change over time. PAFD also wanted to know more about alternative models for prevention and intervention, such as mobile integrated health care practices.
The Solution:
Actionable Insights employed a period life table approach, using governmental population projections, city planning scenarios, and PAFD’s own 911 call data to predict how demographic trends would affect its 911 call volume and composition in the year 2030. Actionable Insights also reviewed scholarly and journalistic literature to identify appropriate models to address the expected changes in call volume/composition.
Benefits and Results:
PAFD learned that between 2015 and 2030, the volume of 911 calls from the city was predicted to increase by 10-15% overall, with EMS calls increasing by 27-31%. It also learned that the total number of EMS patients from Palo Alto was predicted to rise from about 4,300 to between 5,300 and 5,600, with 66% of patients age 55 or older (who represented only 29% of the population in 2015). As a result of presenting the study to the City Council, PAFD was directed to explore partnerships with local healthcare systems and other community organizations to provide non-emergency treatment, proactive health services, community health education, and non-medical referrals.
“The Palo Alto Fire Department had a vision to identify long-range service impacts to the emergency medical system…Actionable Insights helped make our vision a reality.”
ERIC NICKEL, FORMER FIRE CHIEF, CITY OF PALO ALTO
9. Hispanic-Serving Institution (HSI) Evaluates Novel Pre-Calculus College Program
The Challenge:
Inequitable social context and institutional barriers lead to disparities in STEM achievement among Latinx and low-income students. One of HSI’s programs, the Pre-Calculus College Program (PCCP), is aimed at addressing university STEM attrition that occurs as a result of these disparities among high-potential Latinx students pursuing STEM degrees. PCCP provides small-class instruction, tutoring, and academic advising to support pre-calculus students. The HSI team requested an external evaluation of PCCP.
The Solution:
Actionable Insights (AI) created an evaluation plan with the team to identify and answer certain research questions about the contributions of PCCP to students’ STEM outcomes. AI then assessed the secondary data available on 150 students across two years of PCCP and over 750 non-PCCP students during the same time period to create a matched comparison group of 150 non-PCCP students. AI conducted statistical analyses to compare the differences and understand the roles of various factors on the outcomes of PCCP and non-PCCP students. AI also analyzed pre-/post- program surveys and program data on participation in advising and tutoring to better understand PCCP students’ self-efficacy in math and sense of belonging to the university community
Benefits and Results:
PCCP’s clear academic and social-psychological benefits contribute to STEM persistence among high-potential Latinx and low-income students at the university; for example:
-
- PCCP students showed improvements in their attitudes and behaviors related to their sense of belonging in the university community and their self-efficacy in math. These are important insights because both of these elements have been shown to contribute to students’ perseverance in STEM majors.
- PCCP contributed to students’ achievement in pre-calculus, a key STEM course. PCCP participation was the largest and most significant factor in predicting students’ grades in this class, and PCCP students’ grades in pre-calculus were significantly better (earning an average grade of B, compared to a C for non-PCCP students).
- PCCP students did significantly better in foundational STEM courses overall, earning higher GPAs compared to non-PCCP students.
- The data suggest that PCCP contributes to the retention of high-potential Latinx and low-income students in STEM; a significantly greater proportion of PCCP students than non-PCCP students declared a STEM major.
Continuing the PCCP program would allow the expansion of educational equity and would increase access to STEM careers for future students.