Measuring Impact to Craft Your Story

Dr. Kari L. Jordan is the Executive Director of The Carpentries, a non-profit project that provides training in software development and data science skills to promote effective work and career development. The Carpentries builds community and local capacity for teaching and learning data skills as they work toward their goal of being the leading inclusive community teaching data and coding skills.

Prior to becoming Executive Director, Kari was the project’s Senior Director of Equity and Assessment, where she applied her experience in engineering education to evaluating the success of Carpentries workshops.

This post summarizes a series of four steps that Kari recommends event organizers work through to develop an impact evaluation strategy.

1. Write down your event’s purpose and goals and identify concrete impact factors to measure success. These factors may include:

• Reactions: Were instructions easy to follow? What were the strengths and weaknesses of the event? Were the organizers able to accommodate participants so that everyone could fully contribute?
• Learnings: Did participants learn something new about contributing to a project? Did participants learn a new tool? Did participants learn a new approach? Did participants meet new people?
• Behavior: Are participants more confident in their abilities? Are participants motivated to make contributions in the future? Are participants interested in sharing what they learned?
• Results: What are the deliverables? What percentage of participants followed the contribution guidelines? Were there any code of conduct violations?

For example, The Carpentries measures both short- and long- term success in the following areas.

Short Term: How are workshop participants immediately impacted?	Long Term: What concrete changes are workshop participants adopting?
Increased confidence in tools	Improved coding practices
Motivation to seek more knowledge	Making reproducible analyses
Increased self-efficacy for coding	Receiving professional recognition
Motivation to join a local or the global Carpentries community	Improved research productivity

2. Once you have identified your impact factors, think about how you can measure them. Strategies include:

• Surveys: lists of questions aimed at extracting data from a particular group
• Focus groups: demographically diverse groups of people who are assembled to participate in guided discussions to provide feedback
• Observations: acquiring information by watching behaviors and interactions
• Requests for comments: tools used to solicit feedback on planned actions that affect a broad community

Limit your data collection to information that relates directly to the evaluation of your outcomes and once you have the data, take the time to explore and analyze it.

3. In addition to collecting information at the end of the event about what participants have learned (known as summative assessment), it is also important to use formative assessment to explore how well participants are learning along the way. The difference between formative and summative assessment can be explained using the analogy of a chef tasting soup before serving it to customers. By sampling the soup while they are making it, the chef knows if they have the right balance of ingredients and can make any necessary adjustments. After serving the soup to customers, the chef gets final feedback on how it turned out.

Applying the concept of formative assessment to teaching, consider this quote from Software Carpentry instructor, Kunal Marwaha: “If someone feels it is too slow, they will be a bit bored. If they feel it is too fast, they will never come back to programming.”

By developing a strategy for formative assessment during your event, you will know how well your students are learning while you are still teaching them and uncover misconceptions. Having built-in flexibility allows instructors to modify their approaches to better meet the needs of participants. The Carpentries is well-known for using sticky notes to get anonymous, immediate feedback from participants. This approach gives participants the opportunity to write one positive aspect of the workshop on one sticky note and one constructive comment on another. Having this feedback allows instructors to address common issues with the class.

4. Finalize your strategy.

• Identify your target audience: who are your participants? What backgrounds, expectations, and experiences might they bring with them?
• Decide which impact factors you will measure
• Pick which methods you will use to measure impact

When finalizing your approach, consider what is most important to you and other stakeholders (e.g., funders) in terms of success: is it motivation? Is it a sense of belonging in the community? Get feedback on your strategy from your co-organizers or other colleagues. Lastly, don’t feel like you need to start from scratch. Below are resources you can adapt.

• Slide-deck from Kari's facilitated session: Kari Jordan_Impact
• Feedback: The Carpentries Feedback Ethos
• Surveys: The Carpentries Surveys
• Focus Groups: Data Carpentry Pilot Focus Group
• Observations: The Carpentries Suggested Rubric for Teaching Observations
• Request for Comment: Request for Comment on Carpentries Mission and Vision, Request for Comment on the Future of The Carpentries
• Assessment: The Carpentries Assessment GitHub Repository, The Carpentries Reports Page, Assessment on The Carpentries Blog

Featured Image Photo by Mason Unrau on Unsplash