Summertime Evaluations

Summertime and evalin’ is easy
Surveys are fillin’, and response rates are high
Your dataset’s rich and your graphs are good lookin’
So hush little funder, don’t you cry

(With apologies to the Gershwins and Ella Fitzgerald!)

Despite the song, summertime evaluation has its own challenges. The nicer weather often signals a hiatus to regular programming and an increase in special events such as community BBQ’s and multiple-day festivals, requiring a different approach to engaging participants for their feedback. We also slow down a bit in the summer and limit tasks that seem too heavy – who wants to fill out a long survey when you could be outside having fun?

With that in mind, some thoughts on how to collect useful information when the weather’s nice:

  • Start with the simple metrics, like attendance, ticket sales, or amount of food consumed. They’re easy for stakeholders to understand, but just remember that they can be greatly influenced by factors outside your control (especially if your event is rained out): also, they won’t provide much insight if you’re looking for evidence of a greater impact.
  • Hit the pavement! Set up some volunteers with pencils and clipboards and get them talking with participants. Keep the questions to a minimum (3-4 max) so you’re not taking people away from the event for too long, and consider providing a little reward such as a sticker or coupon for providing their two cents.
  • Alternatively, set up a stationary spot for attendees to come by and participate. This method provides the option for longer surveys or more innovative data collection methods such as dot-voting. The main downside is that you need something to encourage people to come to you: if it’s a hot day a shaded tent and a cup of water may be a strong enough draw, but in any case take a minute to figure out what will appeal to people at your event.
  • Go online! Consider including in your evaluation plan social media statistics such as the number of visitors to the event website, likes on Facebook, and usage of the event hashtag on Twitter. Online conversations through these channels can also provide insights into what’s working and what needs to be changed. Promoting an online survey through social media and at the event itself can help collect data, as long as you remember that participants using these tools may not fully represent everyone who attended the event.
  • Debrief with your team of event organizers, volunteers, staff, and other key partners, using an approach such as the After Action Review. Don’t wait too long to hold it, and remember that your team’s perspectives may not match those of event participants.

Determining which method or methods to use will depend on a number of factors, including the scale of the event and the resources you have available. The main consideration, though, should be the purpose of the evaluation – what do you want to learn from the process, and what does success look like? If you just want to demonstrate that your event is popular, collecting attendance numbers (with perhaps a quick demographics survey) would be sufficient. In contrast, if you’re hoping to see more of an impact such as increased community awareness of your organization or a change in attitudes or behaviour, more time and effort will need to be spent engaging participants.

Got any tips for evaluating in the summer? Share them below!

Seeds for Thought: Negative Results

Whether you are evaluator of a program or someone associated with the initiative being evaluted (the evaluatee?), it’s probably safe to say that everyone hopes for good results: proof that all the planning, effort, and resources that went into the program made a difference. Sadly, that doesn’t always happen, leaving the evaluator to figure out how to present the information accurately and constructively.

Susan Lilley recently compiled a ten-point list (PDF) on this topic, based on discussion from the American Evaluation Association’s listserv (Hat tip to the Better Evaluation blog which provides some more context on the discussion and some commentary on the points). To my eye, all the points are great – in particular, #4 (“Build in time for course correction”) and #8 (“Present results in terms of lessons learned”) provide a great rationale for a developmental evaluation approach that understands from the get-go that some components of any given project will need to be tweaked or changed wholesale in response to changing circumstances. What I really appreciate about this list, however, is the very first point – “Use a participatory approach from the start”. Engaging stakeholders and working as a partner with clients are more than “feel-good” tactics: they help create a sense of ownership of the results and builds that crucial relationship that allows for sharing both good and bad news, as well as having a frank discussion as to what the results mean for future work.

What tip do you think is most crucial for sharing negative results? If you have been on the giving or receiving end of bad evaluation news, what helped turn the episode into something constructive? Share below!

Seeds for Thought: Noble Arsonist

If you follow Strong Roots’ Twitter feed, you may have noticed that I attended the MoSo conference last week here in Saskatoon. It was my first time attending this particular conference but hopefully it won’t be my last, with some great sessions and resources that were relevant to me as both a social entrepreneur and someone connected with the non-profit sector. I live-tweeted from most of the sessions and keynotes that I sat in on, so if you want to see what I took away check out my personal account around June 13-14 for ideas and insights.

Out of all the presentations, it was the very last one I caught that social change agents would probably find the most interesting. Theodora Lamb from Vancouver-based Capulet Communications specializes in social marketing for non-profits: her talk, “The Noble Arsonist: Tips for stoking fires and igniting movements for NGOs (and companies that care)” provided some great seeds for thought on the use of social media, particularly around creating “remarkable” campaigns. In contrast to the regular “heartbeat” relationship with a core audience that keeps them engaged and informed, remarkable campaigns are something out of the ordinary that makes your group stand out and get people talking who normally wouldn’t be interested.

Lamb provided several ideas and examples of how to run a successful remarkable campaign, all of which is covered in her free ebook, The Noble Arsonist. One insight that stuck with me was the benefit of using humour. Given that most (if not all) issues that non-profits and like-minded folk take on are quite serious, humour may seem inappropriate: however, “serious” can easily tip over into “negative” and discourage people from getting involved, whereas some light humour can facilitate conversation and foster broader awareness. This approach is also useful for criticizing a popular brand or figure while avoiding an us-vs-them battle: check out the e-book’s description of Greenpeace UK’s Dark Side campaign against Volkswagon as a prime example.

Over to you now – how does your organization initiative use social media? What worked well and what fizzled? Have you been involved with any remarkable campaigns, either as an organizer or supporter?