Word Counts

In response to my post last week on open-ended questionnaires, Sheila Robinson over at Evaluspheric Perceptions explored some of the risks in interpreting this type of data. Without a systematic approach to analyzing qualitative data, we can fall prey to confirmation bias, which as described in her post, “causes us to remember or focus on that with which we agree, or that which matches our internalized frameworks, understandings, or hypotheses”. Another risk is that we pay too much attention to extreme viewpoints, whether positive or negative, because they are more likely to be remembered. Check out Sheila’s post for more thoughts!

One question that I want to address quickly is what to do if you have collected some data from an open-ended survey and want to avoid these pitfalls, but don’t know where to begin? As with evaluation in general, one of the simplest starting points is counting. Read through all the responses and keep a running tally of how often certain ideas come up. You may already have some ideas in mind for how to categorize responses, which will help in sorting but could leave you open to confirmation bias: take care that you’re not trying to fit a square-shaped response into your round category! If you come across strong or extreme comments, make sure you view it in relation to general trends (having complementary numerical data helps here!) to determine how representative that position is: that’s not to say that you should ignore a point raised by a small number of people, but as in the example raised by Sheila in her post, you don’t need to rush and make sweeping changes to something that’s working for the vast majority of respondents.

If there’s interest, I can share an extended example from my first experience with qualitative analysis – food for a future post!