Eval Cafe

At the 2017 Canadian Evaluation Society conference, my friend and fellow evaluator Carolyn Camman and I decided to start a podcast on the topic of program evaluation. We were inspired by Kylie Hutchison and James Coyle’s Adventures in Evaluation podcast, which had wrapped up in 2014, and also by the conversations we fell into whenever we met up in person.

Thus, Eval Cafe was born, featuring informal chats on evaluation-related topics – the kind you might overhear at your favourite coffeeshop, if it was frequented by evaluators.

While the two of us eat, live, and breathe evaluation, we do our best to keep the conversation accessible to anyone who has an interest in the field, even if they don’t identify with the term “evaluator”. In our short run thus far (16 episodes and counting!), we have discussed a range of topics, from participatory approaches in evaluation to how Star Trek and evaluation connect.

Our latest episode features Chris Corrigan, where we talk about the intersection between evaluation and facilitation as well as the role of evaluation in community change more generally. I think that if you work at all in the non-profit or social change field, you’ll definitely enjoy this episode!

You can listen to the audio directly in your browser (below), or simply search for “Eval Cafe” in your podcast app/platform of choice and you should be able to find us.

Also, a quick plug for Chris’ upcoming online course, The Art of Evaluation in Complexity, starting February 5th and featuring Carolyn as one of the presenters!

Online and Learning

Without denying the various challenges affecting the non-profit / for-impact sector, it’s good to recognize the strengths, capabilities, and resources that we have access to today that would have been beyond our reach as recently as a few decades ago. In particular, I’m thinking about the explosion of online learning resources, ranging from blogs to webinars to full-fledged courses. Offerings in that last category, usually referred to as MOOCs (Massive Open Online Courses), are led by instructors from around the world and can include thousands of participants: for example, the Infographics course I enrolled in last winter reached its enrollment cap of 5,000 participants, representing students from 133 countries.

Like any educational method, MOOCs have both strengths and drawbacks, including a “one-size” approach that might not work with your style of learning. However, online learning has two big advantages for the busy non-profit with a small budget: they are usually free, and they can be done on your own timeline. At the very least, a MOOC can provide an overview of a topic as well as help identify knowledge and skill gaps.

If you’re in the for-impact sector, there are three online courses starting in the next week that might be of interest:

  • Principles of Project Management (Instructor: Sue Dawson, Polytechnic West [Australia]). An important topic for anyone in management or supervisory positions, but I think it would be generally useful for anyone in a non-profit setting given how much of our work is project-based. This course started this past week, so if you’re interested it’s best to sign up sooner than later!
  • How to Change the World (Instructor: Michael S. Roth, Wesleyan University). If your ambitions stretch beyond an individual project, check out this six-week offering starting January 20th. Based on discussions from the Social Good Summit, topics include poverty, climate change, disease, and education, with a focus on what we know, why should we care, and what we can do.
  • Introduction to Learning Technologies (Instructor: Heather Ross, University of Saskatchewan). A local offering that’s being called (perhaps slightly tongue-in-cheek, given the season) a TOOC, or a Truly Open Online Course. The focus is on using common online tools and platforms to support teaching and learning, including blogs, podcasts, wikis, Twitter, and more. Although the primary audience is the higher education sector, it looks like a great course that could provide useful insights for changemakers, especially since education and awareness-building activities are crucial for creating broader impact. This course officially starts January 21st, but in line with the “truly open” ethos, all materials are already online. (Full disclosure: My signficant other works in the same office as the course instructor.)

Looking for other options? Lifehacker has a list of interesting courses coming up, as well as sources for finding more options if those listed don’t suit your fancy. Personally, I’ve signed up for the Learning Technologies TOOC with the hope that some of the skills I develop there will be useful in growing Strong Roots’ web and social media presence. I might sign up for one or both of the other options, but I want to make sure I don’t over-commit and I can give the material the attention it deserves.

Are you enrolled in any MOOC’s or other learning opportunities? Share them in the comments below!

SPECs and Vision

As the end of the year rolls closer, I have been taking some time to reflect on the work I have accomplished so far through Strong Roots Consulting and where I hope to go. My recent post about Amazing YXE got me thinking about the strengths-based approach that has consistently served as one of my guiding principles – in case you missed it, here’s the pertinent part:

In supporting for-impact groups and organizations, I have always preferred to take a strengths-based approach that recognizes, celebrates, and builds on existing skills and assets. Although it would be foolish to deny that there are real needs and challenges in our communities today, focusing exclusively on deficits can lead to disillusionment, cynicism, and hopelessness: looking at strengths and existing resources (which may not always be measurable in dollars and cents) can help create confidence and point a way forward for individuals and neighbourhoods. More than that, this approach affirms the fact that every individual has something to contribute and some way of making a difference for our world.

In the above quote, I bolded the line about deficits-focus because the trend to emphasize needs and challenges is all too prevelant in our world today, including (and perhaps especially) in the traditional non-profit sector. Charities and service providers often work to address some type of problem, such as the effects of poverty, illness, violence, bereavement, or family conflict. Blame it on the prevailing Western model of medicine, where our idea of “health” has been formulated as the absence of illness, but even organizations that aim for a more positive and affirmative outcome like helping children achieve their potential can end up working to address impediments over encouraging well-being.

Very few people I know who work to make the world a better place want to focus exclusively on slapping bandaids on the wounds of individuals, families, and communities. While recognizing the necessity of dealing with acute problems that comes from living in an imperfect world, we want to build capacity, nurture strengths, and empower people and neighbourhoods. Over time and in the face of shrinking budgets and a never-ending flood of problems, it becomes too easy to focus on immediate needs and leave those aspirations for broader change on the back burner until that near-mythical future point where funds, time, and the will to use them are abundant.
Continue reading

Building Social Enterprises

If you were following Strong Roots’s Twitter feed yesterday, you probably noticed that I was at a workshop on Building Social Enterprises for non-profits. Hosted by the United Way of Saskatoon and Area and Affinity Credit Union and featuring David LePage of Enterprising Non-Profits (enp), the day provided some great food for thought for non-profits interested in starting a social enterprise as well as outlining the steps to move from an idea to reality.

The idea of social enterprises run by non-profits is not new or unusual. Examples run the full gamut of business ventures including museum and hospital gift shops, theatres, credit unions, recycling depots, thrift shops, cleaning agencies, dental clinics – name any sector and you can probably find an example of a social enterprise. What differentiates these businesses from the traditional model, besides the fact that they’re run by non-profits, is the focus on creating a broader impact in the community beyond narrow economic measures. Whether it’s expanding job opportunities for individuals traditionally un- or under-employed, decreasing social isolation, building cultural capital, or encouraging environmental sustainability, social enterprises aim to make a difference.

A quick note about definitions, since the concept of “social enterprise” is growing in popularity. LePage defined the term as only those businesses run by a non-profit, whereas others would expand social enterprise to include for-profit ventures with an explicit social mission (what LePage called “socially responsible businesses”). Personally I’m in the latter camp and consider Strong Roots Consulting as a social enterprise, despite its structure as a for-profit company (look for a post on this topic in the future!), but I can understand where the presenter was coming from.

I won’t go into too many of the details from the presentation here as enp provides a great resource, the Canadian Social Enterprise Guide, as a free PDF through their website: instead, I’ll highlight some key ideas and insights that I took away from the presentation.

Oreo vs. Pesto

At the heart of social enterprise is the focus on both financial and social returns on investment, or making a difference and a profit. There are two approaches to take towards meeting those goals, the “oreo” model and the “pesto” model. Some ventures will clearly separate out the financial and the social impact: for example, a company may donate a portion of its profits to charity, or a non-profit may run a business for the primary purpose of providing resources for its programs and services. In this model you can look at the the profit and the social purpose separately, just like the cookie and the filling are different parts of the oreo.

The other approach is to blend or integrate the pieces so that they’re inseparably. A social enterprise that hires former gang members to provide them with employable skills or sells fair-trade goods exclusively as part of an international development mission marries the financial and social: you can’t separate out the components any more than you can remove the basil or pine nuts or garlic from a pesto. Neither approach is automatically better than the other: it all depends on your purpose in starting the business.

Know Your Motivation

The reason for starting a social enterprise will generally fall in one of three categories: filling a community need in the market, advancing the mission of the parent non-profit, or contributing to the parent’s financial sustainability, or in simpler terms, market, mission, or money. Great social enterprises may end up contributing to all three outcomes, but it’s more likely that one of them will be the primary motivational driver, particularly in the start-up phase.

An exercise from the workshop clearly demonstrated the importance of being clear on purpose. Event participants were divided into four groups and tasked with developing a business plan for a thrift store (or at least starting a plan, since we only had 15 minutes!). The catch was that each group had a different motivation for launching the store: employing people with disabilities, earning profit for the parent organization, selling affordable clothing and household items in a low-income neighbourhood, or providing employment training for newcomers to Canada. My group was assigned the last one in the list, and while we included considerations around language support and teaching workplace cultural norms, we soon drifted to thinking about other goals. Should our store sell business attire at low cost for immigrants seeking work? Should we partner with international development agencies to share resources and build mainstream community interest? Could this business contribute back to our agency’s bottom line?

At some point, our group realized that it would be very difficult to incorporate all of those ideas and still have a business that was even remotely financially viable. Listening to the other teams present, it became apparent that purpose drove all sorts of “on the ground” decisions from business structure to location to human resource practices: for example, a venture to generate profit may attempt to limit staff turnover, while our concept would include exit or graduation policies so that new participants could continually enter the program. The store selling items to low-income families would try to keep prices as low as possible, in contrast to the enterprise employing people with disabilities that wanted to provide a living wage. As the business became established, the other goals could potentially be added but the primary motivation would continue to guide operational considerations and planning, particularly in the face of challenges and dilemmas.

Planning for The Long Haul

As LePage pointed out, a social enterprise is not a silver bullet or a quick and easy source of cash. Just like a for-profit business, there’s a start-up phase that can last up to five years before the model stabilizes sufficiently to generate regular returns (financial or social). A non-profit interesting in taking this route needs to do a lot of planning and preparation work, starting with developing internal buy-in and capacity, brainstorming business ideas, and ensuring that these opportunities are aligned with their strategy and mission.

A particularly important task is conducting a feasibility study: can your idea become a viable business while meeting the desired social outcomes? LePage shared an example where he was part of a small group planning a garden centre in an inner city neighbourhood. At this stage of determining feasibility, it soon became apparent that, outside of those leading the project, very few people in the community were interested in gardening. Morale of the story? It’s important to know and engage with your community!

Once feasibility is established, drawing up a business plan can help bring the details into place and serve as a tool for management, communications, marketing, and raising capital. One notable divergence from the traditional business plan is the inclusion of cost calculations associated with the social mission, such as providing supports to employees or accepting lower profit margins on the products and services sold. Since social enterprises led by non-profits can rely on support from their parent organization, these resources can also be included in the plan: on the flip side, the plan also has to consider what risks it poses to the parent, such as affecting its reputation or financial picture.

The Road Ahead

I greatly enjoyed attending the event, and not just for the extensive knowledge and personal experiences shared by the presenter (though that was definitely appreciated!). It was encouraging to see representatives for a wide range of non-profits at the presentation and to hear stories about social enterprises in the works, as well as the support and interest of the host organizations. Today I chatted with some colleagues who are interested in moving forward with supporting social entrepreneurship in Saskatoon: stay tuned!

Quick plug – Strong Roots Consulting works with non-profits and social entrepreneurs alike to support project creation and development, from planning and building the case to evaluating your efforts and helping you adapt in response. If you’re interested in starting a social enterprise and need a hand, let’s talk!

Evaluation Planning and Grant Applications

There are tons of great podcasts out there focusing on the non-profit world and evaluations, enough that I can do a full post on that topic sometime. Perusing the back episodes of the Chronicle of Philanthropy’s “Making Change” podcast, I came across a great interview with evaluation mastermind Michael Quinn Patton. He spoke at length about developmental evaluation and the growing movement for accountability, but one story he shared near the beginning (around the 6:30 mark) caught my attention. Patton related a common experience where a non-profit calls him up to say that they need an evaluation and were recommended to him. He asked what kind of evaluation they need, and after a confused silence they replied “A kind?”. He went on to tell the befuddled caller that just like there are different kinds of restaurants and computers and cars, there are different kinds of evaluation. In response the non-profit representative explained that their three-year foundation grant is coming to an end and they just noticed that they are supposed to do an evaluation as part of their agreement with the funder, at which point Patton responded, “I don’t do that kind of evaluation”.

Funders are increasingly asking applying organizations to evaluate their programs and projects: for example, the Ontario Trillium Foundation, a major funder in that province, has a dedicated page on creating an evaluation plan. Thankfully, OTF points out that evaluation planning should be incorporated in the program design process and take place before the project starts: as Patton’s story implies, leaving the evaluation to the last minute is not an ideal situation for either the evaluator or the organization.

With the approach of fall signalling the beginning of a new grant application season, now’s a great time to think about how evaluation fits in to the process. Every funding body and applying organization brings unique considerations, but there are three general principles to consider when you reach the “Evaluation” component of a grant.

1. Be clear as to what funder is looking for

This point is crucial, not just for the evaluation piece but the grant as a whole. If you don’t meet the requirements or understand what they’re asking for, all that time planning and writing will go for naught. Fortunately, almost all funding calls will list a contact person who can answer your questions and oftentimes provide some informal feedback on your application (or at the very least let you know if your idea is completely out in left field). Seek clarification on any points of uncertainty – ignorance is not bliss!

2. Know what you’re getting into

From the safety of the planning process, it can be easy to create an elaborate plan – sure, we can survey all of our participants four times during the program and run six focus groups! Although you should ensure that you are meeting the minimum requirements of the grant (following point 1 and clarifying with the funder directly if necessary), it’s better to underpromise and overdeliver instead of vice versa. Make sure you have the capacity to follow through on what you’re promising in terms of staff time and skills to plan the evaluation and collect and analyze data, as well as having access to the necessary tools such as online survey platforms or statistical software. Check whether you can use a portion of the grant funding to pay for tools and additional staff time or to hire an external consultant to conduct the evaluation.

3. Make sure the evaluation meets your needs too!

It’s all too easy to see the evaluation component as yet another hoop to jump through to get access to a pool of funds, but there are many benefits to your organization. What do you want to learn about your program, your participants, and your community? If the funder simply checks that you completed the evaluation and dumps it in a file-drawer with no impact on future funding, was the process still useful for you? An evaluation can contribute to the ongoing planning and development of your program, as well as demonstrate effectiveness and impact to your staff, participants, community partners, and future potential funders. In fact, ideally the evaluation planning should happen independently of any grant: when you get to that part of the application, it should just be the case of tweaking the pre-existing plan to meet any requirements of the specific grant.

Any other advice that I’m missing here? Share in the comments below or on Twitter!

If you represent a non-profit organization in the Saskatoon area that’s looking for help with evaluation planning, drop me a line – the initial conversation is (and will always be) free of charge!

Summertime Evaluations

Summertime and evalin’ is easy
Surveys are fillin’, and response rates are high
Your dataset’s rich and your graphs are good lookin’
So hush little funder, don’t you cry

(With apologies to the Gershwins and Ella Fitzgerald!)

Despite the song, summertime evaluation has its own challenges. The nicer weather often signals a hiatus to regular programming and an increase in special events such as community BBQ’s and multiple-day festivals, requiring a different approach to engaging participants for their feedback. We also slow down a bit in the summer and limit tasks that seem too heavy – who wants to fill out a long survey when you could be outside having fun?

With that in mind, some thoughts on how to collect useful information when the weather’s nice:

  • Start with the simple metrics, like attendance, ticket sales, or amount of food consumed. They’re easy for stakeholders to understand, but just remember that they can be greatly influenced by factors outside your control (especially if your event is rained out): also, they won’t provide much insight if you’re looking for evidence of a greater impact.
  • Hit the pavement! Set up some volunteers with pencils and clipboards and get them talking with participants. Keep the questions to a minimum (3-4 max) so you’re not taking people away from the event for too long, and consider providing a little reward such as a sticker or coupon for providing their two cents.
  • Alternatively, set up a stationary spot for attendees to come by and participate. This method provides the option for longer surveys or more innovative data collection methods such as dot-voting. The main downside is that you need something to encourage people to come to you: if it’s a hot day a shaded tent and a cup of water may be a strong enough draw, but in any case take a minute to figure out what will appeal to people at your event.
  • Go online! Consider including in your evaluation plan social media statistics such as the number of visitors to the event website, likes on Facebook, and usage of the event hashtag on Twitter. Online conversations through these channels can also provide insights into what’s working and what needs to be changed. Promoting an online survey through social media and at the event itself can help collect data, as long as you remember that participants using these tools may not fully represent everyone who attended the event.
  • Debrief with your team of event organizers, volunteers, staff, and other key partners, using an approach such as the After Action Review. Don’t wait too long to hold it, and remember that your team’s perspectives may not match those of event participants.

Determining which method or methods to use will depend on a number of factors, including the scale of the event and the resources you have available. The main consideration, though, should be the purpose of the evaluation – what do you want to learn from the process, and what does success look like? If you just want to demonstrate that your event is popular, collecting attendance numbers (with perhaps a quick demographics survey) would be sufficient. In contrast, if you’re hoping to see more of an impact such as increased community awareness of your organization or a change in attitudes or behaviour, more time and effort will need to be spent engaging participants.

Got any tips for evaluating in the summer? Share them below!

Seeds for Thought: Noble Arsonist

If you follow Strong Roots’ Twitter feed, you may have noticed that I attended the MoSo conference last week here in Saskatoon. It was my first time attending this particular conference but hopefully it won’t be my last, with some great sessions and resources that were relevant to me as both a social entrepreneur and someone connected with the non-profit sector. I live-tweeted from most of the sessions and keynotes that I sat in on, so if you want to see what I took away check out my personal account around June 13-14 for ideas and insights.

Out of all the presentations, it was the very last one I caught that social change agents would probably find the most interesting. Theodora Lamb from Vancouver-based Capulet Communications specializes in social marketing for non-profits: her talk, “The Noble Arsonist: Tips for stoking fires and igniting movements for NGOs (and companies that care)” provided some great seeds for thought on the use of social media, particularly around creating “remarkable” campaigns. In contrast to the regular “heartbeat” relationship with a core audience that keeps them engaged and informed, remarkable campaigns are something out of the ordinary that makes your group stand out and get people talking who normally wouldn’t be interested.

Lamb provided several ideas and examples of how to run a successful remarkable campaign, all of which is covered in her free ebook, The Noble Arsonist. One insight that stuck with me was the benefit of using humour. Given that most (if not all) issues that non-profits and like-minded folk take on are quite serious, humour may seem inappropriate: however, “serious” can easily tip over into “negative” and discourage people from getting involved, whereas some light humour can facilitate conversation and foster broader awareness. This approach is also useful for criticizing a popular brand or figure while avoiding an us-vs-them battle: check out the e-book’s description of Greenpeace UK’s Dark Side campaign against Volkswagon as a prime example.

Over to you now – how does your organization initiative use social media? What worked well and what fizzled? Have you been involved with any remarkable campaigns, either as an organizer or supporter?

Seeds for Thought: Scale

Today’s Seed for Thought comes from the Stanford Social Innovation Review’s blog, a site that covers (as the name suggests) social innovation and related conepts like philanthropy, social entrepreneurship, and nonprofit organizational development. In the latter category, an article today provided a five-question checklist for nonprofits to assess their readiness to scale and increase their impact. The second question asked whether your program model has been tested: according to a survey of American nonprofits, “only 39 percent of nonprofits that are scaling or intending to scale have evaluated the impact of their work”. To me, that’s a surprising result – in my mind, before growing a program or initiative you should take some time to make sure it’s actually achieving the results that you think it is!

Although I’m glad that evaluation is included in the list, I think there’s a danger that evaluation and research is relegated to a one-time “check it off the list” task. In scaling a program to new communities or populations, organizations are bound to run into unexpected challenges. Elements and approaches that were beneficial in the initial program may be less useful or even detrimental in new situations. One example from my own experience was with an educational support program that had its roots in a dense urban core and was being scaled to other smaller cities. The new site I was involved with was very different from the original program site in terms of geography, history, and demographics: for example, the original site was very ethnically diverse, while the families in the catchment area for the new site were primarily white and had lived in Canada for multiple generations. As a result, our new site did not have to do much work around English as a Second/Additional Language, but we did face unique challenges such as around parent and family engagement. Collecting and analyzing information about our neighbourhood, both government sources like the Census and on-the-ground knowledge from teachers, service providers, and community members, helped us to understand the context and respond appropriately.

Funds, resources, and organizational practices are important elements to consider when scaling up: at the same time, nonprofits need the capacity to recognize the changes that come with growth and adapt accordingly. One tool that can be helpful in this case is developmental evaluation, which as recognized in Michael Quinn Patton’s handbook on the subject, can help organizations identify effective principles from earlier work and determine when it’s better to adapt to local conditions rather than adhere to acontextual “best practices”. By integrating relevant and timely data collection and sense-making into the process, developmental evaluation can help nonprofits learn more about the new situations they are entering, avoid potential pitfalls, and successfully scale.

What else would you add to the checklist?

Word Counts

In response to my post last week on open-ended questionnaires, Sheila Robinson over at Evaluspheric Perceptions explored some of the risks in interpreting this type of data. Without a systematic approach to analyzing qualitative data, we can fall prey to confirmation bias, which as described in her post, “causes us to remember or focus on that with which we agree, or that which matches our internalized frameworks, understandings, or hypotheses”. Another risk is that we pay too much attention to extreme viewpoints, whether positive or negative, because they are more likely to be remembered. Check out Sheila’s post for more thoughts!

One question that I want to address quickly is what to do if you have collected some data from an open-ended survey and want to avoid these pitfalls, but don’t know where to begin? As with evaluation in general, one of the simplest starting points is counting. Read through all the responses and keep a running tally of how often certain ideas come up. You may already have some ideas in mind for how to categorize responses, which will help in sorting but could leave you open to confirmation bias: take care that you’re not trying to fit a square-shaped response into your round category! If you come across strong or extreme comments, make sure you view it in relation to general trends (having complementary numerical data helps here!) to determine how representative that position is: that’s not to say that you should ignore a point raised by a small number of people, but as in the example raised by Sheila in her post, you don’t need to rush and make sweeping changes to something that’s working for the vast majority of respondents.

If there’s interest, I can share an extended example from my first experience with qualitative analysis – food for a future post!

What’s in a Question (type)?

While on Facebook earlier today (I was connecting with some colleagues on a work-related issue, honest!), I came across a survey for a local non-profit initiative. As someone who both identifies as a researcher and generally likes filling out surveys, I eagerly clicked the link … and found myself looking at ten open-ended, fill-in-the-blank questions.

Now, I don’t have anything against this style of question: indeed, as I noted in an earlier post, it’s good to provide space for respondents to share their own perspectives and stories without being boxed into a particular set of responses. In my opinion, though, inviting only written responses is a move too far in the other direction. Some respondents may not have the time to write down their thoughts, while others may feel pressured to provide insightful, well-crafted responses to each question and decide to take a pass on the survey as a result. I remember a conversation with a community group where one member personally disliked open-ended questions: this person’s view was perhaps a bit extreme, but it brings up the good point that individuals may simply have preferences for one question type over another. Accessibility is also a potential concern: will people who have low literacy skills or other challenges around writing feel comfortable participating? A final consideration is analyzing this type of data, which takes more time and effort compared to compiling statistics from multiple choice or rating questions.

Again, I have nothing against open-ended questions: depending on the intended audience and purpose of the survey, it may even be completely appropriate to only use that type of response. For most general surveys, though, a little bit of variety is probably a good thing.