Seeds for Thought: Scale

Posted on by
Reply

Today’s Seed for Thought comes from the Stanford Social Innovation Review’s blog, a site that covers (as the name suggests) social innovation and related conepts like philanthropy, social entrepreneurship, and nonprofit organizational development. In the latter category, an article today provided a five-question checklist for nonprofits to assess their readiness to scale and increase their impact. The second question asked whether your program model has been tested: according to a survey of American nonprofits, “only 39 percent of nonprofits that are scaling or intending to scale have evaluated the impact of their work”. To me, that’s a surprising result – in my mind, before growing a program or initiative you should take some time to make sure it’s actually achieving the results that you think it is!

Although I’m glad that evaluation is included in the list, I think there’s a danger that evaluation and research is relegated to a one-time “check it off the list” task. In scaling a program to new communities or populations, organizations are bound to run into unexpected challenges. Elements and approaches that were beneficial in the initial program may be less useful or even detrimental in new situations. One example from my own experience was with an educational support program that had its roots in a dense urban core and was being scaled to other smaller cities. The new site I was involved with was very different from the original program site in terms of geography, history, and demographics: for example, the original site was very ethnically diverse, while the families in the catchment area for the new site were primarily white and had lived in Canada for multiple generations. As a result, our new site did not have to do much work around English as a Second/Additional Language, but we did face unique challenges such as around parent and family engagement. Collecting and analyzing information about our neighbourhood, both government sources like the Census and on-the-ground knowledge from teachers, service providers, and community members, helped us to understand the context and respond appropriately.

Funds, resources, and organizational practices are important elements to consider when scaling up: at the same time, nonprofits need the capacity to recognize the changes that come with growth and adapt accordingly. One tool that can be helpful in this case is developmental evaluation, which as recognized in Michael Quinn Patton’s handbook on the subject, can help organizations identify effective principles from earlier work and determine when it’s better to adapt to local conditions rather than adhere to acontextual “best practices”. By integrating relevant and timely data collection and sense-making into the process, developmental evaluation can help nonprofits learn more about the new situations they are entering, avoid potential pitfalls, and successfully scale.

What else would you add to the checklist?

Seeds for Thought: Attributing Laughs

Posted on by
1

“I don’t care about the details, just tell me what impact you had!” The impact of interventions that deal with complex social issues often can’t be boiled down to a simple yay/nay vote, especially when examining longer-term outcomes: unfortunately, explaining that fact to stakeholders and funders without seeming evasive can be difficult.

Chris Lysy over at Fresh Spectrum has penned five humourous illustrations on this difficult topic, which I’m keeping in my back pocket the next time I have a discussion with anyone about attributing impact. I find the idea of a “logic model repair shop” (#3) to be hilarious (and I’ve seen models that look that complex!), and the image of someone asking a whole community to stop the good work they’re doing to avoid messing up his or her impact assessment raises a good point that nothing we do happens in isolation. That being said, I think my favourite is the quote from John Mayne: “We need to accept the fact that what we are doing is measuring with the aim of reducing the uncertainty about the contribution made, not proving the contribution made.”

What’s your favourite out of the five?

Word Counts

Posted on by
1

In response to my post last week on open-ended questionnaires, Sheila Robinson over at Evaluspheric Perceptions explored some of the risks in interpreting this type of data. Without a systematic approach to analyzing qualitative data, we can fall prey to confirmation bias, which as described in her post, “causes us to remember or focus on that with which we agree, or that which matches our internalized frameworks, understandings, or hypotheses”. Another risk is that we pay too much attention to extreme viewpoints, whether positive or negative, because they are more likely to be remembered. Check out Sheila’s post for more thoughts!

One question that I want to address quickly is what to do if you have collected some data from an open-ended survey and want to avoid these pitfalls, but don’t know where to begin? As with evaluation in general, one of the simplest starting points is counting. Read through all the responses and keep a running tally of how often certain ideas come up. You may already have some ideas in mind for how to categorize responses, which will help in sorting but could leave you open to confirmation bias: take care that you’re not trying to fit a square-shaped response into your round category! If you come across strong or extreme comments, make sure you view it in relation to general trends (having complementary numerical data helps here!) to determine how representative that position is: that’s not to say that you should ignore a point raised by a small number of people, but as in the example raised by Sheila in her post, you don’t need to rush and make sweeping changes to something that’s working for the vast majority of respondents.

If there’s interest, I can share an extended example from my first experience with qualitative analysis – food for a future post!

Seeds for Thought: Big Data

Posted on by
Reply

During the span of a week, I come across lots of interesting stories, resources, and sites online that may be of interest to those in the non-profit-sector. In line with my approach of connecting people with resources and sharing information, I’m thinking about starting a weekly feature to highlight some of those links – consider this the pilot edition!

This week, I’m highlighting a trio of posts from the Harvard Business Review’s Blog Network, a site I recently started following. Although the focus is primarily on for-profit organizations, I’ve already seen content on social enterprises, philanthropy, and international development, as well as resources and trends that would be equally applicable on the non-profit side.

All three articles below relate to managing and using data, particularly “Big Data”. The term recognizes that collectively we are producing and storing exponentially-greater amounts of data in recent years than at any other point in human history – the first article cites research that 90% of data currently in existence was created in the past two years! This explosion in information can help grow our understanding of practically every facet of life, but there are challenges in analyzing and interpretating these giant data sources as well as limits to how much we can learn from them.

  • Jeff Bladt and Bob Filbin’s article title says it all – A Data Scientist’s Real Job: Storytelling. It’s similar to a truism I learned from a great professor during my undergraduate education, that all research projects have to tell a story: we start at some point of knowledge, we run an experiment or collect some information, and we learn something as a result. Tables of numbers and statistical tests are essential tools, but by themselves they do not advance our knowledge. As Bladt and Filbin put it, “Data gives you the what, but humans know the why“.
  • Presenting data in an accurate, easily-comprehensible visual form has become a field in its own right. If you’re not sure where to start in sharing information, Nancy Durante gives a simple suggestion: When Presenting Your Data, Get to the Point Fast. Check her post for some good tips on how to help your audience focus on the key numbers (hint: tables of numbers and pie charts are not in the cards!).
  • Finally, Kate Crawford explores The Hidden Biases in Big Data. Even databases with millions of records may not cover the full spectrum of a phenomenon: Crawford gives the example of the 20 million tweets generated during Hurricane Sandy, the majority of which came from tech-connected Manhattan compared to harder-hit neighbourhoods. Her prescription? “Take a page from social scientists”: pay attention to where the data comes from, examine your cognitive biases in interpreting the data, and utilize a diverse range of methods including qualitative approaches like interviews to complement the quanatitative data findings.

If you have any thoughts or additional links to share on this topic, I’d love to see them! You can use the comments field below or find me on Twitter. Also, any feedback or suggestions on this approach of weekly annotated links would be greatly appreciated.

What’s in a Question (type)?

Posted on by
4

While on Facebook earlier today (I was connecting with some colleagues on a work-related issue, honest!), I came across a survey for a local non-profit initiative. As someone who both identifies as a researcher and generally likes filling out surveys, I eagerly clicked the link … and found myself looking at ten open-ended, fill-in-the-blank questions.

Now, I don’t have anything against this style of question: indeed, as I noted in an earlier post, it’s good to provide space for respondents to share their own perspectives and stories without being boxed into a particular set of responses. In my opinion, though, inviting only written responses is a move too far in the other direction. Some respondents may not have the time to write down their thoughts, while others may feel pressured to provide insightful, well-crafted responses to each question and decide to take a pass on the survey as a result. I remember a conversation with a community group where one member personally disliked open-ended questions: this person’s view was perhaps a bit extreme, but it brings up the good point that individuals may simply have preferences for one question type over another. Accessibility is also a potential concern: will people who have low literacy skills or other challenges around writing feel comfortable participating? A final consideration is analyzing this type of data, which takes more time and effort compared to compiling statistics from multiple choice or rating questions.

Again, I have nothing against open-ended questions: depending on the intended audience and purpose of the survey, it may even be completely appropriate to only use that type of response. For most general surveys, though, a little bit of variety is probably a good thing.

Buzzwords for Non-profits

Posted on by
Reply

From social enterprise to strategy and planning principles, the non-profit sector is borrowing a lot from the business world. Unfortunately, one less positive adoption is what the Harvard Business Review dubs “bizspeak“: their list of words that should be blacklisted, from “actionable” to “win-win”, has definitely made it into our vocabulary. To be fair, non-profits have long had a language of their own, especially acronyms: moving to a new city or province, or connecting with an organization from a different area (youth services compared to health, for example) requires a mental recalibration and some awkward “What does that mean?” moments.

As a lighthearted end to the week, I’d like to hear from you – share some non-profitese that you find particularly aggravating, amusing, or both!

Time to Count

Posted on by
Reply

As the one and only person working for Strong Roots Consulting, there are many business elements I have to deal with as part of the trade. There’s various regulatory and legal requirements to fulfill, finances to manage, and – a personal “favourite” on the necessary evil list – time tracking. My general preference is to create a proposal with a set project fee, instead of charging by the hour: however, I still need to determine how much a project should cost. A simple starting point is to estimate the number of hours that I would need to complete the work and multiply that number by a per-hour rate. Time tracking then becomes a data collection method to help me assess the accuracy of my initial estimate – or in other words, the first step in an evaluation.

For many people, conducting an evaluation seems like a complex undertaking. Where do you start? Do you need to create a logic model first? What should you measure? What data collection methods should you use? Quantitative, qualitative, or mixed methods? How do you analyze and present the data you collected? A search for “evaluation” books on Amazon turned up over 128,682 results, while a Google search returned “about 357,000,000 results”, so not much help from those sources (or rather, too much help).

One piece of advice I heard recently (and for the life of me I can’t remember where) is that one of the easiest first steps to take in evaluation is counting. It makes a lot of sense: we learn counting at an early age, after all, and it’s pretty easy to come up with questions that can be answered with a number. How many clients are we serving? How many referrals are we making? How much staff time was dedicated to a certain project? How many people indicated through a client survey that they were happy with our services? I bet if you took a minute right now you could come up with similar questions for your professional or personal life (how many hours of TV do I watch a day?) that can be easily answered by tallying up numbers.
Continue reading →

Emergence

Posted on by
Reply

A week or so ago, I received an email out of the blue from a manager who works for a local non-profit agency. She was interested in social innovation and had done some work in that area, and recently learned about the First Tuesday session I had hosted in January on the topic. We met for coffee and had a great conversation, coming up with some ideas on how to promote and support social innovation in Saskatoon. Our next step was to reach out and connect with some other people locally who might be interested in helping out with the planning: although I was already expecting a positive response, I was a bit surprised by the strength of interest expressed and the ease by which we were able to convene an in-person meeting (past experience in the non-profit sector has taught me that finding a date and time for everyone to meet is easily two-thirds the battle!).

This experience of going from a couple of blog posts and a one-off session on the topic to finding an ally, developing some concrete ideas, and quickly connecting with a group of co-conspirators reminded me of the concept of emergence, as described in Getting to Maybe. One of the social innovation examples that the authors drew on was Irish rocker Bob Geldof’s work organizing the Live Aid concert to help alleviate famine conditions in Ethiopia. In describing his experience, Geldof noted that once he started on the project, momentum and energy flowed in, almost beyond his control. “No one particularly stood in my way,” Geldof recalled; “On the contrary, doors impenetrable a week earlier swung open effortlessly.” In Getting to Maybe, this and similar experiences are held up as examples of emergence. Based on ideas from complexity science and in contrast to traditional views of the heroic individual or the deliberate plan that is followed inevitably to a logical conclusion, emergence recognizes that disparate actions from a variety of actors can unexpectedly come together and multiply one’s efforts. New and surprising outcomes often result, while cause and effect can become hopelessly tangled.

In a similar vein, this developing movement on social innovation in Saskatoon has demonstrated emergent properties. Did I plan that my blog posts and the First Tuesday session would be found by someone who would contact me, and that we would subsequently find such a good reception amongst others? Although I may have hoped for such an outcome, I saw my work as simply laying the ground and waiting to see what happened. The situation could have turned out differently – someone else could have contacted me, or perhaps at a different time, or maybe someone else in the city might have started something similar that I would have learned about later and joined in. On the other hand, it would be foolish to think that I had no impact, that this issue was fated to happen regardless of my individual action. If I didn’t write those posts and hosted that discussion, would anything have happened?

Although emergence can seem like something inherently uncontrollable, there are some means to encourage it, or at the very least to be prepared to recognize and act on the opportunities that come along. Getting to Maybe articulates a number of principles in this regard, several of which jump out at me: speaking passionately about the issue, practicing and developing the expression of one’s vision, and supporting intense interactions, networking, and information exchange among those who are interested. I think I was (unconsciously) following those principles through writing on this blog, holding the First Tuesday session, developing my own learning and understanding on the topic (such as by reading and commenting on Getting to Maybe), and building connections with like-minded people in Saskatoon.

Going back to our ideas and plans, I’m going to hold off on sharing specifics for the time being because we’re still very much at the beginning stage: I’m hoping to have something more concrete to announce before the end of the month. That being said, if you are interested in supporting and promoting social innovation in Saskatoon (especially if you are or have connections with powerful strangers), please drop me a line! I think there are some real possibilities here, and I’m excited to see where this venture will lead.

A Little Help

Posted on by
Reply

If nothing else, my recent Achilles’ tendon injury is providing some good food for thought (and blog posts). One obvious effect of being on crutches is that I can’t do everything I used to: driving is right out, going long distances on foot is not an option (and even a short trip outside is dicey given icy sidewalks), and transporting anything that can’t fit in a backpack won’t be happening. At the same time, there are some things I can still do, but I have to ask myself: should I be using up my limited energy and risk further injury? It’s been a learning curve for me to realize that sometimes it’s better to take up an offer of help even if I could theoretically do it myself.

Earlier this week I was having a conversation about the role of non-profit boards of directors and realized the same lesson could easily apply, especially to relatively young and growing organizations. As a non-profit experiences some initial success, starts developing a structure, and becomes incorporated, its founders often become the initial members of the board. At this stage, even when the organization has a staffperson or the resources to contract someone for certain tasks, the board members may be tempted to do the work themselves. Although well-intentioned, this approach can lead to underutilizing the organization’s resources and wasting the time of the board.

As an example, one board I sat on a few years ago was coming out of a time of change and had recently hired a new executive director. At one of our meetings we were talking about an upcoming event and the discussion turned to the relative merits of various floral shops to supply flowers for this event. One of my colleagues on the board became a bit exasperated at this point and asked (quite rightly in my opinion) why we were spending time talking about flowers when we had a capable staff person who could look into the various options and provide a recommendation or even go ahead with a decision? From that point on, whenever our board meetings started going in the direction of doing work that could easily be delegated to our ED or volunteers, somebody would crack a joke about flowers as a reminder of where to spend our time and energy.

There’s a dichotomy between “working boards” that get more involved in day-to-day operations and “strategic boards” with a focus on the bigger picture: a board may play either role through its lifetime as the situation dictates. Regardless of what role it’s currently playing, a board is made up of a finite number of people with finite time and energy, and as an organization and its activities and responsibilities grow, it can only do so much. That’s not to say that board members shouldn’t be contributing their specific expertise or skills for the betterment of the organization. However, just like I’ve had to reach out to others for help with tasks that I could theoretically do myself, a board should likewise take a good look at where it should spend its time and energy and don’t hesitate to ask someone else for a hand when needed.

Another View

Posted on by
2

“Walk a mile in my shoes.” It’s a common metaphor, almost to the point of cliche, but one that I’ve been living, quite literally, for the past few months. Back in November, a moment of inattention on a set of stairs sent me tumbling and ended with a fractured metatarsal on my left foot and me in a walking cast for a month. Just as I was finishing my recovery from that injury, I ruptured my Achilles’ tendon in my other foot playing badminton: that incident was a bit more serious, requiring surgery to repair (which I had just last week) and me looking at a six-month recovery period before I’m back to one hundred percent.

Both of these injuries have had an impact on my daily life, particularly my mobility. Normal tasks became much more difficult, especially with the Achilles’ injury as I’m not yet at the stage where I can put weight on that foot. I’m adjusting to life with crutches, and while I can go short distances, walking to work won’t be an option for some time. Both injuries also created difficulties with events or routines that I had previously taken for granted: take a community event or networking night, which often takes place in crowded spaces with few options for sitting. I also started noticing barriers in the world. Those two steps into a building that I would be hard-pressed to notice before turned into an obstacle. When the local transit agency substituted an older bus onto a route that usually had low-floor vehicles, I noticed (and cursed). Searching for an elevator only to discover it’s out of service (or non-existent) would definitely knock the wind out of my sails.

These two injuries have provided me with just a tiny bit of insight into what life is like with a physical disability that impacts mobility. Unlike many others, I know that this status will (hopefully!) not be permanent, and I have many supports that can help me through the next few months. However, the lessons from these experiences won’t leave me soon – while physical accessibility was always on my checklist when planning an event I’ll definitely be more conscious of those barriers, both now and after I’m back to walking on two feet.

In the non-profit and social services sector, we don’t often have the opportunity to take a literal walk in another’s shoes (though there are some neat roleplaying exercises that try to provide those experiences). When we think about evaluation or program feedback or research, the first thing that usually comes to mind is a questionnaire or survey: while these set questions have their uses, there is the risk that we’re reducing people to a couple of numbers on predetermined scales, boxing them into our predefined concepts, and ignoring important contextual differences. If I was running an organization that helped people with mobility issues but based our practice solely on my experience with temporary injuries, our service would be of little use to seniors, single parents, or people with multiple health issues (just to provide a few examples). Although there would be some commonalities, I would not claim that they will follow the same path, or that what helped me would help them.
Continue reading →