ANDE Impact Report

In a previous blog post, Devyani Srinivasan wrote about the Dangers of Poor Research. In a follow-up post, she examines another research report brought out by the Aspen Network of Development Entrepreneurs (ANDE), drawing our attention to some limitations of the report. Devyani is an independent consultant. This article was originally published on her personal blog, Devyani Writes.

In an earlier post, I had pointed out some weaknesses in the research methodologies of the Ashoka and Beyond Profit surveys of social entrepreneurs.  At around the same time that the results of these two surveys were published, ANDE also published its own impact report.  ANDE is the Aspen Network of Development Entrepreneurs.  How does their report compare to the Ashoka and Beyond Profit surveys?

In brief, the ANDE report is more transparent about how it arrived at its results than the Ashoka and Beyond Profit surveys.  However, there is some variability in the completeness and clarity with which the data is presented.

Although ANDE’s report describes itself as an “impact report”, it is really only two sections that are about ANDE’s work.  In addition, there is no mention of the methodology used to collect the data in these two sections.  Despite the strengths of the report, this is a major weakness.

The two sections that are on ANDE’s work are titled, “ANDE’s Role and the Impact of Our Members”, and “ANDE’s Efforts To Grow The Sector To Scale”.  Of these, the latter section consists of straightforward reporting on ANDE’s activities over the year.  Therefore I will focus only on the section titled, “ANDE’s Role and the Impact of Our Members”.

In this section, it is Figures 9-14 that I would like to comment on.  All figures report results in terms of percentages.  In my earlier post I had said that the problem with using percentages alone is that we have no way of knowing if these results are due to chance, because these results were not tested using a test like the chi-square or t-test.  However, I realized that I need to explain this further.

Chi-squares or t-tests are needed if you are collecting data from a sample of respondents, and using your results to generalize about the larger population that they (supposedly) represent.  Using percentages is not a problem if you have collected data on the entire universe that you are studying.  Look at Figure 13 of the ANDE report, for example.  The figure is titled, “How ANDE Members Fund SGBs’ Financial Needs”.  We are told earlier in the text that ANDE members who invest in SGBs (Small and Growing Businesses) manage 51 funds (23/24).  The N=51 at the bottom of Figure 13 tells us that data from all 51 funds is represented in this figure.

Similarly, Figure 11 is also quite clear.  It shows how many ANDE member funds have a target return range of 0-5%, 5-20% and above 20%.  Here, as in Figure 13, the universe should be 51, as that is the number of funds that ANDE members investing in SGBs manage.  However, it is explained that for this figure N=48, as three ANDE member funds did not provide their target IRR range.  The only flaw in this figure is its titling.  It is titled, “ANDE intermediary target benefit: Percent of ANDE member funds with target IRR range”.  This suggests that respondents were asked a yes/no question, such as, “Do you have a target IRR range?”  Instead, respondents were probably asked a question like, “What is your target IRR range?” and their responses indicate the spread of ranges.  In addition, IRR is not defined in the figure, nor is there a glossary.

Other figures are more ambiguous.  There is no N given for Figure 10.  Since the title of the figure is, “ANDE intermediary target size: Percent of ANDE member funds with target average investment size”, should we assume that N=51?  The title also suggests a yes/no question as with Figure 11.  Figures 12 and 14 both say that N=70, and includes both funds and capacity building providers.  Yet the term “capacity building providers” is not explained anywhere else in this section.

The text in this section is well-supported by footnotes, and doesn’t leave room for misinterpretation.  Where the report says that, “ANDE members have made 2,499 investments in SGBs totaling $830 million (26/27)”, we can see from footnote 15 that this information was collected from all 51 ANDE member funds.  Where it says that, “33 ANDE members spent $96.8 million on technical-assistance activities (26/27)”, footnote 17 tells us that data was not available for the remaining ANDE members.  And the $1.7 billion in additional funding that ANDE portfolio companies have secured is, as footnote 19 tells us, based on reporting by 21 funds.

It is important to note that, as in some of the above examples, ANDE has most likely underreported their results rather than extrapolate where data is not available.  This is to be appreciated.  The one exception I found, where the footnote did not explain the text well, was 16.  If you read the text and the footnote together, it says, “Among those funds that reported historical investment-size information for these past investments, 96 percent of the total number of investments made were investments under $2 million (26/27)”, excluding one ANDE member fund representing 450 investments that did not report quantity under $2 million.  We don’t know what the total number of respondents was (those that reported historical investment-size information), and we also don’t know why one fund was excluded.

The Dangers of Poor Research

In this blog post, Devyani Srinivasan, an indepdent consultant, takes a look at some of the issues to be aware of while conducting social enterprise research. This post was originally published on her personal blog, Devyani Writes.

In the last year, Ashoka and Intellecap’s Beyond Profit have both conducted surveys of social entrepreneurs. Ashoka describes itself as the global association of the world’s leading social entrepreneurs, whom it elects as Fellows. The Ashoka survey was conducted to understand how their Fellows have changed systems. This phrasing already suggests a bias that Ashoka Fellows have changed systems, one that is carried through in the way the survey was designed and conducted.

Beyond Profit is the social enterprise magazine of Intellecap, a social investment advisory firm. Their survey was conducted to better understand social enterprises, and the people who lead them, in India. While India is described as having a high degree of social entrepreneurship, it lags behind in research in this field. Therefore, initiatives to conduct research on social entrepreneurship in India are much needed. However, this initiative by Beyond Profit, along with the survey by Ashoka, are marred by weak research methods.

The first limitation to both the Beyond Profit and Ashoka surveys is that neither of them go beyond percentages in reporting on the results. For example, the Beyond Profit survey reports on the percentage of respondents who are men (and women), who fall within a certain age group, and whose work falls within a certain sector. The Ashoka survey says of their Fellows that, “these people are incredibly focused on achieving their goals with 93% pursuing their original objective after 10 years. 80% of them are seen as leaders in their field and 90+% of their ideas are replicated by other groups” (4/6). The Ashoka survey also calculates the percentage of Fellows who have changed the system in one of five ways.

The problem is that we have no way of knowing if these results are due to chance. If the authors had conducted a chi square or t-test on their results, it would give us the probability that these results are due to chance. For example, a probability of 0.001 would mean that the results are highly significant in statistical terms, that is, the results are very probably true.

In addition, these tests only work if you have a random sample, and there is good reason to believe that in neither the Ashoka nor Beyond Profits surveys was this the case. Out of all the Ashoka Fellows elected in 1998,1999, 2003 and 2004, the total number of Fellows with current contact information from those years is 315. 172 of those Fellows returned surveys, and this was used to calculate a response rate of 55%. However, we don’t know how many Fellows current contact information is not available for. This is especially important because it is quite likely that the Fellows for whom current contact information is not available may be those whose social enterprises have closed down or are inactive.

The Beyond Profit survey was conducted using an online platform, and was distributed to Intellecap’s database of social entrepreneurs in India, as well as to the networks of Ashoka, Dasra and Unltd India. The sample size for this survey is 118, as that is the number of people who responded. However, as readers we do not know the universe from which this sample was selected. How many people was the survey distributed to? Do Intellecap’s database, as well as the networks of Ashoka, Dasra and Unltd India, cover all social entrepreneurs in India? Alternatively, was the survey only distributed to a sample of social entrepreneurs to begin with?

As it was up to these social entrepreneurs to respond to the survey, it is quite likely that all those who did are similar to one another in some way. For example, they might be all in a younger age group, and therefore more comfortable with using the Internet to respond to surveys. Therefore this sample is unlikely to be random, and most probably suffers from what is known as self-selection bias.

There is one section in the Beyond Profit survey in which it is acknowledged that those omitted from the survey are likely to have influenced its results. The report states that, “One element to keep in mind is what the data doesn’t tell us. Because we didn’t survey people who almost became entrepreneurs, but didn’t follow through because of negative reactions from family, it is difficult to judge just how prevalent family pressure is” (5/7). However, those omitted from the survey are likely to have influenced all of its results, and this is not acknowledged throughout most of the report.

As long as social enterprises in India do not have their own dedicated legal form(s), it may be difficult to know how many social entrepreneurs there actually are. In a context in which a comprehensive database of social entrepreneurs in India is not available, it makes sense for Intellecap to use their own database and other networks to contact potential respondents. In fact, this is a legitimate research method and is known as snowball sampling. Snowball sampling is suitable for qualitative research, where the main purpose is to gain a rich and complex understanding of a specific social context or phenomenon. The problem is that the Intellecap survey seems to use this method for quantitative research, where the emphasis is on eliciting data that can be generalized to other geographical areas or populations.

For example, the Intellecap survey says that “…there are actually more men than women in social enterprise today” (2/4). Similar statements, which generalize from the sample to the universe of Indian social entrepreneurs, are made throughout the report, including with regard to age, experience, motivation, revenue generation and sector. However, without a random sample, and without testing for the statistical significance of the findings, it is misleading to make these generalizations.

Another limitation that both the Ashoka and Beyond Profit surveys suffer from is the lack of triangulation. In the social sciences, triangulation refers to using more than two methods in a study to double (or triple) check the results. The Ashoka survey, in which Fellows were self-reporting on their achievements, could have certainly benefited from triangulation. In the Beyond Profit survey, triangulation could have been particularly useful in cross-checking certain pieces of information, such as on revenue generation.

A final area in which the Beyond Profit survey errs is in the statement:
…Not surprisingly, people from a for-profit background are more likely to choose a
for-profit structure for their own social enterprise. 63% of respondents who came
from a for-profit business background chose to work in a for-profit structure, while
only 17% of people with non-profit experience switched to a for-profit structure (3/5).”
The claim that, “people from a for-profit background are more likely to choose a for-profit structure for their own social enterprise” leads us to believe that there is a correlation between the background of social entrepreneurs and the legal structure they choose for their social enterprises. However, establishing a correlation between these two phenomena requires regression analysis.

The simplest form of regression is linear. If you plot the data collected on a graph, regression analysis will create a single line that best summarizes the distribution of points. The typical distance between the line and all the points indicates whether the regression analysis has captured a relationship that is strong or weak. There is no evidence of regression analysis in the Beyond Profit report.

In addition, the statement discussed above begins with the words “not surprisingly”. This suggests that it is because a social entrepreneur used to work in a for-profit that he / she chose the same legal structure for his / her social enterprise. When one variable (in this case, legal structure of social enterprise) is inferred to be because of another variable (in this case, background of social entrepreneur), this is known as causation. Causation cannot be measured from this study because both the variables were measured together in a setting.

One of my former colleagues described the Ashoka report as impressive and inspiring, and I don’t mean to detract from the achievements of their Fellows by pointing out the weaknesses in the research. A strong research methodology would have made these achievements even more impressive, as they would have been supported by firm evidence.