The Biodynamic Association of India is currently being run from Kodaikanal by David Hogg (dhoggn@yahoo.com). In previous blog entries I have written about my experiences with preparations, chromatograms and the calendar. For those of you who found that interesting please note that there is a Biodynamic Association of India and their work statement is below along with the link to the website.

Bio-dynamic agriculture is an advanced form of organic farming, with techniques to ‘farm the air’ as well as farm the soil, and is the oldest organic farming movement practiced in over 20 countries in the world. It includes the normal organic farming practices, such as the use of compost, green manures, and crop rotation. In addition, Bio-dynamic agriculture uses a series of Preparations numbered from 500 to 508 which are based on various mineral, plant, and animal substances. These enhance all the bacterial, fungal and mineral processes that are found in the organic farming system. Placing great importance on the auspicious positions of the moon, sun and planets, a Planting Calendar is used for applying the biodynamic preparations, sowing seeds, planting plants, applying liquid manures, spraying fruit trees and crops, and other farming activities. Experience has shown that use of the Bio-dynamic techniques can make all organic farming processes work more quickly and better. http://www.biodynamics.in/

There will be short pieces on water and water treatment technologies from time to time in this blog. The world is going to experience many different types of problems with respect to fresh water. The availability of fresh water is one issue. The more important one is around what the quality of the water will be like. The quality will decide what treatment needs to be done. With the rise of air pollution around the world even rain water cannot be considered safe, as small particles will form part of the rain drop.

The simplest, cheapest and oldest technology for treating water is filtration. Filtration is a process by which water flows through a membrane like cloth. It is one of the oldest water treatment technologies. It is almost always used as a first step before more complex processes are implemented. Filtration removes macro level contaminants like dust and is very versatile as very large filtration systems can be built at relatively low cost. Filtration systems need regular maintenance otherwise they can end up contaminating the water they are meant to treat. Important design consideration for a water filtration system therefore is how quickly and cheaply can the maintenance be done.

The biggest drawback for a conventional filtration system is that it cannot get rid of chemicals dissolved in the water (like salt). More recent technologies can do this too but then the cost would be higher and the quantity of water the system can treat would be limited and maintenance would be costlier. The following link gives more technical update. http://www.allaboutwater.org/filtration.html

I wrote this piece for Wake Up India – It was printed in the July- September 2011 edition and that has not come online as of yet.

(http://www.ts-adyar.org/content/magazines#Wake_Up_India)

Renewable Energy. Can it work for India?

Most people will confuse energy with electricity. This confusion has spread throughout media and can even be found in government planning. This may have to do with the way electricity is delivered to us through a hole in the wall. Electricity does not have an origin in plain sight as with any other fuel and yet we will equate it to primary energy.

The requirement for energy and electricity depends on where you live in India. We all know that there is an availability problem for electricity in rural areas and that urban area become paralysed without electricity. The primary fuels (energy sources) that are used to create electricity in India are coal, natural gas and oil. These fuels are commonly called ‘fossil fuels’ since they are mined from the earth and represent dead plant life from millennia before and us and are finite in availability.

Fossil fuels are also responsible for the phenomenon known as global warming since it is believed that the carbon dioxide they produce after combustion cannot be absorbed by the plant life of the earth at the same rate that they enter the atmosphere. This build up of carbon dioxide and other combustion products create a green house effect since they absorb and retain heat that should otherwise be rejected to outer space. This is how global warming happens and the end result is violent and unpredictable changes in rainfall, hurricanes and prolonged heating of certain parts of the world.

When you live in rural India your biggest energy consumption is for cooking and then for lighting. Cooking uses biomass in various forms like firewood, agriculture residue, twigs, and dung cake. This biomass for cooking is a sizable chunk (more than 2/3^rd ) of rural energy consumption. However very rarely will this very important source of energy find its way into any planning document. The focus is always on electricity

If you live in urban India and depending whether you are poor or rich your energy consumption will be different. The rich will use electricity and Liquefied Petroleum Gas (LPG) and in some cases kerosene, primarily for cooking. If you are poor in urban India there are good chances of using firewood as the primary cooking fuel. This will explain the numerous firewood depots that can be found near the slums of any metropolis. Firewood is also the fuel of choice for rural eateries.

Most planning documents prepared by governmental or international agencies will talk about electricity and developing means to increase power generation. Though biomass is one of the biggest contributors to the energy consumption of the majority of our population, attempts made for the sustainable supply of biomass is limited.

When we talk about renewable energy what pops into the mind are windmills, solar devices, hydropower and for some strange reason nuclear power. Rarely does the renewable energy sector talk about biomass. Again this is due to the concentration on providing electricity rather than energy. India is also very rich in electricity generated by hydro power which as mentioned is a renewable resource.

The way to understand renewable energy is to think about it as the result of what is available in nature right now, rather than as a finite remnant from a distant past. Thus if we can convert or capture all the sunlight that falls on the world on a daily basis there would be sufficient energy available for everybody’s needs across the globe.

To quote from wikipedia: (http://en.wikipedia.org/wiki/Solar_energy)

“The total solar energy absorbed by Earth’s atmosphere, oceans and land masses is approximately 3,850,000 exajoules (EJ) per year.”  (The solar energy that falls on the world in one hour is equivalent to the world energy consumption in 2002).
“Photosynthesis captures approximately 3,000 EJ per year in biomass. The amount of solar energy reaching the surface of the planet is so vast that in one year it is about twice as much as will ever be obtained from all of the Earth’s non-renewable resources of coal, oil, natural gas, and mined uranium combined.”

There are various problems associated with renewable energy that have prevented its widespread use. One of the biggest challenges is to create technologies that will allow the energy captured to be stored for use at a later time. The other challenges are with the variable nature of the input.

Take for instance wind energy, all of us know that wind speeds are highly variable and can change both speed and direction minute to minute. This gives designers of windmills many challenges. They have to account for the wide variation in speeds and the changes in direction. If you drive past any wind farm where there are multiple windmills installed, there are good chances that you will see rotor blades broken and therefore windmills not working. These catastrophic failures are an indication of how difficult it is to design across a highly variable speed and direction regime. Extremely modern designs make use of advances in material technology that help them build lighter and stronger wind mills that overcome the earlier flaws and perform better.

In the case of solar energy too there are variations in the input due to cloud cover or overcast days where the amount of sunlight reaching the earth is not sufficient for meaningful use. Sunlight can be captured and used for heating or otherwise through photovoltaic cells converted to electricity. However the storage problem is there for both conversions. Some of these conversion devices also tend to be very expensive and have poor efficiencies. Much of the research work around converting sunlight to electricity is around making high efficiency low cost photo voltaic cells. If you think about it all plants are using sunlight and fixing carbon dioxide thus increasing their mass (referred to as biomass). So a plant by its photosynthetic process overcomes the problem of conversion of sunlight, build up of carbon dioxide and storage of energy.

The answer to the question posed in the title is that Renewable Energy is already working in a big way for India. Biomass energy is providing a very large part of the energy requirement of the rural population and for the poor in urban areas. Hydro power provides around 14% of the electricity produced in the country according to the International Energy Agency¹ and 25% according to Asian Development Bank². The potential identified and not exploited could contribute over 60% of electricity requirement but the investment requirements will keep this sector contributing between 20 to 30% of current requirement. In a report by the Global Energy Network Institute³ states “the technical potential of these renewables exceeds the present installed generation capacity.”

The potential of biomass has again been missed in many of these reports. The cooking energy requirement if met by anaerobic digestion of all household waste can free biomass that can potentially create small power generating units of 25kW to 50 kW in each village of India. This means that these villages after meeting their internal requirements could sell power to the national grid.

Mistaking electricity for energy has led to this situation whereby planners have missed on how simple biogas technology at a household level can free up sufficient biomass to create as much energy as the country needs for current and future demand. In simpler terms renewable energy can work for India.

Ref 1. (http://www.iea.org/textbase/nppdf/free/2010/key_stats_2010.pdf pg21)

Ref 2. (http://www.adb.org/Documents/Reports/Hydropower-Devt-India/Hydropower-Devt-India.pdf pg 15)

Ref 3. (http://www.geni.org/globalenergy/research/renewable-energy-potential-of-india/Renewable%20Energy%20Potential%20for%20India.pdf)

Indian farmers use a “panchangam” as a guide to farming activities. “Krishi Panchangam or Agro-almanac or Agro-panchang may be defined as ‘basic astro-agricultural guide book/ calendar published annually, giving calendrical information on various aspects of agriculture and allied activities, basically suggesting region-wise, season-wise and crop-wise crop strategy based on astro meterological predictions, giving auspicious/ inauspicious time for undertaking / avoiding various farm related operations, along with a list of performing religious rites, festival, observation fasts and some non-astrological guidance, primarily useful for the farming communities and persons having interest in agricultural development” (from http://agritech.tnau.ac.in/itk/almanac_types.html ).

The Biodynamic system of agriculture too has an active calendar and largely follows the cycles of the moon. There are three methods for planting by the moon, the Synodic, or waxing and waning cycle, the Sidereal, and the Biodynamic cycle. These can be seen at http://www.thegardenerscalendar.com/Moon_Planting.asp. The Biodynamic Association of India has its own calendar that can be seen at http://www.biodynamics.in/Oct11.htm

The theory behind the use of the panchangam and the biodynamic calendar is based on some scientific truth. The tides on our ocean are controlled by the proximity of the moon to the earth. When the moon is closer its gravitational attraction is higher and it pulls the water towards it causing high tide.  A good animation can be seen at

http://www.mmscrusaders.com/newscirocks/tides/tideanim.htm

In a similar fashion it is suggested that after a seed is put into the ground the gravitational attraction of the moon can make it easier for the germination to take place. The idea is that the moon helps pull the germinating seed to break ground. There is a skeptics view presented at http://skeptics.stackexchange.com/questions/4643/is-there-any-evidence-to-support-the-benefits-of-lunar-planting. It also good to see:

http://www.ehow.com/facts_5731145_do-phases-affect-plant-germination_.html?ref=Track2&utm_source=ask

The use of chromatograms in Biodynamic Agriculture was supposed to show the presence of a spiritual component a “formative force”. The chromatogram is an image developed from an extract of substances like soil, dung or food using simple commonly available chemicals. (see http://www.biodynamics.in/chrom.htm )

I first saw the chromatograms at a Biodynamic workshop held at Kodaikanal, Tamil Nadu, by C Jeyakaran and friends. I was accompanied by Dr. K Perumal and we were both representing Shri A M M Murugapa Chettiar Research Centre (MCRC), Chennai. Perumal and I, were both fascinated by the chromatograms and we discussed the similarity of the technique with conventional chromatography. I suggested to Perumal it would be good to see how important soil nutrients like Nitrogen (N), Phosphorous (P) and Potassium (K) looked like with this technique.

Perumal kept working on this and a couple of years later after I had left MCRC we met again and discussed the technique. By then I was with n-Logue Communications Private Limited. n-Logue was putting up internet kiosks in villages and I was searching for special applications to put on the internet for the kiosks set up in the villages.

I suggested that Perumal should create a library/database of chromatograms. The IIT Madras had a bunch of researchers working on pattern recognition. If the chromatogram kits could be standardized then a chromatogram of the soil at a farmers field could be compared with images in the database using pattern recognition software through an internet connection at the village. The comparison and subsequent interpretation would be used to give guidelines on the quantity and type of nutrient the soil needed to become healthy.

These were just ideas we were throwing around at that time but Perumal continued to work on it and he even got research funding from Department of Science and Technology, Government of India. He worked and built a kit that gives repeated and reliable results. He also created a vast library of images. Validation of results used conventional techniques as well.

Today this technology is to be used to advise farmers on their soil nutrient requirements. Fo more information contact Dr. K Perumal (perumalk@mcrc.murugappa.org) at Shri AMM Murugappa Chettiar Research Centre. (http://www.amm-mcrc.org/)

About 20 years ago my guru Dr. C. V. Seshadri was approached by a young and enthusiastic NRI who wanted to set up Institutes of excellence similar to the IIT’s (Indian Institute of Technology). These institutes would focus on information technology and were to be named IIIT (Indian Institute of Information Technology). The young NRI asked my guru what would be the pressing challenge facing Indian farmers that these Institutes could address. Without batting an eyelid my guru said every farmer wanted to know when his cow would enter estrous.

Much later I understood how useful and really important this piece of technology could be. Farmers who keep small number of cattle need many sources of income to make keeping cattle viable. Chief source of income is the milk. The milk is available as long as the cow is nurturing a young calf. For a farmer it is important to keep getting a yield of milk from the first calving. This means that after the birth of the calf and within a 2 month window the farmer would want the cow to be pregnant again. This is the tricky part. The estrous cycle is about 21 days long. There is a window of less than 6 hours and upto (sometimes) 24 hours within which insemination should take place. Otherwise the farmer has to wait for another 21 days. That means a 21 day loss of milk yield. (see  http://beefrepro.unl.edu/pdfs/estrouscycle.pdf for technical details).

The first technology revolution for cattle was pioneered in India by Bharatiya Agro Industries Foundation (BAIF). “Dr. Manibhai Desai, a freedom fighter came to Urulikanchan near Pune with Mahatma Gandhi in 1946 to manage the Nature Cure Centre to promote community development and sustainable livelihood.   Based on the successful experience, he established the Bharatiya Agro Industries Foundation (BAIF) and motivated small farmers to take up agro-based development, as their business enterprise with sound management. Later in 1989, BAIF was renamed as BAIF Development Research Foundation to re-focus on development research for rural prosperity. Cattle development was the initial preference to serve small farmers effectively.”  (see: http://dev.ikf.in/baif/about_us_about_baif.asp )

BAIF is most well known for their Livestock Development programme. “This flagship programme, launched in early 70’s, is presently serving over 4.4 million families through a network of over 3500 cattle development centres spread over 70,000 villages in the states of Maharashtra, Karnataka, Gujarat, Rajasthan, Punjab, Uttar Pradesh, Uttarakhand, Madhya Pradesh, Andhra Pradesh, Bihar, Orissa, Jharkhand and Chhattisgarh. Each centre covering 15-20 villages in a radius of 10 km, is operated by a local youth, trained to provide services such as artificial insemination, pregnancy diagnosis, vaccination, minor veterinary care, forage production, good feeding and management practices. BAIF’s Central Research Station at Urulikanchan provides frozen semen produced from elite bulls of foreign and Indian breeds to these centres, apart from conducting applied research, technology development, training and supply of critical inputs such as cattle feed, mineral mixture and seeds of elite forage varieties.” (see http://dev.ikf.in/baif/our_programmes_livestock_development.asp)

With establishment of these centres, artificial insemination of cattle has allowed farmers to get their cows pregnant in the shortest time. However they must recognize that their cow is ready to be inseminated. A technology that would inform a farmer that his cow is ready for insemination would go a long way in reducing the loss of revenue from milk, and this is what Dr. C V Seshadri had said was critical to farmers. Any technologists readyto take up the challenge?

When I first came across Biodynamic Gardening I was 22 years old and understood it as a form of kitchen gardening that would maximize yield by close planting and stay healthy with companion planting all done on raised beds and was watchful of planetary rhythms. This was at Vadakadambadi village campus of the Shri A M M Murugappa Chettiar Research Centre outside Mahabalipuram in Tamil Nadu. (You can see the campus: http://www.wikimapia.org/#lat=12.6208577&lon=80.1562285&z=18&l=0&m=b&v=8)

Almost 2 decades later I came to understand that the Biodynamic system was evolved by Rudolph Steiner (and his mysterious companion) who was part of the Theosophical movement led by Annie Besant. I learnt that the system had many other aspects including Biodynamic preparations and analytical techniques using chromatograms.

There are critics of the techniques and also many studies done to compare organic and conventional systems of agriculture with biodynamic agriculture. Most of these comparisons are incomplete and don’t really answer the question of which method is superior. However the results show that the biodynamic system works with minimal inputs and produces yields similar to the other systems of agriculture. Interestingly analysis of the soil in lands where biodynamic system is practiced showed “The Biodynamic farms proved in most enterprises to have soils of higher biological and physical quality: significantly greater in organic matter, content and microbial activity, more earthworms, better soil structure, lower bulk density, easier penetrability, and thicker topsoil.” ( see: http://en.wikipedia.org/wiki/Biodynamic_agriculture)

To any technocrat the methods of making the biodynamic prepartions seem more like magic than science. As a technocrat I have to say that despite my great skepticism the results I saw in the field were exceptional. While inspecting a tea garden where the system was practiced, one of the preparations called bark paste was applied on the tea bush, in a pruned section. In a neighbouring pruned section the paste was yet to be applied. Both sections had been pruned at the same time. The new bud generation on the section where bark paste had been applied was at least double or triple compared to that on the section where it had not been applied. This same garden has consistently yielded higher than the average for the district.

In an earlier post, I had critiqued the research methodologies of 2 surveys of social entrepreneurs, by Ashoka and Intellecap.  Most of my suggestions for improvement were directed towards helping Ashoka and Intellecap better answer the question, “How representative are my results of a larger community (the universe) of social entrepreneurs?”  However, at the end of the post I had also mentioned the concept of causation.  Causation is important when trying to answer a different question, which is, “How likely is it that my results were caused by a particular intervention, rather than by something else?”

From the 25^th to the 28^th of October I attended the Evaluation Conclave 2010 in Delhi, where I had the chance to explore these issues further.  In an evaluation context, it’s often the word attribution, rather than causation, that is used.  Attribution is an issue at many levels.  I’ll try to illustrate this using a hypothetical example of a social enterprise.

This social enterprise produces insecticide-treated bed nets.  Orders for the bed nets are placed through a microfinance institution, and the bed nets are delivered to customers through kirana stores (corner shops).  The social enterprise requires funds for its social marketing campaigns, to cover costs until it reaches break-even, and to develop talent within the organization.  It is able to find donors to meet each of these needs.

An evaluation is conducted of the social enterprise, and it is found that incidences of malaria have decreased in the households in which customers have bought bed nets.  The donor that funded the talent development program wants to know how its money contributed to the reduction in malaria.  However, it is not possible to isolate talent development from the rest of the organization, as the people developed through this program contribute to social marketing, production and all other aspects of the enterprise.

What all three donors can agree on is that they want to know to what extent the reduction in malaria can be attributed to the bed nets.  While it’s easy to assume that bed nets resulted in a reduction in malaria, this may not be the case.  For example, the government could have embarked on a mosquito eradication program, and sprayed the villages in which the bed nets were also sold.

How do we find out the extent to which the reduction in malaria can be attributed to the bed nets?  The strongest evaluation designs that answer this question should have two characteristics.  The first is that they should include both the project group, and comparison (non-equivalent control) group.  The second is that both groups should be “observed” at both the start and end of the project.  If time and budget permit, both groups can also be “observed” during implementation of the project and after it has been operating for some time.

What is it about these evaluation designs that make them appropriate for the social enterprise in the example?

• The project group is “observed” both at the start and at the end of the project. In our example, the project group is households that have bought bed nets.  A figure for the incidence of malaria at the end of the project is meaningless if we do not know what the incidence of malaria was at the beginning.  The only exception is if the incidence of malaria is 0.  However, since this would require all the bed nets sold to be in perfect condition, to be used all the time, and by all members of the family, this is highly unlikely to be the case.
• There is a comparison (non-equivalent control) group, which is also “observed” at the start and end of the project. Even if the incidence of malaria has reduced in our project group, how do we know that this was due to the bed nets and not other factors?  The answer is by selecting a comparison (non-equivalent control) group.  This group should comprise of households that did not buy the bed nets, but are as similar as possible to the households which bought bed nets.  This will enable the evaluator to compare whether malaria declined at a similar rate in the project group and comparison group.  If it did, then it is likely that malaria declined due to factors other than the bed nets.

The bad news about evaluation is that it can be quite complicated to successfully meet the criteria for a strong evaluation design.  The good news is that between strong and weak evaluation designs, there are a range of options that are frequently adequate.  These options are summarized in this overview of Real World Evaluation, a book by Michael Bamberger, Jim Rugh and Linda Mabry.  I was lucky enough to attend a workshop that spanned two days, and was facilitated by Jim Rugh, at the Conclave.  Based on what I learned at the workshop, I’m going to propose my own design to evaluate the social enterprise in my example above.

If you remember, orders for the bed nets are placed through a microfinance institution (MFI).  The social enterprise is still building its distribution network, and has only reached the kirana shops in one geographical area.  However, it asks the MFI to aggregate orders from all of the geographical areas in which it has a presence.  The customers who have ordered the bed nets, but whom the social enterprise’s distribution network has not reached as yet, will serve as the comparison group.

It is important for the evaluation design that the comparison group also comprises of clients of the MFI.  This is because it is likely that MFI clients are more enterprising than other members of their communities.  It is possible that even without bed nets, they have devised other solutions to avoid getting bitten by mosquitoes.  Therefore, in order to understand whether it is bed nets that have made the difference, the households where bed nets were bought must be compared to other enterprising households, and not to the average population.

The “observation” of the project and comparison groups at the start of the project can come in part from the MFI’s records, as they are likely to have already collected a lot of data on their clients.  Any additional data that needs to be collected can be done at the time that orders for the bed nets are placed, before they have been bought and have had the chance to have an effect.

The Real World Evaluation book uses terms such as during, upon completion of, and after the project.  In the case of a social enterprise, it is more likely that the evaluation will be of an ongoing business, than of a project that has a start, middle and end.  However, as one of Paul Polak’s principles of Designing for the Other 90 Percent is that the design should pay for itself in the first year, one year may be a suitable duration to designate as the project period.

Of course, in this case I have chosen the business model of the social enterprise so that it lends itself to a strong evaluation design.  Designing evaluations for actual social enterprises will be less easy.  However, I hope my example demonstrates that while difficult, designing a strong, or at least adequate, evaluation is not impossible, and that practical solutions can be found to real world constraints.

Finally, there are some instances in which attribution may either not be possible or not necessary.  As I discussed earlier, it may not be possible to attribute the effects of a talent development program within a social enterprise to a reduction in malaria.  One example of where attribution may not be necessary is in the evaluation of a sector.  Returning to our example, let us assume that an epidemic hits the Asian continent.  The government is not prepared for this epidemic.  Therefore, in the year the epidemic hits, while all of the government’s planned health programs are successful, the health of the population decreases overall.

In such an event, the overall trend in the health of the population may not be discovered through evaluations of individual interventions or organizations.  In a sectoral review, positing an entire nation as a project group and another nation as a comparison group may not be meaningful.

While the example of the epidemic may seem dramatic and unlikely, at the Conclave there was an interesting exchange between a representative of the Rockefeller Foundation and the Asian Development Bank on whether donors should, in addition to evaluating the interventions that they fund, evaluate whole sectors as well.  However, sectoral evaluations are likely to be quite expensive, complicated, and therefore rare.  For the vast majority of evaluations, issues of attribution and comparison will remain important.

In an earlier post, I had pointed out some weaknesses in the research methodologies of the Ashoka and Beyond Profit surveys of social entrepreneurs.  At around the same time that the results of these two surveys were published, ANDE also published its own impact report.  ANDE is the Aspen Network of Development Entrepreneurs.  How does their report compare to the Ashoka and Beyond Profit surveys?

In brief, the ANDE report is more transparent about how it arrived at its results than the Ashoka and Beyond Profit surveys.  However, there is some variability in the completeness and clarity with which the data is presented.

Although ANDE’s report describes itself as an “impact report”, it is really only two sections that are about ANDE’s work.  In addition, there is no mention of the methodology used to collect the data in these two sections.  Despite the strengths of the report, this is a major weakness.

The two sections that are on ANDE’s work are titled, “ANDE’s Role and the Impact of Our Members”, and “ANDE’s Efforts To Grow The Sector To Scale”.  Of these, the latter section consists of straightforward reporting on ANDE’s activities over the year.  Therefore I will focus only on the section titled, “ANDE’s Role and the Impact of Our Members”.

In this section, it is Figures 9-14 that I would like to comment on.  All figures report results in terms of percentages.  In my earlier post I had said that the problem with using percentages alone is that we have no way of knowing if these results are due to chance, because these results were not tested using a test like the chi-square or t-test.  However, I realized that I need to explain this further.

Chi-squares or t-tests are needed if you are collecting data from a sample of respondents, and using your results to generalize about the larger population that they (supposedly) represent.  Using percentages is not a problem if you have collected data on the entire universe that you are studying.  Look at Figure 13 of the ANDE report, for example.  The figure is titled, “How ANDE Members Fund SGBs’ Financial Needs”.  We are told earlier in the text that ANDE members who invest in SGBs (Small and Growing Businesses) manage 51 funds (23/24).  The N=51 at the bottom of Figure 13 tells us that data from all 51 funds is represented in this figure.

Similarly, Figure 11 is also quite clear.  It shows how many ANDE member funds have a target return range of 0-5%, 5-20% and above 20%.  Here, as in Figure 13, the universe should be 51, as that is the number of funds that ANDE members investing in SGBs manage.  However, it is explained that for this figure N=48, as three ANDE member funds did not provide their target IRR range.  The only flaw in this figure is its titling.  It is titled, “ANDE intermediary target benefit: Percent of ANDE member funds with target IRR range”.  This suggests that respondents were asked a yes/no question, such as, “Do you have a target IRR range?”  Instead, respondents were probably asked a question like, “What is your target IRR range?” and their responses indicate the spread of ranges.  In addition, IRR is not defined in the figure, nor is there a glossary.

Other figures are more ambiguous.  There is no N given for Figure 10.  Since the title of the figure is, “ANDE intermediary target size: Percent of ANDE member funds with target average investment size”, should we assume that N=51?  The title also suggests a yes/no question as with Figure 11.  Figures 12 and 14 both say that N=70, and includes both funds and capacity building providers.  Yet the term “capacity building providers” is not explained anywhere else in this section.

The text in this section is well-supported by footnotes, and doesn’t leave room for misinterpretation.  Where the report says that, “ANDE members have made 2,499 investments in SGBs totaling $830 million (26/27)”, we can see from footnote 15 that this information was collected from all 51 ANDE member funds.  Where it says that, “33 ANDE members spent $96.8 million on technical-assistance activities (26/27)”, footnote 17 tells us that data was not available for the remaining ANDE members.  And the $1.7 billion in additional funding that ANDE portfolio companies have secured is, as footnote 19 tells us, based on reporting by 21 funds.

It is important to note that, as in some of the above examples, ANDE has most likely underreported their results rather than extrapolate where data is not available.  This is to be appreciated.  The one exception I found, where the footnote did not explain the text well, was 16.  If you read the text and the footnote together, it says, “Among those funds that reported historical investment-size information for these past investments, 96 percent of the total number of investments made were investments under $2 million (26/27)”, excluding one ANDE member fund representing 450 investments that did not report quantity under $2 million.  We don’t know what the total number of respondents was (those that reported historical investment-size information), and we also don’t know why one fund was excluded.

In the last year, Ashoka and Intellecap’s Beyond Profit have both conducted surveys of social entrepreneurs. Ashoka describes itself as the global association of the world’s leading social entrepreneurs, whom it elects as Fellows. The Ashoka survey was conducted to understand how their Fellows have changed systems. This phrasing already suggests a bias that Ashoka Fellows have changed systems, one that is carried through in the way the survey was designed and conducted.

Beyond Profit is the social enterprise magazine of Intellecap, a social investment advisory firm. Their survey was conducted to better understand social enterprises, and the people who lead them, in India. While India is described as having a high degree of social entrepreneurship, it lags behind in research in this field. Therefore, initiatives to conduct research on social entrepreneurship in India are much needed. However, this initiative by Beyond Profit, along with the survey by Ashoka, are marred by weak research methods.

The first limitation to both the Beyond Profit and Ashoka surveys is that neither of them go beyond percentages in reporting on the results. For example, the Beyond Profit survey reports on the percentage of respondents who are men (and women), who fall within a certain age group, and whose work falls within a certain sector. The Ashoka survey says of their Fellows that, “these people are incredibly focused on achieving their goals with 93% pursuing their original objective after 10 years. 80% of them are seen as leaders in their field and 90+% of their ideas are replicated by other groups” (4/6). The Ashoka survey also calculates the percentage of Fellows who have changed the system in one of five ways.

The problem is that we have no way of knowing if these results are due to chance. If the authors had conducted a chi square or t-test on their results, it would give us the probability that these results are due to chance. For example, a probability of 0.001 would mean that the results are highly significant in statistical terms, that is, the results are very probably true.

In addition, these tests only work if you have a random sample, and there is good reason to believe that in neither the Ashoka nor Beyond Profits surveys was this the case. Out of all the Ashoka Fellows elected in 1998,1999, 2003 and 2004, the total number of Fellows with current contact information from those years is 315. 172 of those Fellows returned surveys, and this was used to calculate a response rate of 55%. However, we don’t know how many Fellows current contact information is not available for. This is especially important because it is quite likely that the Fellows for whom current contact information is not available may be those whose social enterprises have closed down or are inactive.

The Beyond Profit survey was conducted using an online platform, and was distributed to Intellecap’s database of social entrepreneurs in India, as well as to the networks of Ashoka, Dasra and Unltd India. The sample size for this survey is 118, as that is the number of people who responded. However, as readers we do not know the universe from which this sample was selected. How many people was the survey distributed to? Do Intellecap’s database, as well as the networks of Ashoka, Dasra and Unltd India, cover all social entrepreneurs in India? Alternatively, was the survey only distributed to a sample of social entrepreneurs to begin with?

As it was up to these social entrepreneurs to respond to the survey, it is quite likely that all those who did are similar to one another in some way. For example, they might be all in a younger age group, and therefore more comfortable with using the Internet to respond to surveys. Therefore this sample is unlikely to be random, and most probably suffers from what is known as self-selection bias.

There is one section in the Beyond Profit survey in which it is acknowledged that those omitted from the survey are likely to have influenced its results. The report states that, “One element to keep in mind is what the data doesn’t tell us. Because we didn’t survey people who almost became entrepreneurs, but didn’t follow through because of negative reactions from family, it is difficult to judge just how prevalent family pressure is” (5/7). However, those omitted from the survey are likely to have influenced all of its results, and this is not acknowledged throughout most of the report.

As long as social enterprises in India do not have their own dedicated legal form(s), it may be difficult to know how many social entrepreneurs there actually are. In a context in which a comprehensive database of social entrepreneurs in India is not available, it makes sense for Intellecap to use their own database and other networks to contact potential respondents. In fact, this is a legitimate research method and is known as snowball sampling. Snowball sampling is suitable for qualitative research, where the main purpose is to gain a rich and complex understanding of a specific social context or phenomenon. The problem is that the Intellecap survey seems to use this method for quantitative research, where the emphasis is on eliciting data that can be generalized to other geographical areas or populations.

For example, the Intellecap survey says that “…there are actually more men than women in social enterprise today” (2/4). Similar statements, which generalize from the sample to the universe of Indian social entrepreneurs, are made throughout the report, including with regard to age, experience, motivation, revenue generation and sector. However, without a random sample, and without testing for the statistical significance of the findings, it is misleading to make these generalizations.

Another limitation that both the Ashoka and Beyond Profit surveys suffer from is the lack of triangulation. In the social sciences, triangulation refers to using more than two methods in a study to double (or triple) check the results. The Ashoka survey, in which Fellows were self-reporting on their achievements, could have certainly benefited from triangulation. In the Beyond Profit survey, triangulation could have been particularly useful in cross-checking certain pieces of information, such as on revenue generation.

A final area in which the Beyond Profit survey errs is in the statement:
…Not surprisingly, people from a for-profit background are more likely to choose a
for-profit structure for their own social enterprise. 63% of respondents who came
from a for-profit business background chose to work in a for-profit structure, while
only 17% of people with non-profit experience switched to a for-profit structure (3/5).”
The claim that, “people from a for-profit background are more likely to choose a for-profit structure for their own social enterprise” leads us to believe that there is a correlation between the background of social entrepreneurs and the legal structure they choose for their social enterprises. However, establishing a correlation between these two phenomena requires regression analysis.

The simplest form of regression is linear. If you plot the data collected on a graph, regression analysis will create a single line that best summarizes the distribution of points. The typical distance between the line and all the points indicates whether the regression analysis has captured a relationship that is strong or weak. There is no evidence of regression analysis in the Beyond Profit report.

In addition, the statement discussed above begins with the words “not surprisingly”. This suggests that it is because a social entrepreneur used to work in a for-profit that he / she chose the same legal structure for his / her social enterprise. When one variable (in this case, legal structure of social enterprise) is inferred to be because of another variable (in this case, background of social entrepreneur), this is known as causation. Causation cannot be measured from this study because both the variables were measured together in a setting.

One of my former colleagues described the Ashoka report as impressive and inspiring, and I don’t mean to detract from the achievements of their Fellows by pointing out the weaknesses in the research. A strong research methodology would have made these achievements even more impressive, as they would have been supported by firm evidence.