December 12, 2008

"THE TRUTH": EPISTEMOLOGICAL, PRACTICAL AND ETHICAL CONSIDERATIONS IN CASE STUDY RESEARCH

I'll be in Tokyo next week, delivering a paper at a conference of the International Sociological Association. Drawing on examples of research I've done, in both Kenya and North America, the paper discusses issues faced by researchers who undertake critical investigations of the way political power is wielded. It looks at the problem of how to get at "the truth", as well as some obstacles posed by inappropriate research ethics protocols. Following is a brief summary of the paper, or, if you prefer, download the paper itself.

ABSTRACT

Case study research has unjustifiably acquired a reputation for being semi-anecdotal investigation of the small details of individual circumstances, research that is incapable of generating significant empirical or theoretical advances in knowledge. Drawing on a research methods literature, and on my own experience with case study research on two continents, I argue that case studies can serve as an indispensable tool, both for testing the validity of claims made for policy initiatives and political decisions, and occasionally for the generation of theoretically significant insights.

Case studies that involve critical assessments of the use of political power pose serious challenges for researchers because power holders may be in a position to exercise control over both the opinions of a researcher’s interview subjects and some of the contents of the documentary record. The challenge to the researcher is to evaluate the significance of various interviewees’ interpretations of the state of affairs or sequence of events under study, while separating out facts from interpretations, and testing the validity of factual accounts against the representation of those same facts in the documentary record. This study makes the case that power is unlikely to be so absolute as to withstand a scrutiny that includes careful cross-checking of the representation of facts in different, independent sources.

The study then turns to the problems posed by institutional ethics reviews, which, though well-intentioned, inhibit the kind of critical investigation that case study research requires, both by effectively legitimizing any efforts power-holders may make to conceal facts and obfuscate analysis, and by failing adequately to protect subordinates and ordinary people from reprisals for research findings that offend power-holders.


Posted by leo-c at 7:41 AM

March 26, 2008

WHEN WE INTERVIEW RESEARCH SUBJECTS, HOW DO WE KNOW WE'RE GETTING THE TRUTH?

I was having a drink with a couple of colleagues, who, like me, are engaged in case study research, and the conversation turned to interviews. One of my colleagues mentioned some questionable propositions that had been put to him in one of those interviews. "I don't believe that," he said, "but if that's what they say, what are you going to do?"

I knew the answer to the question: triangulation. But it took some excavation of my own research experience to remember how I had arrived at that answer. The idea of triangulation never actually occurred to me. It presented itself, in the form of a puzzle I encountered as a graduate student immersing myself in my first primary research project, a study of Kenya's Million-Acre Settlement Scheme, the starting point for a book I later published under the title Land and Class in Kenya.

The Million-Acre Scheme was a giant development project, in which a large proportion of land previously occupied by European settlers was subdivided and conveyed to African smallholders. As I cooled my heels in Nairobi, Kenya's capital, waiting for research clearance from the Kenya government, I read everything I could get my hands on about Kenya's recent history, its agriculture and rural society, in order to gain an understanding of the research I was embarking upon.

I gathered from my reading that, although there were two types of settlement schemes, high-density and low-density, the differences between them were relatively minor. All of them, I understood from the readings, had been settled by Africans in need of land. The high-density schemes had been less successful than the low-density, but - beyond the fact that low-density settlers made a down-payment on their land, while high-density settlers did not - the reasons were not altogether clear. A colonial agriculturalist, writing about African agriculture, explained them in terms of the fable of the grasshopper and the ant. Some Africans, in his opinion, were simply harder-working and more prudent than others.

Once I got my research clearance, I was in a position to do my own work on the settlement scheme. The heart of my planned research was two sets of interviews with large random samples of the farmers in one high-density and one low-density settlement scheme. As I began to talk to people, it quickly became evident that the differences between the two schemes, and the people who got the land, were dramatic. A preponderance of the low-density settlers were upwardly mobile: absentee landlords in many cases and often owners of multiple pieces of land or businesses in different parts of the country.

The high-density settlers, by contrast, were mostly poor. All settlers, high and low-density, were required to pay for their land in instalments, and although this clearly posed no problems for the low-density settlers, many of their high-density counterparts were scratching a living from manifestly inferior soil, and struggling, or failing, to meet the payments. In short, the literature said one thing and the farmers were telling me something very different. What to do?

My puzzlement drove me to the Kenya National Archives in search of documentary evidence of who these two types of settlers were and how they had come to the land they occupied. The archives helped me to solve the puzzle. The solution is complicated, but, long story short, in doing my two sets of interviews, I had stumbled upon Kenya's rural class system, and found the key to a fascinating and disturbing story of colonial duplicity and manipulation. My discovery, and the follow-up work in the archives, transformed my study from a development politics case into a study of the evolution of Kenya's agrarian class system through the colonial and post-colonial eras.

As you might guess, the farmers and the archives told different aspects of the same story. My archival sources confirmed the observations I made on the ground. The history I was able to read out of them confirmed that the two types of settlement schemes did indeed serve two different social classes and explained how that came to be. In short, the archives and the interview findings agreed that the secondary sources from which I gained my initial impressions were wrong.

My research taught me a great deal about Kenya's history and its agrarian class system. It also focused my mind, once and for all, on the fact that, if you can find three independent sources for the same story, you have a very good chance of working out who's got the story right. That was my first experience with triangulation, which, ever since then, has become my modus operandi. It has served me well. In this blog, I relate two cases, that of the Eaton Centre in Edmonton and the Norwood Bridge in Winnipeg, where triangulation helped me get to the bottom of stories in which facts were being shaded or misrepresented.

In the Kenya case, my interviewees had the story right. More frequently, it is interviews that are least likely to be a reliable source of factual information. As a rule, I treat interviews as an invaluable source of background and context, because no one understands a situation better than those who have lived through it.

However, in assembling the facts - times, places, precise questions at issue - I always bear in mind that the facts cited in interviews are likely to be shaded by the frailties of human memory and the reality that we are all prone to remembering things as we wish they were, not necessarily as they are. (I think I remember this story as it actually was!)

Accordingly, I look for confirmation in documents or secondary sources of any facts I glean from interviews, and teach my research assistants to do the same. Of course, neither documents nor secondary sources are infallible. The best way to get near the truth is to look to all three types of sources for mutual confirmation, and never to settle for less than two.

Therefore, my answer to my colleague, who was wondering what to do if you suspect your interviewees aren't telling the truth is: Triangulate. Always suspect your sources, and always look to other, independent sources for confirmation or contradiction. This becomes particularly important when your interviewees are power holders who, as Jerry Krase points out in a comment below, may control some of the "independent" sources you are using to check on their stories.

It is a reality of political research that it often impinges on power holders, who invariably have definite ideas about what they want you to say in your report of their activities. The realities of power pose a host of problems for researchers, and for the ethics of politics research, as I point out in another blog entry.

But come what may, it's a rare half-truth or lie that survives a determined search for independent sources.

•••••••••••••••••

Want to find out more? Various parts of my Kenya story are documented in detail in:

Christopher Leo, Land and Class in Kenya. Toronto: University of Toronto Press 1984.

Christopher Leo, Land and Class in Kenya (Southern African Edition). Harare, Zimbabwe: Nehanda 1989.

Christopher Leo, "Failure of the 'Progressive Farmer' in Kenya's Million-Acre Settlement Scheme". Journal of Modern African Studies 16 (4) 1978, 619-38.

Christopher Leo, "Who Benefited from the Million-Acre Scheme? Toward a Class Analysis of Kenya's Transition to Independence." Canadian Journal of African Studies. 15 (2) 1981, 201-22.

Here are some useful sources on research methodology, including interviews:

Robert K. Yin, Case Study Research: Design and Methods. Thousand Oaks: Sage, 2003.

Robert E. Stake, The Art of Case Study Research. Thousand Oaks: Sage, 1995.

Raymond L. Gorden, Interviewing: Strategy, Techniques, and Tactics (Fourth Edition). Chicago: Dorsey Press, 1987.

Another source, pitched especially to ethnographic research, is:

John and Lyn H Lofland, Analysing Social Settings: A Guide to Qualitative Observation and Analysis (Third Edition). London: Wadsworth, 1995.


Posted by leo-c at 10:49 AM | Comments (1)