What have we got to show for the billions invested in ICT4D projects?
By and large, we’re not sure because relatively little impact assessment of ICT4D projects has been undertaken; and what has been undertaken often lacks clear framing and rigour.
Impact assessment is therefore pushing its way up the ICT4D agenda. For example, a number of ICT4D agencies have IA programmes; perhaps the biggest being the joint Gates Foundation/IDRC IPAI programme.
As a feed-in to that programme, staff with the University of Manchester’s Centre for Development Informatics created a “Compendium on Impact Assessment of ICT-for-Development Projects”. IDRC – the sponsor for its creation – has given permission for this Compendium to be shared, and it is attached here (2MB .doc file): idrc-ia-for-ict4d-compendium
The Compendium is arranged into three parts:
· Overview – explains the basis for understanding impact assessment of ICT4D projects (including the ICT4D Value Chain), and the different assessment frameworks that can be used.
· Frameworks – summarises a series of impact assessment frameworks, each one drawing from a different perspective.
· Bibliography – a tabular summary of real-world examples of ICT4D impact assessment.
This is an ongoing work, and comments or pointers to similar resources are welcome.
ICTs for Development 
Hey Richard, I imagine its very difficult to “measure impact”. I know, I should look at the document and your framework. There have been other attempts of which I’m aware, one UNESCO BKK study springs to mind. Then we also tried to develop an evaluative framework for ESD which isn’t too dissimilar. Longitudanality (???) issues spring to mind too – how long can we measure impact for? Where do we draw the boundaries? A small seed planted in the right places could one day grow a forest but we won’t be around to witness it. What is the motivation for doing this? To satisfy funders? Is this a worthy cause? Is this reality? Cheers, http://www.timbarker.org
The Compendium does certainly identify those types of questions (see page 1). I think on the issue of timing, the classic problem has not so much been that of attribution when you assess a long time after implementation, but the opposite – that most ICT4D impact assessment takes place too early, before learning curves and secondary impacts have yet had a chance to bed in.
Hi Richard,
This is a great compendium with loads of useful references. It does remind me of a conversation I had a few years ago with Simon Batchelor when he was writing an overview of evaluation frameworks for Infodev. Then as now, there seems to be some tension between what is evaluation and what is research. For me, in the field of ICT4D, there is a bit of a history of critiquing evaluation as if it were research. So is this a guide for researchers or for evaluators? Obviously there is a substantial overlap but the difference matters don’t you think?
-Steve Song
Thank you very much for sharing the Compendium!
I think that for the evaluation of technology-based marketplaces the details about cost benefit analysis, information economics, and enterprise methodologies would be particularly relevant.
Steve – thanks for your comment, and your rather tricky question! Certainly the intent of the compendium is to guide those who want to find out the impact of ICTs on “D”; be they project staff; donor consultants; students; or academics: anyone who wants to know where to start, or feels there might be a better way to approach IA.
I suppose the compendium contains an implicit criticism of some prior “finding out” (though not all – since many examples are quoted). The critique would be that, because it lacks a framework, this work has a) reduced its external credibility; b) reduced its ability to build on and be compared to other work. So, while it may add to data it does not add to knowledge. Put another way, such framework-less evaluation might fulfill the needs of those involved with the project, but its “shareability” is greatly reduced.
Richard
Hi,
Impressive document. Thanks so much for sharing it.
More than a critique, I’d see the lack of a framework as a challenge, as the next step to be achieved. Of course, this is a most needed step 😉
After a few searches, there’s some examples that might enrich this work.
First one is the assessment of USAID’s Last Mile Initiative in Peru:
http://ictlogy.net/bibciter/reports/projects.php?idp=431
Second one is Gudrun Wikander’s SIMBA model:
http://ictlogy.net/bibciter/reports/projects.php?idp=797 [Ed. note: book chapter, not online]
I think you’re seeing each other soon, so don’t forget to ask her about it 🙂
Anything I would be able to contribute, just ask.
Best,
i.
Hi Richard,
Let me thank you for the compendium of impact assessment methodologies. Having read your papers on the use of ANT perspectives, I am surprised that you chose not to mention it in the compendium. Is there some reason?
Secondly, there is also work in the field of ICTD that does not explicitly use any framework but nevertheless the research has the rigor and the findings are invaluable. Let me refer to a paper by Paven Malhotra, a personal favorite.
Paven Malhotra (2005) ‘Technology and the Politics of Corruption; Andhra Pradesh’s CARD experience’ in R.K.Bagga Et.al (2004) ‘State, IT and Development’ Sage Publications, New Delhi.
His paper relies on interviews but incisively nails down the hype associated with technology in Indian governance like no one did till today. As the field of ICTD attempts to focus on impact assessment, it would be helpful if you could at some point pen your thoughts on this blog on what constitutes good qualitative research using some of the examples of the compendium.
Best,
Srinivas Y
Dear Srinivas
Thank you for your comment. I guess the answer about not using ANT in ICT4D impact assessment is probably that a) it’s a potentially very heavyweight theoretical framework that requires a lot of background reading before application; b) its focus has been largely on the process of projects rather than their impacts.
A couple of examples of application to ICT4D e-government projects would be:
Madon, S, Sahay, S, and Sahay S (2004) ‘Implementing property tax reforms in Bangalore: an actor-network perspective’, Information and Organization 14, 269–295 (http://heim.ifi.uio.no/~sundeeps/Publicationsnew/Sundeep%20webpagework/Taxreform%20in%20banglore.pdf)
and
Heeks. R.B. & Stanforth, C. (2007) ‘Understanding e-government project trajectories from an actor-network perspective’, European Journal of Information Systems, 16(2), 165-177 (an earlier variant is available as paper no.17 at: http://www.sed.manchester.ac.uk/idpm/research/publications/wp/igovernment/index.htm)
I can’t comment on Paven’s work since I don’t have access to a copy but it sounds good; point on research also noted.
Richard H
A very interesting paper on a compendium of IA frameworks. I have extensively researched on ICT4D evaluation frameworks, and one thing I do find missing in such evaluations is its comprehensive nature. Evaluation moves beyond just research when it actually builds into the progressive development of ICT4D programmes or projects. Impact is just one of the domains of an evaluation, and it also significantly relies on previous evaluations associated with the process of implementing an ICTD intevention and its outcomes. However, ICTD evaluations tend to focus on just one domain…nowadays, impact. ICTD is still a working hypothesis, and impact can take many many years to show any effect in a community or marginalised area. However, other evaluations can take place, which eventually build up and contribute to an effective impact assessment in the future.I do hope to share this in some publications in the future.