-
Discussing cutting-edge issues
by Jacques ToulemondeIn this page, INTEVAL group members engage into a series of public discussions about new approaches to (and new viewpoints on) policy and program evaluation. Just register and start interacting. A first pilot discussion has opened on contribution analysis.
Answer this message
-
Contribution analysis
by jtoulemondeContribution analysis is a theory-based approach to impact evaluation. It proceeds through a step-by-step investigation into all cause-and-effect assumptions. It does not require developing a counterfactual. It enables evaluators to reach robust conclusions on the impact of public interventions even where they involve complex and open arrays of causal assumptions, for instance in the case of prevention of health problems or improvement of governance.
This discussion is moderated by Jacques Toulemonde and by John Mayne who introduced the concept of ‘contribution analysis’ in 1999.
Answer this message
-
A brief introduction to the approach
by jtoulemondeA brief introduction to the approach
ILAC_Contribution_Analysis-2.pdf
Answer this message
-
A presentation at a recent conference of the UK Evaluation
by jtoulemondeA presentation at a recent conference of the UK Evaluation
E_Wimbush_UKES_Nov_2009-2.pdf
Answer this message
-
Contribution analysis
by Thomas DelahaisDear Jacques and all,
That’s quite a task to post the first comment here! These first documents are quite general and should be considered as the basis for any discussion about contribution analysis, and actually John Mayne’s ILAC article was the first I used in trying to integrate principles of contribution analysis in my assignments: this was for an evaluation of promotion programmes of agricultural products by DG AGRI of the European Commission we did in 2008, and since then I’ve tried to include more of this in all my projects.
This is probably not the place for an all-out discussion on contribution analysis yet, so I’ll only mention here 3 questions or issues I’ve stumbled upon in the latest 2 years. I’d be happy to hear how other evaluators have been dealing with these.
1. With contribution analysis, the theory of change becomes a “companion” all along the evaluation. With the DG AGRI evaluation it was probably the first time I spent so much time on the theory of change, first by mentioning explicitly for the first time all assumptions and risks I could reckon, second because I discussed it with as many persons as possible until it could be understood and agreed upon by the majority. We modified this intervention logic based on the case studies thanks to collegial discussion with the consultants involved and in the end I think that the logic we presented was reasonably sound and add a true added value when compared to previous evaluation (which were more impressionistic, let’s say). The difficulty we had was to obtain and discuss counter-evidences which the consultants had not been really looking for in the case studies. Later in another evaluation we made sure that consultants would be looking for alternative stories and counter-evidences in the case studies, but still it is more something they would add to their case studies that something that is really part of their work. How do you deal with that in your assignments?
2. Directly related to the previous issue is the fact that creating critical discussion on the provisional contribution story is truly a challenge for many reasons. First, you must have an evaluation team of several persons equally engaged in the work to be able to discuss (which is certainly the case for large evaluations but not for smaller engagements). We have found that case studies are a good way to engage discussion, but it can be more difficult when each consultant in a team is in charge of one data collection instrument, which is often the case. Otherwise, what we can have is more a kind of “Socratic dialogue” between two consultants, which is good but there is still a risk that the evaluation manager makes in the end an expert judgement based on the data available. How do you avoid that kind of risk?
3. A consequence of the revision process of the contribution story is that we have started integrating a step in the evaluation when we actually revise the judgement criteria and indicators that had been decided at the beginning of the evaluation (especially when the programme is evaluated for the first time). This is logical as when the contribution story changes a lot it is not possible to assess the programme as it was initially decided, but it can be difficult to communicate this to a distrustful client. How would you do that?
Regards,
Thomas Delahais
(note: I’m a consultant at Euréval)
Answer this message
-
Contribution analysis
by John MayneThomas,
No comment on your first point. i agree the need for counter-evidence and gathering such data should be part of the evaluation design.
Critical discussion on the contribution story is also important and a challenge and a test of robustness. A good reason to have a variety of stakeholders involved, not just the evaluator.
However, there may be cases where is is useful to keep more than one theory of change as part of the process. Henson and Vedung argue the value of multiple stakeholders contribution stories:
Hansen, M. B. and E. Vedung (2010). Theory-Based Stakeholder Evaluation. American Journal of Evaluation, 31(3): 295-313.
On the third point, presumably the prior contribution story and its indicators were rather weak, and hence would not likely perform well. In this case, programme managers are likely quite happy to make the change. Donors or central groups maybe not so. There may be a need to track both sets of measures, showing yes, original expectations were not met, but indeed the programme did have a certain effect.
Answer this message
-
Contribution analysis
by Thomas DelahaisThanks John.
I like the idea of having several concurrent theories of change or stories during the course of the evaluation. I’m not sure I’ve ever been in a situation when there were two or more really different stories, but I guess too that I’ve been tending to build an "official" theory of change at the beginning of the evaluation and adapt it to different stories, rather than considering the very idea of two competing explanations.
Answer this message
-
-
-
Bibliography
by Jacques ToulemondeThe references are: Mayne, J. (2001). Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly. Canadian Journal of Program Evaluation, 16(1): 1-24.
Mayne, J. (forthcoming) ’Contribution Analysis: Addressing Cause and Effect in Simple and Complex Settings’ in R. Schwartz, K. Forss, and M. Marra (Eds.), Evaluating the Complex. Transaction Publishers
Answer this message
-
One-day training on contribution analysis
by jtoulemondeSlides of my training pre-session at the EES Conference in Prague on 4/10/2010.
ca_one_day_training_CC-3.pdf
Answer this message
-
What do you mean by ’causal mechanisms’?
by Erica WimbushWhat exactly do you mean by the term ’causal mechanism’ - you use this term frequently in your slides but I’m not certain what you mean by it. Are you using the term in the realist sense of ’mechanism’ or do you mean the ’results chain’ or sequence of expected changes as set out in a results chain?
Also when you say ’conditions’, are you referring to the wider context including external influencing factors?
Answer this message
-
Three types of causal mechanisms
by Jacques ToulemondeLet us consider the causal assumption "A causes B". A given item of information may confirm or falsify that assumption in several different ways: 1) A did occur, or not, or partly 2) B did occur, or not, or partly 3) A was the cause of B (fully, partly, or not) 4) Something else than A was the cause of B (fully, partly, or not) 5) A was the cause of B (fully or partly), but only because something else than A made it possible, or A was not the cause of B because something else than A should have made it possible but did not occur.
In the first three cases the causal mechanism is just the assumption itself which is confirmed or falsified.
In the fourth case, the causal mechanism is an alternative explanation such as an external factor or the effect of another policy.
In the fifth case, the causal mechanism is a "condition". The theory does not work, or does not work well as long as something else does not occur. Other terms would be "enabling factor" or "synergy effect".
Up to now, I consider that we do not need more than these three "causal mechanisms" (assumption, alternative explanation, condition) for explaining how a given item of information confirms of falsifies a given causal assumption.
Answer this message
-
Three types of causal mechanisms
by John MayneAstbury and Leeuw discuss mechanisms from an evaluation perspective in a recent article:
Astbury, B. and F. Leeuw (2010). Unpacking black boxes: mechanisms and theory-building in evaluation. American Journal of Evaluation, 31(3): 363-381.
Within this discussion, Jacques is arguing that three classes of mechanisms can be usefully identified.
Answer this message
-
-
-
One-day training on contribution analysis
byThe Swiss Evaluation Society (SEVAL) is organising a two day training on Contribution Analysis in Spring 2011 delivered by John Mayne
Answer this message
-
-
Training case study on contribution analysis analysis
by jtoulemondeCase study of my training pre-session at the EES Conference in Prague on 4/10/2010.
case_study.pdf
Answer this message
-
-
-
Forum
by Seweryn KrupnikHi,
I am writing methodological paper concerning labour market research and after hearing promising presentation in Prague and reading some literature I am curious whether there are some examples of application of contribution analysis in labour market area?
Best Regards
Seweryn Krupnik
Center for Evaluation and Public Policy Analysis
Jagiellonian University Poland
Answer this message
-
Forum
by Jacques ToulemondeHi Seweryn
There is such an example in a recent study for the European Commission (DG EMPL). The study was about the impact of ’human capital investment’ (mainly training) and the methods available for evaluating this impact. This part of the study relies on attribution analyses using couterfactuals.
In addition, we were asked to reflect on how policy-makers learn from impact evaluations and how they could learn more and faster. We answered that question on the basis of a meta-evaluation based on four case studies of impact evaluations. We applied a contribution analysis to this set of information.
Follow the link below and look at Chapter 6 (p95-105)
Answer this message
-
Forum
bycash advance cash advance
Answer this message
-
-
Forum
byThis article is very interesting.
Answer this message
