Inteval Forum

Forum

2012-04-16T13:38:22Z

This article is very interesting.

coach outlet

2012-03-16T09:06:56Z

coach bags outlet

coach outlet

2012-03-16T09:05:35Z

http://www.afar-fiction.com/spip.ph... sdfasdfasdf

Forum

2012-02-16T01:24:39Z

This article is very interesting.

Forum

2011-12-24T11:57:26Z

cash advance cash advance

Forum

2011-01-12T05:01:39Z

Hi Seweryn

There is such an example in a recent study for the European Commission (DG EMPL). The study was about the impact of 'human capital investment' (mainly training) and the methods available for evaluating this impact. This part of the study relies on attribution analyses using couterfactuals.

In addition, we were asked to reflect on how policy-makers learn from impact evaluations and how they could learn more and faster. We answered that question on the basis of a meta-evaluation based on four case studies of impact evaluations. We applied a contribution analysis to this set of information.

Follow the link below and look at Chapter 6 (p95-105)

Contribution analysis

2011-01-05T13:38:01Z

Thanks John.

I like the idea of having several concurrent theories of change or stories during the course of the evaluation. I'm not sure I've ever been in a situation when there were two or more really different stories, but I guess too that I've been tending to build an "official" theory of change at the beginning of the evaluation and adapt it to different stories, rather than considering the very idea of two competing explanations.

Forum

2011-01-04T09:31:54Z

Hi,

I am writing methodological paper concerning labour market research and after hearing promising presentation in Prague and reading some literature I am curious whether there are some examples of application of contribution analysis in labour market area?

Best Regards

Seweryn Krupnik

Center for Evaluation and Public Policy Analysis

Jagiellonian University Poland

One-day training on contribution analysis

2010-12-14T12:13:59Z

The Swiss Evaluation Society (SEVAL) is organising a two day training on Contribution Analysis in Spring 2011 delivered by John Mayne

Contribution analysis

2010-12-02T03:59:25Z

Thomas,

No comment on your first point. i agree the need for counter-evidence and gathering such data should be part of the evaluation design.

Critical discussion on the contribution story is also important and a challenge and a test of robustness. A good reason to have a variety of stakeholders involved, not just the evaluator.

However, there may be cases where is is useful to keep more than one theory of change as part of the process. Henson and Vedung argue the value of multiple stakeholders contribution stories:

Hansen, M. B. and E. Vedung (2010). Theory-Based Stakeholder Evaluation. American Journal of Evaluation, 31(3): 295-313.

On the third point, presumably the prior contribution story and its indicators were rather weak, and hence would not likely perform well. In this case, programme managers are likely quite happy to make the change. Donors or central groups maybe not so. There may be a need to track both sets of measures, showing yes, original expectations were not met, but indeed the programme did have a certain effect.

Three types of causal mechanisms

2010-12-02T03:42:01Z

Astbury and Leeuw discuss mechanisms from an evaluation perspective in a recent article:

Astbury, B. and F. Leeuw (2010). Unpacking black boxes: mechanisms and theory-building in evaluation. American Journal of Evaluation, 31(3): 363-381.

Within this discussion, Jacques is arguing that three classes of mechanisms can be usefully identified.

Three types of causal mechanisms

Jacques Toulemonde — 2010-11-29T16:12:39Z

Let us consider the causal assumption "A causes B". A given item of information may confirm or falsify that assumption in several different ways: 1) A did occur, or not, or partly 2) B did occur, or not, or partly 3) A was the cause of B (fully, partly, or not) 4) Something else than A was the cause of B (fully, partly, or not) 5) A was the cause of B (fully or partly), but only because something else than A made it possible, or A was not the cause of B because something else than A should have made it possible but did not occur.

In the first three cases the causal mechanism is just the assumption itself which is confirmed or falsified.

In the fourth case, the causal mechanism is an alternative explanation such as an external factor or the effect of another policy.

In the fifth case, the causal mechanism is a "condition". The theory does not work, or does not work well as long as something else does not occur. Other terms would be "enabling factor" or "synergy effect".

Up to now, I consider that we do not need more than these three "causal mechanisms" (assumption, alternative explanation, condition) for explaining how a given item of information confirms of falsifies a given causal assumption.

What do you mean by 'causal mechanisms'?

Erica Wimbush — 2010-11-29T15:37:19Z

What exactly do you mean by the term 'causal mechanism' - you use this term frequently in your slides but I'm not certain what you mean by it. Are you using the term in the realist sense of 'mechanism' or do you mean the 'results chain' or sequence of expected changes as set out in a results chain?

Also when you say 'conditions', are you referring to the wider context including external influencing factors?

Training case study on contribution analysis analysis

jtoulemonde — 2010-10-26T08:48:45Z

Case study of my training pre-session at the EES Conference in Prague on 4/10/2010.

One-day training on contribution analysis

jtoulemonde — 2010-10-26T08:43:14Z

Slides of my training pre-session at the EES Conference in Prague on 4/10/2010.

Bibliography

Jacques Toulemonde — 2010-10-13T12:08:07Z

The references are: Mayne, J. (2001). Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly. Canadian Journal of Program Evaluation, 16(1): 1-24.

Mayne, J. (forthcoming) 'Contribution Analysis: Addressing Cause and Effect in Simple and Complex Settings' in R. Schwartz, K. Forss, and M. Marra (Eds.), Evaluating the Complex. Transaction Publishers

Contribution analysis

2010-06-01T14:44:22Z

Dear Jacques and all,

That's quite a task to post the first comment here! These first documents are quite general and should be considered as the basis for any discussion about contribution analysis, and actually John Mayne's ILAC article was the first I used in trying to integrate principles of contribution analysis in my assignments: this was for an evaluation of promotion programmes of agricultural products by DG AGRI of the European Commission we did in 2008, and since then I've tried to include more of this in all my projects.

This is probably not the place for an all-out discussion on contribution analysis yet, so I'll only mention here 3 questions or issues I've stumbled upon in the latest 2 years. I'd be happy to hear how other evaluators have been dealing with these.

1. With contribution analysis, the theory of change becomes a “companion” all along the evaluation. With the DG AGRI evaluation it was probably the first time I spent so much time on the theory of change, first by mentioning explicitly for the first time all assumptions and risks I could reckon, second because I discussed it with as many persons as possible until it could be understood and agreed upon by the majority. We modified this intervention logic based on the case studies thanks to collegial discussion with the consultants involved and in the end I think that the logic we presented was reasonably sound and add a true added value when compared to previous evaluation (which were more impressionistic, let's say). The difficulty we had was to obtain and discuss counter-evidences which the consultants had not been really looking for in the case studies. Later in another evaluation we made sure that consultants would be looking for alternative stories and counter-evidences in the case studies, but still it is more something they would add to their case studies that something that is really part of their work. How do you deal with that in your assignments?

2. Directly related to the previous issue is the fact that creating critical discussion on the provisional contribution story is truly a challenge for many reasons. First, you must have an evaluation team of several persons equally engaged in the work to be able to discuss (which is certainly the case for large evaluations but not for smaller engagements). We have found that case studies are a good way to engage discussion, but it can be more difficult when each consultant in a team is in charge of one data collection instrument, which is often the case. Otherwise, what we can have is more a kind of “Socratic dialogue” between two consultants, which is good but there is still a risk that the evaluation manager makes in the end an expert judgement based on the data available. How do you avoid that kind of risk?

3. A consequence of the revision process of the contribution story is that we have started integrating a step in the evaluation when we actually revise the judgement criteria and indicators that had been decided at the beginning of the evaluation (especially when the programme is evaluated for the first time). This is logical as when the contribution story changes a lot it is not possible to assess the programme as it was initially decided, but it can be difficult to communicate this to a distrustful client. How would you do that?

Regards,

Thomas Delahais

(note: I'm a consultant at Euréval)

A presentation at a recent conference of the UK Evaluation

jtoulemonde — 2010-06-01T14:33:18Z

A presentation at a recent conference of the UK Evaluation

A brief introduction to the approach

jtoulemonde — 2010-06-01T14:21:05Z

A brief introduction to the approach

Contribution analysis

jtoulemonde — 2010-06-01T12:01:57Z

Contribution analysis is a theory-based approach to impact evaluation. It proceeds through a step-by-step investigation into all cause-and-effect assumptions. It does not require developing a counterfactual. It enables evaluators to reach robust conclusions on the impact of public interventions even where they involve complex and open arrays of causal assumptions, for instance in the case of prevention of health problems or improvement of governance.

This discussion is moderated by Jacques Toulemonde and by John Mayne who introduced the concept of ‘contribution analysis' in 1999.

Discussing cutting-edge issues

2010-05-31T09:10:09Z

In this page, INTEVAL group members engage into a series of public discussions about new approaches to (and new viewpoints on) policy and program evaluation. Just register and start interacting. A first pilot discussion has opened on contribution analysis.

Inteval Forum

Forum

propriétaire nom de domaine

coach outlet

coach outlet

Forum

assurance sante comparateur

Forum

Forum

Contribution analysis

Forum

One-day training on contribution analysis

Contribution analysis

Three types of causal mechanisms

Three types of causal mechanisms

What do you mean by 'causal mechanisms'?

Training case study on contribution analysis analysis

One-day training on contribution analysis

Bibliography

Contribution analysis

A presentation at a recent conference of the UK Evaluation

A brief introduction to the approach

Contribution analysis

Discussing cutting-edge issues