Publications


COMPARATIVE POLICY ANALYSIS SERIES Edited by Ray C. Rist Transaction Publishers Transaction's Comparative Policy Analysis series is a leading forum for the critical examination of cross-national policy issues. General editor Ray C. Rist is advisor for the Economic Development Institute of the World Bank in Washington, D.C. BUDGETING, AUDITING AND EVALUATION : Functions and Intergration in Seven Governments, Andrew Gray, Bill Jenkins, and Bob Segsworth, editors. With a foreword by Ray. C. Rist., (1993-2001) BUILDING EFFECTIVE EVALUATION CAPACITY : Lessons From Practice, Richard Boyle, Donald Lemaire, editors, (1999) CAN GOVERNMENTS LEARN?, Frans. L. Leeuw, Ray C. Rist, and Richard Sonnichsen, editors, (2000) CARROTS, STICKS, AND SERMONS : Policy Instruments and Their Evaluation, Marie-Lousie Bemelmans-Videc, Ray C. Rist, and Evert Vedung, editors, (2003) COLLABORATION IN PUBLIC SERVICES : THE CHALLENGE FOR EVALUATION Andrew Gray, Bill Jenkins, Frans Leeuw and John Mayne, Editors (2003) FROM STUDIES TO STREAMS : MANAGING EVALUATIVE SYSTEMS Ray C. Rist and Nicoletta Stame, Editors (2005) MONITORING PERFORMANCE IN THE PUBLIC SECTOR, John Mayne and Eduardo Zapico-Goñi, editors. With a foreword by Joseph S. Wholey, (2007) OPEN TO THE PUBLIC : EVALUATION IN THE PUBLIC ARENA Peter Dahler-Larsen, Jonathan D.Breul, and Richard Boyle, Editors (2007) MAKING ACCOUNTABILITY WORK : DILEMNAS FOR EVALUATION AND FOR AUDIT Editors : Marie-Louise Bemelmans-Videc, Jeremy Lonsdale and Burt Perrin (Eds.) (2007) PROGRAM EVALUATION AND THE MANAGEMENT OF GOVERNMENT, Ray C. Rist, editor, (1989-1999) PUBLIC POLICY AND PROGRAM EVALUATION, Evert Vedung, (1997-2000) QUALITY MATTERS: SEEKING CONFIDENCE IN EVALUATION, AUDITING AND PERFORMANCE REPORTING Robert Schwartz and John Mayne, Editors, (2005) THE INTERNATIONAL ATLAS OF EVALUATION: Jan-Eric Furubo, Rolf Sandahl and Ray Rist, Editors, (2002) POLITICS AND PRACTICES OF INTERGOVERNMENTAL EVALUATION ,Olaf Rieper and Jacques Toulemonde, editors. With a foreword by Ray C. Rist, (1996) BUDGETING , AUDITING AND EVALUATION : Functions and Intergration in Seven Governments Andrew Gray, Bill Jenkins, and Bob Segsworth, editors. With a foreword by Ray. C. Rist. "Explainations for the failure of 'rational' budgeting reforms abound, yet this edited volume provides a fresh comparative perspective. . . . The case studies reveal the distinctive professional cultures of each function, their institutional boundaries, the impact of changing political and economic conditions on each function, and the importance of constitutional and legal premises." - A. M. Khademian, Choice Table of contents 1-56000-071-6 (cloth) 152 pp. BUILDING EFFECTIVE EVALUATION CAPACITY : Lessons From Practice, Richard Boyle, Donald Lemaire, editors Building effective Evaluation Capacity draws upon the experience of a range of industrialized nations over three decades. A wealth of illustrative case studies of good practice highlight the book. Rather than proposing a single model for building evaluation capacity in new countries or new domains, this comparative research allows readers to apply the findings to their own circumstances. Table of contents 1-56000-396-0 (cloth) 202 pp. CAN GOVERNMENTS LEARN? Frans. L. Leeuw, Ray C. Rist, and Richard Sonnichsen, editors Many citizens view government as inept, arthritic, and dedicated to the preservation of the bureaucratic status quo. The challenge for demographic governments of the 21st century is to become adaptive, flexible, innovative, and creative. This book considers what changes will be necessary if governments are to break away from their traditional ineffective problem-solving techniques, and learn new approaches to find solutions. Table of contents 1-56000-130-5 (cloth) 270 pp. CARROTS , STICKS, AND SERMONS : Policy Instruments and Their Evaluation Marie-Louise Bemelmans-Videc, Ray C. Rist, and Evert Vedung, editors This instructive text offers a comprehensive analysis of the categories and typologies of policy instruments. It classifies sticks, carrots, and sermons - or, more specifically, regulation, economic means, and information - while highlighting the role evaluation plays in the instruments-choice process. Table of contents 1-56000-338-3 (cloth) 325 pp. COLLABORATION IN PUBLIC SERVICES : THE CHALLENGE FOR EVALUATION Andrew Gray, Bill Jenkins, Frans Leeuw and John Mayne, Editors The book examines collaboration in the delivery of public policies and identifies the challenges for policy and propam evaluation. Written by a mix of academics, program managers, evaluators, and auditors, this volume explores the forms and challenges of collaboration in different national contexts. Table of contents ISBN: 0-7658-0183-3 (cloth) 2003 251pp. FROM STUDIES TO STREAMS : MANAGING EVALUATIVE SYSTEMS Ray C. Rist and Nicoletta Stame, Editors The current era is characterized by the emergence of an increasingly global set of pressures for governments to perforrn effectively, not just efficiently, and tIo demonstrate that their performance is producing desired results. Information technology allows enormous quantities of information to be stored, sorted, analyzed, and made available al little or no cost. The result for those in the evaluation community is that, while individual evaluations are still conducted and reported upon, they are a rapidly diminishing dource of information. In the new environment, ever accelerating political and organizational demands and expectations are reframing thinking about the definition of what, fundamentally, constitutes evaluation, and what we understand as its applications. Table of contents ISBN: 0-7658-0287-2 (cloth) 2005 340 pp. MONITORING PERFORMANCE IN THE PUBLIC SECTOR John Mayne and Eduardo Zapico-Goñi, editors. With a foreword by Joseph S. Wholey A host of promising public sector reform efforts are under way throughout the world. Performance monitoring plays a central role in the most important current reform efforts. This book explains why there is an interest in performance monitoring in a given setting, and identifies criteria for improving its design and use. ISBN: 1-56000-292-1 (cloth) 293 pp. ISBN: 1-4128-0632-9 (paper) Table of contents MAKING ACCOUNTABILITY WORK : DILEMNAS FOR EVALUATION AND FOR AUDIT Editors : Marie-Louise Bemelmans-Videc, Jeremy Lonsdale and Burt Perrin (Eds.) (2007) Like honesty and clean water, "accountability" is invariably seen as a "good thing"; few argue against it, and the absence of accountability is associated with many of the greatest abuses in history. Accountability is closely linked with the exercise of power and the legitimacy of policies, and goes beyond technical practices. Issues of accountability involve ethical dilemmas and choices of values that impinge upon the trust needed in viable democracies. Yet in spite of more accountability-related activities than ever before, at the same time there is much debate lamenting what is seen as a lack of actual accountability. This book explores this apparent paradox in addressing questions such as the following: Is there a need for different approaches to establishing accountability or can current arrangements be modified to make them more effective? Are some present practices indeed part of the problem, as many of the chapters in the book suggest, that unintentionally may even aversely affect program effectiveness and improvement? How can systems awash with performance information ensure that at least some of it makes sense to an increasingly wide range of potential users and stakeholders? And how is it that greater accountability and transparency - fairly recent phenomena in many countries - can so quickly have become associated with concerns about perverse incentives and be seen by some as a costly burden? This book - written by twelve authors from seven different countries with many years' experience of participating in and observing accountability processes, is aimed at evaluators, auditors, policymakers and the academic community who are interested in ensuring that accountability practices remain relevant and up to date. The book identifies some of the significant changes that have been affecting the very nature of public sector governance. It proposes a new vision for accountability, focusing on the roles of evaluation and of audit, so that these can be aligned with the current realities of a more complex, outcome-oriented public sector. It includes detailed case studies and synthesizes up to date research evidence drawn from very different governmental systems, and provides practical advice for those involved in the accountability processes so that accountability practices can support learning and improvement as well as provide for greater confidence and trust in the integrity of public sector initiatives. In this way, the book identifies how a repositioning of evaluation and audit can assist in reshaping accountability to meet the demands of the twenty-first century. ISBN: 978-0-7658-0399-3 (cloth) 2007 296 pp. Table of contents OPEN TO THE PUBLIC : EVALUATION IN THE PUBLIC ARENA Peter Dahler-Larsen, Jonathan D.Breul, and Richard Boyle, Editors Open to the Public grows out of concern with evaluation in the public arena and the struggle to understand how best to use the information it generates. Many concepts and models of evaluation, how to undertake it, and how to make it more useful, were developed before government performance became of so much interest to the public. In fact, it is arguable that recent changes in the forms, shapes, structures, and media through which the information developed in the process of evaluation becomes public, require new ways of thinking about its role in society. What is the role of evaluative information in the public arena today? How, when, and under what circumstances does the actual use of evaluative information take place, and what are the forces at play? By compiling and comparing international case studies, this book considers forces that make the information produced in evaluations increasing "open to the public." They provide insights into the many factors that influence evaluation and its use in the public arena. Their case studies include such current topics as: "spin doctoring" of information by the media and this practice's relationship to evaluation studies, the hotly debated issue of school performance, and information about it aired in the public arena, and the controversial link between budget processing and government performance. Table of contents ISBN: 978-0-7658-0372-6 (cloth) 2007 380pp. PROGRAM EVALUATION AND THE MANAGEMENT OF GOVERNMENT Ray C. Rist, editor Contributors to this volume, all of whom are involved in the production and use of evaluation findings, offer a detailed account of the organization and use of evaluation in eight western, democratic countries: Canada, Germany, the United Kingdom, the United States, Denmark, Holland, Norway, and Switzerland. Table of contents 0-88738-297-5 (cloth) 378 pp. PUBLIC POLICY AND PROGRAM EVALUATION Evert Vedung An excellent introduction to the field of policy evaluation, this book examines evaluation as a mechanism for monitoring, systematizing, and grading government activities and their results so that public officials, in their future-orientated work, will be able to act responsibly, creatively, and efficiently as possible. Topics discussed include: "Models of Evaluation"; "impact Assessment as Tryout and Social Experimentation"; and "The Eight-Problems Approach to Evaluation." 1-56000-299-9 (cloth) 240 pp. QUALITY MATTERS: SEEKING CONFIDENCE IN EVALUATION, AUDITING AND PERFORMANCE REPORTING Robert Schwartz and John Mayne (Editors) Transaction Publishers (New Brunswick) 2004 Information - regular, systematic, reliable - is the life-blood of democracy and the fuel of effective management. Surely today there is no problem with information for this is the age of information overload. It pours onto our computer screens and out of our printers. Indeed, many governments claim, often with some justificatjon, to be more open and transparent than ever before. But what if the life-blood is contaminated, or the fuel polluted? Then the body politic sickcens and the engine of public management runs rough. It is the vital issue of the quality of the information we receive that this book addresses. Table of contents ISBN: 0-7658-0256-2 (cloth) 2004 289 pp. THE INTERNATIONAL ATLAS OF EVALUATION Jan-Eric Furubo, Rolf Sandahl and Ray Rist, Editors About ten years after the publishing of a first international comparative review of evaluation across eight nations (Rist, 1990), this is the new state of evaluation affairs in about 700 pages. The book includes more than 20 national chapters, plus several chapters describing some evaluation systems in international organizations. Table of contents ISBN: 0-7658-0095-0 (cloth) 2002 471pp. POLITICS AND PRACTICES OF INTERGOVERNMENTAL EVALUATION Jacques Toulemonde and Olaf Rieper, editors. With a foreword by Ray. C. Rist. A surprisingly large proportion of public interventions are of intergovernmental nature. Autonomous governments and layers of administration increasingly interact at municipal, regional, provincial, state, federal and confederal levels. These interactions range from policy making and co-financing to implementation and monitoring. Meanwhile we often keep thinking of evaluation as a single government process. This book is a landmark work in the area of the evaluation of intergovernmental policies, programmes and projects. The material presented not only provides a systematic theoretical treatment of intergovernmental evaluation, but does so with case material in a cross-national perspective. The book includes contributions by Jan Eric Furubo (Sweden), Mary Henkel (United Kingdom), Linda Morra (USA), Robert Segsworth and Dale Poel (Canada), Olaf Rieper (Danemark), Jacques Toulemonde (European Union), Willy Zimmermann and Peter Knoepfel (Switzerland). Table of contents 1-56000-256-5 (cloth) 132 pp.






BUDGETING, AUDITING AND EVALUATION - FUNCTIONS AND INTEGRATION IN SEVEN GOVERNMENTS Andrew Gray, Bill Jenkins, Bob Segsworth With a foreword by : Ray C. Rist, Table of contents Foreword vii Ray C. Rist Acknowledgement xiii 1. Perspectives on Budgeting, Auditing, and Evaluation: An Introduction 1 Andrew Gray, Bill Jenkins, and Bob Segsworth 2. The Separation of Powers and Political Choice: Budgeting, Auditing, and Evaluation in the United States 19 David Mathiasen 3. Separate Developments: Budgeting, Auditing, and Evaluation in the United Kingdom 41 Andrew Gray and Bill Jenkins 4. Two-Track Processes: Budgeting, Auditing and Evaluation in the Federal Republic of Germany 65 Hans-Ulrich Derlien 5. Out of Sequence and Out of Synce: Budgeting, Auditing, and Evaluation in Canada 87 Bob Segsworth 6. Many Reforms, Little Learning: Budgeting, Auditing, and Evaluation in Spain 115 Eduardo Zapico Goñi 7. Connected or Separated? Budgeting, Auditing and Evaluation in Sweden 139 Rolf Sandahl 8. Muddling Through, Too: Evaluative Auditing of Budgeting in Finland 163 Pertti Ahonen and Esa Tammlin 9. Horses to the Water:85 Budgeting, Auditing, and Evaluation in Seven Governments Andrew Gray and Bill Jenkins Glossary 209 Contributors 211 Index 213 BUILDING EFFECTIVE EVALUATION CAPACITY : LESSONS FROM PRATICE Richard Boyle , Donald Lemaire, editors Introduction: Building Evaluation Capacity 1 Richard Boyle, Donald Lemaire, and Ray C. Rist Part 1: Setting the Boundaries for Evaluation Regimes 1. Locating Evaluation: Anchoring Evaluation in the Executive or the Legislature, or Both or Elsewhere? 23 John Mayne, Stan Divorski, and Donald Lemaire 2. Building Evaluation Capacity within Organizations 53 Richard C. Sonnischsen 3. Evaluation Coverage 75 Yoon-Shik Lee Part 2: Evaluation Linkages 4. Linking Evaluation with Strategic Planning, Budgeting, Monitoring, and Auditing 93 Per Oyvind Bastoe 5. Linking Evaluation Utilization and Governance: Fundamental Challenges for Countries Building Evaluation Capacity 111 Ray C. Rist Part 3: Evaluation Capacity Building 6. Professionalizing the Evaluation Function - Human Resource Development and the Building of Evaluation Capacity 135 Richard Boyle 7. Incentives, Constraints, and Culture-building as Instruments for the Development of Evaluation Demand 153 Jacques Toulemonde Part 4: Institutionalizing Evaluation - The Lessons Learnt 8 Evaluation Capacity Development in Developing Countries: Applying the Lessons from Experience 177 R. Pablo Guerrero 9 Postscript: Evaluation Capacity Building - A Journey without an End 195 Donald Lemaire and Richard Boyle Contributors 199 Index 201 CAN GOVERNMENTS LEARN? Comparative Perspectives on Evaluation & Organization Learning Edited by Frans L. Leeuw, Ray C. Rist& Richard C. Sonnichsen Table of contents Introduction : Evaluations and Organizational Learning: International Perspectives Frans L. Leeuw and Richard C. Sonnichsen 1 Part I National Perspectives 1 Utilizing Evaluation in Organizations: The Balancing Act John Mayne (Canada) 17 2 Learning from Evaluations: The Swedish Experience Jan-Eric Furubo (Sweden) 45 3 Policy Evaluation and the Netherland's Government: Scope, Utilization, and Organizational Learning Frans L. Leeuw and Piet J. Rozendal (The Netherlands) 67 Part II Institutional Perspectives 4 Organizational Learning at the U.S. Office of Personnel Management John A. Leitch and Ray C. Rist (United States) 89 5 Formative Evaluation and Organizational Learning: The Case of the Belgium Postal System Philippe Spaey and Fabienne Leloup (Belgium) 107 6 Effective Internal Evaluation: An Approach To Organizational Learning Richard C. Sonnichsen (United States) 125 Viii Can Governments Learn? Part III When Do Governments Learn? 7 Facilitating Organizational Learning: Human Resource Management and Program Evaluation Bjarne Eriksen (Norway) Edie N. Goldenberg (United States) 145 8 The Preconditions for Learning: Lessons from the Public Sector Ray C. Rist (United States) 189 Contributors 207 Index 211 Carrots, Sticks & Sermons; Policy Instruments & Their Evaluation Edited by Marie-Louise Bemelmans-Videc, Ray C. Rist and Evert Vedung Transaction Publishers, New Brunswick (U.S.A.) and London (U.K.) Table of contents Introduction: Policy Instrument Choice and Evaluation Marie-Louise Bemelmans-Videc Part I : Typology of Instruments 1. Policy Instruments: Typologies and Theories Evert Vedung 2. The Stick: Regulation as a Tool of Government Donald Lemaire 3. The Carrot: Subsidies as a Tool of Government - Theory and Practice Frans L. Leeuw 4. The Sermon: Information Programs in the Public Policy Process - Choice, Effects, and Evaluation Evert Vedung and Frans C.J. van der Doelen 5. The "Give-and-Take" Packaging of Policy Instruments: Optimizing Legitimacy and Effectiveness Frans C.J. van der Doelen Part II: Choice and Context 6. Choosing the Right Policy Instrument at the Right Time: The Contextual Challenges of Selection and Implementation Ray C. Rist 7. Contracting-Out and Program Evaluation: A Case Study Joe Hudson, Richard Nutter, and Burt Galaway 8. Government-Sponsored Enterprises as a Credit Allocation Tool in the United States Edward J. DeMarco and Ray C. Rist 9. The Invisible Problem and How to Deal with It: National Policy Styles in Radiation Protection Policy in The Netherlands, England and Belgium Maarten J. Arentsen 10. The Management and Privatization of Korea's Public Enterprises Nam-Kee Lee 11. Conclusions: Policy Instruments Types, Packages, Choices, and Evaluation Marie-Louise Bemelmans-Videc and Evert Vedung. Contributors Index COLLABORATION IN PUBLIC SERVICES : THE CHALLENGE FOR EVALUATION Andrew Gray, Bill Jenkins, Frans Leeuw and John Mayne, Editors Contents 1. Exploring Evaluation and Collaborative Government: An Initial Perambulation Andrew Gray and Frans Leeuw 2. Networks and Partnerships John Mayne, Tom Wileman and Frans Leeuw 3. Quangos and Collaborative Government Bill Jenkins, Frans Leeuw and Sandra Van Thiel 4. The Third Sector and Collaborative Government Reuven Schwartz 5. Consumers and Collaborative Quality John Mayne and Olaf Reiper 6. Collaboration Through Contract Goran Arvidsson 7. Results Based Collaborative Government John Mayne 8. Auditing Collaborative Government Marie-Louise Bemelmans-Videc 9. Holding Collaborative Government to Account Robert Segsworth 10. Conclusion: Forms and Issues in Evaluation and Collaborative Government Andrew Gray and Bill Jenkins FROM STUDIES TO STREAMS : MANAGING EVALUATIVE SYSTEMS Ray C. Rist and Nicoletta Stame, Editors Table of contents Introduction N. Stame Part I - Channelled streams of evaluative knowledge The "E" in Monitoring and Evaluation - Using Evaluation Information to Support a results-Based Management System J. Kusek and R. Rist, How evaluation can help make knowledge management real B. Perrin Part II - Information systems at work for evaluation Management of evaluative knowledge in national health M. Spinatsch Organizing knowledge: Evidence and the construction of evaluative Information systems P. Dahler-Larsen Managing evaluations in the Netherlands and types of knowledge F. Leeuw Implementing Results-based Management P. O. Bastoe Part III - Thematic evaluations and their uses Complex policies and evaluative streams of knowledge N. Stame Evaluating knowledge about the instruments of government: The Canadian Federal experience P. Eliadis and D. Lemaire Why evaluations sometimes can't be used - and why they shouldn't J. E. Furubo Part IV - Strategic budgeting and streams of knowledge Information Use and Information Communication Technology: what is the real issue? Y. S. Lee Evaluative information in the Norwegian ministries M. Waerness, R. Ovrelid Evaluation knowledge for strategic budgeting X. Ballart, E. Zapico Part V - Multi-study evaluation and the learning organization Evaluation, knowledge management and learning : caught between order and disorder K. Forss and C.Rebien, Patterns of evaluative knowledge creation and utilization within the World Bank M. Marra Post-script Theory of knowledge and use of evaluation: Popper's relevance for the concept of streams of evaluation knowledge O. Rieper Conclusion R.Rist MONITORING PERFORMANCE IN THE PUBLIC SECTOR John Mayne and Eduardo Zapico-Goñi, editors. With a foreword by Joseph S. Wholey Table of contents Foreword Joseph S. Wholey Part I: Performance Monitoring: An Overview 1. Effective Performance Monitoring: A Necessary Condition for Public Sector Reform John Mayne and Eduardo Zapico-Gofii Part II: Designing and Implementing Effective Performance Monitoring 2. Establishing Performance Monitoring: 'The Role of the Central Unit Terje Haugli Nilsen 3. Performance Monitoring for Budget Management: A New Role of the Budget Center Eduardo Zapico-Goni 4. Public Sector Reform Strategy: A GiantLeap or a Small Step? Sylvie Trosa 5. Performance-Monitoring Systems: A Basis for Decisions? Rolf Sandahl 6. Accountability for Program Performance: A Key to Effective Performance Monitoring and Reporting John Mayne Part III: Comparing Performance Monitoring in Policy Areas 7. The Performance-Monitoring System in the Korean Government, With Special Reference to Health Care Myoung-soo Kim 8. A System for Monitoring and Control of Health Services: The Case of Mexico Francisco Javier Casas Guzman 9. Measuring Police Performance Richard C. Sonnichsen 10. Monitoring the Efficiency, Quality, and Effectiveness of Policy Advice to Government John Nicholson 11. Performance Monitoring: Implications for the Future Eduardo Zapico-Goni and John Mayne MAKING ACCOUNTABILITY WORK : DILEMNAS FOR EVALUATION AND FOR AUDIT Editors : Marie-Louise Bemelmans-Videc, Jeremy Lonsdale and Burt Perrin (Eds.) (2007) Table of contents Foreword by Amitai Etzioni Part 1 Modern-day Accountability: Evaluation and Audit challenged 1. Introduction; Accountability - the challenges for two professions Jeremy Lonsdale and Marie-Louise Bemelmans-Videc 2. Accountability: a classic concept in modern contexts. Implications for evaluation and for auditing roles Marie-Louise Bemelmans-Videc 3. Towards a new view of accountability Burt Perrin Part 2 The Changing Accountability of the State 4. Evaluation for accountability: Myth or reality ? John Mayne 5. Walking a tightrope ? - the changing role of state audit in accountability regimes in Europe Jeremy Lonsdale 6. Public sector auditing for accountability: New Directions, New Tricks? Peter Wilkins and Jeremy Lonsdale 7. New wine in old bottles ? When audit, accountability and evaluation meet Tom Ling 8. Checking out? Accountability and evaluation in the British regulatory state Andrew Gray and Bill Jenkins Part 3 New Perspectives on Accountability; Case Studies 9. Results-oriented budgeting and policy evaluation: accountable for learning Peter van der Knaap 10. Making policy interventions more effective: the case for accountability up and accountability down John Owen 11. How does evaluation foster accountability for performance? Tracing accountability lines and evaluation impact within the World Bank and the Italian local health-care providers Mita Marra 12. Adapting evaluation and accountability systems to public management contexts Eduardo Zapico-Goñi Part 4 Conclusion 13. How evaluation and auditing can help bring accountability into the twenty-first century Burt Perrin, Marie-Louise Bemelmans-Videc and Jeremy Lonsdale OPEN TO THE PUBLIC : EVALUATION IN THE PUBLIC ARENA Peter Dahler-Larsen, Jonathan D.Breul, and Richard Boyle, Editors Table of contents Introduction: Evaluation in the Public Arena Peter Dahler-Larsen and Jonathan D. Breul Part One: Evaluation in the Media 1. Evaluation and Political Communication in the French Public Employment Service Jean-Claude Barbier 2. Dissemination of Evaluation Reports in Newspapers: The Case of CEEP Evaluations in Geneva, Switzerland Katia Horber-Papazian and Michael Butzer Part Two: The Preparation and Presentation of Public Evaluation Data 3. Community Perspectives on the Preparation and Reporting of Public Sector Performance Indicators Peter Wilkins 4. Who Decides If and How Evaluation Information Reaches the Public Arena? The Case of School Performance Information in Ireland Richard Boyle 5. The Framing of Public Evaluation Data: Transparency and Openness in Danish Schools. Vibeke Normann Andersen and Peter Dahler-Larsen Part Three: Evaluation Decisions and the Political Agenda 6. Using Evaluative Knowledge for Policy Design: A Case Study of the Swiss Addiction Policy Markus Spinatsch 7. The Growing Use of Performance Information in the Budget Process in the US: the Convergence of Performance Budgeting and the Digital Transformation Jonathan D. Breul Part Four: Citizen Participation 8. Creating Public Arenas for Evaluative Information on the Internet: Experience from e-Democracy in Local Governments Olaf Rieper 9. Does Citizen Participation in the Evaluation Processes Make Any Difference? Evaluation of Government Policies and Programs in the Korean Government Myoung-Soo Kim 10. Evaluation and Exclusion from the Public Arena: the Case of the British Deaf Community Sarah Batterbury 11. Calling Citizens to Deliberate on Disputed Evaluation Questions: Implications for Evaluations Jacques Toulemonde, François Mouterde, and Jean Marc Bernardini 12. Afterword Ray Rist PROGRAM EVALUATION AND THE MANAGEMENT OF GOVERNMENT Ray C. Rist, editor Table of contents Page Figures xiii Tables xiii I. Introduction Managing of Evaluations or Managing by Evaluations : Choices and Consequences Ray C. Rist II. Countries in the First Wave of Evaluation Development 1. Policy and Program Evaluation in the Government of Canada R. V. Segsworth 2. Program Evaluation in the Federal Republic of Germany Hans-Ulrich Derlien 3. Policy Evaluation in British Government: From Idealism to Realism? Bill Jenkins and Andrew Gray 4. The Organization and Function of Evaluation in the United States: A Federal Overview Ray C. Rist III. Countries in the second Wave of Evaluation Development 5. Evaluation in Denmark: The State of the Art Erik Albaek and Soren Winter Vii Viii Contents 6. Policy Evaluation in the Netherlands: Institutional Context and State of Affairs M.L. Bemelmans-Videc, R. Elte, and E. Koolhaas 7. Institutional Aspects of Evaluation in Norway Bjarne Eriksen 8. Switzerland: Moving towards Evaluation Katia Horber-Papazian and Laurent Thevoz IV. Overview and Synthesis 9. Gen-sis and Structure of Evaluation Efforts in Comparative Perspective Hans-Ulrich Derlien Index About the Editor Addresses of Authors QUALITY MATTERS: SEEKING CONFIDENCE IN EVALUATION, AUDITING AND PERFORMANCE REPORTING Robert Schwartz and John Mayne (Editors) Transaction Publishers (New Brunswick) 2004 Contents Foreword: Christopher Pollitt Introduction: Assuring the Quality of Evaluative Information John Mayne and Robert Schwartz 1 Part I: Evaluations 1. Devising and Using Evaluation Standards: The French Paradox 26 Jean-Claude Barbier 2. Instruments and Procedures for Assuring Evaluation Quality: A Swiss Perspective 58 Thomas Widmer 3. Triple Check for Top Quality or Triple Burden?: Assessing EU Evaluations 106 Jaques Toulemonde, Hilkka Summa-Pollitt and Neil Usher 4. Quality of Evaluative Information at the World Bank 141 Patrick Grasso 5. The Netherlands Court of Audit and Meta-research: Principles and Practice 176 Andrea Kraan and Helenne van Adrichem 6.Auditing the Evaluation Function in Canada 203 Bob Segsworth and Stellina Volpe 7. Guidelines and Standards: Assuring the Quality of Evaluation and Audit Practice by Instruction 229 M.L. Bemelmans-Videc Part II: Performance Audits 8.Neat And Tidy…..And 100% Correct: Assuring The Quality Of SAI Performance Audit Work 270 Jeremy Lonsdale and John Mayne Part III: Performance Reports 9.Professionals, Self-Evaluation and Information in the UK: The Higher Education Research Assessment Exercise and Clinical Governance 305 Andrew Gray and Bill Jenkins 10. Decentralization Does Not Mean Poor Data Quality: A Case Study from the US Department of Education 326 Alan Ginsburg and Natalia Pane 11."Believe it or not?": The Emergence of Performance Information Auditing 359 John Mayne and Peter Wilkins 12.Can Supreme Audit Institutions Have a Government-Wide Impact on Performance Information Quality? 399 Stan Divorski 13.Assessment of Performance Reports: A comparative perspective 426 Richard Boyle Conclusion 14.Does Quality Matter? Who Cares About the Quality of Evaluative Information 455 Robert Schwartz and John Mayne About the Authors 491 Index THE INTERNATIONAL ATLAS OF EVALUATION Jan-Eric Furubo, Rolf Sandahl and Ray Rist, Editors Table of contents Introduction : A Diffusion Perspective on Global Developments in Evaluation Jan-Eric Furubo and Rolf Sandahl Part 1: Strong External Pressure, Strong Interna1 Pressure - Denmark, Finland, France, Germany, The Netherlands, Sweden, United Kingdom Evaluation in Danish Governance Erik Albæk and Olaf Rieper Evaluation in Finland Pertti Ahonen, Petri Virtanen, and Petri Uusikyla Evaluation in France Céline Fontaine and Eric Monnier Policy Evaluation in Germany: Institutional Continuation and 'Sectoral Activation Hans- Ulrich Derlien Evaluation in The Netherlands 1990-2000: Consolidation and Expansion M. L. Bemelmans-Videc Coordinated Pluralism-The Swedish Case Jan-Eric Furubo and Rolf Sandahl Policy and Program Evaluation in the United Kingdom: A Reflective State? Andrew Gray and Bill Jenkins Part II: Weak External Pressure, Strong Internal Pressue - Australia, Canada, Korea, Norway, United States The Australian Government: Success with a Central, Directive Approach Keith Mackay Evaluation in the Twenty-First Century: Two Perspectives on the Canadian Experience R. V Segsworth Public Policy Evaluation in Korea: In Search for New Direction Yoon-Shik Lee Norway: Toward a Results-Oriented Government Administration? Ragnhild Ovrelid and Per Oyvind Bastoe The Rise and Fall (and Rise Again?) of the Evaluation Function in the U.S. Government Ray C. Rist and Kathleen L. Paliukas Part III: Strong External Pressure, Weak Internal Pressure - China, Ireland, Italy, Spain, Zimbabwe Evaluation Capacity Building in the People's Republic of China: Trends and Prospects Hong Houqi and Ray C. Rist A Two-Tiered Approach: Evaluation Practice in the Republic of Ireland Richard Boyle Evaluation in Italy: An Inverted Sequence frorn Performance Management to Program Evaluation? Nicoletta Stame Program Evaluation in Spain: Taking Off at the Edge of the Twenty -First Century ? Maria Pazos and Eduardo Zapico-Goni Project and Program Monitoring and Evaluation in Zimbabwe Peter Mazikana and Stephen Brushett PART IV . Weak External Pressure, Weak Internal Pressure Israel, Japan, New Zeland, Switzerland Fragmented Yet Wodespread : The practice of evaluation in Israel Robet Swartz The Art of Policy Evaluation in Japan Kiyoshi Yamaya Evaluation in the New Zealand Public Management System David Turner and Sally Washington Evaluation in Switzerland : Moving toward a Decentralized System Markus Spinatsch PART V : Multi-National Organizations and Associations Developmental Aid Evaluation, The European Union, The Wold Bank Evaluation of Developemental Asistance : Its start, Progress, and Current Challenges Niels Dabelstein and Claud C. Rebien Evaluation in the European Union : Adressing Complexity and Ambiguity Hikka Summa and Jacques Toulemonde Evaluation in the World Bank : Antecedents, Methods, and Instruments Robert Picciotto Conclusion Policy Evaluation in International Comparison Hans-Ulrich Derlien and Ray C. Rist About the Authors Index POLITICS AND PRACTICES OF INTERGOVERNMENTAL EVALUATION Olaf Rieper and Jacques Toulemonde, Editors Table of contents Introduction: The Politics and Practice of Inter-Governmental Evaluation Jacques Toulemonde and Olaf Rieper 1 Evaluation and Autonomy in Sweden Jan Eric Furubo 28 Evaluative Institutions in England and Wales: Weak Versons of Inter-Governmental Evaluation Mary Henkel 48 Evaluation in the United States: Cooperative But Not Inter-Governmental Linda G. Morra 69 Two Cases in Inter-Governmental Evaluation in Canada: "Parallel Play" and Cooperation Without Policy Consequences Robert V. Segworth and Dale H.Poel 92 Evaluation on An EU-Funded Regional Development Program in Denmark -- The Case of Inter-Governmental Evaluation of The North Jutland Technology Program Olaf Pieper 125 Europe and the Member States: Co-operating and Competing on Evaluation Grounds Jacques Toulemonde 151 Evaluation of the Newly-formed Swiss Federal Office of Environmental Protection: The Limits to Environmental Protection Across Two Levels of Government Willi Zimmermann and Peter Knoepfel 174 Inter-Governmental Evaluation: Patterns and Prospects Olaf Rieper and Jacques Toulemonde 199