Blue Marble Evaluation

As part of my Master of Evaluation studies at the University of Melbourne, I wrote a book review of Michael Quinn Patton’s Blue Marble Evaluation. I think it’s an interesting read, so why not repurpose my assignment as a blog post? 😊 First published in April 2021.

Expanding perspectives on evaluation in the Anthropocene: A book review of Michael Quinn Patton’s Blue Marble Evaluation

This [blog post] critically reviews Michael Quinn Patton’s newest substantive contribution to the evaluation literature, the book Blue Marble Evaluation: Principles and Premises, published early 2020.

Blue Marble Evaluation isan explicit call for tearing down the siloed barriers between the practice of evaluation and the work needed to bring about global systems transformation. Patton proposes that evaluation has a significant role to play in addressing global systems challenges, and that such a role is best played when evaluation and evaluators are explicitly integrated with the work of design and implementation of this transformation. This represents an expansion of Patton’s earlier calls for evaluation to be more integrated into program development, and contrasts sharply with some other views on the role of evaluation and evaluators.

This paper briefly explores key concepts in Blue Marble Evaluation articulated through 15 principles, and investigates Patton’s call for a transformed evaluation practice to evaluate global systems transformation. The paper then considers how this drives an expanding conceptualisation of the role of the evaluator that builds on Patton’s previous works, contrasting views with Scriven and others. Potential criticisms of the Blue Marble Evaluation approach are reflected on. The paper concludes that the Blue Marble Evaluation approach is likely to be controversial among evaluation theorists, but it represents an interesting and timely call to action for evaluation, and a call to move beyond the traditional program-policy-project evaluative mindset.

Blue Marble Evaluation

Michael Quinn Patton is a prolific contributor to the body of evaluation literature, and is the author of hundreds of papers since the 1970s. A staunch advocate of utilisation-focused evaluation (Patton, 1^st ed. 1978; 4^th ed. 2008), Patton is known for his collaborative approach to evaluation and decision-making, positioning evaluation as “too important to be left to evaluators” alone (Patton 1996, p139).

Blue Marble Evaluation: Premises and Principles is Patton’s eighteenth book, published at the start of 2020 (before the advent of the COVID-19 pandemic). The book draws its title from the famous ‘blue marble’ photograph taken by Apollo 17 astronauts showing the whole earth in a single image for the first time, which shows a view “looking beyond nation-state boundaries and across sector and issue silos to connect the global and the local, connect the human and ecological, and connect evaluative thinking and methods with those trying to bring about global systems transformation.” (p vii). The central premise of Blue Marble Evaluation is that evaluation must be a core part of the way we tackle the global problems of the Anthropocene, and it must adapt to meet this challenge. The book is a call for “transforming evaluation to evaluate the transformations necessary to reverse negative climate change effects and make human life on Earth more sustainable and equitable.” (p vii).

Patton posits that evaluation must progress beyond the program and policy lens to understand global change in the context of the Anthropocene, which he defines as the current geological era as characterised by “dramatic and demonstrable human impacts on the planet” (p13). It is well accepted that traditional approaches to policy and problem-solving are inadequate for complex systems (OECD 2020) and so it follows that traditional project-oriented evaluation may serve to reinforce traditional and ineffective approaches, which is the core of Patton’s claims.

The book is structured around 15 principles that describe the evaluative approach: four overarching principles, eight operating principles, and three global systems transformation principles. The principles build on each other to create a strong call to action, and serve to guide prospective ‘blue marble evaluators’ on decision-making and perspective without being prescriptive. Blue Marble Evaluation clearly draws on Patton’s body of work including Utilisation-Focused Evaluation (2008), Developmental Evaluation (2011) and Principles-Focused Evaluation (2018), and he positions Blue Marble Evaluation as an evaluation approach that can be integrated alongside these other approaches. Patton advocates methodological flexibility or “bricolage methods” (p101) and actively rejects the idea that a certain method can be considered a ‘gold standard’ or ‘best practice’ – a Blue Marble Evaluation should use an eclectic mix of methods appropriate to context in a utilisation-focused way, and be prepared to adapt methods in light of emergent context. 

Evaluation for a changing and changed world

In Utilisation-Focused Evaluation (Patton, 2008), Patton states that evaluators should adaptively alter “evaluation questions and designs in light of […] increased understanding of the situation and changing conditions” (p207) – indeed, Christie & Alkin (2004) note that willingness to be flexible is a key element of Patton’s views of evaluation. Rather than adapting only questions and designs, Blue Marble Evaluation could be considered a proposed adaptation of the evaluation (trans)discipline as a whole in light of our increased understanding of the global situation we face, which has changed and is still changing.

A central tenant of Blue Marble Evaluation is the context of the Anthropocene, acknowledging and understanding the global problems that humanity has caused and now faces. Within this context, Patton calls for global systems transformation to address the extensive problems facing our species and our world, using  “evaluative thinking and evaluation processes to contribute to more sustainable and equitable human/ecosystem trajectories for the future,” (Patton, 2020, p14).

Patton calls for an explicit transformation of evaluation to play a vital role in the global transformation required, providing a 20-item “Transforming Evaluation Vision and Agenda” (p189-190), and a further 20 explicit changes required of evaluation linked to current practices (p191-194) in what Picciotto (2020, p474) calls a “spirited, forward-looking disposition”. This includes calls to action to evaluators, such as “Treat all evaluations at whatever level as open systems affected by global trends”, “Integrate design, engagement, implementation, and evaluation,” and “Move beyond evaluator competence to being world savvy through ongoing learning,” (Patton 2020, p189-190). While acknowledging the value of what has been achieved in evaluation to date, Patton decries the “self-limiting” project mindset (p195), asserting that conventional evaluative tools do not work well for evaluating global transformation. In the current era, Patton’s Blue Marble principles declare that all programs should be considered open systems that contribute to global transformation.

Expanding conceptions of the role of the evaluator

The role of the evaluator is somewhat of a preoccupation for evaluation practitioners (see for example Ryan & Schwandt 2002). Patton has not been a lone voice in calling for a closer relationship between evaluation and other elements of program delivery – Mathison 1994, House 2004, and Mertens 2007 are just a handful of many examples. But Blue Marble Evaluation expands this idea further.

In Blue Marble Evaluation, Patton calls for a principles-oriented recognition from evaluators that we all have “skin in the game” (p136). Every one of us lives here on this planet, and we all have a stake in how global systems transform. Patton asserts that evaluators must recognise they are not independent but interdependent and “internal participants in change” (p136).

The question of evaluator independence is debated and ongoing. In 1972, Weiss described “a long history of controversy” on the topic (p21), and in the nearly 5 decades since, this has not been resolved. Michael Scriven is known as a critic of evaluation that does not maintain “distance” from the evaluand (see for example, Scriven 1997). Picciotto (2013) draws strongly on the works of Scriven to advocate strongly for evaluator independence. Mertens (2007) finds fault with distanced approaches to evaluation and explains a need for partnership between researchers and communities.

Interestingly, Patton has drawn on the views of Scriven to support his call for evaluators to recognise and declare their stake in systems transformation. Patton asserts that caring about an issue is not the same as bias, and that caring deeply can in fact push an evaluator to insist upon greater rigour. He quotes Scriven demonstrating that views and preferences are not the same as bias and do not invalidate a study. However Scriven has been famously critical of more integrated evaluator roles. 

Patton and Scriven have clashed on their views of the role of the evaluator. Scriven (1996) has criticised Patton’s developmental evaluation as losing independence, objectivity and credibility. Patton (1996) has said he has felt personally attacked by some of Scriven’s critiques of his approach. Scriven’s (1993) positioning of evaluation as a transdiscipline places the evaluation theorist in the role of defining “tools, methods and approaches” (p8) used across all disciplines, in a way that remains separated and distinct. In Blue Marble Evaluation, Patton claims “a transdisciplinary specialisation” (p73) within a transdiscipline, a concept that cuts across sectors, silos and perspectives for full integration into the global system – a marked difference from Scriven’s conceptualisation of transdiscipline.

Blue Marble Evaluation is a call for evaluators to step into active and proactive roles within a global systems transformation agenda. It expands the role of evaluation from judge of programs to collaborative instigator of global change. Interestingly, despite his 2013 calls for evaluator independence, Picciotto (2020) praises Blue Marble Evaluation, and asserts that this type of evaluation will mean that “the boundaries between evaluation and other knowledge occupations are no longer blurred” (p476).

A new role for evaluation or for the evaluator?

In considering Blue Marble Evaluation’s calls for evaluation transformation, it is important to consider if this represents a change for evaluation, or for evaluators. Arbour (2021) questions the preoccupation of scholars with questions of the role of the evaluator: is there a purpose to examining the role of evaluators rather than evaluation? Does an examination of the roles of evaluators contribute to understanding evaluation theory, or is it self-serving for evaluators?

Evaluation is a practice-oriented field, and so theory is linked to action, and thus actors, or evaluators. “Theory and practice ought to connect,” asserts Weiss (1995, p1), and the connectors are evaluators as practitioners of evaluation. In Blue Marble Evaluation, Patton positions the role of the evaluator as fundamental to global systems transformation. The decision that evaluation concepts should be applied in this way is important, because in a practice-oriented field, application is fundamental to understanding. Restricting conceptualisations of the role of the evaluator to the judgement of merit prevents the application of evaluative thinking to problems where it could make a difference – and this is fundamental to Patton’s calls for transformed evaluation for the Anthropocene.  

Everything needs hammering: is evaluation the right tool for global systems transformation?

In 1997, Pawson and Tilley explained the application of Kaplan’s ‘law of the hammer’ to evaluation – just as a child with a hammer will view everything as needing hammering, so too have evaluators found that everything needs evaluating.

Patton acknowledges it is not always clear if his Blue Marble Evaluation principles constitute an evaluation approach or something else. In explaining his response to this, Patton demonstrates issue with the siloed approach to problems and problem solving that neglects how complex systems interact. Patton’s criticism of siloed projects could be furthered to the practice of evaluation itself, which can reinforce these siloes by adopting a project-based approach. Patton extends this thinking to the siloing of design, implementation and evaluation as separate functions lacking integrated communication. Blue Marble Evaluation, he proposes, sees “evaluation become part of the intervention because it is embedded in and integral to the innovation and change efforts being developed” (p8-9). He adds, “That degree of evaluation engagement is controversial, to be sure, and asks more of the evaluator” (p9).

Despite Patton’s assertions of the need for integration of evaluation with other functions to address global systems transformation, it remains a question: is this an evaluation approach, as he claims, or is this a holistic systems thinking approach using the language of evaluation? Has Patton adopted the role that Pawson and Tilley (1997) describe of the evaluator as a child with a hammer?

Picciotto (2020) claims Patton’s approach will “make evaluation truly matter in a world facing existential risks” (p471) and “enhance the quality, relevance, and impact of the evaluation discipline at a critical time in its history” (p476). In the program implementation world, evaluation is often seen as an optional add-on or a burdensome requirement. By proposing an integrated evaluation as part of the mechanism of global systems change, Patton positions evaluation as a fundamental part of the needed transformation. Blue Marble Evaluation makes integration part of the core of evaluation – and so, while it is fair to say that Patton’s Blue Marble principles can be applied more broadly than within the evaluation transdiscipline alone, it is also fair to say that Blue Marble Evaluation really is about evaluation and the place this vital work must occupy as we move into the future.

Conclusion

Patton’s Blue Marble Evaluation will not be without critics. Just as Patton’s developmental evaluation has been called “an evaluation-related exercise, not […] a different kind of evaluation,” (Scriven 1996, p158), some will understand Blue Marble Evaluation as related to evaluation and using evaluation concepts, but not a true evaluation type or approach.

However, Blue Marble Evaluation’s call for transformed evaluation in the context of the Anthropocene is compelling. By understanding the evaluator’s ‘stake’ as part of the global system, Blue Marble Evaluation calls for a more aware and context-driven evaluation led by evaluators who understand their role within the power structures that drive global systems.

Blue Marble Evaluation’s vision is grand, and the principles described are certainly broadly applicable beyond the discipline of evaluation. For some, this will confirm that the book describes a world view or some other idea rather than an evaluation theory or approach. But in the context of positioning evaluators in the global system, this broad applicability of concepts drives home the central tenant of evaluation being connected to global systems. Evaluation as it is typically practiced can serve to validate and perpetuate existing practice by delivering evidence of impact or effect on a project basis, and defining its bounds within these projects, policies and programs. The transformed evaluation described by Patton could break free of these bounds, with the work of evaluators becoming vital to the creation of more just global systems.

The timing of Blue Marble Evaluation arriving in early 2020 just prior to the COVID-19 pandemic serves to reinforce its message. Never have we been so aware of how integrated and connected our world is. The 1918 influenza pandemic affected half of all countries (Liang et. al 2021); the COVID-19 pandemic has affected every country through not just disease but lockdowns, disruption to trade and global systems, and more. Patton cites the risk of pandemic as one of the realities of the Anthropocene that contextualises Blue Marble Evaluation. This interconnected context drives home the message of the book: we must evaluate with understanding of the ways that every program, policy and project is connected to the greater global system. Without doing this, we as humans will not meet the challenges of the Anthropocene. Evaluation has a significant role to play, and evaluators must now step into this role.

References

Arbour, G. (2021). 5.4 Additional notes in EDUC90715_2021_TM1 Week 5: The role of the evaluator. Retrieved from: https://canvas.lms.unimelb.edu.au/courses/116298/ pages/week-5-the-role-of-the-evaluator?module_item_id=2562469

Christie, C. A., & Alkin, M. C. (2004). An Evaluation Theory tree. In M. C. Alkin (Ed.), Evaluation Roots. pp. 11–57. Sage.

House, E. R. (2004). The role of the evaluator in a political world. The Canadian Journal of Program Evaluation, 19(2), 1.

Liang, S.T., Liang, L.T., Rosen, J.M. (2021). COVID-19: a comparison to the 1918 influenza and how we can defeat it. Postgraduate Medical Journal. Published Online First: 09 February 2021. http://dx.doi.org/10.1136/postgradmedj-2020-139070

Mathison, S. (1994). Rethinking the evaluator role: Partnerships between organizations and evaluators. Evaluation and Program Planning. 17(3), 299-304.

Mertens, D. M. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research. 1(3), 212–225.

OECD Observatory of Public Sector Innovation (2020). Working with Change Systems approaches to public sector challenges (preliminary version). Accessed at https://www.oecd.org/media/oecdorg/satellitesites/opsi/contents/files/SystemsApproachesDraft.pdf

Pawson, R., & Tilley, N. (1997). A history of evaluation in 28 1/2 pages. In Realistic Evaluation. pp. 1–29. Sage.

Patton, M. Q. (1996). A world larger than formative and summative. American Journal of Evaluation, 17(2), 131–144.

Patton, M. Q. (1997). Utilization-focused evaluation (3^rd ed.). Sage Publications.

Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press.

Patton, M. Q. (2018). Principles-focused evaluation: The guide. The Guilford Press.

Patton, M. Q. (2020). Blue Marble Evaluation: Premises and Principles. The Guilford Press.

Picciotto, R. (2013). Evaluation independence in organizations. Journal of Multidisciplinary Evaluation. 9(20), 18-32.

Picciotto R. (2020). Book Review: Blue Marble Evaluation: Premises and Principles. American Journal of Evaluation. 41(3), 471-477.

Ryan, K. E., & Schwandt, T. A. (Eds.). (2002). Exploring Evaluator Role Identity. IAP.

Scriven, M. (1993). The nature of evaluation. In: Hard-won lessons in program evaluation. 1–48. Jossey-Bass.

Scriven, M. (1996). Types of evaluation and types of evaluator. American Journal of Evaluation. 17(2), 151–161.

Scriven, M. (1997). Truth and objectivity in evaluation. In Evaluation for the 21st century: A handbook (pp. 477-500). SAGE Publications, Inc.

Weiss, C. H. (1972). Evaluation research: methods for assessing program effectiveness. Prentice-Hall.

Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. Aspen Institute.