China Medical Board News

It’s a great honor for me to share some reflections about this first conference on the current state and future directions of global health metrics and evaluation. The privilege is all the greater because of many friends and special people here, including co-authors of the classic 1980 Lancet paper by Murray, Murray, Murray, and Murray — where Chris is not the first author! I can well imagine how proud Chris’ parents are of his achievements. More importantly, I want to offer my deep affection to Felicia Knaul and Emmanuela Gakidou who are demonstrating enormous bravery in overcoming health adversity. We love you both!

We are more than 300 participants with diverse backgrounds who have spent two days together in (sunny and rainless!) Seattle — the new “world center” in global health — rapidly overtaking Paris, London, New York, Geneva, or even Boston!

We are here bearing witness to the birth of a new Institute, the IHME. Health metrics and evaluation are hugely important endeavors, because as founding director Chris articulated at the outset, the level of funding and volume of activities in global health have accelerated dramatically. Metrics and evaluation are essential for learning more about what works, what doesn’t, and how we can do better. Chris also noted that institutionally the founding of IHME signals a shift from the inter-governmental “bureaucratic” mode of data-information-analysis of the 1980s to more pluralistic “open scientific” modes of the 21st century. Lancet editor Richard Horton proposed that IHME should pave the path for mobilizing data and information to inform public reasoning and discourse, foundational practices in democratic societies.

Metrics and evaluation, I would add, are also timely because of changing norms, high politics, and big business. In this era of globalization with increasing recognition of human inter-connectedness, the global public is increasingly distressed and intolerant of unnecessary human suffering and premature death. Gross inequalities in human survival violate the fundamental human right of universal access to basic health services. Health consequently is finding itself the beneficiary of statesmen who wish to demonstrate their global political leadership. The UN Millennium Development Goals show how health has assumed the position of common currency for cooperative international action. President Bush’s legacy — some say his only legacy! — is his commitment to PEPFAR for controlling HIV/AIDS and more recently malaria. Other statesmen are not far behind — UK’s PM Gordon Brown launched the international health partnership; Norway’s PM Stoltenberg announced global action for children’s health; and Japan’s PM Fukuda is proposing to expand global health investments in the upcoming July G8 meeting. These political commitments are transforming global health into a “big business.” Foreign health aid has rapidly increased to nearly $20 billion annually as part of a $4 trillion global health care industry. The work of metrics and evaluation is to assess and improve the performance of this industry.

The conference’s eight sessions began by considering mortality-cause and risk-determinants, followed by evaluation of interventions and national work at the country level. Health systems performance, priority setting, coverage, and quality constituted the remaining sessions. The format has been structured to promote dialogue — each session opening with a thematic presentation followed by commentaries leaving time for open discussions. Just as Prabhat Jha described the “randomization of politicians,” conference organizers have “randomized participants” into informal round tables. Our seating preference, of course, was “self-selected” reflecting “clustering” according to “social networking!” Irrespective of location, all participants were engaged with floor discussions pushing time constraints. I was among those who thirsted for a little more depth and focus at each session’s ending.

As I just flew in from Mandalay Myanmar, my jet lag has been waking me up around 3 am in the mornings. Jet lag was fortunate because the early rising gave me quiet time to grapple with my worries about how to assemble a few coherent reflections. I knew that it would be foolhardy to try a comprehensive conference summary. What came to mind was a sign that Albert Einstein kept in his office.

“Not everything that counts can be counted.
Not everything that is counted counts.”

Bolstered by Einstein’s wisdom, let me pose some key questions. What is counted and how are they counted — the science or field of metrics and evaluation? Who does the counting and who uses the counts — the actors of metrics and evaluation? Why do the counting and what is the counting for — the purposes of metrics and evaluation? These questions constitute the three major reflections — the field, the actors, and the purposes.

The Field

At the outset, we have to ask the question: Why did the founders of IHME call the institute “metrics” and “evaluation”? This week’s Lancet article by Julio Frenk and Chris Murray describe the rationale. The Latin root of the term metrics has the property of both “measurement” as well as “to take measures.” Evaluation describes “assessment” together with the intent to “improve action.” Bringing together metrics and evaluation, thus, doubly underscores appraisal of health as well as action for health.

This dual conceptualization is important, because as Chris Murray reported at the Institute’s opening board meeting, IHME is not simply yet another academic department. As an autonomous institute in a university, IHME has a vision and the mission to advance global health — health impact! This ultimate objective of bringing together metrics and evaluation for impact was nicely captured by Rafael Lorenzo who described the “value chain” of data-information-knowledge-action. Zulfiqar Bhutta agreed in a different way by lamenting the gap in this value chain where knowledge is insufficiently translated into action in so many parts of the world.

A basic question for the conference is whether “metrics and evaluation” can be brought together to establish a scientific field? This challenge is by no means settled, but the conference surfaced several promising signs — defining the core scientific disciplines, developing a culture of science, and building a community of scientists and practitioners.

The field is not dominated by any single discipline. At its core, metrics and evaluation are interdisciplinary — harnessing such established disciplines as epidemiology, demography, economics, policy sciences, medicine, and public health. Despite concentration on quantitative sciences, qualitative methods are also important as in anthropology, management, and case study “vignettes” as described by Peter Smith and John Peabody. One can anticipate that this hybridization process will generate over time new disciplines, just as demography and epidemiology emerged from statistics. Metrics and evaluation also has core skills, competencies, tools and methods — thus much talk about “gold standards” (as well as I presume “tarnished” work). As a field, metrics and evaluation has a core body of knowledge that is continuously changing and constantly growing.

As demonstrated at this conference, the work of metrics and evaluation can be imbedded in a culture of science. Ultimately, such work is a search for truth. Srinath Reddy described the journey of “moving upstream” from mortality, to causes of death, to risk factors, to social determinants — in search for understanding and identification of prevention. Alan Lopez was complimented for his frank incorporation of “uncertainty,” just as Majid Ezzati expressed scientific “skepticism” over efforts to classify risk factors. Indeed, it could be argued that the establishment of IHME will promote the positive culture of science in the emerging field of metrics and evaluation — acceptance of scientific uncertainty, welcoming of scientific skepticism, and constantly revisiting basic scientific assumptions.

Finally, this conference demonstrated that metrics and evaluation can bring together a community of scientists and practitioners. A scientific community is created when participants are able to agree on shared terminologies, definitions, concepts, and language. And a community grows if participants gain something for the time and effort invested in membership. Underlying these processes are also shared community values. These aspects will evolve and strengthen as the community grows. Steve Tollman challenged the community on its values. “Information can be hot” in his country of South Africa. Will the metrics and evaluation community join in “solidarity” to defend its members’ work in the face of political controversy?

The Actors

Most conference participants are producers of data, information, and knowledge. We are mostly researchers! But the users of information, policy-makers and managers, were repeatedly cited as key actors. Users will view metrics and evaluation differently than producers, and each will shape priorities differently. It is the connection between producers and users that must be fostered.

For managers, there is the adage “if you don’t count it, you can’t manage it!” Jim Kim in pushing to create a new field of the “delivery” and “implementation” sciences was essentially adopting a managerial approach. But most knowledge producers also want to influence health policies. Julio Frenk warned that the link between appraisal and policy-making carries political risks — a double edge sword of producers hoping to influence decisions but also the hazard that the messenger of unwelcomed news may be blamed by the users.

There are two other no less important actors — what I will call “brokers” and “audiences.” Brokers are those who transform and communicate metrics and evaluation results — including journal editors like the Lancet, reporters and journalists, and media specialists. Brokers manage the translation and dissemination of the work by both producers and users to wider audiences.

Considering brokers and audiences brings forth additional challenges. Where is the line between science and advocacy? Some have used the term “evidence-based advocacy,” but are the boundaries, limits, and opportunities clear? How does the “climate” or “receptivity” affect audience uptake and use of information? Richard Horton vividly described the markedly different responses the Lancet received of two articles on health consequences of the Iraq war — hostile letters in 2003 and praiseworthy letters in 2006! Brokerage is essential for public understanding. This process is more than simply transparency of data and analysis which is important for scientific quality. Accountability depends upon public understanding and thus the capacity to communicate metrics and evaluation to the lay public. Peter Smith vividly underscored these translational challenges when describing news reports that called his policing studies “baffling”!

Actors are more than individuals; they are also institutions — such as the UN system, academia, agencies, non-governmental organizations, think tanks, and brokering bodies. Each of these has mission, capacities, and interests. Central to the field is interaction among institutional actors — characterized by such descriptions as cooperation, collaboration, complementarity, competition, duplication, and fragmentation. Tis Boerma of WHO argued for exploiting more fully new opportunities for collaboration. He worried that such may be missed because of insufficient investment in cultivating harmonious relationships, partnerships, joint work, and trustful communications. Institutional “adjudication” may also necessary, as described by Richard Horton when he spoke of needs during moments of controversy over contentious research results.

The 800 pound institutional gorillas in the room, of course, are the donors! Many complained about the shortage of resources; some observed that “money calls the shots!” Questions, priorities, and the object of research are often dictated by the investor rather than the scientist. While these are legitimate concerns, we should also applaud those who finance this grossly neglected and under-resource field. The bigger donor problem is under-recognition and under-funding! We want to applaud and thank the Bill and Melinda Gates Foundation for taking the gamble by betting on Julio Frenk and Chris Murray in launching the IHME. We want to make metrics and evaluation good investments for donors, thereby encouraging others to follow Gates.

A climate of mutual respect and trust is essential for building a sound field and a healthy community. Actors in metrics and evaluation, as described as Rafael Lorenzo, are like soccer players — too many players kicking around too many balls on a playing field without goal posts! Suwit reminded us that the playing field is not level! The voice from poorer developing countries is often not heard, or even listened to. The essentiality of building capacity in poorer countries is neglected. Bias of the playing field is due to weak economic and political power of the poorer countries in exercising their fair share of participation in the production, use, and brokering of metrics and evaluation. Suwit argued that support, cooperation, and participation from poorer countries belong as central objectives of developments in this field. How well these imbalances are addressed will determine the ultimate health of the emerging field.

The Purposes

Amartya Sen, the philosopher and economist, urges us to make clear distinctions between “instrumentality” and “intrinsic value.” He often cites Aristotle who distinguished between wealth as a mere instrumentality for what is truly intrinsically-valued — a good life. Metrics and evaluation are clearly only instrumental in function. All metrics and evaluation work must be linked to intrinsic values.

In this regard, we need to understand better the history of purposes of many data-information systems. In his spectacular presentation last night, how did Hans Rosling cite a death rate in Sweden in 1709, one hundred years before Malthus as remembered by Richard Horton? The field of historical demography in Europe depended greatly upon the recording of birth and death in church parish registries. Why did the Swedes register births and deaths? The church wanted to save human “souls” which required that all births and deaths be registered. Hans later described to me how Swedish families would carry a child more than 20 miles for her/his baptism.

Recently health information conferences typically show photos of hapless workers surrounded high piles of files in a darkly lit room. The message is clear: data systems in developing countries are “messy” and poorly managed, overwhelming powerless workers. When I worked in rural Bangladesh, I listed 26 registers that community health workers were required to maintain. Poorly understood is that the Indian and Bangladesh data collection systems which I observed where established under British colonialism where the principal purposes of the system were for taxation and policing. Data systems, therefore, were set up to record aspects relevant to tax revenue and public security — not health! Many contemporary health data systems simply added indicators to these out-dated systems. By the way, in keeping with their purposes, these systems usually sent information upward to the rulers, not downward to the ruled, with data quite often shipped overseas to colonial archives.

What Einstein was suggesting in his adage, I believe, is that counting always has a purpose and that metrics and evaluation cannot be delinked from its purposes or lack thereof. That is why the discussion by Dan Brock on equity and justice aspects of metrics and evaluation is so important. That is also why IHME’s vision and mission should have strong explicit value and ethical bases, for example advancing global health equitably shared while adhering to the three principles of excellence, relevance, and independence.

Navigating into the Future

Hans Rosling concluded last night with his detective work discovering IHME as a “miracle,” at least according to the Finnish language! Like Julio Frenk’s 106 year old grandmother who is thrilled over the “first time,” we can be proud to have participated in the first IHME conference on metrics and evaluation. We will always be able to claim: “we were there at the creation!” We were present at IHME’s “miraculous birth” — not immaculate birth! And the IHME, I believe, “has legs to run.” It will be important and around for a long time, because . . . .

IHME has leadership. Julio Frenk and Chris Murray form a leadership team that has made a life commitment to IHME’s mission and work.

IHME has high ambitions as encouraged by Bill Gates Senior at the conference opening. Julio and Chris “think big,” but they also “think small,” because so much of the devil is in the details, and “think in between” by speaking, studying, acting, and promoting to guide and strengthen their thinking.

IHME is starting on a strongly built platform. Two decades have been invested in constructing this platform, building upon Chris Murray’s burden of disease work in parallel to Julio Frenk’s building of the Mexican Institute of Public Health and Mexican Health Foundation. Together and in sequence, Julio and Chris created and built the Evidence and Information for Policy cluster in the World Health organization, before moving on to academic and political roles in Harvard and Mexico.

IHME has got legs because it’s got big bucks! $125 million have already been secured over the next 10 years, backed by strongly committed partner institutions, the Gates Foundation and the University of Washington. I know the staying power of philanthropy. Every month, I receive a salary check from my foundation, the China Medical Board, that was endowed by last century’s leading philanthropists, John D. Rockefeller Senior. After 94 years of philanthropic giving (now exceeding more than $1 billion in current dollars), the Board is still kicking money out of its doors to advance health in China and Asia. I earnestly believe that the IHME will also enjoy a century of shelf-life!

Into the future, IHME will have to navigate many challenges: How to balance and link science, actors, and purpose? How to retain independence yet promote inclusiveness? How to extend global reach with greater diversity and equity? I hope future conferences will see many more faces, better gender balance, and greater national diversity. I was extremely impressed with IHME’s capacity to attract bright young people reflected by the posters at the IHME inaugural open house. These young people will certainly be future leaders because Chris Murray and Julio Frenk have a demonstrated track record for nurturing the next generation of leaders.

Let me conclude by returning to Einstein. “Not everything that counts can be counted; not everything that is counted counts.” This conference is a statement of what really counts for IHME! In describing Chris Murray’s earliest work, Bill Foege — that other senior fellow of the Gates Foundation along with Julio Frenk — praised the work as having the potential to transform the field of global public health. Foege observed that Chris insists on measuring and harmonizing all deaths, causes, and burdens — of all people. In our shrinking planet, no persons should be missed. IHME’s work therefore explicitly recognizes the unity of global humanity where “every person truly counts!” So to answer Einstein, the health of every person is what counts for IHME! And for the sciences at IHME, every missing person must be counted, every person must be counted once, every birth and death must be counted only once! That is the “true count” our shared humanity!

Back to Top

Go to Home Page