Question

As part of their Global Health Partnerships program (2008-2011), Pfizer Inc., and the Pfizer Foundation supported thirty-one promising and innovative cancer and tobacco-control projects from forty-six countries from Asia to Africa.

The Pfizer Foundation’s investment included the provision of tailored technical assistance provided to each grantee to support successful program implementation, build evaluation capacity, and establish a network that would foster a global community of practitioners to fight cancer. The Institute for Global Tobacco Control at the Johns Hopkins Bloomberg School of Public Health provided the technical assistance, which consisted of in-person, individualized planning sessions; phone and e-mail exchanges; training workshops; attendance at international conferences; webinars; and specially developed educational materials and evaluation training guides. Opportunities were provided for grantees to share information and work peer-to peer through online networking. Grantees were provided with easy access to best practices data and given personalized instruction on how to improve their evaluation efforts.

The Pfizer Foundation used this approach to improve the likelihood that these projects would be successful. This blog post describes and shares the results of an evaluation that we conducted to assess this approach.

A multidimensional Global Health Partnerships program index was developed to assess whether grantees met or exceeded their expected programmatic goals. Criteria of success were established for the index and included

* degree to which projects achieved their stated goal(s),

* quality of their evaluation plans and indicators,

* ability of the grantees to disseminate their outcomes and/or findings,

* how easy it was for the grantees to use partnerships developed through the Global Health Partnerships program to advance project goals,

* degree to which the projects scaled up or expanded their efforts throughout their Global Health Partnerships funding, and

* ability to obtain additional funding for sustainability.

Data to assess these components were obtained from a systematic review of grantee progress reports and anecdotal input from program managers and mentors who had in-depth knowledge of grantees. Additional information was obtained from the grantees to determine how the Bloomberg School’s technical assistance and networking efforts affected grantees’ projects. This was accomplished by soliciting feedback from the grantees regarding the new knowledge gained as a result of the technical assistance, specific improvements made to their projects because of the technical assistance, and components that the grantees thought would be useful for future programmatic efforts. Since our sample size was small, none of these components reached a level of statistical significance; however, successful grantees did report that the following contributed the most to their success:

* gaining specific knowledge of implementation and evaluation (that is, developing an action plan for their project),

* developing measurable indicators,

* improving data collection methods, and

* understanding how evaluation could improve their program sustainability.

We found that grantees that were able to improve their evaluation plans and data collection methods were twice as likely to be in the highest success category, based on the index. Also, those grantees that came to understand the role of evaluation in sustaining their efforts and those who developed a collaborative relationship with at least one other grantee were three times more likely to be among the most successful grantees.

We also found that highly successful projects were able to measure their impact beyond mere counts of participants or activities conducted. In addition, highly successful grantees were better able to integrate their programs into existing health care systems or expand their projects to reach more communities. Many of these grantees also advocated for promoting or improving tobacco control and cancer prevention programs and policies in their countries.

The specific skills that grantees rated as most useful were interacting and networking with other grantees, learning specific evaluation techniques, and learning how to use data to improve their projects. For example, staff at one project, whose efforts funded by the Global Health Partnership focused, in part, on the creation of smoke-free hospitals, reported that after they evaluated their project, they found that 595 indoor-smoking areas were removed during the first year of the project. Consequently, they used this positive outcome to reinforce efforts to make the hospitals smoke-free as well as to encourage other hospitals to implement such policies.

At the end of the Global Health Partnerships program, more than 40 percent of all grantees had secured, or were in the process of securing, new (other) financial support; more than 50 percent of grantees had confirmed plans to continue their programs or use products or tools they had produced in their Global Health Partnerships project; and nearly 80 percent had formed collaborations or alliances to strengthen their reach, innovation, or impact. The Pfizer Foundation, by providing funding for this technical assistance, helped contribute to the effective completion of its grantees’ projects. Based on these findings, we would strongly encourage other funders to adopt a similar approach.

• What are the keys to evaluating the success of a project? Do you think they did a thorough job of articulating the ‘lessons learned’ given the information presented?

• The Global Health partnership program index assessed grantees’ projects success based on a systematic review of progress reports and anecdotal input from program managers and mentors. There is an inherent subjectivity to anecdotal inputs. Explain your thoughts about the impact of subjectivity on the quality of the index given the subjectivity factor and consider both the pros and cons. Given both sides, what is your stance on qualitative vs. quantitative type data in this type of reporting? Explain.

• The grantees of the Global Health Partnership program gave specific feedback about how the technical assistance contributed to their projects. Based on that feedback, do you agree or disagree that these results should be extended to all grantees? And to other similar programs? Explain your perspective.

Answer 1

Keys to evaluate successful projects is based on customer experience, customeroyalty and retention ratio, impact created on customer, customers positive word of mouth .

The lessons learnt are not extremely articulated because sample size is negligible and not yielding significant results.

Qualitative analysis merely focuses on experiences and overall impact created and hence cannot give complete overview. Quantitative analysis on other hand speak of numbers and statistics butcalso lack complete overview if sample size is low. Hence mix of both qualitative and quantitative analysis is important.

The repirt shoukd be extended to all grantees and all programmes and weighted average must be taken to reach conclusive evidence and significant analysis.