Evaluation Team : JSI Research & Training Institute, Inc. (JSI)

Gavi, the Vaccine Alliance is commissioning this independent, multi-year evaluation to assess the relevance, effectiveness, efficiency, outcomes and sustainability of its CCEOP investment from 2018 to 2020. The evaluation is being conducted at the global level (market shaping) and in three countries, Guinea, Kenya and Pakistan.

The market-shaping aspect considers progress made in promoting healthy markets and improved optimal market conditions, as well as any unintended consequences. The country-level implementation focuses on achievements made in upgrading and expanding cold chain equipment (CCE) and creating a more efficient and effective supply chain.

The results of this evaluation are intended to help the Alliance improve the design of the platform, including Gavi’s cold chain equipment (CCE) market shaping strategies, and to help countries in scope and other Gavi countries implementing the Platform.

Approach and Methods

This evaluation is being done both retrospectively and prospectively at the global and country level. A mixed methods approach is being utilised for this evaluation, including data collection from a variety of sources - document review, direct observation, key informant interviews (KIIs), health facility assessments (HFA), and health management information system/logistics information management system (HMIS/LMIS) where possible.

Key high-level findings (further details can be found in the full report)

Market Shaping

• The supply of CCE continues to expand, with 9 of the 18 CCE segments tracked exceeding the original goal to ensure two platform-eligible suppliers of ice lined refrigerators (ILRs) and solar direct drives (SDDs) per size segment.
• Procurement has not kept pace with the initial annual forecasts for CCE that were shared with suppliers at the outset of CCEOP, due to delays in implementation.
• Tenders for the first 25 countries (including Year 2 procurement) have skewed towards two suppliers based on country preferences, undermining the CCEOP market-shaping objectives to create a healthy market.
• Delays on a number of processes have raised questions about the credibility of CCEOP demand forecasts, which may undermine ongoing decisions by suppliers to produce or innovate for this market.
• Some suppliers feel that the tender award process is opaque and is not currently rewarding investments already made in product innovation and lower TCO (total cost of ownership) options.
• It is difficult to accurately assess for various contexts and compare value for money or TCO in selecting equipment due to limited data on longer-term CCE field performance and ability to compare across a wide set of features.
• Questions and concerns persist regarding the tendering and CCE selection processes and role of country preferences.
• The service bundle mandate is the greatest source of conflicted feedback.
• Information flow and transparency among partners, countries and manufacturers have made progress under CCEOP.
• Progress has been made to improve price transparency, however challenges remain.  

Country-level (Guinea, Kenya and Pakistan)

• CCEOP responds to country needs and priorities and is well coordinated by the project management team (PMT) in-country with other partners.
• Proper preparation for CCEOP deployment requires significant investment in time and budget, and needs to be taken into account during planning.
• Not all implementation guidance, policies, or plans are well communicated, understood, or implemented at the country level.
• Stakeholders were generally satisfied with SBP installation of equipment, although in Kenya stakeholders were concerned that the cost was too high for a service the MOH felt they could provide.
• The SBP monitoring system for deployment was effective and allowed for minor deviations.
• Training provided by SBPs was insufficient; training needs and topics need to be clarified.

A summary of the Gavi Alliance Evaluation Management Response (EMR) for this midline evaluation report captures the response to the midline evaluation findings and recommendations, and indicates actions to be taken by the various Alliance partners.

Documents

• Midline assessment summary of Evaluation of the Cold Chain Equipment Optimization Platform (2020)

  Download

• Midline assessment of Evaluation of the Cold Chain Equipment Optimization Platform (2019)

  Download

• Evaluation of the Cold Chain Equipment Optimization Platform: Gavi Alliance Evaluation Management Response

  Download

• Cold Chain Equipment Optimization Platform: Market Shaping Policy Brief

  Download