An ISG drones conversation with…
VillageReach
Version two of the Drone Evidence Generation Toolkit features sample frameworks, indicators and data collection tools
©Henry Sempangi Sanyulye
We spoke to VillageReach’s Dr. Olivier Defawe, Director of Health Systems and Drone Programme Lead and Susie Truog, Senior Manager in the Research, Evidence and Learning team, to learn about drone delivery evidence gaps and the new tool to generate much-needed data.
VillageReach has been involved in multiple drone delivery activities and is currently managing several drone delivery operations in Malawi and Democratic Republic of the Congo. What do you see as the biggest evidence gaps when it comes to the contribution of drone delivery to supply chain performance, health outcomes and cost?
In addition to working directly with governments to introduce drone delivery systems through the UAV for Payload Delivery Working Group (UPDWG), we have also developed an online database of drone implementations, called the Medical Drone Delivery Database (MD3). Through our own experience and evaluating evidence from other implementers submitted to the MD3, we have seen that basic operational data is typically available, such as the number of flights completed or number of health facilities served.
Where we really see a gap is in evidence on the effect drones have on supply chain performance (such as product availability), total supply chain cost and health outcomes. Data should be collected before drone deliveries begin as well as periodically over the course of operations to allow comparisons with the status quo ground transportation system. In addition, to generate this type of evidence, drone operations need to be sustained for at least 12-to-18 months but to date most drone operations have been short-term, lasting for a few weeks or months. Collecting this type of evidence requires expertise in supply chain monitoring and evaluation, and costing. This requires sufficient funding and planning not just for drone delivery services but also to monitor and evaluate the operations.
“We’ve been wanting to measure our effectiveness for a while now and I think this [the EGT] was a really good framework to see the different indicators that we could look out for.
“I think that having data that backs up what we say can push our story a lot further and can resonate with people and show that it can be a viable business case. I think it really sets the standard for our drone company. To push further, you do need to have certain standards in the ecosystem to follow.”
Manon Taylor, Avy drone logistics provider, who tested the EGT in their operations in Botswana
How did the idea of an evidence generation toolkit (EGT) come about and why do we need such a resource?
In response to the priorities identified in the USAID publication UAVs in Global Health: Defining a Collective Path Forward, the Interagency Supply Chain Group (ISG) uncrewed aircraft systems (UAS) coordinating body asked VillageReach to develop guidance for implementers on collecting the right data. Data on the first medical drone deliveries was limited and where it was available it was hard to compare results between projects. The EGT provides resources and guidance for drone operations at different stages of maturity, from early feasibility and safety tests (which we call phase one) to established drone delivery systems (which we call phase two and when scaled we call phase three). The EGT can help all implementers collect the right data for stakeholders, including governments, funders and clients, to make evidence-based investment decisions. The evidence will ultimately enable decision makers to understand the costs of an ongoing drone delivery system and its impact on existing delivery systems.
“The implementation roadmap has been useful as a framework to help pinpoint the current stage of the implementation that’s been ongoing. The EGT has also helped us to understand what types of investments are needed to move the implementation from where it is now into the next iteration; it’s been super helpful for us and I’m excited to see it out there.”
Denise Soesilo, Outsight International, who has been using the EGT in an evaluation of a donor-funded drone delivery programme
Version one of the EGT was released in 2019 and now you are releasing version two. What exactly is different about version two, who should use it and what should it be used for?
The first version of the toolkit, released in 2019, focused on providing the resources and guidance needed to collect data around phase one implementations, or feasibility and safety tests. Since the first version of the toolkit was released the sector has learned a lot about the evidence needed when drone delivery operations are sustained for a year or more. In this second version of the toolkit, released in October 2021, we have updated the logical frameworks, which implementers can use to guide the design of their projects. The frameworks highlight the various components that should be considered when establishing drone operations including regulations, stakeholder engagement, supply chain benefits and costs, local capacity and flight operations. The framework suggests sample activities and the outputs and outcomes that should be achieved at each phase of implementation, from early feasibility and safety testing to scaled operations.
The toolkit also provides indicators that can be used to measure progress towards achieving these results, which can be used to communicate results publicly. The updated version provides new, proven indicators and sample data collection tools for all indicators.
It’s important to note that the toolkit provides sample frameworks, indicators and data collection tools, but adapting these resources to meet the needs of a particular project and then analysing the results requires expertise in supply chain monitoring and evaluation, as well as supply chain costing.
How do you think the EGT benefits the donors interested in drone delivery for supply chain delivery?
The EGT provides donors with the resources and guidance that can assist their grantees to generate the evidence required to measure progress towards outcomes and to inform future investment decisions. Donors should consider providing their grantees sufficient funding to collect, analyse and share this data, which may require contracting an experienced supply chain monitoring and evaluation expert to ensure that supply chain performance and cost are rigorously evaluated. Baseline data collection must be planned from the beginning of any project. Donors should also require that their grantees submit drone project data to the MD3 and allow for the complete submission of generated results to benefit the sector as a whole. This may mean putting in place data sharing agreements to align expectations and ensure approval to share findings publicly.
The toolkit is available here.
