Drug development practices to improve public health policy 

Dr Andrew Garrett, Executive Vice President of Scientific Operations at ICON, explores the benefits of integrating some of the validation and standardisation practices already developed for PIs into the practices around NPIs and public health data; outlines some of the practices that can be implemented in order to do this; and highlights the need for interdisciplinary collaboration between pharmaceutical and non-pharmaceutical researchers and public health centres to consider the overlap between NPIs and PIs and identify where standards can be improved and enforced. 

As a result of clear US Food and Drug Administration (FDA) guidelines, the leading vaccine development programmes for Covid-19 were all remarkably similar.¹ Vaccine development was conducted at great pace and within a year of the first infection being reported, the first Covid-19 vaccine to report phase III clinical trial data demonstrated unexpectedly high efficacy and cleared regulatory efficacy criteria with ease.² Meanwhile, the implementation of non-pharmaceutical interventions (NPIs) – such as masking, social distancing and lock-downs – proceeded on the basis of observational public health data that lacked clear decision criteria.

In fact, the World Health Organization has recognised that regional public health responses could have benefited from established criteria for generating public health data and evaluating NPIs.³ The following article reviews a select number of practices for pharmaceutical interventions that should be considered for NPIs and public health data generation going forward.  

Pre-specification of effect size  

During Covid-19 vaccine development, guidelines pre-specified the primary efficacy criteria that defined success in a placebo-controlled efficacy trial – notably, estimated vaccine efficacy of at least 50% with the lower confidence limit greater than 30%. Ideally, research assessing NPIs would also pre-specify the effect size, which could inform their implementation or withdrawal, especially when weighed against socioeconomic costs.  

However, pre-specification of effect size is typically the preserve of randomised studies, where data are prospectively generated to meet specific objectives, whereas research directed at NPIs relies on observational data where the data generating process is less controlled. 

Indeed, few randomised trials of NPIs have been undertaken. For instance, a review of cloth face masking identified just two reported randomised controlled trials comparing face mask intervention to no mask control with respect to the spread of Covid-19, both of which pre-specified the effect size.⁴ 

Even when randomisation is impractical, observational studies of NPIs should clarify at the outset what magnitude of effect would constitute public benefit.  

Quasi-experimental and other randomised trial designs  

One way that NPIs research could introduce greater objectivity is by using quasi-experimental methods. These methods are aimed at establishing a cause-and-effect relationship without randomly assigned control and experimental groups. When using retrospective data, researchers can implement quasi-experimental designs that take advantage of how public policy is implemented. One example is the regression discontinuity design.  

Regression discontinuity designs can be used to create two broadly comparable groups around the threshold of an intervention. This is possible if the data are limited to a narrow range around a threshold, where participants may be regarded as similar. For example, in a retrospective study of vaccine efficacy, age requirements for vaccine eligibility enabled researchers to assign an experimental group of those aged between 80-84 years, who were first eligible for Covid-19 vaccination, and a control group of those aged between 75-79 years who were ineligible.⁵ This design also has been used to assess the impact of NPIs. The Bank of England, for example, used a discontinuity design to assess the impact of Covid-19 lockdowns on business activity, while in Japan, it was used to assess the impact of Covid-19 school closures.^6,7 While regression discontinuity designs have the potential to tease out the impact of NPIs in high-volume observational settings, they can be biased if the model assumptions do not hold.⁸

Another design option, that was originally aimed at tackling challenges such as logistics in service delivery type interventions and resource constraints in developing countries, is stepped-wedge cluster randomisation. These designs should be considered for both pharmaceutical and non-pharmaceutical research and involve the random and sequential crossover of clusters from control to intervention, until all clusters are exposed.⁹ Each cluster includes individuals who receive the same intervention. Initially, no cluster is exposed to the intervention, and then clusters receive the intervention at regular intervals. This design can be used when the availability of healthcare, the geographic spread of the population, or variations in public health policy implementation means that the intervention is rolled out over a prolonged time period. For example, care homes could have been selected at random to receive a Covid-19 vaccine, since not all care homes could be visited at once. This could have generated experimental data on both the efficacy and safety of the vaccines.   

Data storage and modelling standards 

To conduct observational public health research, researchers must have reliable access to trustworthy data that have been collected, stored and documented using established standards. At times, this proved a major roadblock during Covid-19, where similar datasets contained unexplained differences and inconsistencies. For instance, in August 2020, Public Health England modified its definition of Covid-19 deaths after it became apparent that deaths were being attributed to Covid-19 if a person had tested positive for Covid-19, regardless of how long it had been since the test was conducted.¹⁰ As a result, reported cumulative Covid-19 deaths reduced by 13% overnight.  

For statisticians working in drug development, such ambiguity is an anathema. Detailed meta-data (the data about data) is a requirement of working in a regulated industry – enabling reproducibility and traceability. The Clinical Data Interchange Standards Consortium provides a suite of standards (naming, formatting and structure) supporting the clinical and non-clinical, end-to-end processes.¹¹ These standards are required by the FDA and the global nature of drug development means that they have now become the de facto standard.  

In addition to data standards, there are also established standards for modelling and simulation.¹² In drug development, derived datasets and outputs are often independently programmed from detailed specifications and compared electronically to systematically identify and investigate differences. While error-free statistical modelling is not guaranteed, standardised practices allow others to navigate the data sets generated and perform independent checks. They also allow corrections to be made in a timely and ordered manner.  More importantly, there is an audit trail that provides reassurance and builds trust. 

To ensure NPIs research and public-health decision-making is both reliable and transparent, standards for health-related data and public health modelling should be jointly developed. Key institutions that could drive this are the UK Statistics Authority, the Medicines and Healthcare products Regulatory Agency, and the UK Health Security Agency. These standards could draw from already established standards in drug development – including those for data collection and reporting, and those for statistical modelling and simulation.  

Conclusion 

Given the differing requirements and context for the research of pharmaceutical and non-pharmaceutical interventions, the development of standardised practices will require interdisciplinary collaboration between pharmaceutical and non-pharmaceutical researchers and public health centres. Such cross-disciplinary thinking and collaboration could prove beneficial for all involved and help to lay a foundation for a more aligned and effective public health response to future threats.   

Biography 

Andrew Garrett is Executive Vice President, Scientific Operations at ICON, responsible for the strategic direction and operational execution of ICON’s Global Scientific Operations. He is President Elect of the Royal Statistical Society, having previously been Chair and Founder of its Data Science Section, VP/Honorary Secretary of the organisation and Chair of its Long Term Strategy Group. He has worked extensively in the area of rare diseases and has a portfolio of published papers on the topics of non-inferiority trials, subgroup analysis, data transparency and modelling and simulation. Dr Garrett has a BSc in Economics, an MSc in Medical Statistics and a PhD in Applied Statistics. 

References 

1. Development of licensure of vaccines to prevent COVID-19: Guidance for industry. Food and Drug Administration. Center for Biological Evaluation and Research. June 2020. 
2. Pfizer and BioNTech Conclude Phase 3 Study of COVID-19 Vaccine Candidate, Meeting All Primary Efficacy Endpoints. November 2020.  
3. WHO. Calibrating long-term non-pharmaceutical interventions for COVID-19. Principles and facilitation tools. 30 July 2021. 
4. Liu IT, Prasad V and Darrow JJ. Evidence for community cloth face masking to limit the spread of SARS-CoV-2: A critical review. Cato Working Paper. Cato Institute. No. 64. 2021. 
5. Bermingham C, Morgan J, Ayoubkhani D, et al. Estimating the effectiveness of first dose pf COVID-19 vaccine against mortality in England: a quasi-experimental study. medRxiv 2021. doi: https://doi.org/10.1101/2021.07.12.21260385 
6. Hurley J, Walker D. Staff working paper no. 943. Did the COVID-19 lockdown reduce business activity Evidence from UK SMEs. Bank of England. November 2021. https://www.bankofengland.co.uk/-/media/boe/files/working-paper/2021/did-the-COVID-19-local-lockdowns-reduce-business-activity-evidence-from-uk-smes.pdf?la=en&hash=E8E116D61061BB71B018643E983FD306D8553C80 
7. Takaku R, Yokoyama I. What the COVID-19 school closure left in its wake: Evidence from a regression discontinuity analysis in Japan. Journal of Public Economics 2021:195;1-10. https://doi.org/10.1016/j.jpubeco.2020.104364 
8. Senn S. A brief note regarding group matching in medical research. February 2020. http://www.senns.uk/Stats_Notes/Matching.pdf. 
9. Hemming K, Haines TP, Chilton PJ, et al. The stepped wedge cluster randomised trial: rationale, design, analysis and reporting. BMJ 2015: 350;h391. 
10. Public Health England. Technical summary: Public Health England data series on deaths in people with COVID-19 death. 12 August 2020. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1052203/UKHSA-technical-summary-update-February-2022.pdf 
11. Clinical Data Interchange Standards Consortium. https://www.cdisc.org 
12. Smith M, Marshall A. Importance of protocols for simulation studies in clinical drug development. Stat Methods Med Res 2011; 20(6):613-22. DOI:10.1177/0962280210378949