Design Of Experiments: Useful statistical tool in assay development or vendor disconnect

Design Of Experiments: Useful statistical tool in assay development or vendor disconnect By Dr John Comley The use of design of experiments (DOE) in assay development (AD) has the potential to speed up assay optimisation (ie reduce assay development bottlenecks) and to facilitate a more thorough evaluation of assay variables. Only one liquid handling vendor currently offers application specific software and support for investigating DOE in biological assays. Although standalone DOE software packages are available, these were not written specifically for biological applications and they vary in their suitability for AD. DOE needs to be simpler to implement to make a major impact on AD. A market opportunity exists for a turnkey solution that directly links statistical design with automated liquid handler programming and also feeds the assay readout directly into the statistical analysis, to suggest and facilitate further iterative retesting. Until new tools or more encompassing solutions emerge, the full impact of DOE on AD is unlikely to be realised. Design of experiments (DOE) is a well established and proven statistical method which has broad application across many disciplines and industries. Typically there are levels of design which can be applied: these range from the simplest fractional factorial (which includes experiments to identify which factors are most critical), followed by full factorial (which enables identification of significant interactions between factors), and the more complex surface area design (which facilitates finer optimisation of factors). Assay development (AD) has become a bottleneck in many pharmaceutical organisation’s lead discovery operations, with assays typically taking in excess of a month or more to develop using a traditional approach (ie changing one setting at a time or sequential design). A typical AD lab may be expected to develop in excess of five assays per year, with around one in 10 assays never achieving the desired assay quality parameters/signal window (the main criteria of development success). Using the traditional approach up to around 10 different combinations of assay conditions (factors) may be explored using either manual liquid handling or a basic automated liquid handler set up. One of the methods companies are increasingly exploring to compress AD times without compromising on quality is DOE. However, there is paucity of factual information around the application of DOE in AD, although many anecdotes prevail in the industry. With this in mind and as part HTStec’s tracking of emerging life science marketplaces, a survey was undertaken in June 2009. The objectives were to examine: 1) how widely DOE approaches in AD are used 2) what is the current level of understanding of DOE 3) what views people hold about DOE and its benefits 4) what success has been achieved 5) which aspects of DOE are problematic 6) what restricts its wider implementation today. Some of the findings of this market report (1) form the basis of this article and the setting to review the tools that currently support DOE in AD. Current use of design of experiments in assay development The survey showed that only around 5% of all assays developed today were done using DOE, although the expectation was that significant cost savings (3x) would be achieved by applying DOE. Figure 1 shows the current level of use of DOE in AD among respondents to the HTStec survey. Some 45% of survey respondents were using DOE in some aspects of AD today, with a further 23% planning to implement it in the future. An additional 12% had tried DOE, but are not currently using it today. That left 20% that have never used DOE and remain to be convinced about its utility in AD. To date DOE has shown most promise with biochemical assays, however it is increasingly being applied to cell-based assays, although often limited here to discrete steps or to fewer variables. Main benefits if design of experiments in assay development The majority (77%) of survey respondents perceived the main benefit of DOE in AD to be faster assay optimisation (ie reduce assay development bottlenecks) (Figure 2). This was followed by more thorough evaluation of assay variables (71% responding); reveals unexpected interactions between components (60% responding); and more robust (stable) assays (60% responding). Improving the skills set/repertoire of assay biologists and enabling results that can be standardised and presented in a language that cuts across disciplines, were both perceived as the least important benefits of DOE in AD. Why has design of experiments not been widely implemented in assay development? The reason that had most prevented survey respondents from using or considering DOE was too hard to implement. This was followed by lack of integrated solutions; too hard to persuade others of the power of DOE; I don’t know how to do it; and I don’t fully understand the benefits (Figure 3). Design of experiments software packages Survey respondents rated their familiarity with commercial DOE software packages and approaches as highest for Beckman Coulter AAO (Automated Assay Optimization) (Figure 4). This was followed by SAS JMP™ and Stat-Ease Design- Expert®. However, it should be pointed out that the level of familiarity with all commercial products listed was between only moderately familiar and unfamiliar (ie they don’t know it). The biggest concerns of those respondents using and/or familiar with DOE, was that commercial DOE software packages can lead to illogical biology recommendations (29% concerned) or were not biology user friendly (26% concerned) (Figure 5). Together these findings suggest that DOE software vendors currently are not doing a very good job making assay developers aware of their products, and most vendors have limited understanding of biology applications and screening requirements for their products. Of those survey respondents using and/or familiar with DOE, nearly all have found the needed to integrate different DOE software packages using Microsoft Excel or a similar tool, with the majority (37%) using an Excel template to describe the factors to be investigated (Figure 6). If end users have to resort to this type of fix to make DOE approaches doable in AD, it further highlights that commercial software vendors need to better understand the application and that what is on offer today only partial supports AD. The need for better DOE training Greater than 50% of respondents admitted to having never been properly trained in DOE. This may reflect the fact that the type of DOE training on offer currently does not meet their needs. The majority (41%) of respondents thought that the most useful DOE training was a fully integrated approach on-site (ie one which included interfacing the DOE software package with an automated liquid handler and the biological assay), this was followed by biology application(s) specific training (35% responding) (Figure 7). What’s needed to drive wider uptake of design of experiments? Survey respondents rated better understanding/ greater knowledge of DOE as what was most needed to drive the wider uptake or greater use of DOE. This was followed by easier implementation, easier interpretation of data, and then software that directly links the statistical design and automated liquid handler programming (Figure 8). However, it should be noted that the ratings for all factors considered in this analysis were greater than three, ie all options were considered needed (ie desirable) to drive greater uptake of DOE. Overall these findings suggest that the current DOE offerings and the way in which they are offered to assay developers (eg mainly as standalone components) does not currently meet their needs and there is a significant vendor disconnect in this respect. Other applications of design of experiments In addition to AD, survey respondents were asked to comment on other drug discovery-related areas where DOE approaches might be usefully applied (Figure 9). Optimising liquid handler dispensing settings was the area where the majority (38%) of respondents plan to implement DOE. This was followed by optimising reader settings and cell culture media optimisation (both equal at 31% responding); then sample preparation (26% responding) and protein expression (21% responding). Latest liquid handling tools for design of experiments The following vendor snapshots provide details of some of the latest liquid handling tools that support DOE in AD: Beckman Coulter (www.beckman.com) pioneered assay optimisation using DOE on the Biomek® 2000 Liquid Handling Workstation with the introduction of SAGIAN™ Automated Assay Optimization (AAO) software, developed in collaboration with SmithKline Beecham in 1999. The AAO software was later adapted to the Biomek FX instrument to take advantage of the FX’s individual eight-channel pipetting head to significantly increase pipetting speed over the Biomek 2000 workstation. The latest adaptation of the AAO software is a version for Beckman Coulter’s BioRAPTR™ Microfluidic Dispenser. The BioRAPTR can deliver up to eight different reagents simultaneously to the wells of high-density microplates. The system executes accurate, contact- free dispensing of the reagents for volumes ranging between 100nL and 60μL into an entire experiment plate in just a few minutes. The BioRAPTR AAO software supports both 384- and 1536-well formats which enables assay developers to explore a wide range of assay factors allowing thousands of experimental conditions to be tested simultaneously, effectively enabling the use of complex response surface designs for rapid assay optimisation. The AAO software provides a ‘Wizard- Style’ interface to translate DOE design tables from the most common statistical software packages into Biomek or BioRAPTR methods. The experimental design can be enhanced within AAO by adding specific background wells for every set of experimental conditions. The software can also create quality control wells such as positive or negative controls. Running the Biomek or BioRAPTR methods and collecting data on a plate reader can be interleaved with other operations necessary for the assay such as shaking or incubation. This functionality allows inclusion of non-reagent experimental factors into the experimental design with no limitation to pipetting factors. Examples of non-reagent factors include incubation time and temperature, plate shaking and wash protocols. AAO has been reported to significantly decrease the time and cost of assay development as well as increase the robustness and sensitivity of high throughput screens (2). Biomek FX AAO has also been successfully utilised in media optimisation for production cell lines of biopharmaceuticals (3) (Figures 10 and 11). Formulatrix (www.formulatrix.com) offers two liquid handlers (the Tempest and the Formulator) that have potential utility for DOE. Both use microfluidic dispensing technology which provides these liquid handlers with faster dispenses and a compact design as compared to their traditional counterparts. The Tempest is a flexible reagent dispenser that can dispense dilutions and factorial DOEs and highly complex plate patterns from 12 reagent inputs. This high-throughput device’s independent channel control over all 96 channels enables it to dispense any volume of any input into any well of 96-, 384-, 1536-, or 3456-well plates. CVs below 10%, a typical dead volume of 400μL, and a minimum dispense volume of 200nL with no upper limit make this an ideal instrument for highthroughput screening and precious sample dispensing. The Tempest uses positive displacement instead of pressurised bottles. The Formulator is a high-viscosity reagent dispenser with up to 34 different reagent inputs, from either 50mL Falcon tubes or 125mL and 250mL Nalgene bottles. At the core of the Formulator is a patent-pending microfluidic chip that can measure and dispense discrete volumes of liquid. The chip has 96 outputs each with its own microfluidic valve cluster. Each valve cluster has two micro-diaphragms (200nL and 2.5μL) that can fill and dispense as fast as three times per second. Combining multiple dispenses of each micro-diaphragm allows any volume down to 200nL to be dispensed with a resolution of 200nL. The Formulator’s positive displacement microdiaphragms handle viscous and non-viscous liquids with ease and precision. The Formulator was recently used by scientists at Novartis to investigate the DOE of kinase and protease assays (4). Formulatrix hopes to launch the Tempest for DOE later in 2010 (Figure 12). Hewlett-Packard Specialty Printing Systems (www.hp.com) is developing a high-performance dispense solution for dose-response analysis. The new dispense technology, which is leveraged from HP’s established printing franchise, is currently undergoing pre-commercial field testing at BioPharma sites. Lab Scientists in assay development, lead optimisation, ADMET and other similar groups are key end-users targeted to use HP’s picolitre-to-microlitre dispense technology (Figure 1). HP’s dispense solution addresses a significant product gap in high-performance titration of DMSO-based small-molecule concentrate. Biologists are currently taking advantage of the HP technology’s dynamic range to titrate compounds directly to assay plates, thus eliminating serial-dilution and other painful intermediate steps. Endusers are also enjoying results within minutes for small-batch biochemical and cell-based assays. Inhibitors with different potencies can each be evaluated against bio-assay input variables for outputs such as IC50, Hill slope, signal strength, noise and DMSO content. The HP dispense solution enables non-contact dispensing of any compound at any dose into any well in 96 and 384-well plate formats making creative layouts in any row/column direction easy to implement. This new capability enables end-users to overcome previous barriers to conducting complex assays. For example, elaborate synergy or allosteric modulation experiments are easy to plan, programme and achieve. The technology also enables very fine titration spacing which facilitates better data in general and easier enzyme binding titrations to quantify the active fraction. Assay developers will also appreciate the exceptional flexibility of HP’s dispense technology, which can rapidly enable multi-factorial experimental design. Preliminary results suggest HP’s dispense solution could be promising a new tool for DOE (Figure 13). DOE software The following vendor snapshots provide details some standalone software packages available to support DOE in AD: Clinical and pharmaceutical assay development is a complex process. Minitab® statistical software (www.minitab.co.uk) can help the assay developer take advantage of statistical design of experiments by selecting the best matrix of experimental conditions to gain knowledge about the system being studied with a minimum number of experiments and a maximum degree of accuracy in the estimates. A wide variety of design models such as two-level factorials, D-optimal, robust designs and others are available to check for interactions between reagents and ensure that the full parameter space is covered. Some designs such as Plackett- Burman designs are used to screen multiple factors while others, such as Response Surface Designs, enable researchers to study the significant factors in more detail, including their non-linear effects. Minitab software can assist in determining the appropriate number of experiments and replicates to ensure that the maximum amount of information is obtained from the results. Statistically, significant factors are identified through the ANOVA table or through graphs such as Pareto and normal plots. Minitab also generates graphs showing the behaviour of the response, such as main effects and interactions plots as well as cube, contour or threedimensional response surface plots to assist in interpreting results. Once the key factors are revealed, Minitab guides scientists through the optimisation tool to select the reagent combinations that will result in optimal critical quality attributes of the assay. Minitab can also assist the researcher in making assays more robust to nuisance factors and in reducing the amount of variability in the output (Figure 14). DOE is the most effective method to achieve product and process knowledge and optimisation. MODDE from MKS Umetrics (www.umetrics.com) is a state-of-the-art design of experiments software package that allows investigators to achieve three important stages of DOE, which are also critical requirements during assay development. These stages are: identifying the most important factors and their ranges (screening), locating an optimal factor combination which can be used as a future set-point (optimisation) and investigating the sensitivity of the set-point to changes in the important factors (robustness testing). The latest release, MODDE 9, introduces a novel approach for Design Space Estimation (DSE) and validation by taking a quantum leap towards fulfilling the objective of the Quality by Design (QbD) paradigm, which defines design space as “the multidimensional combination and interaction of input variables (eg material attributes) that have been demonstrated to provide assurance of quality” (5). Based on Monte Carlo simulation, and unique to MODDE 9, the DSE can be utilised in robustness testing and validation during assay development as it can provide estimates of the largest possible design space and give quality or probability estimates for a safe region of operability for future results (Figure 15). Developing useful assays through DOE relies on two things. One is your understanding of some key concepts such as experimental treatments, experimental units, randomisation, replication and blocking. The other is the use of statistical technology that allows you to quickly construct designs that adequately express the biological context of the assay and the nuances of its operational setting, and then to easily analyse the resulting data to rapidly draw the best conclusions. JMP®, a desktop product from SAS (www.jmp.com), gives scientists and technicians this capability, providing a single, unified environment that allows you to generate and analyse efficient designs that are customised to your specific situation rather than insisting you pick an omnibus design from a pre-existing library. JMP’s computer-generated designs allow you to take specific account of constraints in your factors, include mixture and process factors in the same design, and correctly handle the hard and very hard to change factors needed when randomisation is restricted. JMP provides extensive diagnostics for the D and I optimal designs it generates for screening and optimisation, allowing you to see in advance if your experimental budget gives you a good chance of finding out what you need to know. Through its extensive and powerful scripting language, JMP also allows you to build customised workflows that integrate its capabilities into your environment (Figure 16). Stat-Ease (www.statease.com) offers software, training and consulting that make DOE easy for assay developers. Our Design-Expert® software provides test matrices for screening up to 50 factors. Built in to its design-building wizard is a handy power calculator – a valuable tool for establishing the number of test runs needed. Graphical effects analyses make it obvious which effects stand out. Design-Expert then lays out a complete analysis of variance (ANOVA) for establishing statistical significance. A broad array of graphical diagnostics reveals any abnormalities or outliers in the data. Based on the validated predictive models, a numerical optimiser pinpoints the most desirable combination of factors. The program then displays the ‘sweet spot’ where all requirements can be achieved. This frames the design space for those adhering to quality by design (QbD), as promoted by the FDA. The entire process of design, analysis and optimisation is catalysed via Design-Expert software, which is backed by world-class experts on DOE who stand ready to provide statistical support. A great deal of valuable information on DOE is available free at the Stat-Ease website, including a Primer on Mixture Design that is a must for work on assay formulations (Figure 17). StatSoft’s STATISTICA DOE module provides comprehensive design and analysis capabilities for all standard fractional, response surface and mixture designs, as well as constrained surfaces, split plots, Latin squares, Taguchi, D-, A- and T-optimal designs and more. Handling unbalanced, incomplete and ‘botched’ designs, the analyses will automatically determine estimable effects, report aliases and compute the parameter estimates for all non-redundant effects. Users can manually toggle specific effects in and out of the current model quickly and easily, observing the effect on the overall fit. A large number of output options are provided to review parameter estimates, effect sizes and response characteristics, in both tabular and graphical form. Response/desirability profiling allows users to optimise across multiple competing response variables, with user-defined desirability profiles for each response variable. In addition to the standard set of designs supported by most DOE software, STATISTICA also provides much more versatile analytical capabilities via general/generalised linear models, variance components and other analytical and graphical functions, allowing users to handle a much wider range of designs and response types. Beyond these capabilities lie virtually unlimited expansion options into more advanced multivariate modelling and data mining methods with enterprise-class deployment of predictive models and web-based client-server operation (Figure 18). Conclusions What is immediately apparent from this review is that only one liquid handling vendor (Beckman) currently offers any application specific software or support for investigating DOE. It should be stated that all other established liquid handling vendors contacted when writing this review indicated they had nothing to contribute to the article. Although several new entrants to the liquid handing market (Formulatrix and HP) appear to be developing solutions with novel and enabling dispensing capabilities which could be advantageously deployed for some aspects of DOE, neither has so far launched products specifically focused on this application, so it is too early to say what each may contribute. Conversely, there are a number of vendors (Minitab, MKS Umetrics, SAS, Stat-Ease, StatSoft) that offer standalone DOE software packages. For the most part these are generic offerings that were not written specifically with biological applications in mind, and they vary in their suitability for assay development. What is noticeable from the software vendor snapshots is that none actually mentions interfacing statistical design with setting up assays and the associated liquid handling. Herein lies the assay development biologists dilemma: limited automated liquid handling tools are available that can be simply programmed to perform full factorial analysis over a wide dynamic volumetric range. While available, DOE software, for all its sophistication and complex statistical analysis, is not entirely suited to their needs nor does it seamlessly interface with the liquid handling instruments they need to use to set up experiments to investigate assay variables. The need the make DOE simpler to implement was a major take home message from the survey. A fully integrated approach (ie one which seamlessly interfaces the DOE software package, with an automated liquid handler and the biological assay set-up and subsequent readout) that is simple to implement clearly would be beneficial. There exists a market opportunity for a provider of an automated liquid handling workstation that can be directly interfaced with a microplate reader to combine its technology and expertise with a DOE software vendor to develop a turnkey solution that fully supports DOE for AD. That platform ideally should offer software that directly links the statistical design with the automated liquid handler programming and also feeds the assay readout directly into the statistical analysis package, to suggest and facilitate further iterative retesting. Arguably until this happens the solutions/tools on offer will at best be piecemeal components that only partially address the problem and will inevitably represent a compromise. Beckman Coulter’s AAO module for the Biomek FX, and now for the BioRAPTR, must be congratulated on coming close to meeting user expectations for a DOE solution; however, it only accepts DOE design table input from some third party software and statistical analysis of the assay readout is not addressed. Until new and more comprehensive ‘total’ solutions emerge, the full impact of DOE on AD is unlikely to be realised. --- This article originally featured in the DDW Winter 2009/10 Issue --- Dr John Comley is Managing Director of HTStec Limited, an independent market research consultancy whose focus is on assisting clients delivering novel enabling platform technologies (liquid handling, laboratory automation, detection instrumentation and assay reagent technologies) to drug discovery and the life sciences. Since its formation six years ago, HTStec has published more than 50 market reports on drug discovery technologies and Dr Comley has authored 30 review articles in Drug Discovery World. Further information on accessing the market report DOE in Assay Development Trends can be obtained by visiting www.htstec.com or by emailing john.comley@htstec.com to receive a free copy of the Report’s Executive Summary and Table of Contents. References 1 DOE in Assay Development Trends 2009 Report, published by HTStec Limited, Cambridge, UK, July 2009. 2 Taylor, Paul. Optimizing assays for automated platforms. Modern Drug Discovery, December 2002, 37-39. 3 Voisard, Damien. Application of a high-throughput semiautomated platform for cell culture media optimization. Presented at San Francisco ACS meeting, September 2006. 4 Ho, Samuel et al (2009). The Use of the Formulatrix Formulator™34 for Design of Experiment Approaches. Poster Presented at 15th Annual SBS Meeting Lille, France April 26-30, 2009. 5 International Conference on Harmonisation (ICH); Draft Guidance: Q8(R1) Pharmaceutical Development Revision 1 see http://www.fda.gov/cber/ich/ichguid.htm.