In response to the need to increase the return on investment in R&D, many large Pharma companies are shifting from an internal fixed resource model to more flexible, external relationships. These partnerships are diverse, ranging from basic scientific investigations, through establishing new technologies and methods, to the discovery, development and commercialisation of therapeutic agents.

A host of partnering opportunities now exist, including in-licensing of existing assets and co-discovery of new ones, as well as development of fundamental science and technology in a pre-competitive manner. In reality, the opportunities available far exceed the resources that could be allocated by a single Pharma company; hence, choices between them must be made.

Historically available tools for assessing the value generated by projects mainly address those with tangible deliverables (eg, the discovery of a new drug candidate). They also rely heavily on the quantitative, usually riskadjusted, values that can be assigned to these outputs (2). More recently, fuzzy approaches to real option valuation and decision trees using Boolean algebra have been developed, but these methods still rely on hard commercial valuations that are not realistic for early stage R&D projects (3).

Partnerships with less tangible deliverables, such as fundamental science or technology development, are more difficult to value, particularly in a way that would enable comparisons across a diverse portfolio of projects. An additional complication is that some partnerships are comprised of multiple projects and can have an impact on different business areas, including those outside of R&D.

Without new assessment tools that enable comparison of such partnerships of disparate nature, R&D organisations will find it challenging to assess their relative value for investment decisions. It was therefore timely to develop a new assessment tool that would allow for differences in the various forms of partnerships and their outputs. In addition, such a tool could be valuable for setting expectations over the lifetime of a project and tracking progress.

Developing a new assessment tool

Our approach was constructed around three core principles that ensured the scoring methodology was both representative of the value added by an individual partnership and that each element of the assessment contributed to the overall score at an appropriate level:

1. Enable meaningful comparison of partnerships Partnerships address a diverse range of scientific disciplines across all phases of R&D, from target identification to clinical studies, with diverse deliverables, such as basic scientific results, best practice recommendations or therapeutic agents. Estimating values based on cost of generating the deliverables relative to other means can be misleading, and ignore the fact that these savings alone are no indication of how important the deliverable is to the sponsoring organisation. Therefore, an assessment tool should not calculate the fiscal value generated from the partnership, but instead assess the impact that the output would have on the sponsoring organisation.

2. Apply weighting profiles to highlight important factors Not all elements of an assessment are necessarily of equal importance. For example, while the impact on non-R&D objectives may be worth considering, the impact on an R&D objective may be of greater importance. Each element of the assessment therefore needs to be weighted according to its individual importance to the business. The actual weightings applied to individual questions are highly dependent on the sponsoring organisation’s strategy. Business requirements may change over time and the ability to change the associated weightings enables the assessment tool to be adapted for subsequent reviews should the strategy change. However, provided the weightings are agreed by a stakeholder group at the start of each process, and fixed throughout the assessment, the analysis should be consistent across projects.

3. Recognise that the value a partnership delivers often changes over time Recently started partnerships may not have had sufficient time, compared to older ones, to deliver the expected value, and this could restrict the utility of a scoring model based solely on tangible deliverables. A number of elements that address both the current value of the partnership and the perceived future value (the value considered realistically achievable within two years) were therefore included in the assessment tool. This understanding of the ‘as is’ and ‘should be’ scenarios enables a more realistic visualisation of the value a portfolio can deliver at a subsequent date, avoiding the danger of making decisions based on current value only. It also helps to identify the key areas where focus will be required to ensure that this future value is realised.

Using the assessment tool

Whilst this approach is applicable for general use across a range of partnerships, in the remainder of this article its use will be exemplified with a sub-set of Public-Private Partnerships (PPPs, also commonly referred to as research consortia) currently supported by GSK. In the context of this discussion, PPPs are defined as partnerships involving more than one pharmaceutical company and at least one public funding agency.

The sub-set of PPPs selected, while diverse in nature, are generally focused on areas of pre-competitive science where the deliverables are less tangible than for therapeutic agent partnerships. Therefore, they are well-placed to demonstrate the value of this new assessment tool. A three-step approach was used:

Stage 1: Build the assessment tool

The assessment tool was based around a weighted scoring methodology, comprising a list of questions (or criteria) within four major categories (Figure 1).

The criteria were carefully selected, worded and defined by the needs of the business, then organised into the categories summarised in Table 1. This enables the criteria to be closely aligned with specific R&D requirements and partnership strategy.

The shaded items in Figure 1 represent timedependent criteria, where the present and future values could potentially be differentiated. The Impact on R&D Productivity and Access to Capability and Talent categories are completely populated with time-dependent criteria, while Strategic Interests and Delivery on Objectives are less so. This difference reflects the observation that alignment of partnership objectives to Strategic Interests, and the way they are managed to deliver on those objectives, should not change considerably over a two-year period.

If this were not the case then the purpose of entering the partnership, or continuing it, should be questioned. Indeed, the only way to increase the value that these categories deliver would be to restructure the partnership. In the case of the Impact on R&D Productivity and Access to Capability and Talent categories, the value realised may change over time, particularly with recently-formed partnerships where the outputs have yet to be adopted or utilised extensively.

It is essential that assessment tools such as these are owned and promoted by senior executives and that the extended business units are fully engaged with both the need for assessment and the methodology employed. Working directly with members of the R&D executive, the criteria were tested and validated for reliability through an iterative series of workshops and feedback sessions where the criteria were challenged, reviewed, weighted and accepted by stakeholders.

Stage 2: Gather the data

Many of the IT solutions used to gather information from key stakeholders do not address the level of complexity that often exists when balancing the needs of a company against the perceived benefits of an external venture. As a result, it is often easy to contradict preceding answers or simply assign a ‘score’ to a criterion without thinking through all the elements of potential value such as strategic alignment or organisational development.

For this reason, data were gathered through personal interviews with key stakeholders for each partnership using the criterion set as a basis for discussion. Each criterion was scored by the interviewee based on the discussion and their supporting comments were gathered.

To avoid middle-ground responses and force decisions, only four possible outcomes to each criterion were allowed, ranging in score from zero to three, which were assigned to a value relevant to that criterion. For example, in the Enhanced Disease and Patient Knowledge criterion, scores zero to three corresponded to ‘Little or no Impact’, ‘Enhanced understanding of disease area’, ‘Significant impact on disease area or enhanced understanding of multiple disease areas’, and ‘Sets new direction in disease area or significant impact on several disease areas’. In each case, the selected score was supported by a full documentation of the collected comments of the interviewees.

Stage 3: Analyse and interpret

The most useful portfolio analysis tools allow the user to explore and debate project data from a number of different angles. In the case of this assessment tool, there was a need to investigate various sub-sets of partnerships or different scoring categories from a number of perspectives. To facilitate data aggregation and analysis, a Microsoft Excel VBA tool was developed to import and analyse the criterion scores and comments that formed the output of key stakeholder interviews. This simple application also provided a convenient route to deliver a suite of graphical outputs that can easily be transferred to other Office applications for presentation and reporting.

Example: Evaluation of Public-Private Partnerships (PPPs)

Once the evaluation tool had been built using criteria appropriate for PPPs, and data gathered via interviews, the final step of analysis and interpretation could take place. Visualising the data in a number of formats, some of which are shown in the next few sections, enabled conclusions to be drawn, at differing levels of detail, either individually or as a portfolio.

An example analysis of individual partnerships is shown in Figure 2, focusing on three partnerships at different stages of their lifecycle, and with markedly different future prospects.

Partnership A

Partnership A (Figure 2a) is a PPP that was initiated recently. It is clear that the partnership is already rating well in the Strategic Interests and Delivery on Objectives categories. However, scores in the other two categories (Impact on R&D Productivity and Access to Capability and Talent) are less impressive.

It was previously suggested that the Strategic Interests and Delivery on Objectives categories often score higher earlier in the PPP lifecycle than the Impact on R&D Productivity and Access to Capability and Talent categories, and the results for Partnership A are in line with this. The future value plot suggests that all four categories will eventually score highly, and Partnership A exemplifies the need to consider the future value in addition to the present value of PPPs throughout the assessment.

It is important to note with Partnership A that it was recognised that full value would probably not be delivered within two years. However, it was felt that estimating the value beyond two years into the future would be so compromised by uncertainty as to be of little value.

Partnership B

The example in (Figure 2b) (Partnership B) is of a PPP started before Partnership A; it is delivering low or no value in all categories and experience thus far suggests it will not improve over time. Partnership B requires a more detailed analysis to determine why the prospect for delivery of value is so poor, both to see if it is salvageable and to understand what lessons could be learned for future partnerships.

Partnership C

Partnership C (Figure 2c) has been established for over a decade; it scores relatively well against all categories and very highly for Delivery on Objectives. Although delivery of value is not expected to increase over the next two years, the value it is already delivering means this is not a cause for concern. Analysis of the assessment data leads to key three observations:

1. New PPPs need time to mature in order to deliver their full potential. This is especially true for those with a primary focus on generating results through new experimental work, where the time taken to recruit staff and plan, perform and interpret experiments is considerable. Partnership A is an example of this. As a result, when accessing relatively new PPPs it is important to recognise the potential value they could deliver, rather than solely the present value. It was also apparent that the time it takes a partnership to deliver full value is commonly underestimated.

2. Established PPPs that are performing less well compared to others need further assessment. PPPs with a low current delivery value and a higher future value, should be monitored to ensure that the expected increase does occur, if necessary providing assistance to ensure this. Where both the present and future value is perceived to be low, such as Partnership B, more detailed analysis is required to understand why the PPP could not deliver better value and whether continued investment is appropriate.

3 . The value delivered by a PPP does not have to increase continually over its lifetime provided it is delivering on the objectives. Established PPPs that deliver consistently high value (like Partnership C) are likely to be the ones that are the most stable and are ideally positioned to draw lessons from.

The partnership value dashboard

In line with the previously-explained concerns about measuring the precise value of deliverables from each partnership, semi-quantitative estimates of value in terms of impact on the R&D pipeline were obtained from combining the scores in the four categories outlined in Figure 1. Presenting these as a dashboard (Figure 3) produces a portfolio view and enables comparisons between partnerships.

Figure 3 shows the overall present value of a sub-set of the PPP portfolio. The oldest PPPs are located at the top of the plot with the newest located at the bottom.

In the following discussion, reference is made to the same three partnerships (A, B and C) as discussed earlier in Figure 2.

The highest value is currently resulting from Partnership C, which is the second oldest PPP in the portfolio sub-set. It is clearly well established and, as highlighted previously, is delivering significant value across all four categories. Similarly, PPPs initiated more recently, in general, deliver lower current value. The tendency for more value to be delivered by older partnerships is not surprising, as all PPPs will take time to deliver maximum value.

In addition, older partnerships that have not been delivering value will have been terminated or not renewed, meaning that an older partnership has to be good to have survived. Hence, in comparing the value of partnerships the likely improvement in delivery should also be considered. That is, focus is placed on the questions about the future value of each PPP (the grey items in Figure 1).

A combined dashboard was developed that shows both the present and future value of the PPP portfolio (Figure 4).

PPPs that were not scored as delivering an increase in overall value are marked with a white X; all the others are shown with the present and future values connected with an arrow in the direction of improvement. There were no examples of PPPs decreasing in expected future value.
This plot allows easy visualisation of the current and future value of the portfolio of partnerships and identifies those where more attention would be beneficial:

• The expected improvement in delivery value for Partnership A suggests that in two years’ time it should be delivering the highest value.
• Lower value PPPs, with the exception of Partnership B, are expected to improve sufficiently over the next two years. These PPPs have all been initiated more recently, so the fact that they will improve is not necessarily surprising. However, it will clearly be important to track their progress to ensure that the expected delivery occurs.
• Partnership C is unlikely to improve further but it is already performing very well.
  
• The current value delivered by Partnership D could be considered acceptable if it was early in its lifecycle, and value is expected to improve on a twoyear time scale. This is not the case, however, and it is appropriate to consider how the value delivered by this PPP could be improved or whether investment in this partnership should be terminated.

Partnership D is delivering almost to its full potential in the Delivery on Objectives category, but not in the others. This suggests it is delivering what it was established to do, but these deliverables are of limited value to the business. Partnership D has been running for more than 10 years, so it is not surprising that Strategic Interests will have changed over this timeframe, which, in turn, could limit the ability of the PPP to provide Access to Capability and Talent, and Impact on R&D Productivity. There are two approaches to consider in order to increase the value realised from Partnership D:

• Change externally: Adapt the objectives of the PPP to align better with current Strategic Interests, rather than those in place at the start of the PPP. This may also enable the partnership to deliver increased value in the Impact on R&D Productivity and Access to Capability and Talent categories.
• Change internally: Realign how the PPP output is utilised internally (eg, bringing the PPP to the attention of new internal groups) to increase the impact of the investment

Conclusions and benefits of a structured assessment tool for partnerships

An assessment tool has been developed and used to evaluate major partnership commitments, identifying those that are:

• Delivering lower value than expected. In this case, programmes of work have been initiated to understand why value is being lost and changes to the partnerships have been implemented to ensure that the deliverables can be realised in the shortest amount of time. Of course, it may not be possible to recover fully some partnerships and, in these cases, sponsors can formalise an exit plan based on a well-constructed business case.
• Delivering high value. Such partnerships are important, not only because they deliver value directly to the R&D pipeline, but also because they are essential tools to learn from to guide the evolution of partnerships and general partnering behaviours. Partnerships such as Partnership C, that deliver consistent value, and Partnership A, that is considered to improve dramatically over time, will be used as benchmarks in future assessments of the portfolio

Conclusion – Assessing the Value of R&D Partnerships

With R&D organisations engaging in many partnerships, it is essential that value creation is identified early, so sponsors can learn from them and evolve their partnering strategy. It is also important that organisations can identify where value is not being realised such that they can take the appropriate action to either improve performance or redirect sponsorship to alternative partnerships of higher value. As such, the assessment process should be conducted regularly to ensure that the predicted increase in the value delivered from the portfolio of partnerships is realised and to build a long-term understanding of the partnering behaviours that drive successful collaborations.

This assessment tool is adaptable to the needs of differing portfolios through adjustment of the criteria and weighting profiles so that they are aligned with the strategic requirements of individual portfolios and organisations. It is also apparent that this assessment tool is applicable to other situations where diverse projects have multiple points of impact across R&D, in both academia and industry, and in sectors beyond Pharma. These share common themes with the partnerships described above:

• Benefits change over time.
• Value is difficult to estimate precisely.
• There is a need to maximise the value of the entire portfolio and make hard choices about future investment.

In summary, this assessment tool provides a mechanism that can:

• Flag easily issues in a portfolio of partnerships and aid the understanding of common factors contributing to success or failure throughout their lifecycle.
• Predict the likely value of partnerships both prior to engagement, and when the full return on investment has yet to be realised, and thereby avoid investment in low value ones.
• Identify areas for key focus in managing existing partnerships to deliver.
• Provide a framework to allow sponsors to consider the value of both intangible and tangible deliverables when a collaboration is under consideration.

Acknowledgements
The authors would like to thank the following for their help with building, testing and implementing this assessment tool: Elaine Irving and Jeff Kipling of Science Environment Development at GSK R&D; Zoe Jones and Ken Fyvie of Life Sciences and Healthcare Practice, PA Consulting Group, and Lyn Scott and Wil Schoenmakers of Technology and Innovation Practice, PA Consulting Group. Additionally, the authors would like to acknowledge the contributions of the partnership representatives across GSK R&D who participated in the interviews and data collection exercise. 

—

This article originally featured in the DDW Winter 2010 Issue

—

Dr David A Pardoe received his PhD in Medicinal and Computational Chemistry from the University of Bristol and has worked in the pharmaceutical industry for more than 20 years. After a research career at GlaxoSmithKline focused on the discovery of small molecule drugs for the treatment of neurological and respiratory conditions, he moved into developing and applying new technologies to address drug discovery cycle times using novel microfluidic and computational tools. Dr Pardoe left GSK in 2006 to pursue a career in technology and innovation consulting where he assists companies in the understanding and adoption of open innovation and externalisation of science in order to access global capability, capacity and knowledge capital. He is the author of several papers and patents in the areas of medicinal and computational chemistry, laboratory automation, microfluidics and novel business models. Dr Pardoe can be contacted at david.pardoe@paconsulting.com

Dr Jackie Hunter has worked in the pharmaceutical industry for more than 25 years and was the former SVP and Head of the Neurology and Gastrointestinal Centre of Excellence for Drug Discovery at GlaxoSmithKline. In 2008 she became Head of Science Environment Development where she was responsible for developing GSK’s Open Innovation strategy for Pharmaceutical R&D. In 2010 she left GSK forming Pharmivation Ltd to concentrate on open innovation in bioscience. She is Non-Executive Director of Proximagen Neuroscience, was named as one of 2010’s Women of Outstanding Achievement in Science, Engineering and Technology for her contribution to innovation and entrepreneurship and was awarded the CBE in June for her contributions to the pharmaceutical industry.

Dr Robert M Cooke received his BSc and PhD in chemistry from the University of Sydney. Following a post-doctorate in the Department of Biochemistry at Oxford, Dr Cooke joined Glaxo in 1989. Since then he has led UK and transatlantic departments in a range of disciplines (including protein production, structural biology, computational chemistry and analytical chemistry). He has chaired matrix committees for portfolios of drug discovery programmes and pioneered external collaborations. He initiated the Structural Genomics Consortium, chairing the SAB throughout its first phase of operations, and is now a Board member and Director of the SGC company. Dr Cooke is also on the advisory boards of several academic ventures including the European Bioinformatics Institute. Dr Cooke can be contacted at rob.m.cooke@gsk.com

Dr Paul Barrett has worked in the consulting industry with PA Consulting Group for 18 years, specialising in assignments in innovation and technology management for international clients in a variety of sectors including healthcare, government and consumer goods. His current areas of activity include R&D transformation, open innovation, development of innovation strategy and portfolio management. Prior to PA, his industrial experience was with process, analytical and medical instrumentation companies in R&D and Marketing management and Business Unit leadership. Dr Barrett has a PhD in Low Temperature Physics from Bristol University and an MBA from the University of Warwick.

References

1 Mullin, R. Chemical and Engineering News, 88 (8), 12-18, 2010.

2 Cooper, RG, Edgett, SJ, Kleinschmidt, EJ. In: Portfolio Management for New Products, Addison-Wesley
Longman Inc. MA,1998.

3 Porkolab, L. Drug Disc. Today, 7 (4),230-231, 2002.