Business
The Reproducibility Crisis
The Reproducibility Crisis - How did we get here?

The Reproducibility Crisis - How did we get here?

By Dr William Olds and Dr Deepa Shankar
Volume 21 Issue 2 - Spring 2020

While the antibody market has evolved into a multibillion-dollar industry enabling the discovery of cures, disease diagnosis and basic science acceleration, this has allowed low quality antibodies to hit the market. With a large crisis in reproducibility, the need for high quality antibodies has never been greater.

Since their development, targeted antibodies have played an essential role in biomedical research. Their strong affinity for specific proteins and assay versatility have made them critical for understanding protein function in academia and pharma. Their research applications are numerous and growing, including Western blotting, chromatin immunoprecipitation and immunohistochemistry. Consequently, a large amount of biology rests upon the shoulders of these reagents and the trust that they are specific and high quality. Scientific advancement is only achievable if previous work forms a solid foundation. The best test of these foundations is reproducibility, but the crumbling of this foundation has been discovered through the ‘reproducibility crisis’. Due to being large workhorses of biology, antibodies have shouldered the blame for the lack of reproducibility.

It is easy to blame corporate greed and malfeasance on the part of antibody vendors, but that is a shortcut and neither captures the totality of the causes nor offers the necessary palliative solutions. To see the path forward, it is necessary to look at the origins of the antibody market and how economic and sociological forces took us to our current situation.

The beginnings of the antibody market

Antibodies in the research setting became widespread with Kohler and Milstein’s Nobel Prize-winning work on developing monoclonal antibody technology in the 1970s (1). Capitalising on and iterating this technology, individual labs then generated antibodies to advance their own research. At this stage, quality was ensured by peer review and academic reputation. Irreproducible results using a poor antibody could jeopardise both a scientist’s standing and his or her professional relationships.

As scientists realised antibodies’ potential for application across a wide range of fields, demand soared. Consequently, devoting time and resources to making antibodies in an academic setting became an untenable situation. It became more efficient for commercial suppliers to produce and distribute antibodies to meet scientists’ demand. Due to the volume and range of antibodies needed, the market was eventually large enough to support the emergence of hundreds of suppliers. Though this development enabled more research than ever before, it created new vulnerabilities. The first is lower quality control.

In academia, the peer review process and social pressure kept quality high as mentioned above. In the commercial setting, the major forces are economic in nature. As long as a company’s antibodies sell and the margins are solid, they will stay in business. In an ideal world, scientists punish companies for poor quality by buying a better offering by a competitor. However, a variety of forces are involved in a customer’s buying process, not just quality. Aspects such as price, stock availability, advertising, search engine optimisation, etc play large roles. Quality then is just one factor affecting businesses’ bottom lines and viability. Thus, the pressure to maintain high quality is lessened in the commercial realm compared to academia.

The second issue is the practice of rebranding other manufacturer’s antibodies. To compete in the market and meet the demands of scientists exploring new targets, suppliers were pushed to have substantial coverage of the human proteome. Developing antibodies is expensive and time-consuming, so many suppliers were incentivised to bulk up their catalogue by buying antibodies from a series of contract manufacturers. These contracts were usually non-exclusive so manufacturers often worked with multiple suppliers. These antibodies were then rebranded for sale to scientists with no mention of source, so scientists are led to believe that different suppliers’ antibodies are unique products. They have no way of knowing that two suppliers may be selling the exact same antibody from the exact same source (Figure 1).

Figure 1 An illustration of the complicated antibody marketplace

This can have disastrous effects on science. For example, a common way to verify antibody A’s results is to compare it with antibody B towards the same target. If the results agree, then antibody A is likely specific and the results can be trusted. This is based upon the premise that it is unlikely that two different antibodies – made from two distinct manufacturers using different immunogens and processes – are both incorrect. However, antibodies A and B, despite coming from two different suppliers, may be from the same poor manufacturer, breaking the entire premise of the experiment. Not only does this practice make purchasing confusing for the customer and thereby make the market imbalanced, it also makes quality control challenging. Putting pressure on the contract manufacturer or finding a new manufacturer is not as effective as strict quality control for in-house production. Proteintech was founded as a response to this poor business practice and sells only its own manufactured products to ensure quality control.

Reproducibility crisis increases pressure for high quality antibodies

An inefficient market with a lack of transparency and feedback loop for quality combined to create the reproducibility crisis facing biology today. This crisis bubbled to the surface in 2012 with Glenn Begley’s article showing that only 11% of high-impact studies could be reproduced (2). According to the Global Biological Standards Institute (GBSI), antibodies are responsible for more than a third of irreproducible results (3). In the years that followed Begley’s article, popular scientific (4) and public outlets (5) raised awareness and made this a high priority issue. Eventually, journals, peer reviewers and funders added requirements to ensure data reproducibility. These changes from the gatekeepers of academic science forced scientists to demand better antibodies and more proof of their specificity. Though it is terrible that so much hard work from scientists may not be reproducible due to forces out of their control, finally bringing this issue to light has brought quality back to the forefront and discourages poor business practices.

In 2016, GBSI brought together funders, journals, scientists and suppliers together to discuss global standards for antibodies in Asilomar, CA. After intense sessions covering the spectrum of perspectives, the stakeholder consensus was to promote genetic knockdown and knockout (KD/KO) as the best indicator of specificity. Though several companies have implemented this, Proteintech has made the strongest commitment to this standard. It was the first antibody supplier to implement this standard (and did so in 2015, prior to the GBSI meeting) and still leads the industry in amount and coverage of KD/KO validated antibodies. Recently, Edfors and colleagues developed the pillars of enhanced validation to supplement KD/KO validation6. Besides genetic manipulation, they highlighted orthogonal strategies, comparison of independent antibodies, recombinant expression strategies and capture mass spectrometry.

At Asilomar and in the larger scientific community, rabbit polyclonal antibodies have been blamed for the reproducibility crisis due to the chance of high lot-to-lot variability. This is unfortunate as polyclonal antibodies are great reagents with generally higher affinity and versatility than monoclonal antibodies. They can be highly consistent lot-to-lot. Unacceptable variability is due to poor quality control and can be minimised by using many rabbits for each target and comparing performance to past lots. Proteintech, for instance, tests each lot against all previous lots to ensure quality and has the lowest complaint rate in the industry (Figure 2).

Figure 2 Proteintech's polyclonal antibody validation flowchart

Another consequence of the reproducibility crisis is the increased demand for monoclonal and recombinant antibodies. Both have low lot-to-lot variation. Monoclonal antibodies are not perfect. Hybridomas occasionally incorporate endogenous chains from the myeloma fusion partner. Additionally, hybridomas are subject to genetic drift. Consequently, players are taking a page from the therapeutic industry and developing recombinant antibodies. The most successful method is rabbit recombinant antibodies. In this method, B cells are harvested from immunised rabbits and the antibody-encoded sequences are cloned to form a library. These antibodies are then expressed in a mammalian cell line and screened in a similar process as monoclonal antibody production.

There are a few advantages to the recombinant approach. First, the antibodies are genetically encoded, so the stability and reproducibility are ensured. It also expands the toolkit for antibodies because, like any transgene, they can be encoded with tags like GFP. Lastly, rabbits have a higher immunological diversity than mice, which can result in more specific antibodies. This approach is not without major disadvantages. The first is cost. A single recombinant antibody can cost up to $50,000, compared to a few thousand dollars for polyclonal and monoclonal antibodies. A related disadvantage is production time. A recombinant takes much longer to develop than monoclonal and polyclonal antibodies. Just like previous technologies, the quality of the antibody depends upon the skill and quality control of the manufacturer. Recombinant are not inherently ‘better’ than monoclonal and polyclonal antibodies. Every antibody has its place and it is important for customers to continue to demand high quality.

What does the future hold for the antibody industry?

The long-term effects of the reproducibility crisis will likely manifest itself in a few ways. Recombinants will become a larger portion of the market because demand has now created a market for these antibodies. Currently, costs and product time make development on a large-scale cost prohibitive. This forces companies to strategically choose a small subset of targets with high sales potential.

Fortunately, these issues will likely be resolved over time, allowing for more than high-profile targets to be developed. There will also be a culling of the market. Due to the demand for higher quality and global coverage, many manufacturers will no longer be able to compete and will struggle to stay in business. Only the top players will be able to satisfy both needs. While this development means that scientists will have good antibodies for whatever they are studying, it has the unfortunate consequence of creating high barriers for entry and competing long-term in the market. A start-up would have a huge challenge to quickly produce a large portfolio of high quality antibodies before their investor money runs dry. A lack of competition may precipitate issues in the future.

There will also be increased demand for company transparency. Since scientists are being pushed to use better reagents, they are being more skeptical about their purchasing and adopting a peer reviewer mindset poring over the data. They want to see the raw film for Western blots, lab notebook pages for experiments online and immunogen sequences. Only companies that are transparent and manufacture their own products can satisfy customers. Extracting this information from contract manufacturers is challenging and time-consuming. Proteintech has led the way in transparency. Their lab notebook pages for all their experiments displayed online are readily available for customers to view. Additionally, their immunogen sequences are displayed online and original raw data for both negative and positive testing are available.

Antibodies have come a long way from their humble beginnings as niche laboratory tools. The market has evolved into a multibillion-dollar enterprise that helps discover cures, diagnose disease, and accelerate basic science. This development has allowed for low quality antibodies to flood the market, unfortunately. However, the reproducibility crisis has given the market a course correction. The awareness of the situation and the elevation of high quality as a major purchasing factor will boost the strong suppliers and punish the poor suppliers. Ultimately, this situation may realise the dream of specific antibodies for all applications and enable better drug development and discoveries. DDW

---

Dr William Olds is the Scientific Officer at Proteintech Group. He is involved in the marketing and strategy of Proteintech’s portfolio of antibodies, human cell-expressed cytokines and ELISA kits. He also hosts The Incubator podcast for career development and exploration for young scientists.

Dr Deepar Shankar is the head of global Research and Development (R&D) at Proteintech. Prior to that she served as Director of R&D and Head of R&D and Manufacturing in South Asia for Thermo Fisher. Before joining Thermo Fisher, Dr Shankar managed 11 oncology programmes as Associate Director of the Oncology Therapeutics Group at the drug discovery company Jubilant Life Sciences, in collaboration with Eli Lilly and Company. Dr Deepa received her PhD in molecular biology and immunology at the University of Southern California in Los Angeles receiving several awards and accolades such as the Heidelberger Award for Cancer Research, UC Chancellor’s research award and the Boyer-Parvin award. During her postdoctoral work, she had opportunities to work with pharmaceutical companies gaining exposure to the clinical research industry due to her strong passion for biology and her focus on cancer research, later developing clinically-tested disease models for leukaemia.

References

1 https://www. ncbi.nlm.nih.gov/pubmed/ 1172191.

2 https://www. nature.com/ articles/483531a.

3 https://www.ncbi. nlm.nih.gov/pubmed/ 26057340.

4 https://www.nature.com/ collections/prbfkwmwvz.

5 https://www.wamc. org/post/nprs-richard-harris-warns-shoddy-science-rigor-mortis.

6 https://www.ncbi.nlm.nih. gov/pubmed/30297845.