Is there a reproducibility crisis in industrial animal cell culture?

*Corresponding Author:

Matthew Croughan
Bioprocessing Program
Amgen Bioprocessing Center
Keck Graduate Institute
Claremont, CA, USA
E-mail: croughan@kgi.edu

Abstract

Keywords

reproducibility, recombinant proteins, cell therapy

Reproducibility crisis in the fundamental sciences and preclinical drug discovery

Among researchers in the fundamental sciences, as well as preclinical drug discovery, the issue of reproducibility has arisen over roughly the last decade as a topic of high interest and, for many, a crisis [1-4]. Ioannidis presents a modelling framework to support his statement that most research findings are false, due to investigator bias and other factors [1]. Prinz and colleagues at Bayer Healthcare reported that only 25% of published preclinical studies could be sufficiently validated to justify further pursuit of the approach [2]. Begley and colleagues in hematology and oncology preclinical drug discovery at Amgen were able to confirm only 6 out of 54 (11%) of published research findings, even though attempts were made to contact original authors, get their advice, exchange reagents, and occasionally even attempt to repeat the experiment in the lab of the original investigator [3]. Surveys have been conducted showing that only 10% of scientists believe there is no reproducibility crisis or don’t know [4].

In response, efforts have been undertaken at the NIH, as well as many research journals, to improve reproducibility in the fundamental sciences [5,6]. Many of those who helped identify the reproducibility problem have presented recommendations on how to improve the situation [1-7]. Papers have been published regarding how to avoid false positives and not overestimate the reliability of p-values [8,9]. Detailed advice has been given to improve scientific reproducibility in research methods, such as for example small-scale, laboratory-based animal cell culture [10].

Is there a reproducibility crisis in industrial animal cell culture?

Although the reproducibility crisis discussed above may well lead to significant changes in how fundamental academic research is funded, assessed, and published, it has not received broad attention at conferences attended by, or publications read by, those of us in the industrial animal cell culture field. Do we have the same problem or maybe a similar problem? Is there a broad, systemic reproducibility crisis in industrial animal cell culture?

In an article for which comments and advice were collected from both those in industrial cell culture and those working on the reproducibility crisis in cancer research and the fundamental sciences, Hernandez [11] states: “The specific topic of reproducibility in bioprocessing has not been examined in much depth …”. Hernandez presents some interesting ideas on how to enhance bioprocessing efficiencies through run reproducibility.

To date, industrial animal cell culture is primarily utilized to manufacture therapeutic agents, such as vaccines, recombinant proteins, gene therapy vectors, therapeutic cells, and tissues for transplantation. Unlike fundamental academic research, it is performed utilizing current Good Manufacturing Practices (cGMPs). These practices have been developed and improved over many years to help ensure that our manufacturing processes reproducibly provide safe, efficacious, and consistent products. Our processes must be fully characterized, validated, and proven to be robust across the expected range of process variability and inputs, e.g. [12-14]. Our raw materials are subjected to strict quality control testing prior to use. Our assays and reagents are tested, validated, and improved; they are supplemented with other assays, when shown to be inadequate, e.g. [15]. Our staff is thoroughly trained by cGMP trainers. Our equipment must pass installation, operational, and performance qualification (IQ, OQ, and PQ), put on regular maintenance and calibration schedules, and have its performance tracked over time. This all stands in stark contrast to what many of us remember about our graduate or postdoctoral research days, wherein we often had to make due with older equipment, not necessarily regularly maintained nor calibrated, using raw materials not tested prior to use. Many of us had to largely train ourselves through a somewhat painful and time-consuming amount of trial and error.

With strict adherence to cGMPs, it would certainly be shocking if there was a broad, systemic reproducibility crisis in industrial animal cell culture, especially with regard to processes starting from Master Cell Banks (MCBs). For such processes, such as those used for production of recombinant proteins, the starter cultures derive from MCBs already proven free of adventitious agents [13-15] and containing a clonally-derived population of cells already proven to provide stable production of consistent and safe product over the validated range of in-vitro cell age since thaw from the MCB [12-15]. For such products, usually made by fed-batch cell culture, most manufacturing plants run with production culture success rates in the range of 95-100%, standard deviations in the range of 10 – 15% for titer at harvest, and overall gross margins of at least 80-90% on full cost of goods sold, including all aspects of drug substance and drug product costs. All that said, we have had, and will continue to have, certain problems to solve over the years, including many that catch us off guard and result in a very real crisis at the time.

Industrial cell culture is already years ahead in terms of reproducibility efforts

Approximately 30-35 years ago, when industrial CHO cell culture was first being employed to manufacture recombinant proteins for clinical trials, one of the first problems to solve was contamination arising from passaging or feeding of small-scale cultures, such as flasks, spinners, and/or roller bottles, in hoods or biological safety cabinets. With regard to such small-scale cell culture, many of the issues and solutions recently published in 2015 by Freedman et al. [10] were, respectively, identified and implemented back in the 1980s. More extensive measures were also implemented, such as the use of dedicated suites (with only one cell line at any given time), maintained under positive pressure with hepa-filtered air, accessed only by heavily-gowned, highlytrained personnel [16]. Such approaches have been widely employed for two to three decades in industrial cell culture, but apparently are not yet widely known or applied by those in the fundamental sciences and preclinical drug discovery.

There are many others ways in which industrial cell culture is very likely years ahead of common cell culture in the fundamental sciences. Some further examples include our frequent use of

• chemically-defined medium, free of serum and other animal-derived components, e.g. [14,17,18]

• high-temperature, short-time (HTST) treatment of culture medium, for inactivation of microorganisms, including virus, not removed through 0.2-micron or 0.1-micron filters [19-21]

• virally-retentive filters, for process streams not run through HTST systems [19-21]

• well-mixed bioreactors with automatic feed-back control and continuous monitoring of environmental parameters such as pH, dissolved oxygen, and temperature, e.g. [12,17]

• advanced, validated instruments for objective, accurate, reproducible, and frequent assessment of many key cell culture parameters, such as cell number, cell size, and cell viability, as well as the levels of glucose, lactate, ammonia, and many other chemical species in the culture medium, e.g. [12,17,18].

• Cells with complete cell line history files [21], tested and proven to be free of adventitious contaminants including other cell lines [13-15].

Industry versus academia

Although they occasionally undertake certain fundamental studies and technology R & D projects, industrial scientists and engineers typically spend most of their time developing and manufacturing products. This work is performed according to strict timelines and budgets. To make sure most or all of the projects stay on track, substantial efforts are undertaken, such those discussed in the previous two sections above, to ensure predictability and reproducibility. Wellproven development and manufacturing technology platforms are often repeatedly used for multiple products.

In contrast, academic scientists and engineers spend most of their time trying to make ground-breaking discoveries to advance science and engineering. Although it is impossible to predict what will be discovered, by when, and with what impact, certain assurances along those lines are typically required in advance to procure funding. The resulting funding is typically not adequate to cover both the core research and also all of the ongoing reproducibility efforts described in the previous two sections. Furthermore, each project tends to be somewhat unique, and thus not amendable to execution primarily using only well-established platforms. Accordingly, the reproducibility challenges are greater than in industry.

As a somewhat unique group, academics in the field of industrial cell culture should have access to facilities and protocols that meet at least R & D standards in industry. Furthermore, they should more frequently disclose reproducibility issues and receive funding to directly address those issues. They should also receive funding to address universality issues, as discussed further below.

Reproducibility versus universality

In response to the reproducibility crisis in the fundamental sciences and preclinical drug discovery, there has been some push-back from those subjected to increased scrutiny. When notified that his published paper was reported as not reproduced as part of a large reproducibility study, one author mentioned that at least a dozen other labs had successfully replicated his findings [22]. Bissell brings up the point: “But who will evaluate the evaluators?” [23]. The contention partly has to do with the lack of a common definition of “reproducibility”, clearly distinct and separate from related terms, such as “universality”. To address this shortfall, the following definitions are proposed:

Reproducibility: ability to reproduce the finding in any qualified lab or facility, with adequately-trained staff and well maintained, calibrated equipment, using the same cell line, medium, reagents, and methods employed by the original authors.

Universality: ability to reproduce the finding in any qualified lab or facility, with adequately-trained staff and well maintained, calibrated equipment, across a broad range of similarly representative cell lines, medium formulations, and conditions.

To illustrate the definitions above, two examples are given below:

the phenomena reported by Le et al. [24], wherein there is substantial variation in lactic acid accumulation between seemingly identical runs, all with the same cell line and process in the same facility, is one of reproducibility.

the phenomena reported by Luo et al. [17] and Yuk et al. [18], wherein certain cell lines do not shift to a lactate consumption phenotype even when copper is above a typical minimum threshold level, is one of universality.

The field of industrial cell culture typically employs a high level of expectations regarding reproducibility but a pragmatic, mixed view regarding universality. Under an assumption of low universality, we expect different clones and cell lines to behave differently. We thus often isolate, test, and select clones based upon their exhibition of certain individual characteristics, such as high specific productivity with long term stability, high growth rate with extended high viability in bioreactors, lactic consumption phenotype with low ammonia production, and the ability to provide certain product quality profiles [14,17,18]. Under an assumption of some universality, we expect to successfully find a clone that exhibits most or all of these traits in our platform processes, for nearly all product candidates, when engineered into a pre-adapted host cell line. In contrast, in the fundamental sciences, there are greater expectations regarding universality. Researchers expect to find similar results using similarly representative cell lines and protocols, i.e., those viewed as similarly representative of the stated case or scenario. If they don’t find similar results, who decides which cells and/or protocols provide the best representative case for further fundamental investigation?

Avoiding reproducibility crises in future industrial cell culture operations

For industrial animal cell culture processes starting from Master Cell Banks, there is currently no reproducibility crisis. This happy situation has been created, and will hopefully be maintained, through the talent and dedication of those in field, as well as our own unique “culture”. Specific reproducibility problems are often openly revealed to industry colleagues, e.g. [24,25]. Academic, regulatory agency, and/or other government collaborators are brought in to help analyze and/or solve problems, e.g. [11,15,24]. Problem solving approaches and solutions are often shared with others in industry, academia, and government, e.g. [19,24,25]. Industry wide consensus views, based on strong scientific foundations, are brought together and published as needed to address certain key topics, e.g. [14]. When universality is not initially observed, further characterization and mechanism of action studies are often undertaken to understand why, e.g. [17,18].

Many new industrial cell culture processes, such as ones to manufacture certain cell or tissue therapies, do not start from a single Master Cell Bank that will last the entire market lifetime of the product. Instead, cells may be repeatedly sourced on a patientby- patient basis or repeatedly sourced from, for instance, different donors. These cell sourcing strategies presents much larger reproducibility challenges. Given the training, orientation, and track record of those involved, as fellow members of our industrial cell culture family, it is the opinion of this author that they are up to the task of addressing those challenges.

References

1. Ioannidis JPA. Why most published research findings are false. PloS. Med. 2(8), e124 (2005).
2. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets. Nature. Rev. Drug. Discov. 10, 712-713 (2011).
3. Begley CG, Ellis LM. Raise standards for preclinical cancer research. Nature. 483, 531-533 (2012).
4. Baker M. Is there a reproducibility crisis? Nature. 533, 452-454 (2016).
5. Collins FS, Tabak LA. NIH plans to increase reproducibility. Nature. 505, 612-612 (2014).
6. Editorial. Journals united for reproducibility. Nature. 515, 7, (2014).
7. Ioannidis JPA. How to make published research true. PloS. Med. 11(10), e1001747 (2014).
8. MacArthur D. Face up to false positives. Nature. 487, 427-428 (2012).
9. Nuzzo R. Statistical errors. Nature. 506, 150-152 (2014).
10. Freeman LP, Gibson MG, Neve, RM. Changing the culture of cell culture: applying best practices and authentication to ensure scientific reproducibility. Biopharm. Intl. 28(10), 14-21 (2015).
11. Hernandez R. Enhancing bioprocessing efficiencies through run reproducibility. Biopharm. Intl. 30(5), 14-29, (2017).
12. Horvath B, Mun M, Laird MW. Characterization of a monoclonal antibody cell culture production process using a quality by design approach. Mol. Biotechnol. 45, 203-206 (2010).
13. Numerous regulatory guidance documents over many years, e.g., Guidance for industry, Q5A viral safety evaluation of biotechnology products derived from cell lines of human or animal origin. United States Food and Drug Administration. Sept (1998).
14. Frye C, Deshpande R, Estes S et al. Industry view on the relative importance of “clonality” of biopharmaceutical-producing cell lines. Biol. 44, 117-122 (2016).
15. Chandler DKF, Volokhov DV, Chizhikov VE. Historical overview of mycoplasma testing for production of biologics. Am. Pharm. Rev. (2011)
16. Croughan M, Delfosse S, Svay K. Microbial contamination in industrial animal cell culture operations. Pharm. Bioprocess. 2(1), 23-25 (2014).
17. Luo J, Vijayasankaran N, Autsen J et al. Comparative metabolite analysis to understand lactate metabolism shift in Chinese hamper ovary cell culture process. Biotech. Bioeng. 109, 146-156 (2012).
18. Yuk IH, Russell S, Tang Y et al. Effect of cooper on CHO cells: cellular requirements and product quality considerations. Biotechnol. Prog. 31(1), 226-238 (2015).
19. Kiss R. Practicing safe cell culture: applied process designs for minimizing virus contamination risks. PDA J. Pharm. Sci. Technol. 65, 715-729 (2011).
20. Delfosse S, Sathavipat M, Hsu N et al. Trends regarding viral barrier implementation in animal cell culture processes. Pharm. Bioprocess. 1(4), 351-360 (2013).
21. Liu S, Croughan M, Stacey G. Microbial contamination: challenges and perspectives. Pharm. Bioprocess. 2(4), 307-310 (2014).
22. Hernandez R. Reproducibility project only partially able to validate findings of prominent cancer studies. Biopharm. Intl. (2017)
23. Bissell M. The risks of the replication drive. Nature. 503, 333-334 (2013).
24. Le H, Kabbur S, Pollastrini L et al. Multivariate analysis of cell culture bioprocess data - lactate consumption as process indicator. J. Biotech. 162, 210-223 (2012).
25. Garnick RL. Experience with viral contamination in cell culture. Dev. Biol. Stand. 88, 49-56 (1996).