On the low reproducibility of cancer studies

On the low reproducibility of cancer studies INTRODUCTION Previous reports have suggested that close to 90% of cancer biology publications are irreproducible. The low number has recently been corroborated by five detailed replication studies in eLife, which have been commented on in Nature (Replication studies offer much more than technical details. Nature 2017;541:259–60). While the irreproducibility is often attributed to human factors, which are remediable, the reason might be biological and the irreproducibility is intrinsic to such studies. The low reproducibility, reflecting the diversity in the evolutionary pathways of tumourigenesis, will likely significantly impact clinical strategies. Reproducibility is the foundation of experimental science. While there are many factors afflicting different fields of inquiry to varying degrees [1], cancer biology studies appear to stand out. In an earlier report, only 6 of 53 published findings in cancer biology could be confirmed [2], a rate approaching an alarmingly low 10% of reproducibility. According to the report, such a low rate is common in the pharmaceutical industry. The low rate is of particular concern because these studies are generally published in ‘high-impact’ journals, thus having real consequences in both clinical practice and basic research. In this critique, we first review recent updates on the extent of reproducibility. Second, the reasons underlying low reproducibility are explored. While concerns about low reproducibility are often expressed in terms of human factors [2–5], there are deeper biological reasons. Third, a distinction is made between studies that should be reproducible and those that are intrinsically irreproducible. Fourth, suggestions are made to cope with irreproducibility issues. UPDATES ON REPRODUCIBILITY INVESTIGATION: THE REPRODUCIBILITY PROJECT: CANCER BIOLOGY The Reproducibility Project: Cancer Biology (RP: CB) was launched to address the issue rigorously [6]. RP: CB has selected 30 cancer studies from high-impact journals for reproducibility investigation. Because failure regarding reproducibility is most likely due to technical factors, ruling out these obvious reasons is the salient contribution of RP: CB. Prior to the actual replication is a phase of Registered Reports, which outline the replication designs. The replication studies themselves only begin after the designs have been reviewed and approved. Since both sides agree on the principle of reproducibility, the challenge is to assure that the materials and methods can be accurately replicated. Now, the first batch of reports has been released [7]. Reproducibility here means an ‘actionable and repeatable therapeutic strategy’ [2], as each of the original studies has a therapeutic proposal. According to the eLife editorial [4], two studies are ‘reproducible in important parts’, one being ‘not reproducible’ and the remaining two being ‘uninterpretable’. Another news report characterizes the overall picture as ‘muddy’ [5]. We found these readings of the replication results to be overly cautious. In a more conventional assessment (see Table 1 for a summary), four of the five original studies should be classified as ‘not reproducible’ and one as ‘uninterpretable’. We shall start with the two reports that the eLife editorial considers to be uninterpretable due to irreproducible controls. As the failure to reproduce the control is no different from the failure to reproduce the experiments, we reclassify the two cases as ‘not reproducible’. In Berger et al. [8], mutations in the PREX2 gene are found to be common in human melanomas (15 out of 107). Furthermore, the introduction of a PREX2 mutation into human melanocytes was reported to substantially reduce the tumour-free survival in xenograft mice. In the replication study, Horrigan et al. [9] (see also Davis [10]) cited recent studies that failed to support the prevalence of PREX2 mutations in human melanomas. In addition, Horrigan et al. failed to corroborate the reduction in tumour-free survival caused by PREX2 mutation because the controls without the mutation die just as rapidly (1 week), in contrast with the original report of 9-week median survival in the control. In the other study, Willingham et al. [11] suggest that the CD47 protein is overexpressed on the membrane of most cancer cells. Since CD47 signals to macrophages to withhold attack, blocking CD47 reduces the mass of orthotopic breast tumours by ∼10 fold in immune competent mice that have been injected with MT1A2 mouse cells, relative to the control. In the replication by Horrigan [12], the tumour sizes are curiously reversed between the CD47- and IgG (control)-treated mice. Horrigan noted that the tumour mass is highly variable, often with a 5-fold difference in the same setting. Such high variability is common is tumour evolution but is often treated as noise. It should be noted that Willingham et al. obtained similar results by transplanting human tumour cells into immune-compromised mice. Horrigan also reported that the xenograft experiments have been reproduced only in some follow-up studies. Table 1. Summary of the first batch of RP: CB studies. Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Yes – reproducible; No – Not reproducible; UI – Uninterpretable. View Large Table 1. Summary of the first batch of RP: CB studies. Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Yes – reproducible; No – Not reproducible; UI – Uninterpretable. View Large The one study that was deemed not to be reproducible was that by Sugahara et al. [13], who showed an increase in drug permeability when tumours were treated with the iRGD peptide. In their study, tumours grow from xenografted prostate cancer cells. Mantis et al. [14] could not reproduce the results but did report some successful replication by other studies [15–17]. While the study by Sugahara et al. was the only study whereby the experiments could not be reproduced, it is possible that either the control or the experiment can be more variable. Thus, it seems curious to consider one type of irreproducibility to be more serious than another. For the remaining two studies that were declared to be ‘essentially reproducible’, one [18] would not be considered reproducible under most circumstances. In Sirota et al., the bioinformatic analysis of drug applications on cell lines led to the identification of cimetidine as an effective agent against lung adenocarcinoma cells. While the replication by Kandela et al. [19] observed the same trend, the difference was not significantly different from the control. The reason, as noted by Dang [20], is that the effect of cimetidine reported by Sirota et al. is too weak to be considered biologically significant. Overall, the variance, in comparison with the small difference in mean, makes it difficult to justify the conclusion that cimetidine is an effective new drug against lung cancer. The only reproducible study in the set of five studies is that of Delmore et al. [21]. In the original study, the molecule (+)-JQ1 was reported to downregulate MYC transcription and reduce the burden of multiple myeloma tumours, resulting in the improved survival of xenograft mice. While the results were successfully replicated by Aird et al. (with some variations) [22], a negative control using an enantiomer (−)-JQ1, which does not impact MYC transcription, showed the same biological effect as (+)-JQ1. Given that the negative control also yielded the (unexpected) benefit, we suggest that the original study should be considered uninterpretable. CAUSES OF IRREPRODUCIBILITY The five replications corroborate the earlier report of reproducibility at a rate of 6 out of 53 [2]. While previous reports have hinted at technical (or even ethical) lapses [2,4–6], the factors cited are common across biological disciplines. Furthermore, given the care invested in RP: CB, the replication efforts should be quite adequate. We therefore seek biological explanations below. Whether or not any experimental study can be replicated depends on the measurements being reproduced. In coin tossing, seeing the same side five times in a row will be reproducible no more than 10% of the time. In cancer biology, reproducibility would mean that tumour progression corresponds to highly constrained courses, akin to tissue development. However, if tumour progression follows evolutionary trajectories, the outcome may be highly variable. The course of evolution is often a multi-step process requiring a suite of genetic changes, each of which is governed by stochastic factors including mutation emergence, random drift, and divergent selective pressures. Since each step is contingent on the previous steps taken, divergent outcomes may result from even a small deviation in an earlier stage. In his book ‘Wonderful Life’, S. J. Gould [23] raised the issue of the reproducibility of evolution itself. He wondered if the same evolutionary trajectory would be followed had ‘the tape of life’ been rewound (see also Conway Morris [24]). ‘Rewinding the tape of life’ back to the time of the Cambrian explosion is of course mere fantasy, but there are indeed evolutionary processes that are continually reiterated. The best example may be the evolution of cancers [25–28]. It is hence curious that the word ‘evolution’ does not appear in the RP:CB registered/replication reports, editorials, and commentaries. Reproducibility of evolution would be equivalent to convergent evolution in which a dominant pathway is repeatedly taken. In convergent evolution, the distinction between phenotype and genotype is crucial. Phenotypic convergence is common in natural populations. Similarly, morphological convergence is a basis on which pathologists define malignancy. The central issue is genotypic convergence: whether the genetic changes underlying the phenotypes are themselves convergent. With constant references to somatic mutation, gene expression, and target therapy, cancer biology publications apparently consider genotypic convergence plausible. With this backdrop, the Cancer Genome Atlas (TCGA) project [29–31] has attempted to identify genes that are commonly mutated in tumours. The results show that genetic convergence is much less frequent than had been hoped for [29]. For example, across 12 cancer types, only two genes are mutated in more than 10% of cases: TP53 and PIK3CA; the former is an outlier in the human genome [32,33] and the latter is a very large gene. The number of frequently mutated genes (in >10% of cases) for a given type of cancer is generally around 10 [29,30]. With such low genic convergence, two cases of the same cancer type usually have few mutated genes in common, or may share no mutated genes at all. These observations suggest that, from a very similar starting point (two human beings), the evolution of cancer usually takes different courses. Even cancer cells from the same starting point (within the same person) would continue to diverge, leading to substantial genetic diversity [34,35] and variable responses to therapeutic treatments [36]. The TCGA results have their parallel in natural populations. While genotypic convergence may be observed for highly specialized traits such as echolocation [37], it is nevertheless rare. For adaptations that do not involve highly specialized constructs, diverse molecular mechanisms may operate and genotypic convergence is not expected. For example, human populations living in the high altitudes of the Tibetan, Ethiopian, and Andean Altiplano plateaus have different genetic mutations for hypoxic adaptation [38]. Recently, the search for molecular convergence has expanded to finding signals in the entire genome [37,39–44]. Again, even with the aid of multiple genomes in the same environment, molecular convergence is rare and the signals rarely exceed those of the background noise. Reproducibility of cancer progression and convergent evolution in organisms is nevertheless observable but the conditions are stringent, i.e. when there is a dominant evolutionary pathway leading to an end state. For example, when the organisms are genetically simple (e.g. viruses), there would often be few genetic solutions. Alternatively, if the selective pressure for specific genetic changes is strong (e.g. [39]), then convergence is a likely outcome. Wu et al. speculate that the selective pressure may be particularly high in ‘liquid tumours’ where cells with a proliferative advantage can spread rapidly and widely [26]. Indeed, chronic myelogenous leukaemia remains one of the best examples of cancer convergence at the genic level with the BCR-ABL translocation being a diagnostic feature [45,46]. In general, when we take into account the multi-phenotypic and multi-genic nature of tumour evolution (see Hanahan and Weinberg on cancer hallmarks [33] and Kandoth [29] on cancer driver genes), as well as the complexity of the mammalian genome, molecular convergence in cancer progression would likely be the exception rather than the rule [26]. As TCGA reveals the low convergence in real-life tumourigenesis, one might still expect a high level of convergence in mouse models when most conditions are under control. Now, the RP: CB studies have cast doubt on the predictability of evolution even in simple models. In the five RP: CB reports, cells from cancer cell lines are transplanted into mice, as xenografts or autografts. In these studies, experimental (E) and control (C) samples are collected, and designated (E1, C1) for the original studies and (E2, C2) for the replications. Tumour growth in each mouse is the culmination of two evolutionary processes. First, the cell populations have been evolving prior to transplantation [47]. Second, these cells subsequently evolve as xeno(auto)-grafts into tumours. While the second stage is widely discussed (see Wu et al. [26] for references), the first stage has been neglected even though cell lines do evolve continuously. This first stage is reminiscent of the classic ‘Luria/Delbruck fluctuations’ [48]. Results from E1, C1, E2, and C2 are all conditional distributions (Fig. 1). The final analysis of the RP: CB rests on the comparisons between E1–C1 and E2–C2. In those replication reports that fail to reproduce the original results, C1 and C2 have not evolved along the same path in two reports, and E1 and E2 have evolved divergently in one (see Table 1). In another report, E1–C1 is too small to be biologically or statistically significant. Tumour evolution in RP: CB may be sketched by a simple genetic model (Fig. 1) that frames evolutionary pathways as conditional probabilities. Each stage of evolution is conditional on prior steps of evolution via segregation of existing polymorphisms and the emergence of de novo mutations. A slight difference in the early stage may pave the way for a much greater divergence at a later time. In stage 2 in Fig. 1, the two replications overlap little in their trajectories and very different tumour phenotypes emerge as a result. A more realistic model than that of Fig. 1 will likely yield even more diverse patterns. We suggest that cancer biology studies should develop explicit evolutionary models, rather than assume simple and reproducible outcomes. Figure 1. View largeDownload slide A model of pathway diversity in tumor evolution. Each step of the pathway is the realization from a probability distribution that is conditional on the previous steps taken. In this model of 12 loci (a–l), one locus may change at each stage. The vertical bar below indicates the evolvable locus, which may change from, say, e to E (further changes are allowed.). The locus that actually changes is marked in red. It is assumed that each locus has positive fitness epistasis with the two adjacent loci on each side (e.g. E interacts positively with C, D, F, and G). Stage 1 represents cell line evolution and stage 2 represents the evolution of these cells into tumours. The evolution of two populations (replications) is portrayed. The evolved genotypes in stage 2 determine the tumour's phenotypes (bottom of the figure), which show no overlap between populations. If 10 samples are taken from each replication, as shown by the small arrows, the two replications would appear to be totally irreproducible. Figure 1. View largeDownload slide A model of pathway diversity in tumor evolution. Each step of the pathway is the realization from a probability distribution that is conditional on the previous steps taken. In this model of 12 loci (a–l), one locus may change at each stage. The vertical bar below indicates the evolvable locus, which may change from, say, e to E (further changes are allowed.). The locus that actually changes is marked in red. It is assumed that each locus has positive fitness epistasis with the two adjacent loci on each side (e.g. E interacts positively with C, D, F, and G). Stage 1 represents cell line evolution and stage 2 represents the evolution of these cells into tumours. The evolution of two populations (replications) is portrayed. The evolved genotypes in stage 2 determine the tumour's phenotypes (bottom of the figure), which show no overlap between populations. If 10 samples are taken from each replication, as shown by the small arrows, the two replications would appear to be totally irreproducible. WHICH STUDIES ARE, OR ARE NOT, REPRODUCIBLE? The RP: CB reports give a glimpse of what may or may not be reproducible [7]. If the phenotype being assayed does not evolve in the course of the experiments, the reproducibility is generally high. For example, the treatment of cells with the chemical JQ1 leads to the reproducible downregulation of MYC transcription [21,22]. Similarly, Horrigan [12] was able to reproduce the toxicity effect resulting in mild anaemia in normal mice, whose tissues have not evolved. In the evolution of tumours, reproducibility would be a function of the number, strength, and length of the evolutionary pathways. The TCGA data suggest that the number of genetic pathways for tumourigenesis must be quite large. As the number of steps in each pathway increases, the number of possible alternatives increases exponentially, rendering many observations irreproducible. We should note that the sort of contingent evolution depicted in Fig. 1 results in variable outcomes that may be difficult to capture by increasing the sample size. Facing the diversity of pathways, cancer biology studies often attempt to isolate cases that share part of their pathways; for examples, lung cancer cases sharing the mutated EGFR gene. These cases may indeed show robust and more reproducible outcomes in responding to EGFR inhibitors [49]. However, such partially defined genetic pathways are still quite diverse and irreproducible results, as in drug resistance, are not uncommon. CONCLUSIONS It is curious that the RP: CB project [4,6], together with the earlier reports [2], are met with near total silence. Perhaps the prevailing view that low reproducibility is attributable to human factors [3,5] does not call for intellectual discourse. This view, which has not been supported by any evidence, may have obscured the more fundamental reason behind irreproducibility. From an evolutionary perspective, low reproducibility is intrinsic to such studies because tumourigenesis does not usually traverse the same evolutionary pathway. Reproducibility is nevertheless the central tenet of cancer biology, which assumes convergent pathways with relatively well-defined genetic changes. Thus, in basic research, genetic changes are identified and therapies are then developed to target these changes. The continual evolution of the underlying genetic architecture means that mutations are ‘moving targets’, both between and within individuals. From this perspective, target–gene therapy operates against evolutionary rules. Traits that do not evolve or that evolve slowly may be better targets. Indeed, the evolution of drug resistance continues to be a major impediment in targeted cancer therapy. The efficacies of the top three monoclonal antibody drugs—bevacizumab, trastuzumab, and rituximab—are instructive [50]. Bevacizumab and trastuzumab targeting VEGF and HER-2, respectively, and have limited success [50,51]. In contrast, rituximab, which targets CD20 on the cell surface of all pre-B cells, significantly improves survival in patients with B-cell lymphoma [52]. The efficacy of rituximab may be due to the fact that it does not target a product of cellular evolution. Recent strategies that have targeted the basal transcription machinery [53,54] are compatible with this view favoring non-moving targets. Finally, many diseases are the (by)products of evolution from the viewpoint of Darwinian medicine [55]. Tumourigenesis is different as it is not merely a product of evolution [56]; it is the process itself, or evolution in action [57]. Lewontin [58] pointed out that the essence of evolution is its variability. The diversity, rather than a standard type, is the subject of interest. In cancer studies, the diversity in evolutionary trajectories is also clinically significant. H. J. Muller remarked that biology has gone too long without evolutionary thinking at the centennial of the publication of ‘On the Origin of Species’. Five decades later, Muller's remarks remain relevant to studies of cancers. Finally, it should be noted that four additional reports were released by eLife [59–62] at or after the time of this submission. The eLife editorial considers three to be reproducible in ‘important parts’ and one as reproducible in ‘some parts’. Among the four studies, all reproducible experiments were carried out on cell lines growing in dishes and the time units for collecting data after treatment were either minutes or hours. The only xenograft tumour model examined was in Shan et al. [59], which tested the efficacy of I-BET151, an inhibitor of the BET bromodomain, as a treatment for mixed-lineage leukaemia (MLL)-fusion leukaemia. The experiment took 2 months, thus allowing for plenty of mutation and selection events. Not unexpectedly, the replication group failed to reproduce the original finding of increased survival in I-BET151-treated mice. The timescale-dependent reproducibility of cancer research strengthens the evolutionary arguments of our report. Acknowledgements This work was supported by the National Natural Science Foundation of China (31730046, 91731302, 91731301), the National Basic Research Program (973 Program) of China (2014CB542006 and 2014CB542005), the 985 Project (33000-18821105), and the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB13040300). REFERENCES 1. Baker M. Nature 2016 ; 533 : 452 – 4 . https://doi.org/10.1038/533452a Crossref Search ADS PubMed 2. Begley CG , Ellis LM . Nature 2012 ; 483 : 531 – 3 . https://doi.org/10.1038/483531a Crossref Search ADS PubMed 3. Nature E . Nature 2017 ; 541 : 259 – 60 . https://doi.org/10.1038/541259b 4. Editorial . Elife 2017 ; 6 : e23693 . Crossref Search ADS PubMed 5. Baker M , Dolgin E . Nature 2017 ; 541 : 269 – 70 . https://doi.org/10.1038/541269a Crossref Search ADS PubMed 6. Errington TM , Iorns E , Gunn W et al. Elife 2014 ; 3 : e04333 . https://doi.org/10.7554/eLife.04333 Crossref Search ADS 7. Nosek BA , Errington TM . Elife 2017 ; 6 : e23383 . https://doi.org/10.7554/eLife.23383 Crossref Search ADS PubMed 8. Berger MF , Hodis E , Heffernan TP et al. Nature 2012 ; 485 : 502 – 6 . https://doi.org/10.1038/nature11071 Crossref Search ADS PubMed 9. Horrigan SK , Courville P , Sampey D et al. Elife 2017 ; 6 : e21634 . https://doi.org/10.7554/eLife.21634 Crossref Search ADS PubMed 10. Davis RJ . Elife 2017 ; 6 : e22662 . https://doi.org/10.7554/eLife.22662 Crossref Search ADS PubMed 11. Willingham SB , Volkmer JP , Gentles AJ et al. Proc Natl Acad Sci USA 2012 ; 109 : 6662 – 7 . https://doi.org/10.1073/pnas.1121623109 Crossref Search ADS PubMed 12. Horrigan SK . Elife 2017 ; 6 : e18173 . https://doi.org/10.7554/eLife.18173 Crossref Search ADS PubMed 13. Sugahara KN , Teesalu T , Karmali PP et al. Science 2010 ; 328 : 1031 – 5 . https://doi.org/10.1126/science.1183057 Crossref Search ADS PubMed 14. Mantis C , Kandela I , Aird F et al. Elife 2017 ; 6 : e17584 . https://doi.org/10.7554/eLife.17584 Crossref Search ADS PubMed 15. Peng ZH , Kopecek J . J Am Chem Soc 2015 ; 137 : 6726 – 9 . https://doi.org/10.1021/jacs.5b00922 Crossref Search ADS PubMed 16. Deng C , Zhang Q , Fu Y et al. ACS Appl Mater Interfaces 2017 ; 9 : 1280 – 92 . https://doi.org/10.1021/acsami.6b13738 Crossref Search ADS PubMed 17. Schmithals C , Koberle V , Korkusuz H et al. Cancer Res 2015 ; 75 : 3147 – 54 . https://doi.org/10.1158/0008-5472.CAN-15-0395 Crossref Search ADS PubMed 18. Sirota M , Dudley JT , Kim J et al. Sci Transl Med 2011 ; 3 : 96ra77 . https://doi.org/10.1126/scitranslmed.3001318 Crossref Search ADS PubMed 19. Kandela I , Aird F , Reproducibility Project: Cancer, B . Elife 2017 ; 6 : e17044 . https://doi.org/10.7554/eLife.17044 Crossref Search ADS PubMed 20. Dang CV . Elife 2017 ; 6 : e22661 . https://doi.org/10.7554/eLife.22661 Crossref Search ADS PubMed 21. Delmore JE , Issa GC , Lemieux ME et al. Cell 2011 ; 146 : 904 – 17 . https://doi.org/10.1016/j.cell.2011.08.017 Crossref Search ADS PubMed 22. Aird F , Kandela I , Mantis C et al. Elife 2017 ; 6 : e21253 . https://doi.org/10.7554/eLife.21253 Crossref Search ADS PubMed 23. Gould SJ . Wonderful Life: The Burgess Shale And The Nature Of History . New York : W.W. Norton , 1989 . 24. Conway Morris S . The Crucible Of Creation: The Burgess Shale And The Rise Of Animals . Oxford; New York : Oxford University Press , 1998 . 25. Merlo LMF , Pepper JW , Reid BJ et al. Nat Rev Cancer 2006 ; 6 : 924 – 35 . https://doi.org/10.1038/nrc2013 Crossref Search ADS PubMed 26. Wu CI , Wang HY , Ling S et al. Annu Rev Genet 2016 ; 50 : 347 – 69 . https://doi.org/10.1146/annurev-genet-112414-054842 Crossref Search ADS PubMed 27. Chen H , He X . Mol Biol Evol 2016 ; 33 : 4 – 12 . https://doi.org/10.1093/molbev/msv212 Crossref Search ADS PubMed 28. Wang H-Y , Chen Y , Tong D et al. Natl Sci Rev 2018 https://doi.org/10.1093/nsr/nwx076 ; 5 : 15 – 7 . https://doi.org/10.1093/nsr/nwx076 Crossref Search ADS 29. Kandoth C , McLellan MD , Vandin F et al. Nature 2013 ; 502 : 333 – 9 . https://doi.org/10.1038/nature12634 Crossref Search ADS PubMed 30. Lawrence MS , Stojanov P , Polak P et al. Nature 2013 ; 499 : 214 – 8 . https://doi.org/10.1038/nature12213 Crossref Search ADS PubMed 31. Vogelstein B , Papadopoulos N , Velculescu VE et al. Science 2013 ; 339 : 1546 – 58 . https://doi.org/10.1126/science.1235122 Crossref Search ADS PubMed 32. Weinberg R. The Biology of Cancer : New York : Garland Science , 2006 . 33. Hanahan D , Weinberg RA . Cell 2011 ; 144 : 646 – 74 . https://doi.org/10.1016/j.cell.2011.02.013 Crossref Search ADS PubMed 34. Sottoriva A , Kang H , Ma Z et al. Nat Genet 2015 ; 47 : 209 – 16 . https://doi.org/10.1038/ng.3214 Crossref Search ADS PubMed 35. Ling S , Hu Z , Yang Z et al. Proc Natl Acad Sci USA 2015 ; 112 : E6496 – 505 . https://doi.org/10.1073/pnas.1519556112 Crossref Search ADS PubMed 36. Turajlic S , Swanton C . Nat Rev Drug Discov 2017 ; 16 : 441 – 2 . https://doi.org/10.1038/nrd.2017.78 Crossref Search ADS PubMed 37. Li Y , Liu Z , Shi P et al. Curr Biol 2010 ; 20 : R55 – 6 . https://doi.org/10.1016/j.cub.2009.11.042 Crossref Search ADS PubMed 38. Udpa N , Ronen R , Zhou D et al. Genome Biol 2014 ; 15 : R36 . https://doi.org/10.1186/gb-2014-15-2-r36 Crossref Search ADS PubMed 39. Xu S , He Z , Guo Z et al. Mol Biol Evol 2017 ; 34 : 1008 – 15 . PubMed 40. Parker J , Tsagkogeorga G , Cotton JA et al. Nature 2013 ; 502 : 228 – 31 . https://doi.org/10.1038/nature12511 Crossref Search ADS PubMed 41. Foote AD , Liu Y , Thomas GW et al. Nat Genet 2015 ; 47 : 272 – 5 . https://doi.org/10.1038/ng.3198 Crossref Search ADS PubMed 42. Thomas GWC , Hahn MW . Mol Biol Evol 2015 ; 32 : 1232 – 6 . https://doi.org/10.1093/molbev/msv013 Crossref Search ADS PubMed 43. Xu SH , He ZW , Zhang Z et al. Natl Sci Rev 2017 ; 4 : 721 – 34 . 44. Zou ZT , Zhang JZ . Mol Biol Evol 2015 ; 32 : 1237 – 41 . https://doi.org/10.1093/molbev/msv014 Crossref Search ADS PubMed 45. Salesse S , Verfaillie CM . Oncogene 2002 ; 21 : 8547 – 59 . https://doi.org/10.1038/sj.onc.1206082 Crossref Search ADS PubMed 46. Ren R . Nat Rev Cancer 2005 ; 5 : 172 – 83 . https://doi.org/10.1038/nrc1567 Crossref Search ADS PubMed 47. Nguyen LV , Cox CL , Eirew P et al. Nat Comms 2014 ; 5 : 5871 . https://doi.org/10.1038/ncomms6871 Crossref Search ADS 48. Luria SE , Delbruck M . Genetics 1943 ; 28 : 491 – 511 . PubMed 49. Casas-Selves M , Kim J , Zhang Z et al. Cancer Res 2012 ; 72 : 4154 – 64 . https://doi.org/10.1158/0008-5472.CAN-11-2848 Crossref Search ADS PubMed 50. Aggarwal S . Nat Rev Drug Discov 2010 ; 9 : 427 – 8 . https://doi.org/10.1038/nrd3186 Crossref Search ADS PubMed 51. Pohlmann PR , Mayer IA , Mernaugh R . Clin Cancer Res 2009 ; 15 : 7479 – 91 . https://doi.org/10.1158/1078-0432.CCR-09-0636 Crossref Search ADS PubMed 52. Coiffier B , Lepage E , Brière J et al. N Engl J Med 2002 ; 346 : 235 – 42 . https://doi.org/10.1056/NEJMoa011795 Crossref Search ADS PubMed 53. Chen Y , Wen H , Wu CI . Oncotarget 2017 ; 8 : 38642 – 9 . https://doi.org/10.18632/oncotarget.16957 PubMed 54. Wang Y , Zhang T , Kwiatkowski N et al. Cell 2015 ; 163 : 174 – 86 . https://doi.org/10.1016/j.cell.2015.08.063 Crossref Search ADS PubMed 55. Williams GC , Nesse RM . Q Rev Biol 1991 ; 66 : 1 – 22 . https://doi.org/10.1086/417048 Crossref Search ADS PubMed 56. Cairns J . Nature 1975 ; 255 : 197 – 200 . https://doi.org/10.1038/255197a0 Crossref Search ADS PubMed 57. Nowell PC . Science 1976 ; 194 : 23 – 8 . https://doi.org/10.1126/science.959840 Crossref Search ADS PubMed 58. Lewontin RC . Am J Hum Genet 1974 ; 26 : 400 – 11 . PubMed 59. Shan X , Fung JJ , Kosaka A et al. Elife 2017 ; doi: 10.7554/eLife.25306 . 60. Showalter MR , Hatakeyama J , Cajka T et al. Elife 2017 ; doi: 10.7554/eLife.26030 . 61. Vanden Heuvel , Maddox E , Maalouf SW et al. Elife 2018 ; doi: 10.7554/eLife.29747 . 62. Lewis LM , Edwards MC , Meyers ZR et al. Elife 2018 ; doi: 10.7554/eLife.30274 . © The Author(s) 2018. Published by Oxford University Press on behalf of China Science Publishing & Media Ltd. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png National Science Review Oxford University Press

On the low reproducibility of cancer studies

Loading next page...
 
/lp/ou_press/on-the-low-reproducibility-of-cancer-studies-l5v0jOBU97
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of China Science Publishing & Media Ltd.
ISSN
2095-5138
eISSN
2053-714X
D.O.I.
10.1093/nsr/nwy021
Publisher site
See Article on Publisher Site

Abstract

INTRODUCTION Previous reports have suggested that close to 90% of cancer biology publications are irreproducible. The low number has recently been corroborated by five detailed replication studies in eLife, which have been commented on in Nature (Replication studies offer much more than technical details. Nature 2017;541:259–60). While the irreproducibility is often attributed to human factors, which are remediable, the reason might be biological and the irreproducibility is intrinsic to such studies. The low reproducibility, reflecting the diversity in the evolutionary pathways of tumourigenesis, will likely significantly impact clinical strategies. Reproducibility is the foundation of experimental science. While there are many factors afflicting different fields of inquiry to varying degrees [1], cancer biology studies appear to stand out. In an earlier report, only 6 of 53 published findings in cancer biology could be confirmed [2], a rate approaching an alarmingly low 10% of reproducibility. According to the report, such a low rate is common in the pharmaceutical industry. The low rate is of particular concern because these studies are generally published in ‘high-impact’ journals, thus having real consequences in both clinical practice and basic research. In this critique, we first review recent updates on the extent of reproducibility. Second, the reasons underlying low reproducibility are explored. While concerns about low reproducibility are often expressed in terms of human factors [2–5], there are deeper biological reasons. Third, a distinction is made between studies that should be reproducible and those that are intrinsically irreproducible. Fourth, suggestions are made to cope with irreproducibility issues. UPDATES ON REPRODUCIBILITY INVESTIGATION: THE REPRODUCIBILITY PROJECT: CANCER BIOLOGY The Reproducibility Project: Cancer Biology (RP: CB) was launched to address the issue rigorously [6]. RP: CB has selected 30 cancer studies from high-impact journals for reproducibility investigation. Because failure regarding reproducibility is most likely due to technical factors, ruling out these obvious reasons is the salient contribution of RP: CB. Prior to the actual replication is a phase of Registered Reports, which outline the replication designs. The replication studies themselves only begin after the designs have been reviewed and approved. Since both sides agree on the principle of reproducibility, the challenge is to assure that the materials and methods can be accurately replicated. Now, the first batch of reports has been released [7]. Reproducibility here means an ‘actionable and repeatable therapeutic strategy’ [2], as each of the original studies has a therapeutic proposal. According to the eLife editorial [4], two studies are ‘reproducible in important parts’, one being ‘not reproducible’ and the remaining two being ‘uninterpretable’. Another news report characterizes the overall picture as ‘muddy’ [5]. We found these readings of the replication results to be overly cautious. In a more conventional assessment (see Table 1 for a summary), four of the five original studies should be classified as ‘not reproducible’ and one as ‘uninterpretable’. We shall start with the two reports that the eLife editorial considers to be uninterpretable due to irreproducible controls. As the failure to reproduce the control is no different from the failure to reproduce the experiments, we reclassify the two cases as ‘not reproducible’. In Berger et al. [8], mutations in the PREX2 gene are found to be common in human melanomas (15 out of 107). Furthermore, the introduction of a PREX2 mutation into human melanocytes was reported to substantially reduce the tumour-free survival in xenograft mice. In the replication study, Horrigan et al. [9] (see also Davis [10]) cited recent studies that failed to support the prevalence of PREX2 mutations in human melanomas. In addition, Horrigan et al. failed to corroborate the reduction in tumour-free survival caused by PREX2 mutation because the controls without the mutation die just as rapidly (1 week), in contrast with the original report of 9-week median survival in the control. In the other study, Willingham et al. [11] suggest that the CD47 protein is overexpressed on the membrane of most cancer cells. Since CD47 signals to macrophages to withhold attack, blocking CD47 reduces the mass of orthotopic breast tumours by ∼10 fold in immune competent mice that have been injected with MT1A2 mouse cells, relative to the control. In the replication by Horrigan [12], the tumour sizes are curiously reversed between the CD47- and IgG (control)-treated mice. Horrigan noted that the tumour mass is highly variable, often with a 5-fold difference in the same setting. Such high variability is common is tumour evolution but is often treated as noise. It should be noted that Willingham et al. obtained similar results by transplanting human tumour cells into immune-compromised mice. Horrigan also reported that the xenograft experiments have been reproduced only in some follow-up studies. Table 1. Summary of the first batch of RP: CB studies. Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Yes – reproducible; No – Not reproducible; UI – Uninterpretable. View Large Table 1. Summary of the first batch of RP: CB studies. Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Studies eLife editorial This study Comments Sirota et al. [18] Kandela et al. [19] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al. [21] Aird et al. [22] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al. [13] Mantis et al. [14] No No Co-administration of iRGD with chemo-agent enhances drug uptake by tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al. [8] Horrigan et al. [9] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al. [11] Horrigan [12] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Yes – reproducible; No – Not reproducible; UI – Uninterpretable. View Large The one study that was deemed not to be reproducible was that by Sugahara et al. [13], who showed an increase in drug permeability when tumours were treated with the iRGD peptide. In their study, tumours grow from xenografted prostate cancer cells. Mantis et al. [14] could not reproduce the results but did report some successful replication by other studies [15–17]. While the study by Sugahara et al. was the only study whereby the experiments could not be reproduced, it is possible that either the control or the experiment can be more variable. Thus, it seems curious to consider one type of irreproducibility to be more serious than another. For the remaining two studies that were declared to be ‘essentially reproducible’, one [18] would not be considered reproducible under most circumstances. In Sirota et al., the bioinformatic analysis of drug applications on cell lines led to the identification of cimetidine as an effective agent against lung adenocarcinoma cells. While the replication by Kandela et al. [19] observed the same trend, the difference was not significantly different from the control. The reason, as noted by Dang [20], is that the effect of cimetidine reported by Sirota et al. is too weak to be considered biologically significant. Overall, the variance, in comparison with the small difference in mean, makes it difficult to justify the conclusion that cimetidine is an effective new drug against lung cancer. The only reproducible study in the set of five studies is that of Delmore et al. [21]. In the original study, the molecule (+)-JQ1 was reported to downregulate MYC transcription and reduce the burden of multiple myeloma tumours, resulting in the improved survival of xenograft mice. While the results were successfully replicated by Aird et al. (with some variations) [22], a negative control using an enantiomer (−)-JQ1, which does not impact MYC transcription, showed the same biological effect as (+)-JQ1. Given that the negative control also yielded the (unexpected) benefit, we suggest that the original study should be considered uninterpretable. CAUSES OF IRREPRODUCIBILITY The five replications corroborate the earlier report of reproducibility at a rate of 6 out of 53 [2]. While previous reports have hinted at technical (or even ethical) lapses [2,4–6], the factors cited are common across biological disciplines. Furthermore, given the care invested in RP: CB, the replication efforts should be quite adequate. We therefore seek biological explanations below. Whether or not any experimental study can be replicated depends on the measurements being reproduced. In coin tossing, seeing the same side five times in a row will be reproducible no more than 10% of the time. In cancer biology, reproducibility would mean that tumour progression corresponds to highly constrained courses, akin to tissue development. However, if tumour progression follows evolutionary trajectories, the outcome may be highly variable. The course of evolution is often a multi-step process requiring a suite of genetic changes, each of which is governed by stochastic factors including mutation emergence, random drift, and divergent selective pressures. Since each step is contingent on the previous steps taken, divergent outcomes may result from even a small deviation in an earlier stage. In his book ‘Wonderful Life’, S. J. Gould [23] raised the issue of the reproducibility of evolution itself. He wondered if the same evolutionary trajectory would be followed had ‘the tape of life’ been rewound (see also Conway Morris [24]). ‘Rewinding the tape of life’ back to the time of the Cambrian explosion is of course mere fantasy, but there are indeed evolutionary processes that are continually reiterated. The best example may be the evolution of cancers [25–28]. It is hence curious that the word ‘evolution’ does not appear in the RP:CB registered/replication reports, editorials, and commentaries. Reproducibility of evolution would be equivalent to convergent evolution in which a dominant pathway is repeatedly taken. In convergent evolution, the distinction between phenotype and genotype is crucial. Phenotypic convergence is common in natural populations. Similarly, morphological convergence is a basis on which pathologists define malignancy. The central issue is genotypic convergence: whether the genetic changes underlying the phenotypes are themselves convergent. With constant references to somatic mutation, gene expression, and target therapy, cancer biology publications apparently consider genotypic convergence plausible. With this backdrop, the Cancer Genome Atlas (TCGA) project [29–31] has attempted to identify genes that are commonly mutated in tumours. The results show that genetic convergence is much less frequent than had been hoped for [29]. For example, across 12 cancer types, only two genes are mutated in more than 10% of cases: TP53 and PIK3CA; the former is an outlier in the human genome [32,33] and the latter is a very large gene. The number of frequently mutated genes (in >10% of cases) for a given type of cancer is generally around 10 [29,30]. With such low genic convergence, two cases of the same cancer type usually have few mutated genes in common, or may share no mutated genes at all. These observations suggest that, from a very similar starting point (two human beings), the evolution of cancer usually takes different courses. Even cancer cells from the same starting point (within the same person) would continue to diverge, leading to substantial genetic diversity [34,35] and variable responses to therapeutic treatments [36]. The TCGA results have their parallel in natural populations. While genotypic convergence may be observed for highly specialized traits such as echolocation [37], it is nevertheless rare. For adaptations that do not involve highly specialized constructs, diverse molecular mechanisms may operate and genotypic convergence is not expected. For example, human populations living in the high altitudes of the Tibetan, Ethiopian, and Andean Altiplano plateaus have different genetic mutations for hypoxic adaptation [38]. Recently, the search for molecular convergence has expanded to finding signals in the entire genome [37,39–44]. Again, even with the aid of multiple genomes in the same environment, molecular convergence is rare and the signals rarely exceed those of the background noise. Reproducibility of cancer progression and convergent evolution in organisms is nevertheless observable but the conditions are stringent, i.e. when there is a dominant evolutionary pathway leading to an end state. For example, when the organisms are genetically simple (e.g. viruses), there would often be few genetic solutions. Alternatively, if the selective pressure for specific genetic changes is strong (e.g. [39]), then convergence is a likely outcome. Wu et al. speculate that the selective pressure may be particularly high in ‘liquid tumours’ where cells with a proliferative advantage can spread rapidly and widely [26]. Indeed, chronic myelogenous leukaemia remains one of the best examples of cancer convergence at the genic level with the BCR-ABL translocation being a diagnostic feature [45,46]. In general, when we take into account the multi-phenotypic and multi-genic nature of tumour evolution (see Hanahan and Weinberg on cancer hallmarks [33] and Kandoth [29] on cancer driver genes), as well as the complexity of the mammalian genome, molecular convergence in cancer progression would likely be the exception rather than the rule [26]. As TCGA reveals the low convergence in real-life tumourigenesis, one might still expect a high level of convergence in mouse models when most conditions are under control. Now, the RP: CB studies have cast doubt on the predictability of evolution even in simple models. In the five RP: CB reports, cells from cancer cell lines are transplanted into mice, as xenografts or autografts. In these studies, experimental (E) and control (C) samples are collected, and designated (E1, C1) for the original studies and (E2, C2) for the replications. Tumour growth in each mouse is the culmination of two evolutionary processes. First, the cell populations have been evolving prior to transplantation [47]. Second, these cells subsequently evolve as xeno(auto)-grafts into tumours. While the second stage is widely discussed (see Wu et al. [26] for references), the first stage has been neglected even though cell lines do evolve continuously. This first stage is reminiscent of the classic ‘Luria/Delbruck fluctuations’ [48]. Results from E1, C1, E2, and C2 are all conditional distributions (Fig. 1). The final analysis of the RP: CB rests on the comparisons between E1–C1 and E2–C2. In those replication reports that fail to reproduce the original results, C1 and C2 have not evolved along the same path in two reports, and E1 and E2 have evolved divergently in one (see Table 1). In another report, E1–C1 is too small to be biologically or statistically significant. Tumour evolution in RP: CB may be sketched by a simple genetic model (Fig. 1) that frames evolutionary pathways as conditional probabilities. Each stage of evolution is conditional on prior steps of evolution via segregation of existing polymorphisms and the emergence of de novo mutations. A slight difference in the early stage may pave the way for a much greater divergence at a later time. In stage 2 in Fig. 1, the two replications overlap little in their trajectories and very different tumour phenotypes emerge as a result. A more realistic model than that of Fig. 1 will likely yield even more diverse patterns. We suggest that cancer biology studies should develop explicit evolutionary models, rather than assume simple and reproducible outcomes. Figure 1. View largeDownload slide A model of pathway diversity in tumor evolution. Each step of the pathway is the realization from a probability distribution that is conditional on the previous steps taken. In this model of 12 loci (a–l), one locus may change at each stage. The vertical bar below indicates the evolvable locus, which may change from, say, e to E (further changes are allowed.). The locus that actually changes is marked in red. It is assumed that each locus has positive fitness epistasis with the two adjacent loci on each side (e.g. E interacts positively with C, D, F, and G). Stage 1 represents cell line evolution and stage 2 represents the evolution of these cells into tumours. The evolution of two populations (replications) is portrayed. The evolved genotypes in stage 2 determine the tumour's phenotypes (bottom of the figure), which show no overlap between populations. If 10 samples are taken from each replication, as shown by the small arrows, the two replications would appear to be totally irreproducible. Figure 1. View largeDownload slide A model of pathway diversity in tumor evolution. Each step of the pathway is the realization from a probability distribution that is conditional on the previous steps taken. In this model of 12 loci (a–l), one locus may change at each stage. The vertical bar below indicates the evolvable locus, which may change from, say, e to E (further changes are allowed.). The locus that actually changes is marked in red. It is assumed that each locus has positive fitness epistasis with the two adjacent loci on each side (e.g. E interacts positively with C, D, F, and G). Stage 1 represents cell line evolution and stage 2 represents the evolution of these cells into tumours. The evolution of two populations (replications) is portrayed. The evolved genotypes in stage 2 determine the tumour's phenotypes (bottom of the figure), which show no overlap between populations. If 10 samples are taken from each replication, as shown by the small arrows, the two replications would appear to be totally irreproducible. WHICH STUDIES ARE, OR ARE NOT, REPRODUCIBLE? The RP: CB reports give a glimpse of what may or may not be reproducible [7]. If the phenotype being assayed does not evolve in the course of the experiments, the reproducibility is generally high. For example, the treatment of cells with the chemical JQ1 leads to the reproducible downregulation of MYC transcription [21,22]. Similarly, Horrigan [12] was able to reproduce the toxicity effect resulting in mild anaemia in normal mice, whose tissues have not evolved. In the evolution of tumours, reproducibility would be a function of the number, strength, and length of the evolutionary pathways. The TCGA data suggest that the number of genetic pathways for tumourigenesis must be quite large. As the number of steps in each pathway increases, the number of possible alternatives increases exponentially, rendering many observations irreproducible. We should note that the sort of contingent evolution depicted in Fig. 1 results in variable outcomes that may be difficult to capture by increasing the sample size. Facing the diversity of pathways, cancer biology studies often attempt to isolate cases that share part of their pathways; for examples, lung cancer cases sharing the mutated EGFR gene. These cases may indeed show robust and more reproducible outcomes in responding to EGFR inhibitors [49]. However, such partially defined genetic pathways are still quite diverse and irreproducible results, as in drug resistance, are not uncommon. CONCLUSIONS It is curious that the RP: CB project [4,6], together with the earlier reports [2], are met with near total silence. Perhaps the prevailing view that low reproducibility is attributable to human factors [3,5] does not call for intellectual discourse. This view, which has not been supported by any evidence, may have obscured the more fundamental reason behind irreproducibility. From an evolutionary perspective, low reproducibility is intrinsic to such studies because tumourigenesis does not usually traverse the same evolutionary pathway. Reproducibility is nevertheless the central tenet of cancer biology, which assumes convergent pathways with relatively well-defined genetic changes. Thus, in basic research, genetic changes are identified and therapies are then developed to target these changes. The continual evolution of the underlying genetic architecture means that mutations are ‘moving targets’, both between and within individuals. From this perspective, target–gene therapy operates against evolutionary rules. Traits that do not evolve or that evolve slowly may be better targets. Indeed, the evolution of drug resistance continues to be a major impediment in targeted cancer therapy. The efficacies of the top three monoclonal antibody drugs—bevacizumab, trastuzumab, and rituximab—are instructive [50]. Bevacizumab and trastuzumab targeting VEGF and HER-2, respectively, and have limited success [50,51]. In contrast, rituximab, which targets CD20 on the cell surface of all pre-B cells, significantly improves survival in patients with B-cell lymphoma [52]. The efficacy of rituximab may be due to the fact that it does not target a product of cellular evolution. Recent strategies that have targeted the basal transcription machinery [53,54] are compatible with this view favoring non-moving targets. Finally, many diseases are the (by)products of evolution from the viewpoint of Darwinian medicine [55]. Tumourigenesis is different as it is not merely a product of evolution [56]; it is the process itself, or evolution in action [57]. Lewontin [58] pointed out that the essence of evolution is its variability. The diversity, rather than a standard type, is the subject of interest. In cancer studies, the diversity in evolutionary trajectories is also clinically significant. H. J. Muller remarked that biology has gone too long without evolutionary thinking at the centennial of the publication of ‘On the Origin of Species’. Five decades later, Muller's remarks remain relevant to studies of cancers. Finally, it should be noted that four additional reports were released by eLife [59–62] at or after the time of this submission. The eLife editorial considers three to be reproducible in ‘important parts’ and one as reproducible in ‘some parts’. Among the four studies, all reproducible experiments were carried out on cell lines growing in dishes and the time units for collecting data after treatment were either minutes or hours. The only xenograft tumour model examined was in Shan et al. [59], which tested the efficacy of I-BET151, an inhibitor of the BET bromodomain, as a treatment for mixed-lineage leukaemia (MLL)-fusion leukaemia. The experiment took 2 months, thus allowing for plenty of mutation and selection events. Not unexpectedly, the replication group failed to reproduce the original finding of increased survival in I-BET151-treated mice. The timescale-dependent reproducibility of cancer research strengthens the evolutionary arguments of our report. Acknowledgements This work was supported by the National Natural Science Foundation of China (31730046, 91731302, 91731301), the National Basic Research Program (973 Program) of China (2014CB542006 and 2014CB542005), the 985 Project (33000-18821105), and the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB13040300). REFERENCES 1. Baker M. Nature 2016 ; 533 : 452 – 4 . https://doi.org/10.1038/533452a Crossref Search ADS PubMed 2. Begley CG , Ellis LM . Nature 2012 ; 483 : 531 – 3 . https://doi.org/10.1038/483531a Crossref Search ADS PubMed 3. Nature E . Nature 2017 ; 541 : 259 – 60 . https://doi.org/10.1038/541259b 4. Editorial . Elife 2017 ; 6 : e23693 . Crossref Search ADS PubMed 5. Baker M , Dolgin E . Nature 2017 ; 541 : 269 – 70 . https://doi.org/10.1038/541269a Crossref Search ADS PubMed 6. Errington TM , Iorns E , Gunn W et al. Elife 2014 ; 3 : e04333 . https://doi.org/10.7554/eLife.04333 Crossref Search ADS 7. Nosek BA , Errington TM . Elife 2017 ; 6 : e23383 . https://doi.org/10.7554/eLife.23383 Crossref Search ADS PubMed 8. Berger MF , Hodis E , Heffernan TP et al. Nature 2012 ; 485 : 502 – 6 . https://doi.org/10.1038/nature11071 Crossref Search ADS PubMed 9. Horrigan SK , Courville P , Sampey D et al. Elife 2017 ; 6 : e21634 . https://doi.org/10.7554/eLife.21634 Crossref Search ADS PubMed 10. Davis RJ . Elife 2017 ; 6 : e22662 . https://doi.org/10.7554/eLife.22662 Crossref Search ADS PubMed 11. Willingham SB , Volkmer JP , Gentles AJ et al. Proc Natl Acad Sci USA 2012 ; 109 : 6662 – 7 . https://doi.org/10.1073/pnas.1121623109 Crossref Search ADS PubMed 12. Horrigan SK . Elife 2017 ; 6 : e18173 . https://doi.org/10.7554/eLife.18173 Crossref Search ADS PubMed 13. Sugahara KN , Teesalu T , Karmali PP et al. Science 2010 ; 328 : 1031 – 5 . https://doi.org/10.1126/science.1183057 Crossref Search ADS PubMed 14. Mantis C , Kandela I , Aird F et al. Elife 2017 ; 6 : e17584 . https://doi.org/10.7554/eLife.17584 Crossref Search ADS PubMed 15. Peng ZH , Kopecek J . J Am Chem Soc 2015 ; 137 : 6726 – 9 . https://doi.org/10.1021/jacs.5b00922 Crossref Search ADS PubMed 16. Deng C , Zhang Q , Fu Y et al. ACS Appl Mater Interfaces 2017 ; 9 : 1280 – 92 . https://doi.org/10.1021/acsami.6b13738 Crossref Search ADS PubMed 17. Schmithals C , Koberle V , Korkusuz H et al. Cancer Res 2015 ; 75 : 3147 – 54 . https://doi.org/10.1158/0008-5472.CAN-15-0395 Crossref Search ADS PubMed 18. Sirota M , Dudley JT , Kim J et al. Sci Transl Med 2011 ; 3 : 96ra77 . https://doi.org/10.1126/scitranslmed.3001318 Crossref Search ADS PubMed 19. Kandela I , Aird F , Reproducibility Project: Cancer, B . Elife 2017 ; 6 : e17044 . https://doi.org/10.7554/eLife.17044 Crossref Search ADS PubMed 20. Dang CV . Elife 2017 ; 6 : e22661 . https://doi.org/10.7554/eLife.22661 Crossref Search ADS PubMed 21. Delmore JE , Issa GC , Lemieux ME et al. Cell 2011 ; 146 : 904 – 17 . https://doi.org/10.1016/j.cell.2011.08.017 Crossref Search ADS PubMed 22. Aird F , Kandela I , Mantis C et al. Elife 2017 ; 6 : e21253 . https://doi.org/10.7554/eLife.21253 Crossref Search ADS PubMed 23. Gould SJ . Wonderful Life: The Burgess Shale And The Nature Of History . New York : W.W. Norton , 1989 . 24. Conway Morris S . The Crucible Of Creation: The Burgess Shale And The Rise Of Animals . Oxford; New York : Oxford University Press , 1998 . 25. Merlo LMF , Pepper JW , Reid BJ et al. Nat Rev Cancer 2006 ; 6 : 924 – 35 . https://doi.org/10.1038/nrc2013 Crossref Search ADS PubMed 26. Wu CI , Wang HY , Ling S et al. Annu Rev Genet 2016 ; 50 : 347 – 69 . https://doi.org/10.1146/annurev-genet-112414-054842 Crossref Search ADS PubMed 27. Chen H , He X . Mol Biol Evol 2016 ; 33 : 4 – 12 . https://doi.org/10.1093/molbev/msv212 Crossref Search ADS PubMed 28. Wang H-Y , Chen Y , Tong D et al. Natl Sci Rev 2018 https://doi.org/10.1093/nsr/nwx076 ; 5 : 15 – 7 . https://doi.org/10.1093/nsr/nwx076 Crossref Search ADS 29. Kandoth C , McLellan MD , Vandin F et al. Nature 2013 ; 502 : 333 – 9 . https://doi.org/10.1038/nature12634 Crossref Search ADS PubMed 30. Lawrence MS , Stojanov P , Polak P et al. Nature 2013 ; 499 : 214 – 8 . https://doi.org/10.1038/nature12213 Crossref Search ADS PubMed 31. Vogelstein B , Papadopoulos N , Velculescu VE et al. Science 2013 ; 339 : 1546 – 58 . https://doi.org/10.1126/science.1235122 Crossref Search ADS PubMed 32. Weinberg R. The Biology of Cancer : New York : Garland Science , 2006 . 33. Hanahan D , Weinberg RA . Cell 2011 ; 144 : 646 – 74 . https://doi.org/10.1016/j.cell.2011.02.013 Crossref Search ADS PubMed 34. Sottoriva A , Kang H , Ma Z et al. Nat Genet 2015 ; 47 : 209 – 16 . https://doi.org/10.1038/ng.3214 Crossref Search ADS PubMed 35. Ling S , Hu Z , Yang Z et al. Proc Natl Acad Sci USA 2015 ; 112 : E6496 – 505 . https://doi.org/10.1073/pnas.1519556112 Crossref Search ADS PubMed 36. Turajlic S , Swanton C . Nat Rev Drug Discov 2017 ; 16 : 441 – 2 . https://doi.org/10.1038/nrd.2017.78 Crossref Search ADS PubMed 37. Li Y , Liu Z , Shi P et al. Curr Biol 2010 ; 20 : R55 – 6 . https://doi.org/10.1016/j.cub.2009.11.042 Crossref Search ADS PubMed 38. Udpa N , Ronen R , Zhou D et al. Genome Biol 2014 ; 15 : R36 . https://doi.org/10.1186/gb-2014-15-2-r36 Crossref Search ADS PubMed 39. Xu S , He Z , Guo Z et al. Mol Biol Evol 2017 ; 34 : 1008 – 15 . PubMed 40. Parker J , Tsagkogeorga G , Cotton JA et al. Nature 2013 ; 502 : 228 – 31 . https://doi.org/10.1038/nature12511 Crossref Search ADS PubMed 41. Foote AD , Liu Y , Thomas GW et al. Nat Genet 2015 ; 47 : 272 – 5 . https://doi.org/10.1038/ng.3198 Crossref Search ADS PubMed 42. Thomas GWC , Hahn MW . Mol Biol Evol 2015 ; 32 : 1232 – 6 . https://doi.org/10.1093/molbev/msv013 Crossref Search ADS PubMed 43. Xu SH , He ZW , Zhang Z et al. Natl Sci Rev 2017 ; 4 : 721 – 34 . 44. Zou ZT , Zhang JZ . Mol Biol Evol 2015 ; 32 : 1237 – 41 . https://doi.org/10.1093/molbev/msv014 Crossref Search ADS PubMed 45. Salesse S , Verfaillie CM . Oncogene 2002 ; 21 : 8547 – 59 . https://doi.org/10.1038/sj.onc.1206082 Crossref Search ADS PubMed 46. Ren R . Nat Rev Cancer 2005 ; 5 : 172 – 83 . https://doi.org/10.1038/nrc1567 Crossref Search ADS PubMed 47. Nguyen LV , Cox CL , Eirew P et al. Nat Comms 2014 ; 5 : 5871 . https://doi.org/10.1038/ncomms6871 Crossref Search ADS 48. Luria SE , Delbruck M . Genetics 1943 ; 28 : 491 – 511 . PubMed 49. Casas-Selves M , Kim J , Zhang Z et al. Cancer Res 2012 ; 72 : 4154 – 64 . https://doi.org/10.1158/0008-5472.CAN-11-2848 Crossref Search ADS PubMed 50. Aggarwal S . Nat Rev Drug Discov 2010 ; 9 : 427 – 8 . https://doi.org/10.1038/nrd3186 Crossref Search ADS PubMed 51. Pohlmann PR , Mayer IA , Mernaugh R . Clin Cancer Res 2009 ; 15 : 7479 – 91 . https://doi.org/10.1158/1078-0432.CCR-09-0636 Crossref Search ADS PubMed 52. Coiffier B , Lepage E , Brière J et al. N Engl J Med 2002 ; 346 : 235 – 42 . https://doi.org/10.1056/NEJMoa011795 Crossref Search ADS PubMed 53. Chen Y , Wen H , Wu CI . Oncotarget 2017 ; 8 : 38642 – 9 . https://doi.org/10.18632/oncotarget.16957 PubMed 54. Wang Y , Zhang T , Kwiatkowski N et al. Cell 2015 ; 163 : 174 – 86 . https://doi.org/10.1016/j.cell.2015.08.063 Crossref Search ADS PubMed 55. Williams GC , Nesse RM . Q Rev Biol 1991 ; 66 : 1 – 22 . https://doi.org/10.1086/417048 Crossref Search ADS PubMed 56. Cairns J . Nature 1975 ; 255 : 197 – 200 . https://doi.org/10.1038/255197a0 Crossref Search ADS PubMed 57. Nowell PC . Science 1976 ; 194 : 23 – 8 . https://doi.org/10.1126/science.959840 Crossref Search ADS PubMed 58. Lewontin RC . Am J Hum Genet 1974 ; 26 : 400 – 11 . PubMed 59. Shan X , Fung JJ , Kosaka A et al. Elife 2017 ; doi: 10.7554/eLife.25306 . 60. Showalter MR , Hatakeyama J , Cajka T et al. Elife 2017 ; doi: 10.7554/eLife.26030 . 61. Vanden Heuvel , Maddox E , Maalouf SW et al. Elife 2018 ; doi: 10.7554/eLife.29747 . 62. Lewis LM , Edwards MC , Meyers ZR et al. Elife 2018 ; doi: 10.7554/eLife.30274 . © The Author(s) 2018. Published by Oxford University Press on behalf of China Science Publishing & Media Ltd. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Journal

National Science ReviewOxford University Press

Published: Sep 1, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off