Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

On the low reproducibility of cancer studies

On the low reproducibility of cancer studies Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 National Science Review 5: 619–624, 2018 CRITIQUE & DEBATE doi: 10.1093/nsr/nwy021 Advance access publication 2 February 2018 MOLECULAR BIOLOGY & GENETICS 1 2 1 1,3,∗ Haijun Wen , Hurng-Yi Wang , Xionglei He and Chung-I Wu distinction is made between studies that We found these readings of the repli- INTRODUCTION should be reproducible and those that are cation results to be overly cautious. In Previous reports have suggested that intrinsically irreproducible. Fourth, sug- a more conventional assessment (see close to 90% of cancer biology pub- gestions are made to cope with irrepro- Table 1 for a summary), four of the lications are irreproducible. The low ducibility issues. five original studies should be classi- number has recently been corroborated fied as ‘not reproducible’ and one as by five detailed replication studies in ‘uninterpretable’. eLife, which have been commented on We shall start with the two reports in Nature (Replication studies offer UPDATES ON REPRODUCIBILITY that the eLife editorial considers to be un- much more than technical details. INVESTIGATION: THE interpretable due to irreproducible con- Nature 2017;541:259–60). While the REPRODUCIBILITY PROJECT: trols. As the failure to reproduce the irreproducibility is often attributed to CANCER BIOLOGY control is no different from the fail- human factors, which are remediable, the ure to reproduce the experiments, we The Reproducibility Project: Cancer reason might be biological and the irre- reclassify the two cases as ‘not repro- Biology (RP: CB) was launched to producibility is intrinsic to such studies. ducible’. In Berger et al.[8], muta- address the issue rigorously [6]. RP: The low reproducibility, reflecting the tions in the PREX2 gene are found to diversity in the evolutionary pathways of CB has selected 30 cancer studies from be common in human melanomas (15 tumourigenesis, will likely significantly high-impact journals for reproducibility out of 107). Furthermore, the introduc- impact clinical strategies. investigation. Because failure regarding tion of a PREX2 mutation into human reproducibility is most likely due to tech- Reproducibility is the foundation of melanocytes was reported to substan- nical factors, ruling out these obvious experimental science. While there are tially reduce the tumour-free survival in reasons is the salient contribution of many factors afflicting different fields of xenograft mice. In the replication study, RP: CB. Prior to the actual replication inquiry to varying degrees [1], cancer bi- Horrigan et al.[9] (see also Davis [10]) is a phase of Registered Reports, which ology studies appear to stand out. In an cited recent studies that failed to sup- outline the replication designs. The earlier report, only 6 of 53 published find- port the prevalence of PREX2 mutations replication studies themselves only begin ings in cancer biology could be confirmed in human melanomas. In addition, Hor- after the designs have been reviewed and [2], a rate approaching an alarmingly low rigan et al. failed to corroborate the re- approved. Since both sides agree on the 10% of reproducibility. According to the duction in tumour-free survival caused principle of reproducibility, the chal- report, such a low rate is common in the by PREX2 mutation because the controls lenge is to assure that the materials and pharmaceutical industry. without the mutation die just as rapidly methods can be accurately replicated. The low rate is of particular con- (1 week), in contrast with the original Now, the first batch of reports has cern because these studies are generally report of 9-week median survival in the been released [7]. Reproducibility here published in ‘high-impact’ journals, thus control. means an ‘actionable and repeatable ther- having real consequences in both clin- In the other study, Willingham et al. apeutic strategy’ [2], as each of the ical practice and basic research. In this [11] suggest that the CD47 protein original studies has a therapeutic pro- critique, we first review recent updates is overexpressed on the membrane of posal. According to the eLife editorial [4], on the extent of reproducibility. Second, most cancer cells. Since CD47 signals two studies are ‘reproducible in impor- the reasons underlying low reproducibil- to macrophages to withhold attack, ity are explored. While concerns about tant parts’, one being ‘not reproducible’ blocking CD47 reduces the mass of low reproducibility are often expressed and the remaining two being ‘uninter- orthotopic breast tumours by ∼10 fold in terms of human factors [2–5], there pretable’. Another news report character- in immune competent mice that have izes the overall picture as ‘muddy’ [5]. are deeper biological reasons. Third, a The Author(s) 2018. Published by Oxford University Press on behalf of China Science Publishing & Media Ltd. All rights reserved. For permissions, plea se e-mail: journals.permissions@oup.com Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 620 Natl Sci Rev, 2018, Vol. 5, No. 5 CRITIQUE & DEBATE Table 1. Summary of the rst fi batch of RP: CB studies. Studies eLife editorial This study Comments Sirota et al.[18] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is Kandela et al.[19] weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al.[21] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is Aird et al.[22] reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al.[13] No No Co-administration of iRGD with chemo-agent enhances drug uptake by Mantis et al.[14] tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al.[8] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster Horrigan et al.[9] as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al.[11] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by Horrigan [12] blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Yes – reproducible; No – Not reproducible; UI – Uninterpretable. reproducible’, one [18] would not be been injected with MT1A2 mouse cells, CAUSES OF IRREPRODUCIBILITY considered reproducible under most cir- relative to the control. In the replication The five replications corroborate the ear- cumstances. In Sirota et al., the bioin- by Horrigan [12], the tumour sizes are lier report of reproducibility at a rate of formatic analysis of drug applications on curiously reversed between the CD47- 6out of 53 [2]. While previous reports cell lines led to the identification of cime- and IgG (control)-treated mice. Horri- have hinted at technical (or even ethical) tidine as an effective agent against lung gan noted that the tumour mass is highly lapses [2,4–6], the factors cited are com- adenocarcinoma cells. While the replica- variable, often with a 5-fold difference in mon across biological disciplines. Fur- tion by Kandela et al.[19] observed the the same setting. Such high variability is thermore, given the care invested in RP: same trend, the difference was not signifi- common is tumour evolution but is of- CB, the replication efforts should be quite cantly different from the control. The rea- ten treated as noise. It should be noted adequate. We therefore seek biological son, as noted by Dang [20], is that the ef- that Willingham et al. obtained similar explanations below. fect of cimetidine reported by Sirota et al. results by transplanting human tumour Whether or not any experimental is too weak to be considered biologically cells into immune-compromised mice. study can be replicated depends on the significant. Overall, the variance, in com- Horrigan also reported that the xenograft measurements being reproduced. In coin parison with the small difference in mean, experiments have been reproduced only tossing, seeing the same side five times makes it difficult to justify the conclusion in some follow-up studies. in a row will be reproducible no more that cimetidine is an effective new drug The one study that was deemed not than 10% of the time. In cancer biology, against lung cancer. to be reproducible was that by Sugahara reproducibility would mean that tumour The only reproducible study in the set et al.[13], who showed an increase in progression corresponds to highly con- of five studies is that of Delmore et al. drug permeability when tumours were strained courses, akin to tissue develop- [21]. In the original study, the molecule treated with the iRGD peptide. In their ment. However, if tumour progression (+)-JQ1 was reported to downregulate study, tumours grow from xenografted follows evolutionary trajectories, the out- MYC transcription and reduce the bur- prostate cancer cells. Mantis et al.[14] come may be highly variable. The course den of multiple myeloma tumours, result- could not reproduce the results but did of evolution is often a multi-step process ing in the improved survival of xenograft report some successful replication by requiring a suite of genetic changes, each mice. While the results were successfully other studies [15–17]. While the study of which is governed by stochastic fac- replicated by Aird et al. (with some vari- by Sugahara et al. was the only study tors including mutation emergence, ran- ations) [22], a negative control using whereby the experiments could not be dom drift, and divergent selective pres- an enantiomer (−)-JQ1, which does not reproduced, it is possible that either the sures. Since each step is contingent on the impact MYC transcription, showed the control or the experiment can be more previous steps taken, divergent outcomes same biological effect as ( +)-JQ1. Given variable. Thus, it seems curious to con- may result from even a small deviation in that the negative control also yielded the sider one type of irreproducibility to be an earlier stage. (unexpected) benefit, we suggest that the more serious than another. In his book ‘Wonderful Life’, S. J. original study should be considered unin- For the remaining two studies Gould [23] raised the issue of the terpretable. that were declared to be ‘essentially Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 CRITIQUE & DEBATE Wen et al. 621 reproducibility of evolution itself. He diverge, leading to substantial genetic one might still expect a high level wondered if the same evolutionary tra- diversity [34,35] and variable responses of convergence in mouse models jectory would be followed had ‘the tape to therapeutic treatments [36]. when most conditions are under con- of life’ been rewound (see also Conway The TCGA results have their paral- trol. Now, the RP: CB studies have Morris [24]). ‘Rewinding the tape of lel in natural populations. While geno- cast doubt on the predictability of life’ back to the time of the Cambrian typic convergence may be observed for evolution even in simple models. In the explosion is of course mere fantasy, but highly specialized traits such as echolo- five RP: CB reports, cells from cancer there are indeed evolutionary processes cation [37], it is nevertheless rare. For cell lines are transplanted into mice, as that are continually reiterated. The best adaptations that do not involve highly xenografts or autografts. In these stud- example may be the evolution of cancers specialized constructs, diverse molecular ies, experimental (E) and control (C) [25–28]. It is hence curious that the mechanisms may operate and genotypic samples are collected, and designated word ‘evolution’ does not appear in the convergence is not expected. For exam- (E1, C1) for the original studies and RP:CB registered/replication reports, ple, human populations living in the high (E2, C2) for the replications. Tumour editorials, and commentaries. altitudes of the Tibetan, Ethiopian, and growth in each mouse is the culmi- Reproducibility of evolution would Andean Altiplano plateaus have differ- nation of two evolutionary processes. be equivalent to convergent evolution in ent genetic mutations for hypoxic adap- First, the cell populations have been which a dominant pathway is repeatedly tation [38]. Recently, the search for evolving prior to transplantation [47]. taken. In convergent evolution, the dis- molecular convergence has expanded to Second, these cells subsequently evolve tinction between phenotype and geno- finding signals in the entire genome as xeno(auto)-grafts into tumours. While type is crucial. Phenotypic convergence [37,39–44]. Again, even with the aid of the second stage is widely discussed (see is common in natural populations. Sim- multiple genomes in the same environ- Wu et al.[26] for references), the first ilarly, morphological convergence is a ment, molecular convergence is rare and stage has been neglected even though basis on which pathologists define malig- the signals rarely exceed those of the cell lines do evolve continuously. This nancy. The central issue is genotypic con- background noise. first stage is reminiscent of the classic vergence: whether the genetic changes Reproducibility of cancer progression ‘Luria/Delbruck fluctuations’ [ 48]. underlying the phenotypes are them- and convergent evolution in organisms Results from E1, C1, E2, and C2 selves convergent. With constant refer- is nevertheless observable but the con- are all conditional distributions (Fig. 1). ences to somatic mutation, gene expres- ditions are stringent, i.e. when there is The final analysis of the RP: CB rests sion, and target therapy, cancer biology a dominant evolutionary pathway lead- on the comparisons between E1–C1 and publications apparently consider geno- ing to an end state. For example, when E2–C2. In those replication reports that typic convergence plausible. the organisms are genetically simple (e.g. fail to reproduce the original results, With this backdrop, the Cancer viruses), there would often be few ge- C1 and C2 have not evolved along the Genome Atlas (TCGA) project [29–31] netic solutions. Alternatively, if the selec- same path in two reports, and E1 and has attempted to identify genes that are tive pressure for specific genetic changes E2 have evolved divergently in one (see commonly mutated in tumours. The is strong (e.g. [39]), then convergence Table 1). In another report, E1–C1 is too results show that genetic convergence is a likely outcome. Wu et al. speculate small to be biologically or statistically sig- is much less frequent than had been that the selective pressure may be par- nificant. hoped for [29]. For example, across ticularly high in ‘liquid tumours’ where Tumour evolution in RP: CB may 12 cancer types, only two genes are cells with a proliferative advantage can be sketched by a simple genetic model mutated in more than 10% of cases: spread rapidly and widely [26]. Indeed, (Fig. 1) that frames evolutionary path- TP53 and PIK3CA; the former is an chronic myelogenous leukaemia remains ways as conditional probabilities. Each outlier in the human genome [32,33] one of the best examples of cancer con- stage of evolution is conditional on prior and the latter is a very large gene. The vergence at the genic level with the BCR- steps of evolution via segregation of ex- number of frequently mutated genes ABL translocation being a diagnostic fea- isting polymorphisms and the emergence (in >10% of cases) for a given type of ture [45,46]. In general, when we take of de novo mutations. A slight difference cancer is generally around 10 [29,30]. into account the multi-phenotypic and in the early stage may pave the way for a With such low genic convergence, two multi-genic nature of tumour evolution much greater divergence at a later time. cases of the same cancer type usually (see Hanahan and Weinberg on cancer In stage 2 in Fig. 1, the two replications have few mutated genes in common, hallmarks [33] and Kandoth [29] on can- overlap little in their trajectories and very or may share no mutated genes at all. cer driver genes), as well as the complex- different tumour phenotypes emerge as a These observations suggest that, from a ity of the mammalian genome, molecular result. A more realistic model than that of very similar starting point (two human convergence in cancer progression would Fig. 1 will likely yield even more diverse beings), the evolution of cancer usually likely be the exception rather than the patterns. We suggest that cancer biology takes different courses. Even cancer cells rule [26]. studies should develop explicit evolution- from the same starting point (within As TCGA reveals the low con- ary models, rather than assume simple the same person) would continue to vergence in real-life tumourigenesis, and reproducible outcomes. Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 622 Natl Sci Rev, 2018, Vol. 5, No. 5 CRITIQUE & DEBATE sharing the mutated EGFR gene. These Genotype of Overlap: Genotype of Population1 pop 1 vs. pop2 Population 2 cases may indeed show robust and more (First study) (highlight) (Replication) reproducible outcomes in responding to EGFR inhibitors [49]. However, such Stage 0 abcde f g h i j k l All possible ab c d efg h i j k l (Original partially defined genetic pathways are still pathways cell lines) |||||||||||| |||||||||||| overlapped quite diverse and irreproducible results, as in drug resistance, are not uncommon. Stage 1 abcd E fghijkl 4/5 abcde F ghijkl pathways Cell lines || | | | | | ||| overlapped evolution CONCLUSIONS No Stage 2 ab C dEfghijkl abcdeFg H ijkl overlapp Tumor It is curious that the RP: CB project | ||| | || ||| evolution [4,6], together with the earlier reports [2], are met with near total silence. Per- haps the prevailing view that low repro- ducibility is attributable to human fac- tors [3,5] does not call for intellectual discourse. This view, which has not been A F supported by any evidence, may have ob- scured the more fundamental reason be- hind irreproducibility. From an evolu- tionary perspective, low reproducibility is intrinsic to such studies because tumouri- genesis does not usually traverse the same evolutionary pathway. Reproducibility is nevertheless the Tumor phenotype (size, for example) central tenet of cancer biology, which assumes convergent pathways with rela- Figure 1. A model of pathway diversity in tumor evolution. Each step of the pathway is the tively well-defined genetic changes. Thus, realization from a probability distribution that is conditional on the previous steps taken. In this in basic research, genetic changes are model of 12 loci (a–l), one locus may change at each stage. The vertical bar below indicates the identified and therapies are then devel- evolvable locus, which may change from, say, e to E (further changes are allowed.). The locus oped to target these changes. The con- that actually changes is marked in red. It is assumed that each locus has positive tness fi epistasis tinual evolution of the underlying ge- with the two adjacent loci on each side (e.g. E interacts positively with C, D, F, and G). Stage 1 netic architecture means that mutations represents cell line evolution and stage 2 represents the evolution of these cells into tumours. The evolution of two populations (replications) is portrayed. The evolved genotypes in stage are ‘moving targets’, both between and 2 determine the tumour’s phenotypes (bottom of the figure), which show no overlap between within individuals. From this perspec- populations. If 10 samples are taken from each replication, as shown by the small arrows, the tive, target–gene therapy operates against two replications would appear to be totally irreproducible. evolutionary rules. Traits that do not evolve or that evolve slowly may be better targets. WHICH STUDIES ARE, OR ARE number, strength, and length of the evo- Indeed, the evolution of drug resistance NOT, REPRODUCIBLE? lutionary pathways. The TCGA data sug- continues to be a major impediment in gest that the number of genetic pathways The RP: CB reports give a glimpse of what targeted cancer therapy. The efficacies for tumourigenesis must be quite large. may or may not be reproducible [7]. If of the top three monoclonal antibody As the number of steps in each path- the phenotype being assayed does not drugs—bevacizumab, trastuzumab, way increases, the number of possible evolve in the course of the experiments, and rituximab—are instructive [50]. alternatives increases exponentially, ren- the reproducibility is generally high. For Bevacizumab and trastuzumab targeting dering many observations irreproducible. example, the treatment of cells with the VEGF and HER-2, respectively, and We should note that the sort of contin- chemical JQ1 leads to the reproducible have limited success [50,51]. In contrast, gent evolution depicted in Fig. 1 results downregulation of MYC transcription rituximab, which targets CD20 on the in variable outcomes that may be difficult [21,22]. Similarly, Horrigan [12]was cell surface of all pre-B cells, significantly to capture by increasing the sample size. able to reproduce the toxicity effect re- improves survival in patients with B-cell Facing the diversity of pathways, can- sulting in mild anaemia in normal mice, lymphoma [52]. The efficacy of ritux- cer biology studies often attempt to iso- whose tissues have not evolved. imab may be due to the fact that it does late cases that share part of their path- In the evolution of tumours, repro- not target a product of cellular evolution. ways; for examples, lung cancer cases ducibility would be a function of the Recent strategies that have targeted the Phenotypic distribution for each genotype Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 CRITIQUE & DEBATE Wen et al. 623 Priority Research Program of the Chinese Academy 23. Gould SJ. Wonderful Life: The Burgess Shale And basal transcription machinery [53,54] of Sciences (XDB13040300). The Nature Of History. New York: W.W. Norton, are compatible with this view favoring non-moving targets. 1989. Finally, many diseases are the 24. Conway Morris S. The Crucible Of Creation: The 1 2 1 Haijun Wen , Hurng-Yi Wang , Xionglei He and (by)products of evolution from the Burgess Shale And The Rise Of Animals. Oxford; 1,3, ∗ Chung-I Wu viewpoint of Darwinian medicine [55]. New York: Oxford University Press, 1998. State Key Laboratory of Biocontrol, College of Tumourigenesis is different as it is not 25. Merlo LMF, Pepper JW and Reid BJ et al. Nat Rev Ecology and Evolution, Sun Yat-Sen University, merely a product of evolution [56]; it is Cancer 2006; 6: 924–35. Guangzhou the process itself, or evolution in action 26. Wu CI, Wang HY and Ling S et al. Annu Rev Genet Graduate Institute of Clinical Medicine and [57]. Lewontin [58] pointed out that the 2016; 50: 347–69. Hepatitis Research Center, Taiwan University and essence of evolution is its variability. The 27. Chen H and He X. MolBiolEvol 2016; 33: 4–12. Hospital, Taipei diversity, rather than a standard type, is 28. Wang H-Y, Chen Y and Tong D et al. Natl Sci Rev Department of Ecology and Evolution, University the subject of interest. In cancer studies, 2018 ; 5: 15–7. of Chicago, Chicago the diversity in evolutionary trajectories 29. Kandoth C, McLellan MD and Vandin F et al. Na- Corresponding author. E-mails: is also clinically significant. H. J. Muller ture 2013; 502: 333–9. wuci@big.ac.cn; cw16@uchicago.edu remarked that biology has gone too long 30. Lawrence MS, Stojanov P and Polak P et al. Na- without evolutionary thinking at the ture 2013; 499: 214–8. 31. Vogelstein B, Papadopoulos N and Velculescu VE centennial of the publication of ‘On the et al. Science 2013; 339: 1546–58. Origin of Species’. Five decades later, REFERENCES 32. Weinberg R. The Biology of Cancer: New York: Muller’s remarks remain relevant to Garland Science, 2006. studies of cancers. 1. Baker M. Nature 2016; 533: 452–4. 33. Hanahan D and Weinberg RA. Cell 2011; 144: Finally, it should be noted that four 2. Begley CG and Ellis LM. Nature 2012; 483: 531–3. 646–74. additional reports were released by 3. Nature E. Nature 2017; 541: 259–60. 34. Sottoriva A, Kang H and Ma Z et al. Nat Genet eLife [59–62] at or after the time of this 4. Editorial. Elife 2017; 6: e23693. submission. The eLife editorial considers 2015; 47: 209–16. 5. Baker M and Dolgin E. Nature 2017; 541: 269–70. three to be reproducible in ‘important 35. Ling S, Hu Z and Yang Z et al. Proc Natl Acad Sci 6. Errington TM, Iorns E and Gunn W et al. Elife parts’ and one as reproducible in ‘some USA 2015; 112: E6496–505. 2014; 3: e04333. parts’. Among the four studies, all repro- 36. Turajlic S and Swanton C. Nat Rev Drug Discov 7. Nosek BA and Errington TM. Elife 2017; 6: ducible experiments were carried out on 2017; 16: 441–2. e23383. cell lines growing in dishes and the time 37. Li Y, Liu Z and Shi P et al. Curr Biol 2010; 20: R55– 8. Berger MF, Hodis E and Heffernan TP et al. Nature units for collecting data after treatment 2012; 485: 502–6. 6. were either minutes or hours. The only 9. Horrigan SK, Courville P and Sampey D et al. Elife 38. Udpa N, Ronen R and Zhou D et al. Genome Biol xenograft tumour model examined was 2017; 6: e21634. 2014; 15: R36. in Shan et al.[59], which tested the 10. Davis RJ. Elife 2017; 6: e22662. 39. Xu S, He Z and Guo Z et al. Mol Biol Evol 2017; efficacy of I-BET151, an inhibitor of 11. Willingham SB, Volkmer JP and Gentles AJ et al. 34: 1008–15. the BET bromodomain, as a treatment Proc Natl Acad Sci USA 2012; 109: 6662–7. 40. Parker J, Tsagkogeorga G and Cotton JA et al. Na- for mixed-lineage leukaemia (MLL)- 12. Horrigan SK. Elife 2017; 6: e18173. ture 2013; 502: 228–31. fusion leukaemia. The experiment took 13. Sugahara KN, Teesalu T and Karmali PP et al. Sci- 41. Foote AD, Liu Y and Thomas GW et al. Nat Genet 2 months, thus allowing for plenty of ence 2010; 328: 1031–5. 2015; 47: 272–5. 42. Thomas GWC and Hahn MW. Mol Biol Evol 2015; mutation and selection events. Not 14. Mantis C, Kandela I and Aird F et al. Elife 2017; 6: 32: 1232–6. unexpectedly, the replication group e17584. 43. Xu SH, He ZW and Zhang Z et al. Natl Sci Rev failed to reproduce the original finding of 15. Peng ZH and Kopecek J. J Am Chem Soc 2015; 2017; 4: 721–34. increased survival in I-BET151-treated 137: 6726–9. 44. Zou ZT and Zhang JZ. MolBiolEvol 2015; 32: mice. The timescale-dependent repro- 16. Deng C, Zhang Q and Fu Y et al. ACS Appl Mater 1237–41. ducibility of cancer research strengthens Interfaces 2017; 9: 1280–92. 45. Salesse S and Verfaillie CM. Oncogene 2002; 21: the evolutionary arguments of our 17. Schmithals C, Koberle V and Korkusuz H et al. report. 8547–59. Cancer Res 2015; 75: 3147–54. 46. Ren R. Nat Rev Cancer 2005; 5: 172–83. 18. Sirota M, Dudley JT and Kim J et al. Sci Transl 47. Nguyen LV, Cox CL and Eirew P et al. Nat Comms Med 2011; 3: 96ra77. 2014; 5: 5871. 19. Kandela I, Aird F and Reproducibility Project: ACKNOWLEDGEMENTS 48. Luria SE and Delbruck M. Genetics 1943; 28: 491– Cancer, B . Elife 2017; 6: e17044. This work was supported by the National Nat- 20. Dang CV. Elife 2017; 6: e22661. ural Science Foundation of China (31730046, 21. Delmore JE, Issa GC and Lemieux ME et al. Cell 49. Casas-Selves M, Kim J and Zhang Z et al. Cancer 91731302, 91731301), the National Basic 2011; 146: 904–17. Res 2012; 72: 4154–64. Research Program (973 Program) of China 22. Aird F, Kandela I and Mantis C et al. Elife 2017; 6: 50. Aggarwal S. Nat Rev Drug Discov 2010; 9: (2014CB542006 and 2014CB542005), the 985 Project (33000-18821105), and the Strategic e21253. 427–8. Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 624 Natl Sci Rev, 2018, Vol. 5, No. 5 CRITIQUE & DEBATE 51. Pohlmann PR, Mayer IA and Mernaugh R. Clin 55. Williams GC and Nesse RM. Q Rev Biol 1991; 66: 60. Showalter MR, Hatakeyama J and Cajka Cancer Res 2009; 15: 7479–91. 1–22. T et al. Elife 2017; doi: 10.7554/eLife. 52. Coiffier B, Lepage E and Bri ere ` J et al. N Engl J 56. Cairns J. Nature 1975; 255: 197–200. 26030. Med 2002; 346: 235–42. 57. Nowell PC. Science 1976; 194: 23–8. 61. Vanden Heuvel, Maddox E and Maalouf 53. Chen Y, Wen H and Wu CI. Oncotarget 2017; 8: 58. Lewontin RC. Am J Hum Genet 1974; 26: SW et al. Elife 2018; doi: 10.7554/eLife. 38642–9. 400–11. 29747. 54. Wang Y, Zhang T and Kwiatkowski N et al. Cell 59. Shan X, Fung JJ and Kosaka A et al. Elife 2017; 62. Lewis LM, Edwards MC and Meyers ZR et al. Elife 2015; 163: 174–86. doi: 10.7554/eLife.25306. 2018; doi: 10.7554/eLife.30274. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png National Science Review Oxford University Press

On the low reproducibility of cancer studies

Download PDF
6 pages

Loading next page...
 
/lp/ou_press/on-the-low-reproducibility-of-cancer-studies-l5v0jOBU97
Publisher
Oxford University Press
Copyright
Copyright © 2022 China Science Publishing & Media Ltd. (Science Press)
ISSN
2095-5138
eISSN
2053-714X
DOI
10.1093/nsr/nwy021
Publisher site
See Article on Publisher Site

Abstract

Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 National Science Review 5: 619–624, 2018 CRITIQUE & DEBATE doi: 10.1093/nsr/nwy021 Advance access publication 2 February 2018 MOLECULAR BIOLOGY & GENETICS 1 2 1 1,3,∗ Haijun Wen , Hurng-Yi Wang , Xionglei He and Chung-I Wu distinction is made between studies that We found these readings of the repli- INTRODUCTION should be reproducible and those that are cation results to be overly cautious. In Previous reports have suggested that intrinsically irreproducible. Fourth, sug- a more conventional assessment (see close to 90% of cancer biology pub- gestions are made to cope with irrepro- Table 1 for a summary), four of the lications are irreproducible. The low ducibility issues. five original studies should be classi- number has recently been corroborated fied as ‘not reproducible’ and one as by five detailed replication studies in ‘uninterpretable’. eLife, which have been commented on We shall start with the two reports in Nature (Replication studies offer UPDATES ON REPRODUCIBILITY that the eLife editorial considers to be un- much more than technical details. INVESTIGATION: THE interpretable due to irreproducible con- Nature 2017;541:259–60). While the REPRODUCIBILITY PROJECT: trols. As the failure to reproduce the irreproducibility is often attributed to CANCER BIOLOGY control is no different from the fail- human factors, which are remediable, the ure to reproduce the experiments, we The Reproducibility Project: Cancer reason might be biological and the irre- reclassify the two cases as ‘not repro- Biology (RP: CB) was launched to producibility is intrinsic to such studies. ducible’. In Berger et al.[8], muta- address the issue rigorously [6]. RP: The low reproducibility, reflecting the tions in the PREX2 gene are found to diversity in the evolutionary pathways of CB has selected 30 cancer studies from be common in human melanomas (15 tumourigenesis, will likely significantly high-impact journals for reproducibility out of 107). Furthermore, the introduc- impact clinical strategies. investigation. Because failure regarding tion of a PREX2 mutation into human reproducibility is most likely due to tech- Reproducibility is the foundation of melanocytes was reported to substan- nical factors, ruling out these obvious experimental science. While there are tially reduce the tumour-free survival in reasons is the salient contribution of many factors afflicting different fields of xenograft mice. In the replication study, RP: CB. Prior to the actual replication inquiry to varying degrees [1], cancer bi- Horrigan et al.[9] (see also Davis [10]) is a phase of Registered Reports, which ology studies appear to stand out. In an cited recent studies that failed to sup- outline the replication designs. The earlier report, only 6 of 53 published find- port the prevalence of PREX2 mutations replication studies themselves only begin ings in cancer biology could be confirmed in human melanomas. In addition, Hor- after the designs have been reviewed and [2], a rate approaching an alarmingly low rigan et al. failed to corroborate the re- approved. Since both sides agree on the 10% of reproducibility. According to the duction in tumour-free survival caused principle of reproducibility, the chal- report, such a low rate is common in the by PREX2 mutation because the controls lenge is to assure that the materials and pharmaceutical industry. without the mutation die just as rapidly methods can be accurately replicated. The low rate is of particular con- (1 week), in contrast with the original Now, the first batch of reports has cern because these studies are generally report of 9-week median survival in the been released [7]. Reproducibility here published in ‘high-impact’ journals, thus control. means an ‘actionable and repeatable ther- having real consequences in both clin- In the other study, Willingham et al. apeutic strategy’ [2], as each of the ical practice and basic research. In this [11] suggest that the CD47 protein original studies has a therapeutic pro- critique, we first review recent updates is overexpressed on the membrane of posal. According to the eLife editorial [4], on the extent of reproducibility. Second, most cancer cells. Since CD47 signals two studies are ‘reproducible in impor- the reasons underlying low reproducibil- to macrophages to withhold attack, ity are explored. While concerns about tant parts’, one being ‘not reproducible’ blocking CD47 reduces the mass of low reproducibility are often expressed and the remaining two being ‘uninter- orthotopic breast tumours by ∼10 fold in terms of human factors [2–5], there pretable’. Another news report character- in immune competent mice that have izes the overall picture as ‘muddy’ [5]. are deeper biological reasons. Third, a The Author(s) 2018. Published by Oxford University Press on behalf of China Science Publishing & Media Ltd. All rights reserved. For permissions, plea se e-mail: journals.permissions@oup.com Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 620 Natl Sci Rev, 2018, Vol. 5, No. 5 CRITIQUE & DEBATE Table 1. Summary of the rst fi batch of RP: CB studies. Studies eLife editorial This study Comments Sirota et al.[18] Yes No Cimetidine slowing down lung adenocarcinoma growth. Reported effect is Kandela et al.[19] weak in the original report and the weak effect is found to be insignificant in the replication. Delmore et al.[21] Yes UI JQ1 binding MYC and slowing down myeloma growth. The experiment is Aird et al.[22] reproducible but a negative control using (–)JQ1 has the same effect on tumour growth. Sugahara et al.[13] No No Co-administration of iRGD with chemo-agent enhances drug uptake by Mantis et al.[14] tumour cells. Neither drug uptake nor tumour growth is reproduced. Berger et al.[8] UI No Transplanted melanoma cells expressing a mutated PREX2 gene grow faster Horrigan et al.[9] as tumours, speeding up death. In replication, the control cells have the same lethal effect. Willingham et al.[11] UI No Anti-CD47 antibody promotes growth of mouse breast cancer cells by Horrigan [12] blocking phagocytosis, vis-a-vis the IgG control. The opposite effect is observed in the replication Yes – reproducible; No – Not reproducible; UI – Uninterpretable. reproducible’, one [18] would not be been injected with MT1A2 mouse cells, CAUSES OF IRREPRODUCIBILITY considered reproducible under most cir- relative to the control. In the replication The five replications corroborate the ear- cumstances. In Sirota et al., the bioin- by Horrigan [12], the tumour sizes are lier report of reproducibility at a rate of formatic analysis of drug applications on curiously reversed between the CD47- 6out of 53 [2]. While previous reports cell lines led to the identification of cime- and IgG (control)-treated mice. Horri- have hinted at technical (or even ethical) tidine as an effective agent against lung gan noted that the tumour mass is highly lapses [2,4–6], the factors cited are com- adenocarcinoma cells. While the replica- variable, often with a 5-fold difference in mon across biological disciplines. Fur- tion by Kandela et al.[19] observed the the same setting. Such high variability is thermore, given the care invested in RP: same trend, the difference was not signifi- common is tumour evolution but is of- CB, the replication efforts should be quite cantly different from the control. The rea- ten treated as noise. It should be noted adequate. We therefore seek biological son, as noted by Dang [20], is that the ef- that Willingham et al. obtained similar explanations below. fect of cimetidine reported by Sirota et al. results by transplanting human tumour Whether or not any experimental is too weak to be considered biologically cells into immune-compromised mice. study can be replicated depends on the significant. Overall, the variance, in com- Horrigan also reported that the xenograft measurements being reproduced. In coin parison with the small difference in mean, experiments have been reproduced only tossing, seeing the same side five times makes it difficult to justify the conclusion in some follow-up studies. in a row will be reproducible no more that cimetidine is an effective new drug The one study that was deemed not than 10% of the time. In cancer biology, against lung cancer. to be reproducible was that by Sugahara reproducibility would mean that tumour The only reproducible study in the set et al.[13], who showed an increase in progression corresponds to highly con- of five studies is that of Delmore et al. drug permeability when tumours were strained courses, akin to tissue develop- [21]. In the original study, the molecule treated with the iRGD peptide. In their ment. However, if tumour progression (+)-JQ1 was reported to downregulate study, tumours grow from xenografted follows evolutionary trajectories, the out- MYC transcription and reduce the bur- prostate cancer cells. Mantis et al.[14] come may be highly variable. The course den of multiple myeloma tumours, result- could not reproduce the results but did of evolution is often a multi-step process ing in the improved survival of xenograft report some successful replication by requiring a suite of genetic changes, each mice. While the results were successfully other studies [15–17]. While the study of which is governed by stochastic fac- replicated by Aird et al. (with some vari- by Sugahara et al. was the only study tors including mutation emergence, ran- ations) [22], a negative control using whereby the experiments could not be dom drift, and divergent selective pres- an enantiomer (−)-JQ1, which does not reproduced, it is possible that either the sures. Since each step is contingent on the impact MYC transcription, showed the control or the experiment can be more previous steps taken, divergent outcomes same biological effect as ( +)-JQ1. Given variable. Thus, it seems curious to con- may result from even a small deviation in that the negative control also yielded the sider one type of irreproducibility to be an earlier stage. (unexpected) benefit, we suggest that the more serious than another. In his book ‘Wonderful Life’, S. J. original study should be considered unin- For the remaining two studies Gould [23] raised the issue of the terpretable. that were declared to be ‘essentially Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 CRITIQUE & DEBATE Wen et al. 621 reproducibility of evolution itself. He diverge, leading to substantial genetic one might still expect a high level wondered if the same evolutionary tra- diversity [34,35] and variable responses of convergence in mouse models jectory would be followed had ‘the tape to therapeutic treatments [36]. when most conditions are under con- of life’ been rewound (see also Conway The TCGA results have their paral- trol. Now, the RP: CB studies have Morris [24]). ‘Rewinding the tape of lel in natural populations. While geno- cast doubt on the predictability of life’ back to the time of the Cambrian typic convergence may be observed for evolution even in simple models. In the explosion is of course mere fantasy, but highly specialized traits such as echolo- five RP: CB reports, cells from cancer there are indeed evolutionary processes cation [37], it is nevertheless rare. For cell lines are transplanted into mice, as that are continually reiterated. The best adaptations that do not involve highly xenografts or autografts. In these stud- example may be the evolution of cancers specialized constructs, diverse molecular ies, experimental (E) and control (C) [25–28]. It is hence curious that the mechanisms may operate and genotypic samples are collected, and designated word ‘evolution’ does not appear in the convergence is not expected. For exam- (E1, C1) for the original studies and RP:CB registered/replication reports, ple, human populations living in the high (E2, C2) for the replications. Tumour editorials, and commentaries. altitudes of the Tibetan, Ethiopian, and growth in each mouse is the culmi- Reproducibility of evolution would Andean Altiplano plateaus have differ- nation of two evolutionary processes. be equivalent to convergent evolution in ent genetic mutations for hypoxic adap- First, the cell populations have been which a dominant pathway is repeatedly tation [38]. Recently, the search for evolving prior to transplantation [47]. taken. In convergent evolution, the dis- molecular convergence has expanded to Second, these cells subsequently evolve tinction between phenotype and geno- finding signals in the entire genome as xeno(auto)-grafts into tumours. While type is crucial. Phenotypic convergence [37,39–44]. Again, even with the aid of the second stage is widely discussed (see is common in natural populations. Sim- multiple genomes in the same environ- Wu et al.[26] for references), the first ilarly, morphological convergence is a ment, molecular convergence is rare and stage has been neglected even though basis on which pathologists define malig- the signals rarely exceed those of the cell lines do evolve continuously. This nancy. The central issue is genotypic con- background noise. first stage is reminiscent of the classic vergence: whether the genetic changes Reproducibility of cancer progression ‘Luria/Delbruck fluctuations’ [ 48]. underlying the phenotypes are them- and convergent evolution in organisms Results from E1, C1, E2, and C2 selves convergent. With constant refer- is nevertheless observable but the con- are all conditional distributions (Fig. 1). ences to somatic mutation, gene expres- ditions are stringent, i.e. when there is The final analysis of the RP: CB rests sion, and target therapy, cancer biology a dominant evolutionary pathway lead- on the comparisons between E1–C1 and publications apparently consider geno- ing to an end state. For example, when E2–C2. In those replication reports that typic convergence plausible. the organisms are genetically simple (e.g. fail to reproduce the original results, With this backdrop, the Cancer viruses), there would often be few ge- C1 and C2 have not evolved along the Genome Atlas (TCGA) project [29–31] netic solutions. Alternatively, if the selec- same path in two reports, and E1 and has attempted to identify genes that are tive pressure for specific genetic changes E2 have evolved divergently in one (see commonly mutated in tumours. The is strong (e.g. [39]), then convergence Table 1). In another report, E1–C1 is too results show that genetic convergence is a likely outcome. Wu et al. speculate small to be biologically or statistically sig- is much less frequent than had been that the selective pressure may be par- nificant. hoped for [29]. For example, across ticularly high in ‘liquid tumours’ where Tumour evolution in RP: CB may 12 cancer types, only two genes are cells with a proliferative advantage can be sketched by a simple genetic model mutated in more than 10% of cases: spread rapidly and widely [26]. Indeed, (Fig. 1) that frames evolutionary path- TP53 and PIK3CA; the former is an chronic myelogenous leukaemia remains ways as conditional probabilities. Each outlier in the human genome [32,33] one of the best examples of cancer con- stage of evolution is conditional on prior and the latter is a very large gene. The vergence at the genic level with the BCR- steps of evolution via segregation of ex- number of frequently mutated genes ABL translocation being a diagnostic fea- isting polymorphisms and the emergence (in >10% of cases) for a given type of ture [45,46]. In general, when we take of de novo mutations. A slight difference cancer is generally around 10 [29,30]. into account the multi-phenotypic and in the early stage may pave the way for a With such low genic convergence, two multi-genic nature of tumour evolution much greater divergence at a later time. cases of the same cancer type usually (see Hanahan and Weinberg on cancer In stage 2 in Fig. 1, the two replications have few mutated genes in common, hallmarks [33] and Kandoth [29] on can- overlap little in their trajectories and very or may share no mutated genes at all. cer driver genes), as well as the complex- different tumour phenotypes emerge as a These observations suggest that, from a ity of the mammalian genome, molecular result. A more realistic model than that of very similar starting point (two human convergence in cancer progression would Fig. 1 will likely yield even more diverse beings), the evolution of cancer usually likely be the exception rather than the patterns. We suggest that cancer biology takes different courses. Even cancer cells rule [26]. studies should develop explicit evolution- from the same starting point (within As TCGA reveals the low con- ary models, rather than assume simple the same person) would continue to vergence in real-life tumourigenesis, and reproducible outcomes. Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 622 Natl Sci Rev, 2018, Vol. 5, No. 5 CRITIQUE & DEBATE sharing the mutated EGFR gene. These Genotype of Overlap: Genotype of Population1 pop 1 vs. pop2 Population 2 cases may indeed show robust and more (First study) (highlight) (Replication) reproducible outcomes in responding to EGFR inhibitors [49]. However, such Stage 0 abcde f g h i j k l All possible ab c d efg h i j k l (Original partially defined genetic pathways are still pathways cell lines) |||||||||||| |||||||||||| overlapped quite diverse and irreproducible results, as in drug resistance, are not uncommon. Stage 1 abcd E fghijkl 4/5 abcde F ghijkl pathways Cell lines || | | | | | ||| overlapped evolution CONCLUSIONS No Stage 2 ab C dEfghijkl abcdeFg H ijkl overlapp Tumor It is curious that the RP: CB project | ||| | || ||| evolution [4,6], together with the earlier reports [2], are met with near total silence. Per- haps the prevailing view that low repro- ducibility is attributable to human fac- tors [3,5] does not call for intellectual discourse. This view, which has not been A F supported by any evidence, may have ob- scured the more fundamental reason be- hind irreproducibility. From an evolu- tionary perspective, low reproducibility is intrinsic to such studies because tumouri- genesis does not usually traverse the same evolutionary pathway. Reproducibility is nevertheless the Tumor phenotype (size, for example) central tenet of cancer biology, which assumes convergent pathways with rela- Figure 1. A model of pathway diversity in tumor evolution. Each step of the pathway is the tively well-defined genetic changes. Thus, realization from a probability distribution that is conditional on the previous steps taken. In this in basic research, genetic changes are model of 12 loci (a–l), one locus may change at each stage. The vertical bar below indicates the identified and therapies are then devel- evolvable locus, which may change from, say, e to E (further changes are allowed.). The locus oped to target these changes. The con- that actually changes is marked in red. It is assumed that each locus has positive tness fi epistasis tinual evolution of the underlying ge- with the two adjacent loci on each side (e.g. E interacts positively with C, D, F, and G). Stage 1 netic architecture means that mutations represents cell line evolution and stage 2 represents the evolution of these cells into tumours. The evolution of two populations (replications) is portrayed. The evolved genotypes in stage are ‘moving targets’, both between and 2 determine the tumour’s phenotypes (bottom of the figure), which show no overlap between within individuals. From this perspec- populations. If 10 samples are taken from each replication, as shown by the small arrows, the tive, target–gene therapy operates against two replications would appear to be totally irreproducible. evolutionary rules. Traits that do not evolve or that evolve slowly may be better targets. WHICH STUDIES ARE, OR ARE number, strength, and length of the evo- Indeed, the evolution of drug resistance NOT, REPRODUCIBLE? lutionary pathways. The TCGA data sug- continues to be a major impediment in gest that the number of genetic pathways The RP: CB reports give a glimpse of what targeted cancer therapy. The efficacies for tumourigenesis must be quite large. may or may not be reproducible [7]. If of the top three monoclonal antibody As the number of steps in each path- the phenotype being assayed does not drugs—bevacizumab, trastuzumab, way increases, the number of possible evolve in the course of the experiments, and rituximab—are instructive [50]. alternatives increases exponentially, ren- the reproducibility is generally high. For Bevacizumab and trastuzumab targeting dering many observations irreproducible. example, the treatment of cells with the VEGF and HER-2, respectively, and We should note that the sort of contin- chemical JQ1 leads to the reproducible have limited success [50,51]. In contrast, gent evolution depicted in Fig. 1 results downregulation of MYC transcription rituximab, which targets CD20 on the in variable outcomes that may be difficult [21,22]. Similarly, Horrigan [12]was cell surface of all pre-B cells, significantly to capture by increasing the sample size. able to reproduce the toxicity effect re- improves survival in patients with B-cell Facing the diversity of pathways, can- sulting in mild anaemia in normal mice, lymphoma [52]. The efficacy of ritux- cer biology studies often attempt to iso- whose tissues have not evolved. imab may be due to the fact that it does late cases that share part of their path- In the evolution of tumours, repro- not target a product of cellular evolution. ways; for examples, lung cancer cases ducibility would be a function of the Recent strategies that have targeted the Phenotypic distribution for each genotype Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 CRITIQUE & DEBATE Wen et al. 623 Priority Research Program of the Chinese Academy 23. Gould SJ. Wonderful Life: The Burgess Shale And basal transcription machinery [53,54] of Sciences (XDB13040300). The Nature Of History. New York: W.W. Norton, are compatible with this view favoring non-moving targets. 1989. Finally, many diseases are the 24. Conway Morris S. The Crucible Of Creation: The 1 2 1 Haijun Wen , Hurng-Yi Wang , Xionglei He and (by)products of evolution from the Burgess Shale And The Rise Of Animals. Oxford; 1,3, ∗ Chung-I Wu viewpoint of Darwinian medicine [55]. New York: Oxford University Press, 1998. State Key Laboratory of Biocontrol, College of Tumourigenesis is different as it is not 25. Merlo LMF, Pepper JW and Reid BJ et al. Nat Rev Ecology and Evolution, Sun Yat-Sen University, merely a product of evolution [56]; it is Cancer 2006; 6: 924–35. Guangzhou the process itself, or evolution in action 26. Wu CI, Wang HY and Ling S et al. Annu Rev Genet Graduate Institute of Clinical Medicine and [57]. Lewontin [58] pointed out that the 2016; 50: 347–69. Hepatitis Research Center, Taiwan University and essence of evolution is its variability. The 27. Chen H and He X. MolBiolEvol 2016; 33: 4–12. Hospital, Taipei diversity, rather than a standard type, is 28. Wang H-Y, Chen Y and Tong D et al. Natl Sci Rev Department of Ecology and Evolution, University the subject of interest. In cancer studies, 2018 ; 5: 15–7. of Chicago, Chicago the diversity in evolutionary trajectories 29. Kandoth C, McLellan MD and Vandin F et al. Na- Corresponding author. E-mails: is also clinically significant. H. J. Muller ture 2013; 502: 333–9. wuci@big.ac.cn; cw16@uchicago.edu remarked that biology has gone too long 30. Lawrence MS, Stojanov P and Polak P et al. Na- without evolutionary thinking at the ture 2013; 499: 214–8. 31. Vogelstein B, Papadopoulos N and Velculescu VE centennial of the publication of ‘On the et al. Science 2013; 339: 1546–58. Origin of Species’. Five decades later, REFERENCES 32. Weinberg R. The Biology of Cancer: New York: Muller’s remarks remain relevant to Garland Science, 2006. studies of cancers. 1. Baker M. Nature 2016; 533: 452–4. 33. Hanahan D and Weinberg RA. Cell 2011; 144: Finally, it should be noted that four 2. Begley CG and Ellis LM. Nature 2012; 483: 531–3. 646–74. additional reports were released by 3. Nature E. Nature 2017; 541: 259–60. 34. Sottoriva A, Kang H and Ma Z et al. Nat Genet eLife [59–62] at or after the time of this 4. Editorial. Elife 2017; 6: e23693. submission. The eLife editorial considers 2015; 47: 209–16. 5. Baker M and Dolgin E. Nature 2017; 541: 269–70. three to be reproducible in ‘important 35. Ling S, Hu Z and Yang Z et al. Proc Natl Acad Sci 6. Errington TM, Iorns E and Gunn W et al. Elife parts’ and one as reproducible in ‘some USA 2015; 112: E6496–505. 2014; 3: e04333. parts’. Among the four studies, all repro- 36. Turajlic S and Swanton C. Nat Rev Drug Discov 7. Nosek BA and Errington TM. Elife 2017; 6: ducible experiments were carried out on 2017; 16: 441–2. e23383. cell lines growing in dishes and the time 37. Li Y, Liu Z and Shi P et al. Curr Biol 2010; 20: R55– 8. Berger MF, Hodis E and Heffernan TP et al. Nature units for collecting data after treatment 2012; 485: 502–6. 6. were either minutes or hours. The only 9. Horrigan SK, Courville P and Sampey D et al. Elife 38. Udpa N, Ronen R and Zhou D et al. Genome Biol xenograft tumour model examined was 2017; 6: e21634. 2014; 15: R36. in Shan et al.[59], which tested the 10. Davis RJ. Elife 2017; 6: e22662. 39. Xu S, He Z and Guo Z et al. Mol Biol Evol 2017; efficacy of I-BET151, an inhibitor of 11. Willingham SB, Volkmer JP and Gentles AJ et al. 34: 1008–15. the BET bromodomain, as a treatment Proc Natl Acad Sci USA 2012; 109: 6662–7. 40. Parker J, Tsagkogeorga G and Cotton JA et al. Na- for mixed-lineage leukaemia (MLL)- 12. Horrigan SK. Elife 2017; 6: e18173. ture 2013; 502: 228–31. fusion leukaemia. The experiment took 13. Sugahara KN, Teesalu T and Karmali PP et al. Sci- 41. Foote AD, Liu Y and Thomas GW et al. Nat Genet 2 months, thus allowing for plenty of ence 2010; 328: 1031–5. 2015; 47: 272–5. 42. Thomas GWC and Hahn MW. Mol Biol Evol 2015; mutation and selection events. Not 14. Mantis C, Kandela I and Aird F et al. Elife 2017; 6: 32: 1232–6. unexpectedly, the replication group e17584. 43. Xu SH, He ZW and Zhang Z et al. Natl Sci Rev failed to reproduce the original finding of 15. Peng ZH and Kopecek J. J Am Chem Soc 2015; 2017; 4: 721–34. increased survival in I-BET151-treated 137: 6726–9. 44. Zou ZT and Zhang JZ. MolBiolEvol 2015; 32: mice. The timescale-dependent repro- 16. Deng C, Zhang Q and Fu Y et al. ACS Appl Mater 1237–41. ducibility of cancer research strengthens Interfaces 2017; 9: 1280–92. 45. Salesse S and Verfaillie CM. Oncogene 2002; 21: the evolutionary arguments of our 17. Schmithals C, Koberle V and Korkusuz H et al. report. 8547–59. Cancer Res 2015; 75: 3147–54. 46. Ren R. Nat Rev Cancer 2005; 5: 172–83. 18. Sirota M, Dudley JT and Kim J et al. Sci Transl 47. Nguyen LV, Cox CL and Eirew P et al. Nat Comms Med 2011; 3: 96ra77. 2014; 5: 5871. 19. Kandela I, Aird F and Reproducibility Project: ACKNOWLEDGEMENTS 48. Luria SE and Delbruck M. Genetics 1943; 28: 491– Cancer, B . Elife 2017; 6: e17044. This work was supported by the National Nat- 20. Dang CV. Elife 2017; 6: e22661. ural Science Foundation of China (31730046, 21. Delmore JE, Issa GC and Lemieux ME et al. Cell 49. Casas-Selves M, Kim J and Zhang Z et al. Cancer 91731302, 91731301), the National Basic 2011; 146: 904–17. Res 2012; 72: 4154–64. Research Program (973 Program) of China 22. Aird F, Kandela I and Mantis C et al. Elife 2017; 6: 50. Aggarwal S. Nat Rev Drug Discov 2010; 9: (2014CB542006 and 2014CB542005), the 985 Project (33000-18821105), and the Strategic e21253. 427–8. Downloaded from https://academic.oup.com/nsr/article/5/5/619/4835582 by DeepDyve user on 12 July 2022 624 Natl Sci Rev, 2018, Vol. 5, No. 5 CRITIQUE & DEBATE 51. Pohlmann PR, Mayer IA and Mernaugh R. Clin 55. Williams GC and Nesse RM. Q Rev Biol 1991; 66: 60. Showalter MR, Hatakeyama J and Cajka Cancer Res 2009; 15: 7479–91. 1–22. T et al. Elife 2017; doi: 10.7554/eLife. 52. Coiffier B, Lepage E and Bri ere ` J et al. N Engl J 56. Cairns J. Nature 1975; 255: 197–200. 26030. Med 2002; 346: 235–42. 57. Nowell PC. Science 1976; 194: 23–8. 61. Vanden Heuvel, Maddox E and Maalouf 53. Chen Y, Wen H and Wu CI. Oncotarget 2017; 8: 58. Lewontin RC. Am J Hum Genet 1974; 26: SW et al. Elife 2018; doi: 10.7554/eLife. 38642–9. 400–11. 29747. 54. Wang Y, Zhang T and Kwiatkowski N et al. Cell 59. Shan X, Fung JJ and Kosaka A et al. Elife 2017; 62. Lewis LM, Edwards MC and Meyers ZR et al. Elife 2015; 163: 174–86. doi: 10.7554/eLife.25306. 2018; doi: 10.7554/eLife.30274.

Journal

National Science Review – Oxford University Press

Published: Sep 1, 2018

References

  • Wonderful Life: The Burgess Shale And The Nature Of History
  • The Crucible Of Creation: The Burgess Shale And The Rise Of Animals
  • The Biology of Cancer