The polymerase chain reaction (PCR) is the backbone of contemporary DNA/RNA analysis, ideally enabling detection of one or just a few target molecules. However, when analysing food or forensic samples the analytical procedure is often challenged by low amounts of poor quality template molecules and complex matrices. Applying optimised and validated methods in all steps of the analysis workﬂow, i.e. sampling, sample treatment, DNA/RNA extraction and PCR (including reverse transcription for RNA analysis), is thus necessary to ensure the reliability of analysis. In this paper, we describe how in-house validation can be performed for the different modules of the diagnostic PCR process, providing practical examples as tools for laboratories in their planning of validation studies. The focus is analysis of heterogeneous samples with interfering matrices, with relevance in food testing, forensic DNA analysis, bioterrorism preparedness and veterinary medicine. Our objective is to enable rational in-house validation for reliable and swift quality assurance when results are urgent, for example in the event of a crisis such as a foodborne outbreak or a crime requiring the analysis of a large number of diverse samples. To that end, we explain the performance characteristics associated with method validation from a PCR and biological sample matrix perspective and suggest which characteristics to investigate depending on the type of method to be validated. Also, we include a modular approach to validation within the PCR workﬂow, aiming at efﬁcient validation and a ﬂexible use of methods. Keywords Digital PCR Forensic DNA analysis ISO/IEC 17025 Pre-PCR processing qPCR Quality assurance Validation Introduction The polymerase chain reaction (PCR) is widely applied for the analysis of DNA/RNA from humans and Presented at the Workshop Uncertainty in Qualitative and microorganisms. Hence, PCR techniques are crucial for Quantitative Analysis, Nicosia, Cyprus, 29–30 May 2017. several sectors and applications, such as the investigation & Johannes Hedman of crime and to ensure human safety through analysis of firstname.lastname@example.org foods or suspected bioterrorism samples. In forensics as well as food safety, a false-positive or false-negative Swedish National Forensic Centre, Linkoping, Sweden result can have dire consequences. Optimised and vali- Applied Microbiology, Department of Chemistry, Lund dated analysis workﬂows are necessary to minimise the University, Lund, Sweden risk for such events. One mutual challenge in the analysis Science Division, Biology Department, National Food of food, feed, and forensic samples is the wide variety of Agency, Uppsala, Sweden possible sample types and the heterogeneous nature of the CBRN Defence and Security, Swedish Defence Research samples. Sample matrices can have a negative impact on Agency, Umea, Sweden the analysis by bringing PCR-inhibitory molecules into National Veterinary Institute, Uppsala, Sweden sample extracts or by trapping the target cells/DNA [1, 2]. Chemistry, Materials and Surfaces, Bioscience and Materials, RISE – Research Institutes of Sweden, Boras, Sweden 123 134 Accreditation and Quality Assurance (2018) 23:133–144 Further, the target DNA/RNA is often partially degraded (Fig. 1), focusing on analysis of the challenging samples and present at low levels. encountered in for example food testing, forensic DNA Validation guidelines for chemical analysis have been analysis, bioterrorism preparedness and veterinary medi- developed, for example by Eurachem , to simplify and cine. In these sectors, the sample matrix has a substantial standardise method validation, helping testing and cali- impact on the analytical success. We include a modular bration laboratories to improve their quality assurance and approach to method validation within the chain of analysis, apply for accreditation according to the ISO/IEC 17025 aiming at efﬁcient validation and a ﬂexible use of methods. standard . PCR differs from classical chemical analysis The objective is to enable rational validation of new or as it is based on the capacity of an enzyme, a DNA poly- improved methods, or for analysis of a new sample type merase, to amplify speciﬁc DNA fragments. In PCR, the with an existing method. To that end, we explain the per- target nucleic acid sequence is ampliﬁed and subsequently formance characteristics associated with method validation analysed through a process consisting of physical as well as from a PCR and biological sample matrix perspective and biochemical factors. A few validation guidelines directed propose which characteristics to investigate depending on towards PCR-based analysis have been published, mainly the type of method to be validated. We also suggest for analysis of Genetically Modiﬁed Organism (GMO) experimental setups including which sample types to apply content in foods [5–7] but also for forensic DNA analysis in single-laboratory validation of the different modules. A [8, 9]. Most guidelines have been focused on the actual speciﬁc application of the guide is the need for urgent PCR assay, but some documents incorporate considerations validation in the event of a crisis such as a foodborne also for the upstream modules of the analysis chain, i.e. outbreak. sampling, sample treatment, and DNA/RNA extraction and puriﬁcation. For the individual laboratory, the method validation The workflow in PCR diagnostics process commonly starts with a new demand, creating a need to analyse a certain sample type in a certain way, and The PCR analysis chain can be divided into four modules: ends with a laboratory decision whether or not the (1) sampling, (2) sample treatment, (3) DNA/RNA requirements are fulﬁlled by the applied method . The extraction and puriﬁcation, and (4) PCR-based analysis steps in the validation process are (1) set the requirements, (including reverse transcription for RNA analysis) (Fig. 1). (2) modify an existing method or develop a new method to Sampling must generate a representative sample from a handle new targets or sample types, (3) prepare a method large surface or background material, maximise the uptake instruction, (4) evaluate performance characteristics of target cells/DNA and ideally minimise the uptake of through validation experiments, and ﬁnally (5) a decision PCR inhibitors . Sample treatment serves to concen- regarding ‘‘ﬁtness for purpose’’. If the set requirements are trate target cells, and/or separate them from a background not met, the requirements may need to be updated or the of other cells or matrices prior to cell lysis and nucleic acid method improved. Following implementation in routine extraction. Sample treatment may be performed with dif- analysis, quality control measures are used to ensure the ferent types of methods, e.g. using ultraﬁltration for large continuous performance of the method. Here, we address water samples when testing for pathogenic microbes  the scenario of a DNA/RNA laboratory that has a validated or using laser capture microdissection to pick up individual PCR workﬂow when faced with a new demand, such as a human cells in forensic investigations . Cultivation is request from the Police to start analysing crime scene DNA often needed in food testing to meet the requirement to from new matrices. The validation process may be handled conﬁdently determine the absence of pathogens in 25 g of as a part of the continuous developmental work, or handled background material . However, sample treatment is urgently if connected with a crisis situation. Rational pro- often time-consuming and costly, why performing extrac- cedures for in-house validation are important in both cases. tion/puriﬁcation directly after sampling is preferable, when In many instances, the result of an analysis is critical, since possible. Extensive nucleic acid puriﬁcation should also be actions may have to be taken depending on them; e.g. avoided as it leads to loss of DNA/RNA . An inhibitor- recalls of foods from stores, alerts concerning microbio- tolerant DNA polymerase-buffer system may be applied to logical risks, or identiﬁcation of culprits from crime scene lower the need for puriﬁcation [1, 10]. This approach is samples. Should such an urgent analysis be requested for a part of a concept called pre-PCR processing , aiming at new sample type, for which the existing methods have not reaching an optimal limit of detection for challenging been validated, there will be little time to perform samples and at the same time keeping the analytical pro- validation. cedure efﬁcient and simple. For RNA analysis, a reverse The objective of this paper is to provide validation transcription (RT) step is needed prior to PCR, either as a guidelines for the different modules of the PCR workﬂow stand-alone process or integrated with the PCR. The 123 Accreditation and Quality Assurance (2018) 23:133–144 135 Fig. 1 The PCR analysis chain described by four modules: sampling, sample treatment, DNA/RNA extraction and PCR- based analysis. The sample ﬂow is shown to the left, starting with cells/viruses in a matrix and ending with DNA in the PCR tube. While the sample is processed, the matrix concentration ideally decreases and the analyte concentration increases (middle). Analytical speciﬁcations and performance characteristics (to the right) are included for each module for investigation in method validation (important but in- exhaustive examples) success of the RT-qPCR analysis is to a large extent Module-based method validation determined by the efﬁciency of the reverse transcription , making it vital to control this step in validation. The modular nature of the PCR workﬂow lends itself well Reverse transcription yield is for example highly affected for a modular approach to method validation as proposed by the primer type (e.g. random hexamers or speciﬁc pri- by Holst-Jensen and Berdal  (Fig. 1). There, a module mers), the RNA target and the type of RT enzyme applied is deﬁned as a method to be used in a certain step of the [15, 16]. analysis chain. If the modules are independent, each Depending on the aim of the analysis, one of these three module may be validated separately, not as a part of the technological platforms may be applied in PCR diagnos- complete procedure. This increases ﬂexibility, as a vali- tics: (1) Conventional PCR followed by electrophoresis dated module may be used in several different workﬂows detection (slab gel or capillary) or sequencing of ampli- without the need for re-validation of the whole workﬂow. cons, (2) Real-time PCR (qPCR), or (3) Digital PCR However, the complete independence between modules (dPCR). Nucleic acid analysis may be qualitative or cannot be assumed in all instances. Only limited work has quantitative, depending on the need, the applied platform been directed towards proving the generality of the mod- and the analysis process. qPCR, RT-qPCR and dPCR ular approach, and to the best of our knowledge only in the enable quantitative analysis, but when applied following GMO ﬁeld [20, 21]. Holst-Jensen and Berdal propose to cultivation of bacteria, for example, they are used quali- evaluate the performance of each module by applying non- tatively for detection of the speciﬁc target species. PCR methods, e.g. optical density (OD) absorbance mea- Guidelines for reporting of qPCR and dPCR results have surements to estimate DNA concentration and purity for been published, with the aim to improve the quality the validation of DNA extraction protocols . This tactic is scientiﬁc literature and enable justiﬁed conclusions to be valid if the modules are truly independent. In our case, drawn from PCR results [17, 18]. These guidelines may be analysing samples containing heterogeneous matrices that helpful also in method development and in-house may disturb PCR, it is important to verify the compatibility validation. between the existing PCR workﬂow and the method to be In each module of the PCR workﬂow, there are validated. We, therefore, suggest the application of previ- numerous factors that affect the analytical performance and ously validated methods from the PCR analysis chain when the measurement uncertainty (Fig. 2). For example, the cell validating a new module. Thus, the method performance type and matrix affect sampling and sample treatment, the can be conﬁrmed in a relevant context, without the need for applied cell lysis reagents and thermal conditions affect complete validation of the workﬂow, keeping the ﬂexibility DNA extraction, and standard curve generation and DNA provided by the modular approach. quality affect the PCR measurement. The relevant sources Additionally, methods such as OD or ﬂuorometry for of variation should be considered when designing valida- measuring DNA concentration and purity may not give tion experiments. relevant results with respect to PCR. PCR inhibition, for 123 136 Accreditation and Quality Assurance (2018) 23:133–144 Fig. 2 Sources of variation in PCR diagnostics. Factors that affect the performance, variation and measurement uncertainty of PCR analysis in each of the four modules are shown. Other factors than the ones mentioned may also affect variation, such as the reverse transcription step in RNA analysis example, is largely dependent on the applied DNA poly- together span all the requested properties of the method, merase-buffer system and is not directly reﬂected by OD ensuring that the right target is analysed and that it can be measured sample impurities [22, 23]. Also, for mammalian conﬁdently detected from low level samples containing cells, viruses and some bacteria, for which culture-based relevant matrices. The performance characteristics are methods are not applicable, there are no readily available deﬁned in the International Vocabulary of Metrology  methods for estimating the performance of sampling or and interpreted for validation of analytical chemistry sample treatment without applying DNA/RNA extraction methods by Eurachem [3, 25]. We build on the VIM and and PCR. Eurachem guides and describe the performance character- The impact on total measurement uncertainty from a istics from a PCR perspective, including examples and certain module may be estimated during or after validation, suggested experimental setups. Our descriptions are if necessary for the application. For example, if the varia- intended as support for establishing a validation plan prior tion coming from sample treatment, DNA extraction and to commencing the practical validation work. Different PCR is known, the variation from sampling can be deduced parameters may be important depending on the module to from experiments performed as described above. In the be validated. In Table 1, we suggest which performance PCR community, it is widely accepted that the upstream characteristics to investigate in validation of the different processes of sampling, sample treatment and extraction/ modules in the analysis chain and for the different PCR puriﬁcation, as well as reverse transcription in RNA anal- technologies. ysis, add more to the variation than the PCR assay [15, 19]. Selectivity Performance characteristics In analytical chemistry, selectivity is deﬁned as ‘‘the extent to which the method can be used to determine particular The ﬁrst step in single-laboratory validation of a new or analytes in mixtures or matrices without interferences from improved module in the analysis workﬂow should be to other components of similar behaviour’’ . In the PCR state the requirements on the method. The requirements are context, this is related to the ability of the method to detect generally given as limits for a set of performance charac- target DNA/RNA sequences in a background of non-target teristics, i.e. selectivity, limit of detection (LOD), limit of nucleic acids. In microbial analysis, detecting the variants quantiﬁcation (LOQ), working range, analytical sensitiv- that should be detected is referred to as inclusivity, and ity, trueness, precision, ruggedness, and matrix effects. We excluding those that should not be detected is referred to as have also included contamination risk and carry-over as exclusivity . Here, we choose to separate selectivity these two are important parameters in PCR diagnostics. from matrix effects, i.e. the impact of the matrix substances Ideally, the investigated performance characteristics (here deﬁned as non-nucleic acid content) in the samples. 123 Accreditation and Quality Assurance (2018) 23:133–144 137 Table 1 Performance characteristics to be evaluated in the validation ‘‘?’’, those that may be tested depending on the situation are marked of different modules of the PCR analysis workﬂow. Parameters that with ‘‘?/-’’, and less important/not applicable parameters are marked are important to investigate for a certain module type are marked with with ‘‘-’’ Performance characteristics Module to be validated Sampling Sample treatment DNA/RNA extraction PCR-based analysis Conventional PCR qPCR/RT-qPCR dPCR Selectivity -- - ? ? ? LOD ?/-?/-?/-? ? ? LOQ ?/-?/-?/-?/-? ? Working range ?/-?/-?/-?/-? ? Analytical sensitivity -- - - ?/-?/- Trueness Bias/recovery ?? ? ?/-? ? Precision Repeatability ?? ? ? ? ? Intermediate precision ?? ? ? ? ? Reproducibility -- - ?/-?/-?/- Ruggedness ?? ? ? ? ? Matrix effects ?? ? ? ? ? Contamination risk ?? ? ? ? ? Thus, selectivity testing as described below is performed the food supply chain or in veterinary medicine. The with puriﬁed DNA to distinguish the outcome from matrix identity can be conﬁrmed by applying reference strains or effects. Matrix effects and in particular PCR inhibition are puriﬁed reference material with known DNA proﬁles , further discussed later. or reference methods. As stated in the Eurachem validation An initial step in evaluating selectivity is to ensure that guide , an independent method should be used to con- the generated signal originates from the requested analyte, ﬁrm that the analysis method identiﬁes the analyte it is i.e. conﬁrmation of identity (Fig. 3). In PCR diagnostics, designed to detect. In qPCR, ampliﬁcation curves are the analyte is a speciﬁc DNA or RNA sequence which is generated that should reﬂect the ampliﬁcation of the target. ampliﬁed to enable identiﬁcation of, for example, a par- However, this signal could also be caused by the ampliﬁ- ticular individual in a forensic investigation or pathogen in cation of non-speciﬁc products or artefacts such as primer- dimers, especially when non-speciﬁc DNA binding dyes such as SYBR Green I are used for detection. Here, the conﬁrmation of identity can be achieved by determining that the PCR product has the expected size, for example applying gel electrophoresis (Fig. 3) or melt curve analy- sis. For further conﬁrmation, the product may also be sequenced and identiﬁed in a nucleotide sequence database, if deemed necessary. Conﬁrmation of identity testing should preferably be performed using pure DNA/RNA from a speciﬁc target microorganism or a human individual, depending on the application. Conﬁrmation of identity testing can be per- formed as a limited and simple experiment, often done Fig. 3 Conﬁrmation of identity in PCR-based analysis. In this when the PCR assay is ﬁrst set up at the laboratory. example, the source of the qPCR output (ampliﬁcation curve) is veriﬁed by performing gel electrophoresis to determine the size of the For PCR assays, selectivity is determined primarily by generated DNA fragment. The grey ampliﬁcation curve and gel bands the constructed primers and probes, supposed to bind are the result of correct ampliﬁcation, conﬁrming that the assay speciﬁcally only to the intended sequences of the target detects the target it is supposed to detect. The black ampliﬁcation region(s) (Fig. 4). However, selectivity is also affected by curve, on the other hand, comes from the detection of incorrect (smaller) amplicons (unspeciﬁc products or primer-dimers) 123 138 Accreditation and Quality Assurance (2018) 23:133–144 for the inclusivity test and at least 30 strains for the exclusivity test. For qPCR assays targeting human DNA, a number of human individuals and samples from other species may be tested. Selectivity is generally only rele- vant for the PCR modules (Table 1). For determination of selectivity, an amount of DNA/RNA that does not challenge the limit of detection of the PCR assay should be used. The above-mentioned ISO 22118:2011 standard states that: ‘‘a clearly detectable amount of DNA, e.g. representing DNA of 10 cells, should be used for the selectivity testing’’ . For bacteria, 1 ng of DNA per reaction generally meets this criterion, corresponding to 5 6 approximately 1.5910 – 1.5910 genome copies. Fig. 4 Selectivity of a PCR assay. The samples/strains detected by Limit of detection (LOD), limit of quantification the assay are visualised with the dashed line circle, showing true- (LOQ) and working range positive results (ﬁlled grey circles inside the dashed line), false- negative results (ﬁlled grey circles outside the dashed line), false- positive results (white circles inside the dashed line) and true-negative LOD refers to the smallest concentration of analyte that results (white circles outside the dashed line) can be detected by the method with a given probability. Commonly, both for PCR-based methods and in other physical and chemical factors such as annealing tempera- contexts, LOD is used, which is deﬁned as the lowest ture and the applied concentration of magnesium ions in concentration of analyte at which 95 % of the positive the assay. A lower annealing temperature or higher mag- samples are detected by the analysis method . Limit of nesium ion concentration generally elevates the risk of quantiﬁcation (LOQ) refers to the lowest analyte concen- generating faulty products through increased stability of tration that can be determined with acceptable uncertainty. primer-DNA binding (i.e. the primer may bind to DNA Working range refers to the range of analyte concentrations even if several bases are mismatched). Thus, selectivity that can be quantiﬁed with acceptable accuracy. The lowest must be re-evaluated if changing any of these conditions point in the working range is the LOQ. for a validated method. When an assay is designed, the LOD, LOQ and working range for a PCR assay can be selectivity is usually tested in silico using an appropriate determined by means of a dilution series containing known reference genome sequence database. This gives a predic- amounts of target DNA/RNA (Fig. 5). The dilution series tion of whether or not the designed primers will bind only should include several replicates and concentrations of to the target sequence. However, the true selectivity should nucleic acid to give a useful estimate of the LOD and/or be determined empirically, by PCR analysis of DNA LOQ and working range. More replicates may be extracted from target organisms, not only by in silico analysis . For microbial methods, a panel of nucleic acid samples from relevant strains is usually set up to evaluate inclu- sivity and exclusivity. To determine the inclusivity in pathogen testing (deﬁned as ‘‘the strains or isolates of the target analyte(s) that the method can detect’’ ), the panel should preferably include a diversity of organisms (genus, species, subspecies, serotypes, etc.) that the assay is intended to detect. For exclusivity (deﬁned as ‘‘the non- target strains, which are potentially cross-reactive, that are not detected by the method’’ ), the panel should include: (1) closely related strains, (2) strains that are commonly found in relevant samples and (3) non-related agents which may give similar symptoms or may occur in the same environment . In the ISO 22118:2011 stan- Fig. 5 Determination of LOD, LOQ and working range in qPCR. dard for PCR detection and quantiﬁcation of foodborne Quantiﬁcation cycle (Cq) values from a dilution series of DNA are plotted against log of the DNA concentration to generate a standard pathogens , it is recommended to use at least 50 strains curve covering the working range 123 Accreditation and Quality Assurance (2018) 23:133–144 139 introduced close to the critical levels in order to improve be the DNA concentration of a certiﬁed reference material, the LOD/LOQ estimations. The dilution series may consist e.g. as provided by NIST for human DNA . of pure standard DNA or, preferably, target cells/DNA in a In validation of sampling or DNA/RNA extraction relevant matrix. The latter ensures that ampliﬁcation efﬁ- methods, bias may be measured in recovery experiments. ciency is similar for the prepared samples as for the ‘‘real’’ These spiking tests can be performed by adding a certain samples, making the LOD, LOQ and working range esti- amount of target cells to blank matrices before DNA/RNA mations relevant for the routine analysis situation. In fact, extraction and measure the recovered proportion. In this quantiﬁcation with qPCR builds on the assumption of case, recovery is a measure of the efﬁciency of DNA/RNA identical ampliﬁcation efﬁciencies for standards and extraction. Comparisons may also be made against an unknown samples. In dPCR, no standard curve is needed established reference method, where the reference method for absolute quantiﬁcation, making the technology less result may be set to 100 %. Alternatively, cells can be affected by differing ampliﬁcation efﬁciencies, e.g. due to counted before spiking and the theoretical DNA amount impurities . LOD and LOQ may also be investigated used as a reference value. For example, it is estimated that when validating pre-PCR modules, if deemed necessary one human haploid cell contains around 6 pg DNA . (Table 1). Recovery is also referred to as yield. LOD can be determined for the PCR assay separately, but in general it is more relevant to determine the LOD for Precision the whole analysis chain. Then, more modules and aspects of the workﬂow must be considered and, if relevant, Precision refers to the random variation of a method and included in the tests. A common test design for the eval- may be determined as repeatability, intermediate precision uation of LOD is to spike (i.e. add) target cells (or nucleic or reproducibility, depending on what is most appropriate acid) in different levels to relevant matrices. The samples for the particular module. Distribution measurements such are then processed according to the analytical procedure as standard deviation or coefﬁcient of variation may be which can include sample treatment steps such as culture applied for all the three precision parameters. Precision is enrichment and concentration, and DNA/RNA extraction an important parameter for modules in all steps of the steps such as cell lysis, ﬁltration, and elution. Spiking is not analysis chain (Table 1). as ideal as using real samples, but as real samples are often Repeatability is the variation between analyses con- lacking and also have unknown contents, spiking is often ducted in an identical way, for example replicates within a the best choice available. DNA extraction batch or a PCR run. Thus, the analyses for repeatability testing are performed with identical reagents Analytical sensitivity and applying the same instruments, within a short period of time. Analytical sensitivity refers to the change in instrument Intermediate precision is the variation between analyses response signal as a function of change of analyte con- performed at one laboratory under somewhat different centration. Note that this differs from diagnostic sensitiv- conditions , for example with different persons per- ity, which refers to the ability to diagnose correctly. The forming DNA/RNA extraction or applying different word sensitivity should be avoided when referring to LOD, reagent lots or PCR instruments. Separation in time to avoid any confusion. In general, less importance can be between analyses also counts as intermediate precision given to evaluating analytical sensitivity for PCR-based conditions. analysis; it is rarely interesting to determine which of a pair Reproducibility refers to variation between measure- of unknown samples that contains the highest amount of ments performed at different locations/laboratories . target cells. This is a required part of validation of some newly developed analysis methods, e.g. new qPCR assays tar- Trueness geting pathogens. Inter-laboratory studies are, for example, required in the validation of alternative methods to be used For quantitative methods, trueness is ‘‘an expression of in the ofﬁcial control of food and feed, replacing stan- how close the mean of an inﬁnite number of results (pro- dardised reference methods . Reproducibility may be duced by the method) is to a reference value’’ . Thus, it determined through ring trials, i.e. by analysing replicated is connected with the systematic variation of a method. samples in different laboratories and comparing the results. Trueness cannot be measured directly, but may be esti- For more established methods, it is generally not necessary mated as bias. Bias refers to the proximity between the for the individual laboratory to further investigate repro- measurement value and the true value or, alternatively, a ducibility as part of in-house validation. reference value. For a qPCR assay, the reference value may 123 140 Accreditation and Quality Assurance (2018) 23:133–144 Ruggedness ampliﬁcation (e.g. from humic acid in soil, blood, faeces, feed [10, 35]), and blocked amplicon detection (e.g. from Ruggedness, sometimes referred to as robustness, is the denim fabric, blueberries, soil ). PCR inhibition, i.e. disturbing ampliﬁcation or amplicon method’s insensitivity for small, consciously made changes in the experimental conditions. Ruggedness is evaluated detection, is arguably the most important matrix effect in PCR diagnostics. PCR inhibitory molecules may emanate during validation by varying key parameters or reagent concentrations and studying the effects. For PCR methods, from the sample, the background material, or be added in the effects of slightly varying the temperatures and incu- the analytical chain (Fig. 6). Examples of the latter are bation times during thermal cycling or applying different DNA extraction ingredients such as phenol, SDS, EDTA, primer/probe amounts may be evaluated. For reagents, Chelex, all of which are known PCR inhibitors with dif- ferent modes of disturbing the reaction [10, 37]. All rele- deviations of around ± 10 % from the optimal concentra- tion are frequently applied. This type of test provides vant sources of PCR inhibitors should be investigated in validation, through experiments applying relevant matrices information on how robust the method is in regard to pipetting errors. The outcome of the ruggedness test may at relevant levels. To limit the amount of experiments, matrices with varying effects are preferably chosen. See for be used to determine the limits of the method, for example concerning incubation time ranges in different steps of example Ref.  for a list of PCR inhibitors and their respective mechanisms. The choice of matrices  for DNA/RNA extraction. testing should also be determined by the nature of the target Matrix effects: PCR inhibition to be analysed. For a Francisella tularensis assay, for example, relevant PCR-inhibitory background materials Matrix effects refer to the possibility of obtaining a true include soil, mosquito, water, and clinical samples, as these reﬂect the environments where the bacterium may be found positive result when the analyte is present in a certain matrix, and a true negative result if it is absent. In that way, . Francisella tularensis could also appear as an agent in bioterrorism , with other possible disturbing matrices it resembles selectivity, with the distinction that the focus is on the background material, the matrix, rather than on such as various surfaces (through aerosols) and carcasses. the design of the PCR assay. The matrix effects may improve detection, such as a matrix that acts as a carrier for Contamination risk and carry-over the analyte or a matrix that promotes growth of a target bacterium, but it is more common that a matrix disturbs Contamination risk is the risk of detecting analytes not analysis. A negative matrix effect may cause false-negative derived from the original sample, but instead being added results, partial results, or incorrect quantiﬁcation through along the analysis chain. Contaminating cells/DNA may lowering of ampliﬁcation efﬁciency. Determining the come from the person performing sampling or DNA extraction, especially when human DNA is targeted, or limitations and understanding matrix effects is a vital part of the validation of methods in the PCR workﬂow. Among from consumables and reagents used, such as swabs, plastic tubes and buffers (Fig. 7). From the perspective of food the possible negative effects to look into are: trapping of cells in DNA extraction/puriﬁcation (e.g. cells binding safety, contamination may have a different meaning, i.e. that the tested food stuffs contain the target microorganism tightly to cotton or soil), impaired culture of microorgan- isms (e.g. from heroin samples), inhibition of PCR Fig. 6 Sources of matrix effects in PCR. The sample ﬂow in the PCR grey ampliﬁcation curve signiﬁes ideal ampliﬁcation, and the black analysis chain is shown. Substances that disturb PCR (i.e. PCR curve signiﬁes ampliﬁcation affected by inhibitors (lowered ampli- inhibitors) may be added to the samples in any of the modules. The ﬁcation efﬁciency) 123 Accreditation and Quality Assurance (2018) 23:133–144 141 Fig. 7 Contamination risks in the analytical procedure. Contaminating cells or molecules may be added to the sample in any of the modules leading up to analysis due to poor food hygiene. Here, we use the word con- the method is, for example, whether it is a commercially tamination in the analytical sense described above. available method that has been quality assured by a man- Carry-over refers to the risk that a sample analysed in an ufacturer or a new, in-house method. ISO methods have instrument affects the next test. A speciﬁc carry-over issue in normally been validated through inter-laboratory testing PCR-based analysis is the enormous multiplication of target and do not need to be extensively validated by the testing molecules, creating a risk that amplicons from one reaction laboratory. However, the performance of ISO methods contaminate another prior to ampliﬁcation. Therefore, pre- should be veriﬁed at the laboratory. A recurring question and post-PCR areas must be separated, preferably in differ- for DNA/RNA laboratories, for example in a crisis situa- ent rooms with different air pressures . Carry-over within tion such as a foodborne outbreak, is: ‘‘if a method has capillary electrophoresis instruments can be evaluated by been validated for analysis of agent X in sample type A, analysing blank samples following samples with high can the same method be applied for analysis of sample type amounts of amplicons. In general, the contamination risk is B?’’ If A and B are distinct, it may be necessary to perform investigated by including negative controls in the validation a limited validation study to verify the performance for B. study to monitor the relevant modules. Negative controls are Six different cases for method validation are listed below. treated the same as the samples, with the only difference that In each case, the laboratory must determine how extensive no target cells/DNA are consciously added to them. the validation needs to be in their particular case. 1. Standard method (e.g. ISO) 2. Commercial method/kit, validated by the manufacturer Planning the validation study: practical 3. Method published in scientiﬁc journal considerations 4. In-house developed method 5. Modiﬁed method, of type 1-4 Validation can be a laborious undertaking, creating a need 6. Validated method to be used with new sample for rational validation design, relevant for the method at types/matrices hand. Considering all possible matrices that salmonella or human culprit DNA may appear in, the theoretical scope of Choosing relevant sample types is an important part in a perfect validation study includes an almost inﬁnite the planning of a validation study. Spiking experiments are number of samples. Hence, key (i.e. very common or highly useful to that end, as spiking reduces the number of particularly challenging) sample types should be chosen to unknowns and enables quantitative analysis of the perfor- make validation relevant as well as reasonable concerning mance characteristics including precisions measures. However, it is difﬁcult to mimic the full complexity of time and resources. Another challenge is to limit the samples to a manageable number. Hence, the number of ‘‘real’’ samples with spiking experiments. Therefore, a sample types, nucleic acid levels and replicates must be range of different samples from routine analysis (or pre- determined to get (a) the information needed to assess the pared samples mimicking routine samples) should prefer- performance characteristics, and (b) a feasible experimen- ably be applied to complement the replicated spiking tal setup. For example, analysis of 50 samples in total has experiments. been suggested for the internal validation of commercial To set up a feasible validation study for a pathogen methods in forensic DNA analysis . testing method, a few relevant strains of the target organ- The design of the single-laboratory validation study and ism must be chosen. One such example is the inter-labo- number of analyses performed relies on how established ratory validation of the ISO 10272-1 method for detection 123 142 Accreditation and Quality Assurance (2018) 23:133–144 and enumeration of campylobacter . The method was sampling module (Table 1). Recovery gives an estimate of designed for detection of species of campylobacter in the efﬁciency of sampling, and intermediate precision may samples from the food supply chain. Seventeen laboratories be applied to study variation between individuals per- participated in the validation study. Five different sample forming sampling. It may also be of importance to look types were used (broiler caecal material, frozen spinach, into LOD, e.g. when investigating LOD for the whole frozen minced pork/beef, raw milk, and chicken skin). analysis chain. Each laboratory received eight samples per sample type containing high level, low level or no Campylobacter (i.e. 24 samples in total per sample type). One strain of Validation of a sample treatment method Campylobacter jejuni or of C. coli was used per sample type, presumably to keep the total number of samples at a In many instances, DNA/RNA extraction is performed manageable number. In this example, the bacterial species directly after sampling. However, in some cases sample were chosen since they are relevant food contaminants and treatment may be needed as a link between sampling and also good representatives for their species. Both these DNA/RNA extraction, e.g. to concentrate the target cells. factors should be considered when designing the study. In In validation, it is important to apply relevant samples general, it should also be considered whether or not the concerning both sample matrix, cell type and sampling organism is expected to be persistent (bioﬁlm formation, method. For pathogen testing of water, this may include resistance, etc.) in routine testing and preferably this should applying clean water as well as water with different be reﬂected in the validation experiments. amounts of humic substances . Recovery is the most Through module-based validation it will only be nec- important parameter in relation to sample treatment. essary to validate the actual method that has been added or modiﬁed, not the entire analytical procedure. This saves time and cost. However, it is still necessary to verify the Validation of a DNA/RNA extraction method performance of the whole analysis chain, to ensure the compatibility with the new method. Each step of the The bulk of experiments in validation of DNA/RNA workﬂow has its own speciﬁc challenges concerning vali- extraction methods may be performed applying relevant dation planning. Below we give some practical advice for matrices spiked with known numbers of target cells each of the modules concerning experimental setup and (e.g. microorganism or human cells). This approach makes choice of matrices for testing. it possible to quantify recovery, precision and matrix effects (Table 1). In forensics, for example, cigarette ﬁlter Validation of a sampling method paper from a certain brand may be spiked with a certain volume of saliva to investigate matrix effects. This may be Sampling may be direct, meaning that a piece of a mate- complemented with a set of smoked cigarettes of different rial/matrix is taken directly for further processing, or brands (i.e. real samples) to pick up any other matrix indirect, meaning that a sampling device is used to lift the effects. sample from the material. Swabbing is arguably the most Recovery is a key parameter—how much of the avail- common approach for indirect sampling, in forensic DNA able target DNA is successfully recovered by the method? analysis as well as in microbial testing. To validate a Recovery may be investigated by comparing the amount of sampling method, relevant matrices, free from target ana- target cells or DNA/RNA added to the sample with the lyte, may be spiked with a known amount of target cells/ amount retrieved after extraction, or be calculated as a ratio viruses. In forensic DNA analysis, a certain amount of against a reference method. Any variation linked to the saliva or blood may be put on a relevant surface and technical setup and to individuals performing the pipetting sampled after drying. The outcome may be compared should be investigated, making both repeatability and against a reference method, or against a theoretical value intermediate precision important. Matrix effects may partly coupled to the number of target cells applied. Spiking with be evaluated while performing the DNA/RNA extraction. known amounts of target may not be applicable in all E.g. is the matrix compatible with the reagents and instances, for example in some forensic DNA analysis instruments used for extraction? Some matrices may for applications. When validating a method for sampling of example clog pipette tips and hence disable the method. shed human cells on clothes, reference material may Matrices may also interfere with the downstream analysis, instead be prepared by someone wearing a set of identical e.g. if the extraction method does not remove PCR inhi- garments in a controlled fashion for a speciﬁed amount of bitors in a satisfactory way. This may be analysed by time . Trueness and precision are arguably the most spiking generated extracts (free of target) with a certain important parameters to investigate in the validation of a 123 Accreditation and Quality Assurance (2018) 23:133–144 143 Acknowledgments This study was ﬁnancially supported by the 2:4 amount of pure DNA and investigate if the extracts cause grant distributed by the Swedish Civil Contingencies Agency, project impaired PCR ampliﬁcation. ‘‘Laboratoriediagnostik med ho¨g kapacitet, via effektiv imple- menteringssamordning, inriktad mot framtidens beredskapsanalys’’. Validation of a PCR or RT-PCR method Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creative PCR inhibition is a main limiting factor in PCR diagnostics commons.org/licenses/by/4.0/), which permits unrestricted use, dis- and should be carefully studied in method validation. By tribution, and reproduction in any medium, provided you give spiking PCRs with relevant, homogenised matrices and appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were adding DNA of high quality, PCR inhibition effects may be made. determined in a straightforward and reproducible way. This approach enables quantiﬁcation of matrix effects as well as ensures similar effects over time. One challenge in vali- References dation is to choose appropriate reference materials that together give a broad range of relevant inhibitory effects. 1. Ra˚dstro¨m P, Knutsson R, Wolffs P, Lo¨venklev M, Lo¨fstro¨mC In forensic DNA analysis, manufacturers often validate the (2004) Pre-PCR processing: strategies to generate PCR-compat- ible samples. Mol Biotechnol 26:133–146 inhibitor tolerance of their DNA proﬁling systems by 2. Alaeddini R (2011) Forensic implications of PCR inhibition—a applying puriﬁed hematin as a model for blood and humic review. Forensic Sci Int Genet 6:297–305 acid as a model for soil [45, 46]. An alternative strategy is 3. Magnusson B, Ornemark U (eds) (2014) Eurachem Guide: the to prepare casework-like reference materials containing ﬁtness for purpose of analytical methods—a laboratory guide to method validation and related topics, 2nd edn. ISBN 978-91- solutions of different matrices, such as cigarettes, chewing 87461-59-0. http://www.eurachem.org. Accessed 15 Sept 2017 gum, and soil, giving a more complex content . In RNA 4. ISO/IEC 17025:2005 General requirements for the competence of analysis, the RT step must be included in inhibition testing. testing and calibration laboratories. International Organization for The efﬁciency of the RT, generating complementary DNA Standardization, Geneva 5. Mazzara M, Savini C, Charles Delobel C, Broll H, Damant A, (cDNA), is generally not measured directly due to a lack of Paoletti C, Van Den Eede G (2008) Deﬁnition of minimum suitable methods. Instead, PCR/qPCR is applied for the performance requirements for analytical methods of GMO testing measurements. RT generally adds more to the variation European network of GMO laboratories. Publication ofﬁce of the between measurements than the PCR step . European Union - OPOCE, Luxembourg 6. Broeders S, Huber I, Grohmann L, Berben G, Taverniers I, Mazzara M, Roosens N, Morisset D (2014) Guidelines for vali- dation of qualitative real-time PCR methods. Trends Food Sci Summary Technol 37:115–126 7. Hubner P, Waiblinger HU, Pietsch K, Brodmann P (2001) Vali- dation of PCR methods for quantitation of genetically modiﬁed In this paper, we present guidelines for single-laboratory plants in food. J AOAC Int 84:1855–1864 validation of methods applied in the PCR analysis chain. The 8. SWGDAM (2016) Scientiﬁc Working Group on DNA Analysis speciﬁc focus is analysis of DNA/RNA in sectors such as Methods: Validation Guidelines for DNA Analysis Methods. food safety, forensics, bioterrorism preparedness and vet- Retrieved from https://docs.wixstatic.com/ugd/4344b0_ 813b241e8944497e99b9c45b163b76bd.pdf. 15 Sept 2017 erinary medicine, where the target is often in low levels and 9. ENFSI (2010) Recommended minimum criteria for the validation mixed with high amounts of complex matrices. These of various aspects of the DNA proﬁling process, European Net- guidelines serve to help laboratories to ensure the perfor- work of Forensic Science Institutes, DNA working group. mance of their new or modiﬁed methods using relevant Retrieved from http://enfsi.eu/wp-content/uploads/2016/09/mini mum_validation_guidelines_in_dna_proﬁling_-_v2010_0.pdf.15 sample matrices. The choice of matrices to test during vali- Sept 2017 dation is of great importance. Relevant matrices should be 10. Hedman J, Ra˚dstro¨m P (2013) Overcoming inhibition in real-time chosen based on for example which sample types that are diagnostic PCR. Methods Mol Biol 943:17–48 expected for a certain target in routine analysis. The guide- 11. Polaczyk AL, Narayanan J, Cromeans TL, Hahn D, Roberts JM, Amburgey JE, Hill VR (2008) Ultraﬁltration-based techniques for line may be applied in a crisis situation, such as a foodborne rapid and simultaneous concentration of multiple microbe classes outbreak, requiring urgent analysis of new sample types. In from 100-L tap water samples. J Microbiol Methods 73:92–99 that case, there is little time to perform and evaluate vali- 12. Elliott K, Hill DS, Lambert C, Burroughes TR, Gill P (2003) Use dation experiments, meaning that a strategy for method of laser microdissection greatly improves the recovery of DNA from sperm on microscope slides. Forensic Sci Int 137:28–36 validation should be present beforehand. By applying a 13. EC (2005) Commission Regulation (EC) No 2073/2005 of 15 modular approach to validation the methods can be used with November 2005 on microbiological criteria for foodstuffs (Text more ﬂexibility and the validation studies can be made less with EEA relevance), pp 1–26. Retrieved from: http://eur-lex. laborious. The compatibility between the existing workﬂow europa.eu/eli/reg/2005/2073/oj. 15 Sept 2017 14. Miller DN, Bryant JE, Madsen EL, Ghiorse WC (1999) Evalu- and the new method is veriﬁed by applying previously val- ation and optimization of DNA extraction and puriﬁcation idated methods in the validation study. 123 144 Accreditation and Quality Assurance (2018) 23:133–144 procedures for soil and sediment samples. Appl Environ Micro- quantiﬁcation of foodborne pathogens—Performance character- biol 65:4715–4724 istics. International Organization for Standardization, Geneva ˚ ˚ 15. Stahlberg A, Hakansson J, Xian X, Semb H, Kubista M (2004) 31. Sidstedt M, Romsos EL, Hedell R, Ansell R, Steffen CR, Vallone ˚ ¨ Properties of the reverse transcription reaction in mRNA quan- PM, Radstrom P, Hedman J (2017) Accurate digital polymerase tiﬁcation. Clin Chem 50:509–515 chain reaction quantiﬁcation of challenging samples applying 16. Sta˚hlberg A, Kubista M, Pfafﬂ M (2004) Comparison of reverse inhibitor-tolerant DNA polymerases. Anal Chem 89:1642–1649 transcriptases in gene expression analysis. Clin Chem 32. Kline MC, Duewer DL, Travis JC, Smith MV, Redman JW, 50:1678–1680 Vallone PM, Decker AE, Butler JM (2009) Production and cer- 17. Bustin SA, Benes V, Garson JA, Hellemans J, Huggett J, Kubista tiﬁcation of NIST Standard Reference Material 2372 Human M, Mueller R, Nolan T, Pfafﬂ MW, Shipley GL, Vandesompele DNA Quantitation Standard. Anal Bioanal Chem 394:1183–1192 J, Wittwer CT (2009) The MIQE guidelines: minimum infor- 33. Dolezel J, Bartos J, Voglmayr H, Greilhuber J (2003) Nuclear mation for publication of quantitative real-time PCR experiments. DNA content and genome size of trout and human. Cytometry A Clin Chem 55:611–622 51:127–128; author reply 129 18. Huggett JF, Foy CA, Benes V, Emslie K, Garson JA, Haynes R, 34. ISO 16140:2003 Microbiology of food and animal feeding Hellemans J, Kubista M, Mueller RD, Nolan T, Pfafﬂ MW, stuffs—Protocol for the validation of alternative methods. Inter- Shipley GL, Vandesompele J, Wittwer CT, Bustin SA (2013) The national Organization for Standardization, Geneva digital MIQE guidelines: minimum information for publication of 35. Hedman J, Knutsson R, Ansell R, Ra˚dstro¨m P, Rasmusson B quantitative digital PCR experiments. Clin Chem 59:892–902 (2013) Pre-PCR processing in bioterrorism preparedness: 19. Holst-Jensen A, Berdal KG (2004) The modular analytical pro- improved diagnostic capabilities for laboratory response net- cedure and validation approach and the units of measurement for works. Biosecur Bioterror 11:S87–S101 genetically modiﬁed materials in foods and feeds. J AOAC Int 36. Sidstedt M, Jansson L, Nilsson E, Noppa L, Forsman M, Ra˚d- 87:927–936 stro¨m P, Hedman J (2015) Humic substances cause ﬂuorescence 20. Bellocchi G, De Giacomo M, Foti N, Mazzara M, Palmaccio E, inhibition in real-time polymerase chain reaction. Anal Biochem Savini C, Di Domenicantonio C, Onori R, Van den Eede G 487:30–37 (2010) Testing the interaction between analytical modules: an 37. Boiso L, Hedman J (2017) Overcoming sodium dodecyl sulfate example with Roundup Ready soybean line GTS 40-3-2. BMC induced PCR inhibition. Forensic Sci Int Genet 29:e16–e18 Biotechnol 10:55 38. Keim P, Johansson A, Wagner DM (2007) Molecular epidemi- 21. Lievens A, Jacchia S, Kagkli D, Savini C, Querci M (2016) ology, evolution, and ecology of Francisella. Ann N Y Acad Sci Measuring digital PCR quality: performance parameters and their 1105:30–66 optimization. PLoS ONE 11:e0153317 39. Dennis DT, Inglesby TV, Henderson DA, Bartlett JG, Ascher 22. Roussel Y, Wilks M, Harris A, Mein C, Tabaqchali S (2005) MS, Eitzen E, Fine AD, Friedlander AM, Hauer J, Layton M, Evaluation of DNA extraction methods from mouse stomachs for Lillibridge SR, McDade JE, Osterholm MT, O’Toole T, Parker G, the quantiﬁcation of H. pylori by real-time PCR. J Microbiol Perl TM, Russell PK, Tonat K, Working Group on Civilian B Methods 62:71–81 (2001) Tularemia as a biological weapon: medical and public 23. Gryson N, Dewettinck K, Messens K (2007) Inﬂuence of cocoa health management. JAMA 285:2763–2773 components on the PCR detection of soy lecithin DNA. Eur Food 40. Mifﬂin TE (2007) Setting up a PCR laboratory. CSH Protoc Res Technol 226:247–254 2007:5–14 24. International vocabulary of metrology – Basic and general con- 41. Butler JM (2009) Debunking some urban legends surrounding cepts and associated terms (VIM), JCGM 200:2012, www.bipm. validation within the forensic DNA community. Promega Proﬁles org. A previous version is published as ISO/IEC Guide 99:2007, in DNA 9:3–6 ISO Geneva. Accessed 15 Sept 2017 42. ISO 10272-1:2017 Microbiology of the food chain—Horizontal 25. Barwick VJ, Prichard E (eds) (2011) Eurachem guide: termi- method for detection and enumeration of Campylobacter spp.— nology in analytical measurement—introduction to VIM 3. Part 1: Detection method. International Organization for Stan- ISBN: 978-0-948926-29-7. www.eurachem.org. Accessed 15 dardization, Geneva Sept 2017 43. Forsberg C, Wallmark A, Hedell R, Jansson L, Ansell R, Hedman 26. Vessman J, Stefan R, van Staden J, Danzer K, Lindner W, Burns J (2015) Reference material for comparison of different adhesive D, Fajgelj A, Mu¨ller H (2001) Selectivity in analytical chemistry tapes for forensic DNA sampling. Forensic Sci Int Genet Suppl (IUPAC recommendations 2001). Pure Appl Chem Ser 5:454–455 73:1381–1386 44. Borgma¨stars E, Jazi MM, Persson S, Jansson L, Ra˚dstro¨mP, 27. Brunelle S, LaBudde R, Nelson M, Wehling P (2012) AOAC Simonsson M, Hedman J, Eriksson R (2017) Improved detection INTERNATIONAL Methods Committee Guidelines for Valida- of norovirus and hepatitis A virus in surface water by applying tion of Microbiological Methods for Food and Environmental pre-PCR processing. Food Environ Virol 9:395–405 Surfaces. Retrieved from http://www.aoac.org/aoac_prod_imis/ 45. Green RL, Lagace RE, Oldroyd NJ, Hennessy LK, Mulero JJ AOAC_Docs/StandardsDevelopment/AOAC_Validation_Guide (2013) Developmental validation of the AmpFlSTR NGM SElect lines_for_Food_Microbiology-Prepub_version.pdf. 15 Sept 2017 PCR Ampliﬁcation Kit: a next-generation STR multiplex with the 28. Kline MC, Butts ELR, Hill CR, Coble MD, Duewer DL, Butler SE33 locus. Forensic Sci Int Genet 7:41–51 JM (2011) The new Standard Reference Material1 2391c: PCR- 46. Tucker VC, Hopwood AJ, Sprecher CJ, McLaren RS, Rabbach based DNA proﬁling standard. For Sci Int Genet Suppl Ser DR, Ensenberger MG, Thompson JM, Storts DR (2012) Devel- 3:e355–e356 opmental validation of the PowerPlex ESX 16 and PowerPlex 29. Morales SE, Holben WE (2009) Empirical testing of 16S rRNA ESX 17 Systems. Forensic Sci Int Genet 6:124–131 gene PCR primer pairs reveals variance in target speciﬁcity and 47. Sanga M, Boiso L, Lindsten H, Ra˚dstro¨m P, Ansell R, Hedman J efﬁcacy not suggested by in silico analysis. Appl Environ (2015) A panel of PCR-inhibitory reference materials for quality Microbiol 75:2677–2683 evaluation of multiplex STR analysis kits. Forensic Sci Int Genet 30. ISO 22118:2011 Microbiology of food and animal feeding Supp Ser 5:e317–e319 stuffs—Polymerase chain reaction (PCR) for the detection and
Accreditation and Quality Assurance – Springer Journals
Published: Jun 5, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera