Assessment of Quadrivalent Human Papillomavirus Vaccine Safety Using the Self-Controlled Tree-Temporal Scan Statistic Signal-Detection Method in the Sentinel System

Assessment of Quadrivalent Human Papillomavirus Vaccine Safety Using the Self-Controlled... Abstract The self-controlled tree-temporal scan statistic—a new signal-detection method—can evaluate whether any of a wide variety of health outcomes are temporally associated with receipt of a specific vaccine, while adjusting for multiple testing. Neither health outcomes nor postvaccination potential periods of increased risk need be prespecified. Using US medical claims data in the Food and Drug Administration’s Sentinel system, we employed the method to evaluate adverse events occurring after receipt of quadrivalent human papillomavirus vaccine (4vHPV). Incident outcomes recorded in emergency department or inpatient settings within 56 days after first doses of 4vHPV received by 9- through 26.9-year-olds in 2006–2014 were identified using International Classification of Diseases, Ninth Revision, diagnosis codes and analyzed by pairing the new method with a standard hierarchical classification of diagnoses. On scanning diagnoses of 1.9 million 4vHPV recipients, 2 statistically significant categories of adverse events were found: cellulitis on days 2–3 after vaccination and “other complications of surgical and medical procedures” on days 1–3 after vaccination. Cellulitis is a known adverse event. Clinically informed investigation of electronic claims records of the patients with “other complications” did not suggest any previously unknown vaccine safety problem. Considering that thousands of potential short-term adverse events and hundreds of potential risk intervals were evaluated, these findings add significantly to the growing safety record of 4vHPV. data mining, epidemiologic research design, human papillomavirus recombinant vaccine quadrivalent, types 6, 11, 16, 18, papillomavirus vaccines, vaccination Editor’s note: An invited commentary on this article appears on page 1277, and the authors’ response appears on page 1281. Despite the cancer-preventing promise of human papillomavirus (HPV) vaccines and national recommendations for routine HPV vaccination of females and males at age 11–12 years, HPV vaccine coverage in the United States lags behind that of other adolescent vaccines (1) more than a decade after the first HPV vaccine was licensed. One reason for this is persistent concern about the safety of HPV vaccines on the part of parents and the public (2–4). The scientific literature generally does not bear out these worries. A substantial body of published evidence has accumulated regarding the safety of quadrivalent Gardasil (Merck & Company, Inc., Whitehouse Station, New Jersey) (5–15), which constituted 93% of all HPV vaccine doses distributed in the United States through September 2015 (16), with no confirmed safety problems identified to date, other than syncope and skin infections (5). However, most published studies addressed prespecified outcomes about which theoretical or empirically based concerns had been raised, such as autoimmune diseases, venous thromboembolism, and neurological disease (6–15). More open-ended studies addressing HPV vaccine safety more generally, without prespecifying outcomes of concern, have been fewer and have been somewhat limited in sample size (5) or have tended to rely on spontaneous reports (17, 18), the interpretability of which is hampered by lack of control groups and denominators, underreporting, and reporting biases (19–21). We studied the safety of Gardasil, hereafter called quadrivalent human papillomavirus vaccine (4vHPV), applying a self-controlled tree-temporal scan statistic data-mining method (22–24) to health insurance claims data for signal detection. The method allows a wide variety of unsuspected but potential adverse reactions and a range of potential postvaccination periods of increased risk (“risk windows”) to be simultaneously evaluated, adjusting for the multiple testing involved. METHODS Study population, enrollment criteria, and exposure We conducted the study within the Sentinel system, which was launched by the US Food and Drug Administration in 2009 to conduct postlicensure assessments of the safety of medical products, using a distributed-data-processing approach (25–27). We used claims data from 5 Sentinel Data Partners, including 4 large national health insurance companies. People receiving their first dose of 4vHPV at 9–26.9 years of age within the period June 1, 2006–December 31, 2014 were eligible for inclusion in the study population. To be included, a health plan participant had to be enrolled from 183 days prior to the first 4vHPV dose through 56 days after the first 4vHPV dose. Any apparent enrollment gaps of 45 days or less were treated as continuously enrolled time. 4vHPV vaccination was identified using Current Procedural Terminology code 90649. Hierarchical diagnosis tree Outcomes were identified using International Classification of Diseases, Ninth Revision (ICD-9) codes and a classification of ICD-9 codes into a hierarchical tree structure defined by the Multi-Level Clinical Classifications Software (MLCCS). The MLCCS is a product of the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project (28). The tree has 5 diagnosis levels, although some branches extend only to the second or third level. The first and broadest level identifies 18 broad categories of diagnoses, while the subsequent levels become more and more specific, ending with ICD-9 codes at the finest, “leaf” level. Table 1 presents the classification of “convulsions” as an example. Table 1. Example of the Hierarchical Classification Scheme of the Agency for Healthcare Research and Quality’s Multi-Level Clinical Classifications Softwarea Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b “Convulsions” is a third-level classification without a fourth level. Table 1. Example of the Hierarchical Classification Scheme of the Agency for Healthcare Research and Quality’s Multi-Level Clinical Classifications Softwarea Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b “Convulsions” is a third-level classification without a fourth level. ICD-9 codes representing the following conditions were excluded from the tree and therefore from analysis: outcomes very unlikely to be caused by vaccination (e.g., well-care visits, delivery of a baby, vitamin deficiencies, fracture of a lower limb); some conditions unlikely to manifest themselves within the 56-day follow-up period (e.g., cancer); most infectious diseases with an identified organism (e.g., typhoid fever, tuberculosis, shigellosis); congenital conditions (e.g., sickle cell disease, congenital heart disease); and outcomes that are common and of an unspecific or less serious nature (e.g., fever, croup, acute pharyngitis). The resulting “pruned” tree contained 6,551 ICD-9 codes. Incident diagnoses of interest The study focused on “incident” diagnoses observed in the inpatient or emergency department setting during the 56-day follow-up period. In defining incidence, we sought to exclude repeated diagnoses due to follow-up visits for an earlier episode of illness; in so doing, however, we would not have captured closely spaced exacerbations of preexisting conditions, which are sometimes of interest as potential vaccine reactions. Incidence was defined and determined on the basis of there being no other diagnosis for the patient in the same third-level branch of the MLCCS diagnosis tree in any setting during the prior 183 days. This means that, even if an event was coded with a never-before-seen ICD-9 code, it was not counted if a different ICD-9 code belonging to the same third-level branch was observed for the individual during the prior 183 days. The third level was chosen for determining incidence in order to avoid double-counting and overestimation of incidence, which could otherwise occur if physicians classified the same episode of illness in 2 slightly different ways (e.g., “convulsions” and “febrile convulsions”) in separate patient visits. We allowed each patient to contribute multiple incident diagnoses during his/her follow-up period, as long as they were not part of the same third-level branch of the MLCCS tree. Risk and comparison windows Only health outcomes occurring on days 1–56 after the first apparent dose were included in the analysis. We considered this follow-up period optimal in that it would have included potential adverse reactions occurring (i.e., being diagnosed and coded) up to several weeks after vaccination, while minimizing any time-varying confounding and the likelihood of a second dose of HPV vaccine being received during follow-up. The day of vaccination (day 0) was not included, since 1) a preventive-care visit at which vaccines were given could have generated diagnosis codes for outcomes unrelated to vaccination, such as a health issue noted during the physical examination, and 2) 4vHPV may have been given during a health-care visit that happened due to an illness or other health concern. We evaluated all 665 temporal risk windows that were between 2 and 28 days long, started 1–28 days after vaccination, and ended 2–42 days after vaccination. The comparison period used in the evaluation of each of these potential risk windows consisted of the days within the 56-day follow-up period that were not in the risk window. For example, in evaluating the risk window of days 10–14, the control period was days 1–9 plus days 15–56. The conditional tree-temporal scan statistic With the tree-temporal scan statistic, one performs multiple temporal scans, one for each of the many clinical outcomes and groups of related clinical outcomes (i.e., leaves and branches of the tree). At the same time, one evaluates multiple potential risk windows, comparing the number of events within the risk window with what would be expected by chance if they were randomly distributed over time. Under the null hypothesis, there is no unusual clustering of events within any branch or time interval. Under the alternative hypothesis, there is at least 1 branch of the tree for which there is a temporal cluster of events during some time interval. In a conditional analysis, used for the current study, we condition not only on the number of events observed in each node of the tree during the whole follow-up period but also on the total number of events occurring on the first day after vaccination, on the second day after vaccination, etc. This adjusts for the type of temporal confounding that would occur if there were temporal differences in general health-care-seeking behavior shortly after the vaccination date as compared with longer after the vaccination date. The method adjusts for the multiple testing entailed in evaluating the many branches and time intervals. Each time interval is evaluated on each of the branches, so with our approximately 7,300 nodes (i.e., outcome categories, whether first, second, third, fourth, or fifth level—which include, for example, the codes listed in Table 1) on the tree and our 665 potential time intervals, there were more than 4.8 million potential clusters to evaluate and for which we needed to adjust for multiple testing. With scan statistics, the penalty for adjusting for all of this multiple testing is relatively modest, since many of the potential clusters are highly overlapping with each other. Furthermore, no power is lost (i.e., no α is spent) in scanning nodes where the observed number of events in the follow-up period is less than 2. To implement the conditional tree-temporal scan statistic, we calculate a Poisson generalized log likelihood ratio test statistic for each tree node and time interval. Let n be the number of events in the node, let c be the number of those node events that are also in the time interval, let z be the number of events in the time interval summed over the whole tree, and let C be the total number of events in the tree. The number of events in the cluster, c, is then contrasted with the expected number of events in the cluster under the null hypothesis, which is u = nz/C. When u > 0, the test statistic is calculated as   T={c×ln[c/u]}+{(C−c)×ln[(C−c)/(C−u)]×I(c>u)},where I ( ) is the indication function. I (c > u) = 1 when there are more events than expected in the cluster and 0 otherwise, and it is included to ensure that we are looking for an excess risk of having the outcome rather than a protective decreased risk. The node-interval combination with the maximum test statistic is the most likely cluster of events—that is, the cluster that is least likely to have occurred by chance. The distribution of the test statistic is not known analytically, so there is no simple mathematical formula that can be used to obtain a P value for the detected cluster. To adjust for the multiple testing inherent in the many node-interval combinations considered and evaluate whether the most likely cluster is statistically significant, Monte Carlo hypothesis testing is used. We do this by generating 99,999 random replicates of the data. In each random data set, each node has exactly the same number of events as the real data set, and each day after vaccination has the same number of events when summed over all nodes. The only thing that varies is the pairing of the nodes and times, which is randomized using a permutation approach. The likelihood ratio test statistic from the most likely cluster in the real data set is compared with the likelihood ratio test statistics from the most likely clusters in each of the 99,999 random data sets, and we note its rank. For example, if it has the fifth-highest test statistic, its rank, R, is 5. Note that the most likely cluster will be a different node-interval combination in each of the different data sets, so we are comparing the maxima of the likelihood ratios obtained over all possible node-interval combinations. Since the random data sets were all generated under the null hypothesis, if the null hypothesis is true in the real data set, then the test statistics come from exactly the same probability distribution. This means that, if the null hypothesis is true, the rank test statistic from the real data set will range uniformly from 1 to 100,000, and the probability of having a rank in the top 5% is exactly 5%. If the test statistic from the real data set is in the top 5%, we will reject the null hypothesis at the α = 0.05 level. If the null hypothesis is true, we have a 5% probability of falsely rejecting the null and a 95% probability of not having any alert anywhere on the tree. The analysis was conducted using the free software TreeScan, available at www.treescan.org (29). Calculation of excess risk To calculate excess risks (or attributable risks), we obtained D, the total number of eligible 4vHPV first doses. The attributable risk per 100,000 doses was calculated as (c − u) × 100,000/D. Statistical alert follow-up We use the term “alert” to refer to a cluster with P ≤ 0.05. Data related to alerts were frozen after the analysis was conducted. From the frozen data for 1 alert, a “claims profile” was generated for each patient with an incident diagnosis that contributed to the alert, listing all of the procedures, drug dispensings, and diagnoses captured in the claims data during the period from 56 days before 4vHPV vaccination through 84 days after 4vHPV vaccination. The procedures for generating these reports are explained in detail in the publicly available report Infrastructure for Evaluation of Statistical Alerts Arising From Vaccine Safety Data Mining Activities in Mini-Sentinel (30). Two members of the work group, including an internal medicine physician (M.A.B.), reviewed these claims profiles to see the specific diagnosis codes used, concomitant vaccinations, and other clinical information useful in interpreting the alert. RESULTS A total of 1,903,697 first doses of 4vHPV vaccine were included in analysis. The analysis results are presented in Table 2. All diagnoses with P < 0.05 are shown, along with some others useful for context. There were no diagnoses with P just slightly greater than 0.05—the lowest P value for diagnoses not included in the table was 0.25. There were 2 sets of alerts, described below. Table 2. Details of Statistical Alerts From a Tree-Temporal Scan Statistical Analysis of Quadrivalent Human Papillomavirus Vaccination Among Persons Aged 9–26.9 Years, United States, 2006–2014a Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  a Adapted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “2–4” means days 2–4 after vaccination). c Number of events observed in risk window. d Number of excess cases per 100,000 first vaccine doses. e Some related diagnoses for which there were no alerts are included for context. f All International Classification of Diseases, Ninth Revision, codes in node 12.01.01.03 (just 1 code) and node 16.10.02.07 for which there were any events during days 1–56 are listed. g Blank cells indicate that P = 1 for the respective diagnosis. Table 2. Details of Statistical Alerts From a Tree-Temporal Scan Statistical Analysis of Quadrivalent Human Papillomavirus Vaccination Among Persons Aged 9–26.9 Years, United States, 2006–2014a Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  a Adapted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “2–4” means days 2–4 after vaccination). c Number of events observed in risk window. d Number of excess cases per 100,000 first vaccine doses. e Some related diagnoses for which there were no alerts are included for context. f All International Classification of Diseases, Ninth Revision, codes in node 12.01.01.03 (just 1 code) and node 16.10.02.07 for which there were any events during days 1–56 are listed. g Blank cells indicate that P = 1 for the respective diagnosis. Cellulitis and abscess of arm (node 12.01.01.03) Within “diseases of the skin and subcutaneous tissue,” there were alerts at 4 levels (Table 2, rows 1, 2, 4, and 5). The highest statistical significance was seen at the fourth and fifth levels (rows 4 and 5), for “cellulitis and abscess of arm,” with a risk window of days 2–3 postvaccination, 31 cases, an attributable risk of 1.3 per 100,000 first doses administered, and a P value of 0.00001. (ICD-9 code 682.3 is the only one stemming from node 12.01.01.03, so the results are identical for rows 4 and 5.) Considering especially the statistical significance, these 31 cases appear to be driving the broader “diseases of the skin and subcutaneous tissue” alert (row 1). The 13 cases with ICD-9 code 695.9, “unspecified erythematous condition” (in row 7, within node 12.02, “other inflammatory condition of skin”), on days 2–3, contributed to the broader “diseases of the skin and subcutaneous tissue” alert, too, although there was no alert for the 695.9 code. Because cellulitis is a known adverse reaction to 4vHPV vaccination and is listed as such in the package insert (31), no further investigation was conducted. Other complications of surgical and medical procedures (node 16.10.02.07) There were alerts at 5 levels within “injury and poisoning,” with risk windows all within 6 days after vaccination (Table 2, rows 8–11, 16, 17, and 19). At the first level (row 8), there was a risk window of days 1–3 postvaccination, 48 cases, an attributable risk of 2.2 per 100,000 first doses, and a P value of 0.00001. The second through fourth levels had the same risk window (days 1–3), 36 cases, an attributable risk of 1.8 per 100,000 first doses, and a P value of 0.00001. (The results for rows 9–11 are identical, because there is no branching between nodes 16.10 and 16.10.02.07, due to branches such as “complication of device, implant, or graft” and “postoperative infection” in the full MLCCS tree having been excluded from the “pruned” tree that we used.) These 36 cases in “other complications of surgical and medical procedures” appear to be driving the broader “injury and poisoning” alert, as there are no alerts in other included “injury and poisoning” second-level branches, namely “poisoning” or “other injuries and conditions due to external causes” (not shown in table). There were 3 alerts for specific ICD-9 codes within “other complications of surgical and medical procedures” (rows 16, 17, and 19). Fifty-eight patients had incident diagnoses at the “other complications of surgical and medical procedures” node (Table 3)—36 with their diagnoses in the days 1–3 risk window (as also shown in Table 2, row 11) and 22 with their diagnoses during the days 4–56 control window (Table 3). Thirty-one (86%) of the 36 cases in the risk window and 11 (50%) of the 22 cases in the control window had a code indicating receipt of at least 1 other vaccine on the same day as 4vHPV. Concomitant vaccines included tetanus-diphtheria-acellular pertussis, meningococcal conjugate, varicella, pneumococcal conjugate, hepatitis A, hepatitis B, inactivated influenza, live attenuated influenza, rabies, typhoid, poliomyelitis, and meningococcal polysaccharide vaccines. Nine (25%) of the 36 cases in the risk window and 6 (27%) of the 22 cases outside of the risk window had a claim for a subsequent dose of 4vHPV vaccine within 84 days of the first. Table 3. Distribution of the 58 Adverse Event Cases in the Category “Other Complications of Surgical and Medical Procedures” Among Persons Aged 9–26.9 Years, by Diagnosis Code and Timing After Quadrivalent Human Papillomavirus Vaccination, United States, 2006–2014a ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  Abbreviation: ICD-9, International Classification of Diseases, Ninth Revision. a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “1–3” means days 1–3 after vaccination). Table 3. Distribution of the 58 Adverse Event Cases in the Category “Other Complications of Surgical and Medical Procedures” Among Persons Aged 9–26.9 Years, by Diagnosis Code and Timing After Quadrivalent Human Papillomavirus Vaccination, United States, 2006–2014a ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  Abbreviation: ICD-9, International Classification of Diseases, Ninth Revision. a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “1–3” means days 1–3 after vaccination). Focusing on the 36 cases whose incident diagnosis code fell into the days 1–3 risk window and which therefore contributed to the alert, there were 4, 3, and 2 cases of the specific incident diagnoses of postvaccination fever, generalized vaccinia, and anaphylaxis, respectively (Table 3). Although these 9 cases contributed to the “other complications of surgical and medical procedures” alert, none of these specific diagnoses was associated with an alert of its own (Table 2, rows 12–15). All 4 cases of postvaccination fever (ICD-9 code 780.63) had claims for 1 or more additional vaccines on day 0. Of the 3 cases with the generalized vaccinia ICD-9 code 999.0 as the incident diagnosis, 2 had codes for pain in or swelling of the limb and 1 had codes for allergic urticaria and unspecified urticaria; no additional diagnosis or symptom codes were present for any of the 3 cases. These 3 “vaccinia” patients had claims for at least 2 additional vaccines on day 0, including varicella in all 3 cases (relevant because generalized varicella-like rash, which could plausibly be incorrectly coded as “generalized vaccinia,” has been documented on days 0–23 after varicella dose 2 in adolescents and adults (32)). Regarding the 2 cases of anaphylaxis (ICD-9 code 999.42), it is unclear whether either case was truly anaphylaxis related to 4vHPV vaccination—neither case had claims for epinephrine, and 1 of the patients had received meningococcal conjugate vaccine on the same day as 4vHPV. There were 27 cases in the risk window with a nonspecific incident diagnosis code (ICD-9 codes 999.5, 999.52, 999.59, and 999.9; Table 3). From the patients’ claims profiles, we determined that, of these 27 cases, 8 had codes for pain in and/or swelling of the limb; 5 had codes for local skin reactions and/or unspecific allergic reactions; 1 had a code for cellulitis; and 6 had codes for somewhat diffuse conditions of nausea and/or vomiting, fever, viral exanthem, dizziness and giddiness, headache, and/or unspecified myalgia and myositis, with few or no subsequent medical visits apparent in the claims profile. An additional 3 cases had unspecified symptoms, and in all 3 cases, the next coded visit did not take place until at least 60 days later. The remaining 4 cases had claims for a variety of medical conditions, with no apparent similarity among them. DISCUSSION In our TreeScan analyses of more than 1.9 million recipients of 4vHPV dose 1, we found 2 categories of adverse events within 42 days of vaccination. One was “cellulitis and abscess of the arm.” Cellulitis is listed as an adverse event in the 4vHPV package insert (31) and was not investigated further. The other adverse event category was “other complications of surgical and medical procedures.” Based on the claims data, the clinical characteristics of 29 (81%) of the 36 cases contributing to that alert (the 9 cases with specific incident diagnoses and 20 of the cases with nonspecific incident diagnoses) appeared to conform to what was already known about 4vHPV adverse events (27) (although 4vHPV was not necessarily the cause—most of the 36 patients received 1 or more other vaccines along with 4vHPV). Three cases (8%) with nonspecific incident diagnosis codes and no symptoms specified in the claims profiles had no subsequent coded visits for medical care until at least 60 days later, suggesting that the respective conditions did not require medical follow-up. The other 4 cases (11%) had claims for diverse symptoms, different in each case and therefore not suggestive of a vaccine safety issue. Limited sample size is often an obstacle to the investigation of possible associations between vaccination and rare adverse events. This has been acknowledged in the case of 4vHPV and certain autoimmune disorders, for instance (8, 13). The current data-mining study, with its 1.9 million doses, had good statistical power—detecting, for example, an attributable risk of 6 excess cases per million first 4vHPV doses for “other serum reaction due to vaccination.” In view of the statistical power, the fact that only 2 categories of adverse events were found from more than 7,000 leaves and branches of the hierarchical tree, neither one of which was unexpected, provides reassurance about both the vaccine and the TreeScan conditional temporal-tree scan method. Three adverse events that have drawn the attention of public health authorities in relation to 4vHPV are complex regional pain syndrome (CRPS), postural orthostatic tachycardia syndrome (POTS), and Guillain-Barré syndrome (GBS). CRPS is a syndrome that affects 1 or more extremities and is characterized by persistent pain and swelling disproportionate to any known inciting event and at least 1 sign of autonomic dysfunction in the affected limb(s). In 2013, the Japanese Ministry of Health, Labour and Welfare suspended its recommendation of routine immunization with HPV vaccine for females after some postvaccination reports of serious chronic pain emerged (33). POTS is a heterogeneous and potentially debilitating autonomic disorder whose symptoms can include dizziness, nausea, fatigue, palpitations, weakness, sweating, and sleeping disorders. A case series of POTS occurring after 4vHPV vaccination in Denmark was described, raising concern about 4vHPV vaccine safety (34). In 2015, the European Medicines Agency’s Pharmacovigilance Risk Assessment Committee completed a detailed scientific review and concluded that the evidence did not support a causal link between HPV vaccines and CRPS or POTS (35), yet doubts and concerns persist (18, 36, 37). GBS is a disorder of the peripheral nervous system characterized by symmetrical ascending weakness and abnormal sensations over the course of hours, days, or weeks that can progress to temporary paralysis. In a recent large cohort study conducted in France, Miranda et al. (15) found an association between HPV (mostly 4vHPV) vaccination and GBS, with an adjusted hazard ratio of 3.78 (95% confidence interval: 1.79, 7.98) and an attributable risk of 1–2 excess cases per 100,000 girls vaccinated. They concluded that further studies were needed to confirm the finding. Our method, as applied, may not have been optimal for detecting increased risks of adverse events such as the above 3. One reason is that we considered only risk windows that began between 1 and 28 days postvaccination and ended between 2 and 42 days postvaccination. (While the tree-temporal scan statistic can, in principle, be used for longer follow-up periods, it is as yet untested for such applications.) Thus, we could detect only adverse reactions that manifested themselves within 6 weeks of vaccination, that is, outcomes of relatively acute onset. In the case of CRPS, the interval between the precipitating event and symptom onset can vary widely, with onset typically occurring within 6 months of the injury (38). Regarding GBS, the median time from HPV vaccination to GBS onset was 4.6 months in the Miranda et al. study (15). The second, potentially important limitation regarding syndromes such as CRPS and POTS is that the analysis was done using 1 particular hierarchical tree that included over 6,000 ICD-9 codes organized largely by functional “system” of the body (e.g., endocrine, nervous, circulatory, respiratory, digestive, and musculoskeletal). The ICD-9 and International Classification of Diseases, Tenth Revision, coding systems themselves are organized similarly, although not identically, to the MLCCS tree. However, conditions such as CRPS and POTS have symptoms that manifest in more than 1 functional system, potentially decreasing their detectability when TreeScan is paired with hierarchical structures organized in this way. Nonetheless, from a theoretical as well as an empirical standpoint, the method as we applied it here appears to be appropriate for assessing the many potential adverse reactions of acute onset whose symptoms are concentrated in 1 section of the MLCCS (and International Classification of Diseases) hierarchy. In conclusion, when tree-temporal scan statistics were applied to 1.9 million recipients of 4vHPV dose 1 in the Sentinel system, only 2 categories of adverse events within 42 days of vaccination emerged, and both were consistent with the known safety profile of 4vHPV. Considering the thousands of potential adverse events and hundreds of potential risk intervals evaluated and the good statistical power of this signal detection study, this finding represents a substantive and novel addition to the safety record of 4vHPV vaccine that is accumulating in the scientific literature. ACKNOWLEDGMENTS Author affiliations: Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts (W. Katherine Yih, Judith C. Maro, Meghan A. Baker, Carolyn Balsbaugh, David V. Cole, Inna Dashevsky); Office of Biostatistics and Epidemiology, Center for Biologics Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland (Michael Nguyen); Department of Epidemiology, Center for Biologics Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland (Adamma Mba-Jonas); and Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Harvard Medical School and Brigham and Women’s Hospital, Boston, Massachusetts (Martin Kulldorff). W.K.Y. and J.C.M. contributed equally to this paper and are co–primary authors. This work was funded by the Food and Drug Administration through Department of Health and Human Services Mini-Sentinel contract HHSF223200910006I. We thank Dr. Laura Polakowski for clinical consultation, Megan Reidy for research and administrative assistance, and the following Sentinel Data Partners: Aetna Inc. (Hartford, Connecticut); Harvard Pilgrim Health Care (Boston, Massachusetts); HealthCore (Wilmington, Delaware); Comprehensive Health Insights (Humana Inc., Louisville, Kentucky); and Optum Epidemiology (Eden Prairie, Minnesota). This work was presented at the 32nd International Conference on Pharmacoepidemiology and Therapeutic Risk Management, Dublin, Ireland, August 25–28, 2016. Conflict of interest: none declared. Abbreviations CRPS complex regional pain syndrome GBS Guillain-Barré syndrome HPV human papillomavirus ICD-9 International Classification of Diseases, Ninth Revision MLCCS Multi-Level Clinical Classifications Software POTS postural orthostatic tachycardia syndrome 4vHPV quadrivalent human papillomavirus vaccine REFERENCES 1 Reagan-Steiner S, Yankey D, Jeyarajah J, et al.  . National, regional, state, and selected local area vaccination coverage among adolescents aged 13–17 years—United States, 2015. MMWR Morb Mortal Wkly Rep . 2016; 65( 33): 850– 858. Google Scholar CrossRef Search ADS PubMed  2 Darden PM, Thompson DM, Roberts JR, et al.  . Reasons for not vaccinating adolescents: National Immunization Survey of Teens, 2008–2010. Pediatrics . 2013; 131( 4): 645– 651. Google Scholar CrossRef Search ADS PubMed  3 Stillo M, Carrillo Santisteve P, Lopalco PL. Safety of human papillomavirus vaccines: a review. Expert Opin Drug Saf . 2015; 14( 5): 697– 712. Google Scholar CrossRef Search ADS PubMed  4 Gee J, Weinbaum C, Sukumaran L, et al.  . Quadrivalent HPV vaccine safety review and safety monitoring plans for nine-valent HPV vaccine in the United States. Hum Vaccin Immunother . 2016; 12( 6): 1406– 1417. Google Scholar CrossRef Search ADS PubMed  5 Klein NP, Hansen J, Chao C, et al.  . Safety of quadrivalent human papillomavirus vaccine administered routinely to females. Arch Pediatr Adolesc Med . 2012; 166( 12): 1140– 1148. Google Scholar CrossRef Search ADS PubMed  6 Chao C, Klein NP, Velicer CM, et al.  . Surveillance of autoimmune conditions following routine use of quadrivalent human papillomavirus vaccine. J Intern Med . 2012; 271( 2): 193– 203. Google Scholar CrossRef Search ADS PubMed  7 Arnheim-Dahlström L, Pasternak B, Svanström H, et al.  . Autoimmune, neurological, and venous thromboembolic adverse events after immunisation of adolescent girls with quadrivalent human papillomavirus vaccine in Denmark and Sweden: cohort study. BMJ . 2013; 347: f5906. Google Scholar CrossRef Search ADS PubMed  8 Grimaldi-Bensouda L, Guillemot D, Godeau B, et al.  . Autoimmune disorders and quadrivalent human papillomavirus vaccination of young female subjects. J Intern Med . 2014; 275( 4): 398– 408. Google Scholar CrossRef Search ADS PubMed  9 Langer-Gould A, Qian L, Tartof SY, et al.  . Vaccines and the risk of multiple sclerosis and other central nervous system demyelinating diseases. JAMA Neurol . 2014; 71( 12): 1506– 1513. Google Scholar CrossRef Search ADS PubMed  10 Scheller NM, Svanström H, Pasternak B, et al.  . Quadrivalent HPV vaccination and risk of multiple sclerosis and other demyelinating diseases of the central nervous system. JAMA . 2015; 313( 1): 54– 61. Google Scholar CrossRef Search ADS PubMed  11 Naleway AL, Crane B, Smith N, et al.  . Absence of venous thromboembolism risk following quadrivalent human papillomavirus vaccination, Vaccine Safety Datalink, 2008–2011. Vaccine . 2016; 34( 1): 167– 171. Google Scholar CrossRef Search ADS PubMed  12 Yih WK, Greene SK, Zichittella L, et al.  . Evaluation of the risk of venous thromboembolism after quadrivalent human papillomavirus vaccination among US females. Vaccine . 2016; 34( 1): 172– 178. Google Scholar CrossRef Search ADS PubMed  13 Grimaldi-Bensouda L, Rossignol M, Koné-Paut I, et al.  . Risk of automimmune diseases and human papilloma virus (HPV) vaccines: six years of case-referent surveillance. J Autoimmun . 2017; 79: 84– 90. Google Scholar CrossRef Search ADS PubMed  14 Feiring B, Laake I, Bakken IJ, et al.  . HPV vaccination and risk of chronic fatigue syndrome/myalgic encephalomyelitis: a nationwide register-based study from Norway. Vaccine . 2017; 35( 33): 4203– 4212. Google Scholar CrossRef Search ADS PubMed  15 Miranda S, Chaignot C, Collin C, et al.  . Human papillomavirus vaccination and risk of autoimmune diseases: a large cohort study of over 2 million young girls in France. Vaccine . 2017; 35( 36): 4761– 4768. Google Scholar CrossRef Search ADS PubMed  16 Sukumaran L. Update on HPV vaccine safety. In: Advisory Committee on Immunization Practices (ACIP), Centers for Disease Control and Prevention, Department of Health and Human Services. Meeting of the Advisory Committee on Immunization Practices (ACIP). Summary Report. October 21, 2015. Atlanta, Georgia. Atlanta, GA: Centers for Disease Control and Prevention; 2015:69–73. https://www.cdc.gov/vaccines/acip/meetings/downloads/min-archive/min-2015-10.pdf. Accessed March 1, 2018. 17 Slade BA, Leidel L, Vellozzi C, et al.  . Postlicensure safety surveillance for quadrivalent human papillomavirus recombinant vaccine. JAMA . 2009; 302( 7): 750– 757. Google Scholar CrossRef Search ADS PubMed  18 Chandler RE, Juhlin K, Fransson J, et al.  . Current safety concerns with human papillomavirus vaccine: a cluster analysis of reports in Vigibase®. Drug Saf . 2017; 40( 1): 81– 90. Google Scholar CrossRef Search ADS PubMed  19 Shimabukuro TT, Nguyen M, Martin D, et al.  . Safety monitoring in the Vaccine Adverse Event Reporting System (VAERS). Vaccine . 2015; 33( 36): 4398– 4405. Google Scholar CrossRef Search ADS PubMed  20 Varricchio F, Iskander J, Destefano F, et al.  . Understanding vaccine safety information from the Vaccine Adverse Event Reporting System. Pediatr Infect Dis J . 2004; 23( 4): 287– 294. Google Scholar CrossRef Search ADS PubMed  21 Iskander JK, Miller ER, Chen RT. The role of the Vaccine Adverse Event Reporting System (VAERS) in monitoring vaccine safety. Pediatr Ann . 2004; 33( 9): 599– 606. Google Scholar CrossRef Search ADS PubMed  22 Brown JS, Petronis KR, Bate A, et al.  . Drug adverse event detection in health plan data using the Gamma Poisson Shrinker and comparison to the tree-based scan statistic. Pharmaceutics . 2013; 5( 1): 179– 200. Google Scholar CrossRef Search ADS PubMed  23 Kulldorff M, Dashevsky I, Avery TR, et al.  . Drug safety data mining with a tree-based scan statistic. Pharmacoepidemiol Drug Saf . 2013; 22( 5): 517– 523. Google Scholar CrossRef Search ADS PubMed  24 Kulldorff M, Fang Z, Walsh SJ. A tree-based scan statistic for database disease surveillance. Biometrics . 2003; 59( 2): 323– 331. Google Scholar CrossRef Search ADS PubMed  25 Curtis LH, Weiner MG, Boudreau DM, et al.  . Design considerations, architecture, and use of the Mini-Sentinel distributed data system. Pharmacoepidemiol Drug Saf . 2012; 21( suppl 1): 23– 31. Google Scholar CrossRef Search ADS PubMed  26 Nguyen M, Ball R, Midthun K, et al.  . The Food and Drug Administration’s Post-Licensure Rapid Immunization Safety Monitoring program: strengthening the federal vaccine safety enterprise. Pharmacoepidemiol Drug Saf . 2012; 21( suppl 1): 291– 297. Google Scholar CrossRef Search ADS PubMed  27 Ball R, Robb M, Anderson SA, et al.  . The FDA’s sentinel initiative—a comprehensive approach to medical product surveillance. Clin Pharmacol Ther . 2016; 99( 3): 265– 268. Google Scholar CrossRef Search ADS PubMed  28 Healthcare Cost and Utilization Project, Agency for Healthcare Research and Quality, US Department of Health and Human Services. Clinical Classifications Software (CCS) for ICD-9-CM. http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Updated March 6, 2017. Accessed June 6, 2017. 29 Kulldorff M; Information Management Services, Inc. TreeScan: software for the tree-based scan statistic. Published 2014. Accessed March 1, 2018. 30 Cole DV, Kulldorff M, Baker M, et al.  . Infrastructure for Evaluation of Statistical Alerts Arising from Vaccine Safety Data Mining Activities in Mini-Sentinel. Silver Spring, MD: Sentinel Initiative, Food and Drug Administration; 2016. https://www.sentinelinitiative.org/sites/default/files/Methods/Mini-Sentinel_PRISM_Data-Mining-Infrastructure_Report_0.pdf. Accessed November 14, 2017. 31 Merck & Company, Inc. GARDASIL® [Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) Vaccine, Recombinant]. (Gardasil package insert). Whitehouse Station, NJ: Merck & Company, Inc.; 2006. http://www.fda.gov/downloads/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM111263.pdf. Revised April 2015. Accessed June 6, 2017. 32 Merck & Company, Inc. Varivax®. Varicella Virus Vaccine Live. (Varivax package insert). Whitehouse Station, NJ: Merck & Company, Inc.; 2013. http://www.fda.gov/downloads/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM142812.pdf. Accessed March 1, 2018. 33 Kinoshita T, Abe RT, Hineno A, et al.  . Peripheral sympathetic nerve dysfunction in adolescent Japanese girls following immunization with the human papillomavirus vaccine. Intern Med . 2014; 53( 19): 2185– 2200. Google Scholar CrossRef Search ADS PubMed  34 Brinth LS, Pors K, Theibel AC, et al.  . Orthostatic intolerance and postural tachycardia syndrome as suspected adverse effects of vaccination against human papilloma virus. Vaccine . 2015; 33( 22): 2602– 2605. Google Scholar CrossRef Search ADS PubMed  35 Pharmacovigilance Risk Assessment Committee, European Medicines Agency. Assessment Report. Review Under Article 20 of Regulation (EC) No. 726/2004. Human Papillomavirus (HPV) Vaccines. London, United Kingdom: European Medicines Agency; 2015. (Publication EMA/762033/2015. http://www.ema.europa.eu/docs/en_GB/document_library/Referrals_document/HPV_vaccines_20/Opinion_provided_by_Committee_for_Medicinal_Products_for_Human_Use/WC500197129.pdf. Accessed September 25, 2017. 36 Jefferson T, Jorgensen L. Human papillomavirus vaccines, complex regional pain syndrome, postural orthostatic tachycardia syndrome, and autonomic dysfunction—a review of the regulatory evidence from the European Medicines Agency. Indian J Med Ethics . 2017; 2( 1): 30– 37. Google Scholar PubMed  37 Chandler RE. Safety concerns with HPV vaccines continue to linger: are current vaccine pharmacovigilance practices sufficient? Drug Saf . 2017; 40( 12): 1167– 1170. Google Scholar CrossRef Search ADS PubMed  38 WorkSafeBC Evidence-Based Practice Group, WorkSafeBC, Government of British Columbia. Complex Regional Pain Syndrome (CRPS). What Does the Literature Report Regarding the Time Interval Between the Inciting Trauma and Its Subsequent Diagnosis? Vancouver, British Columbia, Canada: WorkSafeBC; 2011. https://www.worksafebc.com/en/resources/health-care-providers/guides/complex-regional-pain-syndrome-crps-what-does-the-literature-report-regarding-the-time-interval-between-the-inciting-trauma-and-its-subsequent-diagnosis?lang=en. Accessed September 26, 2017. 39 Yih WK, Maro JC, Nguyen M, et al.  . Sentinel CBER/PRISM Methods. Pilot of Self-Controlled Tree-Temporal Scan Analysis for Gardasil Vaccine. Silver Spring, MD: Sentinel Initiative, Food and Drug Administration; 2016. https://www.sentinelinitiative.org/sites/default/files/Methods/Mini-Sentinel_PRISM_Pilot-Self-Controlled-Tree-Temporal-Scan-Analysis-Gardasil-Vaccine-Report.pdf. Accessed November 14, 2017. © The Author(s) 2018. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journalpermissions@oup.com. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png American Journal of Epidemiology Oxford University Press

Assessment of Quadrivalent Human Papillomavirus Vaccine Safety Using the Self-Controlled Tree-Temporal Scan Statistic Signal-Detection Method in the Sentinel System

Loading next page...
 
/lp/ou_press/assessment-of-quadrivalent-human-papillomavirus-vaccine-safety-using-8WbDBPUP5S
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health.
ISSN
0002-9262
eISSN
1476-6256
D.O.I.
10.1093/aje/kwy023
Publisher site
See Article on Publisher Site

Abstract

Abstract The self-controlled tree-temporal scan statistic—a new signal-detection method—can evaluate whether any of a wide variety of health outcomes are temporally associated with receipt of a specific vaccine, while adjusting for multiple testing. Neither health outcomes nor postvaccination potential periods of increased risk need be prespecified. Using US medical claims data in the Food and Drug Administration’s Sentinel system, we employed the method to evaluate adverse events occurring after receipt of quadrivalent human papillomavirus vaccine (4vHPV). Incident outcomes recorded in emergency department or inpatient settings within 56 days after first doses of 4vHPV received by 9- through 26.9-year-olds in 2006–2014 were identified using International Classification of Diseases, Ninth Revision, diagnosis codes and analyzed by pairing the new method with a standard hierarchical classification of diagnoses. On scanning diagnoses of 1.9 million 4vHPV recipients, 2 statistically significant categories of adverse events were found: cellulitis on days 2–3 after vaccination and “other complications of surgical and medical procedures” on days 1–3 after vaccination. Cellulitis is a known adverse event. Clinically informed investigation of electronic claims records of the patients with “other complications” did not suggest any previously unknown vaccine safety problem. Considering that thousands of potential short-term adverse events and hundreds of potential risk intervals were evaluated, these findings add significantly to the growing safety record of 4vHPV. data mining, epidemiologic research design, human papillomavirus recombinant vaccine quadrivalent, types 6, 11, 16, 18, papillomavirus vaccines, vaccination Editor’s note: An invited commentary on this article appears on page 1277, and the authors’ response appears on page 1281. Despite the cancer-preventing promise of human papillomavirus (HPV) vaccines and national recommendations for routine HPV vaccination of females and males at age 11–12 years, HPV vaccine coverage in the United States lags behind that of other adolescent vaccines (1) more than a decade after the first HPV vaccine was licensed. One reason for this is persistent concern about the safety of HPV vaccines on the part of parents and the public (2–4). The scientific literature generally does not bear out these worries. A substantial body of published evidence has accumulated regarding the safety of quadrivalent Gardasil (Merck & Company, Inc., Whitehouse Station, New Jersey) (5–15), which constituted 93% of all HPV vaccine doses distributed in the United States through September 2015 (16), with no confirmed safety problems identified to date, other than syncope and skin infections (5). However, most published studies addressed prespecified outcomes about which theoretical or empirically based concerns had been raised, such as autoimmune diseases, venous thromboembolism, and neurological disease (6–15). More open-ended studies addressing HPV vaccine safety more generally, without prespecifying outcomes of concern, have been fewer and have been somewhat limited in sample size (5) or have tended to rely on spontaneous reports (17, 18), the interpretability of which is hampered by lack of control groups and denominators, underreporting, and reporting biases (19–21). We studied the safety of Gardasil, hereafter called quadrivalent human papillomavirus vaccine (4vHPV), applying a self-controlled tree-temporal scan statistic data-mining method (22–24) to health insurance claims data for signal detection. The method allows a wide variety of unsuspected but potential adverse reactions and a range of potential postvaccination periods of increased risk (“risk windows”) to be simultaneously evaluated, adjusting for the multiple testing involved. METHODS Study population, enrollment criteria, and exposure We conducted the study within the Sentinel system, which was launched by the US Food and Drug Administration in 2009 to conduct postlicensure assessments of the safety of medical products, using a distributed-data-processing approach (25–27). We used claims data from 5 Sentinel Data Partners, including 4 large national health insurance companies. People receiving their first dose of 4vHPV at 9–26.9 years of age within the period June 1, 2006–December 31, 2014 were eligible for inclusion in the study population. To be included, a health plan participant had to be enrolled from 183 days prior to the first 4vHPV dose through 56 days after the first 4vHPV dose. Any apparent enrollment gaps of 45 days or less were treated as continuously enrolled time. 4vHPV vaccination was identified using Current Procedural Terminology code 90649. Hierarchical diagnosis tree Outcomes were identified using International Classification of Diseases, Ninth Revision (ICD-9) codes and a classification of ICD-9 codes into a hierarchical tree structure defined by the Multi-Level Clinical Classifications Software (MLCCS). The MLCCS is a product of the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project (28). The tree has 5 diagnosis levels, although some branches extend only to the second or third level. The first and broadest level identifies 18 broad categories of diagnoses, while the subsequent levels become more and more specific, ending with ICD-9 codes at the finest, “leaf” level. Table 1 presents the classification of “convulsions” as an example. Table 1. Example of the Hierarchical Classification Scheme of the Agency for Healthcare Research and Quality’s Multi-Level Clinical Classifications Softwarea Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b “Convulsions” is a third-level classification without a fourth level. Table 1. Example of the Hierarchical Classification Scheme of the Agency for Healthcare Research and Quality’s Multi-Level Clinical Classifications Softwarea Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  Code  Text Description  06  Diseases of the nervous system and sense organs  06.04  Epilepsy; convulsions  06.04.02  Convulsionsb   780.3   Convulsions   780.31   Febrile convulsions   780.32   Complex febrile convulsions   780.33   Posttraumatic seizures   780.39   Other convulsions  a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b “Convulsions” is a third-level classification without a fourth level. ICD-9 codes representing the following conditions were excluded from the tree and therefore from analysis: outcomes very unlikely to be caused by vaccination (e.g., well-care visits, delivery of a baby, vitamin deficiencies, fracture of a lower limb); some conditions unlikely to manifest themselves within the 56-day follow-up period (e.g., cancer); most infectious diseases with an identified organism (e.g., typhoid fever, tuberculosis, shigellosis); congenital conditions (e.g., sickle cell disease, congenital heart disease); and outcomes that are common and of an unspecific or less serious nature (e.g., fever, croup, acute pharyngitis). The resulting “pruned” tree contained 6,551 ICD-9 codes. Incident diagnoses of interest The study focused on “incident” diagnoses observed in the inpatient or emergency department setting during the 56-day follow-up period. In defining incidence, we sought to exclude repeated diagnoses due to follow-up visits for an earlier episode of illness; in so doing, however, we would not have captured closely spaced exacerbations of preexisting conditions, which are sometimes of interest as potential vaccine reactions. Incidence was defined and determined on the basis of there being no other diagnosis for the patient in the same third-level branch of the MLCCS diagnosis tree in any setting during the prior 183 days. This means that, even if an event was coded with a never-before-seen ICD-9 code, it was not counted if a different ICD-9 code belonging to the same third-level branch was observed for the individual during the prior 183 days. The third level was chosen for determining incidence in order to avoid double-counting and overestimation of incidence, which could otherwise occur if physicians classified the same episode of illness in 2 slightly different ways (e.g., “convulsions” and “febrile convulsions”) in separate patient visits. We allowed each patient to contribute multiple incident diagnoses during his/her follow-up period, as long as they were not part of the same third-level branch of the MLCCS tree. Risk and comparison windows Only health outcomes occurring on days 1–56 after the first apparent dose were included in the analysis. We considered this follow-up period optimal in that it would have included potential adverse reactions occurring (i.e., being diagnosed and coded) up to several weeks after vaccination, while minimizing any time-varying confounding and the likelihood of a second dose of HPV vaccine being received during follow-up. The day of vaccination (day 0) was not included, since 1) a preventive-care visit at which vaccines were given could have generated diagnosis codes for outcomes unrelated to vaccination, such as a health issue noted during the physical examination, and 2) 4vHPV may have been given during a health-care visit that happened due to an illness or other health concern. We evaluated all 665 temporal risk windows that were between 2 and 28 days long, started 1–28 days after vaccination, and ended 2–42 days after vaccination. The comparison period used in the evaluation of each of these potential risk windows consisted of the days within the 56-day follow-up period that were not in the risk window. For example, in evaluating the risk window of days 10–14, the control period was days 1–9 plus days 15–56. The conditional tree-temporal scan statistic With the tree-temporal scan statistic, one performs multiple temporal scans, one for each of the many clinical outcomes and groups of related clinical outcomes (i.e., leaves and branches of the tree). At the same time, one evaluates multiple potential risk windows, comparing the number of events within the risk window with what would be expected by chance if they were randomly distributed over time. Under the null hypothesis, there is no unusual clustering of events within any branch or time interval. Under the alternative hypothesis, there is at least 1 branch of the tree for which there is a temporal cluster of events during some time interval. In a conditional analysis, used for the current study, we condition not only on the number of events observed in each node of the tree during the whole follow-up period but also on the total number of events occurring on the first day after vaccination, on the second day after vaccination, etc. This adjusts for the type of temporal confounding that would occur if there were temporal differences in general health-care-seeking behavior shortly after the vaccination date as compared with longer after the vaccination date. The method adjusts for the multiple testing entailed in evaluating the many branches and time intervals. Each time interval is evaluated on each of the branches, so with our approximately 7,300 nodes (i.e., outcome categories, whether first, second, third, fourth, or fifth level—which include, for example, the codes listed in Table 1) on the tree and our 665 potential time intervals, there were more than 4.8 million potential clusters to evaluate and for which we needed to adjust for multiple testing. With scan statistics, the penalty for adjusting for all of this multiple testing is relatively modest, since many of the potential clusters are highly overlapping with each other. Furthermore, no power is lost (i.e., no α is spent) in scanning nodes where the observed number of events in the follow-up period is less than 2. To implement the conditional tree-temporal scan statistic, we calculate a Poisson generalized log likelihood ratio test statistic for each tree node and time interval. Let n be the number of events in the node, let c be the number of those node events that are also in the time interval, let z be the number of events in the time interval summed over the whole tree, and let C be the total number of events in the tree. The number of events in the cluster, c, is then contrasted with the expected number of events in the cluster under the null hypothesis, which is u = nz/C. When u > 0, the test statistic is calculated as   T={c×ln[c/u]}+{(C−c)×ln[(C−c)/(C−u)]×I(c>u)},where I ( ) is the indication function. I (c > u) = 1 when there are more events than expected in the cluster and 0 otherwise, and it is included to ensure that we are looking for an excess risk of having the outcome rather than a protective decreased risk. The node-interval combination with the maximum test statistic is the most likely cluster of events—that is, the cluster that is least likely to have occurred by chance. The distribution of the test statistic is not known analytically, so there is no simple mathematical formula that can be used to obtain a P value for the detected cluster. To adjust for the multiple testing inherent in the many node-interval combinations considered and evaluate whether the most likely cluster is statistically significant, Monte Carlo hypothesis testing is used. We do this by generating 99,999 random replicates of the data. In each random data set, each node has exactly the same number of events as the real data set, and each day after vaccination has the same number of events when summed over all nodes. The only thing that varies is the pairing of the nodes and times, which is randomized using a permutation approach. The likelihood ratio test statistic from the most likely cluster in the real data set is compared with the likelihood ratio test statistics from the most likely clusters in each of the 99,999 random data sets, and we note its rank. For example, if it has the fifth-highest test statistic, its rank, R, is 5. Note that the most likely cluster will be a different node-interval combination in each of the different data sets, so we are comparing the maxima of the likelihood ratios obtained over all possible node-interval combinations. Since the random data sets were all generated under the null hypothesis, if the null hypothesis is true in the real data set, then the test statistics come from exactly the same probability distribution. This means that, if the null hypothesis is true, the rank test statistic from the real data set will range uniformly from 1 to 100,000, and the probability of having a rank in the top 5% is exactly 5%. If the test statistic from the real data set is in the top 5%, we will reject the null hypothesis at the α = 0.05 level. If the null hypothesis is true, we have a 5% probability of falsely rejecting the null and a 95% probability of not having any alert anywhere on the tree. The analysis was conducted using the free software TreeScan, available at www.treescan.org (29). Calculation of excess risk To calculate excess risks (or attributable risks), we obtained D, the total number of eligible 4vHPV first doses. The attributable risk per 100,000 doses was calculated as (c − u) × 100,000/D. Statistical alert follow-up We use the term “alert” to refer to a cluster with P ≤ 0.05. Data related to alerts were frozen after the analysis was conducted. From the frozen data for 1 alert, a “claims profile” was generated for each patient with an incident diagnosis that contributed to the alert, listing all of the procedures, drug dispensings, and diagnoses captured in the claims data during the period from 56 days before 4vHPV vaccination through 84 days after 4vHPV vaccination. The procedures for generating these reports are explained in detail in the publicly available report Infrastructure for Evaluation of Statistical Alerts Arising From Vaccine Safety Data Mining Activities in Mini-Sentinel (30). Two members of the work group, including an internal medicine physician (M.A.B.), reviewed these claims profiles to see the specific diagnosis codes used, concomitant vaccinations, and other clinical information useful in interpreting the alert. RESULTS A total of 1,903,697 first doses of 4vHPV vaccine were included in analysis. The analysis results are presented in Table 2. All diagnoses with P < 0.05 are shown, along with some others useful for context. There were no diagnoses with P just slightly greater than 0.05—the lowest P value for diagnoses not included in the table was 0.25. There were 2 sets of alerts, described below. Table 2. Details of Statistical Alerts From a Tree-Temporal Scan Statistical Analysis of Quadrivalent Human Papillomavirus Vaccination Among Persons Aged 9–26.9 Years, United States, 2006–2014a Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  a Adapted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “2–4” means days 2–4 after vaccination). c Number of events observed in risk window. d Number of excess cases per 100,000 first vaccine doses. e Some related diagnoses for which there were no alerts are included for context. f All International Classification of Diseases, Ninth Revision, codes in node 12.01.01.03 (just 1 code) and node 16.10.02.07 for which there were any events during days 1–56 are listed. g Blank cells indicate that P = 1 for the respective diagnosis. Table 2. Details of Statistical Alerts From a Tree-Temporal Scan Statistical Analysis of Quadrivalent Human Papillomavirus Vaccination Among Persons Aged 9–26.9 Years, United States, 2006–2014a Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  Row  Node Code  Node Text  Risk Windowb  Observed No. of Eventsc  Attributable Riskd  P Value  1  12  Diseases of the skin and subcutaneous tissue  2–4  214  3.8  0.002  2  12.01  Skin and subcutaneous tissue infections  2–4  111  2.3  0.042  3  12.01.01  Cellulitis and abscesse  2–4  93  2.0  0.204  4  12.01.01.03  Cellulitis and abscess of arm (contains code 682.3 only)f  2–3  31  1.3  0.00001  5   682.3   Cellulitis and abscess of upper arm and forearm  2–3  31  1.3  0.00001  6  12.02  Other inflammatory condition of skine,g          7   695.9   Unspecified erythematous conditione  2–3  13  0.5  0.246  8  16  Injury and poisoning  1–3  48  2.2  0.00001  9  16.10  Complications  1–3  36  1.8  0.00001  10  16.10.02  Complications of surgical procedures or medical care  1–3  36  1.8  0.00001  11  16.10.02.07  Other complications of surgical and medical proceduresf  1–3  36  1.8  0.00001  12   780.63   Postvaccination fevere  1–2  4  0.2  0.306  13   999.0   Generalized vacciniae  1–3  3  0.2  >0.99  14   999.4   Anaphylactic reaction due to serume,g          15   999.42   Anaphylactic reaction due to vaccinatione,g          16   999.5   Other serum reaction not elsewhere classified  1–3  7  0.4  0.011  17   999.52   Other serum reaction due to vaccination  1–2  11  0.6  0.00001  18   999.59   Other serum reactione,g          19   999.9   Other and unspecified complications of medical care  1–6  12  0.6  0.002  a Adapted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “2–4” means days 2–4 after vaccination). c Number of events observed in risk window. d Number of excess cases per 100,000 first vaccine doses. e Some related diagnoses for which there were no alerts are included for context. f All International Classification of Diseases, Ninth Revision, codes in node 12.01.01.03 (just 1 code) and node 16.10.02.07 for which there were any events during days 1–56 are listed. g Blank cells indicate that P = 1 for the respective diagnosis. Cellulitis and abscess of arm (node 12.01.01.03) Within “diseases of the skin and subcutaneous tissue,” there were alerts at 4 levels (Table 2, rows 1, 2, 4, and 5). The highest statistical significance was seen at the fourth and fifth levels (rows 4 and 5), for “cellulitis and abscess of arm,” with a risk window of days 2–3 postvaccination, 31 cases, an attributable risk of 1.3 per 100,000 first doses administered, and a P value of 0.00001. (ICD-9 code 682.3 is the only one stemming from node 12.01.01.03, so the results are identical for rows 4 and 5.) Considering especially the statistical significance, these 31 cases appear to be driving the broader “diseases of the skin and subcutaneous tissue” alert (row 1). The 13 cases with ICD-9 code 695.9, “unspecified erythematous condition” (in row 7, within node 12.02, “other inflammatory condition of skin”), on days 2–3, contributed to the broader “diseases of the skin and subcutaneous tissue” alert, too, although there was no alert for the 695.9 code. Because cellulitis is a known adverse reaction to 4vHPV vaccination and is listed as such in the package insert (31), no further investigation was conducted. Other complications of surgical and medical procedures (node 16.10.02.07) There were alerts at 5 levels within “injury and poisoning,” with risk windows all within 6 days after vaccination (Table 2, rows 8–11, 16, 17, and 19). At the first level (row 8), there was a risk window of days 1–3 postvaccination, 48 cases, an attributable risk of 2.2 per 100,000 first doses, and a P value of 0.00001. The second through fourth levels had the same risk window (days 1–3), 36 cases, an attributable risk of 1.8 per 100,000 first doses, and a P value of 0.00001. (The results for rows 9–11 are identical, because there is no branching between nodes 16.10 and 16.10.02.07, due to branches such as “complication of device, implant, or graft” and “postoperative infection” in the full MLCCS tree having been excluded from the “pruned” tree that we used.) These 36 cases in “other complications of surgical and medical procedures” appear to be driving the broader “injury and poisoning” alert, as there are no alerts in other included “injury and poisoning” second-level branches, namely “poisoning” or “other injuries and conditions due to external causes” (not shown in table). There were 3 alerts for specific ICD-9 codes within “other complications of surgical and medical procedures” (rows 16, 17, and 19). Fifty-eight patients had incident diagnoses at the “other complications of surgical and medical procedures” node (Table 3)—36 with their diagnoses in the days 1–3 risk window (as also shown in Table 2, row 11) and 22 with their diagnoses during the days 4–56 control window (Table 3). Thirty-one (86%) of the 36 cases in the risk window and 11 (50%) of the 22 cases in the control window had a code indicating receipt of at least 1 other vaccine on the same day as 4vHPV. Concomitant vaccines included tetanus-diphtheria-acellular pertussis, meningococcal conjugate, varicella, pneumococcal conjugate, hepatitis A, hepatitis B, inactivated influenza, live attenuated influenza, rabies, typhoid, poliomyelitis, and meningococcal polysaccharide vaccines. Nine (25%) of the 36 cases in the risk window and 6 (27%) of the 22 cases outside of the risk window had a claim for a subsequent dose of 4vHPV vaccine within 84 days of the first. Table 3. Distribution of the 58 Adverse Event Cases in the Category “Other Complications of Surgical and Medical Procedures” Among Persons Aged 9–26.9 Years, by Diagnosis Code and Timing After Quadrivalent Human Papillomavirus Vaccination, United States, 2006–2014a ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  Abbreviation: ICD-9, International Classification of Diseases, Ninth Revision. a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “1–3” means days 1–3 after vaccination). Table 3. Distribution of the 58 Adverse Event Cases in the Category “Other Complications of Surgical and Medical Procedures” Among Persons Aged 9–26.9 Years, by Diagnosis Code and Timing After Quadrivalent Human Papillomavirus Vaccination, United States, 2006–2014a ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  ICD-9 Code  ICD-9 Description  Time Windowb  Days 1–3  Days 4–56  Total  780.63  Postvaccination fever  4  1  5  999.0  Generalized vaccinia  3  0  3  999.4  Anaphylactic shock due to serum  0  1  1  999.42  Anaphylactic reaction due to vaccination  2  3  5  999.5  Serum reaction not elsewhere classified  7  3  10  999.52  Other serum reaction due to vaccination  11  2  13  999.59  Other serum reaction  2  3  5  999.9  Other and unspecified complications of medical care, not elsewhere classified  7  9  16  Total    36  22  58  Abbreviation: ICD-9, International Classification of Diseases, Ninth Revision. a Reprinted from the final project report to the Food and Drug Administration (39), with permission from Harvard Pilgrim Health Care, Inc. (©2016). b Specific days after vaccination (e.g., “1–3” means days 1–3 after vaccination). Focusing on the 36 cases whose incident diagnosis code fell into the days 1–3 risk window and which therefore contributed to the alert, there were 4, 3, and 2 cases of the specific incident diagnoses of postvaccination fever, generalized vaccinia, and anaphylaxis, respectively (Table 3). Although these 9 cases contributed to the “other complications of surgical and medical procedures” alert, none of these specific diagnoses was associated with an alert of its own (Table 2, rows 12–15). All 4 cases of postvaccination fever (ICD-9 code 780.63) had claims for 1 or more additional vaccines on day 0. Of the 3 cases with the generalized vaccinia ICD-9 code 999.0 as the incident diagnosis, 2 had codes for pain in or swelling of the limb and 1 had codes for allergic urticaria and unspecified urticaria; no additional diagnosis or symptom codes were present for any of the 3 cases. These 3 “vaccinia” patients had claims for at least 2 additional vaccines on day 0, including varicella in all 3 cases (relevant because generalized varicella-like rash, which could plausibly be incorrectly coded as “generalized vaccinia,” has been documented on days 0–23 after varicella dose 2 in adolescents and adults (32)). Regarding the 2 cases of anaphylaxis (ICD-9 code 999.42), it is unclear whether either case was truly anaphylaxis related to 4vHPV vaccination—neither case had claims for epinephrine, and 1 of the patients had received meningococcal conjugate vaccine on the same day as 4vHPV. There were 27 cases in the risk window with a nonspecific incident diagnosis code (ICD-9 codes 999.5, 999.52, 999.59, and 999.9; Table 3). From the patients’ claims profiles, we determined that, of these 27 cases, 8 had codes for pain in and/or swelling of the limb; 5 had codes for local skin reactions and/or unspecific allergic reactions; 1 had a code for cellulitis; and 6 had codes for somewhat diffuse conditions of nausea and/or vomiting, fever, viral exanthem, dizziness and giddiness, headache, and/or unspecified myalgia and myositis, with few or no subsequent medical visits apparent in the claims profile. An additional 3 cases had unspecified symptoms, and in all 3 cases, the next coded visit did not take place until at least 60 days later. The remaining 4 cases had claims for a variety of medical conditions, with no apparent similarity among them. DISCUSSION In our TreeScan analyses of more than 1.9 million recipients of 4vHPV dose 1, we found 2 categories of adverse events within 42 days of vaccination. One was “cellulitis and abscess of the arm.” Cellulitis is listed as an adverse event in the 4vHPV package insert (31) and was not investigated further. The other adverse event category was “other complications of surgical and medical procedures.” Based on the claims data, the clinical characteristics of 29 (81%) of the 36 cases contributing to that alert (the 9 cases with specific incident diagnoses and 20 of the cases with nonspecific incident diagnoses) appeared to conform to what was already known about 4vHPV adverse events (27) (although 4vHPV was not necessarily the cause—most of the 36 patients received 1 or more other vaccines along with 4vHPV). Three cases (8%) with nonspecific incident diagnosis codes and no symptoms specified in the claims profiles had no subsequent coded visits for medical care until at least 60 days later, suggesting that the respective conditions did not require medical follow-up. The other 4 cases (11%) had claims for diverse symptoms, different in each case and therefore not suggestive of a vaccine safety issue. Limited sample size is often an obstacle to the investigation of possible associations between vaccination and rare adverse events. This has been acknowledged in the case of 4vHPV and certain autoimmune disorders, for instance (8, 13). The current data-mining study, with its 1.9 million doses, had good statistical power—detecting, for example, an attributable risk of 6 excess cases per million first 4vHPV doses for “other serum reaction due to vaccination.” In view of the statistical power, the fact that only 2 categories of adverse events were found from more than 7,000 leaves and branches of the hierarchical tree, neither one of which was unexpected, provides reassurance about both the vaccine and the TreeScan conditional temporal-tree scan method. Three adverse events that have drawn the attention of public health authorities in relation to 4vHPV are complex regional pain syndrome (CRPS), postural orthostatic tachycardia syndrome (POTS), and Guillain-Barré syndrome (GBS). CRPS is a syndrome that affects 1 or more extremities and is characterized by persistent pain and swelling disproportionate to any known inciting event and at least 1 sign of autonomic dysfunction in the affected limb(s). In 2013, the Japanese Ministry of Health, Labour and Welfare suspended its recommendation of routine immunization with HPV vaccine for females after some postvaccination reports of serious chronic pain emerged (33). POTS is a heterogeneous and potentially debilitating autonomic disorder whose symptoms can include dizziness, nausea, fatigue, palpitations, weakness, sweating, and sleeping disorders. A case series of POTS occurring after 4vHPV vaccination in Denmark was described, raising concern about 4vHPV vaccine safety (34). In 2015, the European Medicines Agency’s Pharmacovigilance Risk Assessment Committee completed a detailed scientific review and concluded that the evidence did not support a causal link between HPV vaccines and CRPS or POTS (35), yet doubts and concerns persist (18, 36, 37). GBS is a disorder of the peripheral nervous system characterized by symmetrical ascending weakness and abnormal sensations over the course of hours, days, or weeks that can progress to temporary paralysis. In a recent large cohort study conducted in France, Miranda et al. (15) found an association between HPV (mostly 4vHPV) vaccination and GBS, with an adjusted hazard ratio of 3.78 (95% confidence interval: 1.79, 7.98) and an attributable risk of 1–2 excess cases per 100,000 girls vaccinated. They concluded that further studies were needed to confirm the finding. Our method, as applied, may not have been optimal for detecting increased risks of adverse events such as the above 3. One reason is that we considered only risk windows that began between 1 and 28 days postvaccination and ended between 2 and 42 days postvaccination. (While the tree-temporal scan statistic can, in principle, be used for longer follow-up periods, it is as yet untested for such applications.) Thus, we could detect only adverse reactions that manifested themselves within 6 weeks of vaccination, that is, outcomes of relatively acute onset. In the case of CRPS, the interval between the precipitating event and symptom onset can vary widely, with onset typically occurring within 6 months of the injury (38). Regarding GBS, the median time from HPV vaccination to GBS onset was 4.6 months in the Miranda et al. study (15). The second, potentially important limitation regarding syndromes such as CRPS and POTS is that the analysis was done using 1 particular hierarchical tree that included over 6,000 ICD-9 codes organized largely by functional “system” of the body (e.g., endocrine, nervous, circulatory, respiratory, digestive, and musculoskeletal). The ICD-9 and International Classification of Diseases, Tenth Revision, coding systems themselves are organized similarly, although not identically, to the MLCCS tree. However, conditions such as CRPS and POTS have symptoms that manifest in more than 1 functional system, potentially decreasing their detectability when TreeScan is paired with hierarchical structures organized in this way. Nonetheless, from a theoretical as well as an empirical standpoint, the method as we applied it here appears to be appropriate for assessing the many potential adverse reactions of acute onset whose symptoms are concentrated in 1 section of the MLCCS (and International Classification of Diseases) hierarchy. In conclusion, when tree-temporal scan statistics were applied to 1.9 million recipients of 4vHPV dose 1 in the Sentinel system, only 2 categories of adverse events within 42 days of vaccination emerged, and both were consistent with the known safety profile of 4vHPV. Considering the thousands of potential adverse events and hundreds of potential risk intervals evaluated and the good statistical power of this signal detection study, this finding represents a substantive and novel addition to the safety record of 4vHPV vaccine that is accumulating in the scientific literature. ACKNOWLEDGMENTS Author affiliations: Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts (W. Katherine Yih, Judith C. Maro, Meghan A. Baker, Carolyn Balsbaugh, David V. Cole, Inna Dashevsky); Office of Biostatistics and Epidemiology, Center for Biologics Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland (Michael Nguyen); Department of Epidemiology, Center for Biologics Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland (Adamma Mba-Jonas); and Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Harvard Medical School and Brigham and Women’s Hospital, Boston, Massachusetts (Martin Kulldorff). W.K.Y. and J.C.M. contributed equally to this paper and are co–primary authors. This work was funded by the Food and Drug Administration through Department of Health and Human Services Mini-Sentinel contract HHSF223200910006I. We thank Dr. Laura Polakowski for clinical consultation, Megan Reidy for research and administrative assistance, and the following Sentinel Data Partners: Aetna Inc. (Hartford, Connecticut); Harvard Pilgrim Health Care (Boston, Massachusetts); HealthCore (Wilmington, Delaware); Comprehensive Health Insights (Humana Inc., Louisville, Kentucky); and Optum Epidemiology (Eden Prairie, Minnesota). This work was presented at the 32nd International Conference on Pharmacoepidemiology and Therapeutic Risk Management, Dublin, Ireland, August 25–28, 2016. Conflict of interest: none declared. Abbreviations CRPS complex regional pain syndrome GBS Guillain-Barré syndrome HPV human papillomavirus ICD-9 International Classification of Diseases, Ninth Revision MLCCS Multi-Level Clinical Classifications Software POTS postural orthostatic tachycardia syndrome 4vHPV quadrivalent human papillomavirus vaccine REFERENCES 1 Reagan-Steiner S, Yankey D, Jeyarajah J, et al.  . National, regional, state, and selected local area vaccination coverage among adolescents aged 13–17 years—United States, 2015. MMWR Morb Mortal Wkly Rep . 2016; 65( 33): 850– 858. Google Scholar CrossRef Search ADS PubMed  2 Darden PM, Thompson DM, Roberts JR, et al.  . Reasons for not vaccinating adolescents: National Immunization Survey of Teens, 2008–2010. Pediatrics . 2013; 131( 4): 645– 651. Google Scholar CrossRef Search ADS PubMed  3 Stillo M, Carrillo Santisteve P, Lopalco PL. Safety of human papillomavirus vaccines: a review. Expert Opin Drug Saf . 2015; 14( 5): 697– 712. Google Scholar CrossRef Search ADS PubMed  4 Gee J, Weinbaum C, Sukumaran L, et al.  . Quadrivalent HPV vaccine safety review and safety monitoring plans for nine-valent HPV vaccine in the United States. Hum Vaccin Immunother . 2016; 12( 6): 1406– 1417. Google Scholar CrossRef Search ADS PubMed  5 Klein NP, Hansen J, Chao C, et al.  . Safety of quadrivalent human papillomavirus vaccine administered routinely to females. Arch Pediatr Adolesc Med . 2012; 166( 12): 1140– 1148. Google Scholar CrossRef Search ADS PubMed  6 Chao C, Klein NP, Velicer CM, et al.  . Surveillance of autoimmune conditions following routine use of quadrivalent human papillomavirus vaccine. J Intern Med . 2012; 271( 2): 193– 203. Google Scholar CrossRef Search ADS PubMed  7 Arnheim-Dahlström L, Pasternak B, Svanström H, et al.  . Autoimmune, neurological, and venous thromboembolic adverse events after immunisation of adolescent girls with quadrivalent human papillomavirus vaccine in Denmark and Sweden: cohort study. BMJ . 2013; 347: f5906. Google Scholar CrossRef Search ADS PubMed  8 Grimaldi-Bensouda L, Guillemot D, Godeau B, et al.  . Autoimmune disorders and quadrivalent human papillomavirus vaccination of young female subjects. J Intern Med . 2014; 275( 4): 398– 408. Google Scholar CrossRef Search ADS PubMed  9 Langer-Gould A, Qian L, Tartof SY, et al.  . Vaccines and the risk of multiple sclerosis and other central nervous system demyelinating diseases. JAMA Neurol . 2014; 71( 12): 1506– 1513. Google Scholar CrossRef Search ADS PubMed  10 Scheller NM, Svanström H, Pasternak B, et al.  . Quadrivalent HPV vaccination and risk of multiple sclerosis and other demyelinating diseases of the central nervous system. JAMA . 2015; 313( 1): 54– 61. Google Scholar CrossRef Search ADS PubMed  11 Naleway AL, Crane B, Smith N, et al.  . Absence of venous thromboembolism risk following quadrivalent human papillomavirus vaccination, Vaccine Safety Datalink, 2008–2011. Vaccine . 2016; 34( 1): 167– 171. Google Scholar CrossRef Search ADS PubMed  12 Yih WK, Greene SK, Zichittella L, et al.  . Evaluation of the risk of venous thromboembolism after quadrivalent human papillomavirus vaccination among US females. Vaccine . 2016; 34( 1): 172– 178. Google Scholar CrossRef Search ADS PubMed  13 Grimaldi-Bensouda L, Rossignol M, Koné-Paut I, et al.  . Risk of automimmune diseases and human papilloma virus (HPV) vaccines: six years of case-referent surveillance. J Autoimmun . 2017; 79: 84– 90. Google Scholar CrossRef Search ADS PubMed  14 Feiring B, Laake I, Bakken IJ, et al.  . HPV vaccination and risk of chronic fatigue syndrome/myalgic encephalomyelitis: a nationwide register-based study from Norway. Vaccine . 2017; 35( 33): 4203– 4212. Google Scholar CrossRef Search ADS PubMed  15 Miranda S, Chaignot C, Collin C, et al.  . Human papillomavirus vaccination and risk of autoimmune diseases: a large cohort study of over 2 million young girls in France. Vaccine . 2017; 35( 36): 4761– 4768. Google Scholar CrossRef Search ADS PubMed  16 Sukumaran L. Update on HPV vaccine safety. In: Advisory Committee on Immunization Practices (ACIP), Centers for Disease Control and Prevention, Department of Health and Human Services. Meeting of the Advisory Committee on Immunization Practices (ACIP). Summary Report. October 21, 2015. Atlanta, Georgia. Atlanta, GA: Centers for Disease Control and Prevention; 2015:69–73. https://www.cdc.gov/vaccines/acip/meetings/downloads/min-archive/min-2015-10.pdf. Accessed March 1, 2018. 17 Slade BA, Leidel L, Vellozzi C, et al.  . Postlicensure safety surveillance for quadrivalent human papillomavirus recombinant vaccine. JAMA . 2009; 302( 7): 750– 757. Google Scholar CrossRef Search ADS PubMed  18 Chandler RE, Juhlin K, Fransson J, et al.  . Current safety concerns with human papillomavirus vaccine: a cluster analysis of reports in Vigibase®. Drug Saf . 2017; 40( 1): 81– 90. Google Scholar CrossRef Search ADS PubMed  19 Shimabukuro TT, Nguyen M, Martin D, et al.  . Safety monitoring in the Vaccine Adverse Event Reporting System (VAERS). Vaccine . 2015; 33( 36): 4398– 4405. Google Scholar CrossRef Search ADS PubMed  20 Varricchio F, Iskander J, Destefano F, et al.  . Understanding vaccine safety information from the Vaccine Adverse Event Reporting System. Pediatr Infect Dis J . 2004; 23( 4): 287– 294. Google Scholar CrossRef Search ADS PubMed  21 Iskander JK, Miller ER, Chen RT. The role of the Vaccine Adverse Event Reporting System (VAERS) in monitoring vaccine safety. Pediatr Ann . 2004; 33( 9): 599– 606. Google Scholar CrossRef Search ADS PubMed  22 Brown JS, Petronis KR, Bate A, et al.  . Drug adverse event detection in health plan data using the Gamma Poisson Shrinker and comparison to the tree-based scan statistic. Pharmaceutics . 2013; 5( 1): 179– 200. Google Scholar CrossRef Search ADS PubMed  23 Kulldorff M, Dashevsky I, Avery TR, et al.  . Drug safety data mining with a tree-based scan statistic. Pharmacoepidemiol Drug Saf . 2013; 22( 5): 517– 523. Google Scholar CrossRef Search ADS PubMed  24 Kulldorff M, Fang Z, Walsh SJ. A tree-based scan statistic for database disease surveillance. Biometrics . 2003; 59( 2): 323– 331. Google Scholar CrossRef Search ADS PubMed  25 Curtis LH, Weiner MG, Boudreau DM, et al.  . Design considerations, architecture, and use of the Mini-Sentinel distributed data system. Pharmacoepidemiol Drug Saf . 2012; 21( suppl 1): 23– 31. Google Scholar CrossRef Search ADS PubMed  26 Nguyen M, Ball R, Midthun K, et al.  . The Food and Drug Administration’s Post-Licensure Rapid Immunization Safety Monitoring program: strengthening the federal vaccine safety enterprise. Pharmacoepidemiol Drug Saf . 2012; 21( suppl 1): 291– 297. Google Scholar CrossRef Search ADS PubMed  27 Ball R, Robb M, Anderson SA, et al.  . The FDA’s sentinel initiative—a comprehensive approach to medical product surveillance. Clin Pharmacol Ther . 2016; 99( 3): 265– 268. Google Scholar CrossRef Search ADS PubMed  28 Healthcare Cost and Utilization Project, Agency for Healthcare Research and Quality, US Department of Health and Human Services. Clinical Classifications Software (CCS) for ICD-9-CM. http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Updated March 6, 2017. Accessed June 6, 2017. 29 Kulldorff M; Information Management Services, Inc. TreeScan: software for the tree-based scan statistic. Published 2014. Accessed March 1, 2018. 30 Cole DV, Kulldorff M, Baker M, et al.  . Infrastructure for Evaluation of Statistical Alerts Arising from Vaccine Safety Data Mining Activities in Mini-Sentinel. Silver Spring, MD: Sentinel Initiative, Food and Drug Administration; 2016. https://www.sentinelinitiative.org/sites/default/files/Methods/Mini-Sentinel_PRISM_Data-Mining-Infrastructure_Report_0.pdf. Accessed November 14, 2017. 31 Merck & Company, Inc. GARDASIL® [Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) Vaccine, Recombinant]. (Gardasil package insert). Whitehouse Station, NJ: Merck & Company, Inc.; 2006. http://www.fda.gov/downloads/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM111263.pdf. Revised April 2015. Accessed June 6, 2017. 32 Merck & Company, Inc. Varivax®. Varicella Virus Vaccine Live. (Varivax package insert). Whitehouse Station, NJ: Merck & Company, Inc.; 2013. http://www.fda.gov/downloads/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM142812.pdf. Accessed March 1, 2018. 33 Kinoshita T, Abe RT, Hineno A, et al.  . Peripheral sympathetic nerve dysfunction in adolescent Japanese girls following immunization with the human papillomavirus vaccine. Intern Med . 2014; 53( 19): 2185– 2200. Google Scholar CrossRef Search ADS PubMed  34 Brinth LS, Pors K, Theibel AC, et al.  . Orthostatic intolerance and postural tachycardia syndrome as suspected adverse effects of vaccination against human papilloma virus. Vaccine . 2015; 33( 22): 2602– 2605. Google Scholar CrossRef Search ADS PubMed  35 Pharmacovigilance Risk Assessment Committee, European Medicines Agency. Assessment Report. Review Under Article 20 of Regulation (EC) No. 726/2004. Human Papillomavirus (HPV) Vaccines. London, United Kingdom: European Medicines Agency; 2015. (Publication EMA/762033/2015. http://www.ema.europa.eu/docs/en_GB/document_library/Referrals_document/HPV_vaccines_20/Opinion_provided_by_Committee_for_Medicinal_Products_for_Human_Use/WC500197129.pdf. Accessed September 25, 2017. 36 Jefferson T, Jorgensen L. Human papillomavirus vaccines, complex regional pain syndrome, postural orthostatic tachycardia syndrome, and autonomic dysfunction—a review of the regulatory evidence from the European Medicines Agency. Indian J Med Ethics . 2017; 2( 1): 30– 37. Google Scholar PubMed  37 Chandler RE. Safety concerns with HPV vaccines continue to linger: are current vaccine pharmacovigilance practices sufficient? Drug Saf . 2017; 40( 12): 1167– 1170. Google Scholar CrossRef Search ADS PubMed  38 WorkSafeBC Evidence-Based Practice Group, WorkSafeBC, Government of British Columbia. Complex Regional Pain Syndrome (CRPS). What Does the Literature Report Regarding the Time Interval Between the Inciting Trauma and Its Subsequent Diagnosis? Vancouver, British Columbia, Canada: WorkSafeBC; 2011. https://www.worksafebc.com/en/resources/health-care-providers/guides/complex-regional-pain-syndrome-crps-what-does-the-literature-report-regarding-the-time-interval-between-the-inciting-trauma-and-its-subsequent-diagnosis?lang=en. Accessed September 26, 2017. 39 Yih WK, Maro JC, Nguyen M, et al.  . Sentinel CBER/PRISM Methods. Pilot of Self-Controlled Tree-Temporal Scan Analysis for Gardasil Vaccine. Silver Spring, MD: Sentinel Initiative, Food and Drug Administration; 2016. https://www.sentinelinitiative.org/sites/default/files/Methods/Mini-Sentinel_PRISM_Pilot-Self-Controlled-Tree-Temporal-Scan-Analysis-Gardasil-Vaccine-Report.pdf. Accessed November 14, 2017. © The Author(s) 2018. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journalpermissions@oup.com.

Journal

American Journal of EpidemiologyOxford University Press

Published: Feb 23, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off