Evaluating Fidelity of reporting in randomized controlled trials on childhood acute lymphocytic leukemia

Abstract

Background

Randomized controlled trials (RCTs) play an integral role in childhood cancer research. Several efforts to improve the quality of reporting of clinical trials have been published in recent years, including the TIDieR checklist. Many reviews have since used TIDieR to evaluate how well RCTs are being reported, but no such study has yet been done in childhood cancer. The aim of this study is to evaluate adherence of RCTs involving acute lymphocytic leukemia (ALL) to the TIDieR checklist.

Methods

The PubMed database was used to screen for RCTs involving ALL published since 2015. Of 1546 articles identified, 46 met study criteria and were then evaluated against the TIDieR 12-point checklist to measure the degree of adherence.

Results

Of the 46 articles included, 9 (19.6%) met full TIDieR criteria. Seven of the 9 reported non-pharmacological interventions, and the remaining 2 reported pharmacological interventions. The average article properly reported 8.98/12 checklist items. Item 5 (intervention provider) was the most poorly reported item, properly reported in only 34.8% of articles.

Conclusion

We conclude that overall TIDieR adherence is low and needs to be adhered to more fully in order to improve research in ALL as well as in all childhood cancers.

Introduction

Acute lymphocytic leukemia (ALL) is the most common type of childhood cancer [1, 2]. In 2016, ALL was newly diagnosed in over 6500 cases in the U.S. and claimed 1400 lives [3]. On a global scale, the overall number of incident diagnoses of ALL is estimated to rise through 2025 [2]. In spite of increased incidence, the 5-year survival rate has steadily risen since the 1960s and recently rose above 90% for the first time [4, 5]. Research continues to yield promising discoveries concerning how ALL is managed. Specifically, randomized controlled trials have been instrumental in optimizing treatment for ALL.

Randomized controlled trials (RCTs) allow scientists to test new theories that may improve clinical practice. As the gold standard for assessing new treatments, RCTs provide valuable information for clinicians who wish to implement evidence-based advancements into their practice [6, 7]. RCTs play a particularly important role in childhood cancer research, with childhood cancer patients participating in RCTs in large numbers relative to other patient populations. Therefore, it is important to maximize the efficacy of RCTs with a thorough, systematized standard.

There has been a strong push in recent years to improve reporting methods in RCTs. Due to “lack of adequate reporting” and “biased results” among reported RCTs, the first Consolidated Standards of Reporting Trials (CONSORT) report was published in 1996 to improve reporting quality [8]. Several evidence-based CONSORT reports have been published since, with the most recent revision published in 2010 [9]. Through a checklist and flowchart, CONSORT guides investigators in proper reporting of RCTs [9]. A vital piece of any clinical trial is the intervention being tested. An extension of CONSORT – the Template for Intervention Description and Replication (TIDieR) – was recently published to address the lack of standardization of intervention reporting [10].

The TIDieR checklist and guideline was published in 2014 to improve reporting and reproducibility of RCTs. Although many prominent medical journals endorse CONSORT [9, 11, 12] and a significant percentage of RCTs report adherence to CONSORT guidelines, “reporting of guidelines [continues to be] deficient,” partially due to “lack of awareness among authors about what comprises a good description” [10]. In response to this deficiency, an international team of experts created the TIDieR checklist with evidence-based research to increase accountability in reporting and reproducibility of interventions [10]. Since the publication of TIDieR in 2014, few studies in different fields of medicine have examined adherence of RCTs to TIDieR guidelines [13,14,15]. However, TIDieR research within the field of oncology is scarce, and no such study has been conducted on an individual childhood cancer disease. Here, we analyze adherence to the TIDieR checklist of RCTs published since 2015 involving childhood acute lymphocytic leukemia.

Methods

PubMed, which includes the MEDLINE collection, was used to search for RCTs involving childhood ALL, defined in our study as ALL occurring before 18 years of age. We chose PubMed to maximize sensitivity when searching medical journals for any aspect of treatment or diagnosis of childhood ALL [16]. To search PubMed with maximum sensitivity for RCTs pertaining to pediatric ALL, we implemented an evidence-based search strategy from The Cochrane Database of Systematic Reviews (CDSR). The CDSR is considered the leading source of systematic reviews in medicine, with each Cochrane review developed by a professional editorial team and peer-reviewed to maintain high quality [17]. We used a PubMed search string for “acute lymphocytic leukemia” developed by a Cochrane Review on ALL [1]. We applied the following PubMed filters to the Cochrane Review search string: “Clinical Trial,” “Full Text,” “2015/01/01 – 2019/04/11,” “Humans,” “English,” and “Cancer.” The choice to exclude articles published before 2015 was made to allow an adequate buffer period for authors to begin adhering to TIDieR guidelines, since the guidelines were published in early 2014. The choice to exclude articles without a “Full text” permits a more accurate evaluation of the studies pulled in our search.

After implementing our CDSR-based PubMed search strategy, we established parameters for inclusion and exclusion of journal articles for final analysis of childhood ALL RCTs. To qualify for inclusion in our study, we used the following criteria: 1) report a randomized controlled trial, 2) contain only human subjects, 3) be published on or after 2015/01/01, 4) be published in English, 5) contain subjects of whom greater than 50% were diagnosed with ALL, and 6) contain subjects of whom greater than 50% of trial subjects were under 18 years of age. Criterion 5 was established as it was deemed unnecessary to exclude articles containing other childhood cancers if the majority of subjects were diagnosed with childhood ALL. Similarly, criterion 6 was established as it was deemed unnecessary to exclude articles containing young adults if the majority of subjects were under 18 years of age.

Each article generated from our CDSR-based PubMed search string was screened for eligibility independently by two investigators (NH and AH). Each excluded article received a label identifying how it did not meet criteria. A collaboration meeting was held between both investigators to review the list of articles for inclusion and exclusion. A consensus was established for each article. The articles that met inclusion criteria were analyzed independently for degree of adherence to the TIDieR checklist.

The TIDieR checklist is a 12-point document containing a brief description of each guideline [10]. It is designed as a checklist that is to be submitted with a publication, requiring authors to provide information on where each checklist item can be found in the publication [10]. A summarized version of the 12-point TIDieR checklist can be found in Table 1. A Google Form was used to score adherence to each item from the TIDieR checklist. Two investigators (NH and AH) independently scored all included articles for adherence to each item of the TIDieR checklist. Grading was achieved with a “Yes” or “No” scale for each item. A “Yes” was selected if an article reported all information required by TIDieR guidelines. Items 9 and 10 contained the option “N/A” (not applicable) since their presence within a study is conditional on the study design and execution. Additionally, we documented the publishing journal, funding, hypothesis, study type, intervention, blinding status, country, clinical trial registry status, and reported adherence to TIDieR. After the two investigators independently screened each article for TIDieR adherence, a second collaboration meeting was held and a consensus was established between the two investigators for each item on every article. A third investigator (MV) resolved discussions that were not easily resolved between the two primary analysts.

Table 1 Summary of TIDieR Checklist Items and Descriptions

Results

The PubMed search yielded 1546 articles. After independent screening and achieving consensus, 46 articles [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63] were deemed to meet inclusion criteria for further evaluation of adherence to TIDieR (see Fig. 1). The most common reason for excluding an article was a wrong population (n = 1033), followed by wrong study design (n = 440). Examples of wrong population include studies that dealt with another disease process and studies with a majority of adult subjects. Examples of wrong study design include non-randomized trials and cohort studies.

Fig. 1

Flow diagram of study selection

Table 2 summarizes the demographic information of our included articles. Of the 46 articles examined, 35 (76.1%) reported pharmacological interventions, while the remaining 11 (23.9%) reported non-pharmacological interventions. 32 trials were not blinded or did not specify blinding type, with 10 being double-blinded and 4 being single-blinded. Thirty-one studies were conducted outside of the U.S., 13 were conducted in the U.S., and the remaining 2 were conducted both in the U.S and outside of the U.S. No studies stated adherence to TIDieR guidelines.

Table 2 Demographic Information for the 46 Included Articles

Figure 2 shows the number of properly reported TIDieR checklist items (out of 12) for the 46 included articles. On average, the articles adhered to TIDieR guidelines on 8.98/12 items, with a range of 5/12–12/12, and a median value of 9/12. Nine articles (19.6%) met full criteria for adherence to TIDieR guidelines. Of those, 7 reported non-pharmacological interventions, and 2 reported pharmacological interventions. Two articles reported only 5 of 12 checklist items correctly.

Fig. 2

TIDieR checklist items adhered to by the 46 articles

Figure 3 contains each item of the TIDieR checklist and the frequency of adherence among the articles. The most common item on the TIDieR checklist incompletely reported was item 5 (intervention provider), which was only present in 34.8% of articles. Item 7 (location) was reported in 50% of articles, while item 6 (mode of delivery) was adequately reported in only 52.2% of articles. Conversely, items 1 (what the intervention is) and 2 (rationale for the intervention) was adequately reported in 100% of the articles we analyzed, while item 8 (time and duration of delivery) was properly reported in 97.8% of articles.

Fig. 3

Percentage of articles that adhered to each TIDieR checklist item

Figure 4 shows a comparison of the year of publication of total articles screened versus articles that met TIDieR checklist standards. Years of publication of the 46 articles spanned from 2015 to 2019, while years of publication of the 9 articles that met full TIDieR checklist criteria spanned from 2015 to 2017. 2016 yielded the highest number of both total articles screened and articles that met TIDieR criteria.

Fig. 4

Head-to-head comparison of the publication year of all RCTs included in the study vs. the RCTs that met TIDieR checklist standards

Discussion

RCTs are the gold standard for advancement of medical research into clinical practice [6, 7]. TIDieR guidelines are intended to increase reproducibility of findings in clinical trials by standardizing reporting of RCTs across all areas of medicine [10]. Our results show that there is still much work to be done. Our study found less than of RCTs that adhered to all TIDieR guidelines when reporting interventions. The average article missed at least 3 checklist items. Additionally, our study shows patterns of underreporting in specific elements of TIDieR. Almost two thirds of the articles neglected to properly report the title and qualifications of the person administering the intervention (item 5). Half of the articles failed to report the mode of delivery (item 6), and nearly half also failed to report a proper title or description of the location of the intervention (item 7). The most likely consequence of continuing inconsistent adherence to TIDieR guidelines is a lack of improvement in study design bias of future RCTs.

Literature shows that reporting of adherence of RCTs to proper intervention varies [8,9,10]. The TIDieR publication cites one study that found that only 11% of articles provided adequate information concerning the intervention [10]. Subsequent TIDieR studies have found similarly poor adherence rates to TIDieR guidelines. For example, Liljeberg et al. reported only 3% of articles that fulfilled all TIDieR criteria in their study [13]. Another study by Hacke et al. reported 0% out of 24 articles complete [64]. while this current study reports a relatively high total adherence rate of 19.6%. We found an average TIDieR adherence rate of 8.98/12, with 9 articles meeting full criteria. This statistic appears to reflect the findings of several previous TIDieR studies. For example, Hacke et al. reported an average TIDieR adherence rate of 61% (7.32/12), while McEwen et al. reported an average of 8.81/12 [15, 64].

A recent study found only 67% of pharmacological interventions were reported properly, compared to 29% of non-pharmacological interventions [10]. Interestingly, our findings showed much better adherence among non-pharmacological intervention studies. Of 11 non-pharmacological RCTs analyzed, 7 (63.6%) were adequately reported. Compare that with the remaining 35 articles reporting pharmacological intervention, of which only 2 (3.6%) were properly reported. Poor adherence in reporting the individual providing the intervention along with his or her qualifications (item 5) was noted among pharmacological RCTs, while good adherence to item 5 was noted among non-pharmacological RCTs. Several factors could be responsible. Many pharmacological RCTs were administered in an inpatient hospital setting. Perhaps authors of pharmacological RCTs assumed the qualifications of those administering the intervention were obvious in a hospital setting. Another consideration is the impact factor of journals from which non-pharmacological vs pharmacological RCTs were pulled, but this does not explain the difference in adherence between the two RCT groups. Further research may be warranted on the reasons for this pattern of adherence to TIDieR guidelines.

Our study suggests that adherence to TIDieR guidelines in reporting interventions in RCTs needs to improve. Furthermore, we found a systemic pattern of inconsistent intervention reporting in RCTs. Almost two thirds of RCTs are not reporting enough information about the location of the intervention to satisfy TIDieR guidelines. As stated in the TIDieR guide, reporting details about the location of intervention such as funding source, facility capabilities, volume of activity, etc. provides crucial and necessary information to whomever wishes to replicate the study [10]. Only half of studies reported enough information about the individual(s) that administered the intervention(s) to satisfy TIDieR guidelines. Properly reporting the title, certification and specific training of professionals delivering an intervention has significant influence on the reliability, validity, accuracy and precision of the RCTs.

We hypothesize that a solution to improve adherence to the TIDieR checklist is to increase awareness. Of the 46 articles analyzed in this study, none explicitly mentioned TIDieR or stated compliance with TIDieR guidelines. Furthermore, the data from Fig. 4 does not suggest a pattern of increasing adherence to the TIDieR checklist among articles based on their publication date. The TIDieR checklist was designed by a team of international professionals using evidence-based research [10] and should therefore be implemented by those who value peer-reviewed research methods.

Our study has limitations to discuss. It is primarily descriptive and is therefore more dependent upon experience and judgment than quantitative research. We attempted to understand that some bias is unavoidable in this type of research and therefore focused on reducing its effects [65, 66]. Although the TIDieR checklist is meant to standardize reporting and eliminate bias when evaluating adherence, our design could not eliminate the possibility of subjective interpretation when grading the adherence of RCTs. Although the TIDieR guide gives examples of successful reporting of several types of interventions for each TIDieR item, explanations are brief and rarely comprehensive enough to eliminate gray areas [10]. In these situations, we attempted to interpret the difficult determinations on a case-by-case basis. In discussing a publication similar to this present study, Liljeberg et al. noted the wide variety in degree of adherence of RCTs to TIDieR protocol and hypothesized that the degree of stringency of individual reviewers likely contributes to the disparity in findings [13]. We aimed to reduce bias by educating investigators with the same video series and discussions with a third party on the main elements of the TIDieR guidelines. Multiple samples were practiced between the two investigators, prior to the start of the study, to eliminate inaccurate interpretations of TIDieR guidelines. Further, both investigators were blinded to the other’s findings until disagreements were revealed at the completion of the study.

Another limitation is the amount of literature pertinent to ALL since the publication of TIDieR. To the knowledge of the authors, no TIDieR study has ever been conducted on RCTs of a single disease process before this current study. ALL was chosen due to its high degree of prevalence in the field of pediatric oncology [1, 2]. With that said, limiting the study to a specific disease limits available data. Whereas 46 articles were analyzed in this study, other TIDieR studies have double our amount or more from which to pull data.

We cannot extrapolate the results of our study to describe the adherence of RCTs involving other types of cancers.

Conclusion

The TIDieR checklist is an effective way of improving the reporting of interventions in randomized controlled trials. As new research continues to use TIDieR guidelines, it is our belief that reporting methods will continue to improve, allowing new interventions to be translated from clinical trials into clinical practice in a safe, efficient, and effective way. It is our hope that further research adhering to the TIDieR checklist will improve treatment of acute lymphocytic leukemia and all other diseases.