A randomized clinical trial is the gold standard for evaluating the efficacy and safety of treatment for eye diseases. Because each subject has two eyes, and many eye diseases (e.g., significant refractive error, dry eye, glaucoma, retinal diseases) can be present in both eyes at same time, ophthalmic trials have some unique features in the trial design and statistical analysis depending on whether two eyes of a participant are enrolled into a trial (e.g., one eye design vs. two-eye design) [1], and whether the two eyes are assigned to the same treatment group or different treatment groups [2-9]. Additionally, since outcome measures from two eyes of the same subject tend to be positively correlated, the appropriate trial design and statistical analysis need to account for their inter-eye correlations [10].

Lee et al. reviewed 69 ophthalmic clinical trial papers published in 4 major clinical ophthalmology journals (Ophthalmology, JAMA Ophthalmology, American Journal of Ophthalmology, and British Journal of Ophthalmology) in 2009, and found substantial heterogeneity in the quality of the trial design and statistical analysis [1]. In particular, they found that among 13 trials with two-eye design, only 5 (38%) trials used the proper statistical methods for adjusting for the inter-eye correlation, and none of them explicitly stated the adjustment of the inter-eye correlation in the sample size and power calculation. In a recent review of 96 ophthalmic clinical trial papers published in these same 4 ophthalmology journals in 2021, Dong et al. found that in 31 trials with two-eye design, only 13 (42%) trials properly adjusted for the inter-eye correlation in the statistical analysis [9].

The clinical trials funded by the National Eye Institute (NEI) went through comprehensive scientific review by biostatisticians, and clinical trialists for trial design and statistical analysis plan before their approval for funding, and the funded trials were led by a team of investigators with experience in conducting ophthalmic trials. It is reasonable to assume that the NEI-funded trials had fewer flaws in the trial design and statistical analysis. Thus, we conducted a review of NEI-funded trials to evaluate their quality in terms of the study design and statistical analysis. This paper reports our findings on the characteristics of design and statistical analysis of NEI-Funded late phase randomized ophthalmic clinical trials conducted after 2000.

On June 1, 2022, we searched for phase 2/3 or phase 3 randomized ophthalmic clinical trials from ClinicalTrials.Gov that were funded by the NEI and started the trial after the year 2000 with the primary result paper (e.g., paper reporting the trial primary outcome results) published by June 1, 2022. In ClinicalTrials.Gov, we searched with the following keywords: “eye diseases” in the condition/disease box, “Phase 3” for phase, “NIH” for funding type, and study start date: “01/01/2000”.

For each trial identified from the search in ClinicalTrials.Gov, two authors (WZ, RZ) independently searched for the primary result paper using the following approaches:

1. PubMed search using NCT number or title of the clinical trial and/or PI name

2. ClinicalTrials.Gov list of publications

3. Google searching using the NCT number or title of the clinical trial and/or PI name

If a primary result paper could not be found from the above search approaches, the trial was not included in this review.

From the primary result paper of each eligible trial, we collected data on the trial design characteristic including the phase of the trial, type of eye disease, type of intervention (drug, medical device, treatment procedure etc.), type of design (one-eye design, two-eye design), number of arms, sample size, statistical power, number of clinical sites, the data type of primary outcome (continuous, categorical, time-to-event), sidedness of statistical test (one-sided or two-sided), type of comparisons (non-inferiority, superiority, equivalence), correction for multiple comparisons, statistical tests for comparison of the primary outcome. In addition, for the trial with two-eye design, we collected information on whether two eyes were in the same or different treatment groups, whether the inter-eye correlation was adjusted for in the trial design and statistical analysis, and the statistical method applied for adjusting for the inter-eye correlation.

All these data extractions were independently performed by two authors (WZ, RZ) and the differences were adjudicated by the senior author (GSY). Descriptive analyses were performed using R to summarize the characteristics of trial design and statistical analysis.

This study was conducted by reviewing published papers; it did not involve any human subjects. Thus, the institutional review board approval and the Declaration of Helsinki do not apply.

The search in Clinicaltrials.gov found a total of 66 clinical trials (Fig. 1). After excluding 8 ongoing trials that have not yet published primary outcome paper, 6 oncology early phase trials, 5 completed trials yet has not published primary result paper, 2 single-arm trials, 2 trials not funded by NEI, and one phase 2 trial, a total of 42 eligible trials were included in the review for this study.

The characteristics of these 42 eligible trials are reported in Table 1. These 42 trials were mostly funded for evaluating the treatment of retinal diseases (45%) and pediatric eye diseases (45%). The treatment evaluated in these trials was mostly for drugs (57%), medical devices (21%), or treatment procedures (14%).

Fig. (1). Flow chart for eligible trials for analysis.

In terms of trial design, the majority (n=39, 93%) of trials used the parallel design. These trials included 35 trials (83.3%) for evaluating superiority, 7 (17%) trials for evaluating non-inferiority, and 1 (2%) trial for equivalence. More than two-thirds (69%) of these trials had two comparison arms, and 13 (31%) had three or four arms. Correction for multiple comparisons from multiple arms was performed in 12 (92%) of these 13 multiple-arm trials using the Bonferroni method (42%), Hochberg procedure (42%), or Tukey’s method (17%).

Sample sizes (based on actual enrollment) of these trials ranged from 19 to 11267 with a median of 265 participants (inter-quartile: 190-528). The number of clinical sites for enrolling participants ranged from one to 128 with a median of 41 (inter-quartile: 18 to 61). With the exception of one pilot trial without providing statistical power, all trials were designed with statistical power at least 80%, with 17 trials (41%) having 80% power, 3 (7%) trials having power 85-90%, and 21 trials (51%) having power 90% or above. Thirty-three (80%) trials were designed with two-sided tests.

The primary outcome measure was continuous in 27 (64%) trials, binary in 10 (24%) trials, and time-to-event outcome in 5 (12%) trials. Masking was implemented in 29 (69.0%) trials, including single masking (21%), double masking (21%), triple (7%), or quadruple (19%) masking. In 41 trials with a comparison of primary outcome, a statistically significant difference in primary outcome measure was found in 19 (46%) trials.

In this study, we reviewed the characteristics of trial design and statistical analysis for forty-two NEI-funded randomized ophthalmic clinical trials conducted since 2000. We found that these trials were designed with good statistical power (≥80%) using a variety of trial designs for evaluating different treatments of eye diseases. About a quarter of these trials used a two-eye design, and statistical analysis was performed properly in almost all the trials with adjustment of inter-eye correlation when data were analyzed at eyelevel.

The outcome measures from the two eyes of a participant are usually positively correlated, proper statistical analyses are necessary to account for the inter-eye correlation [10]. Previous studies found that the correlated eye data are often analyzed inappropriately by ignoring the inter-eye correlation using statistical methods that assume independence in the data [1], [9, 11,12]. Murdoch’s study reviewed 67 clinical science papers published in the British Journal of Ophthalmology in 1995 and found that 70% of 23 papers that used two-eye design did not appropriately adjust for the inter-eye correlation [12]. Similarly, Zhang et al reviewed 56 clinical science papers published in the British Journal of Ophthalmology in 2017 and found that 85% of 39 papers that used two-eye design did not adjust for the inter-eye correlation [11]. Lee et al. reviewed 69 papers from randomized clinical trials published in the four ophthalmology journals in 2009, including Ophthalmology, JAMA Ophthalmology, American Journal of Ophthalmology, and the British Journal of Ophthalmology, and found that 62% of trials of two-eye design did not adjust for the inter-eye correlation [1]. In a recent review of 96 ophthalmic clinical trial papers published in these 4 major ophthalmology journals in 2021, Dong et al found that among 31 trials with two-eye design, 18 (58%) trials did not adjust for the inter-eye correlation [9]. Compared to the high percent of inappropriate statistical analyses of two-eye data in previous studies, the statistical analyses of two-eye data in NEI-funded trials were outstanding in that almost all the trials used two-eye design properly adjusted for the inter-eye correlation when data were analyzed at eye level. The better statistical analysis practice in the NEI-funded trials may be due to the rigorous scientific review process by the expert panel before trials were funded, the rich clinical trial experience among the investigators of the NEI-funded trials, and the support from the data coordinating center who had experience in the design and statistical analyses of similar ophthalmic trials. For the ophthalmic trials that are not funded by NEI, the more stringent oversight of the trial protocol by experienced investigators, and biostatistical consultation of experts for the proper clinical trial design and statistical analyses may improve the practice in ophthalmic trials.

Many eye diseases are bilateral, providing a unique opportunity to enroll both eyes of a participant into a trial. Our review found that about one-quarter of the NEI-funded trials used a two-eye design with less than half of them assigning two eyes to different treatment groups. Although it is advantageous to enroll both eligible eyes into a trial from a statistical perspective, many other factors have to be considered in deciding between one-eye design or two-eye design, and whether assigning two eyes in the same treatment group or different treatment group if two-eye design is used [6,13]. Enrolling both eyes in a trial sometimes can logistically complicate the process of enrollment, treatment, and follow-up, potentially introducing bias in outcome assessment. Ethically, it is sometimes challenging to enroll both eyes into a trial at the same time. In two-eye designs, two eyes assigned to the same treatment group should be considered when there are person-level outcome measures (e.g., quality of life), there is a possibility of cross-over effects in the treatment, or if systemic safety of the treatment is of interest.

This review found that more than half of the trials had continuous primary outcome measures, about a quarter of trials had binary primary outcome and over 10% of trials had time-to-event as a primary outcome. The statistical method for analyzing primary outcome data is dependent on the type of data. For independent outcome data, the analysis of variance or analysis of covariance is commonly used to evaluate the treatment effect for continuous outcome data, the Chi-square test or logistic regression model is used for categorical outcome data, and the Kaplan-Meier analysis and Cox proportional hazards model are used for time-to-event outcome data. When the outcome measures were obtained from both eyes of a participant and data were analyzed at eye level, the inter-eye correlation needed to be accounted for by using a mixed effect model or generalized estimating equations for a continuous outcome, [14] generalized estimating equations for binary outcome [15] and marginal Cox regression model for time-to-event outcome [16]. In these NEI-funded trials, appropriate methods were applied in analyzing different types of correlated two-eye data from trial participants.

In this review, we found that about one-third of the trials had more than two treatment arms which led to multiple pairwise comparisons for evaluating the efficacy of various treatments. It is well-known that multiple comparisons can lead to inflation of type I error rates (i.e., mistakenly claiming that treatment is efficacious when in truth it is not) [17]. Appropriate statistical comparisons are required to account for the multiple comparisons, so that the overall 5% type I error rate is maintained [17,18]. In this review, we found 12 (92%) out of 13 NEI-funded trials properly corrected for multiplicity by comparing multiple arms using the Bonferroni method, Hochberg procedure, or Tukey’s method. This finding suggests that the guidelines on the good practice of clinical trials are closely followed in NEI-funded trials.

Conducting a clinical trial requires a huge investment in time and resources, and failure of identification of effective treatment due to an underpowered clinical trial should be prevented. Late-phase trials are often designed with high statistical power to avoid missing the detection of clinically important treatment differences. Except for one pilot trial without providing statistical power, all the NEI-funded trials were designed with at least 80% power, with more than half of the trials having at least 90% power. However, in spite that the inter-eye correlation was accounted for in the statistical analysis, none of these trials explicitly stated in the primary result paper that the inter-eye correlation was accounted for in the sample size and statistical power calculation, although adjustment for inter-eye correlation could have been made in the trial design. Similar to our findings, Lee’s review of 69 randomized clinical trials published in 4 major clinical ophthalmology journals in 2009 also found none of these trials adjusted for the inter-eye correlation in the sample size calculation [1]. The ignorance of inter-eye correlation in sample size and power calculation could be due to a lack of procedure/software for easy adjustment of the inter-eye correlation, or due to the lack of information on the magnitude of inter-eye correlation that is needed for the precise sample size calculation for the trials with two-eye design. Lack of consideration of the inter-eye correlation could lead to over-estimation or under-estimation of the sample size and statistical power, depending on whether two eyes are assigned to the same treatment group or different treatment groups. Future collaborations are needed, including evaluating the inter-eye correlation of ocular conditions or measures and developing reporting standards to improve the practice of sample size/power calculation for the ophthalmic trials involving two eyes.

This review found that NEI-funded trials followed good practice in trial design and statistical analysis. While the availability of two eyes of a participant may complicate the trial design and statistical analysis, these NEI-funded trials were designed with adequate statistical power, took advantage of including two eyes when appropriate in the design, and properly adjusted for the inter-eye correlation in the statistical analysis. However, these trials have either not explicitly considered or reported the adjustment for the inter-eye correlation in the sample size and power calculation. Future research and education on the appropriate sample size and power calculation that accounts for the inter-eye correlation are needed to improve the design and reporting of ophthalmic trials.

WZ and RZ contributed equally to the design and conduct of the study, data collection, and management, descriptive statistical analysis, as well as results interpretation and wrote the whole paper.

GSY supervised WZ and RZ for the work, adjudicated the differences in data collected by WZ and RZ, provided guidance on the design and conduct of the study, and critically reviewed, revised, and approved the manuscript.

All authors have read and approved the manuscript.