The usefulness of observational studies in the evaluation of drugs is not the same if the aim of the analysis concerns adverse effects or efficacy. Most people would admit the use of observational studies for evaluating the risk of rare adverse events of drugs, because in that case there is a smaller likelihood of prescription bias linked with good prognosis [5] and reject its use for evaluating the efficacy [6–8].

The reason for this is that well-conducted clinical trials are less prone to bias than observational studies. Three main categories of bias can be defined: selection bias, classification bias, and confounding bias [9]. A selection bias can occur when a participant is lost to follow-up in a cohort study, or when the endpoint of interest cannot be evaluated, or when only prevalent cases are recruited in a case-control study. It may also occur when prevalent users are enrolled instead of new users in a cohort study and the risk varies with time [10]. The healthy worker effect in environmental research or the healthy user effect in pharmacoepidemiology is also an example of selection biases. Classification bias may occur when the person in charge of data collection knows the status of the subject. In cohort studies, classification bias may occur when there is a problem in the detection of the outcome of interest or when there is an error in the status of the subject (with or without the outcome of interest). In case-control studies, a classification bias may occur because case and control do no report exposure with similar reliability, or when there is an error on the measurement of the exposure status. Finally, in an observational setting, a confounding bias can occur when a factor is a risk factor for the outcome of interest, and is associated to exposure too. If the two groups compared differed by the frequency of the confounding factor, this may biased the result of the study. A confounding factor may be accounted for by design, using matching, or by analysis, but it is always difficult to warrant than no confounding remains.

In a well-conducted randomized controlled clinical trial, the treatment is randomized between groups. This procedure controls for selection bias and confounding at enrolment. If the trial is blinded, it also controls for differential measurement errors in the outcome of interest, therefore reducing classification bias. However, some selection bias and confounding can still occur if patients are lost to follow-up or if the outcome of interest cannot be assessed on all subjects enrolled.

However, in the real world, trials may be subject to bias, unreliable (too small, low quality, unpublished), not available, not appropriate, not ethical or not possible. For instance, the cost of the WHI was estimated as $628 million over the 15-year period 1992–2007.

The best available knowledge at time of the WHI publication was nicely summarized by Grodstein et al. [11]. For hip fracture, a meta-analysis from 1992 [1] estimated the relative risk in women exposed to HRT as 0.75 (95% CI: 0.68–0.84). For coronary heart diseases (CHD), three meta-analyses were available (Table 1); all found a protective effect of HRT on the risk of CHD [1,12,13]. These three meta-analyses were strongly influenced by the results from the Nurse's Health Study (NHS). The Nurses' Health Study began in 1976 when 121 700 female nurses 30 to 55 years of age completed a mailed questionnaire including questions about their postmenopausal hormone use and medical history, including cardiovascular disease and its risk factors. In 1976, women were asked about use and duration of hormone therapy after menopause. Beginning in 1978, information on the types of hormones taken was collected. All information was updated biennially. In 1976, 21 947 postmenopausal women entered the analysis, and 48 586 women were added during follow-up as they became postmenopausal, for a total of 70 533 participants; 808 825 person-years of follow-up were accrued in the most recent updated publication on this topic before the publication of the WHI trial [14]. The NHS investigators estimated the relative risk of CHD at 0.61 (95% CI: 0.52–0.71), with a relative risk of 0.55 (95% CI: 0.45 to 0.68) for women taking oestrogen alone and 0.64 (95% CI: 0.49 to 0.85) for those taking oestrogen plus progestagen. The same paper reported the relative risk for stroke, with a relative risk between stroke and use of oral conjugated oestrogen alone estimated as 1.18 (95% CI: 0.95 to 1.46), and a 45% higher risk for stroke among women taking oestrogen combined with progestin than in those who had never taken hormone therapy (relative risk, 1.45 [95% CI: 1.10 to 1.92]). For pulmonary embolism, the NHS reported a risk of 2.1 (95% CI: 1.2 to 3.8) [15], and similar findings were observed in a large population-based case-control study [16] for pulmonary embolism and deep vein thrombosis, with a risk of 2.1 (95% CI: 1.4 to 3.2). For breast cancer (Table 2), a large meta-analysis [17] reported a relative risk of 0.99±0.08 for oestrogen-only users for less than five years and 1.15±0.17 for oestrogen-plus-progestagen users for less than five years. For use longer or equal to five years, the respective risks were estimated as 1.34±0.09 and 1.53±0.33. For colorectal cancer, a meta-analysis [18] reported a relative risk for HRT users of 0.66 (95% CI: 0.59–0.74).

Finally, for dementia, a meta-analysis [19] reported a risk reduction of 29% (relative risk 0.71, 95% CI: 0.53–0.96). Overall, benefits were found for fracture, colorectal cancer, coronary heart diseases, and dementia – while there was an estimated increased risk of stroke, venous thromboembolic diseases –, and for long exposure to breast cancers.

The results of both parts of the HRT WHI trial are presented in Table 3. For some endpoints, detailed papers are available for both trial parts (fracture [20,21], coronary heart disease [22,23], stroke [24,25], dementia [26,27], and breast cancer [28,29]); in that case, the results reported here come from these papers. For other endpoints, the oestrogen only trial is not published in a detailed manner (venous thromboembolic disease and colorectal cancer). In that case, I reported the values of the main paper for both trials [3,4].

In terms of benefit, exposure to HRT reduced the risk of fractures, including hip fractures, similarly with oestrogen only and with oestrogen plus progestin. For colorectal cancer, there was a beneficial effect with oestrogen plus progestin, but not with oestrogen only. The risk of ischemic stroke was increased similarly with oestrogen only and with oestrogen plus progestin. The risk of venous thromboembolic disease was increased similarly for deep vein thrombosis and pulmonary embolism, with a slightly higher risk with oestrogen plus progestin than with oestrogen only. The risk of breast cancer was significantly increased only with oestrogen plus progestin, but not with oestrogen. For coronary heart disease, there was an increased risk with oestrogen plus progestin, but not with oestrogen. The risk was increased in the first year of use with oestrogen plus progestin but not with oestrogen. No trends were evidenced for age at enrolment or years since menopause in the oestrogen plus progestin trial. For the oestrogen alone trial, there was a trend with the age at enrolment, with a risk estimated at 0.63 in women aged 50–59 years, 0.94 in women aged 60–69 years and 1.11 in women aged 70–79 years. For years since hysterectomy, there was also a trend, which was not linear (1.13 for women enrolled less than 10 years after hysterectomy, 0.66 for women enrolled less than 10–19 years after, and 1.09 for women enrolled more than 20 years after). Finally, for dementia, in a sub-trial limited to women above 65 years of age without dementia at baseline, the pooled estimate of risk in both parts of the trial was 1.76 (95% CI: 1.19–2.60).

Compared to what was known, results from the oestrogen-plus-progestin trials were surprising for CHD and dementia. The results of the oestrogen trial were less commented. No effect was seen for CHD in this trial. Surprisingly, no increased risk of breast cancer was evidenced, and there was even a trend toward a reduced risk.

A meta-analysis submitted prior to the first publication of the oestrogen plus progestin WHI trial, but published after, re-examined the evidence available on the risk of CHD [30]. In this paper, studies of poor quality, studies with angiography endpoints, and cross-sectional design studies were excluded. Further, the authors found that a cause of heterogeneity between the studies was linked with whether or not the analysis was adjusted on socio-economic status, alcohol consumption, or exercise. Their pooled estimate (Table 1) was 0.87 (0.82–1.21) overall and 1.07 (0.79–1.48) when considering only studies that were adjusted on socio-economic status. For instance, in a good-quality case-control study [31], in which the analysis was adjusted on socio-economic status and alcohol consumption, the estimated relative risk was 0.9 for ever use of unopposed oestrogen (95% CI: 0.7–1.2), and there were insufficient data to evaluate oestrogens taken together with progestagen.

Other confounding effects may also have played a role. Up to the mid 1990s, HRT was contraindicated for women with hypertension, diabetes, and cardiovascular diseases and therefore opening the door to a huge indication bias, unlikely to be well controlled by usual adjustment techniques in most observational studies. In addition, HRT users tend to be healthier than non-users, more educated, with higher social class, leaner, with more positive health behaviours, better levels of several cardiovascular risk factors and better compliance. Better compliance to drugs is a well-known factor of good prognosis.

In WHI, the increased risk of CHD in the oestrogen-plus-progestin treatment was concentrated in the initial year after starting hormones. In the NHS, exposure data were updated biennially. Therefore, duration of exposure was underestimated by one year and this may have contributed to the miss of the early increased risk. In addition, women having already gone through the menopause in 1976 were considered in the analysis, which therefore included prevalent users in addition to incident users. As pointed out by Ray [10], this practice can cause two types of bias, both of which plausibly may have contributed to the discrepancy between observational and randomized studies. First, prevalent users are ‘survivors’ of the early period of pharmacotherapy, which can introduce substantial bias if risk varies with time, just as in studies of operative procedures that enrol patients after they have survived surgery. Second, covariates for drug users at study entry often are plausibly affected by the drug itself, which may introduce confounding. To account for the impact of time-dependent confounding factors, which is linked with the probability of initiating HRT and with the risk of CHD or of other endpoints and is also affected by HRT, and therefore intermediate on the causal pathway from HRT to the endpoint, the usual approaches, such as stratification or regression models may lead to biased results. To overcome this problem, Robins et al. introduced marginal structural models [32,33]. First, the probability of each subject to be treated at each time is estimated, using logistic regression. Second, the weight of each subject, defined as the inverse of each subject's probability of his or her treatment history at each time is derived. Then, a weighted Cox model is fitted for the effect of treatment, controlling for baseline covariates, but not for time-dependent confounders. The weighted analysis creates a statistical population in which the probability of being treated at each time is unrelated to the measured prognostic factors (the time-dependent confounders). The validity of the method depends on all prognostic factors of treatment initiation being recorded. J. Robbins presented a reanalysis of the NHS using this approach in 2004 and showed that the results were then much closer to that of the WHI trial for the risk of CHD [34].

Some authors argued that women who received HRT in real life are much younger that women enrolled in the WHI HRT trials. However, the analysis by age at enrolment and by time since menopause or hysterectomy (Table 2) did not support the view that this might explain the difference. Finally, in sub-group analysis reported in [22], no interaction was seen with the factors studied, including CHD risk factors. In others words, the increased risk was the same in women with and without risk factors, including overweight. These results dismiss the argument of those who consider that we cannot use the results of the WHI trial for women with lower numbers of CHD risk factors.

Of note, a common belief that was used to justify the protective effect of HRT is that risk in women is held low only until the menopause, around age 50 years, when it rebounds, equalling, and later surpassing that in men [35]. This is just not supported by mortality/morbidity data, as shown by Tunstall-Pedoe [36], who evidenced that the risk of death from CHD by age in the United Kingdom is parallel in men and women. Myocardial infarction incidence data from the three French Monica registries in France exhibited a similar trend. There is no change in the age incidence trend in women around or after 50 years of age, and relative risks in women compared to men are similar around 35–39 years (4.7), an age when almost no women has gone though the menopause, and around 60–64 years (4.4), an age when almost all women have, as can be seen in Fig. 1.

Overall, most of the difference between observational studies and WHI trials can be explained by default in analysis such as omission of confounders, bad adjustment for time-dependent confounders, and no good account of the function of risk overtime since initiation of treatment in the design and analysis of the studies. The performed analyses were influenced by the belief that the impact of HRT on CHD was mainly mediated by their positive impact on the lipid profiles, therefore the early impact of HRT was not strongly considered and accounted for in many analyses. It is also quite likely that similar problems explained the difference seen between observational studies and the two WHI HRT trials for dementia risk.

This does not mean that we know all what is needed to take decision as to whether or not use HRT. For instance, the transdermal route of oestrogen administration could have a different impact on the CHD or ischemic stroke risks, as is plausible for venous thromboembolism [37], and deserves further studies.

Since the publication of the first WHI trial, a large study, the Million Women Study (MWS), reported the risk of HRT for breast cancer [38]. This study enrolled women attending mammography screening who fulfilled a questionnaire relative to HRT exposure and potential confounders. The cancer status was evaluated through cancer registries covering the UK population. Because of its size (over 800 000 women included in the analysis), this study allowed one to study the role of different oestrogens and progestagens, as well as different routes of administration. This study, as the WHI trials, showed that the risk associated with exposure to oestrogen plus progestagen is higher than the risk of oestrogen-only regimen. However, the risks were slightly higher in MWS than in the WHI trials (Table 2). For instance, the risk was 1.24 overall in the E+P WHI trial, while it was 2.00 in the MWS. One may however wonder if the best summary of a trial is the relative risk from the intent to treat (ITT) analysis when considering a harmful effect. One may hypothesize that the observational estimates are closer to the on-treatment risk than to the ITT risk. The on-treatment risk for the E+P WHI trial is 1.49 (instead of 1.24 for the ITT risk). On the other hand, for the E WHI trial, no risk was evidenced, and even there was a trend to a beneficial effect. In that case, it might be more prudent and more conservative to keep the ITT risk (0.80) rather than the on-treatment risk (0.67). These results are nonetheless at variance with the estimated increased risk in the MWS (RR of 1.3 for oestrogen-only exposure). Therefore, it remains unclear whether or not the risk is truly augmented after exposure to oestrogen only.

In the MWS study, in oestrogen-only users, the risks were similar by constituent, by dose and by route of administration. Similarly, in oestrogen-plus-progestagen users, no difference was found according to the progestagen constituent or the regimen type (sequential or continuous). One progestagen, which is in common use in France (micronized progesterone), was evaluated neither in the WHI trials nor in the MWS or in recent literature in Nordic countries. In fact, there is only one study reporting separates estimates for micronized progesterone versus progesterone or testosterone derivatives [39]. One may first note that this study, E3N, is the only one using the new users' design [10], and is therefore more likely to be less prone to the confounding described in Section 2 of this paper, although limited at the moment by a short mean follow-up given the new user design (5.8 years). In this study, as shown in Table 2, the risk for exposure to oestrogen only was 1.1 (95% CI: 0.8–1.6) for a mean exposure duration of 2.4 years. For exposure to oestrogen plus progestagen, there was a difference according to the progestagen type, the risk for oestrogen plus micronized progesterone was estimated as 0.9 (95% CI: 0.7–1.2) after a mean exposure of 3.0 years, while for oestrogen plus either progesterone or testosterone derivatives, it was estimated as 1.4 (95% CI: 1.2–1.7) after a mean exposure of 2.9 years. It should be noted that these results are prettily similar to those of the WHI trial, apart for the risk of micronized progesterone, which was not evaluated in the WHI trials, which is quite reassuring for the quality of analysis and design to limit bias and confounding in this study. Overall, with only one relatively short-term study evaluating micronized progesterone, it is difficult to conclude firmly on the potential difference according to the progestagen type with available data, and on whether a less risky regimen can be recommended. Micronized progesterone is also in use in Belgium, Germany, Italy, Spain, and Switzerland, and, hopefully, studies from these countries will help understanding the E3N results.

Of note, both in the WHI trials and in the MWS, breast cancer risks were independent of age and body mass index. Therefore, for this endpoint also, arguments tuning down the results of these studies based on differences between real-life HRT users in France are not scientifically based.

Finally, recent ecological evidence in the USA showed both a dramatic fall in HRT use and a subsequent decline in breast cancer incidence [40], and data from other countries are awaited to confirm this decline.

HRT does not prevent many diseases and prolong life, but is efficacious for menopausal symptoms. A HRT, which could potentially be associated with a much lower overall risk such as transdermal route for oestrogen and micronized progesterone, still needs to be properly evaluated. In addition, the fact that, in the E+P WHI trial, all risks were at least similar or higher than those observed in the E WHI trial, apart from the risk of colorectal cancer, raised some concerns on the safety of progestagen use, namely progestagen used during perimenopause, a well-developed practice in France.

On a more general level, observational data should not replace clinical trials. Cohort studies and clinical trials have complementary roles in the evaluation of treatment. Cohort studies are very useful to assess rare long-term adverse effects of treatment that cannot be evaluated in clinical trials. The main problem with using cohort data to evaluate the effect of treatments is linked with confounding by indication and it is therefore critical to evaluate thoroughly the care taken to account for bias in observational data design and analyses. While validating the proper analysis of observational data by comparison with trial results is of interest, the main focus of such analyses should be on questions not answered, or not answerable by trials. Some recently presented methods, such as marginal structural models and new user design, may help to produce less biased estimates of the treatment effect from observational data, particularly in cases of time-dependent confounding, when no trial data is available.