Trial & Error

Posted on by Mark

The recent impact of the long trial on the benefits of HRT has again rasied the whole  question of clinical trials.

The report: “Schierbeck LL, Rejnmark L, Tofteng CL et al, Effect of hormone replacement therapy on cardiovascular events in recently postmenopausal women: randomised trial BMJ 2012; 345:e6409″ found that after ten years of randomised treatment, women receiving hormone replacement therapy early after the menopause had a significantly reduced risk of mortality, heart failure or myocardial infarction, without any apparent increase in the risk of cancer, venous thromboembolism or stroke.

The trial was an “open label, randomised trial” of 1006 healthy women.

What does all this mean?  It might be useful to describe clinical trials and the limitations of various types of trial.  Any trial seeks to gather evidence about the effectiveness or otherwise of medical treatments.

Advice about any treatment for a patient should be “evidence-based”.  What does this mean?

For instance, any new drug or treatment is not licensed until evidence has been produced that it works.  Gathering evidence is a complex matter and there will be frequent use in the field of diabetes of words such as “observational trials”, “controlled trials” and “randomised trials” and some of these terms will be explained now.

Clinical Trials are essentially tests which seek to provide evidence of the effectiveness or otherwise of interventions. There are two basic types of trials – observational and controlled.

i] Observational trials
These take a group of patients either in prospect or in retrospect and observe what happens or happened.   A good example was the American Nurses Trial.  A large number of nurses were observed over a period of time and divided into those that took hormone replacement therapy (HRT) and those that did not. It became apparent that those who took HRT suffered fewer myocardial infarctions than those who did not take HRT.

What became clear later pin-pointed the major defect of observational trials, namely that those nurses who did not take HRT were actually more at risk of coronary heart disease for other reasons.  Observational trials should therefore always be treated with extreme scepticism.

ii] Controlled trials
These are always prospective. Patients are divided into two groups, half of whom are given an intervention and half are not. The sample is normally ‘randomised’ by dividing them into groups in a truly random manner.

If appropriate, the intervention (usually in drug trials) may either be:
a]    single blind – the researcher knows which intervention is being taken by the patient
b]    double blind in which neither the researcher nor the patient know which intervention is being used
c]    open label where both the researcher and the patient know which intervention is being used

The final term that can be used is ‘cross-over’. In this case, a drug is given to one half of the sample and a placebo to the other half. The benefit is measured and the samples are exchanged with the first half being given the placebo and the other half the active drug.

Hence the most sophisticated but complex drug trial is the “randomised controlled double-blind cross-over trial”; this is often referred to by the abbreviation RCT (randomised controlled trial).

The recent HRT Trial was an open label randomised controlled trial.  It was plainly not appropriate to use the cross-over technique nor, for ethical reasons, was it allowable to make the trial a blind one.

One final thing that should be said about trials is about the size of the sample and the duration of the trial.  A trial should run for a long enough period for significant conclusions to be drawn and the sample should be large enough to enable results which are statistically significant to be achieved.

In connection with this, there are two other terms that you might come across and need explaining:

i]    Meta-analysis – It sometimes happens that there may be several small trials into a particular topic, none of which is large enough to drawn a statistically significant conclusion.  However, if these trials are all pooled together, the n umber may be large enough to draw conclusions.

ii]    Review articles – These are where an expert looks at several trials and attempts to draw overall conclusions about therapies.

A useful example to study might be drawn from the field of diabetes.   In this condition, complications take a long time to develop so any study running for less than ten or even fifteen years cannot enable appropriate conclusions to be drawn about the effectiveness of the therapy that is under investigation.  Likewise only a few patients develop specific complications such as nephropathy.  This is why the Diabetes Control and Complications Trial (DCCT) and the United Kingdom Prospective Diabetes Study (UKPDS) are so well respected.  They both had the ‘power’ to show what they set out to show.

On the other hand when some experts began to question the side effects of the drug, rosiglitazone, no trials were large enough to draw significant conclusions.  However, when a meta-analysis was performed, the conclusions were significant enough to result in the eventual withdrawal of the drug.

Problems with Trials

One of the pitfalls that experts repeatedly fall into is assuming that because an effect is observed with one group of people, it will also occur with another.

In the field of HRT, the Women’s Health Initiative made certain findings in women who were more than ten years post-menopausal.  It was assumed that the same things would happen in those who were newly post-menopausal or peri-menopausal.  We now know that this is not the case.

Other “experts” said that because side-effects had occurred with rosiglitazone, they would also occur with pioglitazone.  There was absolutely no evidence to assume this.  The only reliable evidence to come out of this is that some so-called “experts” are not expert at all.

At least some lessons were learned from this debâcle.  When simvastatin came under the spotlight recently, experts were able to say that the problems with simvastatin did not also happen with atorvastatin but the beneficial effects of simvastatin did.

This sort of thing is what is being referred to when effects of a drug may or may not occur with other similar drugs.  A class effect or not, that is the question.

Looking up References

When you read books or articles about medical subjects, you will come across references to some other papers.  A full reference will usually take this form: Mather H.M., Keen H. 1, The Southall diabetes survey: prevalence of known diabetes in Asians and Europeans. 2 Br Med J, 3 1985; 4 291: 5 1081-4. 6
1    Authors
2    Title of paper
3    Journal
4    Year of publication
5    Volume of journal
6    Page numbers

Reading papers

Scientific papers can look a little forbidding at first glance.  Because they are a means of communicating precise scientific information, they are written in a sort of code which aims to be objective and unambiguous rather than readable.  Unfortunately, most papers adopt a deliberately dry and convoluted style of writing.

The important thing to remember is that the author’s aim in writing up a piece of research is to enable a fellow researcher to reproduce that research, so that the results can be validated by more than one scientist.  This means that the paper has to cover details of the research methods and techniques in some detail.

What you are interested in is the results, which means you don’t always have to read every word of a paper to discover its basic findings.  Conveniently for those who just want to digest papers rapidly, they follow a set pattern.  The summary/abstract does what it says it does, summarising the major findings and how they were obtained.

i]    The introduction explains the thinking which underlines the research and sets out the hypothesis to be tested.
ii]    The methods section explains how the trial or experiment was conducted – how patients were selected, how many they were, how long they were treated for, what tests were performed on them, how placebos were used etc.
iii]    The results section gives all the findings, often in tabular or graphic form.
iv]    The discussion/conclusion offers an interpretation of the results and a conclusion as to whether or not the hypothesis has been confirmed or disproved.
v]    It is always worth checking whether any particular organisation has sponsored the research project.  Unfortunately some research papers are not worth the paper they are written on because the projects they cover are financed by pharmaceutical firms attempting to promote a drug, or the methodologies involved.

Helpful terms

When reports of trials are published, they contain tables of statistics which are not easy to understand and their relevance to every day practice may be difficult to appreciate.

To enable research findings to have more relevance to primary care, two additional terms have been used in an attempt to indicate more clearly the frequency with which events occur or the usefulness of an intervention to prevent it. In all these examples figures are quoted in italics.

i]    Numbers needed to Treat (NNT)
This is defined as “the average number of people from a defined population you would need to treat with a specific intervention (eg drug) for a given period of time (often five years) to prevent one major deleterious life event or achieve one beneficial outcome” – obviously, this includes death!

ii]    Numbers Needed to Harm (NNH)
This is defined as “the average number of people from a defined population you would need to treat with a specific intervention (e.g. drug) for a given period of time (often five years) to cause one additional adverse outcome” obviously, this includes death!  This figure is not often used but can be important if a drug treatment has a potentially serious side-effect – aspirin might be an example.

iii]    Quality Adjusted Life Years (QALY)
This is a measure of the effectiveness of therapies which seeks to combine improvement in life expectancy with the quality of the extended life into a theoretical index.

When all the learned evidence is gathered “guidelines” may be produced.  Sometimes we seem to be drowning in a sea of guidelines, targets and frameworks.  In the field of diabetes, the number has progressed (or deteriorated whichever way you look at it) well into double figures.  What are they for ?   Can any simple sense be made out of them?
The answer to the last question is a simple one – Yes – but not easily.  Whatever the name that is assigned (and for the time being let it be guideline), the aim of all guidelines is to improve patient care.

If a patient is diagnosed with diabetes, we know that they are liable to develop complications but that these can be prevented by good control of the diabetes and other things such as blood pressure.

The basic questions are quite simple:

Research has proved that the lower the overall blood glucose level, the less likely are complications.  However, if the level is too low, unpleasant hypos occur which a patient will not be prepared to put up with.  Therefore guidelines can be drawn up to indicate to professionals what course ideal management should take.  Eventually, the guidelines become so undisputed that pressure is applied to ensure that all professionals comply with them.

The process of drawing up guidelines therefore takes this form :

  1. Research is carried out
  2. Evidence from research is published
  3. Evidence is analysed
  4. Recommendations are made
  5. Guidelines are developed
  6. Targets are set
  7. Pressure to attain targets is applied!

Guideline recommendations

When guidelines are produced, it has become convention to refer to the quality of evidence which leads to the various recommendations,  This convention has now been standardised into five “Levels of evidence” leading to four “Grades of recommendation” and one “Good practice points” as follows :

Levels of evidence
1++    High quality meta-analyses, systematic reviews of randomised controlled trials (RCT), or RCTs with a very low risk of bias
1+    Well conducted meta-analyses, systematic reviews, or RCTs with a low risk of bias
1-    Meta-analyses, systematic reviews, or RCTs with a high risk of bias
2++    High quality systematic reviews of case control or cohort studies or high quality case control or cohort studies with a very low risk of confounding or bias and a high probability that the relationship is causal
2+    Well-conducted case control or cohort studies with a high risk of confounding or bias and a moderate probability that the relationship is causal
2-    Case control or cohort studies with a high risk of confounding or bias and a significant risk that the relationship is not causal
3    Non-analytic studies eg case reports, case series
4    Expert opinion

Grades of recommendation
A    At least one meta-analysis, systematic review, or RCT rated as 1++ and directly applicable to the target population or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population, and demonstrating overall consistency of results
B    A body of evidence including studies rated as 2++, directly applicable to the target population, and demonstrating overall consistency of results or extrapolated evidence from studies rated as 1++ or 1+
C    A body of evidence including studies rated as 2+, directly applicable to the target population and demonstrating overall consistency of results or extrapolated evidence from studies rated as 2++
D    Evidence level 3 or 4 or extrapolated evidence from studies rated as 2+

Good practice points
- Recommended best practice based on the clinical experience of the guideline development group

Who decides ?

The whole process of research and the production of guidelines is very much an ad hoc procedure.  The basis for research products is largely the result of one person or a group of persons deciding that they want to know the answer to a particular question.

The consequent production of guidelines and frameworks is an equally random business.  Probably what has happened with the majority of initiatives in the field of diabetes has been the result of experts perceiving that diabetes is not managed as well as it might be with a resultant effect on patient care.

Who are the experts who decide ?  This is difficult to say, in many cases the same names occur again and again.  Even today, guidelines often have few experts from the field of primary care medicine on their panels.

Some of this article has been reproduced from our distance learning packs. Find out more on one of our university-accredited Distance Learning Courses.