CARRS-Q Research – Part Three

Photo Credit: Dr Paul Martin

Why you should dismiss the CARRS-Q bicycle helmet research

Other Research CARRS-Q Used

This is the third post in a series (Read Part One, Part Two) looking at the non-peer reviewed CARRS-Q publication entitled “Bicycle Helmet Research“, published in 2011 and widely relied upon to support mandatory helmet laws for cyclists.  This publication was commissioned by the State Government to support its policy of mandatory helmet laws in Queensland, Australia, in the face of criticisms of its lack of support for Brisbane’s public bike hire scheme.

Part Two of this series discussed the failings of the Cochrane Review, a review of pre-existing research supporting bicycle helmet efficacy, which was in turn heavily relied upon by the CARRS-Q publication to support the Queensland Government’s policy of mandatory helmet laws for all cyclists.

This post considers other bicycle injury related research utilised by the CARRS-Q publication, highlighting some of the failings of that research and demonstrating how that original research has been misinterpreted by CARRS-Q to support mandatory helmet laws.

The Canadian Report

As well as the Cochrane Review, the CARRS-Q publication relies on Ontario-based research to conclude that helmets greatly reduce the likelihood of bicycle related head injuries (Rowe et al: Bicyclist and environmental factors associated with fatal bicycle-related trauma in Ontario, Canada Medical Association Journal 1995).

The CARRS-Q publication states that:

“Research from Canada examining bicycle fatalities found that the vast majority involved bicyclists not wearing a helmet (96%), although some fatalities did involve helmeted cyclists …” (CARRS-Q, 2011, pages 11 and 12)

Importantly, the CARRS-Q publication does not put this in context:  the Canadian report concerned only 212 cyclist deaths in Ontario when helmet wearing was not compulsory.  The report made no analysis of the helmet wearing rates of the general cycling community.  By quoting the research as finding 96% of fatalities were not wearing a helmet, CARRS-Q is misrepresenting this research as finding helmet use had some statistical relevance to the likelihood of injury, when in reality there is no indication in the Canadian report that wearing a helmet had any effect on the likelihood of death. The Canadian Report itself only concluded that:

“[b]icycle-related deaths result from factors that are generally avoidable.  Identifiable risk factors other than lack of helmet use suggest that additional research is required to determine the benefits of preventative interventions aimed at reducing the number of such deaths.  Age-specific strategies appear warranted.” (Rowe et al, 1995, page 45)

The Canadian report also stated that:

“In all, 193 (91%) of the events were collisions with motor vehicles… Such events were responsible for all the deaths involving children less than 10 years of age and 88% of the deaths involving those 10 to 19.” (Rowe et al, 1995, page 48)

It should be concluded from the Canadian report that a cyclist’s age has more to do with the likelihood of death than whether they are wearing a helmet.

The Canadian report also concluded that:

“Unfortunately, because of our rudimentary understanding of risk factors associated with fatal bicycle-related trauma, countermeasures or preventive interventions have been limited to the promotion of bicycle helmet use. In addition, most of the potential interventions have yet to be subjected to adequate scientific testing.” (Rowe et al, 1995, page 49)

The Canadian Report also found that:

“Most (91%) of the deaths resulted from collisions between motor vehicles and bicycles.  Age did play an important role in crash characteristics.” (Rowe et al, 1995, page 48)

Within a year of the Canadian study being published, Ontario introduced mandatory helmet laws only for children cyclists.  Not adult cyclists.  This was in accordance with those findings.

More recent research by CARRS-Q confirms that teenagers are risk takers.

The general effect of helmet use on injury is mentioned briefly in the Canadian report, with the Canadian report citing current estimates for head and brain injury reduction from helmet wearing.  However these estimates are attributed to earlier studies, specifically the third and fourth studies utilised in the Cochrane Review discussed in a previous post, including a report focusing solely on head injuries for children under the age of 15.

The Melbourne Study

The CARRS-Q publication also relies on a Melbourne study comparing head injury rates between helmeted and non-helmeted cyclists, and to the first object struck by cyclist in a crash.  (McDermott et al: Bicyclist head injury prevention by helmets and mandatory wearing legislation in Victoria, Australia, Annals of the Royal College of Surgeons of England, 1995)

According to the CARRS-Q assessment of this study:

“When fatalities were considered, helmet wearing cyclists had fatality rate of 0.4% while unhelmeted riders had a fatality rate of 0.9%. However this finding was not significant.” (CARRS-Q, 2011, page 12)

While noting that head injuries were less frequent, and less severe, for helmet-wearing bicyclists, the CARRS-Q publication also noted that the frequency of neck injuries was significantly increased by helmet wearing.

The Melbourne study relied on a previous study co-authored by the same author for its comparison of head injury rates between helmeted and non-helmeted cyclists. (McDermott et al: The effectiveness of bicyclist helmets: a study of 1710 casualties, J Trauma 1993).

This earlier study was one of the studies relied upon for the Cochrane Review and discussed in a previous post.  This previous study found helmeted cyclists were more likely to collide with motor vehicles than non-helmeted cyclists, and that there were no significant differences in mortality rates between helmeted and non-helmeted cyclists, and which also identified an increase in neck injuries attributable to helmet use.

The Melbourne study referenced by the CARRS-Q publication also found that total cyclist fatality numbers decreased from 77 for the three years prior to the introduction of mandatory helmet legislation in Victoria, to 41 for the first three years after the introduction.  However, importantly, this study does not take into account the reduced numbers of cyclists during this period, or which age groups had the greatest reductions in cycling rates. These reduced numbers are discussed in the next post.

The Melbourne study referenced by CARRS-Q also noted its results resembled the results of a Seattle study and a study focussing on child bicyclists.  However these studies were the very same studies already discussed in the previous post, the third and fourth studies relied upon for the Cochrane Review.

In short, as well as the actual findings of these studies differing somewhat from the conclusions drawn from them by CARRS-Q, a lot of the claims that these studies support each other’s findings are spurious given that the studies either reference each other or reference the same sources.

The New Zealand Report

The CARRS-Q publication also refers to a New Zealand study that indicates helmet wearing decreased head injuries for non-motor vehicle related crashes.  (Povey et al: Cycle Helmet Effectiveness in New Zealand, Accident Analysis and Prevention, 1999)

The CARRS-Q publication states of the New Zealand report that this reduction of head injuries:

“ … was evident across age groups, being strongest among children of primary school age.  This trend was also found for bicycle crashes involving a motor vehicle, although the effect was not as great.” (CARRS-Q, 2011, page 12)

This New Zealand study only differentiated age groups for non-motor vehicle related crashes. The study was also commissioned by the Land Transport Safety Authority (LTSA), which was the government agency responsible for the introduction of mandatory helmet laws in New Zealand 4 years prior.

This New Zealand study concluded that:

“The relatively large increase in helmet wearing associated with the passing of a compulsory helmet wearing law in 1994 [in New Zealand] reduced head injuries by between 24 and 32% in non-motor vehicle crashes, and by 20% in motor vehicle crashes.” (Povey et al, 1999, abstract)

This reduction is significantly lower than CARRS-Q’s claim that helmets reduce likelihood of head injury by 69%.  This New Zealand study also concluded that:

“No increase or decrease in the severity of head injuries for which cyclists were hospitalised over this period could be detected.  This may have been due to the small and highly variable number of ‘high severity’ injuries.” (Povey et al, 1999, abstract)

This New Zealand study also identified significant disparity in head injuries across different age groups:

“For an increase in 5 percentage points in the helmet wearing rate, the corresponding decreases in head injuries in non-motor vehicle accidents were estimated to be 10.2, 5.3 and 3.2% for children of primary school age (5-12 years), secondary school age (13-18 years), and adults respectively.” (Povey et al, 1999, abstract)

In line with the Canadian report discussed above, there were significant differences in injuries depending on age.

The findings of this New Zealand study has been heavily criticised due to anomalies between voluntary uptake in helmet use prior to mandatory laws, and mandatory helmet use:

“The pre-law increase in adults wearing helmets (from 30% in 1990 to 43% in 1993) was accompanied by a fall of 45 head injuries per 100 limb injuries (i.e. -3.47 for every 1% increase in helmet wearing) compared to a fall of just 11 head injuries when helmet wearing increased from 43% to 93% with the law (-0.23 for every 1% increase in wearing). Unless voluntary wearing is 15 times more effective in reducing head injuries, it seems likely that the claimed 20% – 32% falls in head injuries with increased helmet wearing were, in reality, an artefact caused by failure to fit time trends in their model.” (Robinson: Changes in head injury with the New Zealand bicycle helmet law, Accident Analysis and Prevention, 2001).

A more recent study (covered here), also focusing on the effect of helmet use in New Zealand, found that with inclusion of neck injury, helmet use has no net protective effect on cyclists’ injury.  (Elvik: Publication bias and time-trend bias in meta-analysis of bicycle helmet efficacy: a re-analysis of Attewell, Glase and McFadden, Accident Analysis and Prevention, 2011).

This more recent study is also consistent with the Melbourne study discussed above, on the increased risk of neck injury from helmet wearing.

The Queensland Report

The CARRS-Q publication (p16) also claims mandatory helmet laws have been proven not to reduce head injuries simply because they reduce the average number of cyclists.

The CARRS-Q publication relies on research that compared the reduction of Queensland cyclists’ head injuries after the introduction of mandatory helmet laws, with the reduction of other types of injuries suffered by Queensland cyclists.[1] (King et al: Bicycle helmet legislation and enforcement in Queensland 1991-1993: effects on helmet wearing and crashes, Road User Behaviour Section, Road Transport and Safety Division, Queensland Transport, 1994).

According to CARRS-Q, this Queensland report found a decrease in the rate of cyclists’ head injury compared to other types of reported injury that:

“ … could not be explained as an effect of less cycling.” (CARRS-Q, 2011, page 16)

This Queensland report was written by two employees of the Queensland State Government’s Queensland Transport Department in 1996, including a manager of the Department’s Road Transport Safety Division.  That manager is also one of the four authors of the CARRS-Q publication.

The report was written for presentation at the Third International Conference on Injury Prevention and Control, held in Melbourne in February 1996.  This report was one of a number of pro-helmet presentations amongst a broader programme of social injury prevention presentations.

This report was not peer-reviewed.  It is also important to consider whether the report’s presentation at the Conference should give this report any academic authority. The Conference director has said of this Conference that:

” I saw our intentions to schedule only excellent and innovative papers come falling around our ears.  The reason? A large proportion of submitted papers consisted of injury incidence surveys.  Many had intervention recommendations bearing no relation to the actual study, simply tacked on to the end.  Others were program descriptive papers, many without rigorous evaluation.  … In the end fiscal pragmatism won out – we found a place in the program for all authors in the most creative way we could.”

The majority of the Queensland report focusses on documenting the introduction of mandatory helmet laws in Queensland, and the subsequent enforcement by police.  The reports provides data about the number and type of fines issued by police for infringements of the legislation, and the results of some observational surveys in and around schools and other areas, about helmet wearing rates.

Notably, the report identifies that the number of fines issued for failing to wear a bicycle helmet in 1993 were similar to the number of drink driving offences detected.  It also states that the penalty was set at $30 as this was the approximate cost of a helmet.

The report also presents police crash data from January 1991 to September 1993, broken down into two levels of severity: fatal or hospital treatment being classified as severe, and other medical treatment being classified as less severe. This data is also presented in three categories: before legislation, after legislation but before fines were introduced, and after fines were introduced.  Here is how the data was presented in the report:

The report concludes from this data that:

“… there was clearly a drop in crashes in the first quarter after helmet wearing became compulsory, followed by a rebound to what appears to be a lower quarterly crash rate than previously.”

If one charts the above report, the injury trends appear as follows (with the vertical lines indicating the introduction of helmet laws, and the later introduction of fines for enforcement):

Cyclist crashes by severity: Fatal or hospital treatment compared to other medical treatment (as per King et al)

However a different picture of helmet efficacy emerges if these 1990-1993 figures are merged with similar statistics presented on page 26 the CARRS-Q publication, categorising the latter in the same way as the Queensland publication:

Cyclist crashes by severity: Fatal or hospital treatment compared to other medical treatment (1990 to 1993 as per King et al); Fatalities and hospitalisations compared to medical treatment and minor injury (1994 to 2008 as per CARRS-Q)

Importantly, the above data does not differentiate between head injuries and other injuries.

The Queensland report agreed that it could be argued that the above decreases could be explained solely by reductions in cycling.  The report addressed this by presenting data collected by the Queensland Injury Surveillance and Prevention Program, from a sample of hospitals in the southern metropolitan area of Brisbane.  This data was also categorised as treated (being less serious) and admitted (assumed to be more serious) and also broken up into before legislation, after legislation but before fines were introduced, and after fines were introduced.  Actual data was not presented, just mean quarterly averages:

From this data, the report concludes that:

“The figures show that in every comparison the decrease in head injuries among cyclists was greater than the decrease among injuries to other parts of the body, which suggests that bicycle helmet legislation has been effective in reducing head injuries among cyclists.”

It is clear that the data was, at best, not broadly representative of the Queensland crash and injury statistics.  The CARRS-Q publication itself states that:

“When examining bicycle crash patterns using police-reported data, it is important to acknowledge some of the limitations associated with the data.  Research indicates that there is significant underreporting of bicycle crashes …” (CARRS-Q, 2011, page 25)

In fact the research that CARRS-Q goes on to quote estimates that police data could represent as little as 3.5% of crashes, and over-represents the more serious bicycle crashes.

The hospital data represented above in TABLE IV used by the Queensland report is also limited to only a handful of hospitals, all within the Brisbane metropolitan region.  Additional later data, comparing head injuries to other types of injuries per year since the introduction of helmet laws, was not used by CARRS-Q.  Instead, CARRS-Q only compared the injury type between helmeted and non-helmeted cyclists.  The many problems with this approach are discussed in the last post of this series.

The data used in this Queensland report is also limited to just the three year period before, during and after the introduction of mandatory helmet laws. As demonstrated by the above discussions, limiting data to such a small period can present an unrealistic picture of the efficacy of helmet laws.

It would also appear from the CARRS-Q publication that this report does not take into account the effect of changes to the demographic of cycling caused by helmet laws.  In other words, as evidenced by other studies discussed above and in the next post, mandatory helmet laws deter cycling the most amongst adolescents and young children, which are the age groups most likely to have a collision with a motor vehicle (which in turn are likely to result in more serious injuries or death) or most likely to suffer head injury in a non-vehicle related accident.

A number of other researchers have also been highly critical of this Queensland study.  It has been said with classic academic understatement that a number of State–based studies designed to evaluate the efficacy of helmet legislation, including this Queensland study, were:

“ … less sophisticated in a statistical sense.”

(Hendrie et al, Road Accident Prevention Research Unit, Department of Public Health, University of Western Australia: An economic evaluation of the mandatory bicycle helmet legislation in Western Australia, Insurance Commission of Western Australia Conference on Road Safety, 1999).

One academic has pointed out that the reduction in percentage of head injury compared to other types of injury in the Queensland report was similar across all road user types, including pedestrians, not just cyclists.  This strongly indicates that cyclists’ head injury reduction was due to other road safety measures, not helmets. (Robinson: Is there any reliable evidence that Australian helmet legislation works? Velo Australia International Bicycle Conference, Freemantle, Western Australia, 1996)

It is also not clear without reading the original study whether, like a number of other studies relied upon by CARRS-Q, ‘head injury’ included all types of head injury including minor cuts, bumps and bruises.

This Queensland report also failed to consider any increases in neck injury after the introduction of mandatory helmet laws.

It also appears that the Government Department influenced how the findings of the Queensland report were presented in the CARRS-Q publication. The original draft of the CARRS-Q publication did not rely on this Queensland report for any findings about helmet efficacy, only for findings related to the increase of helmet use as a result of the helmet laws.  It was only after review of the draft CARRS-Q publication by Queensland Transport that the CARRS-Q publication was amended to use the Queensland study in this way.

Looking at the original data of police and hospital data presented in the Queensland report, here is the pattern of injuries

Despite CARRS-Q’s claims, the actual research referenced in the CARRS-Q publication indicates significant differences in helmet efficacy for children and adults. It also identifies motor vehicles as the biggest factor in cyclists’ head injury rates. Even the most minor reduction to cyclists’ exposure to collision with motor vehicles, will prevent far more head injuries than bicycle helmets could ever achieve. This could be done through driver education, awareness and better cycling infrastructure/road layout changes, speed limit reviews and so on, all of which will follow from greater cycling participation.

The next post in this series will look at a series of Victorian-based studies conducted by Monash University, also relied upon by CARRS-Q to support mandatory helmet laws.

---

[1] King, M., Fraine, g., Bicycle helmet legislation and enforcement in Queensland 1991-1993: effects on helmet wearing and crashes, (1994), Road User Behaviour Section, Road Transport and Safety Division, Queensland Transport.