Gun Possession and Assault With a Firearm: Risky Stuff — Or Not?
Over at Greg Laden’s blog, he has notice of a study done via epidemiological procedures to look at the relationship between injury and gun possession. The paper is titled “Investigating the Link Between Gun Possession and Gun Assault “.
Source: Branas, C., Richmond, T., Culhane, D., Ten Have, T., & Wiebe, D. (2009). Investigating the Link Between Gun Possession and Gun Assault American Journal of Public Health DOI: 10.2105/AJPH.2008.143099
The researchers concluded,
On average, guns did not protect those who possessed them from being shot in an assault. Although successful defensive gun uses occur each year, the probability of success may be low for civilian gun users in urban areas. Such users should reconsider their possession of guns or, at least, understand that regular possession necessitates careful safety countermeasures.
I started writing a comment at Greg’s blog, but it just kept going and going, so I’m making a post here instead.
I will start by saying that after looking it over pretty carefully, I am unimpressed with this study. It appears to turn a non-significant result about gun possession into headlines by a series of steps that cannot be replicated from the description in the methods.
I’ve read the full paper, and the logic or math is incompletely described by which they arrived at even their numerical results, much less the further conclusions that they take. Of course, I’ve had only a brief time where I was engaged in epidemiology research, and that was over twenty years ago, so I’ve done a bit of review, too. They used a case-control experimental design. Because most of their data is nominal, not numerical, they employ “conditional logistic regression”. They mention models, but provide no descriptions of the particulars of the models, nor any parameters. (cf. the section on “Statistical Procedures” in the Methods section, where they describe a series of models and regression analyses, but without specificity.) As far as I can tell, even if someone else had a similar dataset to work with, they would not be able to fully replicate the procedure taken in this paper simply from the paper’s description of its methods. The journal site does not have a link for supplemental materials, so there does not appear to be any more extensive description of data, methods, or results than is present in the paper itself.
Their dataset is only comprised of shooting incidents. This limits what question they may reasonably be said to have addressed, as I will go into more detail about later.
this in mind, we conducted a population-based
to investigate the relationship between being
injured with a gun in an assault and an individ-
both fatal and nonfatal outcomes and accounted
for a variety of individual and situational con-
founders also measured at the time of assault.
Even after these
exclusions, the study only needed a subset of
the remaining shootings to test its hypotheses. A
random number was thus assigned to these
remaining shootings, as they presented, to enroll
a representative one third of them.
If this struck you as an strange dataset to base conclusions about the odds of being shot being modulated by having a gun in your possession at the time of an attack, you are not alone. I’ll run through what the case-control method does. A case-control analysis seeks to determine how strong an association there is between a risk factor and an outcome. In epidemiology, this is put in terms of exposure to a risk factor and whether the person has a disease (the outcome). Cases are the instances of people with the outcome/disease. Controls are people without the outcome/disease. The standard way to split things up is in a 2×2 table.
Given numbers to put in for A, B, C, and D, one can then compute an “odds of exposure ratio” for cases (A/B) and controls (C/D), and an “odds ratio” for the risk factor as (A*D)/(B*C). The overall “odds ratio” is an estimate of the relative risk people who are exposed to the risk factor are at. If a higher ratio of people exposed to the risk factor have the disease/outcome than is seen in the ratio of people not exposed to the risk factor, then the odds ratio is greater than 1 and indicates an association of the risk factor with the disease or outcome. If it goes the other way, the ratio is less than 1 and likewise is an association, but indicating that the risk factor is somehow correlated with reduced incidence of the disease or outcome. (Description above based on the fine page here.)
The authors of the present paper were pleased to note that some early work tying smoking as a risk factor to lung cancer was done with case-control analysis. I wasn’t able to grab the early paper cited, but I did find a CDC worksheet that claimed to pass along their numbers. I’ll put that here for an example. It will be laid out somewhat differently, since the work showed multiple categories of exposure, so the “unexposed” numbers I will put in the top row.
|Cigarettes/Day||Cases (had lung cancer)||Controls (did not have lung cancer)||Odds Ratio|
OK, that lays out how a historic use of the case-control method was done, and in a fairly simple way. Part of the reason the smoking/lung cancer study was so influential, though, lay in the categorization with amount of tobacco use, which goes some way toward showing a dosage-response relation on top of the lumped odds ratio of a 9x relative risk of lung cancer with smoking. One reason the case-control method worked for this was that in the smoking/lung cancer association, there was a strong signal in the data. That comes through in the magnitude of the relative risk values. Keep that in mind as we come back to consideration of the present paper.
The present study is in no way simple. Nor are its methods as given clear enough to replicate. There is, though, a table (Table 1) summarizing the basic numbers they started with in the study. It is sufficient to get the raw odds ratios for many of the conditions that they took note of or adjusted for in the study. The remaining ones are given in units that don’t permit easy calculation in terms of ratios. So here are the odds ratios based on the unprocessed, unadjusted numbers from Table 1. (Note: I’ve had to do a lot of transcriptions by hand, so there could be typos or worse, errors. I’ll fix them as they are pointed out to me.)
|Risk Factor||All Shootings||Fatal Shootings||Chance to Resist|
|Illicit drug involvement||1.56||6.12||1.02|
|Occupation: Working class||0.521||0.629||0.507|
One thing that stands out as a huge difference between the case group and the control group is location. The case group (the folks who got shot) were outdoors in 83% of non-fatal shootings and in 71% of fatal shootings. The control group, by contrast, were outdoors at the same time only 9% of the time. That translates into very high relative risk, about 50x the control group by simply going outside. If the authors wanted an attention-grabbing lead, the “being outdoors” risk factor is what they should have played up. “Don’t Go Outside” could have been the headline, validating Philadelphia agarophobics.
For myself, I’d have expected that a dataset to do what was reported would have included assaults with a firearm that either resulted in a shooting of the victim or did not result in the shooting of the victim, and do whatever “adjusting” magic was needed to control for different conditions between those two cases to find the effect of the victim being in possession of a firearm at the time. But this is definitely not the way they’ve described their data and methods.
The fact that exactly the class of assaults where a victim had possession of a firearm and was not shot was excluded from this study would seem to me to be a large methodological hole in the research. That makes the following statement from the discussion all the more bizarre:
Although successful defensive gun uses can
and do occur,33,57 the findings of this study do
not support the perception that such successes
But it seems to me that this was precisely the question that was at issue, and the choice of methodology reduced the authors to this flabby and, so far as I can tell, poorly substantiated claim. Again, maybe I’m missing something here, so if someone could clarify this, I would appreciate it.
Another mysterious item from the discussion:
tively, an individual may bring a gun to an
otherwise gun-free conflict only to have that gun
wrested away and turned on them.
Why is it that this is only stated as a possibility, and not actually quantified? Surely this would have come out in the investigation of the non-fatal shootings, and also in a part of the fatal ones, and could have been presented at least as a raw number, if not analyzed as well to reduce the confounding factors. Given the full pool of over 3,000 shooting incidents that they took their dataset from, this should easily have been characterized as a proportion of the dataset at the very least.
There is no simple relation between the relative numbers and the resultant risk factors that are reported. Since case-control studies are supposed to work by highlighting how a treatment group differs from a control group, it is problematic to see just how the small, non-significant differences between these groups in the particular character of interest get expanded into the fairly large risk factors given in the paper. The raw, unadjusted numbers don’t paint gun possession as being generically risky; in fact, across all cases, it shows a slight association with less relative risk. So it is only through the opaque method of adjustment of confounding factors that the startling relative risk estimates for gun possession come about. That process has a lot to do with how the models mentioned are constructed, and that information is not available via the published paper.
If we assume that all “adjustments” are made to the control data, we can estimate just how much “adjustment” had to occur in order to arrive at the published relative risk numbers for the “gun possession” condition in the three different contexts. I’ll lay this out in a set of tables, too.
|Case (%)||Control (raw %)||Raw Odds Ratio||Adjusted Odds Ratio||Est. Adjusted %||Adjustment Factor||5.92||7.16||0.816||4.46||1.39||0.195|
|Case (%)||Control (raw %)||Raw Odds Ratio||Adjusted Odds Ratio||Est. Adjusted %||Adjustment Factor||8.8||7.85||1.13||4.23||2.14||0.272|
|Chance to Resist Shootings|
|Case (%)||Control (raw %)||Raw Odds Ratio||Adjusted Odds Ratio||Est. Adjusted %||Adjustment Factor||8.28||7.37||1.13||5.45||1.63||0.221|
These tables show that the models used for “adjustment” of the raw data, if applied to the “control” raw data, would have reduced each raw control value for “gun possession” to about one-fifth its original value. This seems pretty aggressive for a model only described in general terms. This information reinforces the point that the startling and headline-grabbing statements about “gun possession” being risky are not founded upon the raw data, but instead come from the model. In order to put trust in the findings, the model needs far more transparency than this paper delivers.
Going back to the conclusions given:
On average, guns did not protect those who possessed them from being shot in an assault.
It seems beyond the scope of the study to say anything about not being shot in an assault with a firearm. The question that the research addresses is limited by the data that was considered, and what the chosen dataset can address is the question, “Is the general population much different in the characteristic of ‘gun possession’ from a sample of people who were shot in an assault with a firearm?” The raw data clearly says “No, there is no significant difference between the groups,” while the model apparently gives the opposite answer with statistical authority. This should cause introspection on the part of the researchers, not broad statements claiming to have established a basis for a sweeping change in perception of the issue.
Methodologically, comparing gun assaults without shootings to those resulting in shootings could have addressed the issue of whether gun possession changed the risk of being shot in an assault, but no such comparison was attempted. This seems distinctly odd, since the research logistical burden would have been lower with that approach, and reducing the research burden was specifically mentioned as a reason for adopting the case-control approach. If the risks estimated with the adjusted data were accurate, comparing gun assaults without shootings to those with would have provided a simple way to independently corroborate the finding. On the other hand, there would be relatively little cause for “adjustment” of the data between those two cases, and it may not corroborate the modeling effort undertaken here. If the Philadelphia police could be prevailed upon to provide a set of numbers from gun assaults without shootings concerning the total number of such cases and the number in which the victim possessed a gun, it would be sufficient to use as the “control” group and provide an odds ratio for comparison to that in the published paper.
Although successful defensive gun uses occur each year, the probability of success may be low for civilian gun users in urban areas.
This is entirely speculative and nothing in this study addresses any quantification of successful defenses against assaults with firearms, much less anything that would rise even to a guess about probability.
Such users should reconsider their possession of guns or, at least, understand that regular possession necessitates careful safety countermeasures.
I think better safety countermeasures is a goal we can all get behind, but I didn’t need a study with opaque and not obviously functional methodology to tell me that. As to reconsidering possession of firearms, I think I’ll take that with a grain of salt until someone is able to explain how that conclusion really follows from the data and methods described. Right now, it looks like a big non sequitur leap to me.
As noted above, it seems that there is a fairly simple way to check the model against reality. If that is done and it validates the model, I’d be somewhat surprised, but I’d be satisfied on the methodological issues that a real result had been obtained. But in the absence of either a transparent model permitting replication of results or the independent check I outlined above, my impression of the study is that it is more a means for assumptions to be converted into conclusions than a solid piece of empirical work.
8 thoughts on “Gun Possession and Assault With a Firearm: Risky Stuff — Or Not?”
Your conclusions confirmed my worst fears about the study, and then some.
I’m not a huge fan of the NRA, but went to their site to hunt for data links. Their official position is that there a lot of dubious studies too easily published on the topic of gun safety. This study appears to be one that confirms the NRA’s assessment.
Thank you for your work on this well-written review. :)
I just finished a course on medical statistics and this caught my eye. I am not prepared to pay $30 to see the original paper, but I think your criticism may be unfair. The “adjustments” are presumably allowing for confounding factors. If people are more inclined to have a gun in their possession when they are inside than when they are outside, and most shootings take place when the victim is outside, then this will strongly bias the odds ratio towards guns preventing shooting. It is vital to adjust for this and other potential confounders. If this is the adjustment then it is a perfectly legitimate procedure and they would have been at fault not to have made the adjustment. Without paying the $30 I can’t see whether this is true or not.
It is also not so unreasonable to use citizens of the same city at that time as controls. It depends whether you want to know if carrying a gun affects your chances of being assaulted and shot, or just whether it affects your chances of being shot if you are assaulted.
To me a bigger concern would be whether they have allowed for all confounders. And of course, association is not causality. It does not follow that dispensing with your gun will reduce your risk..
I guess I’m not seeing offhand in what way I am being “unfair”. If it is supposed to be generically over the fact that I do make a criticism of the paper, I will have to disagree. Published research is open to criticism. If it is supposed to be more specifically about a part of my analysis, we can certainly argue about that. But I’m uncertain as to what part of what I wrote you might be objecting to.
Unfortunately, whether you spend $30 or not will not enable you to dispense with the criticism that their methods are not transparent. In the paper, they give a table of summary data, where the raw odds ratios give no basis for saying much of anything about how gun ownership might interact with being shot in an assault with a firearm. They say that they used a model to adjust for confounding factors, and that they used conditional logistic regression to obtain values to put into their model, but that’s about as much useful detail as you get in the paper itself concerning the “adjustment” itself. One could not, even if provided with the complete raw data set, use the methods section in the paper to replicate their analysis. You would be able to perform a conditional logistic regression, certainly, but how the results interact with the model they describe is indeterminate.
Certainly the particulars of how the “location” attribute interrelates with the “gun ownership” attribute could have a major impact on the way the numbers get “adjusted”. However, given that “location” is precisely the attribute with the greatest disparity between the case and control groups, it seems reasonable to take a major shift in the raw data based on a major adjustment due to such a factor as a reason to place less confidence in the sweeping nature of the conclusions drawn.
Now, why do you suppose that given the cooperation and access to the police records that the researchers had, that they chose not to utilize the information concerning the obvious control group, to wit, those people assaulted with a firearm but who did not get shot in the incident? Is it an unfair criticism to note that an even easier-to-access control group with far fewer potential confounding factors was entirely excluded from the study?
Let me start by saying this is mostly an exercise for me to test my understanding of the correct use of odds ratios.
I accept that they should be have been more explicit about their methods.
I do not accept that the large association between being shot and being outdoors reduces confidence in the association with gun possession. It does make it vital to allow for this as a potential confounding factor. The alternative would have been to thrown away all the data from either indoors or outdoors which would hardly have increased confidence.
It is not obvious to me that the control group should be those people assaulted with a firearm but who did not get shot in the incident. It depends what you are trying to measure – the probability of being shot given you have been assaulted, or the probability of being assaulted and shot. I believe the case for possessing a gun is includes reducing your chances of being assaulted in the first place – doesn’t it?
We’re probably going to have to agree to disagree, which I think is fine. If I recall correctly, the effect of the “adjustment”, if one then figures out what the raw numbers would have had to have been to yield the final odds ratios reported, are that of the 49 people in the control group who were actually gun owners, only 9 of them were admitted to the gun owner group after the “adjustment”. When forty out of forty-nine data points are being treated as if they were outliers, I think that does reduce confidence in the claims that are made on that basis.
You are absolutely right that what data you use defines the question that one is actually addressing. However, that’s one of my points of criticism of the study, since the authors cast their conclusions in terms of probability of being shot generically in an assault with a firearm conditioned on gun possession. Their data set does not allow them to make a statement on that question. Let’s review their statement:
The first sentence, even if one grants for the sake of argument that all their methods are spiffy, is undercut by the fact that they did not incorporate a presumably substantial pool of relevant cases, those people who were in possession of a gun but were not shot in an assault with a firearm. I say presumably because it is my naive belief that most assaults with a firearm do not actually result in a shooting, that most such assaults simply use the firearm for threat value. If I’m mistaken on this point, it would still be useful to get some quantification of just how big a group that is. However large or small the group, though, the authors are not justified in saying that they’ve addressed the question that they are providing the conclusion of the first sentence for. And I’ve already discussed the other sentences in my post above in terms of the disparity between what they studied and what they concluded.
“I believe the case for possessing a gun is includes reducing your chances of being assaulted in the first place – doesn’t it?”
Quick note… the case for possessing a gun is primarily a matter of constitutional law. Public health factors are not, strictly speaking, relevant to the issue in that context. Another point: legal use is called “concealed carry” for a reason, so if people are actually effective in concealing a firearm, one would expect possession or not to have no correlation to whether one is assaulted in the first place where the person assaulting does not have personal knowledge of the victim.
Very happy to agree to disagree. But on a point of statistical correctness I was confused by this:
If I recall correctly, the effect of the “adjustment”, if one then figures out what the raw numbers would have had to have been to yield the final odds ratios reported, are that of the 49 people in the control group who were actually gun owners, only 9 of them were admitted to the gun owner group after the “adjustment”. When forty out of forty-nine data points are being treated as if they were outliers, I think that does reduce confidence in the claims that are made on that basis.
They presumably came up with something like a Manszel-Haenzel odds ratio which is an approved way to calculate the combined odds ratio when the data are stratified by a confounder (and may give very different results from a raw aggregate odds ratio). As far as I know working out what the raw data would have been to give the same odds ratio is a meaningless exercise and doesn’t mean the changed data are being treated as outliers.
The fact that the study excludes people who weren’t shot is very suspicious to me. When NASA engineers were trying to decide to launch the Challenger, they based their decision on statistics that didn’t include O-rings that didn’t fail. After the Challenger exploded, they reviewed the statistics *with* the O-rings, and discovered that they should not have approved the launch.
Seven people died because of that mistake!
Because of this, the NASA team now includes a statistician.
But to have a major study like this, and then to make such sweeping conclusions, and *then* to leave out those who didn’t get shot, I can only conclude: these people were attempting to create a study that put guns in a bad light.
Even if the study were accurate, however, we still have one question left: Who’s life is it anyway? If we want to live, shouldn’t we do everything in our power, including gun training and being aware of our surroundings as much as possible?