Archive for September, 2012

Doing the Numbers, IDPA Nationals Edition

September 25, 2012 2 comments

Let me start by saying that I have a lot of respect for When The Balloon Goes Up. As a successful blog about guns, self-defense, and preparedness–or pretty much anything at all–they kick my ass. Never afraid to tilt at statistical windmills, though, I’d like to make some criticisms of their inconclusive statistical analysis of the IDPA Nationals.

First off, the lead graph in that article is probably the single worst graphical representation of data I’ve ever seen. As best I can tell it’s an effort to pack too much data in one plot, but it ends up just being an incomprehensible gibberish of colors.

As for the author’s wish to use the 2012 IDPA Nationals as a case study to show “a widening gap between CDP and the other Auto divisions as the additional reloads made more of a difference”: every single stage was, either by design or accident, pretty CDP-neutral. There was a single stage with the 18-round minimum dreaded by CDP shooters for inducing a second reload, but everyone started downloaded to three on Stage 8, eliminating any advantage SSP and ESP might have had with their 10+1 loading. In fact, of the 16 stages, 6 involved starting downloaded or reloading off the clock, eliminating magazine size (dis)advantages. Of the rest of the stages, assuming a stage plan with no extra shots, a CDP shooter would only be required to do an extra reload on Stage 2 “Saving Earl III”, requiring a second reload to engage the last plate if all other targets had been fully engaged. And given the cavernous maw of entropy swallowing up stage plans whole that Stage 2 was, a marginal extra mag change would not have a statistically significant effect on the results. To put it briefly, the stage design of this particular match can be seen through inspection to be extremely CDP-neutral. Enviable.

This is not, however, grounds for abolishing the different auto-loader divisions. There are compelling cases to be made in that debate, but the results from one extremely-well designed match isn’t one of them, unless we can also give every club stage designers like Frank Glover, Dean Brevit, and Toni Dandreamatteo.

As for the line, “Assuming the nationals required 180 rds …”: why assume? Five minutes with a calculator and your score sheets will confirm the match required 214 rounds to complete, a statistically significant discrepancy from 180 that changes the denominator used to determine all the numbers like “the average Expert would need to shave .39 seconds off each shot to be an average Master.” The actual number is some 14% lower, .33 seconds per round, for example.

As a final indictment, consider the graph of the proportional makeup of the score of a member of skill class. By simple taking the average of each, you let your analysis be victimized by the Achilles’ heel of using the mean: outliers. Of the nine DMs who shot the match, they collectively accumulated 10 procedurals, for an average of 1.1 procedural per DM. Inspecting the data, however, reveals that a full six of those were racked up by a single DM. Put another way, four DMs had none, four DMs had one, and one DM had six. This gives a median and mode of .5 procedural per DM, half the mean. And consider penalty seconds in total:

Although there are outliers on both sides, the mean of 8.3 seconds per DM is significantly greater than the median of 5. While this may seem like a small difference, my gut says if I were to run all the numbers I’d find similar discrepancies through the averages that would together stack to mask statistically significant insights.

Not having any fancy graphing software or data analysis tools, computing percentiles by hand would be unduly laborious. All the same, not wishing to merely detract from the conversation, what follows are a few statistical trends noticed by my naked eye.

Bob Vogel, SSP DC, had six stage wins, easily the most, and a naive guess would predict a similar number falling to the ESP and CDP division champions. However, of those two DCs, only one CDP DC Glenn Shelby had a stage win. This is not a slight on them, but a praise. It shows that success in IDPA–or at least this match–derives not from a string of breakout performances, but from being able to handle a wide range of stages and scenarios.

Interestingly, if you exclude Bob Vogel, the ultimate outlier, for those with stage wins, their stage wins correlate negatively with their finish. The CDP DC has one, as do the 1st Masters in CDP (Byerly), ESP (Wright), and SSP (Tate) and the ESP 2nd Master (Fuson). Butler who placed last of the three ESP DMs had two stage wins and Perry, 3rd ESP Master had 3. There is no doubt that every single shooter there named is quite skilled, but the inverse relationship between stage wins and overall victory is interesting and unexpected.

To bring everything back around, I don’t want to give the impression that I think the When The Balloon Goes Up post is without merit despite my criticisms. In particular, as an SSP Expert shooting the same Glock 17 with the same night sights since I started out three years ago, this resonates with me:

If you want to get better at this game and you have a reasonable pistol, don’t worry about your gear. Continue to shoot what you have, and get out and practice your movement and weapon manipulation skills!

I’ve mentally committed to not letting me fool myself in to thinking my gear is limiting me. Would Bob Vogel be a bad shooter with any random gun? No. Would I be a statistically significantly better shooter with a Glock 34? Probably not, but falling in to the trap of thinking you can spend your way to better performance is a slippery and costly slope that I’d rather not even set foot on. So to Master class I go with a truly stock service pistol.

The author shot the 2012 IDPA Nationals and placed 9th SSP Expert.

Categories: Uncategorized