Home > Uncategorized > Doing the Numbers, IDPA Nationals Edition

Doing the Numbers, IDPA Nationals Edition

Let me start by saying that I have a lot of respect for When The Balloon Goes Up. As a successful blog about guns, self-defense, and preparedness–or pretty much anything at all–they kick my ass. Never afraid to tilt at statistical windmills, though, I’d like to make some criticisms of their inconclusive statistical analysis of the IDPA Nationals.

First off, the lead graph in that article is probably the single worst graphical representation of data I’ve ever seen. As best I can tell it’s an effort to pack too much data in one plot, but it ends up just being an incomprehensible gibberish of colors.

As for the author’s wish to use the 2012 IDPA Nationals as a case study to show “a widening gap between CDP and the other Auto divisions as the additional reloads made more of a difference”: every single stage was, either by design or accident, pretty CDP-neutral. There was a single stage with the 18-round minimum dreaded by CDP shooters for inducing a second reload, but everyone started downloaded to three on Stage 8, eliminating any advantage SSP and ESP might have had with their 10+1 loading. In fact, of the 16 stages, 6 involved starting downloaded or reloading off the clock, eliminating magazine size (dis)advantages. Of the rest of the stages, assuming a stage plan with no extra shots, a CDP shooter would only be required to do an extra reload on Stage 2 “Saving Earl III”, requiring a second reload to engage the last plate if all other targets had been fully engaged. And given the cavernous maw of entropy swallowing up stage plans whole that Stage 2 was, a marginal extra mag change would not have a statistically significant effect on the results. To put it briefly, the stage design of this particular match can be seen through inspection to be extremely CDP-neutral. Enviable.

This is not, however, grounds for abolishing the different auto-loader divisions. There are compelling cases to be made in that debate, but the results from one extremely-well designed match isn’t one of them, unless we can also give every club stage designers like Frank Glover, Dean Brevit, and Toni Dandreamatteo.

As for the line, “Assuming the nationals required 180 rds …”: why assume? Five minutes with a calculator and your score sheets will confirm the match required 214 rounds to complete, a statistically significant discrepancy from 180 that changes the denominator used to determine all the numbers like “the average Expert would need to shave .39 seconds off each shot to be an average Master.” The actual number is some 14% lower, .33 seconds per round, for example.

As a final indictment, consider the graph of the proportional makeup of the score of a member of skill class. By simple taking the average of each, you let your analysis be victimized by the Achilles’ heel of using the mean: outliers. Of the nine DMs who shot the match, they collectively accumulated 10 procedurals, for an average of 1.1 procedural per DM. Inspecting the data, however, reveals that a full six of those were racked up by a single DM. Put another way, four DMs had none, four DMs had one, and one DM had six. This gives a median and mode of .5 procedural per DM, half the mean. And consider penalty seconds in total:

Although there are outliers on both sides, the mean of 8.3 seconds per DM is significantly greater than the median of 5. While this may seem like a small difference, my gut says if I were to run all the numbers I’d find similar discrepancies through the averages that would together stack to mask statistically significant insights.

Not having any fancy graphing software or data analysis tools, computing percentiles by hand would be unduly laborious. All the same, not wishing to merely detract from the conversation, what follows are a few statistical trends noticed by my naked eye.

Bob Vogel, SSP DC, had six stage wins, easily the most, and a naive guess would predict a similar number falling to the ESP and CDP division champions. However, of those two DCs, only one CDP DC Glenn Shelby had a stage win. This is not a slight on them, but a praise. It shows that success in IDPA–or at least this match–derives not from a string of breakout performances, but from being able to handle a wide range of stages and scenarios.

Interestingly, if you exclude Bob Vogel, the ultimate outlier, for those with stage wins, their stage wins correlate negatively with their finish. The CDP DC has one, as do the 1st Masters in CDP (Byerly), ESP (Wright), and SSP (Tate) and the ESP 2nd Master (Fuson). Butler who placed last of the three ESP DMs had two stage wins and Perry, 3rd ESP Master had 3. There is no doubt that every single shooter there named is quite skilled, but the inverse relationship between stage wins and overall victory is interesting and unexpected.

To bring everything back around, I don’t want to give the impression that I think the When The Balloon Goes Up post is without merit despite my criticisms. In particular, as an SSP Expert shooting the same Glock 17 with the same night sights since I started out three years ago, this resonates with me:

If you want to get better at this game and you have a reasonable pistol, don’t worry about your gear. Continue to shoot what you have, and get out and practice your movement and weapon manipulation skills!

I’ve mentally committed to not letting me fool myself in to thinking my gear is limiting me. Would Bob Vogel be a bad shooter with any random gun? No. Would I be a statistically significantly better shooter with a Glock 34? Probably not, but falling in to the trap of thinking you can spend your way to better performance is a slippery and costly slope that I’d rather not even set foot on. So to Master class I go with a truly stock service pistol.

The author shot the 2012 IDPA Nationals and placed 9th SSP Expert.

Categories: Uncategorized
  1. September 25, 2012 at 9:10 am

    Thanks for the response! I love replies like this for 3 reasons. 1) It shows I made you think (even if you think I’m wrong) 2) It makes me think critically about what I wrote and 3) It lets me know someone is reading the blog 😉 I actually love being wrong because it is an opportunity to learn and improve that success doesn’t provide.
    I’ll try to address each point in sequence…
    The incomprehensible gibberish of colors: The idea of that chart is 2 fold… First to show the total score for each division based on the percentile finish. To get the gist of what is happening there, the numbers don’t actually matter (all three autoloader divisions were exactly the same). The chart was also left with a metric ton of white space that would have been graphically uninteresting on the front page of the blog.
    So decided to super impose the Classification of the shooters at that rank to show that that while SS and EX do finish all over the spectrum, if you are a master you’re probably going to finish in the top half and if you are a marksman, you probably aren’t. While not perfect the classification does give you a fair indicator of expected performance.
    If you try to look at them together, or study the data, I can see how it could made your head hurt. Try looking at it like a flash sight picture. 🙂
    Now for the actual data…
    I really like analyzing data from sets like the Nationals because it is safe to assume that no-one is sand bagging or going through the motions. No one is a new shooter (Ok there was 1) that is greatly throwing off the curves. And everyone is taking it with similar enthusiasm, so shooters of equal skill are equally likely to be reloading et cetera. So this is a pretty homogenous set of data as evidenced by the nice flat curve. If you did the same thing with your local match is would be an “S” curve with at least 2 noticeable kinks.
    Unfortunately, Frank is really good setting up matches and this year is largely responsible for the 3 biggest, so it would be difficult to eliminate his affect.
    That said, IDPA stages are already designed to 6 rd neutral and your telling me that a skilled MD can also make them 9 rd neutral and put on a great, challenging match. You are also ignoring the 2nd part that ESP and SSP were the same and based on the classifier, you would expect ESP to walk away from SSP. Lastly the USMC just went back to the 1911 so I’m going to assume they think it is viable, so I don’t understand giving it IDPA Affirmative Action.
    Do I expect it to go away? Um Joyce’s last name is Wilson… So no.
    You know what happens when you assume (214 vs 180 rounds): I didn’t shoot the match so I didn’t have a match book or I would have done what you suggested. I had also spend 8 hours on the post to that point and didn’t care that much. If the exact numbers would have changed the narrative, I would have figured it out, but the point that shooting faster isn’t going to get you there still holds. (Which you picked up on)
    The final indictment, ignoring outliers: I almost made the joke that the single biggest indicator of IDPA success is having the name Vogel or Miculek, but I decided to let it go.
    Using the mean is not ideal because it hide a lot of the information available in a distribution, but it has 3 distinct advantages. 1) It is accessible to everyone because everyone knows what average means (see what I did there?) 2) It allows for charts with a single point as opposed to the watercolor painting behind my 1st chart 3) It is easier to calculate.
    I think that by trying to identify what data is an outlier I would have been doing it a disservice, because at every match you are going to have outliers and I would venture to guess most are going to be of the type you describe.
    Winning an IDPA match is about high level consistent performance (Bob Vogel, and the other two DC’s). This is why you will hear people say “You can’t win the match on this stage, but you can lose it!”
    If you are in a situation where you cannot win if you shoot to your skill level, the only option you have is to push past it and hope to get lucky. This is like going up to bat and trying to hit a homerun, if you do it you called your shot, but you have increased the likelihood of striking out. This happens at every match (the way DM’s are treated it has to) and at every skill level.
    If you are a NASCAR fan it happens there too. Once a driver is out of the Chase the only thing left are race wins… they tend to crash more.
    As for you gun choice, great job. I am an ESP Expert (same thing) and I compete with the same Glock 19 I carry daily. I have been through a few sets of sights though, each time trending towards simplicity.
    This response is longer than my actual post, but I hope it sheds some light on my thoughts. Let me know if you still think I’m completely off base and comment on what you think about the idea of a slide mounted optics division. I would shoot it with an M&P45c next year!

  2. September 25, 2012 at 9:11 am

    Holy block of text batman! Sorry about that, I assumed my paragraphs would render with white space.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: