Home > Uncategorized > Numbers games

Numbers games

I’ve always had a feeling that judged Olympic events like figure skating were subject to certain nationalist pressures that seem rampant at such events, but the Freakonomics Blog has helped put things in perspective:

In response to allegations of vote-trading and home-country bias among figure-skating judges at the 2002 Winter Olympics in Salt Lake, the International Skating Union changed judging procedures. Scores are now reported anonymously, and not all the scores are used in the final tally. Unfortunately, as Ray Fisman writes in Slate, the reforms have been less than effective: the Dartmouth economist Eric Zitzewitz finds that anonymity actually results in a higher home-country bias, as it allows judges to hide from a “scrutinizing press.”

What? Seriously?

I feel like there should be some axiom that states that if an agency claims to reform itself with measures that do nothing or make the problem worse, for whatever reason, they like the problem.

For the first reform: they’re trying to enforce or promote integrity by granting anonymity? These people really think that a lack of accountability will lead to more responsible judging? Risible.

As for the second reform: let’s analyze this numerically.

Judges A, B, C (from Country A, B, and C, respectively) judge a performer from Country D. It’s an average performance, so, on a scale of one to ten, they give ratings of 5.6, 4.3 and 4.7 respectively. Let’s see what happens when we successively drop one score from each calculation:

Country A B C Total

Scores 5.6 4.3 4.7 14.6

Drop A 4.3 4.7 9.0

Drop B 5.6 4.7 10.3

Drop C 5.6 4.3 9.9

Three scores that all hover around 10, appropriate for a so-so performer.

But now a performer from country B goes on and gives a similarly average performance. But B hasn’t won the gold in this event in 16 years and there is great pressure for them to win, or else lose face. So the judge from country B gives the average performance higher-than-deserved marks.

Their scores are as follows:

Country A B C Total

Scores 4.9 8.3 5.3 18.5

Drop A 8.3 5.3 13.6

Drop B 4.9 5.3 10.2

Drop C 4.9 8.3 13.2

Let’s look first at the case of dropping B’s score: the average performance gets a score of 10.2 out of 20. Fine. In one of three outcomes, there is “fair” judging.

But let’s then consider the other two cases: the judge from Country B has his score kept in and one of the “fair” scores dropped. This means that his unfair score is now being averaged out by one less fair score a majority of the time (2 out of 3 cases). By definition, this means that, instead of his score contributing one-third in 3 out of 3 cases, his score counts one-half in 2 out of 3 cases.

So, statistically, this is a wash: in aggregate, dropping the unbalanced score a few times evens out with it being included relatively often. On a macro level, this policy makes no significant, statistical difference whatsoever.

So given that the counter-measures introduced to increase the “fairness” in judging either a) exacerbate the problem or b) have no effect on solving the problem, that leaves one of two options. Either the International Skating Union is stupider than a 20-year-old recreational statistician, or they aren’t really interested in solving the problem in the first place.

About these ads
Categories: Uncategorized
  1. Bob
    March 1, 2010 at 11:07 am | #1

    I wouldn’t discount the possibility that they are stupider than a 20-year-old recreational statistician. But I wouldn’t be surprised if both of your hypotheses are true.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: