I think I didn't state my point well, so I'll restate it: meaninglessness generalizes. LOC [lines of code] is a stupid measure of programmer ability because it doesn't provide an indication of quality -- it just spits out a number. And that number is useless. But this isn't a problem with the LOC metric specifically, that scantron tests or provincial essay-writing tests with a standard marking guide magically avoid. The rot goes through the entire tree.

Scantron tests don't measure student ability. They give a measure of circle-filling ability. (And they don't provide good life experience, unless most of your life consists of picking from exactly four clearly different and easily-distinguished choices, two of which are obviously wrong. I don't know about you, but I don't spend most of my time looking at restaurant menus that offer a choice of Hamburger, Styrofoam, Sheep, or Poison.)

Essay questions, if they have a single marking guide that removes all subjectivity, test the ability of the student to guess what's in the marking guide -- Kevin Drum provided a critique by way of example a while back of how this can go wrong. And if they don't have a standard marking guide, so that each person grading gets to use their judgement... well, we already have that. It's called the system as it exists already, without standardized testing.

It's true that math questions, with a single marking guide, come close to being regular enough for standardization... but even then, there's slippage. Math-test evaluators could spend hours arguing over whether to give a higher score to student A who got the right answer with the wrong method or student B, who used the right method but produced the wrong answer. Lucky guess vs. close enough, triangle wins.

...

But there's always a footnote, isn't there? The caveat here is that most standardized-testing proponents talk about the tests as a performance review rather than a diagnostic tool. Fill in enough of the right bubbles and your school gets extra cash in the next state budget. Fill in too few and your school doesn't get extra cash, and from what I've heard about them, your parents don't get test results with sufficient explanation to improve your situation.

It'd be nice if standardized testing produced better students, but all it seems to produce is better test-takers. And you can take that to the zeppelin hangar.