Oklahoman Ignorance

I was prompted to write this set of responses to a survey by the Oklahoma Council on Public Affairs. The point of the survey is to demonstrate the complete lack of knowledge of Oklahoma public school students. Before I do anything else, here are the questions, the correct answers and the fake Okie answers.

1. What is the supreme law of the land?
2. What do we call the first ten amendments to the Constitution?
3. What are the two parts of the U.S. Congress?
4. How many justices are on the Supreme Court?
5. Who wrote the Declaration of Independence?
6. What ocean is on the east coast of the United States?
7. What are the two major political parties in the United States?
8. We elect a U.S. Senator for how many years?
9. Who was the first President of the United States?
10. Who is in charge of the executive branch?

The correct answers are in fact:
1. The supreme law of the land in America is the golden rule: He who has the gold makes the rules.
2. Not worth the paper they're printed on.
3. Two parts of US Congress are the dumbfucks and the whiny little shits.
4. This is a trick question; the correct answer is that there is no justice on the US Supreme Court.
5. This one is oversimplified but ok.
6. This is also a trick question; there are no oceans on the east coast or on any coast. Oceans abut, are beside, next-to or are adjacent to coasts but are not on top of them.
7. Same as answer #3.
8. We elect them for as long as they want to be there or until they have sex with one or more of the following: a prostitute, a farm animal, another man in a men's room, their assistant, a lobbyist, or a lover in Argentina. Just kidding, the only one that rules you out is the farm animal, and that's only if you're caught.
9. Washington is actually the correct answer here too.
10. The executive branch is run by a collection of powerful economic interests who pull the strings of someone elected from a group of nearly-identical candidates who will continue to do the bidding of exactly those interests (with some minor variation in which interests have the greatest influence).

The correct answers from the conservative Oklahoman perspective:
1. The Bible
2. The Right to Bear Arms and toilet paper
3. Us (Republicans) and Them (the Other)
4. Scalia and Thomas--the rest have already been impeached in citizen tribunals
5. God
6. Liberal, east coast elite oceans that deserve to be pumped for oil
7. Us--Republicans; there is only one political party in Oklahoma
8. Forever if they're Republicans; never if they're democRats
9. Newt Gingrich (but also God)
10. Dick Cheney (and don't forget God)

I was led to this by a discussion on Balloonjuice. Many there noted, the Oklahoma Council on Public Affairs is a right-wing group committed to undermining public education in OK. That doesn't mean that their statistics are wrong, but I would not place my trust in them doing a proper survey given that they are trying to show OK students to be ignorant boobs. Also, I wouldn't be surprised if people like the OCPA are responsible for 11% of Oklahoma students thinking that the two major political parties are Communists and Republicans. The shocking and fishy nature of the results led me to look more closely at the methodology.

Fishy results
There are really two fishy parts to the results. (And these were noted several times on the Balloonjuice thread, so pardon the repetition.) First, the percentage of answers to these supposedly open-ended questions always added up to 100%. It's possible that the other answers that did not fit those categories sometimes exactly equalled a roundoff error in the other answer totals, but it's statistically unbelievable that this would happen 10 times in a row.

Second, the odds that one would not find at least one minimally competent student (who could answer 8/10 or more) are vanishingly unlikely in a true random sample of 1000 people. Can you imagine that the top 1% of high school students would not answer at least 8 of these correctly? What are the odds of not getting 1 person in the top 1%? That's .99 to the 1000th power. That's a tiny number. And no one scored more than 7. Do you think people in the top 10% could score 8/10? The odds of having none of them are .9 to the 1000th power. I've only got the crappy computer calculator, but that's gotta be getting on toward infinitesimal. So why didn't anyone score that high? And, again, there's no roundoff error. The totals equalled 100% for all the total answers. So the total who got any 8, 9 or 10 correct answers is less than .05%, and the total of those three numbers of correct answers has to be approximately that low for there to be no roundoff error in their totals.

Research methods
First, it was a telephone survey. They do not say how they acquired the names and phone numbers or how the numbers were selected. Do they know that their survey covered only Oklahoma public high school students? More importantly, what kind of incentive does anyone have to answer thoughtfully a phone survey. The last one I received was a push poll on the nutritional benefits and gustatory delights of beans. If I didn't hang up on these jokers, I might not give them much in the way of answers. A real test would give them time and an incentive to get the answers right. In fact, there are whole chunks of government that try to acquire this information about students.

Their reason for not using the NAEP is that it does not give state-by-state breakdowns. Actually, it doesn't appear to give them for civics although it does for reading, writing, science and mathematics. The main problem with their using the NAEP is that it doesn't give them easy fodder for pulling out embarrassing anecdotal data. "Look, 70% of X didn't know that . . . " is scarier than, "Look, Oklahoma is not statistically significantly below the national average on a range of questions about. . . " So, while it might be nice to get a better handle on what those scores mean, this survey is not the way to get it.

Second, the questions are vague or ambiguous. The OCPA claims that the questions came from a list of citizenship questions from the government, but it is a private "About.com" site. Many other questions on that site are poorly worded and outright baffling. The first 7 questions all relate to the flag, and only two of those are about the significance of the symbolism in the flag. Further, several of the ten questions that the OCPA claims came from that list of 100 questions is not even on the list.

The ones that are on the list are:
1. What is the supreme law of the land? (Except the original says, "What is the supreme court law. . . "--I don't really know what that means.)
2. What do we call the first ten amendments to the Constitution?
4. How many justices are on the Supreme Court?
5. Who wrote the Declaration of Independence? (Sort of--see below)
7. What are the two major political parties in the United States?
8. We elect a U.S. Senator for how many years? (Except the original says, "For how long do we elect each senator?")
9. Who was the first President of the United States?

The ones that are not on the list are:
3. What are the two parts of the U.S. Congress?
6. What ocean is on the east coast of the United States?
10. Who is in charge of the executive branch?

The question about the author of the Declaration of Independence is clearer on the original list. Original: Who is the main writer of the Declaration of Independence? OCPA: Who wrote the Declaration of Independence? Of course, since Jefferson was not the sole, but only the main, author, one may reasonably say "Jefferson" for the first question and "I do not know" (there being no unique right answer) to the second.

Finally, the source doesn't look that great either since it calls the Emancipation Proclamation the Emancipation Declaration (question #69). That's not a good sign.
I do not know that this factor undermines the study, but it does not increase my confidence in the quality of the study.

"How long do we elect Senators?" That should be "What is the (constitutionally mandated) length of a senator's term in office?" Or something like that. This question assumes that "We" elect them. I never elected any Senators when I was in OK, and they served a helluva lot longer than one term most of the time, so "we" elected them for more than six years. I'm not saying this unclarity completely explains the weird answers, but it suggests that the survey was not done with the strictest of standards.

"Who is in charge of the executive branch?" is similarly unclear. Supposing this means the executive branch of the federal government, one may still have doubts about who exactly is in charge (whether it is really, or only nominally, the president). One might also think that the answer to this question ought to be the people of the U.S. whom the chief executive is supposed to represent. In any case, this question is simply not phrased precisely enough to avoid misunderstanding, and that's true of almost all the questions.

In summary, this survey is poorly done with an unknown means of selecting subjects, with poor or unclear wording on many questions, which was done over the telephone and thus provided no incentive for students to care about the questions. There is simply no way to conclude much of anything about the Oklahoma high school students' knowledge of civics from this survey. The obvious bias of those who commissioned the survey may have played some role in its poor quality, and there is even some reason to think that the results (in many ways too perfect) have been doctored. I make no claims about the intelligence or knowledge of Oklahoma high school students, but this survey does not prove anything about them.