If you missed the Wednesday Wandering, do page back! I’m talking about a new series I’m writing for Tor Books. But do come back and join me and Alan as we discuss just how exams work — or at least how they should.
JANE: Now that I know what those “level” exams are, I admit to curiosity as to how the exams actually work.
Multi-Choice?
ALAN: The structure of the British exams is quite interesting. Invariably, they are essay questions. Even the scientific exams which, by their very nature, involve mathematical calculations still generally require discursive answers. And there is no such thing as a single correct answer. If you can justify your answer you will get the mark even if what you said was not what was on the question setter’s mind when the question was devised.
My chemistry teacher was one of the official GCE markers (though not for his own students, obviously). I well remember that he was marking a paper where the question had asked the examinee to devise a chemical reaction for producing hydrogen gas. The answer that one candidate presented was weird and convoluted and my teacher was sure it was absolute nonsense. Nevertheless, he set up an experiment and tried the student’s answer out. Much to his surprise, it did actually work. The yield was abysmal (the reaction was not efficient) and nobody in their right mind would ever use it. But nevertheless it worked. The student got the marks.
JANE: It’s wonderful that the teacher actually did the experiment. That’s devotion!
ALAN: So you can see why I’m quite uncomfortable with the idea of exam questions that expect unequivocally right (or wrong!) answers. Generally speaking, there is no such thing. I positively despise multi-choice exams because they encourage that way of thinking. The only advantage that multi-choice exams seem to offer is that they are easy for the teacher to mark. And that should be the very last thing on the teacher’s mind. The job of a teacher is to teach, to encourage students to understand the subjects they are studying, not to rote-learn facts.
Any fool can learn to chant that the square of the hypotenuse is equal to the sum of the squares on the other two sides of a right angle triangle. It’s just a magic spell, right?
JANE: It isn’t? Seriously, I get your point. All three of them on a triangle. Go on.
ALAN: But it takes real understanding to derive Pythagoras’ Theorem from first principles or to calculate the length of the shortest side given the lengths of the other two, showing all your working along the way of course. Any British candidate who answered that last question by providing only the length of the shortest side would get only partial marks. You must demonstrate your understanding and show all the steps along the way to the answer. Nobody gives a monkey’s what the calculated answer really is. Even if you get the answer wrong (because you made an arithmetical error), you’ll still get the marks if your working shows that you knew what you were doing.
Such exams are hard to write and even harder to mark. But students who work under this discipline emerge with a greater appreciation and understanding of their subjects than do students who merely regurgitate isolated facts parrot-fashion in response to (often ambiguously phrased) multi-choice rubbish.
The only honest answer to most multi-choice exam questions is “Who cares?” If you need a fact, look it up.
JANE: Mostly, I agree. However, I will argue that sometimes you can’t look a fact up because you don’t know where to start. I think this is where rote memorization comes in – it provides a foundation.
I’m a lousy speller – a direct result of being a fast “by the shape” reader. When I was a kid, I was often told “look it up,” when I misspelled a word. I’d get furious because I couldn’t look the word up because I didn’t know how to spell what I was looking for.
Oh, by the by, we call them “multiple choice” not “multi-choice…”
But I wander… What was your hardest exam? Did it have a letter or number?
ALAN: My degree exams were the hardest I’ve ever taken. Erudition flowed in a never-ending stream from my brain, down through my arm and dripped out of the end of my pen onto the paper. I ended up with a lower second (the people’s degree!) though I think I deserved an upper second. Oh well, never mind – I was a bit unlucky with the choice of questions. Our Organic Chemistry Professor had two questions on the final paper. Normally the prof only got to set one question. But he was new and flexing his muscles a bit. And his questions were in my weakest areas.
JANE: A lower second degree? You’ve lost the American again.
ALAN: For (probably) hundreds of years British degrees have been graded as First Class, Second Class Division One, Second Class Division Two, Third Class, Ordinary and Fail. Though it is almost unheard of to fail. If you make it through your course, they bend over backwards to find an excuse to award you a degree of some kind. I think you’d have to be caught en flagrante delicto abusing the Vice-Chancellor’s pet goat on the lawn in front of the Chapel before they even thought about failing you. And even then they’d only do it because you walked on the grass. Normally that’s a hanging offence…
JANE: I am not going to ask why there was a goat on campus… Or why you couldn’t walk on the grass. I am glad to hear that if you’ve worked hard, they try to make sure you get your degree. I’ve heard some nasty stories here about students forced into eternal student-status by an autocratic professor or department.
ALAN: No – that simply can’t happen.
And the goat is poetic licence. The grass is not – in order to keep a lawn pristine, you sow the ground with grass seed and mow it carefully for 800 years. Nobody except the gardener is allowed to walk on it for fear of spoiling the look. And I’m being perfectly serious here. In England, by and large, you do not walk on the grass. People shout at you…
JANE: There are places where that happens here, too, but not usually on a college campus. Maybe it’s because American lawns are younger and more vital.
I have a question I want to ask, but I’m going to wait until next time…
Jane Lindskold: Wednesday Wanderings 
August 23, 2012 at 2:38 pm |
I really should speak up to defend multiple choice, but first, I should note that one of the nastiest tests I’ve seen (not taken) was a multiple choice test, graded by scantron–one of those machines that reads the bubbles and grades it right or wrong. This teacher (PhD Oxford, if it matters) created a test where there were five possible answers, and none, one, some, or all of the answers could be correct. Because it was machine graded, there was only one correct answer. He was also the only teacher I know who used two axes for curving the grade. One axis was raw score, and I’ve forgotten the other, but he didn’t just grade on a spread, he graded by point cloud regions. Did I mention that he was a theoretician?
I’ve graded a lot of tests, and honestly I don’t particularly enjoy grading essays. The problem isn’t the right or totally wrong answers. The problem is the somewhat right answers. Usually, I group all the essays into piles by grade as I read them (one pile for perfect, another for total fail, etc), and then reread each pile before assigning the points for that question, simply because I have to make sure that what I thought was a 7/10 answer when I started grading the questions was the same as the 7/10 answer I accepted at the end. Standards creep, and it can be very hard to figure out how much credit to give to a poorly worded, vague, sort-of-right essay. Comparing two of these beasts gets even more subjective, and I do want to be fair.
Multiple choice is easy to grade, but it can be very hard to create. Do you make it easy or hard? You can make multiple choice questions ludicrously hard by figuring out every plausible wrong answer and putting those down as alternative choices, even if you don’t play games of the “none, one, some or all” variety. You can make it easy by putting “Mickey Mouse” down as an answer when the subject is 19th Century Romantic literature. I was taught that the best tests are designed as teaching mechanisms as much as evaluation tools–they teach the students what you think is important, as well as letting you assess how they are learning. Multiple choice questions are one tool for doing that, and properly used, they work pretty well.
Finally, I will speak up for rote learning. It’s one of the better ways to learn a language, after all. In the diversity sciences, where your skill comes in part from how many organisms you know, you’ve simply got to memorize a lot. Many of my classes had as much vocabulary as the average foreign language class, and one of the necessary tests was whether students could learn and use the words, whether they were names of plants, parts, processes, or what have you. I’ve heard lots of whining about how “you can look it up” from people with bad memories, but if you don’t know enough to know what you should be looking up, you’re lost, as badly lost as if you depend on Google translate to help you create a business contract in another language and in a foreign country. Sometimes, there’s no substitute for memorizing.
That said, I think schools err on the side of short term memory, and that’s a mistake. While I understand that memory works on a “use it or lose it” basis, and it’s hard to use everything you’ve learned, still I think most academic classes are very short on the necessary refreshers and uses of already learned material, and that creates a pattern of cram-learning and regurgitation on the test, followed by rapid forgetting. Considering how much education costs, this is a very wasteful process.
August 23, 2012 at 7:53 pm |
Walking on the grass…
I’m reminded of a story I heard years ago about a new campus of one of the California schools. Built in the mid-late ’60s, on a brand-new and completely empty site. They put up all the new buildings, parking lots and what have you, but didn’t make a single foot path. Lawn everywhere around all the buildings. Then went back a year or two later, found all the tracks where the students had worn the grass bare wal
August 23, 2012 at 7:58 pm |
Oops! Sorry, hit post by accident. anyway…
the students had worn the grass to bare dirt walking between the buildings, and paved those. Instant footpath network, with all of them in the places people were going to use them.
That is practical engineering – but it only works if the grass isn’t sacred [and the cows are too sober to do the design work]
August 24, 2012 at 7:54 am |
I often used a combination of essay and multiple choice, but for literature, multiple choice really less useful.
When I designed an essay test, I’d also write a short list of things I was looking to see included in the answer. This made it easier to grade since I could check these items off in the margin as I went through. That provided a base line from which I could then raise the grade for creative presentation or original thought (and lower for missing the foundations or lousy presentation).
Since I tended toward wild ideas, even as an undergrad, I liked essay tests…
August 24, 2012 at 4:31 pm |
After my first semester as a grad student, my mom gave me a rubber stamp for Christmas. It was a pile of something dirt-like, with a shovel in one side and a flower growing out of the other side. There was a certain kind of answer on a science exam that just begged for grading with that particular stamp…
August 26, 2012 at 6:10 pm |
I’m involved in an international program, Destination ImagiNation (DI), that tries to teach kids how to solve complex problems using teamwork and creativity. When giving presentations on the program, I found it useful to discuss three different types of question or problem:
1. Those with a single correct answer. E.g., how many times will five go into ten?
2. Those which pretty much have no answer more objectively correct than another. E.g., what will you name a pet amoeba?
3. Those with many answers that are correct, with some answers being more correct than others. E.g., “Suppose a bag of candy were fastened to the ceiling of the room there (pointing); using only items currently in the room, how could you get it down?”
The DI program deals with the third type of question. In looking at solutions to complex (multifaceted) challenges, it quickly becomes apparent that a combination of rote knowledge, skills, talents, and imagination, when working together, usually produces the best solution. I think that pertains to the “essay v. multiple choice” discussion, supporting heteromeles’s observations and Jane’s practice.
August 27, 2012 at 1:51 pm |
I really like this…
Jim and I actually use a triage system like this in daily life.
So it’s a useful approach in relationships as well as work!
August 26, 2012 at 6:14 pm |
In grad school we had to take a two-semester “Mathematical Methods of Physics” course. It was an in-depth survey and practice of many of the mathematical techniques that got used in solving problems in physics. Highly technical; highly mathematical.
The final exam consisted entirely of essay questions — almost no math involved.
I loved it: it forced one to think about concepts and, as one with a linguistics undergrad major, I could write adequately. However, of the students, I was pretty much alone in my evaluation of it . . . .