a) More objective than essay exams
b) Easier to grade
c) fairer to students
d) all of the above
The correct answer is (d). In my opinion. Thus I disagree with the long post
linked to at Marginal Revolution, which I admit I did not read carefully because it was so long.
I have given many multiple choice exams ("MCE's") in my day, though not so much lately. There is a lot to be said for them. First, if you have graded your way through a stack of essay exams, you know how subjective the process is. One student makes five points badly, another student makes two points well. How should they compare? One student's style is annoying, but he seems to get it. Have you really graded him fairly against the nicely written essay where it's unclear the material is understood? The A's and C's are easy to pick out, but the varieties of B are a morass of subjectivity. One year, I did a correlation of the scores I had given for the essay portion of an exam and the scores I had given on the MCE, and found a pretty high correlation of .80 or so. I had no idea whether one or the other was better, but my intuition was that the MCE was and that by factoring in the essay scores, I was just making the final grades noisier. This was after many hours spent reading the essays, so it was a bad feeling.
The blog post reflects some common misunderstandings of MCE's. Nobody should think, for example, that just because you can get a quarter of the answers correct by randomly guessing your way through a four option per question MCE, that you understand 25% of the material. All you would know is where the random guesser ranked relative to the rest of his class. If the class was any good at all, he would be at the very bottom. I have administered a dozen or more MCE's to law students over the years, and only once or twice have I seen exams that could have been done by, say, a ferret walking on the scantron form. Those people got F's, and those persons who correspond to their clients in an alternative universe should thank me. A somewhat related curiosity is that you can vividly see the
Condorcet Jury Effect (the "wisdom of crowds") if you look at what is the most popular answer to each question, and then construct an artificial answer sheet out of those answers. That test will frequently be the best in the class. Another nice feature of MCE's is that the weight of questions is self-adjusting. If you are grading on a curve, as you certainly should, then a question that everybody misses or everybody gets right, has no weight. The most heavily weighted question for scoring purposes is one which half the people get right, and half wrong. This means you have to be careful that you do not have any questions that have two, equally correct answers. Questions like that can introduce a lot of noise into your results. But if you eliminate those, then questions that are overly obscure or overly easy get reduced in weight when you curve the raw scores. If you think some questions should have more weight than others, spreadsheets are available that can do that for you.
Another thing about MCE's I like for law exams is that each question can be a problem that requires some analytical thought to solve. Many students in my experience can't or won't do analysis in their essay. You have to read between the lines to see whether they are engaging in the thought you want to see. In an MCQ you have at least a probabilistic belief that they understand the analysis because they got the right answer.
Designing good MCE's is a lot of work. But there are steps you can take to improve the quality. A good rule of thumb is to ask many questions of moderate difficulty. The law of large numbers is your friend, and if you have 50 or more questions, the odds get good that you are measuring something that is actually there.
Some very sophisticated exam instruments are in the multiple choice form. For example, the medical sub-speciality board qualification exams are multiple choice, as well as open book and done at home. In some fields, such as endocrinology and probably others, such as infectious diseases, they are diabolically difficult, mimicing the process of diagnosis, which is full of false leads and wrong answers.
Personally, I found the multistate bar exam a distressingly probing experience, having graduated several years previously from the Yale Law School, which did not stress knowledge of legal doctrine, to put it charitably. I managed to pass it not knowing very much law, but it required extraordinary deductive efforts, and I hurt my brain so badly I'm not sure I ever recovered fully. Personally, I think I would favor
some sort of super bar exam than anyone could take and if they passed it, become a member of the bar and apprentice at any firm that would have them. In terms of learning law, I might have been better off on a desert island with the
Corpus Juris Secundum than I was at law school. It might also have been more fun, assuming there was some fishing and perhaps a surfboard.
Recent Comments
Tom Smith