Building Reading Quizzes with AI

As a former literature instructor and a current instructional designer, I’m a fan of reading quizzes. To be more specific, I’m a fan of frequent, low-stakes, objective reading quizzes. I’m a fan of them from high school through grad school. I go into the reasons why in detail in Episode 2 of The Pedagogy Toolkit podcast, so I won’t dwell on it here. I’ll just put in a plug for the book that brought me around to this way of thinking, which is Brown, Roediger, and McDaniel’s Make it Stick: The Science of Successful Learning.

As part of an online class I’m taking (“Teaching with AI” via Auburn’s Biggio Center), my assignment was to find an educationally useful way to leverage AI chatbots for building online classes. I naturally thought of reading quizzes. And then I thought it would be fun to compare the output of various AI chatbots on the same task. I decided to go with Hemingway’s short story, “Soldier’s Home,” which I used to teach in an American Lit II course. It was published in 1925 as part of his collection In Our Time. It’s finally in the public domain, if you’re in the US. It may already be there in other countries.

Here’s the prompt I fed to the AI:

Write five multiple-choice reading quiz questions for Hemingway’s short story, “Soldier’s Home.”

The Prompt I fed to the AI Chatbots

I gave it to ChatGPT first. Here’s its result, reformatted a bit to look better here:

Certainly! Here are five multiple-choice reading quiz questions for Hemingway’s short story, “Soldier’s Home”:

Question 1: What is the protagonist’s name in “Soldier’s Home”?

A) Harold Krebs
B) Henry Johnson
C) William Smith
D) Thomas Anderson

Question 2: Where is Krebs’ hometown in “Soldier’s Home”?

A) Chicago
B) New York
C) Kansas City
D) Oklahoma

Question 3: What war did Harold Krebs fight in?

A) World War II
B) Korean War
C) World War I
D) Vietnam War

Question 4: How does Krebs feel about telling the truth to people when he returns home?

A) He finds it difficult to tell the truth.
B) He enjoys sharing his war experiences.
C) He is always honest about his experiences.
D) He avoids talking about the war entirely.

Question 5: What does Krebs’ mother want him to do after returning from the war?

A) Get a job
B) Travel the world
C) Write a book about his experiences
D) Attend college

Answers:

1. A) Harold Krebs
2. D) Oklahoma
3. C) World War I
4. D) He avoids talking about the war entirely.
5. A) Get a job

Output from ChatGPT-3.5, October 3, 2023

Here is the result from Claude:

Here are five multiple choice reading quiz questions for Hemingway’s “Soldier’s Home”:

1. What war did Krebs fight in?

a) World War I 
b) World War II
c) Korean War
d) Vietnam War

2. How does Krebs feel about his hometown when he returns from war?

a) He is excited to be back
b) He finds it dull and uninteresting
c) He misses his friends from the war
d) He wishes he had never left

3. What does Krebs lie about to girls he meets in town?

a) That he was a war hero
b) That he is married
c) That he owns property
d) That he fought in famous battles

4. How does Krebs’ mother react when he returns home?

a) She is overjoyed
b) She doesn’t recognize him
c) She is upset he waited so long to return
d) She pretends he never left

5. What does Krebs wish to do but lacks the confidence to go through with?

a) Move out of his hometown
b) Start a relationship
c) Get a regular job
d) Re-enlist in the army

Output from Claude, October 3, 2023

Discussion

The first thing to notice is that ChatGPT was smart enough to include an answer key. I didn’t ask for one, but it assumed I’d want one. That’s a nice touch. ChatGPT also successfully intuited that I wanted straight-forward, objective questions about the facts of the story, carefully sidestepping questions that might involve some level of interpretation or be muddled by ambiguity.

While ChatGPT started with a good and obvious question about the protagonist’s name, it then proceeds to use that same name in subsequent questions. That’s a rookie mistake. Claude didn’t bother asking the protagonist’s name, avoiding the difficulty of not referencing it in other questions and their answers. But ChatGPT’s questions were more straight forward–which is another of my unstated requirements. Claude was mostly good on this score. But, thought I’m familiar with the story, I’d have to reread it to decide which of the answers is the best fit for Claude’s Question 2. Option A is obviously wrong, but I’m not sure, off the top of my head, which of B, C, or D are accurate. And Claude’s Question 5 is torturously worded “What does Krebs wish to do but lacks the confidence to go through with?” A human with any eloquence would have put it as “What does Krebs lack the confidence to do?” After all, if you “lack the confidence” to do something, the fact that you “wish to do” that thing is assumed.

The winner of this round is ChatGPT, but the real winners are teachers, because crafting reading quizzes can be incredibly time consuming. Both AI created workable drafts. Pulling a bit from both and doing a little human editing turns a task that can take half an hour into one that takes on a few minutes. In a real-life situation, it would take longer to implement such a reading quiz in a Learning Management System than it did to create it.

A Final Note

My appreciation of Hemingway’s story was greatly deepened by Steven Trout‘s “‘Where Do We Go From Here?’: Ernest Hemingway’s ‘Soldier’s Home’ and American Veterans of World War I” from The Hemingway Review, Volume 20, Number 1, Fall 2000. If you’re teaching it, that’s a fine bit of criticism to include.

Image Credits

The images of robots playing chess were generated by DALL-E 2 with the prompt “Create a painting of two robots playing chess.” DALL-E 2 is owned by OpenAI, the people behind ChatGPT. WordPress crops it as a featured image of the post, here’s the full version: