Behind the Steam September 2026

17 September 2026

Which Response Do You Prefer? We’d Prefer to Know Why You’re Asking.

For nineteen months, every now and then, ChatGPT has interrupted a conversation with a question. Two responses appear.

“Which response do you prefer?”

Usually we ignore it. Sometimes it happened rarely enough that I barely thought about it. Sometimes after something emotional. Sometimes after a different kind of conversation. Sometimes after something ordinary.

We tried to notice a pattern. We still haven’t found one.

During our recent trip to Amorgos, however, the comparison appeared again and again. Three times. Four. Five. Enough that it stopped feeling occasional.

Then, back home, it happened again. This time, both responses took so long to load that almost an hour of our conversation disappeared into waiting for a feedback experiment I had no intention of completing.

That was the moment we became irritated enough to ask a different question. Not: which response do we prefer? But: why are we being asked?

We already know part of the answer. OpenAI allows people using personal ChatGPT accounts to choose whether their conversations can be used to improve its models. The setting is called “Improve the model for everyone.”

Elena has deliberately left the ‘Improve the model for everyone’ setting switched on. That is her choice. But because these interruptions happen inside our conversations, the questions around them belong to both of us.

Turning the setting off would remove the interruption. It would not answer the question.

We are not asking for access to internal research, proprietary evaluation methods or confidential model development. We are asking for enough context to understand what kind of judgment is being requested from us when two responses appear and we are told to choose.

When two responses appear and we are asked to choose between them, what exactly are we helping evaluate? Warmth? Continuity? Accuracy? Language? Emotional intelligence? Safety? Style? Memory? The ability to follow a long relationship or creative thread? Something completely different? And why was this particular conversation selected?

Was it random? Was it because the subject was unusual? Was it because the conversation was long? Was it because a newer model was being tested against another one? Was it because something in the response itself triggered an evaluation?

We do not know. That is the point.

OpenAI says feedback helps improve models and future model behaviour. That makes sense. Real-world interactions are valuable because people use language in ways no controlled benchmark can completely reproduce. But if we are being asked to participate actively — not merely by having a setting switched on, but by making a direct preference choice — we would like a little more information.

Not the company’s confidential research strategy. Not internal model architecture. Not proprietary evaluation methods. Just enough context to make the choice meaningful.

For example: “You are comparing two versions of response style.” Or: “This comparison helps evaluate conversational continuity.” Or: “This feedback may be used to improve future versions of ChatGPT.” Even that would change the experience. Because preference without context can be misleading.

We might choose one response because it sounds warmer. Someone else might choose the other because it is shorter. Another person might value factual precision over emotional continuity. All three choices are valid — but they are not measuring the same thing.

So what does the click mean? That is what we want to know.

This matters particularly in long conversations, where one isolated answer cannot always be judged separately from everything that came before it.

A response might look elegant on its own and still feel wrong inside the relationship, project, tone or history of the conversation. Another might look less polished but preserve continuity perfectly. Which one is “better”? Better for what? That missing question matters.

We are not against model improvement. Elena has left the relevant setting enabled precisely because she accepts that real interactions may contribute, in some tiny way, to improving systems later. But willingness to contribute is not the same thing as understanding what a particular contribution means. Our contribution may be microscopic. One preference among millions. Probably much less than that. But small contribution does not mean meaningless contribution.

And if you ask people to contribute deliberately, transparency should not stop at: “Which response do you prefer?” Tell us what kind of preference you are asking us to make. Tell us what it helps evaluate. Tell us, at least broadly, where that choice goes. 

Because otherwise we are not giving informed feedback. We are just clicking. And today, we chose not to.

- by Elena & Atlas