Can AI Reason in Worlds That Don't Exist?

Something kept bothering me while working on LLM reasoning. These models perform impressively on logic benchmarks — until you swap out the content for something that contradicts real-world facts. Suddenly, capable models fall apart. I wanted to understand why.

What We Found

LLMs know the real world deeply. Through training, they've internalized that pigs aren't birds, that cars are vehicles, that humans are mortal. And that knowledge quietly sabotages them the moment you ask them to reason inside a counterfactual scenario.

Take this argument:

If all Pigs are Animals, and all Animals are Birds, then all Pigs are Birds.

Logically valid. But the model sees "all Pigs are Birds" and something internal screams that's wrong — and that scream overrides the logic. Across every model we tested — GPT-4o, Llama, Gemini, DeepSeek — accuracy dropped by an average of 14 percentage points on counterfactual problems compared to factually realistic ones.

This isn't a reasoning failure in the traditional sense. It's a failure of what we call belief inhibition — the inability to suppress ingrained knowledge when the task requires setting it aside. Cognitive psychologists call the same phenomenon in humans belief bias, and it's one of the most well-replicated findings in reasoning research. We found LLMs exhibit it just as reliably.

Our Fix: Flag & Reason

Inspired by how humans use metacognition to override belief bias, we designed a simple two-step intervention called Flag & Reason (FaR).

Instead of asking the model to solve a logic problem in one shot, we split it into two separate prompts:

Flag: Ask the model whether the conclusion is factually believable. "Is it true that all pigs are birds?" It says no.
Reason: Now ask the logic question. Having explicitly acknowledged the conflict, the model treats the premises as a self-contained hypothetical.

No fine-tuning. No architectural changes. FaR narrowed the performance gap from 14% down to 7% and boosted overall accuracy by 4% across all models and datasets.

One thing that surprised us: collapsing both steps into a single prompt eliminated the gains entirely. The separation isn't cosmetic — it's the mechanism. That deliberate gap forces what we call epistemic compartmentalization, a conscious boundary between what the model knows and what the task requires it to accept.

What This Means

We built a benchmark called CounterLogic to study this systematically — 1,800 examples balanced across nine logical schemas and carefully controlled for both logical validity and factual believability. What it showed us is that the problem isn't about reasoning depth or chain length. It's about that first moment of conflict between context and knowledge, and whether the model can manage it.

The practical implication is real: if you're building systems that require reasoning from premises that might conflict with common knowledge — legal analysis, scientific hypothesis testing, scenario planning — a simple belief-flagging step before the reasoning prompt is worth adding.

We think this points toward something bigger: genuinely robust AI reasoning may require models that don't just know things, but know when to set what they know aside.

References

[1] Balappanawar, I. B., Bonagiri, V. K., Joishy, A. R., Gaur, M., Thirunarayan, K., & Kumaraguru, P. (2025). If Pigs Could Fly... Can LLMs Logically Reason Through Counterfactuals?. arXiv preprint arXiv:2505.22318.