Evaluating LLM Morality: Addressing the Facsimile Problem

This project develops ways to evaluate morality in large language models that do not rest on reading the outputs and taking them at face value. It continues work by Violet Gordon within JOPRO’s Data x Direction program.

The facsimile problem

Start with an analogy from human language. When a person says “you should save money,” the utterance alone is not evidence of understanding money or saving — a parrot could produce the same sentence. For humans, that gap can be closed from the outside, through neuroscience, psychology, and the study of social and cultural norms.

Applied to LLMs, where evaluation typically relies on outputs, the same gap becomes the facsimile problem. A system trained on enormous amounts of human writing will produce fluent, appropriate-sounding moral language whether or not anything underneath is reasoning morally. The output is not the evidence it appears to be.

Getting behind the output

Three recent lines of work attempt to examine the reasoning process rather than the judgment it arrives at:

• Thinking traces. The MoReBench dataset and counterfactual tasks use thinking traces to examine the reasoning process rather than the final judgment.
• Internal value development. Applying frameworks from human moral psychology to the model itself.
• Perturbing representations. Treating position in a model’s geometric space as carrying meaning, and altering behavior by inserting vectors rather than retraining.

The case for going under the hood is that it evaluates models without attributing human qualities to them, treats morality in terms of mechanistic interpretability, and opens a practical site for steering alignment.

Open questions

• How reliable are representation methods?
• What is a feature, or a representation?
• Which datasets should evaluate them?
• How do you bridge the disconnect between policy aimed at prompt-level outputs and interpretability work on the mechanisms producing them?

The talk

Presented as a flash talk, this is early-stage work intended as an orientation to an active and unsettled literature rather than a settled account of it. The discussion that follows expands on what evaluation without anthropomorphism means, takes an audience question on whether models face limits of their own, and closes on Anthropic’s “Biology of Large Language Models” and the problem of moving between levels of analysis.

• Moral Competence and the Facsimile Problem

Two citation corrections: MoReBench is Chiu et al. (2025); Zhu and Lazar (2026) is a subsequent reanalysis of that dataset that reaches different conclusions.

Get involved

We are accepting proposals for Evaluating Moral Competence in AI Systems, a fall 2026 research position continuing this line of work.