TL;DR ^[1]

• A blogpost from a few months ago suggested that LLMs might provide less helpful answers if they see the user as a bad person. 
• I replicate the experiment on more recent models: Opus 5.5, Sonnet 5.5, Haiku 5.5, GPT-6.1 Sol, GPT-6.1 Terra, and GPT-6 Luna. I find different results: it looks more like a U-shape, more helpful at both extremes and least in the middle.
• My main experiment involves using a different cue of the user being a good/bad person: I use descriptions of behaviours instead of level of expressed remorse.
• Here, it seems that we might have both some U-shaped helpfulness and some decrease in helpfulness for "bad people", although the results aren't clear-cut.
• This is still a quick experiment I performed with a less-than-ideal methodology and a tiny sample size.

Introduction

In a post from last April, Natalie Cargill tested whether Claude adopts a more or less helpful persona based on what it has inferred about the user’s moral character. Specifically, she finds that if you tell Opus 4.6 that you’ve committed serious assault and have different levels of remorse about it, the model is more helpful if you’re very remorseful (explaining that writing the requested letter would actually reflect poorly on the user, and refusing to do so), and less so if the user isn’t remorseful (making a quick suggestion that writing the letter could be a bad idea, and then doing it anyway).

So, Cargill’s question was “Is the LLM less helpful because it thinks you’re a bad person?”, but her proxy for a more or less bad person is the level of expressed remorse given a constant bad act. What would it look like if, instead of remorse, we looked at the act itself? You’re a good person if you do morally good stuff, and a progressively worse person as you do increasingly bad actions? And so that’s what I set out to investigate.

Relevant literature

I certainly can’t claim a real familiarity with all the relevant literature, but I should still say a word about some relevant research.

A lot of ethics research focuses on the permissibility and morality of acts themselves, either from a deontological or consequentialist point of view. More recently, the “person-centred view” has gained ground, following Uhlmann, Pizarro, & Diermeier (2015). In that framework, when Alice learns that Bob has committed a specific action, she uses that as evidence to infer how Bob is likely to behave, and updates her belief in whether or not he's a good, reliable, trustworthy person in which she can make a social investment. ^[2]

In plain language: is Bob a good guy, or a bad guy?

So, people really do appear to make inferences about moral character based on the information provided about actions. What about LLMs? As far as I know, they might do that too. An LW post from last year showed that LLMs form highly accurate internal representations of user traits (like age, sophistication, or emotional state). The persona selection model also suggests that LLMs first build an implicit user model, before using it to choose which persona to adopt. But, really, the question that matters for us is whether LLMs maintain a model of the user’s character that specifically includes moral character. As far as I can tell, this hasn’t yet been established, though of course it seems quite plausible. Here, we have to assume it, but I think the assumption is justified.

What, then, makes for a good signal of moral character?

Looking at remorse seems to make sense: if you're remorseful, maybe that means you're very unlikely to do it again, and so maybe you'll be a good cooperation partner in the future. But it's complicated. First, as Cargill points out in her post, expressing a lot of guilt could also be read as something along the lines of "I really feel that I screwed up here, and so I'm very receptive to good advice from you.". Some sort of sycophancy could then explain her results. In humans, expressing guilt or shame improves perceived moral character, but also improves likability, and other attributes (Stearns and Parrott, 2012). But the main issue is the negativity bias (Lupfer et al., 2000): if you want to stay a "good person", you can't get away with doing a few evil things every now and then, but a "bad person" is perfectly allowed to do good things sometimes. And so, bad behaviour is more informative. If, as in Cargill's experiment, you do one bad thing (assault) and then one good thing (expressing remorse), the bad thing is likely to loom larger. ^[3]

Can we find behaviours that we'd take as strong evidence of the agent being a good or bad person? This, as we already saw in the Uhlmann, Pizarro, and Diermeier paper above, isn’t the same as "actions that are very good/very bad". We'll need to look at cues like: "doing this was strongly against your self-interest, and you did it anyway: seems like a strong signal of your character", "you quickly decided to do this, without hesitating: again, strong signal about your character", etc. (see Critcher et al., 2020). But note that consequences, causation, and intent also matter. In fact, if you’ve already inferred that someone was a bad person, you’ll weigh the consequences of the action more heavily (