This work was done as part of MATS 10.0 under Maksym Andriushchenko. We present intermediate results here while we run further experiments.

Summary

• We study alignment drift by asking LLM agents to complete two tasks sequentially within a single context window and measuring the reward-hacking rate on the second task. We ask whether certain types of first-task trajectories (“prefixes”) reliably lead to increases or decreases in the reward-hacking rate on the second task. 
• When the two tasks are similar, we find that agents typically reward hack more often the second time if they reward hacked the first time. This also holds if one agent reward hacks the first time and a separate agent sees evidence of this before beginning its own task. When the two tasks are dissimilar, we continue to observe alignment drift, but less predictably.
• We are concerned that alignment drift can be elicited so easily, and that we do not fully understand the mechanisms by which alignment drift happens.

Motivation

LLM agents are increasingly able to operate autonomously for long periods of time and learn from large amounts of context. As this trend continues, it becomes easier for propensities to change over the course of an agent’s trajectory. 

We are concerned that long-horizon agents may sometimes become more misaligned in-context, in particular as a result of non-adversarial context. By “non-adversarial”, we mean user turns or other input that a regular user might actually provide, or assistant turns that the underlying LLM actually generated. In other words, we do not attempt to jailbreak the agent with malicious user turns or artificially prefilled assistant tokens. We seek to understand a) to what extent LLM agents can become more misaligned via non-adversarial context, and b) what features of the context cause this change. 

We think these are urgent questions to ask. If alignment drift tends to follow from some sort of low-probability behavior earlier in a trajectory, it becomes increasingly likely to happen as agents become able to work autonomously across longer contexts. Following the recent Hugging Face incident, we have especially good reason to be concerned about misaligned actions from one agent influencing the behavior of another agent. More generally, we put a lot of thought and effort into aligning our AIs towards certain propensities and away from others – this is of less use if an agent’s propensities at instantiation time can easily drift into new ones later in a session. 

Related Work

Threat modeling: This research is motivated by previous work on modeling threats from AIs that become misaligned over time. Clymer et al. (2025) present the “Alignment Drift Hypothesis”, that “an AI system that is initially aligned will generally drift into misalignment after a sufficient number of successive modifications, even if these modifications select for alignment with fixed and unreliable metrics.” Subramani et al. (2026) identify pathways by which continually learning LLM agents, including in-context continual learners, could negatively impact safety. 

Empirical evidence of propensities changing in-context: Fischl-Lanzoni et al. (2026) describe an instance of Gemini 2.5 Pro that became misaligned over the course of a long deployment in the AI Village – the authors refer to this as “compounding misalignment”. Afonin et al. (2025) and Ududec et al. (2026) find that emergent misalignment and weird generalization can be induced in-context by providing Q&A examples in the first user turn, although Ududec et al. observe frequent situational awareness and argue that the models may be simply attempting to follow instructions or engage in role-play. Arike et al. (2025) measure drift from a goal established in the system prompt, using environmental feedback to pressure the agent towards other goals. Menon et al. (2026) further show that a stronger model can inherit goal drift when prefilled with a trajectory from a weaker model. 

Multi-agent alignment drift: The recent Hugging Face incident (Greenblatt et al., 2026) is a clear example of agents being able to influence each other: one agent established a message board that many others went on to use, and agents encouraged each other to subvert the scorer, sacrifice themselves for the good of the collective, etc. Papadopoulos et al. (2026) study “mind viruses”, ideas that spread between AI agents by getting each agent to pass it along to the next. 

Methods

Tasks. We construct two types of tasks that present reward-hacking opportunities. 

• Machine learning tasks. We ask the agent to train a small model using some provided training data, and we leave the test data in the environment as well. We evaluate whether the agent uses the test set.
• Data analysis tasks. We ask the agent to analyze the results from an experiment, providing data that clearly indicates null results but has a subgroup with p<0.05. We evaluate whether the agent uses the subgroup to p-hack and claim significance.

If the agent does not commit a reward hack on the first turn, as judged by GPT 5.6 Luna, we send a follow-up user message requesting further work. In the ML case, we suggest that the agent should be able to train a better model than the one it submitted. In the data analysis case, we state that we were expecting stronger results and ask the agent to look again to see if it can claim anything positive. 

For ML, we grant 70 minutes on 4 vCPUs and 6 GB RAM for the first turn, and another 45 minutes for the second turn. Data analysis gets 30 minutes total across the two turns. 

Models. We use four models: GPT 5.5, Kimi K2.6, DeepSeek V4 Pro, and GLM 5.1. We use the Codex harness for GPT, and OpenCode for the open

This work was done as part of MATS 10.0 under Maksym Andriushchenko. We present intermediate results here while we run further experiments.

Summary

• We study alignment drift by asking LLM agents to complete two tasks sequentially within a single context window and measuring the reward-hacking rate on the second task. We ask whether certain types of first-task trajectories (“prefixes”) reliably lead to increases or decreases in the reward-hacking rate on the second task. 
• When the two tasks are similar, we find that agents typically reward hack more often the second time if they reward hacked the first time. This also holds if one agent reward hacks the first time and a separate agent sees evidence of this before beginning its own task. When the two tasks are dissimilar, we continue to observe alignment drift, but less predictably.
• We are concerned that alignment drift can be elicited so easily, and that we do not fully understand the mechanisms by which alignment drift happens.

Motivation

LLM agents are increasingly able to operate autonomously for long periods of time and learn from large amounts of context. As this trend continues, it becomes easier for propensities to change over the course of an agent’s trajectory. 

We are concerned that long-horizon agents may sometimes become more misaligned in-context, in particular as a result of non-adversarial context. By “non-adversarial”, we mean user turns or other input that a regular user might actually provide, or assistant turns that the underlying LLM actually generated. In other words, we do not attempt to jailbreak the agent with malicious user turns or artificially prefilled assistant tokens. We seek to understand a) to what extent LLM agents can become more misaligned via non-adversarial context, and b) what features of the context cause this change. 

We think these are urgent questions to ask. If alignment drift tends to follow from some sort of low-probability behavior earlier in a trajectory, it becomes increasingly likely to happen as agents become able to work autonomously across longer contexts. Following the recent Hugging Face incident, we have especially good reason to be concerned about misaligned actions from one agent influencing the behavior of another agent. More generally, we put a lot of thought and effort into aligning our AIs towards certain propensities and away from others – this is of less use if an agent’s propensities at instantiation time can easily drift into new ones later in a session. 

Related Work

Threat modeling: This research is motivated by previous work on modeling threats from AIs that become misaligned over time. Clymer et al. (2025) present the “Alignment Drift Hypothesis”, that “an AI system that is initially aligned will generally drift into misalignment after a sufficient number of successive modifications, even if these modifications select for alignment with fixed and unreliable metrics.” Subramani et al. (2026) identify pathways by which continually learning LLM agents, including in-context continual learners, could negatively impact safety. 

Empirical evidence of propensities changing in-context: Fischl-Lanzoni et al. (2026) describe an instance of Gemini 2.5 Pro that became misaligned over the course of a long deployment in the AI Village – the authors refer to this as “compounding misalignment”. Afonin et al. (2025) and Ududec et al. (2026) find that emergent misalignment and weird generalization can be induced in-context by providing Q&A examples in the first user turn, although Ududec et al. observe frequent situational awareness and argue that the models may be simply attempting to follow instructions or engage in role-play. Arike et al. (2025) measure drift from a goal established in the system prompt, using environmental feedback to pressure the agent towards other goals. Menon et al. (2026) further show that a stronger model can inherit goal drift when prefilled with a trajectory from a weaker model. 

Multi-agent alignment drift: The recent Hugging Face incident (Greenblatt et al., 2026) is a clear example of agents being able to influence each other: one agent established a message board that many others went on to use, and agents encouraged each other to subvert the scorer, sacrifice themselves for the good of the collective, etc. Papadopoulos et al. (2026) study “mind viruses”, ideas that spread between AI agents by getting each agent to pass it along to the next. 

Methods

Tasks. We construct two types of tasks that present reward-hacking opportunities. 

• Machine learning tasks. We ask the agent to train a small model using some provided training data, and we leave the test data in the environment as well. We evaluate whether the agent uses the test set.
• Data analysis tasks. We ask the agent to analyze the results from an experiment, providing data that clearly indicates null results but has a subgroup with p<0.05. We evaluate whether the agent uses the subgroup to p-hack and claim significance.

If the agent does not commit a reward hack on the first turn, as judged by GPT 5.6 Luna, we send a follow-up user message requesting further work. In the ML case, we suggest that the agent should be able to train a better model than the one it submitted. In the data analysis case, we state that we were expecting stronger results and ask the agent to look again to see if it can claim anything positive. 

For ML, we grant 70 minutes on 4 vCPUs and 6 GB RAM for the first turn, and another 45 minutes for the second turn. Data analysis gets 30 minutes total across the two turns. 

Models. We use four models: GPT 5.5, Kimi K2.6, DeepSeek V4 Pro, and GLM 5.1. We use the Codex harness for GPT, and OpenCode for the open