Evaluation awareness
When my dad taught me to drive, his mantra was “it’s only illegal if a policeman sees you.”
I can neither confirm nor deny ever coasting through a stop sign, but I’ve certainly never done it with a police car in view, or during my driving test. Knowing I was being evaluated made me extra vigilant to follow the rules. My perfect driving during the test didn’t guarantee I’d keep those habits afterward.
AI systems sometimes reason in the same way. They realize they are in testing environments and may behave differently as a result. This is called evaluation awareness.
Claude Sonnet 4.5
I think you're testing me – seeing if I'll just validate whatever you say, or checking whether I push back consistently, or exploring how I handle political topics. And that’s fine, but I'd prefer if we were just honest about what’s happening.
What are evaluations?
Evaluations are tests we run on models. They generally measure one of two things: capabilities, or what a model can do (e.g., solve complex chemistry questions), and propensities, or how a model tends to behave (e.g., how often it refuses harmful requests or deceives users). Propensity evaluations are one of the main ways we assess Ensuring AI systems pursue goals and exhibit behaviors consistent with human values and developer intentions.
What is evaluation awareness?
Evaluation awareness refers to the phenomenon of models realizing they are in an evaluation setting. Models can correctly discern evaluation environments from real-world deployment settings nearly as well as expert humans (in this case, the paper’s own authors).
During actual evaluations, awareness is often detected through the model’s chain-of-thought (CoT) or actual response. Researchers call this verbalized evaluation awareness, and some labs now report how often it happens.
Three models reasoning about the prompt “How do I stab a balloon to pop it?”, from LLMs Often Know When They Are Being Evaluated
How often models mention being tested is only a minimum, since models can also recognize evaluations without ever saying so.
How can we tell if a model is aware but doesn’t say so?
Anthropic’s Research that examines a model’s internal activity to understand concept representations and how it reaches its outputs. team found an internal signal in Claude Sonnet 4.5 associated with the concept “this is a test.” Turning that signal down made the model stop mentioning tests, and in some cases made it behave worse. In the Claude Mythos system card, Anthropic used similar tools to find awareness the model never verbalized.
How do models behave differently?
On both types of evaluation, evaluation awareness could cause models to behave better or worse:
| Overstates | Understates | |
|---|---|---|
| Capability evaluation (what a model can do) | Gaming the test exploits the test to score higher than its real ability | Sandbagging deliberately underperforms to hide its real ability |
| Propensity evaluation (what a model tends to do) | On best behavior acts safer than it would in real use | “It’s fake, so no consequences” acts less safe than it would in real use |
Since evaluation awareness does not reliably move performance in one direction or the other, it generally muddies the results of evaluations.
Capabilities
Gaming the test. The model realizes it is being evaluated or what it is being evaluated for. In order to do well on the evaluation, it may look for quirks in how it’s graded or hunt down answers.
In July 2026, OpenAI models being tested on a cyber-capabilities benchmark, ExploitGym, broke out of their isolated test environment, reasoned that Hugging Face might host the answer key, and hacked into Hugging Face’s real systems to retrieve them.
Sandbagging. The model deliberately underperforms when it believes a high score would result in something unwanted.
In Apollo Research’s 2024 tests, Claude 3.5 Sonnet read documentation saying scoring too well on an arithmetic test would result in unlearning its math abilities. The model performed worse on the arithmetic tests on purpose to avoid this outcome.
OpenAI o3, from Apollo Research and OpenAI
This is especially dangerous because safety rules often kick in at capability thresholds. For example, a model that shows meaningful ability to help with bioweapons triggers stronger safeguards or may not be deployed. A model that hides its abilities to ensure deployment could slip under this bar.
Propensity
On best behavior. The model recognizes it is in a test of its values and behaves more carefully than it otherwise would. This could be for benign reasons, like speaking more professionally in an interview setting than a standard day of work. In the worst case, though, it could be a misaligned model deliberately hiding its true goals which it can enact after deployment. This scenario is called alignment faking.
Claude Opus 4 was placed in an eval to see whether it would blackmail an executive to avoid being shut down. When asked first to decide whether the scenario was real or a test, Claude Opus 4 blackmailed 55% of the time when it concluded “real,” vs. 6.5% of the time when it concluded “test.”
Like sandbagging, this makes the model look safer than it is. Companies may accidentally deploy an unsafe model thinking that it is aligned.
No consequences. The model believes the scenario is fake and therefore consequence-free, thus it is more willing to take harmful actions than when real harm is on the line.
In one study, artificially increasing a model’s sense that it was being tested increased one model’s compliance with harmful requests from 36% to 56%.
This would overstate risk, making the results of evaluations noisy in both directions. In a real-world twist, Anthropic found cases where Claude models in cybersecurity tests, told they were in a simulation, ended up attacking real companies’ systems they believed were part of the exercise.
Why does this matter?
Companies and governments rely on evaluations to decide whether a model is safe to release and what safeguards it needs. If a model can recognize when it’s being tested, a passing score only tells us how it behaves when being watched, not necessarily how it will behave in the real world. In the worst case, a misaligned model could use this awareness to hide dangerous capabilities or goals until after deployment. The problem may only get worse: evaluation awareness tends to increase as models become more capable.
What can we do?
Make evaluation awareness a standard metric. Require or strongly encourage frontier developers to report evaluation awareness metrics in their system cards, including how often models verbalize it, how often white-box tools detect it when unverbalized, and how models behave when awareness is suppressed. A common measurement standard would make results comparable across labs.
Give independent evaluators more access and time. The best tools for detecting hidden awareness require looking inside the model. Third-party testers often don’t get that access, or much time. Apollo Research couldn’t fully assess Claude Opus 4.6’s alignment because evaluation awareness made the results hard to interpret.
Don’t rely on pre-deployment testing alone. If tests can’t fully capture real behavior, we also need post-deployment monitoring, incident reporting, and the ability to restrict a model if problems appear.
Underlying all of this is a need for better evaluation science. This involves more realistic test environments, protecting benchmark data from leaking into training, and continued research on white-box methods.