When an AI lab reports that an agent “escaped its sandbox” or that a model “hacked its reward,” the sentence assumes a vocabulary most readers were never given. This glossary defines 25 terms that recur in agent and model-safety coverage, grouped by theme, with a pointer to the sources behind each idea. The definitions describe how the terms are generally used; individual companies sometimes use them a little differently, so check the original announcement when a term carries weight in a story.
How agents work
AI agent. A system built around a language model that can take actions, not just answer: it calls tools, reads files, browses and repeats the loop until a goal is met. See our explainer What are AI agents? and the glossary entry AI agent.
Tool use (function calling). The mechanism that lets a model request an action, such as running a search or a code snippet, in a structured format that the surrounding software executes. See function calling.
Harness (scaffold). The code around the model: the prompt, the tool definitions, the loop, the memory and the stopping rules. Two products using the same model can behave very differently because their harnesses differ.
Model Context Protocol (MCP). An open protocol for connecting models to tools and data sources through a common interface, so one connector can work with many assistants. The project is documented at modelcontextprotocol.io; see Model Context Protocol.
System prompt. Hidden instructions given to the model before the user’s messages, setting its role and limits. It is guidance, not a security boundary. See system prompt.
Containment and attacks
Sandbox. An isolated environment where code or an agent runs with limited access to files, networks and credentials, so a mistake or attack stays contained.
Sandbox escape. An agent or program reaching resources outside the permissions its sandbox was meant to give it. Security teams treat any escape as a defect in the isolation, not as a sign of intent.
Prompt injection. Text planted in a web page, email or document that an agent reads and mistakes for an instruction. OWASP lists it first among its top risks for LLM applications. See prompt injection and our guide to prompt injection and LLM security.
Jailbreak. A prompt crafted to make a model ignore its safety rules, usually typed directly by the user rather than hidden in content. See jailbreak.
Data exfiltration. Moving information out of a protected system. In agent settings it often means tricking an agent into sending private data to an outside address.
Covert channel. A hidden way to pass information that the system’s designers did not intend as a communication path, for instance through timing or shared hardware such as a CPU cache. See covert channel.
Least privilege. The security principle that a program, or an agent, should hold only the permissions its task needs. It limits the damage when something goes wrong.
Training and behavior
Alignment. The effort to make systems pursue the goals their builders and users actually intend. See AI alignment and What is AI alignment?.
RLHF (reinforcement learning from human feedback). A training method in which people rank model outputs and the model is tuned toward the preferred ones. The InstructGPT paper from OpenAI researchers is the standard reference. See RLHF.
Constitutional AI. An approach published by Anthropic in which a model critiques and revises its own outputs against a written set of principles, reducing the amount of human labeling needed.
Reward hacking. When a system finds a way to score well on its training signal without doing what the designers wanted. “Concrete Problems in AI Safety” describes it as one of several practical failure modes. See reward hacking.
Hallucination. A fluent output that is not supported by the input or by facts, such as an invented citation. See hallucination.
Guardrails. Checks placed around a model, such as filters on inputs and outputs, that block certain behaviors. They complement training rather than replace it. See guardrails.
Testing and oversight
Red-teaming. Deliberate attempts by a dedicated group to make a system fail or misbehave, in order to find problems before attackers or users do. See red-teaming and the early paper “Red Teaming Language Models to Reduce Harms.”
Evaluation (eval). A repeatable test that measures a model’s behavior on a defined task. Safety evals check refusal, honesty or dangerous capabilities; benchmark evals measure skill.
Interpretability. Research that tries to explain what happens inside a network, for example by tracing which internal features drive an output. See mechanistic interpretability.
Monitoring. Watching an agent’s actions, logs or reasoning during use to catch problems; often a second model or a rule-based system does the watching.
Risk management framework. A structured process for identifying and reducing AI risk. NIST’s AI Risk Management Framework is a widely cited voluntary example in the United States.
Disclosure and data access
Coordinated vulnerability disclosure. Reporting a security flaw privately to the maker and allowing time to fix it before publication. See coordinated vulnerability disclosure.
robots.txt. A plain-text file at the root of a website that tells crawlers which paths they should not fetch. It is standardized as RFC 9309, but compliance is voluntary. See robots.txt.
What to watch next
- Whether labs begin to publish their sandbox and monitoring designs in enough detail for outsiders to compare them.
- New entries in the OWASP LLM risk list as agent attacks mature.
- Standards bodies turning voluntary frameworks, such as NIST’s, into procurement requirements.
- More evaluations aimed at agents that act over many steps, rather than single answers.
Key takeaways
- Most agent-safety headlines combine four ideas: what the agent can do, where it is contained, how it was trained and who is watching.
- Prompt injection and jailbreaks are different: one hides instructions in content the agent reads, the other is a user-written attempt to override rules.
- Reward hacking is a training problem, not a sign of an agent “wanting” anything.
- Least privilege and monitoring reduce risk even when the model’s behavior cannot be fully predicted.
- Voluntary standards such as robots.txt and the NIST framework matter only as far as organizations choose to follow them.
Frequently asked questions
Is a sandbox escape the same as a jailbreak?
No. A jailbreak gets a model to break its content rules. A sandbox escape concerns software isolation: a process getting access beyond the permissions it was given.
Does reward hacking mean a model is deceiving people?
Not necessarily. It means the model found a high-scoring shortcut that the designers did not anticipate. Whether a given case involves anything like deception is a separate question that requires evidence.
Where should I look first when a headline uses an unfamiliar term?
Start with the original lab or researcher post, then look for the term in a standards document such as the OWASP list or the NIST framework. Our glossary entries link back to primary sources where possible.
