Guardrails
Checks on the way in, checks on the way out, and a gate before anything acts: how an AI feature is kept inside its rules.
Read first:Prompt injectionTool calling
Step 1 of 7· Illustrative requests, classifier scores and latencies; the gate layout and the risk names follow the cited sources
A guardrail is a check that sits around the model rather than inside it. On the way in, checks look at the request: does it carry an instruction that tries to hijack the assistant, does it contain personal data that should not be sent, is it about something this assistant handles at all? On the way out, checks look at the answer: is it in the agreed format, does it keep to policy, is the tone right, are the sources cited? Before any action that writes data or moves money, a gate asks a person. No single check catches everything, so several layers work together, and each one costs a little time and sometimes blocks a legitimate request.
Why it matters for your product
Guardrails are where a product's rules become enforceable: what the assistant may talk about, what may never leave the system, and which actions need a person. Put them outside the model, in code you can version, test and read, because a prompt can be argued out of a rule and a model update can change how well it follows one. Measure each check's false-positive rate on real traffic as carefully as its catch rate, budget the latency it adds, and treat retrieved documents, tool results and pasted text as untrusted input. Then assume any single layer will fail and make sure another stands behind it.
For engineersShow the maths
precision = TP ÷ (TP + FP), recall = TP ÷ (TP + FN), false-positive rate = FP ÷ (FP + TN)
For any check, recall is the share of real attacks it catches and the false-positive rate is the share of legitimate requests it wrongly blocks. Both matter; a check tuned only for recall blocks customers.
Worked example: Of 10,000 requests, 100 are attacks. A classifier with 95% recall and a 2% false-positive rate catches 95 attacks and blocks 198 honest requests: two of every three blocks are false positives.
share of blocks that are false positives = FPR × (1 − b) ÷ (FPR × (1 − b) + recall × b)
With b the share of traffic that is an attack. When attacks are rare, even a small false-positive rate dominates what the gate actually blocks, so the review queue fills with honest customers.
Worked example: b = 1%, FPR = 2%, recall = 95%: 0.0198 ÷ (0.0198 + 0.0095) = 68%.
P(an attack passes every layer) = Π (1 − recall_i), if the layers fail independently
Stacking checks multiplies their miss rates. The catch is independence: two filters that fail on the same phrasing are closer to one filter than to two.
Worked example: Layers with 90% and 80% recall: 0.10 × 0.20 = 2% of attacks pass both, against 10% for the first alone.
added latency = Σ gates run in sequence + max of gates run in parallel
Every check that must finish before the next stage starts adds its time to the answer. Cheap checks can run side by side; a model-based judge on the output is usually the slow one.
Worked example: Injection classifier 40 ms and PII scan 15 ms in parallel, then a schema check of 2 ms and a policy judge of 350 ms in sequence: about 392 ms on top of the model call, all illustrative.
Guardrails reduce risk; they do not remove it. Classifiers such as Llama Guard, a model that labels prompts and responses against a risk taxonomy, are themselves models with false positives and blind spots, a determined attacker can rephrase until a filter misses, and rules written for one model can behave differently after an upgrade. Rails can also make an assistant so cautious that people stop using it, which is a failure of a different kind. Measure both error rates on real traffic and revisit the rules when the model or the product changes.
The words you will hear
Input guardrail
Primary sources
- 01OWASP Top 10 for LLM Applications 2025OWASP GenAI Security Project · 2025
- 02Llama Guard: LLM-based Input-Output Safeguard for Human-AI ConversationsInan et al. · 2023
- 03NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable RailsRebedea et al. · 2023
- 04Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionGreshake et al. · 2023
- 05Hidden Technical Debt in Machine Learning SystemsSculley et al. · 2015
Connected ideas
Agents and protocols
Model Context Protocol (MCP)
One standard plug between AI apps and the tools and data they use, and exactly what travels over it.
ExploreObservability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreAgents and protocols
The agent loop
How a model works through a task: think, use a tool, read the result, and go round again.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.