Model Monitoring
Model monitoring is the continuous observation of an AI model’s behavior in production to detect degradation, manipulation, and failure. Training tells you how a model performs under controlled conditions. Monitoring tells you how it performs against reality, after deployment, against users it was not designed for, and against attackers who were not in the test set.
A model that is not monitored is a control that cannot be verified. It may be working. It may have been manipulated. It may have drifted silently into unreliability. Without monitoring, those three states are identical from the outside.
What Model Monitoring Tracks
Input distribution monitoring watches for shifts in what the model receives. Unusual prompt patterns, anomalous token distributions, or inputs that structurally resemble known prompt-injection attempts are detectable signals that precede the model’s output and confirm the attack. Input monitoring is the earliest warning layer in the detection stack.
Output monitoring tracks what the model produces across requests over time. Sudden shifts in refusal rates, changes in output length distributions, unexpected content categories, or responses that deviate from the model’s established behavioral profile indicate that something has changed, whether due to manipulation, model drift, or environmental shifts.
Performance drift detection measures whether the model’s accuracy, confidence calibration, or task-specific metrics are degrading on recent real-world data compared to the training distribution. A model trained on last year’s threat data may perform well on last year’s benchmarks and poorly on today’s inputs. Performance drift is silent without explicit measurement.
Latency and resource anomalies surface denial-of-service conditions and adversarial inference exhaustion before they become availability incidents. Inputs crafted to maximize processing time produce statistical outliers in latency and token consumption, detectable as anomalies in resource usage patterns.
Tool call and action monitoring applies to agentic systems. Tracking which tools the model calls, how often, in what order, and with which parameters establishes a baseline for expected agent behavior. Deviations from that baseline, unusual tool combinations, unexpected external calls, and atypical action sequences are the earliest detectable signals of a compromised agent.
What is model monitoring in AI?
Model monitoring is the continuous observation of an AI model’s inputs, outputs, performance, and behavior in production. It detects degradation, manipulation, and drift that pre-deployment testing does not surface, and provides the forensic data that security incident response depends on.
How is model monitoring different from conventional application monitoring?
Conventional application monitoring detects infrastructure failures: errors, crashes, and loss of availability. Model monitoring detects behavioral failures: output drift, input anomalies, performance degradation, and agent behavior deviations. An AI system can be fully operational from an infrastructure perspective while failing behaviourally in ways that only model monitoring exposes.
What is performance drift, and why does it matter for security?
Performance drift occurs when a model’s accuracy or reliability degrades as the real-world input distribution diverges from its training distribution. In security applications, a drifting classifier may miss the threat patterns most prevalent today while continuing to detect those most common when it was trained. Drift is silent without explicit measurement against recent data.
What monitoring data is most critical for AI incident response?
Input logs, output logs, and tool call traces for agentic systems. These three together allow responders to reconstruct what the model received, what it produced, and what actions it took. Without them, investigating a suspected prompt injection or excessive agency failure depends on inference rather than evidence.
