Generative AI

Generative AI refers to artificial intelligence systems that produce new content rather than simply classifying or analyzing existing data. Given a prompt, a generative model can produce text, images, audio, video, code, or synthetic data that did not exist before. The output is not retrieved from a database. It is created, token by token or pixel by pixel, based on patterns learned during training.

Large language models like GPT-4, Claude, and Gemini are the most widely deployed examples of generative AI. Diffusion models that produce photorealistic images, voice synthesis systems that clone speech from seconds of audio, and video generation tools that create footage from text descriptions all fall under the same category.

In cybersecurity, generative AI matters for two reasons that pull in opposite directions. It is being embedded in security tools to accelerate threat detection, code review, and incident response. At the same time, it is lowering the barrier for producing the content that powers the most common attacks: phishing emails, deepfakes, malicious code, and synthetic identities. Understanding the technology is a prerequisite for managing either side of that equation.

How Generative AI Works

Most generative AI systems in wide use today are built on two architectural approaches.

Transformer-based language models generate text by predicting the most likely next token given everything that came before it. They are trained on enormous text corpora, developing internal representations of language, facts, and reasoning that allow them to produce coherent output across an almost unlimited range of topics and tasks. See the Transformer entry in this glossary for a detailed treatment of the architecture and its security implications.

Diffusion models generate images, audio, and video through a different process. During training, they learn to reverse a gradual noise-corruption process applied to real data. At inference time, they start from random noise and iteratively refine it toward a coherent output guided by the input prompt. This approach produces the photorealistic synthetic images and convincing voice clones that have made deepfakes a practical security concern.

Both approaches share a key property: their outputs are probabilistic. The model does not look up the correct answer. It generates the most statistically plausible continuation of its input. This makes generative AI outputs fluent and versatile, but also unreliable as a source of ground truth without independent verification.

The Security Risk Surface of Generative AI

Generative AI introduces risks that span the applications built on it, the content it produces, and the workflows in which it is embedded. OWASP GenAI Security Project addresses the application-layer risks directly through the Top 10 for Large Language Model Applications. The content-layer risks require additional attention.

Application-Layer Risks

Prompt injection is the most critical application-layer risk OWASP GenAI Security Project GenAI Security Project GenAI Security Project identifies. Because generative models process instructions and data in a single input stream, an attacker can embed instructions in the content the model is asked to process, overriding the developer’s intended behavior. In agentic systems that retrieve external content, browse the web, or call APIs on a user’s behalf, indirect prompt injection turns every piece of retrieved content into a potential attack vector.

Insecure output handling occurs when an application treats generative AI output as trusted input to a downstream system. Text rendered as HTML without sanitization becomes an XSS vector. Output passed to a database query becomes a SQL injection vector. The generative model is not the final point of failure. The application’s assumption that its output is safe is.

Sensitive information disclosure arises when a generative model reproduces content from its training data, reveals system prompt instructions it was told to keep confidential, or surfaces data from documents it was asked to process. Models can be prompted in ways that elicit memorized training content, including sensitive records that should not have been included in the training set.

Excessive agency becomes a critical risk when generative AI systems are given tools: the ability to send email, execute code, modify files, or call external APIs. A successful prompt injection against an agent with broad permissions can cause it to take harmful real-world actions. The risk scales directly with the permissions the agent holds and inversely with the constraints placed on its action space.

Content-Layer Risks

Spear phishing at scale is now a practical threat because generative AI can produce highly personalized, grammatically correct phishing content for thousands of targets simultaneously. The indicators that once helped recipients identify phishing, awkward phrasing, generic greetings, and obvious translation errors are no longer reliable signals.

Deepfakes and synthetic media produced by generative AI are being used in business email compromise, CEO fraud, and identity verification bypass. Voice clones created from publicly available audio can impersonate executives in phone-based social engineering. Synthetic video has been used in real fraud cases to deceive employees into authorizing wire transfers.

Malicious code generation is an area where generative AI has materially lowered the skill barrier. Models can generate functional exploit code, write scripts that automate attack steps, and assist in adapting existing malware to evade detection. The same capability that accelerates legitimate development also accelerates adversarial development.

Synthetic identity creation uses generative AI to produce realistic profile photographs, fabricated document images, and coherent personal histories that can pass automated identity verification checks. This is an active concern in financial services fraud and account takeover scenarios.