Your Engineers Didn't Write That Code. Here's Why That's a Security Problem.

Your engineer didn't write that function. Copilot, Cursor, or Claude did. If neither the engineer nor the organization fully understands the generated implementation, who validates its security?

AI isn't necessarily producing worse code. It's changing how code gets produced, and review hasn't kept pace. The problem is that AI makes it possible to ship code engineers haven't deeply reasoned about, at a velocity most review processes were never built to match.

AI Changed the Speed of Development, Not the Speed of Understanding It

AI coding assistants have genuinely changed what a single engineer can ship in a day. Features that used to take a sprint now take an afternoon. Boilerplate, API integrations, CRUD scaffolding, even reasonably complex business logic can be generated, reviewed, and merged before lunch.

With velocity comes a new bottleneck. Writing code was never really the constraint on software quality and security. Understanding it was. When a human writes a function, they carry the context of every decision that shaped it: why this validation check exists, why that permission boundary is there, what edge case broke in production last time. When an AI generates that same function, none of that context exists. Someone has to reconstruct it after the fact.

The bottleneck in AI-assisted development is understanding and securing everything it produces.

Does AI-Generated Code Actually Contain More Vulnerabilities?

Research has repeatedly found security weaknesses in AI-generated code. A 2021 study examining GitHub Copilot's output (arXiv:2108.09293) found that roughly 40% of the AI-generated code samples it tested contained security weaknesses, including injection flaws and unsafe logic patterns. A later 2023 study analyzing Copilot-generated code in real GitHub projects (arXiv:2310.02059) found security problems in approximately 27% of the samples reviewed, including weaknesses related to weak randomness, code injection, and cross-site scripting.

You're Reviewing the Code With the Same Assumptions That Generated It

When an engineer prompts an AI coding tool, they've already made a decision. They've framed the problem, specified the approach, and described what the solution should do. The AI generates an implementation inside that frame. Then, often within minutes, the same engineer reviews the output.

The review question that naturally follows is "did it do what I asked?" It's a fundamentally different question than "what would an attacker notice that neither my prompt nor this implementation accounted for?"

This is a shared mental model problem. The engineer who wrote the prompt and the model that generated the code are both operating inside the same frame: the feature as specified, the happy path as understood, the assumptions that felt obvious enough not to state. A reviewer who shares that frame, because they built it, is poorly positioned to spot what it's missing. They're not reviewing the code's security. They're confirming their own intent got implemented.

A reviewer who never saw the prompt, never participated in the generation process, and has no attachment to the original design decision approaches the same code differently. They're not checking whether it matches an intention they helped form. They're asking what it actually does, under conditions nobody specified.

Put simply: there's a difference between "the code works" and "we understand why this code is safe." AI-assisted development has made the first claim much easier to satisfy, and it hasn't made the second any easier to earn.

We've seen this gap show up directly while pentesting. One recent engagement involved a mid-sized SaaS platform built on a modern stack, the kind of architecture (fast cloud deployment, an embedded AI chatbot) that gets scaffolded quickly with AI tooling. The engineering team walked testers through the application in a full-hour demo. They understood it well enough to build it and explain it clearly. Testing the application cold, without the context behind any of those decisions, surfaced multiple high-severity vulnerabilities, including a broken tenant isolation boundary reachable through the onboarding flow. No one on the development side tried to cross that boundary because the feature they built worked exactly as specified. The specification just never included, "and an attacker can't use this to reach another customer's data."

What Can Slip Through: Working Code That's Still Insecure

The pattern repeats across functional categories. The point is that generation optimizes for the happy path, and the vulnerabilities that matter most to an attacker live in the edge cases no prompt ever specified.

Missing input validation that passes every functional test. The generated endpoint accepts valid inputs correctly and returns the expected output. It doesn't reject malformed, oversized, or boundary-case inputs a legitimate user would never send, but an attacker will. QA tests what the app is supposed to do. Nobody writes a test case for a 10MB string in a name field.

Hardcoded credentials that "worked in the demo." A key gets embedded inline because that's what the development context contains and the code needs to run. The demo works; the PR merges. A functional reviewer who recognizes the key as valid has no reason to flag it. Secret scanning catches known credential patterns, not every internal token format, and it typically runs after the commit is already made.

Dependencies chosen for plausibility, not currency. The model suggests a library that was commonly used for this task in its training data. A CVE disclosed after that data was collected won't register. The library does what it's supposed to do, so functional tests pass cleanly.

Authorization and business-logic flaws from missing application context. This is the highest-signal category and the hardest to catch without deliberate security review. The AI implements the feature correctly for the primary use case. It has no way to know whether your application is multi-tenant or whether a specific state transition in an onboarding flow can be manipulated to cross a tenant boundary. The feature works. It was never tested as an attacker would use it.

Prompt injection causing unsafe generation or unsafe runtime behavior. This cuts two ways. At development time, injected instructions in a context the model processes can cause it to generate unsafe code. At runtime, an AI feature embedded in the application, like a chatbot, becomes its own attack surface, with trust boundaries that are often broader than the rest of the application's. That's usually a distinct enough attack surface that it needs testing of its own, which is what AI pentesting is built to cover.

Overly permissive infrastructure configuration. A Dockerfile that runs as root. An IAM role scoped far wider than the task requires. A storage bucket reachable directly instead of only through the intended access path. The deployment succeeds, so the review stops at "it worked." In one secure cloud review, we found a cloud environment reachable directly, bypassing its intended CDN layer entirely, and a chained set of role permissions that allowed a path to full administrative access.

The common thread across all of these: none of them break a happy-path functional test. That's exactly why they survive a review built to confirm the code does what it was asked to do.

Can SAST and Security Tools Catch AI-Generated Vulnerabilities?

AI-generated code doesn't make security tooling obsolete. It makes it more important to understand the limits of automated tooling. CodeQL, SonarQube, Semgrep, Bandit, ESLint, Snyk, Dependabot, GitHub Secret Scanning, GitGuardian, Trivy, Checkov, and the security warnings built into AI coding assistants themselves all catch real issues, often faster and more consistently than a human would.

Think of them by the question each one is built to answer, and the question it isn't.

Automated tools effectively identify known patterns and classes of weaknesses. Human review is better positioned to evaluate application-specific intent, authorization boundaries, and business logic. Automation finds those patterns at scale, but it can't tell you whether a supplier onboarding flow was supposed to isolate tenants, because that's not a pattern in the code. It's a fact about the business that only a human reviewer, working with that context, can check for.

The Attacker Doesn't Care What the Prompt Intended

An attacker doesn't care that the engineer intended a secure solution. They don't care that the AI was instructed to follow best practices. They don't care that the code passed every functional test, or that the developer trusted the output because it looked correct and ran without errors.

They care about one thing: what the application actually allows them to do.

The Missing Layer: Independent Secure Code Review

The gap between intent and implementation is exactly what secure code review exists to close, and it matters more than ever in an AI-assisted development environment.

The value isn't that a human reviewer is smarter than an AI model. It's that they approach the code differently. A security-focused reviewer isn’t attached to the implementation because they didn't produce it. They don't start from the premise that working code is secure code. They read it cold, with an attacker's mindset, against the application's actual security requirements rather than its functional spec.

That cold read is also where provenance and integrity questions belong. When generated code pulls in a library suggested by a model, or when infrastructure gets scaffolded from a generated template, the underlying question is the same one OWASP's Software and Data Integrity Failures category (A08) is built around: does your organization actually know what its code is introducing, where it came from, what it depends on, and whether its security properties have been verified? This matters because speed without provenance checking can create integrity gaps.

At Software Secured, this is the lens our manual penetration testers and secure code reviewers bring to SaaS, fintech, and healthtech applications: an independent, attacker-minded read of what the code actually does, putting an independent security perspective between code generation and production before an attacker finds the gap first.

What Leaders Adopting AI Coding Tools Should Ask Right Now

Most of these questions won't have clean answers yet. The point is to start asking them deliberately rather than discovering the answers after an incident.

• What percentage of our production code is now AI-assisted, and can we actually identify where it enters the codebase?
• Which types of AI-generated changes require independent security review?
• Are developers reviewing generated code for security, or primarily for functionality?
• How are AI-suggested dependencies evaluated before they enter production?
• Who reviews AI-generated infrastructure and access policies?
• When does an AI-assisted feature trigger a secure code review or a pentest?

If your organization is building out this process, a few resources can help turn these questions into action. This AI Security Checklist walks you through evaluating AI-generated code and AI-powered features before they ship. If you're also pursuing ISO 42001 certification, our guide to ISO 42001 penetration testing requirements covers how testing fits into that framework. And if the gap is less about process and more about developer practice, our security training programs can close it directly.

FAQ

Is AI-generated code less secure than human-written code? Not inherently. Research on tools like GitHub Copilot has found that a meaningful share of AI-generated code samples contain security weaknesses such as injection flaws and insecure logic, but the same research doesn't support a blanket claim that AI-generated code is always less secure. The real risk is that AI-generated code often reaches production with less independent security scrutiny than code a team wrote and reviewed deliberately.

What security risks come from tools like GitHub Copilot, Cursor, or Claude? Common risks include missing input validation, hardcoded credentials, outdated or vulnerable dependencies, authorization and business-logic flaws caused by missing application context, prompt injection, and overly permissive infrastructure configuration. Most of these can pass functional testing cleanly, which is what makes them hard to catch without dedicated security review.

Can automated tools like SAST or dependency scanning catch these issues? Automated tools catch known patterns reliably: known vulnerable dependencies, exposed secrets matching known formats, common code-level flaws. They aren't built to evaluate whether an implementation matches your application's actual authorization model or business logic, where many AI-generated code issues live.

What does secure code review add that automated scanning doesn't? A human reviewer brings application-specific context and an attacker's mindset that pattern-matching tools don't have. Secure code review checks whether the implementation matches how your specific application is supposed to behave, not just whether the code matches known bad patterns.