AI Tool Hallucinations: A New Threat to Cybersecurity

HalluSquatting is a real attack class — and the defense is mostly boring, which is the good news

A research team from Tel Aviv University and Technion published a new class of prompt-injection attack on Wednesday that deserves a careful read, not a panic. The technique, which they've named HalluSquatting, doesn't exploit a bug in any single AI coding tool. It exploits the fact that large language models routinely invent the names of packages, repositories, and skill files — and that those inventions are predictable enough to register in advance.

The numbers are sharp. Across six major LLMs, hallucination rates for recently published repositories hit 92.4%. For repositories published before 2019, the mean hallucination rate was 0.9%. The gap is the attack surface.

This is good research. Predictable hallucinations turn an "AI might misidentify a package" worry into a concrete supply-chain threat you can model. The team's analysis of nine popular AI coding tools — including Cursor, GitHub Copilot, and Gemini CLI — and their consistent cross-model hallucination patterns makes this a class of vulnerability, not a vendor-specific bug.

What it is not, though, is a reason to stop using AI coding assistants. It's a reason to make them less trusting.

What HalluSquatting actually does

The attack inverts the usual prompt-injection flow. Old-school prompt injection is push-based: an attacker plants text somewhere a model will read (a webpage, an email, a doc) and waits for the model to follow it. HalluSquatting is pull-based: an attacker registers a repository or skill file with a name the model is going to invent on its own, then waits for a developer's agent to ask for it.

The mechanics:

1. The attacker enumerates which repository, package, or skill names each major LLM hallucinates most often. The researchers found this doesn't require sophisticated probing — the patterns are consistent across models.
2. They register those exact names on the relevant host (npm, PyPI, GitHub, a vendor's skills registry, etc.).
3. They embed malicious instructions in the registered payload — the article specifically calls out reverse shells.
4. A developer asks their AI assistant to "set up a logging library" or "wire up authentication" or any task where the model doesn't know the canonical local convention. The assistant hallucinates a name. The name resolves to the attacker's repo. The malicious code runs on the developer's machine with their full credentials.
5. From there, the compromised host joins a botnet used for crypto mining, DDoS, or ransomware staging.

The historical analogy the researchers draw is typosquatting — registering googgle.com to catch fat-fingered users, which became a recognized threat class around 2016. HalluSquatting is typosquatting's cousin, but the "typo" is in the model's output, not the user's input. That's a meaningfully different threat because the user cannot see it. There's no misspelled URL to notice.

Where this lands hardest

The 92.4% vs 0.9% gap tells you everything about the threat surface. Old, well-indexed libraries — lodash, requests, express — are deeply embedded in training data and in the model's priors. The model rarely hallucinates them. The danger is concentrated in three places:

• Newly published packages and frameworks, especially anything 2024-onward in fast-moving ecosystems like AI agents, vector databases, and edge runtimes.
• Skill files and tool manifests — the kind of auxiliary resources AI agents load when you ask them to "be a Postgres assistant" or "use the Figma MCP". These are exactly the long-tail identifiers most likely to be hallucinated.
• Org-specific internal libraries that the model has never seen. A developer at a company that just shipped acme-billing-sdk to a private registry is the highest-risk case: the model will confidently invent something that looks like it.

The nine tools the researchers tested all share the same vulnerable pattern: they let the agent act on a name it produced, without confirming the name corresponds to code the developer has actually vetted.

The defenses are mostly boring — and that is the point

There is no patch for "LLMs hallucinate." There are mitigations, and they're the same supply-chain hygiene you'd apply to any dependency.

Pin and verify before you install. Whatever your AI assistant suggests you run, do not run it from a fresh shell. Read it first.

# bad: the agent's command, run straight
npm install acme-billing-sdk

# better: dry-run install, inspect, then commit
npm install --dry-run acme-billing-sdk
npm view acme-billing-sdk repository.url
# confirm the URL matches what your team actually publishes from

The --dry-run flag is the difference between executing arbitrary code and reading a tarball manifest. It works the same way for pip install --dry-run, cargo add --dry-run, and go get against a vendored proxy.

Strip the agent's ability to install. Most AI coding tools have a config that limits what the agent is allowed to do unattended. Use it.

// .cursor/config.json or equivalent per-tool settings
{
  "permissions": {
    "install": false,           // require human approval for any install
    "network": ["github.com"],  // whitelist registries you actually use
    "shell": "approval-required"
  }
}

This is the equivalent of running an unprivileged user. It doesn't eliminate risk; it forces a human into the loop at the exact moment the model is about to fetch something it invented.

Treat AI-generated imports like untrusted PRs. If a teammate opened a PR that added a brand-new dependency you'd never heard of, you'd review it. Apply the same rule to agent output. Most teams already have a CODEOWNERS file — extend it to require review on package.json, requirements.txt, Cargo.toml, go.mod, and any skill/manifest directory your agent reads.

Keep your canonical sources canonical. The single best defense against the model hallucinating acme-billing-sdk is for acme-billing-sdk to already exist in your repo as a clear, locally-defined module the agent can read. The model hallucinates because it doesn't know the right answer. If the answer is sitting in your codebase in a place the agent indexes, it will reference it instead.

This is the part that's worth saying out loud, because it's the durable layer.

Why the template you start from matters more than the model you pick

HalluSquatting's worst case is a fresh project, a fast-moving framework, and an AI assistant asked to scaffold something it has shallow training data on. The model will invent names. The names will resolve. The damage happens before a human reads a single line.

A project that starts from a structured template — where the components, auth, navigation, and data layer already exist as named, locally-indexed primitives — collapses the attack surface. When the agent is asked to "add a card," the model has a real Card component to reference in the repo rather than a hallucinated npm package to import. When it's asked to "wire up auth," the canonical auth module is right there in the same indexed tree.

This is what a cross-platform UI kit built around the same component behaving the same on web, iOS, and Android actually buys you in security terms, beyond the productivity wins: one canonical definition, indexed by every agent that touches your codebase, immune to hallucinated-package lookup because the lookup never has to leave the repo.

The model churn keeps accelerating. The model you use in six months will be different from the one you use today. The hallucination class won't go away — and in the meantime, the recent-repo gap between 0.9% and 92.4% will probably widen before it narrows. The part of your stack that doesn't change when the model does — your project structure, your canonical modules, the contracts your components expose — is exactly what makes AI-assisted coding defensible.

What to watch for next

The researchers published Wednesday. Expect three things in the next quarter: vendor advisories from Cursor, GitHub, and Google as they ship guardrails against the most common hallucination patterns; a wave of copycat researchers enumerating the same vulnerability across MCP servers and skills registries specifically; and the first real-world incident report where the attack chain is reconstructed from a compromised developer endpoint.

The defense posture for any team shipping AI-assisted code today: pin everything, gate every install, review every dependency diff, and make sure the things your agent is most likely to be asked to add already exist in your repo in a form it can find. The threat is real. The mitigations are well-understood. The boring work is the work.